Publications

To understand language, comprehenders must retrieve the meaning associated with the words they perceive from memory and they must integrate retrieved word meanings into a representation of utterance meaning. During incremental comprehension, both processes are constrained by what has been understood so far and hence are expectation-based mechanisms. Psycholinguistic experiments measuring the electrical activity of the brain have provided key evidence that may elucidate how the language comprehension system organises and implements expectation-based retrieval and integration. However, the field has converged neither on a generally accepted formalisation of these processes nor on their mapping to the two most salient components of the event-related potential signal, the N400 and the P600. Retrieval-Integration theory offers a mechanistic account of the underpinnings of language comprehension and posits that retrieval is indexed by the N400 and integration is indexed by the P600. Following these core assumptions, this thesis demonstrates the expectation-based nature of language comprehension in which both retrieval (N400) and integration (P600) are influenced by expectations derived from an incrementally constructed utterance meaning representation. Critically, our results also indicate that lexical association to the preceding context modulates the N400 but not the P600, affirming the relation of the N400 to retrieval, rather than to integration. Zooming in on the role of integration, we reveal an important novel dimension to the interpretation of the P600 by demonstrating that P600 amplitude — and not N400 amplitude — is continuously related to utterance meaning plausibility. Finally, we examine the single-trial dynamics of retrieval and integration, establishing that words that are more effortful to retrieve tend to be more effortful to integrate, as evidenced by a within-trial correlation of N400 and P600 amplitude. These results are in direct opposition to traditional and more recent proposals arguing that (1) the N400 indexes integration processes, (2) integration — as indexed by the N400 — is merely “quasi-compositional”, and (3) the P600 is a reflection of conflicting interpretations generated in a multi-stream architecture. Rather, our findings indicate that (1) integration is continuously indexed by the P600, (2) integration is fully compositional, and (3) a single-stream architecture in which the N400 continuously indexes retrieval and the P600 continuously indexes integration is sufficient to account for the key ERP data. We conclude that retrieval and integration are two central mechanisms underlying language processing and that the N400 and the P600 should be considered part of the default ERP signature of utterance comprehension. Future study of expectation-based language processing should adopt a comprehension-centric view on expectancy and hence focus on integration effort, as indexed by the P600.

---

Um Sprache zu verstehen, müssen Menschen die Bedeutung einzelner Worte abrufen und sie müssen die Bedeutungen dieser Worte in eine Bedeutungsrepräsentation der Äußerung integrieren. Diese Prozesse erfolgen inkrementell: Mehr oder weniger jedes wahrgenommene Wort eines Satzes wird sofort einem Bedeutungsabrufungsprozess unterzogen und die abgerufene Wortbedeutung wird in die Äußerungsbedeutung integriert. Die inkrementelle Sprachverarbeitung ist dabei nicht allein von den wahrgenommen Informationen bestimmt sondern stark erwartungsbasiert: Das bislang Verstandene weckt Erwartungen darüber, was als nächstes kommuniziert wird. Zum Beispiel erleichtert das Verarbeiten des Teilsatzes „Gestern schärfte der Holzfäller die …“ die Bedeutungsabrufung und Bedeutungsintegration für das Wort „Axt“ (Beispiel aus Kapitel 3). Lautet der Teilsatz jedoch „Gestern aß der Holzfäller die …“ sollte keine Erleichterung für Abrufung und Integration desWortes „Axt“ gegeben sein. Zentraler Baustein hierfür ist die inkrementell erstellte Bedeutungsrepräsentation des Teilsatzes. Die Teilsatzbedeutung kann mögliche zukünftigeWortbedeutungen voraktivieren und dadurch deren Abrufung erleichtern. Ebenso kann die bislang erstellte Bedeutung der Äußerung die Integration vonWortbedeutungen in die angepasste Äußerungsbedeutung erleichtern, wenn die neuen Informationen dem Weltwissen gemäß erwartbar sind. Der Einfluss der Bedeutungsrepräsentation einer Äußerung auf Abrufung und Integration lässt sich mit dem generellen Begriff der Erwartbarkeit eines Wortes beschreiben. Diese Dissertation fußt auf der Annahme, dass das Sprachverständnis maßgeblich durch die erwartungsbasierten Prozesse der Bedeutungsabrufung und Bedeutungsintegration geprägt ist. Wenn diese beiden Prozesse tatsächlich maßgebliche Bestandteile des Sprachverständnisses sind, stellt sich die Frage, wie der kognitive Aufwand der Abrufung und der Integration gemessen werden kann. Ein vielversprechender Ansatz um zu verstehen, wie Menschen Bedeutung abrufen und integrieren, wäre es, die „Hardware“, welche diese kognitiven Prozesse implementiert – nämlich das menschliche Gehirn – direkt zu messen, während Versuchspersonen Sprache verarbeiten. In der Tat wurden entscheidende Erkenntnisse über das Wie und Wann des Sprachverständnisses im Gehirn durch die Messung ereigniskorrelierter Potentiale (EKP) gewonnen. EKP werden aus dem Elektroenzephalogramm (EEG) berechnet und offenbaren die auf der Kopfhaut gemessene elektrische Aktivität des Gehirns im Verlauf der Zeit nach der Präsentation eines Stimulus. In den Experimenten, welche für diese Arbeit durchgeführt wurden, werden als Stimuli einzelne Worte, welche zusammen einen Satz formen, präsentiert. Dadurch lässt sich zum Beispiel das EKP erwartbarerWorte mit jenem nicht erwartbarer Worte vergleichen („Gestern [schärfte/aß] der Holzfäller die Axt“). Unterschiede in der Erwartbarkeit eines Wortes gehen im EKP – unter anderem – mit Unterschieden in der Amplitude sogenannter EKP-Komponenten, zeitlich abgegrenzter Teile des EKPs, einher. Zwei EKP-Komponenten haben im Besonderen zu wichtigen Erkenntnissen für die Erforschung des Sprachverständnisses geführt: Die N400-Komponente, ein negativer Ausschlag des EKPs, welcher etwa 400 Millisekunden nach der Präsentation eines Stimulus seine maximale Amplitude erreicht, und die P600-Komponente, eine anhaltende, positive Abweichung des Signals, welche etwa ab 600 Millisekunden nach der Präsentation des Stimulus sichtbar wird. Seit der Entdeckung dieser EKP-Komponenten hat die elektrophysiologische Forschung die Sensitivität beider Komponenten hinsichtlich verschiedener sprachlicher sowie nicht-sprachlicher Variablen untersucht. Trotz der Vielzahl der EKP-Resultate, welche innerhalb der Sprachverarbeitungsforschung vorgelegt wurden, ist das Forschungsfeld weder bei einer allgemein anerkannten formellen Beschreibung der zum Sprachverständnis notwendigen Prozesse (z.B. Abrufung und Integration) noch zu einer unumstrittenen Zuordnung dieser Prozesse zu EKPKomponenten (z.B. N400 und P600) angelangt. Die daraus resultierende Ungewissheit behindert Fortschritte in der Beschreibung der neurokognitiven Implementation des Sprachverständnisses, was in der Konsequenz die effektive Entwicklung experimenteller Sprachstudien sowie deren eindeutige Auswertung erschwert. Zur Lösung dieses Problems können komputationale Modelle des Sprachverständnisprozesses entwickelt werden, welche, erstens, die enthaltenen Prozesse (z.B. Abrufung und Integration) mit mathematischer Genauigkeit beschreiben. Aufgrund dieser exakten Beschreibungen können dann, zweitens, explizite und überprüfbare Vorhersagen für neuronale Indikatoren (z.B. N400 und P600) getroffen werden. Die zu Anfang ausgeführte Beschreibung des Sprachverarbeitungsprozesses durch die Funktionen der Bedeutungsabrufung und der Bedeutungsintegration entspricht dem komputationalem Retrieval-Integration-Modells der Elektrophysiologie des Sprachverständnisses (Brouwer et al., 2017; Brouwer et al., 2012, kurz RI-Modell). Gemäß dem RI-Modell indiziert die Amplitude der N400 die kognitive Leistung beim Abrufen von Wortbedeutungen, wobei negativere Werte höherem Aufwand entsprechen. Die Amplitude der P600 wird als Index der kognitiven Leistung bei der Beudeutungsintegration betrachtet, wobei positivere Werte höherem Aufwand entsprechen. Das Ziel dieser Dissertation ist es, spezifische Vorhersagen des RIModells empirisch zu validieren, wobei diese mit alternativen Interpretationen der N400 und P600 sowie mit alternativen Modellen verglichen werden. Zu diesem Zwecke werden zunächst die EKP-Methode sowie wegweisende Resultate zusammengefasst (Kapitel 2). Basierend auf diesem Überblick werden die funktionalen Interpretationen der N400 und P600 sowie deren Rolle in Modellen der Elektrophysiologie der Sprachverarbeitung nachgezeichnet. Dem folgen drei Studien, welche entscheidende Hypothesen des RI-Modells empirisch untersuchen. Die erste Hypothese betrifft die zentrale Rolle, welche die erwartungsbasierte Sprachverarbeitung innerhalb des RI-Modells einnimmt: Der Aufwand sowohl von Abrufung als auch von Integration sollte stark durch die Erwartbarkeit eines Wortes moduliert werden. Neue erhobene EKP-Daten zeigen (Kapitel 3), dass unerwartete Worte tatsächlich sowohl die N400 als auch die P600 modulieren („Gestern [schärfte/aß] der Holzfäller […] die Axt“). Die gleichzeitige Modulation von N400 und P600 bedeutet jedoch, dass aufgrund dieser Daten alleine nicht entschieden werden kann, welchem Prozess – Abrufung oder Integration – die beiden EKPKomponenten entsprechen. Um dieses Problem zu lösen, wurde zusätzlich eine Manipulation der lexikalischen Assoziation vorgenommen („Gestern [schärfte/aß] der Holzfäller, [bevor er das Holz stapelte/bevor er den Film schaute], die Axt“). Der eingeschobene, assoziierte Nebensatz („bevor er das Holz stapelte“) sollte die Wortbedeutung des Zielwortes („Axt“) voraktivieren und dadurch dessen Abrufung zusätzlich erleichtern, jedoch ohne dabei Einfluss auf den Aufwand der Bedeutungsintegration zu nehmen. Die Ergebnisse zeigen, dass die Präsentation lexikalisch assoziierter Worte zu einer weiteren Reduktion der N400 führt, aber keinen Einfluss auf die P600 hat, was darauf hindeutet, dass die N400 Bedeutungsabrufung indiziert, während die P600 eindeutig der Bedeutungsintegration zuordenbar ist. Nachfolgend wurden Verhaltensstudien durchgeführt, in denen Lesezeiten gemessen wurden, welche ermitteln, wie lange Leser auf einzelnen Worten verweilen, was Aufschluss über den kognitiven Aufwand bei der Sprachverarbeitung geben kann. Diese Verhaltensdaten ähneln den Modulationsmustern der P600, was eine direkte Verbindung von Lesezeiten und der P600 mit dem Aufwand bei der Wortintegration nahelegt. Modulationen der Lesezeiten durch lexikalische Assoziation fielen kürzer und weniger reliabel aus, was es möglich erscheinen lässt, dass die etablierte Verbindung von Lesezeiten zur N400 nur korrelativ sein könnte. In der Summe stützen die erhobenen Lesezeitdaten die oben ausgeführte Interpretation der EKP Daten. Eine zentrale Vorhersage des RI-Modells ist, dass die P600-Komponente von jedem Wort innerhalb einer Äußerung erzeugt wird und dass die Amplitude der P600 kontinuierlich den Aufwand der Integration indiziert. Als Teil dieser Dissertation werden erstmals EKP-Daten, welche diese Hypothese unterstützen, präsentiert. Eine post-hoc Analyse der EKP-Daten des ersten Experiments zeigt, dass sowohl die N400 als auch die P600 bei Zielworten der Kontrollkondition, welche keiner Manipulation unterlag, graduell mit der Erwartbarkeit des Zielwortes variieren. Dies würde nahelegen, dass die P600 nicht allein durch eindeutig unplausible Sätze hervorgerufen wird, sondern tatsächlich einen kontinuierlichen Index des Integrationsaufwandes darstellt. Die zweite experimentelle Studie ist speziell der Erforschung dieser Hypothese gewidmet (Kapitel 4). In diesem Experiment wird zunächst ein Kontextparagraph präsentiert, welcher den Beginn einer kurzen Geschichte enthält: „Ein Tourist wollte seinen riesigen Koffer mit in das Flugzeug nehmen. Der Koffer war allerdings so schwer, dass die Dame am Check-in entschied, dem Touristen eine extra Gebühr zu berechnen. Daraufhin öffnete der Tourist seinen Koffer und warf einige Sachen hinaus. Somit wog der Koffer des einfallsreichen Touristen weniger als das Maximum von 30 Kilogramm.“ Diesem Kontextparagraphen folgen abschließende Sätze, in welchen das Zielwort („Tourist“) plausibel, weniger plausibel, oder implausibel ist („Dann [verabschiedete / wog / unterschrieb] die Dame den Touristen…“). Eine zuerst durchgeführte Verhaltensstudie zeigt Verlangsamungen der Lesezeit als Funktion der Plausibilität, was die erfolgreiche Manipulation der Stimuli unterstreicht. Die Ergebnisse der danach durchgeführten EKP-Studie demonstrieren eindeutig, dass die Amplitude der P600 kontinuierlich als Funktion der Plausibilität variiert. Das experimentelle Design erlaubt zudem die Interpretation der N400 als Index der Bedeutungsabrufung zu überprüfen: Die wiederholte Präsentation des Zielwortes im vorangegangenen Kontextparagraph sollte die Bedeutungsabrufung in allen drei Konditionen gleichermaßen erleichtern – unabhängig von Unterschieden in der Plausibilität. In der Tat zeigen die EKP-Daten keinerlei Modulation der N400, was also die Zuordnung dieser EKP-Komponente zum Abrufungsprozess stützt. Zusätzlich testet dieses Design die Vorhersagen einer Gruppe von alternativen Modellen des Sprachverständnisses, sogenannten Multi-Stream-Modellen. Multi- Stream-Modelle sagen eine verstärkte N400 für eine Kondition („Dann unterschrieb die Dame den Tourist“) und eine verstärkte P600 für eine andere Kondition („Dann wog die Dame den Tourist“) vorher. Dies ist abhängig davon, ob der implausible Satz eine alternative, plausible Interpretation nahelegt („Dann wog die Dame den Koffer“ anstelle von „Dann wog die Dame den Touristen“) oder nicht („Dann unterschrieb die Dame den Koffer“). Da keine der Konditionen eine verstärkte N400 hervorruft, wurde die Vorhersage der Multi-Stream-Modelle durch dieses zweite Experiment falsifiziert. Stattdessen bestätigen die Ergebnisse die Vorhersagen des Single- Stream RI-Modells und stellen starke Evidenzen für die Interpretation der P600 als kontinuierlichen Index der Bedeutungsintegration bereit. Aus der Architektur des RI-Modells und der Erkenntnis, dass sowohl Bedeutungsabrufung als auch Bedeutungsintegration stark erwartungsbasiert sind, folgt eine weitere Vorhersage: Die Amplitude der N400 (je negativer die Amplitude desto höher der Abrufungsaufwand) und die Amplitude der P600 (je positiver die Amplitude desto höher der Integrationsaufwand) müssen negativ korreliert sein. Auf Prozessebene bedeutet dies: Worte, welche mehr Bedeutungsabrufung erfordern, sollten generell auch schwieriger zu integrieren sein. Diese Vorhersage steht wiederum im Kontrast zu Multi-Stream-Modellen, welche vorhersagen, dass durch jedes Wort entweder eine Verstärkung der N400 oder der P600 produziert werden sollte. Diese unterschiedlichen Hypothesen werden in neuen statistischen Analysen zuvor erhobener EKP-Daten überprüft (Kapitel 5). Die Resultate zeigen erstmals, dass die Amplituden der N400 und der P600 auf der Ebene einzelner EEG-Signale – und nicht nur auf der Ebene von durchschnittlichen EKP – korreliert sind. Diese Ergebnisse stärken damit weiter das RI-Modell und sind schwer mit der Architektur eines Multi-Stream-Modells zu vereinbaren. Zusammengefasst zeigt diese Doktorarbeit die separierbaren Einflüsse von lexikalischer Assoziation und Erwartbarkeit auf die N400. Die P600 wird dagegen nicht durch lexikalische Assoziationen moduliert, sondern reagiert darauf, wie stark die Satzbedeutung als Funktion der Erwartbarkeit und Plausibilität angepasst werden muss. Dabei ist die P600 keine kategorische Reaktion auf implausible Stimuli, sondern stellt einen kontinuierlichen Index des Bedeutungsintegrationsaufwandes dar. Des Weiteren konnte gezeigt werden, dass graduelle Modulationen der N400 und der P600 innerhalb einzelner EEG-Signale korrelieren, was auf die Organisation der erwartungsbasierten Prozesse Abrufung und Integration in einer Single- Stream-Architektur hindeutet. Für beide experimentellen Designs wurden neben EKP-Daten auch Lesezeitdaten erhoben, welche im Kontext verständnisbasierter Erwartbarkeit eine direkte Verbindung von Lesezeiten mit der P600 nahelegen. Die Ergebnisse dieser Dissertation sind unvereinbar mit traditionellen sowie neueren Theorien, welche argumentieren, dass die N400 Aspekte der Bedeutungsintegration indiziert. Im Speziellen widersprechen die Ergebnisse mehreren Schlüsselhypothesen von Multi-Stream-Modellen, welche aussagen, dass die N400 strukturunsensible Integration indiziert, während die P600 Konflikte zwischen strukturunsensibler und struktursensibler Integration widerspiegelt. Stattdessen lassen sich die Resultate mit wesentlich weniger Annahmen durch das Single-Stream-Modell der Retrieval-Integration-Theorie erklären (siehe Diskussion in Kapitel 6). Demnach fußt das Sprachverständnis imWesentlichen auf den Mechanismen der Bedeutungsabrufung sowie der Bedeutungsintegration, welche im EKP-Signal als N400- und P600-Komponente messbar sind. Beide Komponenten werden standardmäßig durch jedes Wort einer Äußerung hervorgerufen, wobei ihre Amplituden kontinuierlich den kognitiven Aufwand der Bedeutungsabrufung (N400) sowie der Bedeutungsintegration (P600) indizieren. Basierend auf den Ergebnissen dieser Dissertation ziehe ich den Schluss, dass eine an Erkenntnissen über das Sprachverständnis interessierte Forschung der P600 zentrale Bedeutung beimessen sollte. Anhang A enthält eine theorieneutrale Abhandlung über die rERP Methode (Smith & Kutas, 2015a), einem statistischen Analyseverfahren, welches in der gesamten Dissertation zur Auswertung von EKP- und Lesezeitdaten zum Einsatz kommt. Alle Daten und sämtlicher Code, welche zur Reproduktion der Analysen und Graphiken dieser Arbeit, einschließlich des Anhangs, notwendig sind, werden im Thesis Repository bereitgestellt (https://www.github.com/caurnhammer/ AurnhammerThesis). Jedwede Studien, welche mit menschlichen Partizipanten durchgeführt wurden, erhielten eine Ethik-Zulassung durch die Deutsche Gesellschaft für Sprachwissenschaft (DGfS). Teile dieser Arbeit basieren auf Veröffentlichungen in wissenschaftlichen Journalen (Kapitel 3: Aurnhammer et al., 2021; Kapitel 4: Aurnhammer, Delogu, et al., 2023; Kapitel 5: Aurnhammer, Crocker, and Brouwer, 2023).

• https://doi.org/10.22028/D291-41500
• https://doi.org/10.22028/D291-41500

@phdthesis{aurnhammer2024thesis,
title = {Expectation-based retrieval and integration in language comprehension},
author = {Christoph Aurnhammer},
url = {https://doi.org/10.22028/D291-41500},
doi = {https://doi.org/10.22028/D291-41500},
year = {2024},
date = {2024},
school = {Saarland University},
publisher = {Saarl{\"a}ndische Universit{\"a}ts- und Landesbibliothek},
address = {Saarbruecken, Germany},
abstract = {To understand language, comprehenders must retrieve the meaning associated with the words they perceive from memory and they must integrate retrieved word meanings into a representation of utterance meaning. During incremental comprehension, both processes are constrained by what has been understood so far and hence are expectation-based mechanisms. Psycholinguistic experiments measuring the electrical activity of the brain have provided key evidence that may elucidate how the language comprehension system organises and implements expectation-based retrieval and integration. However, the field has converged neither on a generally accepted formalisation of these processes nor on their mapping to the two most salient components of the event-related potential signal, the N400 and the P600. Retrieval-Integration theory offers a mechanistic account of the underpinnings of language comprehension and posits that retrieval is indexed by the N400 and integration is indexed by the P600. Following these core assumptions, this thesis demonstrates the expectation-based nature of language comprehension in which both retrieval (N400) and integration (P600) are influenced by expectations derived from an incrementally constructed utterance meaning representation. Critically, our results also indicate that lexical association to the preceding context modulates the N400 but not the P600, affirming the relation of the N400 to retrieval, rather than to integration. Zooming in on the role of integration, we reveal an important novel dimension to the interpretation of the P600 by demonstrating that P600 amplitude — and not N400 amplitude — is continuously related to utterance meaning plausibility. Finally, we examine the single-trial dynamics of retrieval and integration, establishing that words that are more effortful to retrieve tend to be more effortful to integrate, as evidenced by a within-trial correlation of N400 and P600 amplitude. These results are in direct opposition to traditional and more recent proposals arguing that (1) the N400 indexes integration processes, (2) integration — as indexed by the N400 — is merely “quasi-compositional”, and (3) the P600 is a reflection of conflicting interpretations generated in a multi-stream architecture. Rather, our findings indicate that (1) integration is continuously indexed by the P600, (2) integration is fully compositional, and (3) a single-stream architecture in which the N400 continuously indexes retrieval and the P600 continuously indexes integration is sufficient to account for the key ERP data. We conclude that retrieval and integration are two central mechanisms underlying language processing and that the N400 and the P600 should be considered part of the default ERP signature of utterance comprehension. Future study of expectation-based language processing should adopt a comprehension-centric view on expectancy and hence focus on integration effort, as indexed by the P600.

---

Um Sprache zu verstehen, m{\"u}ssen Menschen die Bedeutung einzelner Worte abrufen und sie m{\"u}ssen die Bedeutungen dieser Worte in eine Bedeutungsrepr{\"a}sentation der {\"A}u{\ss}erung integrieren. Diese Prozesse erfolgen inkrementell: Mehr oder weniger jedes wahrgenommene Wort eines Satzes wird sofort einem Bedeutungsabrufungsprozess unterzogen und die abgerufene Wortbedeutung wird in die {\"A}u{\ss}erungsbedeutung integriert. Die inkrementelle Sprachverarbeitung ist dabei nicht allein von den wahrgenommen Informationen bestimmt sondern stark erwartungsbasiert: Das bislang Verstandene weckt Erwartungen dar{\"u}ber, was als n{\"a}chstes kommuniziert wird. Zum Beispiel erleichtert das Verarbeiten des Teilsatzes „Gestern sch{\"a}rfte der Holzf{\"a}ller die ...“ die Bedeutungsabrufung und Bedeutungsintegration f{\"u}r das Wort „Axt“ (Beispiel aus Kapitel 3). Lautet der Teilsatz jedoch „Gestern a{\ss} der Holzf{\"a}ller die ...“ sollte keine Erleichterung f{\"u}r Abrufung und Integration desWortes „Axt“ gegeben sein. Zentraler Baustein hierf{\"u}r ist die inkrementell erstellte Bedeutungsrepr{\"a}sentation des Teilsatzes. Die Teilsatzbedeutung kann m{\"o}gliche zuk{\"u}nftigeWortbedeutungen voraktivieren und dadurch deren Abrufung erleichtern. Ebenso kann die bislang erstellte Bedeutung der {\"A}u{\ss}erung die Integration vonWortbedeutungen in die angepasste {\"A}u{\ss}erungsbedeutung erleichtern, wenn die neuen Informationen dem Weltwissen gem{\"a}{\ss} erwartbar sind. Der Einfluss der Bedeutungsrepr{\"a}sentation einer {\"A}u{\ss}erung auf Abrufung und Integration l{\"a}sst sich mit dem generellen Begriff der Erwartbarkeit eines Wortes beschreiben. Diese Dissertation fu{\ss}t auf der Annahme, dass das Sprachverst{\"a}ndnis ma{\ss}geblich durch die erwartungsbasierten Prozesse der Bedeutungsabrufung und Bedeutungsintegration gepr{\"a}gt ist. Wenn diese beiden Prozesse tats{\"a}chlich ma{\ss}gebliche Bestandteile des Sprachverst{\"a}ndnisses sind, stellt sich die Frage, wie der kognitive Aufwand der Abrufung und der Integration gemessen werden kann. Ein vielversprechender Ansatz um zu verstehen, wie Menschen Bedeutung abrufen und integrieren, w{\"a}re es, die „Hardware“, welche diese kognitiven Prozesse implementiert – n{\"a}mlich das menschliche Gehirn – direkt zu messen, w{\"a}hrend Versuchspersonen Sprache verarbeiten. In der Tat wurden entscheidende Erkenntnisse {\"u}ber das Wie und Wann des Sprachverst{\"a}ndnisses im Gehirn durch die Messung ereigniskorrelierter Potentiale (EKP) gewonnen. EKP werden aus dem Elektroenzephalogramm (EEG) berechnet und offenbaren die auf der Kopfhaut gemessene elektrische Aktivit{\"a}t des Gehirns im Verlauf der Zeit nach der Pr{\"a}sentation eines Stimulus. In den Experimenten, welche f{\"u}r diese Arbeit durchgef{\"u}hrt wurden, werden als Stimuli einzelne Worte, welche zusammen einen Satz formen, pr{\"a}sentiert. Dadurch l{\"a}sst sich zum Beispiel das EKP erwartbarerWorte mit jenem nicht erwartbarer Worte vergleichen („Gestern [sch{\"a}rfte/a{\ss}] der Holzf{\"a}ller die Axt“). Unterschiede in der Erwartbarkeit eines Wortes gehen im EKP – unter anderem – mit Unterschieden in der Amplitude sogenannter EKP-Komponenten, zeitlich abgegrenzter Teile des EKPs, einher. Zwei EKP-Komponenten haben im Besonderen zu wichtigen Erkenntnissen f{\"u}r die Erforschung des Sprachverst{\"a}ndnisses gef{\"u}hrt: Die N400-Komponente, ein negativer Ausschlag des EKPs, welcher etwa 400 Millisekunden nach der Pr{\"a}sentation eines Stimulus seine maximale Amplitude erreicht, und die P600-Komponente, eine anhaltende, positive Abweichung des Signals, welche etwa ab 600 Millisekunden nach der Pr{\"a}sentation des Stimulus sichtbar wird. Seit der Entdeckung dieser EKP-Komponenten hat die elektrophysiologische Forschung die Sensitivit{\"a}t beider Komponenten hinsichtlich verschiedener sprachlicher sowie nicht-sprachlicher Variablen untersucht. Trotz der Vielzahl der EKP-Resultate, welche innerhalb der Sprachverarbeitungsforschung vorgelegt wurden, ist das Forschungsfeld weder bei einer allgemein anerkannten formellen Beschreibung der zum Sprachverst{\"a}ndnis notwendigen Prozesse (z.B. Abrufung und Integration) noch zu einer unumstrittenen Zuordnung dieser Prozesse zu EKPKomponenten (z.B. N400 und P600) angelangt. Die daraus resultierende Ungewissheit behindert Fortschritte in der Beschreibung der neurokognitiven Implementation des Sprachverst{\"a}ndnisses, was in der Konsequenz die effektive Entwicklung experimenteller Sprachstudien sowie deren eindeutige Auswertung erschwert. Zur L{\"o}sung dieses Problems k{\"o}nnen komputationale Modelle des Sprachverst{\"a}ndnisprozesses entwickelt werden, welche, erstens, die enthaltenen Prozesse (z.B. Abrufung und Integration) mit mathematischer Genauigkeit beschreiben. Aufgrund dieser exakten Beschreibungen k{\"o}nnen dann, zweitens, explizite und {\"u}berpr{\"u}fbare Vorhersagen f{\"u}r neuronale Indikatoren (z.B. N400 und P600) getroffen werden. Die zu Anfang ausgef{\"u}hrte Beschreibung des Sprachverarbeitungsprozesses durch die Funktionen der Bedeutungsabrufung und der Bedeutungsintegration entspricht dem komputationalem Retrieval-Integration-Modells der Elektrophysiologie des Sprachverst{\"a}ndnisses (Brouwer et al., 2017; Brouwer et al., 2012, kurz RI-Modell). Gem{\"a}{\ss} dem RI-Modell indiziert die Amplitude der N400 die kognitive Leistung beim Abrufen von Wortbedeutungen, wobei negativere Werte h{\"o}herem Aufwand entsprechen. Die Amplitude der P600 wird als Index der kognitiven Leistung bei der Beudeutungsintegration betrachtet, wobei positivere Werte h{\"o}herem Aufwand entsprechen. Das Ziel dieser Dissertation ist es, spezifische Vorhersagen des RIModells empirisch zu validieren, wobei diese mit alternativen Interpretationen der N400 und P600 sowie mit alternativen Modellen verglichen werden. Zu diesem Zwecke werden zun{\"a}chst die EKP-Methode sowie wegweisende Resultate zusammengefasst (Kapitel 2). Basierend auf diesem {\"U}berblick werden die funktionalen Interpretationen der N400 und P600 sowie deren Rolle in Modellen der Elektrophysiologie der Sprachverarbeitung nachgezeichnet. Dem folgen drei Studien, welche entscheidende Hypothesen des RI-Modells empirisch untersuchen. Die erste Hypothese betrifft die zentrale Rolle, welche die erwartungsbasierte Sprachverarbeitung innerhalb des RI-Modells einnimmt: Der Aufwand sowohl von Abrufung als auch von Integration sollte stark durch die Erwartbarkeit eines Wortes moduliert werden. Neue erhobene EKP-Daten zeigen (Kapitel 3), dass unerwartete Worte tats{\"a}chlich sowohl die N400 als auch die P600 modulieren („Gestern [sch{\"a}rfte/a{\ss}] der Holzf{\"a}ller [...] die Axt“). Die gleichzeitige Modulation von N400 und P600 bedeutet jedoch, dass aufgrund dieser Daten alleine nicht entschieden werden kann, welchem Prozess – Abrufung oder Integration – die beiden EKPKomponenten entsprechen. Um dieses Problem zu l{\"o}sen, wurde zus{\"a}tzlich eine Manipulation der lexikalischen Assoziation vorgenommen („Gestern [sch{\"a}rfte/a{\ss}] der Holzf{\"a}ller, [bevor er das Holz stapelte/bevor er den Film schaute], die Axt“). Der eingeschobene, assoziierte Nebensatz („bevor er das Holz stapelte“) sollte die Wortbedeutung des Zielwortes („Axt“) voraktivieren und dadurch dessen Abrufung zus{\"a}tzlich erleichtern, jedoch ohne dabei Einfluss auf den Aufwand der Bedeutungsintegration zu nehmen. Die Ergebnisse zeigen, dass die Pr{\"a}sentation lexikalisch assoziierter Worte zu einer weiteren Reduktion der N400 f{\"u}hrt, aber keinen Einfluss auf die P600 hat, was darauf hindeutet, dass die N400 Bedeutungsabrufung indiziert, w{\"a}hrend die P600 eindeutig der Bedeutungsintegration zuordenbar ist. Nachfolgend wurden Verhaltensstudien durchgef{\"u}hrt, in denen Lesezeiten gemessen wurden, welche ermitteln, wie lange Leser auf einzelnen Worten verweilen, was Aufschluss {\"u}ber den kognitiven Aufwand bei der Sprachverarbeitung geben kann. Diese Verhaltensdaten {\"a}hneln den Modulationsmustern der P600, was eine direkte Verbindung von Lesezeiten und der P600 mit dem Aufwand bei der Wortintegration nahelegt. Modulationen der Lesezeiten durch lexikalische Assoziation fielen k{\"u}rzer und weniger reliabel aus, was es m{\"o}glich erscheinen l{\"a}sst, dass die etablierte Verbindung von Lesezeiten zur N400 nur korrelativ sein k{\"o}nnte. In der Summe st{\"u}tzen die erhobenen Lesezeitdaten die oben ausgef{\"u}hrte Interpretation der EKP Daten. Eine zentrale Vorhersage des RI-Modells ist, dass die P600-Komponente von jedem Wort innerhalb einer {\"A}u{\ss}erung erzeugt wird und dass die Amplitude der P600 kontinuierlich den Aufwand der Integration indiziert. Als Teil dieser Dissertation werden erstmals EKP-Daten, welche diese Hypothese unterst{\"u}tzen, pr{\"a}sentiert. Eine post-hoc Analyse der EKP-Daten des ersten Experiments zeigt, dass sowohl die N400 als auch die P600 bei Zielworten der Kontrollkondition, welche keiner Manipulation unterlag, graduell mit der Erwartbarkeit des Zielwortes variieren. Dies w{\"u}rde nahelegen, dass die P600 nicht allein durch eindeutig unplausible S{\"a}tze hervorgerufen wird, sondern tats{\"a}chlich einen kontinuierlichen Index des Integrationsaufwandes darstellt. Die zweite experimentelle Studie ist speziell der Erforschung dieser Hypothese gewidmet (Kapitel 4). In diesem Experiment wird zun{\"a}chst ein Kontextparagraph pr{\"a}sentiert, welcher den Beginn einer kurzen Geschichte enth{\"a}lt: „Ein Tourist wollte seinen riesigen Koffer mit in das Flugzeug nehmen. Der Koffer war allerdings so schwer, dass die Dame am Check-in entschied, dem Touristen eine extra Geb{\"u}hr zu berechnen. Daraufhin {\"o}ffnete der Tourist seinen Koffer und warf einige Sachen hinaus. Somit wog der Koffer des einfallsreichen Touristen weniger als das Maximum von 30 Kilogramm.“ Diesem Kontextparagraphen folgen abschlie{\ss}ende S{\"a}tze, in welchen das Zielwort („Tourist“) plausibel, weniger plausibel, oder implausibel ist („Dann [verabschiedete / wog / unterschrieb] die Dame den Touristen...“). Eine zuerst durchgef{\"u}hrte Verhaltensstudie zeigt Verlangsamungen der Lesezeit als Funktion der Plausibilit{\"a}t, was die erfolgreiche Manipulation der Stimuli unterstreicht. Die Ergebnisse der danach durchgef{\"u}hrten EKP-Studie demonstrieren eindeutig, dass die Amplitude der P600 kontinuierlich als Funktion der Plausibilit{\"a}t variiert. Das experimentelle Design erlaubt zudem die Interpretation der N400 als Index der Bedeutungsabrufung zu {\"u}berpr{\"u}fen: Die wiederholte Pr{\"a}sentation des Zielwortes im vorangegangenen Kontextparagraph sollte die Bedeutungsabrufung in allen drei Konditionen gleicherma{\ss}en erleichtern - unabh{\"a}ngig von Unterschieden in der Plausibilit{\"a}t. In der Tat zeigen die EKP-Daten keinerlei Modulation der N400, was also die Zuordnung dieser EKP-Komponente zum Abrufungsprozess st{\"u}tzt. Zus{\"a}tzlich testet dieses Design die Vorhersagen einer Gruppe von alternativen Modellen des Sprachverst{\"a}ndnisses, sogenannten Multi-Stream-Modellen. Multi- Stream-Modelle sagen eine verst{\"a}rkte N400 f{\"u}r eine Kondition („Dann unterschrieb die Dame den Tourist“) und eine verst{\"a}rkte P600 f{\"u}r eine andere Kondition („Dann wog die Dame den Tourist“) vorher. Dies ist abh{\"a}ngig davon, ob der implausible Satz eine alternative, plausible Interpretation nahelegt („Dann wog die Dame den Koffer“ anstelle von „Dann wog die Dame den Touristen“) oder nicht („Dann unterschrieb die Dame den Koffer“). Da keine der Konditionen eine verst{\"a}rkte N400 hervorruft, wurde die Vorhersage der Multi-Stream-Modelle durch dieses zweite Experiment falsifiziert. Stattdessen best{\"a}tigen die Ergebnisse die Vorhersagen des Single- Stream RI-Modells und stellen starke Evidenzen f{\"u}r die Interpretation der P600 als kontinuierlichen Index der Bedeutungsintegration bereit. Aus der Architektur des RI-Modells und der Erkenntnis, dass sowohl Bedeutungsabrufung als auch Bedeutungsintegration stark erwartungsbasiert sind, folgt eine weitere Vorhersage: Die Amplitude der N400 (je negativer die Amplitude desto h{\"o}her der Abrufungsaufwand) und die Amplitude der P600 (je positiver die Amplitude desto h{\"o}her der Integrationsaufwand) m{\"u}ssen negativ korreliert sein. Auf Prozessebene bedeutet dies: Worte, welche mehr Bedeutungsabrufung erfordern, sollten generell auch schwieriger zu integrieren sein. Diese Vorhersage steht wiederum im Kontrast zu Multi-Stream-Modellen, welche vorhersagen, dass durch jedes Wort entweder eine Verst{\"a}rkung der N400 oder der P600 produziert werden sollte. Diese unterschiedlichen Hypothesen werden in neuen statistischen Analysen zuvor erhobener EKP-Daten {\"u}berpr{\"u}ft (Kapitel 5). Die Resultate zeigen erstmals, dass die Amplituden der N400 und der P600 auf der Ebene einzelner EEG-Signale – und nicht nur auf der Ebene von durchschnittlichen EKP – korreliert sind. Diese Ergebnisse st{\"a}rken damit weiter das RI-Modell und sind schwer mit der Architektur eines Multi-Stream-Modells zu vereinbaren. Zusammengefasst zeigt diese Doktorarbeit die separierbaren Einfl{\"u}sse von lexikalischer Assoziation und Erwartbarkeit auf die N400. Die P600 wird dagegen nicht durch lexikalische Assoziationen moduliert, sondern reagiert darauf, wie stark die Satzbedeutung als Funktion der Erwartbarkeit und Plausibilit{\"a}t angepasst werden muss. Dabei ist die P600 keine kategorische Reaktion auf implausible Stimuli, sondern stellt einen kontinuierlichen Index des Bedeutungsintegrationsaufwandes dar. Des Weiteren konnte gezeigt werden, dass graduelle Modulationen der N400 und der P600 innerhalb einzelner EEG-Signale korrelieren, was auf die Organisation der erwartungsbasierten Prozesse Abrufung und Integration in einer Single- Stream-Architektur hindeutet. F{\"u}r beide experimentellen Designs wurden neben EKP-Daten auch Lesezeitdaten erhoben, welche im Kontext verst{\"a}ndnisbasierter Erwartbarkeit eine direkte Verbindung von Lesezeiten mit der P600 nahelegen. Die Ergebnisse dieser Dissertation sind unvereinbar mit traditionellen sowie neueren Theorien, welche argumentieren, dass die N400 Aspekte der Bedeutungsintegration indiziert. Im Speziellen widersprechen die Ergebnisse mehreren Schl{\"u}sselhypothesen von Multi-Stream-Modellen, welche aussagen, dass die N400 strukturunsensible Integration indiziert, w{\"a}hrend die P600 Konflikte zwischen strukturunsensibler und struktursensibler Integration widerspiegelt. Stattdessen lassen sich die Resultate mit wesentlich weniger Annahmen durch das Single-Stream-Modell der Retrieval-Integration-Theorie erkl{\"a}ren (siehe Diskussion in Kapitel 6). Demnach fu{\ss}t das Sprachverst{\"a}ndnis imWesentlichen auf den Mechanismen der Bedeutungsabrufung sowie der Bedeutungsintegration, welche im EKP-Signal als N400- und P600-Komponente messbar sind. Beide Komponenten werden standardm{\"a}{\ss}ig durch jedes Wort einer {\"A}u{\ss}erung hervorgerufen, wobei ihre Amplituden kontinuierlich den kognitiven Aufwand der Bedeutungsabrufung (N400) sowie der Bedeutungsintegration (P600) indizieren. Basierend auf den Ergebnissen dieser Dissertation ziehe ich den Schluss, dass eine an Erkenntnissen {\"u}ber das Sprachverst{\"a}ndnis interessierte Forschung der P600 zentrale Bedeutung beimessen sollte. Anhang A enth{\"a}lt eine theorieneutrale Abhandlung {\"u}ber die rERP Methode (Smith & Kutas, 2015a), einem statistischen Analyseverfahren, welches in der gesamten Dissertation zur Auswertung von EKP- und Lesezeitdaten zum Einsatz kommt. Alle Daten und s{\"a}mtlicher Code, welche zur Reproduktion der Analysen und Graphiken dieser Arbeit, einschlie{\ss}lich des Anhangs, notwendig sind, werden im Thesis Repository bereitgestellt (https://www.github.com/caurnhammer/ AurnhammerThesis). Jedwede Studien, welche mit menschlichen Partizipanten durchgef{\"u}hrt wurden, erhielten eine Ethik-Zulassung durch die Deutsche Gesellschaft f{\"u}r Sprachwissenschaft (DGfS). Teile dieser Arbeit basieren auf Ver{\"o}ffentlichungen in wissenschaftlichen Journalen (Kapitel 3: Aurnhammer et al., 2021; Kapitel 4: Aurnhammer, Delogu, et al., 2023; Kapitel 5: Aurnhammer, Crocker, and Brouwer, 2023).},
pubstate = {published},
type = {phdthesis}
}

Copy BibTeX to Clipboard

Theories of the electrophysiology of language comprehension are mostly informed by event-related potential effects observed between condition averages. We here argue that a dissociation between competing effect-level explanations of event-related potentials can be achieved by turning to predictions and analyses at the single-trial level. Specifically, we examine the single-trial dynamics in event-related potential data that exhibited a biphasic N400–P600 effect pattern. A group of multi-stream models can explain biphasic effects by positing that each individual trial should induce either an N400 increase or a P600 increase, but not both. An alternative, single-stream account, Retrieval-Integration theory, explicitly predicts that N400 amplitude and P600 amplitude should be correlated at the single-trial level. In order to investigate the single-trial dynamics of the N400 and the P600, we apply a regression-based technique in which we quantify the extent to which N400 amplitudes are predictive of the electroencephalogram in the P600 time window. Our findings suggest that, indeed, N400 amplitudes and P600 amplitudes are inversely correlated within-trial and, hence, the N400 effect and the P600 effect in biphasic data are driven by the same trials. Critically, we demonstrate that this finding also extends to data which exhibited only monophasic effects between conditions. In sum, the observation that the N400 is inversely correlated with the P600 on a by-trial basis supports a single stream view, such as Retrieval-Integration theory, and is difficult to reconcile with the processing mechanisms proposed by multi-stream models.

• s11571-023-09983-7     (4.2MB)
• https://link.springer.com/article/10.1007/s11571-023-09983-7
• https://doi.org/10.1007/s11571-023-09983-7

@article{aurnhammer2023singletrial,
title = {Single-trial neurodynamics reveal N400 and P600 coupling in language comprehension},
author = {Christoph Aurnhammer and Matthew W. Crocker and Harm Brouwer},
url = {https://link.springer.com/article/10.1007/s11571-023-09983-7},
doi = {https://doi.org/10.1007/s11571-023-09983-7},
year = {2023},
date = {2023},
journal = {Cognitive Neurodynamics},
abstract = {Theories of the electrophysiology of language comprehension are mostly informed by event-related potential effects observed between condition averages. We here argue that a dissociation between competing effect-level explanations of event-related potentials can be achieved by turning to predictions and analyses at the single-trial level. Specifically, we examine the single-trial dynamics in event-related potential data that exhibited a biphasic N400–P600 effect pattern. A group of multi-stream models can explain biphasic effects by positing that each individual trial should induce either an N400 increase or a P600 increase, but not both. An alternative, single-stream account, Retrieval-Integration theory, explicitly predicts that N400 amplitude and P600 amplitude should be correlated at the single-trial level. In order to investigate the single-trial dynamics of the N400 and the P600, we apply a regression-based technique in which we quantify the extent to which N400 amplitudes are predictive of the electroencephalogram in the P600 time window. Our findings suggest that, indeed, N400 amplitudes and P600 amplitudes are inversely correlated within-trial and, hence, the N400 effect and the P600 effect in biphasic data are driven by the same trials. Critically, we demonstrate that this finding also extends to data which exhibited only monophasic effects between conditions. In sum, the observation that the N400 is inversely correlated with the P600 on a by-trial basis supports a single stream view, such as Retrieval-Integration theory, and is difficult to reconcile with the processing mechanisms proposed by multi-stream models.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

The integration of word meaning into an unfolding utterance representation is a core operation of incremental language comprehension. There is considerable debate, however, as to which component of the ERP signal—the N400 or the P600—directly reflects integrative processes, with far reaching consequences for the temporal organization and architecture of the comprehension system. Multi-stream models maintaining the N400 as integration crucially rely on the presence of a semantically attractive plausible alternative interpretation to account for the absence of an N400 effect in response to certain semantic anomalies, as reported in previous studies. The single-stream Retrieval–Integration account posits the P600 as an index of integration, further predicting that its amplitude varies continuously with integrative effort. Here, we directly test these competing hypotheses using a context manipulation design in which a semantically attractive alternative is either available or not, and target word plausibility is varied across three levels. An initial self-paced reading study revealed graded reading times for plausibility, suggesting differential integration effort. A subsequent ERP study showed no N400 differences across conditions, and that P600 amplitude is graded for plausibility. These findings are inconsistent with the interpretation of the N400 as an index of integration, as no N400 effect emerged even in the absence of a semantically attractive alternative. By contrast, the link between plausibility, reading times, and P600 amplitude supports the view that the P600 is a continuous index of integration effort. More generally, our results support a single-stream architecture and eschew the need for multi-stream accounts.

• Psychophysiology - 2023 - Aurnhammer - The P600 as a continuous index of integration effort     (11.04MB)
• https://onlinelibrary.wiley.com/doi/10.1111/psyp.14302
• https://doi.org/10.1111/psyp.14302

@article{aurnhammer2023continuous,
title = {The P600 as a Continuous Index of Integration Effort},
author = {Christoph Aurnhammer and Francesca Delogu and Harm Brouwer and Matthew W. Crocker},
url = {https://onlinelibrary.wiley.com/doi/10.1111/psyp.14302},
doi = {https://doi.org/10.1111/psyp.14302},
year = {2023},
date = {2023},
journal = {Psychophysiology},
abstract = {The integration of word meaning into an unfolding utterance representation is a core operation of incremental language comprehension. There is considerable debate, however, as to which component of the ERP signal—the N400 or the P600—directly reflects integrative processes, with far reaching consequences for the temporal organization and architecture of the comprehension system. Multi-stream models maintaining the N400 as integration crucially rely on the presence of a semantically attractive plausible alternative interpretation to account for the absence of an N400 effect in response to certain semantic anomalies, as reported in previous studies. The single-stream Retrieval–Integration account posits the P600 as an index of integration, further predicting that its amplitude varies continuously with integrative effort. Here, we directly test these competing hypotheses using a context manipulation design in which a semantically attractive alternative is either available or not, and target word plausibility is varied across three levels. An initial self-paced reading study revealed graded reading times for plausibility, suggesting differential integration effort. A subsequent ERP study showed no N400 differences across conditions, and that P600 amplitude is graded for plausibility. These findings are inconsistent with the interpretation of the N400 as an index of integration, as no N400 effect emerged even in the absence of a semantically attractive alternative. By contrast, the link between plausibility, reading times, and P600 amplitude supports the view that the P600 is a continuous index of integration effort. More generally, our results support a single-stream architecture and eschew the need for multi-stream accounts.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

Electrophysiological studies suggest that situational event knowledge plays an important role in language processing, but often fail to distinguish whether observed effects are driven by combinatorial expectations, or simple association with the context. In two ERP experiments, participants read short discourses describing ongoing events. We manipulated the situational expectancy of the target word continuing the event as well as the presence of an associated, but inactive event in the context. In both experiments we find an N400 effect for unexpected compared to expected target words, but this effect is significantly attenuated when the unexpected target is nonetheless associated with non-occurring context events. Our findings demonstrate that the N400 is simultaneously influenced by both simple association with – and combinatorial expectations derived from – situational event knowledge. Thus, experimental investigations and comprehension models of the use of event knowledge must accommodate the role of both expectancy and association in electrophysiological measures.

• https://www.tandfonline.com/doi/full/10.1080/23273798.2021.2022171?src=
• https://doi.org/10.1080/23273798.2021.2022171

@article{doi:10.1080/23273798.2021.2022171,
title = {Situational expectancy or association? The influence of event knowledge on the N400},
author = {Elisabeth Rabs and Francesca Delogu and Heiner Drenhaus and Matthew W. Crocker},
url = {https://www.tandfonline.com/doi/full/10.1080/23273798.2021.2022171?src=},
doi = {https://doi.org/10.1080/23273798.2021.2022171},
year = {2022},
date = {2022-01-16},
journal = {Language, Cognition and Neuroscience},
pages = {1-19},
publisher = {Routledge},
abstract = {Electrophysiological studies suggest that situational event knowledge plays an important role in language processing, but often fail to distinguish whether observed effects are driven by combinatorial expectations, or simple association with the context. In two ERP experiments, participants read short discourses describing ongoing events. We manipulated the situational expectancy of the target word continuing the event as well as the presence of an associated, but inactive event in the context. In both experiments we find an N400 effect for unexpected compared to expected target words, but this effect is significantly attenuated when the unexpected target is nonetheless associated with non-occurring context events. Our findings demonstrate that the N400 is simultaneously influenced by both simple association with – and combinatorial expectations derived from – situational event knowledge. Thus, experimental investigations and comprehension models of the use of event knowledge must accommodate the role of both expectancy and association in electrophysiological measures.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

Expectation-based theories of language processing, such as Surprisal theory, are supported by evidence of anticipation effects in both behavioural and neurophysiological measures. Online measures of language processing, however, are known to be influenced by factors such as lexical association that are distinct from—but often confounded with—expectancy. An open question therefore is whether a specific locus of expectancy related effects can be established in neural and behavioral processing correlates. We address this question in an event-related potential experiment and a self-paced reading experiment that independently cross expectancy and lexical association in a context manipulation design. We find that event-related potentials reveal that the N400 is sensitive to both expectancy and lexical association, while the P600 is modulated only by expectancy. Reading times, in turn, reveal effects of both association and expectancy in the first spillover region, followed by effects of expectancy alone in the second spillover region. These findings are consistent with the Retrieval-Integration account of language comprehension, according to which lexical retrieval (N400) is facilitated for words that are both expected and associated, whereas integration difficulty (P600) will be greater for unexpected words alone. Further, an exploratory analysis suggests that the P600 is not merely sensitive to expectancy violations, but rather, that there is a continuous relation. Taken together, these results suggest that the P600, like reading times, may reflect a meaning-centric notion of Surprisal in language comprehension.

• https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0257430
• https://doi.org/10.1371/journal.pone.0257430

@article{aurnhammer2021retrieval,
title = {Retrieval (N400) and Integration (P600) in Expectation-based Comprehension},
author = {Christoph Aurnhammer and Francesca Delogu and Miriam Schulz and Harm Brouwer and Matthew W. Crocker},
url = {https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0257430},
doi = {https://doi.org/10.1371/journal.pone.0257430},
year = {2021},
date = {2021-09-28},
journal = {PLoS ONE},
pages = {e0257430},
volume = {16},
number = {9},
abstract = {Expectation-based theories of language processing, such as Surprisal theory, are supported by evidence of anticipation effects in both behavioural and neurophysiological measures. Online measures of language processing, however, are known to be influenced by factors such as lexical association that are distinct from—but often confounded with—expectancy. An open question therefore is whether a specific locus of expectancy related effects can be established in neural and behavioral processing correlates. We address this question in an event-related potential experiment and a self-paced reading experiment that independently cross expectancy and lexical association in a context manipulation design. We find that event-related potentials reveal that the N400 is sensitive to both expectancy and lexical association, while the P600 is modulated only by expectancy. Reading times, in turn, reveal effects of both association and expectancy in the first spillover region, followed by effects of expectancy alone in the second spillover region. These findings are consistent with the Retrieval-Integration account of language comprehension, according to which lexical retrieval (N400) is facilitated for words that are both expected and associated, whereas integration difficulty (P600) will be greater for unexpected words alone. Further, an exploratory analysis suggests that the P600 is not merely sensitive to expectancy violations, but rather, that there is a continuous relation. Taken together, these results suggest that the P600, like reading times, may reflect a meaning-centric notion of Surprisal in language comprehension.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

The problem of spatiotemporal overlap between event-related potential (ERP) components is generally acknowledged in language research. However, its implications for the interpretation of experimental results are often overlooked. In a previous experiment on the functional interpretation of the N400 and P600, it was argued that a P600 effect to implausible words was largely obscured – in one of two implausible conditions – by an overlapping N400 effect of semantic association. In the present ERP study, we show that the P600 effect of implausibility is uncovered when the critical condition is tested against a proper baseline condition which elicits a similar N400 amplitude, while it is obscured when tested against a baseline condition producing an N400 effect. Our findings reveal that component overlap can result in the apparent absence or presence of an effect in the surface signal and should therefore be carefully considered when interpreting ERP patterns. Importantly, we show that, by factoring in the effects of spatiotemporal overlap between the N400 and P600 on the surface signal, which we reveal using rERP analysis, apparent inconsistencies in previous findings are easily reconciled, enabling us to draw unambiguous conclusions about the functional interpretation of the N400 and P600 components. Overall, our results provide compelling evidence that the N400 reflects lexical retrieval processes, while the P600 indexes compositional integration of word meaning into the unfolding utterance interpretation.

• https://www.sciencedirect.com/science/article/pii/S0006899321003711
• https://doi.org/10.1016/j.brainres.2021.147514

@article{DELOGU2021147514,
title = {When components collide: Spatiotemporal overlap of the N400 and P600 in language comprehension},
author = {Francesca Delogu and Harm Brouwer and Matthew W. Crocker},
url = {https://www.sciencedirect.com/science/article/pii/S0006899321003711},
doi = {https://doi.org/10.1016/j.brainres.2021.147514},
year = {2021},
date = {2021},
journal = {Brain Research},
pages = {147514},
volume = {1766},
abstract = {The problem of spatiotemporal overlap between event-related potential (ERP) components is generally acknowledged in language research. However, its implications for the interpretation of experimental results are often overlooked. In a previous experiment on the functional interpretation of the N400 and P600, it was argued that a P600 effect to implausible words was largely obscured – in one of two implausible conditions – by an overlapping N400 effect of semantic association. In the present ERP study, we show that the P600 effect of implausibility is uncovered when the critical condition is tested against a proper baseline condition which elicits a similar N400 amplitude, while it is obscured when tested against a baseline condition producing an N400 effect. Our findings reveal that component overlap can result in the apparent absence or presence of an effect in the surface signal and should therefore be carefully considered when interpreting ERP patterns. Importantly, we show that, by factoring in the effects of spatiotemporal overlap between the N400 and P600 on the surface signal, which we reveal using rERP analysis, apparent inconsistencies in previous findings are easily reconciled, enabling us to draw unambiguous conclusions about the functional interpretation of the N400 and P600 components. Overall, our results provide compelling evidence that the N400 reflects lexical retrieval processes, while the P600 indexes compositional integration of word meaning into the unfolding utterance interpretation.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

Theories and models of spoken word recognition aim to explain the process of accessing lexical knowledge given an acoustic realization of a word form. There is consensus that phonological and semantic information is crucial for this process. However, there is accumulating evidence that orthographic information could also have an impact on auditory word recognition. This paper presents two models of spoken word recognition that instantiate different hypotheses regarding the influence of orthography on this process. We show that these models reproduce human-like behavior in different ways and provide testable hypotheses for future research on the source of orthographic effects in spoken word recognition.

• https://aclanthology.org/2021.eacl-srw.3

@inproceedings{macher-etal-2021-read,
title = {Do we read what we hear? Modeling orthographic influences on spoken word recognition},
author = {Nicole Macher and Badr M. Abdullah and Harm Brouwer and Dietrich Klakow},
url = {https://aclanthology.org/2021.eacl-srw.3},
year = {2021},
date = {2021},
booktitle = {Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop},
pages = {16-22},
publisher = {Association for Computational Linguistics},
address = {Online},
abstract = {Theories and models of spoken word recognition aim to explain the process of accessing lexical knowledge given an acoustic realization of a word form. There is consensus that phonological and semantic information is crucial for this process. However, there is accumulating evidence that orthographic information could also have an impact on auditory word recognition. This paper presents two models of spoken word recognition that instantiate different hypotheses regarding the influence of orthography on this process. We show that these models reproduce human-like behavior in different ways and provide testable hypotheses for future research on the source of orthographic effects in spoken word recognition.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

Natural language semantics has recently sought to combine the complementary strengths of formal and distributional approaches to meaning. However, given the fundamentally different ‘representational currency’ underlying these approaches—models of the world versus linguistic co-occurrence—their unification has proven extremely difficult. Here, we define Distributional Formal Semantics, which integrates distributionality into a formal semantic system on the level of formal models. This approach offers probabilistic, distributed meaning representations that are inherently compositional, and that naturally capture fundamental semantic notions such as quantification and entailment. Furthermore, we show how the probabilistic nature of these representations allows for probabilistic inference, and how the information-theoretic notion of “information” (measured in Entropy and Surprisal) naturally follows from it. Finally, we illustrate how meaning representations can be derived incrementally from linguistic input using a recurrent neural network model, and how the resultant incremental semantic construction procedure intuitively captures key semantic phenomena, including negation, presupposition, and anaphoricity.

• https://www.sciencedirect.com/science/article/pii/S089054012100078X
• https://doi.org/10.1016/j.ic.2021.104763

@article{venhuizen2021distributional,
title = {Distributional formal semantics},
author = {Noortje Venhuizen and Petra Hendriks and Matthew W. Crocker and Harm Brouwer},
url = {https://www.sciencedirect.com/science/article/pii/S089054012100078X},
doi = {https://doi.org/10.1016/j.ic.2021.104763},
year = {2021},
date = {2021},
journal = {Information and Computation},
pages = {104763},
abstract = {Natural language semantics has recently sought to combine the complementary strengths of formal and distributional approaches to meaning. However, given the fundamentally different ‘representational currency’ underlying these approaches—models of the world versus linguistic co-occurrence—their unification has proven extremely difficult. Here, we define Distributional Formal Semantics, which integrates distributionality into a formal semantic system on the level of formal models. This approach offers probabilistic, distributed meaning representations that are inherently compositional, and that naturally capture fundamental semantic notions such as quantification and entailment. Furthermore, we show how the probabilistic nature of these representations allows for probabilistic inference, and how the information-theoretic notion of “information” (measured in Entropy and Surprisal) naturally follows from it. Finally, we illustrate how meaning representations can be derived incrementally from linguistic input using a recurrent neural network model, and how the resultant incremental semantic construction procedure intuitively captures key semantic phenomena, including negation, presupposition, and anaphoricity.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

While there is a substantial amount of evidence for language processing being a highly incremental and predictive process, we still know relatively little about how top-down discourse based expectations are combined with bottom-up information such as discourse connectives. The present article reports on three experiments investigating this question using different methodologies (visual world paradigm and ERPs) in two languages (German and English). We find support for highly incremental processing of causal and concessive discourse connectives, causing anticipation of upcoming material. Our visual world study shows that anticipatory looks depend on the discourse connective; furthermore, the German ERP study revealed an N400 effect on a gender-marked adjective preceding the target noun, when the target noun was inconsistent with the expectations elicited by the combination of context and discourse connective. Moreover, our experiments reveal that the facilitation of downstream material based on earlier connectives comes at the cost of reversing original expectations, as evidenced by a P600 effect on the concessive relative to the causal connective.

• https://doi.org/10.1515/ling-2021-0011
• https://doi.org/doi:10.1515/ling-2021-0011

@article{koehne2021online,
title = {The online processing of causal and concessive discourse connectives},
author = {Judith K{\"o}hne-Fuetterer and Heiner Drenhaus and Francesca Delogu and Vera Demberg},
url = {https://doi.org/10.1515/ling-2021-0011},
doi = {https://doi.org/doi:10.1515/ling-2021-0011},
year = {2021},
date = {2021-03-04},
journal = {Linguistics},
pages = {417-448},
volume = {59},
number = {2},
abstract = {While there is a substantial amount of evidence for language processing being a highly incremental and predictive process, we still know relatively little about how top-down discourse based expectations are combined with bottom-up information such as discourse connectives. The present article reports on three experiments investigating this question using different methodologies (visual world paradigm and ERPs) in two languages (German and English). We find support for highly incremental processing of causal and concessive discourse connectives, causing anticipation of upcoming material. Our visual world study shows that anticipatory looks depend on the discourse connective; furthermore, the German ERP study revealed an N400 effect on a gender-marked adjective preceding the target noun, when the target noun was inconsistent with the expectations elicited by the combination of context and discourse connective. Moreover, our experiments reveal that the facilitation of downstream material based on earlier connectives comes at the cost of reversing original expectations, as evidenced by a P600 effect on the concessive relative to the causal connective.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

Expectation-based theories of language comprehension, in particular Surprisal Theory, go a long way in accounting for the behavioral correlates of word-by-word processing difficulty, such as reading times. An open question, however, is in which component(s) of the Event-Related brain Potential (ERP) signal Surprisal is reflected, and how these electrophysiological correlates relate to behavioral processing indices. Here, we address this question by instantiating an explicit neurocomputational model of incremental, word-by-word language comprehension that produces estimates of the N400 and the P600 – the two most salient ERP components for language processing – as well as estimates of `comprehension-centric‘ Surprisal for each word in a sentence. We derive model predictions for a recent experimental design that directly investigates `world-knowledge‘-induced Surprisal. By relating these predictions to both empirical electrophysiological and behavioral results, we establish a close link between Surprisal, as indexed by reading times, and the P600 component of the ERP signal. The resultant model thus offers an integrated neurobehavioral account of processing difficulty in language comprehension.

• https://www.frontiersin.org/articles/10.3389/fpsyg.2021.615538/full
• https://doi.org/10.3389/fpsyg.2021.615538

@article{Brouwer2021,
title = {Neurobehavioral Correlates of Surprisal in Language Comprehension: A Neurocomputational Model},
author = {Harm Brouwer and Francesca Delogu and Noortje Venhuizen and Matthew W. Crocker},
url = {https://www.frontiersin.org/articles/10.3389/fpsyg.2021.615538/full},
doi = {https://doi.org/10.3389/fpsyg.2021.615538},
year = {2021},
date = {2021-02-11},
journal = {Frontiers in Psychology},
abstract = {Expectation-based theories of language comprehension, in particular Surprisal Theory, go a long way in accounting for the behavioral correlates of word-by-word processing difficulty, such as reading times. An open question, however, is in which component(s) of the Event-Related brain Potential (ERP) signal Surprisal is reflected, and how these electrophysiological correlates relate to behavioral processing indices. Here, we address this question by instantiating an explicit neurocomputational model of incremental, word-by-word language comprehension that produces estimates of the N400 and the P600 - the two most salient ERP components for language processing - as well as estimates of `comprehension-centric' Surprisal for each word in a sentence. We derive model predictions for a recent experimental design that directly investigates `world-knowledge'-induced Surprisal. By relating these predictions to both empirical electrophysiological and behavioral results, we establish a close link between Surprisal, as indexed by reading times, and the P600 component of the ERP signal. The resultant model thus offers an integrated neurobehavioral account of processing difficulty in language comprehension.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

Event‐related potentials (ERPs) provide a multidimensional and real‐time window into neurocognitive processing. The typical Waveform‐based Component Structure (WCS) approach to ERPs assesses the modulation pattern of components—systematic, reoccurring voltage fluctuations reflecting specific computational operations—by looking at mean amplitude in predetermined time‐windows.

This WCS approach, however, often leads to inconsistent results within as well as across studies. It has been argued that at least some inconsistencies may be reconciled by considering spatiotemporal overlap between components; that is, components may overlap in both space and time, and given their additive nature, this means that the WCS may fail to accurately represent its underlying latent component structure (LCS). We employ regression‐based ERP (rERP) estimation to extend traditional approaches with an additional layer of analysis, which enables the explicit modeling of the LCS underlying WCS. To demonstrate its utility, we incrementally derive an rERP analysis of a recent study on language comprehension with seemingly inconsistent WCS‐derived results.

Analysis of the resultant regression models allows one to derive an explanation for the WCS in terms of how relevant regression predictors combine in space and time, and crucially, how individual predictors may be mapped onto unique components in LCS, revealing how these spatiotemporally overlap in the WCS. We conclude that rERP estimation allows for investigating how scalp‐recorded voltages derive from the spatiotemporal combination of experimentally manipulated factors. Moreover, when factors can be uniquely mapped onto components, rERPs may offer explanations for seemingly inconsistent ERP waveforms at the level of their underlying latent component structure.

• Brouwer2020SplittingEvent     (3.73MB)
• https://onlinelibrary.wiley.com/doi/10.1111/ejn.14961
• https://doi.org/10.1111/ejn.14961

@article{Brouwer2020,
title = {Splitting event‐related potentials: Modeling latent components using regression‐based waveform estimation},
author = {Harm Brouwer and Francesca Delogu and Matthew W. Crocker},
url = {https://onlinelibrary.wiley.com/doi/10.1111/ejn.14961},
doi = {https://doi.org/10.1111/ejn.14961},
year = {2020},
date = {2020-09-08},
journal = {European Journal of Neuroscience},
abstract = {Event‐related potentials (ERPs) provide a multidimensional and real‐time window into neurocognitive processing. The typical Waveform‐based Component Structure (WCS) approach to ERPs assesses the modulation pattern of components—systematic, reoccurring voltage fluctuations reflecting specific computational operations—by looking at mean amplitude in predetermined time‐windows. This WCS approach, however, often leads to inconsistent results within as well as across studies. It has been argued that at least some inconsistencies may be reconciled by considering spatiotemporal overlap between components; that is, components may overlap in both space and time, and given their additive nature, this means that the WCS may fail to accurately represent its underlying latent component structure (LCS). We employ regression‐based ERP (rERP) estimation to extend traditional approaches with an additional layer of analysis, which enables the explicit modeling of the LCS underlying WCS. To demonstrate its utility, we incrementally derive an rERP analysis of a recent study on language comprehension with seemingly inconsistent WCS‐derived results. Analysis of the resultant regression models allows one to derive an explanation for the WCS in terms of how relevant regression predictors combine in space and time, and crucially, how individual predictors may be mapped onto unique components in LCS, revealing how these spatiotemporally overlap in the WCS. We conclude that rERP estimation allows for investigating how scalp‐recorded voltages derive from the spatiotemporal combination of experimentally manipulated factors. Moreover, when factors can be uniquely mapped onto components, rERPs may offer explanations for seemingly inconsistent ERP waveforms at the level of their underlying latent component structure.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

Language is processed on a more or less word-by-word basis, and the processing difficulty induced by each word is affected by our prior linguistic experience as well as our general knowledge about the world. Surprisal and entropy reduction have been independently proposed as linking theories between word processing difficulty and probabilistic language models. Extant models, however, are typically limited to capturing linguistic experience and hence cannot account for the influence of world knowledge. A recent comprehension model by Venhuizen, Crocker, and Brouwer (2019, Discourse Processes) improves upon this situation by instantiating a comprehension-centric metric of surprisal that integrates linguistic experience and world knowledge at the level of interpretation and combines them in determining online expectations. Here, we extend this work by deriving a comprehension-centric metric of entropy reduction from this model. In contrast to previous work, which has found that surprisal and entropy reduction are not easily dissociated, we do find a clear dissociation in our model. While both surprisal and entropy reduction derive from the same cognitive process – the word-by-word updating of the unfolding interpretation – they reflect different aspects of this process: state-by-state expectation (surprisal) versus end-state confirmation (entropy reduction).

• entropy-21-01159-v2-1     (1.29MB)
• https://www.mdpi.com/1099-4300/21/12/1159
• https://doi.org/10.3390/e21121159

@article{Venhuizen2019,
title = {Semantic Entropy in Language Comprehension},
author = {Noortje Venhuizen and Matthew W. Crocker and Harm Brouwer},
url = {https://www.mdpi.com/1099-4300/21/12/1159},
doi = {https://doi.org/10.3390/e21121159},
year = {2019},
date = {2019-11-27},
journal = {Entropy},
pages = {1159},
volume = {21},
number = {12},
abstract = {Language is processed on a more or less word-by-word basis, and the processing difficulty induced by each word is affected by our prior linguistic experience as well as our general knowledge about the world. Surprisal and entropy reduction have been independently proposed as linking theories between word processing difficulty and probabilistic language models. Extant models, however, are typically limited to capturing linguistic experience and hence cannot account for the influence of world knowledge. A recent comprehension model by Venhuizen, Crocker, and Brouwer (2019, Discourse Processes) improves upon this situation by instantiating a comprehension-centric metric of surprisal that integrates linguistic experience and world knowledge at the level of interpretation and combines them in determining online expectations. Here, we extend this work by deriving a comprehension-centric metric of entropy reduction from this model. In contrast to previous work, which has found that surprisal and entropy reduction are not easily dissociated, we do find a clear dissociation in our model. While both surprisal and entropy reduction derive from the same cognitive process - the word-by-word updating of the unfolding interpretation - they reflect different aspects of this process: state-by-state expectation (surprisal) versus end-state confirmation (entropy reduction).},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

The processing difficulty of each word we encounter in a sentence is affected by both our prior linguistic experience and our general knowledge about the world. Computational models of incremental language processing have, however, been limited in accounting for the influence of world knowledge.

We develop an incremental model of language comprehension that constructs – on a word-by-word basis – rich, probabilistic situation model representations. To quantify linguistic processing effort, we adopt Surprisal Theory, which asserts that the processing difficulty incurred by a word is inversely proportional to its expectancy (Hale, 2001; Levy, 2008). In contrast with typical language model implementations of surprisal, the proposed model instantiates a novel comprehension-centric metric of surprisal that reflects the likelihood of the unfolding utterance meaning as established after processing each word. Simulations are presented that demonstrate that linguistic experience and world knowledge are integrated in the model at the level of interpretation and combine in determining online expectations.

• Venhuizen2019ExpectationComprehension     (0.97MB)
• https://www.tandfonline.com/doi/full/10.1080/0163853X.2018.1448677
• https://doi.org/10.1080/0163853X.2018.1448677

@article{Venhuizen2019,
title = {Expectation-based Comprehension: Modeling the interaction of world knowledge and linguistic experience},
author = {Noortje Venhuizen and Matthew W. Crocker and Harm Brouwer},
url = {https://www.tandfonline.com/doi/full/10.1080/0163853X.2018.1448677},
doi = {https://doi.org/10.1080/0163853X.2018.1448677},
year = {2019},
date = {2019},
journal = {Discourse Processes},
pages = {229-255},
volume = {56},
number = {3},
abstract = {The processing difficulty of each word we encounter in a sentence is affected by both our prior linguistic experience and our general knowledge about the world. Computational models of incremental language processing have, however, been limited in accounting for the influence of world knowledge. We develop an incremental model of language comprehension that constructs - on a word-by-word basis - rich, probabilistic situation model representations. To quantify linguistic processing effort, we adopt Surprisal Theory, which asserts that the processing difficulty incurred by a word is inversely proportional to its expectancy (Hale, 2001; Levy, 2008). In contrast with typical language model implementations of surprisal, the proposed model instantiates a novel comprehension-centric metric of surprisal that reflects the likelihood of the unfolding utterance meaning as established after processing each word. Simulations are presented that demonstrate that linguistic experience and world knowledge are integrated in the model at the level of interpretation and combine in determining online expectations.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

The functional interpretation of two salient language-sensitive ERP components – the N400 and the P600 – remains a matter of debate. Prominent alternative accounts link the N400 to processes related to lexical retrieval, semantic integration, or both, while the P600 has been associated with syntactic reanalysis or, alternatively, to semantic integration. The often overlapping predictions of these competing accounts in extant experimental designs, however, has meant that previous findings have failed to clearly decide among them. Here, we present an experiment that directly tests the competing hypotheses using a design that clearly teases apart the retrieval versus integration view of the N400, while also dissociating a syntactic reanalysis/reprocessing account of the P600 from semantic integration. Our findings provide support for an integrated functional interpretation according to which the N400 reflects context-sensitive lexical retrieval – but not integration – processes. While the observed P600 effects were not predicted by any account, we argue that they can be reconciled with the integration view, if spatio-temporal overlap of ERP components is taken into consideration.

• delogu     (2.96MB)
• https://www.sciencedirect.com/science/article/pii/S0278262618304299
• https://doi.org/10.1016/j.bandc.2019.05.007

@article{delogu2019event,
title = {Event-related potentials index lexical retrieval (N400) and integration (P600) during language comprehension},
author = {Francesca Delogu and Harm Brouwer and Matthew W. Crocker},
url = {https://www.sciencedirect.com/science/article/pii/S0278262618304299},
doi = {https://doi.org/10.1016/j.bandc.2019.05.007},
year = {2019},
date = {2019},
journal = {Brain and Cognition},
volume = {135},
abstract = {The functional interpretation of two salient language-sensitive ERP components - the N400 and the P600 - remains a matter of debate. Prominent alternative accounts link the N400 to processes related to lexical retrieval, semantic integration, or both, while the P600 has been associated with syntactic reanalysis or, alternatively, to semantic integration. The often overlapping predictions of these competing accounts in extant experimental designs, however, has meant that previous findings have failed to clearly decide among them. Here, we present an experiment that directly tests the competing hypotheses using a design that clearly teases apart the retrieval versus integration view of the N400, while also dissociating a syntactic reanalysis/reprocessing account of the P600 from semantic integration. Our findings provide support for an integrated functional interpretation according to which the N400 reflects context-sensitive lexical retrieval - but not integration - processes. While the observed P600 effects were not predicted by any account, we argue that they can be reconciled with the integration view, if spatio-temporal overlap of ERP components is taken into consideration.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

Contrary to the Gricean maxims of quantity (Grice, in: Cole, Morgan (eds) Syntax and semantics: speech acts, vol III, pp 41-58, Academic Press, New York, 1975), it has been repeatedly shown that speakers often include redundant information in their utterances (over-specifications). Previous research on referential communication has long debated whether this redundancy is the result of speaker-internal or addressee-oriented processes, while it is also unclear whether referential redundancy hinders or facilitates comprehension.

We present an information-theoretic explanation for the use of over-specification in visually-situated communication, which quantifies the amount of uncertainty regarding the referent as entropy (Shannon in Bell Syst Tech J 5:10, https://doi.org/10.1002/j.1538-7305.1948.tb01338.x, 1948). Examining both the comprehension and production of over-specifications, we present evidence that (a) listeners’ processing is facilitated by the use of redundancy as well as by a greater reduction of uncertainty early on in the utterance, and (b) that at least for some speakers, listeners’ processing concerns influence their encoding of over-specifications: Speakers were more likely to use redundant adjectives when these adjectives reduced entropy to a higher degree than adjectives necessary for target identification.

• Rational-over-specification-in-visually-situated-comprehension-and-production     (1.72MB)
• https://link.springer.com/article/10.1007%2Fs41809-019-00032-6
• https://doi.org/10.1007/s41809-019-00032-6

@article{Tourtouri2019,
title = {Rational over-specification in visually-situated comprehension and production},
author = {Elli Tourtouri and Francesca Delogu and Les Sikos and Matthew W. Crocker},
url = {https://link.springer.com/article/10.1007%2Fs41809-019-00032-6},
doi = {https://doi.org/10.1007/s41809-019-00032-6},
year = {2019},
date = {2019},
journal = {Journal of Cultural Cognitive Science},
pages = {175-202},
volume = {3},
number = {2},
abstract = {Contrary to the Gricean maxims of quantity (Grice, in: Cole, Morgan (eds) Syntax and semantics: speech acts, vol III, pp 41-58, Academic Press, New York, 1975), it has been repeatedly shown that speakers often include redundant information in their utterances (over-specifications). Previous research on referential communication has long debated whether this redundancy is the result of speaker-internal or addressee-oriented processes, while it is also unclear whether referential redundancy hinders or facilitates comprehension. We present an information-theoretic explanation for the use of over-specification in visually-situated communication, which quantifies the amount of uncertainty regarding the referent as entropy (Shannon in Bell Syst Tech J 5:10, https://doi.org/10.1002/j.1538-7305.1948.tb01338.x, 1948). Examining both the comprehension and production of over-specifications, we present evidence that (a) listeners’ processing is facilitated by the use of redundancy as well as by a greater reduction of uncertainty early on in the utterance, and (b) that at least for some speakers, listeners’ processing concerns influence their encoding of over-specifications: Speakers were more likely to use redundant adjectives when these adjectives reduced entropy to a higher degree than adjectives necessary for target identification.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

Formal semantics and distributional semantics offer complementary strengths in capturing the meaning of natural language. As such, a considerable amount of research has sought to unify them, either by augmenting formal semantic systems with a distributional component, or by defining a formal system on top of distributed representations.

Arriving at such a unified framework has, however, proven extremely challenging. One reason for this is that formal and distributional semantics operate on a fundamentally different ‘representational currency’: formal semantics defines meaning in terms of models of the world, whereas distributional semantics defines meaning in terms of linguistic co-occurrence. Here, we pursue an alternative approach by deriving a vector space model that defines meaning in a distributed manner relative to formal models of the world.

We will show that the resulting Distributional Formal Semantics offers probabilistic distributed representations that are also inherently compositional, and that naturally capture quantification and entailment. We moreover show that, when used as part of a neural network model, these representations allow for capturing incremental meaning construction and probabilistic inferencing. This framework thus lays the groundwork for an integrated distributional and formal approach to meaning.

• https://link.springer.com/chapter/10.1007%2F978-3-662-59533-6_39
• https://doi.org/https://doi.org/10.1007/978-3-662-59533-6_39

@inproceedings{Venhuizen2019b,
title = {A Framework for Distributional Formal Semantics},
author = {Noortje Venhuizen and Petra Hendriks and Matthew W. Crocker and Harm Brouwer},
editor = {Rosalie Iemhoff and Michael Moortgat and Ruy de Queiroz},
url = {https://link.springer.com/chapter/10.1007%2F978-3-662-59533-6_39},
doi = {https://doi.org/https://doi.org/10.1007/978-3-662-59533-6_39},
year = {2019},
date = {2019-06-09},
booktitle = {Logic, Language, Information, and Computation, Proceedings of the 26th International Workshop WoLLIC 2019},
pages = {633-646},
abstract = {Formal semantics and distributional semantics offer complementary strengths in capturing the meaning of natural language. As such, a considerable amount of research has sought to unify them, either by augmenting formal semantic systems with a distributional component, or by defining a formal system on top of distributed representations. Arriving at such a unified framework has, however, proven extremely challenging. One reason for this is that formal and distributional semantics operate on a fundamentally different ‘representational currency’: formal semantics defines meaning in terms of models of the world, whereas distributional semantics defines meaning in terms of linguistic co-occurrence. Here, we pursue an alternative approach by deriving a vector space model that defines meaning in a distributed manner relative to formal models of the world. We will show that the resulting Distributional Formal Semantics offers probabilistic distributed representations that are also inherently compositional, and that naturally capture quantification and entailment. We moreover show that, when used as part of a neural network model, these representations allow for capturing incremental meaning construction and probabilistic inferencing. This framework thus lays the groundwork for an integrated distributional and formal approach to meaning.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

When reading a text describing an everyday activity, comprehenders build a model of the situation described that includes prior knowledge of the entities, locations, and sequences of actions that typically occur within the event. Previous work has demonstrated that such knowledge guides the processing of incoming information by making event boundaries more or less expected.

In the present ERP study, we investigated whether comprehenders’ expectations about event boundaries are influenced by how elaborately common events are described in the context. Participants read short stories in which a common activity (e.g., washing the dishes) was described either in brief or in an elaborate manner. The final sentence contained a target word referring to a more predictable action marking a fine event boundary (e.g., drying) or a less predictable action, marking a coarse event boundary (e.g., jogging). The results revealed a larger N400 effect for coarse event boundaries compared to fine event boundaries, but no interaction with description length. Between 600 and 1000 ms, however, elaborate contexts elicited a larger frontal positivity compared to brief contexts.

This effect was largely driven by less predictable targets, marking coarse event boundaries. We interpret the P600 effect as indexing the updating of the situation model at event boundaries, consistent with Event Segmentation Theory (EST). The updating process is more demanding with coarse event boundaries, which presumably require the construction of a new situation model. Electronic supplementary material The online version of this article (10.3758/s13421-017-0766-4) contains supplementary material, which is available to authorized users.

• On-the-predictability-of-event-boundaries-in-discourse     (0.73MB)
• https://www.researchgate.net/publication/321175597_On_the_predictability_of_event_boundaries_in_discourse_An_ERP_investigation
• https://doi.org/https://doi.org/10.3758/s13421-017-0766-4

@article{Delogu2018,
title = {On the predictability of event-boundaries in discourse: an ERP investigation},
author = {Francesca Delogu and Heiner Drenhaus and Matthew W. Crocker},
url = {https://www.researchgate.net/publication/321175597_On_the_predictability_of_event_boundaries_in_discourse_An_ERP_investigation},
doi = {https://doi.org/https://doi.org/10.3758/s13421-017-0766-4},
year = {2018},
date = {2018},
journal = {Memory and Cognition},
pages = {315-325},
volume = {46},
number = {2},
abstract = {When reading a text describing an everyday activity, comprehenders build a model of the situation described that includes prior knowledge of the entities, locations, and sequences of actions that typically occur within the event. Previous work has demonstrated that such knowledge guides the processing of incoming information by making event boundaries more or less expected. In the present ERP study, we investigated whether comprehenders’ expectations about event boundaries are influenced by how elaborately common events are described in the context. Participants read short stories in which a common activity (e.g., washing the dishes) was described either in brief or in an elaborate manner. The final sentence contained a target word referring to a more predictable action marking a fine event boundary (e.g., drying) or a less predictable action, marking a coarse event boundary (e.g., jogging). The results revealed a larger N400 effect for coarse event boundaries compared to fine event boundaries, but no interaction with description length. Between 600 and 1000 ms, however, elaborate contexts elicited a larger frontal positivity compared to brief contexts. This effect was largely driven by less predictable targets, marking coarse event boundaries. We interpret the P600 effect as indexing the updating of the situation model at event boundaries, consistent with Event Segmentation Theory (EST). The updating process is more demanding with coarse event boundaries, which presumably require the construction of a new situation model. Electronic supplementary material The online version of this article (10.3758/s13421-017-0766-4) contains supplementary material, which is available to authorized users.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

A word’s predictability or surprisal, as determined by cloze probabilities or language models (e.g. Frank, Otten, Galli, & Vigliocco, 2015) is related to processing effort, in that less expected words take more effort to process (e.g. Hale, 2001). A words surprisal, however, may also be influenced by the non-linguistic context, such as visual cues: In the visual world paradigm (VWP), for example, anticipatory eye movements suggest that comprehenders exploit the scene to predict what will be mentioned next (Altmann & Kamide, 1999).

How visual context affects word surprisal and processing effort, however, remains unclear. Here, we present evidence that visually-determined probabilistic expectations for a spoken target word predict graded processing effort for that word, in both pupillometric (ICA) and ERP (N400) measures. These findings demonstrate that the non-linguistic context can immediately influence both lexical expectations, and surprisal-based processing effort.

• https://www.researchgate.net/publication/325644935_Multimodal_Surprisal_in_the_N400_and_the_Index_of_Cognitive_Activity

@inproceedings{Ankener2018,
title = {Multimodal Surprisal in the N400 and the Index of Cognitive Activity},
author = {Christine Ankener and Heiner Drenhaus and Matthew W. Crocker and Maria Staudte},
url = {https://www.researchgate.net/publication/325644935_Multimodal_Surprisal_in_the_N400_and_the_Index_of_Cognitive_Activity},
year = {2018},
date = {2018},
booktitle = {Proceedings of the 40th Annual Cognitive Science Society Meeting},
pages = {94-100},
publisher = {The Cognitive Science Society},
address = {Madison, Wisconsin},
abstract = {A word’s predictability or surprisal, as determined by cloze probabilities or language models (e.g. Frank, Otten, Galli, & Vigliocco, 2015) is related to processing effort, in that less expected words take more effort to process (e.g. Hale, 2001). A words surprisal, however, may also be influenced by the non-linguistic context, such as visual cues: In the visual world paradigm (VWP), for example, anticipatory eye movements suggest that comprehenders exploit the scene to predict what will be mentioned next (Altmann & Kamide, 1999). How visual context affects word surprisal and processing effort, however, remains unclear. Here, we present evidence that visually-determined probabilistic expectations for a spoken target word predict graded processing effort for that word, in both pupillometric (ICA) and ERP (N400) measures. These findings demonstrate that the non-linguistic context can immediately influence both lexical expectations, and surprisal-based processing effort.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

In situated communication, reference can be established with expressions conveying either precise (Minimally-Specified, MS) or redundant (Over-Specified, OS) information. For example, while in Figure 1, “Find the blue ball” identifies exactly one object in all panels, only in the top displays is the adjective required. There is no consensus, however, concerning whether OS hinders processing (e.g., Engelhardt et al., 2011) or not (e.g., Tourtouri et al., 2015). Additionally, as incoming words incrementally restrict the referential domain, they contribute to the reduction of uncertainty regarding the target (i.e., referential entropy). Depending on the distribution of objects, the same utterance results in different entropy reduction profiles: “blue” reduces entropy by 1.58 bits in the right panels, and by .58 bits in the left ones, while “ball” reduces entropy by 1 and 2 bits, respectively. Thus, the adjective modulates the distribution of entropy reduction, resulting in uniform (UR) or non-uniform (NR) reduction profiles. This study seeks to establish whether referential processing is facilitated: a) by the use of redundant pre-nominal modification (OS), b) by the uniform reduction of entropy (cf. Jaeger, 2010), and c) when these two factors interact. Results from inspection probabilities and the Index of Cognitive Activity — a pupillometric measure of cognitive workload (Demberg & Sayeed, 2016) — indicate that processing was facilitated for both OS and UR, while fixation probabilities show a greater advantage for OS-UR. In conclusion, efficient processing is determined by both informativity of the reference and the rate of entropy reduction.

• https://www.researchgate.net/publication/322556329_The_interplay_of_specificity_and_referential_entropy_reduction_in_situated_communication

@inproceedings{Tourtourietal2017d,
title = {The interplay of specificity and referential entropy reduction in situated communication},
author = {Elli Tourtouri and Francesca Delogu and Matthew W. Crocker},
url = {https://www.researchgate.net/publication/322556329_The_interplay_of_specificity_and_referential_entropy_reduction_in_situated_communication},
year = {2017},
date = {2017},
booktitle = {10th Annual Embodied and Situated Language (ESLP) Conference},
publisher = {Higher School of Economics},
address = {Moscow, Russia},
abstract = {In situated communication, reference can be established with expressions conveying either precise (Minimally-Specified, MS) or redundant (Over-Specified, OS) information. For example, while in Figure 1, “Find the blue ball” identifies exactly one object in all panels, only in the top displays is the adjective required. There is no consensus, however, concerning whether OS hinders processing (e.g., Engelhardt et al., 2011) or not (e.g., Tourtouri et al., 2015). Additionally, as incoming words incrementally restrict the referential domain, they contribute to the reduction of uncertainty regarding the target (i.e., referential entropy). Depending on the distribution of objects, the same utterance results in different entropy reduction profiles: “blue” reduces entropy by 1.58 bits in the right panels, and by .58 bits in the left ones, while “ball” reduces entropy by 1 and 2 bits, respectively. Thus, the adjective modulates the distribution of entropy reduction, resulting in uniform (UR) or non-uniform (NR) reduction profiles. This study seeks to establish whether referential processing is facilitated: a) by the use of redundant pre-nominal modification (OS), b) by the uniform reduction of entropy (cf. Jaeger, 2010), and c) when these two factors interact. Results from inspection probabilities and the Index of Cognitive Activity — a pupillometric measure of cognitive workload (Demberg & Sayeed, 2016) — indicate that processing was facilitated for both OS and UR, while fixation probabilities show a greater advantage for OS-UR. In conclusion, efficient processing is determined by both informativity of the reference and the rate of entropy reduction.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

Over-specifications (OS) are expressions that provide more information than minimally required for the identification of a referent, thereby violating Grice’s 2nd Quantity Maxim [1]. In Figure 1, for example, the expression “Find the blue ball” identifies exactly one object in all panels, but only in the top displays is the adjective required to disambiguate the target. In recent years, psycholinguistic research has tried to test the empirical validity of Grice’s Maxim, resulting in conflicting findings. That is, there is evidence both that OS hinders [2,3] and that it facilitates [4,5] referential processing. The current study investigates the influence of OS on visually-situated processing, when the context allows both a minimally-specified (MS) and an OS interpretation of pre-nominal adjectives (cf. Fig.1). Additionally, as the utterance unfolds over time, incoming words incrementally restrict the search space. In this sense, information on “blue” and “ball” is determined not only by their probability to occur in this context, but also by the amount of uncertainty about the target they reduce — in information theoretic terms [6]. A greater reduction of the referential set size on the adjective (A&C) results in a more uniform reduction profile (Uniform Reduction, UR), as the adjective reduces entropy by 1.58 bits and the noun by 1 bit. On the other hand, a moderate reduction of the set size on the adjective (B&D) results in a less uniform reduction profile (Nonuniform Reduction, NR): the adjective reduces entropy by .58 bits and the noun by 2 bits. This study examines whether, above and beyond any effects of specificity, the rate at which incoming words reduce visual entropy also affects referential processing.

• TourtouriDeloguCrocker     (0.54MB)
• https://www.researchgate.net/publication/322571202_Over-specification_Uniform_Reduction_of_Visual_Entropy_Facilitate_Referential_Processing

@inproceedings{Tourtourietal2017b,
title = {Overspecification and uniform reduction of visual entropy facilitate referential processing},
author = {Elli Tourtouri and Francesca Delogu and Matthew W. Crocker},
url = {https://www.researchgate.net/publication/322571202_Over-specification_Uniform_Reduction_of_Visual_Entropy_Facilitate_Referential_Processing},
year = {2017},
date = {2017},
booktitle = {XPrag2017},
address = {Cologne, Germany},
abstract = {Over-specifications (OS) are expressions that provide more information than minimally required for the identification of a referent, thereby violating Grice’s 2nd Quantity Maxim [1]. In Figure 1, for example, the expression “Find the blue ball” identifies exactly one object in all panels, but only in the top displays is the adjective required to disambiguate the target. In recent years, psycholinguistic research has tried to test the empirical validity of Grice’s Maxim, resulting in conflicting findings. That is, there is evidence both that OS hinders [2,3] and that it facilitates [4,5] referential processing. The current study investigates the influence of OS on visually-situated processing, when the context allows both a minimally-specified (MS) and an OS interpretation of pre-nominal adjectives (cf. Fig.1). Additionally, as the utterance unfolds over time, incoming words incrementally restrict the search space. In this sense, information on “blue” and “ball” is determined not only by their probability to occur in this context, but also by the amount of uncertainty about the target they reduce — in information theoretic terms [6]. A greater reduction of the referential set size on the adjective (A&C) results in a more uniform reduction profile (Uniform Reduction, UR), as the adjective reduces entropy by 1.58 bits and the noun by 1 bit. On the other hand, a moderate reduction of the set size on the adjective (B&D) results in a less uniform reduction profile (Nonuniform Reduction, NR): the adjective reduces entropy by .58 bits and the noun by 2 bits. This study examines whether, above and beyond any effects of specificity, the rate at which incoming words reduce visual entropy also affects referential processing.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard