Publications

Given the sentence “On sunny days, Rose rides to work with her …”, it is likely that you predict the word “bicycle” before reading it. Notably, not only adults, but even children from an early age predict language, which is seen as one reason of why language comprehension is remarkably fast and accurate. In everyday-life language is usually received in visual contexts which can influence what a comprehender predicts. Imagine processing the above sentence while looking at the picture of a bicycle. This could make you even more likely to predict the noun “bicycle”. Thus, prediction research often applies the Visual World Paradigm. Here, participants listen to predictable sentences like the above while looking at visual scenes that show one visual prediction option that is (e.g., bicycle) and one distractor object that is not (e.g., cake) consistent with the predictive sentence context. When participants show an increase in fixations to the visual prediction option after the predictive cue was played (e.g., “ride”), but prior to the target noun, this indexes prediction. Cognitive models argue that visually situated prediction involves two mechanisms. Predictive linguistic cues (e.g., the semantically constraining verb “ride”) cause the pre-activation of the mental representations of prediction options such as “bicycle” in long-term memory. If a visual context allows to commit to a prediction option, this option is pre-updated (i.e., pre-processed) in working memory. Given this, individual differences in verbal and cognitive abilities could influence visually situated prediction. That is, language experience could determine which long-term memory representations can be pre-activated, while working memory capacity could affect the ability to pre-update prediction options. Since children have smaller language experience and working memory capacity than adults, we used a developmental approach and compared children and adults in their prediction behavior in the visual world to test the above model assumptions. First, we compared children and adults in their ability to make multiple predictions in parallel. With the Visual World Paradigm, adults have already been shown to rely on visual contexts to make multiple predictions: When hearing the sentence (“Rose rides to work with her …”) while looking at multiple “ridable” objects, adults have been shown to predict up to four sentence continuations in parallel. We examined whether also children can follow a multiple predictions pattern, or whether their limited language experience and cognitive capacity prevent them from doing so. Besides, since working memory engages more mental resources when more stimuli are processed, we examined whether children and adults show an increase in cognitive load to pre-update multiple versus only single prediction options in working memory. We examined whether this effect is more prominent in children given their smaller cognitive capacity. We finally investigated whether processing load of a predictable target word (e.g., “bicycle”) is smaller when that word was pre-updated alone or among multiple competitors. In Chapter 1 we outline the theoretical background of this work. This is followed by an empirical section that addresses the above questions. We conducted two studies in which children and adults were presented with sentences with semantically constraining verbs and predictable target nouns (e.g., “The father eats the waffle”) in visual scenes of four object pictures each. Across four conditions, the scenes varied in predictability: Either 0, 1, 3, or 4 visual objects were consistent with the verb constraints and thus viewed as visual prediction options. Chapter 2 shows a pretest of the sentences and the scenes with young children (4–6 years). Experiment 1 was an eye-tracking study in which children (5–6) and adults listened to the sentences while looking at the visual scenes. In Chapter 3, we used their anticipatory object fixations as an index of prediction behavior. Chapter 4 presents data collected in the same study. Here, the Index of Cognitive Activity (ICA) and pupil sizes were used as a measure of cognitive load engaged in sentence processing in the different visual conditions. Chapter 5 presents Experiment 2, where literate children (8–12 years) and adults were presented with the same sentences and scenes in a self-paced reading task. They read the sentences word-by-word while inspecting the scenes. We relied on word processing times as an index of cognitive load. Their anticipatory object fixations (Experiment 1) showed that children and adults followed a multiple predictions pattern. For children, this ability was positively related to their language experience, supporting the view that prediction involves the pre-activation of mental representations in long-term memory. We found no consistent evidence of whether children and adults engaged higher cognitive load to make multiple predictions. Both age groups’ ICA and pupil size values did not (Experiment 1) but their word processing times did (Experiment 2) suggest additional processing costs for multiple predictions. The latter result is in line with the view that prediction involves the pre-updating of input in the cognitive system. Finally, both studies found children and adults to engage less processing load for target nouns that could be pre-updated alone versus among multiple competitors. In sum, we provide indication that visual contexts can influence the ease of (predictive) language processing, which is discussed beyond cognitive perspectives of prediction in Chapter 6. Here, we also consider which questions about predictive language processing still remain open, in particular for children.

---

Stellen Sie sich folgenden Satz vor: „Um an sonnigen Tagen zur Arbeit zu kommen, fährt Rosa mit ihrem …“. Vermutlich haben Sie das Wort „Fahrrad“ antizipiert, ohne es gelesen zu haben. Dies wird prädiktive Sprachverarbeitung genannt und als ein Grund für die enorme Genauigkeit und Geschwindigkeit des Sprachverständnisses gesehen. Bemerkenswerterweise weisen nicht nur Erwachsene, sondern auch Kinder, die Fähigkeit zur sprachlichen Vorhersage auf. Im Alltag wird Sprache oft in visuellen Kontexten rezipiert, welche die Vorhersage beeinflussen. Stellen Sie sich vor, Sie hören obigen Satz, während Sie das Bild eines Fahrrades betrachten. Dies könnte die Wahrscheinlichkeit erhöhen, dass Sie das Wort „Fahrrad“ vorhersagen. Empirische Studien zur sprachlichen Vorhersage nutzen daher häufig das Visual World Paradigma. Hier hören Versuchspersonen vorhersagbare Sätze, wie den obigen, während sie visuelle Szenen betrachten. Diese zeigen typischerweise eine visuelle Vorhersageoption (z.B. das Bild eines Fahrrades) und ein weiteres Objekt, das inkonsistent mit dem prädiktiven Satzkontext ist (z.B. das Bild eines Kuchens). Dieses Paradigma weist sprachliche Vorhersage nach, wenn Versuchspersonen bereits nach dem prädiktive Hinweisreiz (z.B. „fahren“) und vor dem Zielwort (z.B. „Fahrrad“) einen Anstieg an Fixationen der visuellen Vorhersageoption im Vergleich zum inkonsistenten Objekt zeigen. Kognitive Modelle postulieren, dass zwei Mechanismen an der Vorhersage im visuellen Kontext beteiligt sind. Prädiktive sprachliche Hinweisreize (z.B. das Verb „fahren“) erwirken die Voraktivierung von Vorhersageoptionen (z.B. Fortbewegungsmitteln) im Langzeitgedächtnis. Wenn zudem eine visuelle Vorhersageoption verfügbar ist (z.B. das Bild eines Fahrrades), wird diese Option im Arbeitsgedächtnis vorverarbeitet. Infolgedessen könnten verbale und kognitive Fähigkeiten die sprachliche Vorhersage im visuellen Kontext beeinflussen. So könnte die Spracherfahrung bestimmen, welche Informationen im Langzeitgedächtnis voraktiviert werden können. Die Arbeitsgedächtniskapazität hingegen könnte die Fähigkeit zur Vorverarbeitung von Vorhersageoptionen beeinflussen. Da Kinder im Vergleich zu Erwachsenen über eine geringere Spracherfahrung sowie Kapazität des Arbeitsgedächtnisses verfügen, nutzte diese Arbeit einen entwicklungspsychologischen Ansatz, um obige Annahmen zur sprachlichen Vorhersage zu prüfen. Zunächst wurden Kinder und Erwachsene in ihrer Fähigkeit verglichen, mehrere Vorhersagen gleichzeitig zu treffen. Mit dem Visual World Paradigma wurde bereits gezeigt, dass Erwachsene visuelle Kontexte nutzen, um mehrere Vorhersagen zu treffen: Erwachsene, die obigen Beispielsatz hören und gleichzeitig mehrere „fahrbare“ Objekte betrachten, konnten nachweislich bis zu vier potentielle Zielwörter gleichzeitig vorhersagen. Diese Arbeit untersucht, ob auch Kinder mehrere Vorhersagen gleichzeig treffen oder ob ihre geringe Spracherfahrung und kognitive Kapazität ein solches Muster der Vorhersage einschränken. Weiterhin wird geprüft, ob Kinder und Erwachsene eine höhere kognitive Belastung zeigen, wenn sie mehrere, statt nur einer Vorhersageoption, vorverarbeiten. Dies wäre plausibel, da das Arbeitsgedächtnis in der Regel mehr mentale Ressourcen beansprucht, wenn es mehr Informationen verarbeitet. Zudem wird untersucht, ob dieser Effekt bei Kindern aufgrund ihrer geringen kognitiven Kapazität stärker ausgeprägt ist als bei Erwachsenen. Zuletzt wird ermittelt, ob mehr mentale Ressourcen zur Verarbeitung eines Zielwortes benötigt werden, wenn dieses Wort mit weiteren Vorhersageoptionen (statt als einzige Option) vorverarbeitet wurde. Kapitel 1 präsentiert den theoretischen Hintergrund dieser Arbeit. Es folgt ein empirischer Teil, in dem obige Fragen adressiert werden. Dieser umfasst zwei Studien, in denen Kindern und Erwachsenen Sätze mit prädiktiven Verben und Zielwörtern gezeigt wurden (z.B. „Der Vater isst die Waffel“). Die Sätze wurden zusammen mit visuellen Szenen präsentiert, die jeweils vier Bilder von Objekten zeigten. Die Szenen variierten in ihrer Vorhersagbarkeit: Basierend auf dem prädiktiven Verb stellten 0, 1, 3 oder 4 der Objekte eine visuelle Vorhersageoption dar. Kapitel 2 zeigt eine Studie, in der die Sätze und Szenen mit Kindern (4–6 Jahre) normiert wurden. Experiment 1 war eine Eye-Tracking Studie, in der Kinder (5–6 Jahre) und Erwachsene die Szenen betrachteten, während ihnen die Sätze vorgespielt wurden. In Kapitel 3 wurden die Objektfixationen der Versuchspersonen als Index für das Vorhersageverhalten verwendet. Kapitel 4 präsentiert Daten, die in derselben Studie erhoben wurden. Hier wurde die Pupillengröße sowie der Index of Cognitive Activity (ICA) als Maß für die kognitive Belastung der Satzverarbeitung in den verschiedenen visuellen Konditionen verwendet. Kapitel 5 präsentiert Experiment 2. Hier wurden Kindern (8–12 Jahre) und Erwachsenen dieselben Sätze und Szenen präsentiert, jedoch wurden die Sätze auf dem Bildschirm innerhalb der Szenen gezeigt und Wort für Wort gelesen. Die Wortverarbeitungszeit wurde als Maß für die kognitive Belastung gewertet. Anhand der Objektfixationen zeigte Experiment 1, dass beide Altersgruppen mehrere Vorhersagen gleichzeitig trafen. Bei Kindern stand diese Fähigkeit in positiver Relation zu ihrer Spracherfahrung. Wir fanden keine konsistente Evidenz, dass Kinder und Erwachsene eine höhere kognitive Belastung zeigen, wenn sie mehrere Vorhersagen gleichzeitig treffen. Dieser Effekt wurde durch die Wortverarbeitungszeiten beider Altersgruppen nachgewiesen (Experiment 2), nicht jedoch durch ihre Pupillengrößen und ICA-Daten (Experiment 1). In beiden Studien zeigten Kinder und Erwachsene eine höhere kognitive Belastung bei der Verarbeitung von Zielwörtern, die mit mehreren Vorhersageoptionen (statt als einzige Option) antizipiert wurden. Insgesamt zeigen die Ergebnisse dieser Arbeit, dass visuelle Kontexte einen Einfluss auf die prädiktive Sprachverarbeitung und ihre Leichtigkeit haben können. Dies wird in Kapitel 6 vor dem Hintergrund kognitiver Modelle der Vorhersage diskutiert. Hier werden zudem offene Fragen zur sprachlichen Vorhersage, insbesondere bei Kindern, thematisiert.

• Dissertation_Sommerfeld_15.05.24_FINAL_online published version     (3.57MB)
• https://jahrbib.sulb.uni-saarland.de/handle/20.500.11880/37808
• https://doi.org/10.22028/D291-42078

@phdthesis{Sommerfeld_Diss,
title = {Predictive language processing in the complex visual world in children and adults},
author = {Linda Sommerfeld},
url = {https://jahrbib.sulb.uni-saarland.de/handle/20.500.11880/37808},
doi = {https://doi.org/10.22028/D291-42078},
year = {2024},
date = {2024},
school = {Saarland University},
publisher = {Saarl{\"a}ndische Universit{\"a}ts- und Landesbibliothek},
address = {Saarbruecken, Germany},
abstract = {Given the sentence “On sunny days, Rose rides to work with her …”, it is likely that you predict the word “bicycle” before reading it. Notably, not only adults, but even children from an early age predict language, which is seen as one reason of why language comprehension is remarkably fast and accurate. In everyday-life language is usually received in visual contexts which can influence what a comprehender predicts. Imagine processing the above sentence while looking at the picture of a bicycle. This could make you even more likely to predict the noun “bicycle”. Thus, prediction research often applies the Visual World Paradigm. Here, participants listen to predictable sentences like the above while looking at visual scenes that show one visual prediction option that is (e.g., bicycle) and one distractor object that is not (e.g., cake) consistent with the predictive sentence context. When participants show an increase in fixations to the visual prediction option after the predictive cue was played (e.g., “ride”), but prior to the target noun, this indexes prediction. Cognitive models argue that visually situated prediction involves two mechanisms. Predictive linguistic cues (e.g., the semantically constraining verb “ride”) cause the pre-activation of the mental representations of prediction options such as “bicycle” in long-term memory. If a visual context allows to commit to a prediction option, this option is pre-updated (i.e., pre-processed) in working memory. Given this, individual differences in verbal and cognitive abilities could influence visually situated prediction. That is, language experience could determine which long-term memory representations can be pre-activated, while working memory capacity could affect the ability to pre-update prediction options. Since children have smaller language experience and working memory capacity than adults, we used a developmental approach and compared children and adults in their prediction behavior in the visual world to test the above model assumptions. First, we compared children and adults in their ability to make multiple predictions in parallel. With the Visual World Paradigm, adults have already been shown to rely on visual contexts to make multiple predictions: When hearing the sentence (“Rose rides to work with her …”) while looking at multiple “ridable” objects, adults have been shown to predict up to four sentence continuations in parallel. We examined whether also children can follow a multiple predictions pattern, or whether their limited language experience and cognitive capacity prevent them from doing so. Besides, since working memory engages more mental resources when more stimuli are processed, we examined whether children and adults show an increase in cognitive load to pre-update multiple versus only single prediction options in working memory. We examined whether this effect is more prominent in children given their smaller cognitive capacity. We finally investigated whether processing load of a predictable target word (e.g., “bicycle”) is smaller when that word was pre-updated alone or among multiple competitors. In Chapter 1 we outline the theoretical background of this work. This is followed by an empirical section that addresses the above questions. We conducted two studies in which children and adults were presented with sentences with semantically constraining verbs and predictable target nouns (e.g., “The father eats the waffle”) in visual scenes of four object pictures each. Across four conditions, the scenes varied in predictability: Either 0, 1, 3, or 4 visual objects were consistent with the verb constraints and thus viewed as visual prediction options. Chapter 2 shows a pretest of the sentences and the scenes with young children (4–6 years). Experiment 1 was an eye-tracking study in which children (5–6) and adults listened to the sentences while looking at the visual scenes. In Chapter 3, we used their anticipatory object fixations as an index of prediction behavior. Chapter 4 presents data collected in the same study. Here, the Index of Cognitive Activity (ICA) and pupil sizes were used as a measure of cognitive load engaged in sentence processing in the different visual conditions. Chapter 5 presents Experiment 2, where literate children (8–12 years) and adults were presented with the same sentences and scenes in a self-paced reading task. They read the sentences word-by-word while inspecting the scenes. We relied on word processing times as an index of cognitive load. Their anticipatory object fixations (Experiment 1) showed that children and adults followed a multiple predictions pattern. For children, this ability was positively related to their language experience, supporting the view that prediction involves the pre-activation of mental representations in long-term memory. We found no consistent evidence of whether children and adults engaged higher cognitive load to make multiple predictions. Both age groups’ ICA and pupil size values did not (Experiment 1) but their word processing times did (Experiment 2) suggest additional processing costs for multiple predictions. The latter result is in line with the view that prediction involves the pre-updating of input in the cognitive system. Finally, both studies found children and adults to engage less processing load for target nouns that could be pre-updated alone versus among multiple competitors. In sum, we provide indication that visual contexts can influence the ease of (predictive) language processing, which is discussed beyond cognitive perspectives of prediction in Chapter 6. Here, we also consider which questions about predictive language processing still remain open, in particular for children.

---

Stellen Sie sich folgenden Satz vor: „Um an sonnigen Tagen zur Arbeit zu kommen, f{\"a}hrt Rosa mit ihrem ...“. Vermutlich haben Sie das Wort „Fahrrad“ antizipiert, ohne es gelesen zu haben. Dies wird pr{\"a}diktive Sprachverarbeitung genannt und als ein Grund f{\"u}r die enorme Genauigkeit und Geschwindigkeit des Sprachverst{\"a}ndnisses gesehen. Bemerkenswerterweise weisen nicht nur Erwachsene, sondern auch Kinder, die F{\"a}higkeit zur sprachlichen Vorhersage auf. Im Alltag wird Sprache oft in visuellen Kontexten rezipiert, welche die Vorhersage beeinflussen. Stellen Sie sich vor, Sie h{\"o}ren obigen Satz, w{\"a}hrend Sie das Bild eines Fahrrades betrachten. Dies k{\"o}nnte die Wahrscheinlichkeit erh{\"o}hen, dass Sie das Wort „Fahrrad“ vorhersagen. Empirische Studien zur sprachlichen Vorhersage nutzen daher h{\"a}ufig das Visual World Paradigma. Hier h{\"o}ren Versuchspersonen vorhersagbare S{\"a}tze, wie den obigen, w{\"a}hrend sie visuelle Szenen betrachten. Diese zeigen typischerweise eine visuelle Vorhersageoption (z.B. das Bild eines Fahrrades) und ein weiteres Objekt, das inkonsistent mit dem pr{\"a}diktiven Satzkontext ist (z.B. das Bild eines Kuchens). Dieses Paradigma weist sprachliche Vorhersage nach, wenn Versuchspersonen bereits nach dem pr{\"a}diktive Hinweisreiz (z.B. „fahren“) und vor dem Zielwort (z.B. „Fahrrad“) einen Anstieg an Fixationen der visuellen Vorhersageoption im Vergleich zum inkonsistenten Objekt zeigen. Kognitive Modelle postulieren, dass zwei Mechanismen an der Vorhersage im visuellen Kontext beteiligt sind. Pr{\"a}diktive sprachliche Hinweisreize (z.B. das Verb „fahren“) erwirken die Voraktivierung von Vorhersageoptionen (z.B. Fortbewegungsmitteln) im Langzeitged{\"a}chtnis. Wenn zudem eine visuelle Vorhersageoption verf{\"u}gbar ist (z.B. das Bild eines Fahrrades), wird diese Option im Arbeitsged{\"a}chtnis vorverarbeitet. Infolgedessen k{\"o}nnten verbale und kognitive F{\"a}higkeiten die sprachliche Vorhersage im visuellen Kontext beeinflussen. So k{\"o}nnte die Spracherfahrung bestimmen, welche Informationen im Langzeitged{\"a}chtnis voraktiviert werden k{\"o}nnen. Die Arbeitsged{\"a}chtniskapazit{\"a}t hingegen k{\"o}nnte die F{\"a}higkeit zur Vorverarbeitung von Vorhersageoptionen beeinflussen. Da Kinder im Vergleich zu Erwachsenen {\"u}ber eine geringere Spracherfahrung sowie Kapazit{\"a}t des Arbeitsged{\"a}chtnisses verf{\"u}gen, nutzte diese Arbeit einen entwicklungspsychologischen Ansatz, um obige Annahmen zur sprachlichen Vorhersage zu pr{\"u}fen. Zun{\"a}chst wurden Kinder und Erwachsene in ihrer F{\"a}higkeit verglichen, mehrere Vorhersagen gleichzeitig zu treffen. Mit dem Visual World Paradigma wurde bereits gezeigt, dass Erwachsene visuelle Kontexte nutzen, um mehrere Vorhersagen zu treffen: Erwachsene, die obigen Beispielsatz h{\"o}ren und gleichzeitig mehrere „fahrbare“ Objekte betrachten, konnten nachweislich bis zu vier potentielle Zielw{\"o}rter gleichzeitig vorhersagen. Diese Arbeit untersucht, ob auch Kinder mehrere Vorhersagen gleichzeig treffen oder ob ihre geringe Spracherfahrung und kognitive Kapazit{\"a}t ein solches Muster der Vorhersage einschr{\"a}nken. Weiterhin wird gepr{\"u}ft, ob Kinder und Erwachsene eine h{\"o}here kognitive Belastung zeigen, wenn sie mehrere, statt nur einer Vorhersageoption, vorverarbeiten. Dies w{\"a}re plausibel, da das Arbeitsged{\"a}chtnis in der Regel mehr mentale Ressourcen beansprucht, wenn es mehr Informationen verarbeitet. Zudem wird untersucht, ob dieser Effekt bei Kindern aufgrund ihrer geringen kognitiven Kapazit{\"a}t st{\"a}rker ausgepr{\"a}gt ist als bei Erwachsenen. Zuletzt wird ermittelt, ob mehr mentale Ressourcen zur Verarbeitung eines Zielwortes ben{\"o}tigt werden, wenn dieses Wort mit weiteren Vorhersageoptionen (statt als einzige Option) vorverarbeitet wurde. Kapitel 1 pr{\"a}sentiert den theoretischen Hintergrund dieser Arbeit. Es folgt ein empirischer Teil, in dem obige Fragen adressiert werden. Dieser umfasst zwei Studien, in denen Kindern und Erwachsenen S{\"a}tze mit pr{\"a}diktiven Verben und Zielw{\"o}rtern gezeigt wurden (z.B. „Der Vater isst die Waffel“). Die S{\"a}tze wurden zusammen mit visuellen Szenen pr{\"a}sentiert, die jeweils vier Bilder von Objekten zeigten. Die Szenen variierten in ihrer Vorhersagbarkeit: Basierend auf dem pr{\"a}diktiven Verb stellten 0, 1, 3 oder 4 der Objekte eine visuelle Vorhersageoption dar. Kapitel 2 zeigt eine Studie, in der die S{\"a}tze und Szenen mit Kindern (4–6 Jahre) normiert wurden. Experiment 1 war eine Eye-Tracking Studie, in der Kinder (5–6 Jahre) und Erwachsene die Szenen betrachteten, w{\"a}hrend ihnen die S{\"a}tze vorgespielt wurden. In Kapitel 3 wurden die Objektfixationen der Versuchspersonen als Index f{\"u}r das Vorhersageverhalten verwendet. Kapitel 4 pr{\"a}sentiert Daten, die in derselben Studie erhoben wurden. Hier wurde die Pupillengr{\"o}{\ss}e sowie der Index of Cognitive Activity (ICA) als Ma{\ss} f{\"u}r die kognitive Belastung der Satzverarbeitung in den verschiedenen visuellen Konditionen verwendet. Kapitel 5 pr{\"a}sentiert Experiment 2. Hier wurden Kindern (8–12 Jahre) und Erwachsenen dieselben S{\"a}tze und Szenen pr{\"a}sentiert, jedoch wurden die S{\"a}tze auf dem Bildschirm innerhalb der Szenen gezeigt und Wort f{\"u}r Wort gelesen. Die Wortverarbeitungszeit wurde als Ma{\ss} f{\"u}r die kognitive Belastung gewertet. Anhand der Objektfixationen zeigte Experiment 1, dass beide Altersgruppen mehrere Vorhersagen gleichzeitig trafen. Bei Kindern stand diese F{\"a}higkeit in positiver Relation zu ihrer Spracherfahrung. Wir fanden keine konsistente Evidenz, dass Kinder und Erwachsene eine h{\"o}here kognitive Belastung zeigen, wenn sie mehrere Vorhersagen gleichzeitig treffen. Dieser Effekt wurde durch die Wortverarbeitungszeiten beider Altersgruppen nachgewiesen (Experiment 2), nicht jedoch durch ihre Pupillengr{\"o}{\ss}en und ICA-Daten (Experiment 1). In beiden Studien zeigten Kinder und Erwachsene eine h{\"o}here kognitive Belastung bei der Verarbeitung von Zielw{\"o}rtern, die mit mehreren Vorhersageoptionen (statt als einzige Option) antizipiert wurden. Insgesamt zeigen die Ergebnisse dieser Arbeit, dass visuelle Kontexte einen Einfluss auf die pr{\"a}diktive Sprachverarbeitung und ihre Leichtigkeit haben k{\"o}nnen. Dies wird in Kapitel 6 vor dem Hintergrund kognitiver Modelle der Vorhersage diskutiert. Hier werden zudem offene Fragen zur sprachlichen Vorhersage, insbesondere bei Kindern, thematisiert.},
pubstate = {published},
type = {phdthesis}
}

Copy BibTeX to Clipboard

• https://srcd.onlinelibrary.wiley.com/doi/10.1111/cdep.12515
• https://doi.org/10.1111/cdep.12515

@article{Kray_etal_2024,
title = {The role of prediction error in the development of language learning and memory},
author = {Jutta Kray and Linda Sommerfeld and Arielle Borovsky and Katja H{\"a}user},
url = {https://srcd.onlinelibrary.wiley.com/doi/10.1111/cdep.12515},
doi = {https://doi.org/10.1111/cdep.12515},
year = {2024},
date = {2024},
journal = {Child Development Perspectives},
pages = {1–14},
abstract = {

},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

• https://www.sciencedirect.com/science/article/pii/S0022096523000668
• https://doi.org/10.1016/j.jecp.2023.105690

@article{Sommerfeld_etal_children_2023,
title = {Even young children make multiple predictions in the complex visual world},
author = {Linda Sommerfeld and Maria Staudte and Nivedita Mani and Jutta Kray},
url = {https://www.sciencedirect.com/science/article/pii/S0022096523000668},
doi = {https://doi.org/10.1016/j.jecp.2023.105690},
year = {2023},
date = {2023},
journal = {Journal of Experimental Child Psychology},
volume = {235},
number = {105690},
abstract = {

},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

Developmental and longitudinal studies with children increasingly use pictorial stimuli in cognitive, psychologic, and psycholinguistic research. To enhance validity and comparability within and across those studies, the use of normed pictures is recommended. Besides, creating picture sets and evaluating them in rating studies is very time consuming, in particular regarding samples of young children in which testing time is rather limited. As there is an increasing number of studies that investigate young German children’s semantic language processing with colored clipart stimuli, this work provides a first set of 247 colored cliparts with ratings of German native speaking children aged 4 to 6 years. We assessed two central rating aspects of pictures: Name agreement (Do pictures elicit the intended name of an object?) and semantic categorization (Are objects classified as members of the intended semantic category?). Our ratings indicate that children are proficient in naming and even better in semantic categorization of objects, whereas both seems to improve with increasing age of young childhood. Finally, this paper discusses some features of pictorial objects that might be important for children’s name agreement and semantic categorization and could be considered in future picture rating studies.

 

• 1-s2.0-S0001691822000737-main     (0.78MB)
• https://www.sciencedirect.com/science/article/pii/S0001691822000737
• https://doi.org/https://doi.org/10.1016/j.actpsy.2022.103558

@article{Sommerfeld_of_2022,
title = {Ratings of name agreement and semantic categorization of 247 colored clipart pictures by young German children},
author = {Linda Sommerfeld and Maria Staudte and Jutta Kray},
url = {https://www.sciencedirect.com/science/article/pii/S0001691822000737},
doi = {https://doi.org/https://doi.org/10.1016/j.actpsy.2022.103558},
year = {2022},
date = {2022},
journal = {Acta Psychologica},
pages = {103558},
volume = {226},
abstract = {Developmental and longitudinal studies with children increasingly use pictorial stimuli in cognitive, psychologic, and psycholinguistic research. To enhance validity and comparability within and across those studies, the use of normed pictures is recommended. Besides, creating picture sets and evaluating them in rating studies is very time consuming, in particular regarding samples of young children in which testing time is rather limited. As there is an increasing number of studies that investigate young German children's semantic language processing with colored clipart stimuli, this work provides a first set of 247 colored cliparts with ratings of German native speaking children aged 4 to 6 years. We assessed two central rating aspects of pictures: Name agreement (Do pictures elicit the intended name of an object?) and semantic categorization (Are objects classified as members of the intended semantic category?). Our ratings indicate that children are proficient in naming and even better in semantic categorization of objects, whereas both seems to improve with increasing age of young childhood. Finally, this paper discusses some features of pictorial objects that might be important for children's name agreement and semantic categorization and could be considered in future picture rating studies.},
keywords = {Name agreement, Semantic categorization, Picture naming, Picture ratings, Children, Age differences},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

Semantic extension plays a key role in language change and grammaticalisation. Here we use a dyadic interaction paradigm to study semantic extension of novel labels in controlled circumstances. We ask whether participants will be able to (i) use highly accessible associations in the perceptual environment (colour-shape associations) to converge on a meaning for the novel labels, and (ii) extend these meanings to apply to both concrete targets (objects) and abstract targets (emotions). Further, given the argument that both metonymy and metaphor are important drivers of language change, we investigate whether participants will be able to draw on relations of contiguity (‘metonymic’ associations, e.g. colour-shape or object-colour) and relations of similarity (‘metaphorical’ associations, e.g. emotion-colour) to extend the meaning of labels.

• eScholarship UC item 7w22b8gm     (0.34MB)
• https://escholarship.org/uc/item/7w22b8gm

@inproceedings{HaeuserKray2022,
title = {Uninvited and unwanted: False memories for words predicted but not seen},
author = {Katja H{\"a}user and Jutta Kray},
editor = {Jennifer Culbertson and Andrew Perfors and Hugh Rabagliati and Veronica Ramenzoni},
url = {https://escholarship.org/uc/item/7w22b8gm},
year = {2022},
date = {2022},
booktitle = {Proceedings of the 44th Annual Conference of the Cognitive Science Society, Toronto, Canada (27 Jul 2022 - 30 Jul 2022)},
pages = {2401-2408},
abstract = {Semantic extension plays a key role in language change and grammaticalisation. Here we use a dyadic interaction paradigm to study semantic extension of novel labels in controlled circumstances. We ask whether participants will be able to (i) use highly accessible associations in the perceptual environment (colour-shape associations) to converge on a meaning for the novel labels, and (ii) extend these meanings to apply to both concrete targets (objects) and abstract targets (emotions). Further, given the argument that both metonymy and metaphor are important drivers of language change, we investigate whether participants will be able to draw on relations of contiguity (‘metonymic’ associations, e.g. colour-shape or object-colour) and relations of similarity (‘metaphorical’ associations, e.g. emotion-colour) to extend the meaning of labels.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

• collabra_2022_8_1_36945     (1.55MB)
• https://online.ucpress.edu/collabra/article/8/1/36945/187814/Hedging-Bets-in-Linguistic-Prediction-Younger-and
• https://doi.org/10.1525/collabra.36945

@article{Haeuseretal22,
title = {Hedging Bets in Linguistic Prediction: Younger and Older Adults Vary in the Breadth of Predictive Processing},
author = {Katja H{\"a}user and Jutta Kray and Arielle Borovsky},
url = {https://online.ucpress.edu/collabra/article/8/1/36945/187814/Hedging-Bets-in-Linguistic-Prediction-Younger-and},
doi = {https://doi.org/10.1525/collabra.36945},
year = {2022},
date = {2022},
journal = {Collabra: Psychology},
volume = {8(1):36945},
abstract = {

},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

How do violations of predictability and plausibility affect online language processing? How does it affect longer-term memory and learning when predictions are disconfirmed by plausible or implausible words? We investigated these questions using a self-paced sentence reading and noun recognition task. Critical sentences violated predictability or plausibility or both, for example, “Since Anne is afraid of spiders, she doesn’t like going down into the … basement (predictable, plausible), garden (unpredictable, somewhat plausible), moon (unpredictable, deeply implausible).” Results from sentence reading showed earlier-emerging effects of predictability violations on the critical noun, but later-emerging effects of plausibility violations after the noun. Recognition memory was exclusively enhanced for deeply implausible nouns. The earlier-emerging predictability effect indicates that having word form predictions disconfirmed is registered very early in the processing stream, irrespective of semantics. The later-emerging plausibility effect supports models that argue for a staged architecture of reading comprehension, where plausibility only affects a post-lexical integration stage. Our memory results suggest that, in order to facilitate memory and learning, a certain magnitude of prediction error is required.

• how-odd-diverging-effects-of-predictability-and-plausibility-violations-on-sentence-reading-and-word-memory     (0.53MB)
• https://www.cambridge.org/core/journals/applied-psycholinguistics/article/how-odd-diverging-effects-of-predictability-and-plausibility-violations-on-sentence-reading-and-word-memory/D8E12864E47CE24E62297ABF5BA2BED0
• https://doi.org/10.1017/S0142716422000364

@article{HaeuserKray22,
title = {How odd: Diverging effects of predictability and plausibility violations on sentence reading and word memory},
author = {Katja H{\"a}user and Jutta Kray},
url = {https://www.cambridge.org/core/journals/applied-psycholinguistics/article/how-odd-diverging-effects-of-predictability-and-plausibility-violations-on-sentence-reading-and-word-memory/D8E12864E47CE24E62297ABF5BA2BED0},
doi = {https://doi.org/10.1017/S0142716422000364},
year = {2022},
date = {2022},
journal = {Applied Psycholinguistics},
pages = {1193-1220},
volume = {43(5)},
abstract = {How do violations of predictability and plausibility affect online language processing? How does it affect longer-term memory and learning when predictions are disconfirmed by plausible or implausible words? We investigated these questions using a self-paced sentence reading and noun recognition task. Critical sentences violated predictability or plausibility or both, for example, “Since Anne is afraid of spiders, she doesn’t like going down into the … basement (predictable, plausible), garden (unpredictable, somewhat plausible), moon (unpredictable, deeply implausible).” Results from sentence reading showed earlier-emerging effects of predictability violations on the critical noun, but later-emerging effects of plausibility violations after the noun. Recognition memory was exclusively enhanced for deeply implausible nouns. The earlier-emerging predictability effect indicates that having word form predictions disconfirmed is registered very early in the processing stream, irrespective of semantics. The later-emerging plausibility effect supports models that argue for a staged architecture of reading comprehension, where plausibility only affects a post-lexical integration stage. Our memory results suggest that, in order to facilitate memory and learning, a certain magnitude of prediction error is required.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

Not only adults, but already children anticipate upcoming input in sentences with semantically constraining verbs. In the visual world, the sentence context is used to anticipatorily fixate the only object matching potential sentence continuations. Adults process multiple visual cues in parallel when predicting language. Here, we examined whether young children can also process multiple visual objects fitting the constraining verb of a sentence. We also examined if their processing of multiple potential referents is affected by language skills, given that vocabulary size modulates children’s prediction.

• 119     (0.28MB)
• https://amlap2021.github.io/program/119.pdf

@miscellaneous{sommerfeld2021children,
title = {Children and adults integrate complex visual contexts in language prediction},
author = {Linda Sommerfeld and Maria Staudte and Nivedita Mani and Jutta Kray},
url = {https://amlap2021.github.io/program/119.pdf},
year = {2021},
date = {2021},
booktitle = {Architectures and Mechanisms for Language Processing AMLAP Annual Meeting},
abstract = {Not only adults, but already children anticipate upcoming input in sentences with semantically constraining verbs. In the visual world, the sentence context is used to anticipatorily fixate the only object matching potential sentence continuations. Adults process multiple visual cues in parallel when predicting language. Here, we examined whether young children can also process multiple visual objects fitting the constraining verb of a sentence. We also examined if their processing of multiple potential referents is affected by language skills, given that vocabulary size modulates children’s prediction.},
pubstate = {published},
type = {miscellaneous}
}

Copy BibTeX to Clipboard

The current work investigates how preexisting mental representations of the meaning of an utterance (top-down processing) affect the comprehension of external perceptual properties of the linguistic input (bottom-up processing). When it comes to top-down bottom-up processing in the brain previous findings report a division of focus between left and right hemispheric mechanisms. The PARLO sentence comprehension model posits that the LH employs top-down mechanisms which allow for efficient anticipatory processing, while the RH relies more on bottom-up mechanisms. A shortcoming of the PARLO model is that it’s based on experiments manipulating solely top-down contextual constraint, leading to conclusions that hemispheric asymmetries are a function of differences in the efficiency of top-down rather than bottom-up mechanisms. Up until now, there has been no investigation of asymmetries in bottom-up processing, nor an investigation of the potential interactions between that and top-down processing for each hemisphere. This thesis consists of four event-related potential (ERP) experiments divided into two parts. Experiments 1 (central presentation) and 2 (hemispheric presentation) manipulate the bottom-up lexical frequency of critical words in high and low predictability contexts. Experiments 3 (central presentation) and 4 (hemispheric presentation) manipulate bottom-up word status, presenting critical words and pseudowords in the same high and low predictability contexts. The results allow us to extend previous findings and present the Spotlight Theory of Hemispheric Comprehension. We argue that the LH employs a kind of spotlight focus, which affords very efficient top-down processing of the expected input, since only highly predictable inputs receive additional facilitation based their bottom-up features. Alternatively, the RH lack of spotlight mechanism and focus on bottom-up lexical properties allows for the reliable processing of less predictable and irregular inputs. In combination, these complementary processing strategies provide the comprehension system with the efficiency and robustness required in a wide range of communicative situations.

• https://publikationen.sulb.uni-saarland.de/handle/20.500.11880/31806
• https://doi.org/https://dx.doi.org/10.22028/D291-33976

@phdthesis{Vergilova_Diss_2021,
title = {The Lateralization of Expectations: Hemispheric Differences in Top-down and Bottom-up Word Processing in Context},
author = {Yoana Vergilova},
url = {https://publikationen.sulb.uni-saarland.de/handle/20.500.11880/31806},
doi = {https://doi.org/https://dx.doi.org/10.22028/D291-33976},
year = {2021},
date = {2021},
school = {Saarland University},
address = {Saarbruecken, Germany},
abstract = {The current work investigates how preexisting mental representations of the meaning of an utterance (top-down processing) affect the comprehension of external perceptual properties of the linguistic input (bottom-up processing). When it comes to top-down bottom-up processing in the brain previous findings report a division of focus between left and right hemispheric mechanisms. The PARLO sentence comprehension model posits that the LH employs top-down mechanisms which allow for efficient anticipatory processing, while the RH relies more on bottom-up mechanisms. A shortcoming of the PARLO model is that it’s based on experiments manipulating solely top-down contextual constraint, leading to conclusions that hemispheric asymmetries are a function of differences in the efficiency of top-down rather than bottom-up mechanisms. Up until now, there has been no investigation of asymmetries in bottom-up processing, nor an investigation of the potential interactions between that and top-down processing for each hemisphere. This thesis consists of four event-related potential (ERP) experiments divided into two parts. Experiments 1 (central presentation) and 2 (hemispheric presentation) manipulate the bottom-up lexical frequency of critical words in high and low predictability contexts. Experiments 3 (central presentation) and 4 (hemispheric presentation) manipulate bottom-up word status, presenting critical words and pseudowords in the same high and low predictability contexts. The results allow us to extend previous findings and present the Spotlight Theory of Hemispheric Comprehension. We argue that the LH employs a kind of spotlight focus, which affords very efficient top-down processing of the expected input, since only highly predictable inputs receive additional facilitation based their bottom-up features. Alternatively, the RH lack of spotlight mechanism and focus on bottom-up lexical properties allows for the reliable processing of less predictable and irregular inputs. In combination, these complementary processing strategies provide the comprehension system with the efficiency and robustness required in a wide range of communicative situations.},
pubstate = {published},
type = {phdthesis}
}

Copy BibTeX to Clipboard

Recently, Ankener et al. (Frontiers in Psychology, 9, 2387, 2018) presented a visual world study which combined both attention and pupillary measures to demonstrate that anticipating a target results in lower effort to integrate that target (noun). However, they found no indication that the anticipatory processes themselves, i.e., the reduction of uncertainty about upcoming referents, results in processing effort (cf. Linzen and Jaeger, Cognitive Science, 40(6), 1382–1411, 2016). In contrast, Maess et al. (Frontiers in Human Neuroscience, 10, 1–11, 2016) found that more constraining verbs elicited a higher N400 amplitude than unconstraining verbs. The aim of the present study was therefore twofold: Firstly, we examined whether the graded ICA effect, which was previously found on the noun as a result of a likelihood manipulation, replicates in ERP measures. Secondly, we set out to investigate whether the processes leading to the generation of expectations (derived during verb and scene processing) induce an N400 modulation. Our results confirm that visual context is combined with the verb’s meaning to establish expectations about upcoming nouns and that these expectations affect the retrieval of the upcoming noun (modulated N400 on the noun). Importantly, however, we find no evidence for different costs in generating more or less specific expectations for upcoming nouns. Thus, the benefits of generating expectations are not associated with any costs in situated language comprehension.

• https://link.springer.com/article/10.3758/s13423-020-01827-3
• https://doi.org/10.3758/s13423-020-01827-3

@article{staudte2021,
title = {Graded expectations in visually situated comprehension: Costs and benefits as indexed by the N400},
author = {Maria Staudte and Christine Ankener and Heiner Drenhaus and Matthew W. Crocker},
url = {https://link.springer.com/article/10.3758/s13423-020-01827-3},
doi = {https://doi.org/10.3758/s13423-020-01827-3},
year = {2021},
date = {2021},
journal = {Psychonomic Bulletin & Review},
pages = {624-631},
publisher = {Springer},
volume = {28},
number = {2},
abstract = {Recently, Ankener et al. (Frontiers in Psychology, 9, 2387, 2018) presented a visual world study which combined both attention and pupillary measures to demonstrate that anticipating a target results in lower effort to integrate that target (noun). However, they found no indication that the anticipatory processes themselves, i.e., the reduction of uncertainty about upcoming referents, results in processing effort (cf. Linzen and Jaeger, Cognitive Science, 40(6), 1382–1411, 2016). In contrast, Maess et al. (Frontiers in Human Neuroscience, 10, 1–11, 2016) found that more constraining verbs elicited a higher N400 amplitude than unconstraining verbs. The aim of the present study was therefore twofold: Firstly, we examined whether the graded ICA effect, which was previously found on the noun as a result of a likelihood manipulation, replicates in ERP measures. Secondly, we set out to investigate whether the processes leading to the generation of expectations (derived during verb and scene processing) induce an N400 modulation. Our results confirm that visual context is combined with the verb’s meaning to establish expectations about upcoming nouns and that these expectations affect the retrieval of the upcoming noun (modulated N400 on the noun). Importantly, however, we find no evidence for different costs in generating more or less specific expectations for upcoming nouns. Thus, the benefits of generating expectations are not associated with any costs in situated language comprehension.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

• https://www.cambridge.org/core/journals/journal-of-the-international-neuropsychological-society/article/abs/patients-with-amnestic-mci-fail-to-adapt-executive-control-when-repeatedly-tested-with-semantic-verbal-fluency-tasks/E09D9B7801DA02360B056E34E0BD96F7

@article{troger2021patients,
title = {Patients with amnestic MCI Fail to Adapt Executive Control When Repeatedly Tested with Semantic Verbal Fluency Tasks},
author = {Johannes Tr{\"o}ger and Hali Lindsay and Mario Mina and Nicklas Linz and Stefan Kl{\"o}ppel and Jutta Kray and Jessica Peter},
url = {https://www.cambridge.org/core/journals/journal-of-the-international-neuropsychological-society/article/abs/patients-with-amnestic-mci-fail-to-adapt-executive-control-when-repeatedly-tested-with-semantic-verbal-fluency-tasks/E09D9B7801DA02360B056E34E0BD96F7},
year = {2021},
date = {2021-06-30},
journal = {Journal of the International Neuropsychological Society},
pages = {1-8},
publisher = {Cambridge University Press},
abstract = {

},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

Prediction facilitates word processing in the moment, but the longer-term consequences of prediction remain unclear. We investigated whether prediction error during language encoding enhances memory for words later on. German-speaking participants read sentences in which the gender marking of the pre-nominal article was consistent or inconsistent with the predictable noun. During subsequent word recognition, we probed participants’ recognition memory for predictable and unpredictable nouns. Our results indicate that individuals who demonstrated early prediction error during sentence reading, showed enhanced recognition memory for nouns overall. Results from an exploratory step-wise regression showed that prenominal prediction error and general reading speed were the best proxies for recognition memory. Hence, prediction error may facilitate recognition by furnishing memory traces built during initial reading of the sentences. Results are discussed in the light of hypotheses positing that predictable words show a memory disadvantage because they are processed less thoroughly.

• https://www.tandfonline.com/doi/full/10.1080/23273798.2021.1924387
• https://doi.org/10.1080/23273798.2021.1924387

@article{haeuser2021effects,
title = {Effects of prediction error on episodic memory retrieval: evidence from sentence reading and word recognition},
author = {Katja H{\"a}user and Jutta Kray},
url = {https://www.tandfonline.com/doi/full/10.1080/23273798.2021.1924387},
doi = {https://doi.org/10.1080/23273798.2021.1924387},
year = {2021},
date = {2021},
journal = {Language, Cognition and Neuroscience},
pages = {1-17},
publisher = {Taylor & Francis},
abstract = {Prediction facilitates word processing in the moment, but the longer-term consequences of prediction remain unclear. We investigated whether prediction error during language encoding enhances memory for words later on. German-speaking participants read sentences in which the gender marking of the pre-nominal article was consistent or inconsistent with the predictable noun. During subsequent word recognition, we probed participants’ recognition memory for predictable and unpredictable nouns. Our results indicate that individuals who demonstrated early prediction error during sentence reading, showed enhanced recognition memory for nouns overall. Results from an exploratory step-wise regression showed that prenominal prediction error and general reading speed were the best proxies for recognition memory. Hence, prediction error may facilitate recognition by furnishing memory traces built during initial reading of the sentences. Results are discussed in the light of hypotheses positing that predictable words show a memory disadvantage because they are processed less thoroughly.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

Language processing is predictive in nature. But how do people balance multiple competing options as they predict upcoming meanings? Here, we investigated whether readers generate graded predictions about grammatical gender of nouns. Sentence contexts were manipulated so that they strongly biased people’s expectations towards two or more nouns that had the same grammatical gender (single bias condition), or they biased multiple genders from different grammatical classes (multiple bias condition). Our expectation was that unexpected articles should lead to elevated reading times (RTs) in the single-bias condition when probabilistic expectations towards a particular gender are violated. Indeed, the results showed greater sensitivity among language users towards unexpected articles in the single-bias condition, however, RTs on unexpected gendermarked articles were facilitated, and not slowed. Our data confirm that difficulty in sentence processing is modulated by uncertainty about predicted information, and suggest that readers make graded predictions about grammatical gender.

• https://link.springer.com/article/10.3758/s13415-015-0340-0
• https://doi.org/10.3758/s13415-015-0340-0

@inproceedings{haeuser2020great,
title = {Great expectations: Evidence for graded prediction of grammatical gender},
author = {Katja H{\"a}user and Jutta Kray and Arielle Borovsky},
url = {https://link.springer.com/article/10.3758/s13415-015-0340-0},
doi = {https://doi.org/10.3758/s13415-015-0340-0},
year = {2020},
date = {2020},
booktitle = {CogSci},
abstract = {Language processing is predictive in nature. But how do people balance multiple competing options as they predict upcoming meanings? Here, we investigated whether readers generate graded predictions about grammatical gender of nouns. Sentence contexts were manipulated so that they strongly biased people's expectations towards two or more nouns that had the same grammatical gender (single bias condition), or they biased multiple genders from different grammatical classes (multiple bias condition). Our expectation was that unexpected articles should lead to elevated reading times (RTs) in the single-bias condition when probabilistic expectations towards a particular gender are violated. Indeed, the results showed greater sensitivity among language users towards unexpected articles in the single-bias condition, however, RTs on unexpected gendermarked articles were facilitated, and not slowed. Our data confirm that difficulty in sentence processing is modulated by uncertainty about predicted information, and suggest that readers make graded predictions about grammatical gender.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

A word’s predictability or surprisal in linguistic context, as determined by cloze probabilities or languagemodels (e.g., Frank, 2013a) is related to processing effort, in that less expected words take more effort to process (e.g., Hale, 2001). This shows how, in purely linguistic contexts, rational approaches have been proven valid to predict and formalise results from language processing studies. However, the surprisal (or predictability) of a word may also be influenced by extra-linguistic factors, such as visual context information, as given in situated language processing. While, in the case of linguistic contexts, it is known that the incrementally processed information affects the mental model (e.g., Zwaan and Radvansky, 1998) at each word in a probabilistic way, no such observations have been made so far in the case of visual context information. Although it has been shown that in the visual world paradigm (VWP), anticipatory eye movements suggest that listeners exploit the scene to predict what will be mentioned next (Altmann and Kamide, 1999), it is so far unclear how visual information actually affects expectations for and processing effort of target words. If visual context effects on word processing effort can be observed, we hypothesise that rational concepts can be extended in order to formalise these effects, hereby making them statistically accessible for language models. In a line of experiments, I hence observe how visual information – which is inherently different from linguistic context, for instance in its non-incremental-at once-accessibility– affects target words. Our findings are a clear and robust demonstration that the non-linguistic context can immediately influence both lexical expectations, and surprisal-based processing effort as assessed by two different on-line measures of effort (a pupillary and an EEG one). Finally, I use surprisal to formalise the measured results and propose an extended formula to take visual information into account.

• https://publikationen.sulb.uni-saarland.de/handle/20.500.11880/27905
• https://doi.org/https://dx.doi.org/10.22028/D291-28451

@phdthesis{Ankener_Diss_2019,
title = {The influence of visual information on word predictability and processing effort},
author = {Christine Ankener},
url = {https://publikationen.sulb.uni-saarland.de/handle/20.500.11880/27905},
doi = {https://doi.org/https://dx.doi.org/10.22028/D291-28451},
year = {2019},
date = {2019},
school = {Saarland University},
address = {Saarbruecken, Germany},
abstract = {A word’s predictability or surprisal in linguistic context, as determined by cloze probabilities or languagemodels (e.g., Frank, 2013a) is related to processing effort, in that less expected words take more effort to process (e.g., Hale, 2001). This shows how, in purely linguistic contexts, rational approaches have been proven valid to predict and formalise results from language processing studies. However, the surprisal (or predictability) of a word may also be influenced by extra-linguistic factors, such as visual context information, as given in situated language processing. While, in the case of linguistic contexts, it is known that the incrementally processed information affects the mental model (e.g., Zwaan and Radvansky, 1998) at each word in a probabilistic way, no such observations have been made so far in the case of visual context information. Although it has been shown that in the visual world paradigm (VWP), anticipatory eye movements suggest that listeners exploit the scene to predict what will be mentioned next (Altmann and Kamide, 1999), it is so far unclear how visual information actually affects expectations for and processing effort of target words. If visual context effects on word processing effort can be observed, we hypothesise that rational concepts can be extended in order to formalise these effects, hereby making them statistically accessible for language models. In a line of experiments, I hence observe how visual information – which is inherently different from linguistic context, for instance in its non-incremental-at once-accessibility– affects target words. Our findings are a clear and robust demonstration that the non-linguistic context can immediately influence both lexical expectations, and surprisal-based processing effort as assessed by two different on-line measures of effort (a pupillary and an EEG one). Finally, I use surprisal to formalise the measured results and propose an extended formula to take visual information into account.},
pubstate = {published},
type = {phdthesis}
}

Copy BibTeX to Clipboard

Behavioral studies have shown that speaker gaze to objects in a co-present scene can influence listeners’ sentence comprehension. To gain deeper insight into the mechanisms involved in gaze processing and integration, we conducted two ERP experiments (N = 30, Age: [18, 32] and [19, 33] respectively). Participants watched a centrally positioned face performing gaze actions aligned to utterances comparing two out of three displayed objects. They were asked to judge whether the sentence was true given the provided scene. We manipulated the second gaze cue to be either Congruent (baseline), Incongruent or Averted (Exp1)/Mutual (Exp2). When speaker gaze is used to form lexical expectations about upcoming referents, we found an attenuated N200 when phonological information confirms these expectations (Congruent). Similarly, we observed attenuated N400 amplitudes when gaze-cued expectations (Congruent) facilitate lexical retrieval. Crucially, only a violation of gaze-cued lexical expectations (Incongruent) leads to a P600 effect, suggesting the necessity to revise the mental representation of the situation. Our results support the hypothesis that gaze is utilized above and beyond simply enhancing a cued object’s prominence. Rather, gaze to objects leads to their integration into the mental representation of the situation before they are mentioned.

• https://www.sciencedirect.com/science/article/pii/S0278262619300120
• https://doi.org/10.1016/j.bandc.2019.05.009

@article{Jachmann2019b,
title = {Influence of speakers’ gaze on situated language comprehension: Evidence from Event-Related Potentials},
author = {Torsten Jachmann and Heiner Drenhaus and Maria Staudte and Matthew W. Crocker},
url = {https://www.sciencedirect.com/science/article/pii/S0278262619300120},
doi = {https://doi.org/10.1016/j.bandc.2019.05.009},
year = {2019},
date = {2019},
journal = {Brain and cognition},
pages = {103571},
publisher = {Elsevier},
volume = {135},
abstract = {Behavioral studies have shown that speaker gaze to objects in a co-present scene can influence listeners’ sentence comprehension. To gain deeper insight into the mechanisms involved in gaze processing and integration, we conducted two ERP experiments (N = 30, Age: [18, 32] and [19, 33] respectively). Participants watched a centrally positioned face performing gaze actions aligned to utterances comparing two out of three displayed objects. They were asked to judge whether the sentence was true given the provided scene. We manipulated the second gaze cue to be either Congruent (baseline), Incongruent or Averted (Exp1)/Mutual (Exp2). When speaker gaze is used to form lexical expectations about upcoming referents, we found an attenuated N200 when phonological information confirms these expectations (Congruent). Similarly, we observed attenuated N400 amplitudes when gaze-cued expectations (Congruent) facilitate lexical retrieval. Crucially, only a violation of gaze-cued lexical expectations (Incongruent) leads to a P600 effect, suggesting the necessity to revise the mental representation of the situation. Our results support the hypothesis that gaze is utilized above and beyond simply enhancing a cued object’s prominence. Rather, gaze to objects leads to their integration into the mental representation of the situation before they are mentioned.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

A word’s predictability or surprisal in linguistic context, as determined by cloze probabilities or languagemodels (e.g. Frank, 2013a) is related to processing effort, in that less expected words take more effort to process (e.g. Hale, 2001). This shows how, in purely linguistic contexts, rational approaches have been proven valid to predict and formalise results from language processing studies. However, the surprisal (or predictability) of a word may also be influenced by extra-linguistic factors, such as visual context information, as given in situated language processing. While, in the case of linguistic contexts, it is known that the incrementally processed information affects the mental model (e.g. Zwaan and Radvansky, 1998) at each word in a probabilistic way, no such observations have been made so far in the case of visual context information. Although it has been shown that in the visual world paradigm (VWP), anticipatory eye movements suggest that listeners exploit the scene to predict what will be mentioned next (Altmann and Kamide, 1999), it is so far unclear how visual information actually affects expectations for and processing effort of target words. If visual context effects on word processing effort can be observed, we hypothesise that rational concepts can be extended in order to formalise these effects, hereby making them statistically accessible for language models. In a line of experiments, I hence observe how visual information – which is inherently different from linguistic context, for instance in its non-incremental-at once-accessibility– affects target words. Our findings are a clear and robust demonstration that the non-linguistic context can immediately influence both lexical expectations, and surprisal-based processing effort as assessed by two different on-line measures of effort (a pupillary and an EEG one). Finally, I use surprisal to formalise the measured results and propose an extended formula to take visual information into account.

• http://nbn-resolving.de/urn:nbn:de:bsz:291--ds-284518
• https://doi.org/http://dx.doi.org/10.22028/D291-28451

@phdthesis{Ankener_diss_2019,
title = {The influence of visual information on word predictability and processing effort},
author = {Christine Ankener},
url = {http://nbn-resolving.de/urn:nbn:de:bsz:291--ds-284518},
doi = {https://doi.org/http://dx.doi.org/10.22028/D291-28451},
year = {2019},
date = {2019},
school = {Saarland University},
address = {Saarbruecken, Germany},
abstract = {A word’s predictability or surprisal in linguistic context, as determined by cloze probabilities or languagemodels (e.g. Frank, 2013a) is related to processing effort, in that less expected words take more effort to process (e.g. Hale, 2001). This shows how, in purely linguistic contexts, rational approaches have been proven valid to predict and formalise results from language processing studies. However, the surprisal (or predictability) of a word may also be influenced by extra-linguistic factors, such as visual context information, as given in situated language processing. While, in the case of linguistic contexts, it is known that the incrementally processed information affects the mental model (e.g. Zwaan and Radvansky, 1998) at each word in a probabilistic way, no such observations have been made so far in the case of visual context information. Although it has been shown that in the visual world paradigm (VWP), anticipatory eye movements suggest that listeners exploit the scene to predict what will be mentioned next (Altmann and Kamide, 1999), it is so far unclear how visual information actually affects expectations for and processing effort of target words. If visual context effects on word processing effort can be observed, we hypothesise that rational concepts can be extended in order to formalise these effects, hereby making them statistically accessible for language models. In a line of experiments, I hence observe how visual information – which is inherently different from linguistic context, for instance in its non-incremental-at once-accessibility– affects target words. Our findings are a clear and robust demonstration that the non-linguistic context can immediately influence both lexical expectations, and surprisal-based processing effort as assessed by two different on-line measures of effort (a pupillary and an EEG one). Finally, I use surprisal to formalise the measured results and propose an extended formula to take visual information into account.},
pubstate = {published},
type = {phdthesis}
}

Copy BibTeX to Clipboard

Questions under Discussion (QUDs) have been suggested to influence the integration of individual utterances into a discourse-level representation. Previous work has shown that processing ungrammatical ellipses is facilitated when the elided material addresses an implicit QUD raised through a nonactuality implicature (NAIs). It is not clear, however, if QUDs influence discourse coherence during comprehension of fully acceptable discourse. We present two ERP studies examining the effects of QUDs introduced by NAIs using two-sentence discourses. Experiment 1 showed that processing definite NPs with inaccessible antecedents is facilitated when their content is relevant to the QUD. Using acceptable discourses, Experiment 2 showed that definite NPs failing to address a QUD elicit increased processing cost. Overall, our results indicate that QUDs raise the expectation that the following discourse will address them, providing unambiguous evidence that their influence is not limited to the processing of ungrammatical input.

• https://doi.org/10.1080/0163853X.2019.1575140
• https://doi.org/10.1080/0163853X.2019.1575140

@article{Delogu2019b,
title = {Discourse Expectations Are Sensitive to the Question Under Discussion: Evidence From ERPs},
author = {Francesca Delogu and Torsten Jachmann and Maria Staudte and Francesco Vespignani and Nicola Molinaro},
url = {https://doi.org/10.1080/0163853X.2019.1575140},
doi = {https://doi.org/10.1080/0163853X.2019.1575140},
year = {2019},
date = {2019},
journal = {Discourse Processes},
pages = {1-19},
abstract = {Questions under Discussion (QUDs) have been suggested to influence the integration of individual utterances into a discourse-level representation. Previous work has shown that processing ungrammatical ellipses is facilitated when the elided material addresses an implicit QUD raised through a nonactuality implicature (NAIs). It is not clear, however, if QUDs influence discourse coherence during comprehension of fully acceptable discourse. We present two ERP studies examining the effects of QUDs introduced by NAIs using two-sentence discourses. Experiment 1 showed that processing definite NPs with inaccessible antecedents is facilitated when their content is relevant to the QUD. Using acceptable discourses, Experiment 2 showed that definite NPs failing to address a QUD elicit increased processing cost. Overall, our results indicate that QUDs raise the expectation that the following discourse will address them, providing unambiguous evidence that their influence is not limited to the processing of ungrammatical input.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

A large body of contemporary psycholinguistic research utilizes the information-theoretic notions related to the transmission of information in an attempt to better understand and formalize the regularities of language production and comprehension. The overarching hypothesis is that prediction is a core mechanism underlying language comprehension. Anticipating what is likely to be mentioned next based on the previous context is what is assumed to allow for smooth and effortless communication. The anticipation of linguistic units that fit the current context reduces the uncertainty about the upcoming material, which consequently facilitates the processing of that material, in a typically noisy channel. Situated language processing allows for the integration of not only linguistic but also non-linguistic visual information that contribute to establishing the context, and facilitate the creation of anticipations regarding the upcoming linguistic material. Moreover, noticing that our interlocutor is directing her attention to a certain object, inspires a shift in our visual attention towards the same entity. Since what is relevant for our interlocutor is highly likely to be relevant for us, too, whether simply conversationally, or more importantly, even existentially (Emery, 2000). Hence, following the speaker’s referential gaze cue towards an object relevant for the current conversation has been shown to benefit listeners’ language processing, measured by shorter reaction times on subsequent tasks (e.g., Staudte & Crocker, 2011; Staudte, Crocker, Heloir, & Kipp, 2014; Knoeferle & Kreysa, 2012; Macdonald & Tatler, 2013, 2014). The present thesis aimed to provide an insight into the mechanisms behind this facilitation. We examined the dynamics of combining visual and linguistic information in creating anticipation for a specific object to be mentioned, and the effect this has on language processing. To this end we used a pupillary measure of cognitive load that is robust enough to allow for free eye movements (the Index of Cognitive Activity; Marshall, 2000). This enabled us to measure not only the visual attention during language comprehension, but also the immediately induced cognitive load at various relevant points during the auditory presentation of the linguistic stimulus. Eight experiments were conducted towards addressing our research questions. The initial three experiments established the ICA measurement in the context of our linguistic manipulation. This series of experiments included reading, cognitive load during listening, and the examination of visual attention together with cognitive load in the visual world paradigm (VWP). Subsequently, we conducted five eye tracking experiments in the VWP where the linguistic context was further enriched by a referential gaze cue. All five experiments simultaneously assessed both visual attention and the immediate cognitive load induced at different stages of sentence processing. We manipulated the existence of the referential gaze cue (Exp. 4), the probability of mention of the cued object (Exp. 4, 5), the congruency of the gaze cue and the subsequent referring expression (Exp. 6), as well as the number of cued objects with equal probability of mention (Exp. 7, 8). Finally, we examined whether the gaze cue can take the role of fully disambiguating the target referent (Exp. 8). We quantified the importance of the visual context in language processing, and showed that if a certain object from the visual context has a higher likelihood of mention given the linguistic context, its processing is facilitated, in comparison to the processing of the same sentence without the visual context. Furthermore, our results support the previous findings that the referential gaze cue leads to a shift in visual attention towards the cued object, thereby facilitating language processing. We expanded these findings by showing that it is the processing of the linguistic reference, that is the referent noun, that is facilitated by gaze-following. Importantly, perceiving and following the gaze cue did not prove costly in terms of cognitive effort, unless the cued object did not fit the verb selectional preferences. This is true regardless of the number of objects cued, or the lower likelihood of mention of the cued object. We conclude that listeners strategically use visual information to reduce the referential uncertainty for upcoming nouns but that the visual cues, such as the referential gaze cue, do not underly the same kinds of expectations (and resulting cognitive costs) as linguistic references. We did not find evidence that the gaze cue is processed in a manner comparable to noun processing, rather, it is likely perceived as a relevant piece of information introduced in addition to the linguistic material, in order to aid language processing, but, importantly, not there to substitute it.

• http://nbn-resolving.de/urn:nbn:de:bsz:291--ds-285651
• https://doi.org/http://dx.doi.org/10.22028/D291-28565

@phdthesis{Sekicki_diss_0919,
title = {Exploiting referential gaze for uncertainty reduction in situated language processing: an information-theoretic approach},
author = {Mirjana Sekicki},
url = {http://nbn-resolving.de/urn:nbn:de:bsz:291--ds-285651},
doi = {https://doi.org/http://dx.doi.org/10.22028/D291-28565},
year = {2019},
date = {2019},
school = {Saarland University},
address = {Saarbr{\"u}cken},
abstract = {A large body of contemporary psycholinguistic research utilizes the information-theoretic notions related to the transmission of information in an attempt to better understand and formalize the regularities of language production and comprehension. The overarching hypothesis is that prediction is a core mechanism underlying language comprehension. Anticipating what is likely to be mentioned next based on the previous context is what is assumed to allow for smooth and effortless communication. The anticipation of linguistic units that fit the current context reduces the uncertainty about the upcoming material, which consequently facilitates the processing of that material, in a typically noisy channel. Situated language processing allows for the integration of not only linguistic but also non-linguistic visual information that contribute to establishing the context, and facilitate the creation of anticipations regarding the upcoming linguistic material. Moreover, noticing that our interlocutor is directing her attention to a certain object, inspires a shift in our visual attention towards the same entity. Since what is relevant for our interlocutor is highly likely to be relevant for us, too, whether simply conversationally, or more importantly, even existentially (Emery, 2000). Hence, following the speaker’s referential gaze cue towards an object relevant for the current conversation has been shown to benefit listeners’ language processing, measured by shorter reaction times on subsequent tasks (e.g., Staudte & Crocker, 2011; Staudte, Crocker, Heloir, & Kipp, 2014; Knoeferle & Kreysa, 2012; Macdonald & Tatler, 2013, 2014). The present thesis aimed to provide an insight into the mechanisms behind this facilitation. We examined the dynamics of combining visual and linguistic information in creating anticipation for a specific object to be mentioned, and the effect this has on language processing. To this end we used a pupillary measure of cognitive load that is robust enough to allow for free eye movements (the Index of Cognitive Activity; Marshall, 2000). This enabled us to measure not only the visual attention during language comprehension, but also the immediately induced cognitive load at various relevant points during the auditory presentation of the linguistic stimulus. Eight experiments were conducted towards addressing our research questions. The initial three experiments established the ICA measurement in the context of our linguistic manipulation. This series of experiments included reading, cognitive load during listening, and the examination of visual attention together with cognitive load in the visual world paradigm (VWP). Subsequently, we conducted five eye tracking experiments in the VWP where the linguistic context was further enriched by a referential gaze cue. All five experiments simultaneously assessed both visual attention and the immediate cognitive load induced at different stages of sentence processing. We manipulated the existence of the referential gaze cue (Exp. 4), the probability of mention of the cued object (Exp. 4, 5), the congruency of the gaze cue and the subsequent referring expression (Exp. 6), as well as the number of cued objects with equal probability of mention (Exp. 7, 8). Finally, we examined whether the gaze cue can take the role of fully disambiguating the target referent (Exp. 8). We quantified the importance of the visual context in language processing, and showed that if a certain object from the visual context has a higher likelihood of mention given the linguistic context, its processing is facilitated, in comparison to the processing of the same sentence without the visual context. Furthermore, our results support the previous findings that the referential gaze cue leads to a shift in visual attention towards the cued object, thereby facilitating language processing. We expanded these findings by showing that it is the processing of the linguistic reference, that is the referent noun, that is facilitated by gaze-following. Importantly, perceiving and following the gaze cue did not prove costly in terms of cognitive effort, unless the cued object did not fit the verb selectional preferences. This is true regardless of the number of objects cued, or the lower likelihood of mention of the cued object. We conclude that listeners strategically use visual information to reduce the referential uncertainty for upcoming nouns but that the visual cues, such as the referential gaze cue, do not underly the same kinds of expectations (and resulting cognitive costs) as linguistic references. We did not find evidence that the gaze cue is processed in a manner comparable to noun processing, rather, it is likely perceived as a relevant piece of information introduced in addition to the linguistic material, in order to aid language processing, but, importantly, not there to substitute it.},
pubstate = {published},
type = {phdthesis}
}

Copy BibTeX to Clipboard

Referential gaze has been shown to benefit language processing in situated communication in terms of shifting visual attention and leading to shorter reaction times on subsequent tasks. The present study simultaneously assessed both visual attention and, importantly, the immediate cogni-tive load induced at different stages of sentence processing. We aimed to examine the dynamics of combining visual and linguistic information in creating anticipation for a specific object and the effect this has on language processing. We report evidence from three visual-world eye-tracking experiments, showing that referential gaze leads to a shift in visual attention toward the cued object, which consequently lowers the effort required for processing the linguistic reference. Importantly, perceiving and following the gaze cue did not prove costly in terms of cognitive effort, unless the cued object did not fit the verb selectional preferences.

• https://onlinelibrary.wiley.com/doi/full/10.1111/cogs.12682
• https://doi.org/10.1111/cogs.12682

@article{ Sekicki2018,
title = {Eye'll help you out! How the gaze cue reduces the cognitive load required for reference processing},
author = {Mirjana Sekicki and Maria Staudte},
url = {https://onlinelibrary.wiley.com/doi/full/10.1111/cogs.12682},
doi = {https://doi.org/10.1111/cogs.12682},
year = {2018},
date = {2018},
journal = {Cognitive Science},
pages = {1-40},
volume = {42},
abstract = {Referential gaze has been shown to benefit language processing in situated communication in terms of shifting visual attention and leading to shorter reaction times on subsequent tasks. The present study simultaneously assessed both visual attention and, importantly, the immediate cogni-tive load induced at different stages of sentence processing. We aimed to examine the dynamics of combining visual and linguistic information in creating anticipation for a specific object and the effect this has on language processing. We report evidence from three visual-world eye-tracking experiments, showing that referential gaze leads to a shift in visual attention toward the cued object, which consequently lowers the effort required for processing the linguistic reference. Importantly, perceiving and following the gaze cue did not prove costly in terms of cognitive effort, unless the cued object did not fit the verb selectional preferences.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

@miscellaneous{Ankener2018b,
title = {Visually informed prediction: How combining lexical and visual information affects surprisal},
author = {Maria Staudte and Christine Ankener},
year = {2018},
date = {2018-10-17},
booktitle = {31st Annual Conference on Sentence Processing (CUNY)},
address = {UC Davis, USA},
pubstate = {published},
type = {miscellaneous}
}

Copy BibTeX to Clipboard