Publications

Developmental and longitudinal studies with children increasingly use pictorial stimuli in cognitive, psychologic, and psycholinguistic research. To enhance validity and comparability within and across those studies, the use of normed pictures is recommended. Besides, creating picture sets and evaluating them in rating studies is very time consuming, in particular regarding samples of young children in which testing time is rather limited. As there is an increasing number of studies that investigate young German children’s semantic language processing with colored clipart stimuli, this work provides a first set of 247 colored cliparts with ratings of German native speaking children aged 4 to 6 years. We assessed two central rating aspects of pictures: Name agreement (Do pictures elicit the intended name of an object?) and semantic categorization (Are objects classified as members of the intended semantic category?). Our ratings indicate that children are proficient in naming and even better in semantic categorization of objects, whereas both seems to improve with increasing age of young childhood. Finally, this paper discusses some features of pictorial objects that might be important for children’s name agreement and semantic categorization and could be considered in future picture rating studies.

• 1-s2.0-S0001691822000737-main     (0.78MB)
• https://www.sciencedirect.com/science/article/pii/S0001691822000737
• https://doi.org/https://doi.org/10.1016/j.actpsy.2022.103558

@article{Sommerfeld_of_2022,
title = {Ratings of name agreement and semantic categorization of 247 colored clipart pictures by young German children},
author = {Linda Sommerfeld and Maria Staudte and Jutta Kray},
url = {https://www.sciencedirect.com/science/article/pii/S0001691822000737},
doi = {https://doi.org/https://doi.org/10.1016/j.actpsy.2022.103558},
year = {2022},
date = {2022},
journal = {Acta Psychologica},
pages = {103558},
volume = {226},
abstract = {Developmental and longitudinal studies with children increasingly use pictorial stimuli in cognitive, psychologic, and psycholinguistic research. To enhance validity and comparability within and across those studies, the use of normed pictures is recommended. Besides, creating picture sets and evaluating them in rating studies is very time consuming, in particular regarding samples of young children in which testing time is rather limited. As there is an increasing number of studies that investigate young German children's semantic language processing with colored clipart stimuli, this work provides a first set of 247 colored cliparts with ratings of German native speaking children aged 4 to 6 years. We assessed two central rating aspects of pictures: Name agreement (Do pictures elicit the intended name of an object?) and semantic categorization (Are objects classified as members of the intended semantic category?). Our ratings indicate that children are proficient in naming and even better in semantic categorization of objects, whereas both seems to improve with increasing age of young childhood. Finally, this paper discusses some features of pictorial objects that might be important for children's name agreement and semantic categorization and could be considered in future picture rating studies.},
keywords = {Name agreement, Semantic categorization, Picture naming, Picture ratings, Children, Age differences},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

Semantic extension plays a key role in language change and grammaticalisation. Here we use a dyadic interaction paradigm to study semantic extension of novel labels in controlled circumstances. We ask whether participants will be able to (i) use highly accessible associations in the perceptual environment (colour-shape associations) to converge on a meaning for the novel labels, and (ii) extend these meanings to apply to both concrete targets (objects) and abstract targets (emotions). Further, given the argument that both metonymy and metaphor are important drivers of language change, we investigate whether participants will be able to draw on relations of contiguity (‘metonymic’ associations, e.g. colour-shape or object-colour) and relations of similarity (‘metaphorical’ associations, e.g. emotion-colour) to extend the meaning of labels.

• eScholarship UC item 7w22b8gm     (0.34MB)
• https://escholarship.org/uc/item/7w22b8gm

@inproceedings{HaeuserKray2022,
title = {Uninvited and unwanted: False memories for words predicted but not seen},
author = {Katja H{\"a}user and Jutta Kray},
editor = {Jennifer Culbertson and Andrew Perfors and Hugh Rabagliati and Veronica Ramenzoni},
url = {https://escholarship.org/uc/item/7w22b8gm},
year = {2022},
date = {2022},
booktitle = {Proceedings of the 44th Annual Conference of the Cognitive Science Society, Toronto, Canada (27 Jul 2022 - 30 Jul 2022)},
pages = {2401-2408},
abstract = {Semantic extension plays a key role in language change and grammaticalisation. Here we use a dyadic interaction paradigm to study semantic extension of novel labels in controlled circumstances. We ask whether participants will be able to (i) use highly accessible associations in the perceptual environment (colour-shape associations) to converge on a meaning for the novel labels, and (ii) extend these meanings to apply to both concrete targets (objects) and abstract targets (emotions). Further, given the argument that both metonymy and metaphor are important drivers of language change, we investigate whether participants will be able to draw on relations of contiguity (‘metonymic’ associations, e.g. colour-shape or object-colour) and relations of similarity (‘metaphorical’ associations, e.g. emotion-colour) to extend the meaning of labels.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

• collabra_2022_8_1_36945     (1.55MB)
• https://online.ucpress.edu/collabra/article/8/1/36945/187814/Hedging-Bets-in-Linguistic-Prediction-Younger-and
• https://doi.org/10.1525/collabra.36945

@article{Haeuseretal22,
title = {Hedging Bets in Linguistic Prediction: Younger and Older Adults Vary in the Breadth of Predictive Processing},
author = {Katja H{\"a}user and Jutta Kray and Arielle Borovsky},
url = {https://online.ucpress.edu/collabra/article/8/1/36945/187814/Hedging-Bets-in-Linguistic-Prediction-Younger-and},
doi = {https://doi.org/10.1525/collabra.36945},
year = {2022},
date = {2022},
journal = {Collabra: Psychology},
volume = {8(1):36945},
abstract = {

Copy BibTeX to Clipboard

How do violations of predictability and plausibility affect online language processing? How does it affect longer-term memory and learning when predictions are disconfirmed by plausible or implausible words? We investigated these questions using a self-paced sentence reading and noun recognition task. Critical sentences violated predictability or plausibility or both, for example, “Since Anne is afraid of spiders, she doesn’t like going down into the … basement (predictable, plausible), garden (unpredictable, somewhat plausible), moon (unpredictable, deeply implausible).” Results from sentence reading showed earlier-emerging effects of predictability violations on the critical noun, but later-emerging effects of plausibility violations after the noun. Recognition memory was exclusively enhanced for deeply implausible nouns. The earlier-emerging predictability effect indicates that having word form predictions disconfirmed is registered very early in the processing stream, irrespective of semantics. The later-emerging plausibility effect supports models that argue for a staged architecture of reading comprehension, where plausibility only affects a post-lexical integration stage. Our memory results suggest that, in order to facilitate memory and learning, a certain magnitude of prediction error is required.

• how-odd-diverging-effects-of-predictability-and-plausibility-violations-on-sentence-reading-and-word-memory     (0.53MB)
• https://www.cambridge.org/core/journals/applied-psycholinguistics/article/how-odd-diverging-effects-of-predictability-and-plausibility-violations-on-sentence-reading-and-word-memory/D8E12864E47CE24E62297ABF5BA2BED0
• https://doi.org/10.1017/S0142716422000364

@article{HaeuserKray22,
title = {How odd: Diverging effects of predictability and plausibility violations on sentence reading and word memory},
author = {Katja H{\"a}user and Jutta Kray},
url = {https://www.cambridge.org/core/journals/applied-psycholinguistics/article/how-odd-diverging-effects-of-predictability-and-plausibility-violations-on-sentence-reading-and-word-memory/D8E12864E47CE24E62297ABF5BA2BED0},
doi = {https://doi.org/10.1017/S0142716422000364},
year = {2022},
date = {2022},
journal = {Applied Psycholinguistics},
pages = {1193-1220},
volume = {43(5)},
abstract = {How do violations of predictability and plausibility affect online language processing? How does it affect longer-term memory and learning when predictions are disconfirmed by plausible or implausible words? We investigated these questions using a self-paced sentence reading and noun recognition task. Critical sentences violated predictability or plausibility or both, for example, “Since Anne is afraid of spiders, she doesn’t like going down into the … basement (predictable, plausible), garden (unpredictable, somewhat plausible), moon (unpredictable, deeply implausible).” Results from sentence reading showed earlier-emerging effects of predictability violations on the critical noun, but later-emerging effects of plausibility violations after the noun. Recognition memory was exclusively enhanced for deeply implausible nouns. The earlier-emerging predictability effect indicates that having word form predictions disconfirmed is registered very early in the processing stream, irrespective of semantics. The later-emerging plausibility effect supports models that argue for a staged architecture of reading comprehension, where plausibility only affects a post-lexical integration stage. Our memory results suggest that, in order to facilitate memory and learning, a certain magnitude of prediction error is required.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

Not only adults, but already children anticipate upcoming input in sentences with semantically constraining verbs. In the visual world, the sentence context is used to anticipatorily fixate the only object matching potential sentence continuations. Adults process multiple visual cues in parallel when predicting language. Here, we examined whether young children can also process multiple visual objects fitting the constraining verb of a sentence. We also examined if their processing of multiple potential referents is affected by language skills, given that vocabulary size modulates children’s prediction.

• 119     (0.28MB)
• https://amlap2021.github.io/program/119.pdf

@miscellaneous{sommerfeld2021children,
title = {Children and adults integrate complex visual contexts in language prediction},
author = {Linda Sommerfeld and Maria Staudte and Nivedita Mani and Jutta Kray},
url = {https://amlap2021.github.io/program/119.pdf},
year = {2021},
date = {2021},
booktitle = {Architectures and Mechanisms for Language Processing AMLAP Annual Meeting},
abstract = {Not only adults, but already children anticipate upcoming input in sentences with semantically constraining verbs. In the visual world, the sentence context is used to anticipatorily fixate the only object matching potential sentence continuations. Adults process multiple visual cues in parallel when predicting language. Here, we examined whether young children can also process multiple visual objects fitting the constraining verb of a sentence. We also examined if their processing of multiple potential referents is affected by language skills, given that vocabulary size modulates children’s prediction.},
pubstate = {published},
type = {miscellaneous}
}

Copy BibTeX to Clipboard

The current work investigates how preexisting mental representations of the meaning of an utterance (top-down processing) affect the comprehension of external perceptual properties of the linguistic input (bottom-up processing). When it comes to top-down bottom-up processing in the brain previous findings report a division of focus between left and right hemispheric mechanisms. The PARLO sentence comprehension model posits that the LH employs top-down mechanisms which allow for efficient anticipatory processing, while the RH relies more on bottom-up mechanisms. A shortcoming of the PARLO model is that it’s based on experiments manipulating solely top-down contextual constraint, leading to conclusions that hemispheric asymmetries are a function of differences in the efficiency of top-down rather than bottom-up mechanisms. Up until now, there has been no investigation of asymmetries in bottom-up processing, nor an investigation of the potential interactions between that and top-down processing for each hemisphere. This thesis consists of four event-related potential (ERP) experiments divided into two parts. Experiments 1 (central presentation) and 2 (hemispheric presentation) manipulate the bottom-up lexical frequency of critical words in high and low predictability contexts. Experiments 3 (central presentation) and 4 (hemispheric presentation) manipulate bottom-up word status, presenting critical words and pseudowords in the same high and low predictability contexts. The results allow us to extend previous findings and present the Spotlight Theory of Hemispheric Comprehension. We argue that the LH employs a kind of spotlight focus, which affords very efficient top-down processing of the expected input, since only highly predictable inputs receive additional facilitation based their bottom-up features. Alternatively, the RH lack of spotlight mechanism and focus on bottom-up lexical properties allows for the reliable processing of less predictable and irregular inputs. In combination, these complementary processing strategies provide the comprehension system with the efficiency and robustness required in a wide range of communicative situations.

• https://publikationen.sulb.uni-saarland.de/handle/20.500.11880/31806
• https://doi.org/https://dx.doi.org/10.22028/D291-33976

@phdthesis{Vergilova_Diss_2021,
title = {The Lateralization of Expectations: Hemispheric Differences in Top-down and Bottom-up Word Processing in Context},
author = {Yoana Vergilova},
url = {https://publikationen.sulb.uni-saarland.de/handle/20.500.11880/31806},
doi = {https://doi.org/https://dx.doi.org/10.22028/D291-33976},
year = {2021},
date = {2021},
school = {Saarland University},
address = {Saarbruecken, Germany},
abstract = {The current work investigates how preexisting mental representations of the meaning of an utterance (top-down processing) affect the comprehension of external perceptual properties of the linguistic input (bottom-up processing). When it comes to top-down bottom-up processing in the brain previous findings report a division of focus between left and right hemispheric mechanisms. The PARLO sentence comprehension model posits that the LH employs top-down mechanisms which allow for efficient anticipatory processing, while the RH relies more on bottom-up mechanisms. A shortcoming of the PARLO model is that it’s based on experiments manipulating solely top-down contextual constraint, leading to conclusions that hemispheric asymmetries are a function of differences in the efficiency of top-down rather than bottom-up mechanisms. Up until now, there has been no investigation of asymmetries in bottom-up processing, nor an investigation of the potential interactions between that and top-down processing for each hemisphere. This thesis consists of four event-related potential (ERP) experiments divided into two parts. Experiments 1 (central presentation) and 2 (hemispheric presentation) manipulate the bottom-up lexical frequency of critical words in high and low predictability contexts. Experiments 3 (central presentation) and 4 (hemispheric presentation) manipulate bottom-up word status, presenting critical words and pseudowords in the same high and low predictability contexts. The results allow us to extend previous findings and present the Spotlight Theory of Hemispheric Comprehension. We argue that the LH employs a kind of spotlight focus, which affords very efficient top-down processing of the expected input, since only highly predictable inputs receive additional facilitation based their bottom-up features. Alternatively, the RH lack of spotlight mechanism and focus on bottom-up lexical properties allows for the reliable processing of less predictable and irregular inputs. In combination, these complementary processing strategies provide the comprehension system with the efficiency and robustness required in a wide range of communicative situations.},
pubstate = {published},
type = {phdthesis}
}

Copy BibTeX to Clipboard

Recently, Ankener et al. (Frontiers in Psychology, 9, 2387, 2018) presented a visual world study which combined both attention and pupillary measures to demonstrate that anticipating a target results in lower effort to integrate that target (noun). However, they found no indication that the anticipatory processes themselves, i.e., the reduction of uncertainty about upcoming referents, results in processing effort (cf. Linzen and Jaeger, Cognitive Science, 40(6), 1382–1411, 2016). In contrast, Maess et al. (Frontiers in Human Neuroscience, 10, 1–11, 2016) found that more constraining verbs elicited a higher N400 amplitude than unconstraining verbs. The aim of the present study was therefore twofold: Firstly, we examined whether the graded ICA effect, which was previously found on the noun as a result of a likelihood manipulation, replicates in ERP measures. Secondly, we set out to investigate whether the processes leading to the generation of expectations (derived during verb and scene processing) induce an N400 modulation. Our results confirm that visual context is combined with the verb’s meaning to establish expectations about upcoming nouns and that these expectations affect the retrieval of the upcoming noun (modulated N400 on the noun). Importantly, however, we find no evidence for different costs in generating more or less specific expectations for upcoming nouns. Thus, the benefits of generating expectations are not associated with any costs in situated language comprehension.

• https://link.springer.com/article/10.3758/s13423-020-01827-3
• https://doi.org/10.3758/s13423-020-01827-3

@article{staudte2021,
title = {Graded expectations in visually situated comprehension: Costs and benefits as indexed by the N400},
author = {Maria Staudte and Christine Ankener and Heiner Drenhaus and Matthew W. Crocker},
url = {https://link.springer.com/article/10.3758/s13423-020-01827-3},
doi = {https://doi.org/10.3758/s13423-020-01827-3},
year = {2021},
date = {2021},
journal = {Psychonomic Bulletin & Review},
pages = {624-631},
publisher = {Springer},
volume = {28},
number = {2},
abstract = {Recently, Ankener et al. (Frontiers in Psychology, 9, 2387, 2018) presented a visual world study which combined both attention and pupillary measures to demonstrate that anticipating a target results in lower effort to integrate that target (noun). However, they found no indication that the anticipatory processes themselves, i.e., the reduction of uncertainty about upcoming referents, results in processing effort (cf. Linzen and Jaeger, Cognitive Science, 40(6), 1382–1411, 2016). In contrast, Maess et al. (Frontiers in Human Neuroscience, 10, 1–11, 2016) found that more constraining verbs elicited a higher N400 amplitude than unconstraining verbs. The aim of the present study was therefore twofold: Firstly, we examined whether the graded ICA effect, which was previously found on the noun as a result of a likelihood manipulation, replicates in ERP measures. Secondly, we set out to investigate whether the processes leading to the generation of expectations (derived during verb and scene processing) induce an N400 modulation. Our results confirm that visual context is combined with the verb’s meaning to establish expectations about upcoming nouns and that these expectations affect the retrieval of the upcoming noun (modulated N400 on the noun). Importantly, however, we find no evidence for different costs in generating more or less specific expectations for upcoming nouns. Thus, the benefits of generating expectations are not associated with any costs in situated language comprehension.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

• https://www.cambridge.org/core/journals/journal-of-the-international-neuropsychological-society/article/abs/patients-with-amnestic-mci-fail-to-adapt-executive-control-when-repeatedly-tested-with-semantic-verbal-fluency-tasks/E09D9B7801DA02360B056E34E0BD96F7

@article{troger2021patients,
title = {Patients with amnestic MCI Fail to Adapt Executive Control When Repeatedly Tested with Semantic Verbal Fluency Tasks},
author = {Johannes Tr{\"o}ger and Hali Lindsay and Mario Mina and Nicklas Linz and Stefan Kl{\"o}ppel and Jutta Kray and Jessica Peter},
url = {https://www.cambridge.org/core/journals/journal-of-the-international-neuropsychological-society/article/abs/patients-with-amnestic-mci-fail-to-adapt-executive-control-when-repeatedly-tested-with-semantic-verbal-fluency-tasks/E09D9B7801DA02360B056E34E0BD96F7},
year = {2021},
date = {2021-06-30},
journal = {Journal of the International Neuropsychological Society},
pages = {1-8},
publisher = {Cambridge University Press},
abstract = {

Copy BibTeX to Clipboard

Prediction facilitates word processing in the moment, but the longer-term consequences of prediction remain unclear. We investigated whether prediction error during language encoding enhances memory for words later on. German-speaking participants read sentences in which the gender marking of the pre-nominal article was consistent or inconsistent with the predictable noun. During subsequent word recognition, we probed participants’ recognition memory for predictable and unpredictable nouns. Our results indicate that individuals who demonstrated early prediction error during sentence reading, showed enhanced recognition memory for nouns overall. Results from an exploratory step-wise regression showed that prenominal prediction error and general reading speed were the best proxies for recognition memory. Hence, prediction error may facilitate recognition by furnishing memory traces built during initial reading of the sentences. Results are discussed in the light of hypotheses positing that predictable words show a memory disadvantage because they are processed less thoroughly.

• https://www.tandfonline.com/doi/full/10.1080/23273798.2021.1924387
• https://doi.org/10.1080/23273798.2021.1924387

@article{haeuser2021effects,
title = {Effects of prediction error on episodic memory retrieval: evidence from sentence reading and word recognition},
author = {Katja H{\"a}user and Jutta Kray},
url = {https://www.tandfonline.com/doi/full/10.1080/23273798.2021.1924387},
doi = {https://doi.org/10.1080/23273798.2021.1924387},
year = {2021},
date = {2021},
journal = {Language, Cognition and Neuroscience},
pages = {1-17},
publisher = {Taylor & Francis},
abstract = {Prediction facilitates word processing in the moment, but the longer-term consequences of prediction remain unclear. We investigated whether prediction error during language encoding enhances memory for words later on. German-speaking participants read sentences in which the gender marking of the pre-nominal article was consistent or inconsistent with the predictable noun. During subsequent word recognition, we probed participants’ recognition memory for predictable and unpredictable nouns. Our results indicate that individuals who demonstrated early prediction error during sentence reading, showed enhanced recognition memory for nouns overall. Results from an exploratory step-wise regression showed that prenominal prediction error and general reading speed were the best proxies for recognition memory. Hence, prediction error may facilitate recognition by furnishing memory traces built during initial reading of the sentences. Results are discussed in the light of hypotheses positing that predictable words show a memory disadvantage because they are processed less thoroughly.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

Language processing is predictive in nature. But how do people balance multiple competing options as they predict upcoming meanings? Here, we investigated whether readers generate graded predictions about grammatical gender of nouns. Sentence contexts were manipulated so that they strongly biased people’s expectations towards two or more nouns that had the same grammatical gender (single bias condition), or they biased multiple genders from different grammatical classes (multiple bias condition). Our expectation was that unexpected articles should lead to elevated reading times (RTs) in the single-bias condition when probabilistic expectations towards a particular gender are violated. Indeed, the results showed greater sensitivity among language users towards unexpected articles in the single-bias condition, however, RTs on unexpected gendermarked articles were facilitated, and not slowed. Our data confirm that difficulty in sentence processing is modulated by uncertainty about predicted information, and suggest that readers make graded predictions about grammatical gender.

• https://link.springer.com/article/10.3758/s13415-015-0340-0
• https://doi.org/10.3758/s13415-015-0340-0

@inproceedings{haeuser2020great,
title = {Great expectations: Evidence for graded prediction of grammatical gender},
author = {Katja H{\"a}user and Jutta Kray and Arielle Borovsky},
url = {https://link.springer.com/article/10.3758/s13415-015-0340-0},
doi = {https://doi.org/10.3758/s13415-015-0340-0},
year = {2020},
date = {2020},
booktitle = {CogSci},
abstract = {Language processing is predictive in nature. But how do people balance multiple competing options as they predict upcoming meanings? Here, we investigated whether readers generate graded predictions about grammatical gender of nouns. Sentence contexts were manipulated so that they strongly biased people's expectations towards two or more nouns that had the same grammatical gender (single bias condition), or they biased multiple genders from different grammatical classes (multiple bias condition). Our expectation was that unexpected articles should lead to elevated reading times (RTs) in the single-bias condition when probabilistic expectations towards a particular gender are violated. Indeed, the results showed greater sensitivity among language users towards unexpected articles in the single-bias condition, however, RTs on unexpected gendermarked articles were facilitated, and not slowed. Our data confirm that difficulty in sentence processing is modulated by uncertainty about predicted information, and suggest that readers make graded predictions about grammatical gender.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

A word’s predictability or surprisal in linguistic context, as determined by cloze probabilities or languagemodels (e.g., Frank, 2013a) is related to processing effort, in that less expected words take more effort to process (e.g., Hale, 2001). This shows how, in purely linguistic contexts, rational approaches have been proven valid to predict and formalise results from language processing studies. However, the surprisal (or predictability) of a word may also be influenced by extra-linguistic factors, such as visual context information, as given in situated language processing. While, in the case of linguistic contexts, it is known that the incrementally processed information affects the mental model (e.g., Zwaan and Radvansky, 1998) at each word in a probabilistic way, no such observations have been made so far in the case of visual context information. Although it has been shown that in the visual world paradigm (VWP), anticipatory eye movements suggest that listeners exploit the scene to predict what will be mentioned next (Altmann and Kamide, 1999), it is so far unclear how visual information actually affects expectations for and processing effort of target words. If visual context effects on word processing effort can be observed, we hypothesise that rational concepts can be extended in order to formalise these effects, hereby making them statistically accessible for language models. In a line of experiments, I hence observe how visual information – which is inherently different from linguistic context, for instance in its non-incremental-at once-accessibility– affects target words. Our findings are a clear and robust demonstration that the non-linguistic context can immediately influence both lexical expectations, and surprisal-based processing effort as assessed by two different on-line measures of effort (a pupillary and an EEG one). Finally, I use surprisal to formalise the measured results and propose an extended formula to take visual information into account.

• https://publikationen.sulb.uni-saarland.de/handle/20.500.11880/27905
• https://doi.org/https://dx.doi.org/10.22028/D291-28451

@phdthesis{Ankener_Diss_2019,
title = {The influence of visual information on word predictability and processing effort},
author = {Christine Ankener},
url = {https://publikationen.sulb.uni-saarland.de/handle/20.500.11880/27905},
doi = {https://doi.org/https://dx.doi.org/10.22028/D291-28451},
year = {2019},
date = {2019},
school = {Saarland University},
address = {Saarbruecken, Germany},
abstract = {A word’s predictability or surprisal in linguistic context, as determined by cloze probabilities or languagemodels (e.g., Frank, 2013a) is related to processing effort, in that less expected words take more effort to process (e.g., Hale, 2001). This shows how, in purely linguistic contexts, rational approaches have been proven valid to predict and formalise results from language processing studies. However, the surprisal (or predictability) of a word may also be influenced by extra-linguistic factors, such as visual context information, as given in situated language processing. While, in the case of linguistic contexts, it is known that the incrementally processed information affects the mental model (e.g., Zwaan and Radvansky, 1998) at each word in a probabilistic way, no such observations have been made so far in the case of visual context information. Although it has been shown that in the visual world paradigm (VWP), anticipatory eye movements suggest that listeners exploit the scene to predict what will be mentioned next (Altmann and Kamide, 1999), it is so far unclear how visual information actually affects expectations for and processing effort of target words. If visual context effects on word processing effort can be observed, we hypothesise that rational concepts can be extended in order to formalise these effects, hereby making them statistically accessible for language models. In a line of experiments, I hence observe how visual information – which is inherently different from linguistic context, for instance in its non-incremental-at once-accessibility– affects target words. Our findings are a clear and robust demonstration that the non-linguistic context can immediately influence both lexical expectations, and surprisal-based processing effort as assessed by two different on-line measures of effort (a pupillary and an EEG one). Finally, I use surprisal to formalise the measured results and propose an extended formula to take visual information into account.},
pubstate = {published},
type = {phdthesis}
}

Copy BibTeX to Clipboard

Behavioral studies have shown that speaker gaze to objects in a co-present scene can influence listeners’ sentence comprehension. To gain deeper insight into the mechanisms involved in gaze processing and integration, we conducted two ERP experiments (N = 30, Age: [18, 32] and [19, 33] respectively). Participants watched a centrally positioned face performing gaze actions aligned to utterances comparing two out of three displayed objects. They were asked to judge whether the sentence was true given the provided scene. We manipulated the second gaze cue to be either Congruent (baseline), Incongruent or Averted (Exp1)/Mutual (Exp2). When speaker gaze is used to form lexical expectations about upcoming referents, we found an attenuated N200 when phonological information confirms these expectations (Congruent). Similarly, we observed attenuated N400 amplitudes when gaze-cued expectations (Congruent) facilitate lexical retrieval. Crucially, only a violation of gaze-cued lexical expectations (Incongruent) leads to a P600 effect, suggesting the necessity to revise the mental representation of the situation. Our results support the hypothesis that gaze is utilized above and beyond simply enhancing a cued object’s prominence. Rather, gaze to objects leads to their integration into the mental representation of the situation before they are mentioned.

• https://www.sciencedirect.com/science/article/pii/S0278262619300120
• https://doi.org/10.1016/j.bandc.2019.05.009

@article{Jachmann2019b,
title = {Influence of speakers’ gaze on situated language comprehension: Evidence from Event-Related Potentials},
author = {Torsten Jachmann and Heiner Drenhaus and Maria Staudte and Matthew W. Crocker},
url = {https://www.sciencedirect.com/science/article/pii/S0278262619300120},
doi = {https://doi.org/10.1016/j.bandc.2019.05.009},
year = {2019},
date = {2019},
journal = {Brain and cognition},
pages = {103571},
publisher = {Elsevier},
volume = {135},
abstract = {Behavioral studies have shown that speaker gaze to objects in a co-present scene can influence listeners’ sentence comprehension. To gain deeper insight into the mechanisms involved in gaze processing and integration, we conducted two ERP experiments (N = 30, Age: [18, 32] and [19, 33] respectively). Participants watched a centrally positioned face performing gaze actions aligned to utterances comparing two out of three displayed objects. They were asked to judge whether the sentence was true given the provided scene. We manipulated the second gaze cue to be either Congruent (baseline), Incongruent or Averted (Exp1)/Mutual (Exp2). When speaker gaze is used to form lexical expectations about upcoming referents, we found an attenuated N200 when phonological information confirms these expectations (Congruent). Similarly, we observed attenuated N400 amplitudes when gaze-cued expectations (Congruent) facilitate lexical retrieval. Crucially, only a violation of gaze-cued lexical expectations (Incongruent) leads to a P600 effect, suggesting the necessity to revise the mental representation of the situation. Our results support the hypothesis that gaze is utilized above and beyond simply enhancing a cued object’s prominence. Rather, gaze to objects leads to their integration into the mental representation of the situation before they are mentioned.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

A word’s predictability or surprisal in linguistic context, as determined by cloze probabilities or languagemodels (e.g. Frank, 2013a) is related to processing effort, in that less expected words take more effort to process (e.g. Hale, 2001). This shows how, in purely linguistic contexts, rational approaches have been proven valid to predict and formalise results from language processing studies. However, the surprisal (or predictability) of a word may also be influenced by extra-linguistic factors, such as visual context information, as given in situated language processing. While, in the case of linguistic contexts, it is known that the incrementally processed information affects the mental model (e.g. Zwaan and Radvansky, 1998) at each word in a probabilistic way, no such observations have been made so far in the case of visual context information. Although it has been shown that in the visual world paradigm (VWP), anticipatory eye movements suggest that listeners exploit the scene to predict what will be mentioned next (Altmann and Kamide, 1999), it is so far unclear how visual information actually affects expectations for and processing effort of target words. If visual context effects on word processing effort can be observed, we hypothesise that rational concepts can be extended in order to formalise these effects, hereby making them statistically accessible for language models. In a line of experiments, I hence observe how visual information – which is inherently different from linguistic context, for instance in its non-incremental-at once-accessibility– affects target words. Our findings are a clear and robust demonstration that the non-linguistic context can immediately influence both lexical expectations, and surprisal-based processing effort as assessed by two different on-line measures of effort (a pupillary and an EEG one). Finally, I use surprisal to formalise the measured results and propose an extended formula to take visual information into account.

• http://nbn-resolving.de/urn:nbn:de:bsz:291--ds-284518
• https://doi.org/http://dx.doi.org/10.22028/D291-28451

@phdthesis{Ankener_diss_2019,
title = {The influence of visual information on word predictability and processing effort},
author = {Christine Ankener},
url = {http://nbn-resolving.de/urn:nbn:de:bsz:291--ds-284518},
doi = {https://doi.org/http://dx.doi.org/10.22028/D291-28451},
year = {2019},
date = {2019},
school = {Saarland University},
address = {Saarbruecken, Germany},
abstract = {A word’s predictability or surprisal in linguistic context, as determined by cloze probabilities or languagemodels (e.g. Frank, 2013a) is related to processing effort, in that less expected words take more effort to process (e.g. Hale, 2001). This shows how, in purely linguistic contexts, rational approaches have been proven valid to predict and formalise results from language processing studies. However, the surprisal (or predictability) of a word may also be influenced by extra-linguistic factors, such as visual context information, as given in situated language processing. While, in the case of linguistic contexts, it is known that the incrementally processed information affects the mental model (e.g. Zwaan and Radvansky, 1998) at each word in a probabilistic way, no such observations have been made so far in the case of visual context information. Although it has been shown that in the visual world paradigm (VWP), anticipatory eye movements suggest that listeners exploit the scene to predict what will be mentioned next (Altmann and Kamide, 1999), it is so far unclear how visual information actually affects expectations for and processing effort of target words. If visual context effects on word processing effort can be observed, we hypothesise that rational concepts can be extended in order to formalise these effects, hereby making them statistically accessible for language models. In a line of experiments, I hence observe how visual information – which is inherently different from linguistic context, for instance in its non-incremental-at once-accessibility– affects target words. Our findings are a clear and robust demonstration that the non-linguistic context can immediately influence both lexical expectations, and surprisal-based processing effort as assessed by two different on-line measures of effort (a pupillary and an EEG one). Finally, I use surprisal to formalise the measured results and propose an extended formula to take visual information into account.},
pubstate = {published},
type = {phdthesis}
}

Copy BibTeX to Clipboard

Questions under Discussion (QUDs) have been suggested to influence the integration of individual utterances into a discourse-level representation. Previous work has shown that processing ungrammatical ellipses is facilitated when the elided material addresses an implicit QUD raised through a nonactuality implicature (NAIs). It is not clear, however, if QUDs influence discourse coherence during comprehension of fully acceptable discourse. We present two ERP studies examining the effects of QUDs introduced by NAIs using two-sentence discourses. Experiment 1 showed that processing definite NPs with inaccessible antecedents is facilitated when their content is relevant to the QUD. Using acceptable discourses, Experiment 2 showed that definite NPs failing to address a QUD elicit increased processing cost. Overall, our results indicate that QUDs raise the expectation that the following discourse will address them, providing unambiguous evidence that their influence is not limited to the processing of ungrammatical input.

• https://doi.org/10.1080/0163853X.2019.1575140
• https://doi.org/10.1080/0163853X.2019.1575140

@article{Delogu2019b,
title = {Discourse Expectations Are Sensitive to the Question Under Discussion: Evidence From ERPs},
author = {Francesca Delogu and Torsten Jachmann and Maria Staudte and Francesco Vespignani and Nicola Molinaro},
url = {https://doi.org/10.1080/0163853X.2019.1575140},
doi = {https://doi.org/10.1080/0163853X.2019.1575140},
year = {2019},
date = {2019},
journal = {Discourse Processes},
pages = {1-19},
abstract = {Questions under Discussion (QUDs) have been suggested to influence the integration of individual utterances into a discourse-level representation. Previous work has shown that processing ungrammatical ellipses is facilitated when the elided material addresses an implicit QUD raised through a nonactuality implicature (NAIs). It is not clear, however, if QUDs influence discourse coherence during comprehension of fully acceptable discourse. We present two ERP studies examining the effects of QUDs introduced by NAIs using two-sentence discourses. Experiment 1 showed that processing definite NPs with inaccessible antecedents is facilitated when their content is relevant to the QUD. Using acceptable discourses, Experiment 2 showed that definite NPs failing to address a QUD elicit increased processing cost. Overall, our results indicate that QUDs raise the expectation that the following discourse will address them, providing unambiguous evidence that their influence is not limited to the processing of ungrammatical input.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

A large body of contemporary psycholinguistic research utilizes the information-theoretic notions related to the transmission of information in an attempt to better understand and formalize the regularities of language production and comprehension. The overarching hypothesis is that prediction is a core mechanism underlying language comprehension. Anticipating what is likely to be mentioned next based on the previous context is what is assumed to allow for smooth and effortless communication. The anticipation of linguistic units that fit the current context reduces the uncertainty about the upcoming material, which consequently facilitates the processing of that material, in a typically noisy channel. Situated language processing allows for the integration of not only linguistic but also non-linguistic visual information that contribute to establishing the context, and facilitate the creation of anticipations regarding the upcoming linguistic material. Moreover, noticing that our interlocutor is directing her attention to a certain object, inspires a shift in our visual attention towards the same entity. Since what is relevant for our interlocutor is highly likely to be relevant for us, too, whether simply conversationally, or more importantly, even existentially (Emery, 2000). Hence, following the speaker’s referential gaze cue towards an object relevant for the current conversation has been shown to benefit listeners’ language processing, measured by shorter reaction times on subsequent tasks (e.g., Staudte & Crocker, 2011; Staudte, Crocker, Heloir, & Kipp, 2014; Knoeferle & Kreysa, 2012; Macdonald & Tatler, 2013, 2014). The present thesis aimed to provide an insight into the mechanisms behind this facilitation. We examined the dynamics of combining visual and linguistic information in creating anticipation for a specific object to be mentioned, and the effect this has on language processing. To this end we used a pupillary measure of cognitive load that is robust enough to allow for free eye movements (the Index of Cognitive Activity; Marshall, 2000). This enabled us to measure not only the visual attention during language comprehension, but also the immediately induced cognitive load at various relevant points during the auditory presentation of the linguistic stimulus. Eight experiments were conducted towards addressing our research questions. The initial three experiments established the ICA measurement in the context of our linguistic manipulation. This series of experiments included reading, cognitive load during listening, and the examination of visual attention together with cognitive load in the visual world paradigm (VWP). Subsequently, we conducted five eye tracking experiments in the VWP where the linguistic context was further enriched by a referential gaze cue. All five experiments simultaneously assessed both visual attention and the immediate cognitive load induced at different stages of sentence processing. We manipulated the existence of the referential gaze cue (Exp. 4), the probability of mention of the cued object (Exp. 4, 5), the congruency of the gaze cue and the subsequent referring expression (Exp. 6), as well as the number of cued objects with equal probability of mention (Exp. 7, 8). Finally, we examined whether the gaze cue can take the role of fully disambiguating the target referent (Exp. 8). We quantified the importance of the visual context in language processing, and showed that if a certain object from the visual context has a higher likelihood of mention given the linguistic context, its processing is facilitated, in comparison to the processing of the same sentence without the visual context. Furthermore, our results support the previous findings that the referential gaze cue leads to a shift in visual attention towards the cued object, thereby facilitating language processing. We expanded these findings by showing that it is the processing of the linguistic reference, that is the referent noun, that is facilitated by gaze-following. Importantly, perceiving and following the gaze cue did not prove costly in terms of cognitive effort, unless the cued object did not fit the verb selectional preferences. This is true regardless of the number of objects cued, or the lower likelihood of mention of the cued object. We conclude that listeners strategically use visual information to reduce the referential uncertainty for upcoming nouns but that the visual cues, such as the referential gaze cue, do not underly the same kinds of expectations (and resulting cognitive costs) as linguistic references. We did not find evidence that the gaze cue is processed in a manner comparable to noun processing, rather, it is likely perceived as a relevant piece of information introduced in addition to the linguistic material, in order to aid language processing, but, importantly, not there to substitute it.

• http://nbn-resolving.de/urn:nbn:de:bsz:291--ds-285651
• https://doi.org/http://dx.doi.org/10.22028/D291-28565

@phdthesis{Sekicki_diss_0919,
title = {Exploiting referential gaze for uncertainty reduction in situated language processing: an information-theoretic approach},
author = {Mirjana Sekicki},
url = {http://nbn-resolving.de/urn:nbn:de:bsz:291--ds-285651},
doi = {https://doi.org/http://dx.doi.org/10.22028/D291-28565},
year = {2019},
date = {2019},
school = {Saarland University},
address = {Saarbr{\"u}cken},
abstract = {A large body of contemporary psycholinguistic research utilizes the information-theoretic notions related to the transmission of information in an attempt to better understand and formalize the regularities of language production and comprehension. The overarching hypothesis is that prediction is a core mechanism underlying language comprehension. Anticipating what is likely to be mentioned next based on the previous context is what is assumed to allow for smooth and effortless communication. The anticipation of linguistic units that fit the current context reduces the uncertainty about the upcoming material, which consequently facilitates the processing of that material, in a typically noisy channel. Situated language processing allows for the integration of not only linguistic but also non-linguistic visual information that contribute to establishing the context, and facilitate the creation of anticipations regarding the upcoming linguistic material. Moreover, noticing that our interlocutor is directing her attention to a certain object, inspires a shift in our visual attention towards the same entity. Since what is relevant for our interlocutor is highly likely to be relevant for us, too, whether simply conversationally, or more importantly, even existentially (Emery, 2000). Hence, following the speaker’s referential gaze cue towards an object relevant for the current conversation has been shown to benefit listeners’ language processing, measured by shorter reaction times on subsequent tasks (e.g., Staudte & Crocker, 2011; Staudte, Crocker, Heloir, & Kipp, 2014; Knoeferle & Kreysa, 2012; Macdonald & Tatler, 2013, 2014). The present thesis aimed to provide an insight into the mechanisms behind this facilitation. We examined the dynamics of combining visual and linguistic information in creating anticipation for a specific object to be mentioned, and the effect this has on language processing. To this end we used a pupillary measure of cognitive load that is robust enough to allow for free eye movements (the Index of Cognitive Activity; Marshall, 2000). This enabled us to measure not only the visual attention during language comprehension, but also the immediately induced cognitive load at various relevant points during the auditory presentation of the linguistic stimulus. Eight experiments were conducted towards addressing our research questions. The initial three experiments established the ICA measurement in the context of our linguistic manipulation. This series of experiments included reading, cognitive load during listening, and the examination of visual attention together with cognitive load in the visual world paradigm (VWP). Subsequently, we conducted five eye tracking experiments in the VWP where the linguistic context was further enriched by a referential gaze cue. All five experiments simultaneously assessed both visual attention and the immediate cognitive load induced at different stages of sentence processing. We manipulated the existence of the referential gaze cue (Exp. 4), the probability of mention of the cued object (Exp. 4, 5), the congruency of the gaze cue and the subsequent referring expression (Exp. 6), as well as the number of cued objects with equal probability of mention (Exp. 7, 8). Finally, we examined whether the gaze cue can take the role of fully disambiguating the target referent (Exp. 8). We quantified the importance of the visual context in language processing, and showed that if a certain object from the visual context has a higher likelihood of mention given the linguistic context, its processing is facilitated, in comparison to the processing of the same sentence without the visual context. Furthermore, our results support the previous findings that the referential gaze cue leads to a shift in visual attention towards the cued object, thereby facilitating language processing. We expanded these findings by showing that it is the processing of the linguistic reference, that is the referent noun, that is facilitated by gaze-following. Importantly, perceiving and following the gaze cue did not prove costly in terms of cognitive effort, unless the cued object did not fit the verb selectional preferences. This is true regardless of the number of objects cued, or the lower likelihood of mention of the cued object. We conclude that listeners strategically use visual information to reduce the referential uncertainty for upcoming nouns but that the visual cues, such as the referential gaze cue, do not underly the same kinds of expectations (and resulting cognitive costs) as linguistic references. We did not find evidence that the gaze cue is processed in a manner comparable to noun processing, rather, it is likely perceived as a relevant piece of information introduced in addition to the linguistic material, in order to aid language processing, but, importantly, not there to substitute it.},
pubstate = {published},
type = {phdthesis}
}

Copy BibTeX to Clipboard

Referential gaze has been shown to benefit language processing in situated communication in terms of shifting visual attention and leading to shorter reaction times on subsequent tasks. The present study simultaneously assessed both visual attention and, importantly, the immediate cogni-tive load induced at different stages of sentence processing. We aimed to examine the dynamics of combining visual and linguistic information in creating anticipation for a specific object and the effect this has on language processing. We report evidence from three visual-world eye-tracking experiments, showing that referential gaze leads to a shift in visual attention toward the cued object, which consequently lowers the effort required for processing the linguistic reference. Importantly, perceiving and following the gaze cue did not prove costly in terms of cognitive effort, unless the cued object did not fit the verb selectional preferences.

• https://onlinelibrary.wiley.com/doi/full/10.1111/cogs.12682
• https://doi.org/10.1111/cogs.12682

@article{ Sekicki2018,
title = {Eye'll help you out! How the gaze cue reduces the cognitive load required for reference processing},
author = {Mirjana Sekicki and Maria Staudte},
url = {https://onlinelibrary.wiley.com/doi/full/10.1111/cogs.12682},
doi = {https://doi.org/10.1111/cogs.12682},
year = {2018},
date = {2018},
journal = {Cognitive Science},
pages = {1-40},
volume = {42},
abstract = {Referential gaze has been shown to benefit language processing in situated communication in terms of shifting visual attention and leading to shorter reaction times on subsequent tasks. The present study simultaneously assessed both visual attention and, importantly, the immediate cogni-tive load induced at different stages of sentence processing. We aimed to examine the dynamics of combining visual and linguistic information in creating anticipation for a specific object and the effect this has on language processing. We report evidence from three visual-world eye-tracking experiments, showing that referential gaze leads to a shift in visual attention toward the cued object, which consequently lowers the effort required for processing the linguistic reference. Importantly, perceiving and following the gaze cue did not prove costly in terms of cognitive effort, unless the cued object did not fit the verb selectional preferences.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

@miscellaneous{Ankener2018b,
title = {Visually informed prediction: How combining lexical and visual information affects surprisal},
author = {Maria Staudte and Christine Ankener},
year = {2018},
date = {2018-10-17},
booktitle = {31st Annual Conference on Sentence Processing (CUNY)},
address = {UC Davis, USA},
pubstate = {published},
type = {miscellaneous}
}

Copy BibTeX to Clipboard

@inproceedings{Sekicki2018c,
title = {Reference resolution and the integration of referential visual cues},
author = {Maria Staudte andMirjana Sekicki},
year = {2018},
date = {2018-10-17},
booktitle = {SSLP (pre-AMLaP) workshop 2018},
address = {Berlin, Germany},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

Gaze Cues in face-to-face interactions

• Speakers‘ direct their gaze toward an object approximately 800ms before mentioning. (Griffin & Bock, 2000)
• Previous studies showed that listeners utilize speakers‘ gaze to form predictions about the unfolding sentence. (Jachmann et al., 2017)
• Do listeners utilize this external cue to validate expectations about the unfolding sentence? And, if so, how does this effect the comprehension of the noun?

• https://www.researchgate.net/publication/327623334_DISCONFIRMATION_OF_LINGUISTIC_PREDICTION_BY_NON-LINGUISTIC_CUES

@miscellaneous{Jachmann2018,
title = {(Dis-)confirmation of linguistic prediction by non-linguistic cues},
author = {Torsten Jachmann and Heiner Drenhaus and Maria Staudte and Matthew W. Crocker},
url = {https://www.researchgate.net/publication/327623334_DISCONFIRMATION_OF_LINGUISTIC_PREDICTION_BY_NON-LINGUISTIC_CUES},
year = {2018},
date = {2018},
booktitle = {24th Annual Conference on Architectures and Mechanisms for Language Processing (AMLaP)},
address = {Berlin},
abstract = {Gaze Cues in face-to-face interactions

• Speakers‘ direct their gaze toward an object approximately 800ms before mentioning. (Griffin & Bock, 2000)
• Previous studies showed that listeners utilize speakers‘ gaze to form predictions about the unfolding sentence. (Jachmann et al., 2017)
• Do listeners utilize this external cue to validate expectations about the unfolding sentence? And, if so, how does this effect the comprehension of the noun?

Copy BibTeX to Clipboard

@inproceedings{Sekicki2018,
title = {Visual cues and the graded reduction of referential uncertainty},
author = {Maria Staudte andMirjana Sekicki},
year = {2018},
date = {2018-10-17},
booktitle = {24th Annual Conference on Architectures and Mechanisms for Language Processing (AMLaP)},
address = {Berlin},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard