Publications

Referential gaze has been shown to benefit language processing in situated communication in terms of shifting visual attention and leading to shorter reaction times on subsequent tasks. The present study simultaneously assessed both visual attention and, importantly, the immediate cognitive load induced at different stages of sentence processing.

We aimed to examine the dynamics of combining visual and linguistic information in creating anticipation for a specific object and the effect this has on language processing. We report evidence from three visual‐world eye‐tracking experiments, showing that referential gaze leads to a shift in visual attention toward the cued object, which consequently lowers the effort required for processing the linguistic reference.

Importantly, perceiving and following the gaze cue did not prove costly in terms of cognitive effort, unless the cued object did not fit the verb selectional preferences.

• https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6585668/
• https://doi.org/10.1111/cogs.12682

@article{Sekicki2018c,
title = {Eye’ll help you out! How the gaze cue reduces the cognitive load required for reference processing},
author = {Maria Staudte andMirjana Sekicki},
url = {https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6585668/},
doi = {https://doi.org/10.1111/cogs.12682},
year = {2018},
date = {2018-10-17},
journal = {Cognitive science},
pages = {2418-2458},
volume = {42},
number = {8},
abstract = {Referential gaze has been shown to benefit language processing in situated communication in terms of shifting visual attention and leading to shorter reaction times on subsequent tasks. The present study simultaneously assessed both visual attention and, importantly, the immediate cognitive load induced at different stages of sentence processing. We aimed to examine the dynamics of combining visual and linguistic information in creating anticipation for a specific object and the effect this has on language processing. We report evidence from three visual‐world eye‐tracking experiments, showing that referential gaze leads to a shift in visual attention toward the cued object, which consequently lowers the effort required for processing the linguistic reference. Importantly, perceiving and following the gaze cue did not prove costly in terms of cognitive effort, unless the cued object did not fit the verb selectional preferences.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

Speakers tend to fixate objects they are about to mention, while listeners inspect those objects that they believe to be intended referents of the speaker. These production- and comprehension-contingent gaze behaviors may form an integral part of the signal itself, making it inherently reciprocal.

Here, we present work that has investigated the interplay of gaze and language and assessed the role of speaker gaze for language comprehension as well as the utility of listener gaze for an instruction giver. Both lines of research make use of artificial interaction partners which increases experimental control while maintaining a dynamic interactive setting. Thus, the reciprocal nature of situated dialogue becomes a tractable aspect in the enterprise of dealing with human (gaze) behavior.

• https://benjamins.com/catalog/ais.10.05sta
• https://doi.org/10.1075/ais.10.05sta

@incollection{Staudte2018,
title = {On the role of gaze for successful and efficient communication},
author = {Maria Staudte and Matthew W. Crocker},
url = {https://benjamins.com/catalog/ais.10.05sta},
doi = {https://doi.org/10.1075/ais.10.05sta},
year = {2018},
date = {2018-10-17},
booktitle = {Eye-tracking in Interaction: Studies on the role of eye gaze in dialogue},
publisher = {John Benjamins Publishing Company},
abstract = {Speakers tend to fixate objects they are about to mention, while listeners inspect those objects that they believe to be intended referents of the speaker. These production- and comprehension-contingent gaze behaviors may form an integral part of the signal itself, making it inherently reciprocal. Here, we present work that has investigated the interplay of gaze and language and assessed the role of speaker gaze for language comprehension as well as the utility of listener gaze for an instruction giver. Both lines of research make use of artificial interaction partners which increases experimental control while maintaining a dynamic interactive setting. Thus, the reciprocal nature of situated dialogue becomes a tractable aspect in the enterprise of dealing with human (gaze) behavior.},
pubstate = {published},
type = {incollection}
}

Copy BibTeX to Clipboard

A word’s predictability or surprisal, as determined by cloze probabilities or language models (Frank, 2013) is related to processing effort, in that less expected words take more effort to process (Hale, 2001; Lau et al., 2013).

A word’s surprisal, however, may also be influenced by the non-linguistic context, such as visual cues: In the visual world paradigm (VWP), anticipatory eye movements suggest that listeners exploit the scene to predict what will be mentioned next (Altmann and Kamide, 1999). How visual context affects surprisal and processing effort, however, remains unclear.

Here, we present a series of four studies providing evidence on how visually-determined probabilistic expectations for a spoken target word, as indicated by anticipatory eye movements, predict graded processing effort for that word, as assessed by a pupillometric measure (the Index of Cognitive Activity, ICA). These findings are a clear and robust demonstration that the non-linguistic context can immediately influence both lexical expectations, and surprisal-based processing effort.

• https://www.frontiersin.org/articles/10.3389/fpsyg.2018.02387/full
• https://doi.org/10.3389/fpsyg.2018.02387

@article{Ankener2018b,
title = {The Influence of Visual Uncertainty on Word Surprisal and Processing Effort},
author = {Maria Staudte andMirjana Sekicki and Christine Ankener},
url = {https://www.frontiersin.org/articles/10.3389/fpsyg.2018.02387/full},
doi = {https://doi.org/10.3389/fpsyg.2018.02387},
year = {2018},
date = {2018-10-17},
journal = {Frontiers in Psychology},
pages = {2387},
volume = {9},
abstract = {A word’s predictability or surprisal, as determined by cloze probabilities or language models (Frank, 2013) is related to processing effort, in that less expected words take more effort to process (Hale, 2001; Lau et al., 2013). A word’s surprisal, however, may also be influenced by the non-linguistic context, such as visual cues: In the visual world paradigm (VWP), anticipatory eye movements suggest that listeners exploit the scene to predict what will be mentioned next (Altmann and Kamide, 1999). How visual context affects surprisal and processing effort, however, remains unclear. Here, we present a series of four studies providing evidence on how visually-determined probabilistic expectations for a spoken target word, as indicated by anticipatory eye movements, predict graded processing effort for that word, as assessed by a pupillometric measure (the Index of Cognitive Activity, ICA). These findings are a clear and robust demonstration that the non-linguistic context can immediately influence both lexical expectations, and surprisal-based processing effort.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

A word’s predictability or surprisal, as determined by cloze probabilities or language models (e.g. Frank, Otten, Galli, & Vigliocco, 2015) is related to processing effort, in that less expected words take more effort to process (e.g. Hale, 2001). A words surprisal, however, may also be influenced by the non-linguistic context, such as visual cues: In the visual world paradigm (VWP), for example, anticipatory eye movements suggest that comprehenders exploit the scene to predict what will be mentioned next (Altmann & Kamide, 1999).

How visual context affects word surprisal and processing effort, however, remains unclear. Here, we present evidence that visually-determined probabilistic expectations for a spoken target word predict graded processing effort for that word, in both pupillometric (ICA) and ERP (N400) measures. These findings demonstrate that the non-linguistic context can immediately influence both lexical expectations, and surprisal-based processing effort.

• https://www.researchgate.net/publication/325644935_Multimodal_Surprisal_in_the_N400_and_the_Index_of_Cognitive_Activity

@inproceedings{Ankener2018,
title = {Multimodal Surprisal in the N400 and the Index of Cognitive Activity},
author = {Christine Ankener and Heiner Drenhaus and Matthew W. Crocker and Maria Staudte},
url = {https://www.researchgate.net/publication/325644935_Multimodal_Surprisal_in_the_N400_and_the_Index_of_Cognitive_Activity},
year = {2018},
date = {2018},
booktitle = {Proceedings of the 40th Annual Cognitive Science Society Meeting},
pages = {94-100},
publisher = {The Cognitive Science Society},
address = {Madison, Wisconsin},
abstract = {A word’s predictability or surprisal, as determined by cloze probabilities or language models (e.g. Frank, Otten, Galli, & Vigliocco, 2015) is related to processing effort, in that less expected words take more effort to process (e.g. Hale, 2001). A words surprisal, however, may also be influenced by the non-linguistic context, such as visual cues: In the visual world paradigm (VWP), for example, anticipatory eye movements suggest that comprehenders exploit the scene to predict what will be mentioned next (Altmann & Kamide, 1999). How visual context affects word surprisal and processing effort, however, remains unclear. Here, we present evidence that visually-determined probabilistic expectations for a spoken target word predict graded processing effort for that word, in both pupillometric (ICA) and ERP (N400) measures. These findings demonstrate that the non-linguistic context can immediately influence both lexical expectations, and surprisal-based processing effort.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

1. Does following a gaze cue influence the cognitive load required for processing the corresponding linguistic referent?
2. Is considering the gaze cue costly? Is there a distribution of cognitive load between the cue and the referent?
3. Can a gaze cue have a disruptive effect on processing the linguistic referent?

• Sekicki_Staudte_CUNY2017 (1)     (2.54MB)

@miscellaneous{Sekicki2017,
title = {Cognitive load in the visual world: The facilitatory effect of gaze},
author = {Mirjana Sekicki and Maria Staudte},
year = {2017},
date = {2017},
publisher = {39th Annual Meeting of the Cognitive Science Society},
address = {London, UK},
abstract = {

1. Does following a gaze cue influence the cognitive load required for processing the corresponding linguistic referent?
2. Is considering the gaze cue costly? Is there a distribution of cognitive load between the cue and the referent?
3. Can a gaze cue have a disruptive effect on processing the linguistic referent?

Copy BibTeX to Clipboard

• https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6302025/

@inproceedings{Ankener2016,
title = {The influence of visual context on predictions in sentence processing: Evidence from ICA},
author = {Maria Staudte},
url = {https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6302025/},
year = {2016},
date = {2016},
booktitle = {Proceedings at the Language and Perception International Conference},
address = {Trondheim, Norwegen},
abstract = {

Copy BibTeX to Clipboard

• ESLP2016_book_of_abstracts_2     (10.01MB)

@inproceedings{sekicki2016b,
title = {Cost and Gains of Using Visual Context for Referent Prediction},
author = {Maria Staudte},
year = {2016},
date = {2016-10-18},
booktitle = {Proceedings of the 9th Embodied and Situated Language Processing Conference (ESLP)},
address = {Pucón},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

We investigated the effect of visual context on cognitive load (CL) that is induced by prediction forming during sentence processing, using a novel measure of CL: the Index of Cognitive Activity. We conducted two experiments, one including only linguistic stimuli (LING) and one with the additional visual context of four potential target objects (VIS). Noun predictability was modulated by verb constraint (ironable vs. describable objects) and thematic fit; and further by visual competitors (two ironable vs. four describable objects).
“The woman (1) irons / (2) describes soon the (a) t-shirt / (b) sock.“
We found lower CL on the noun in (1a) compared to (1b) in both studies, suggesting that after “iron“, “t-shirt“ was more predictable, and hence easier to process, than “sock“. More importantly, VIS findings show higher CL on “iron“ compared to “describe“, suggesting that visual context allowed for active exclusion of two non-ironable targets. Conversely, CL on nouns was lower when following “iron“ than “describe“, due to only one ironable competitor compared to three describable competitors. These findings suggest that the presence of visual context alters the distribution of CL during sentence processing. Future work includes gaze cues as additional information, potentially further affecting CL distribution.

@inproceedings{sekicki2016,
title = {Language Processing: Cognitive Load with(out) Visual Context},
author = {Mirjana Sekicki and Christine Ankener and Maria Staudte},
year = {2016},
date = {2016-10-18},
booktitle = {Proceedings at the 22nd Annual Conference on Architectures and Mechanisms for Language Processing (AMLaP)},
address = {Bilbao, Spain},
abstract = {We investigated the effect of visual context on cognitive load (CL) that is induced by prediction forming during sentence processing, using a novel measure of CL: the Index of Cognitive Activity. We conducted two experiments, one including only linguistic stimuli (LING) and one with the additional visual context of four potential target objects (VIS). Noun predictability was modulated by verb constraint (ironable vs. describable objects) and thematic fit; and further by visual competitors (two ironable vs. four describable objects). ''The woman (1) irons / (2) describes soon the (a) t-shirt / (b) sock.'' We found lower CL on the noun in (1a) compared to (1b) in both studies, suggesting that after ''iron'', ''t-shirt'' was more predictable, and hence easier to process, than ''sock''. More importantly, VIS findings show higher CL on ''iron'' compared to ''describe'', suggesting that visual context allowed for active exclusion of two non-ironable targets. Conversely, CL on nouns was lower when following ''iron'' than ''describe'', due to only one ironable competitor compared to three describable competitors. These findings suggest that the presence of visual context alters the distribution of CL during sentence processing. Future work includes gaze cues as additional information, potentially further affecting CL distribution.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

Contexts that constrain upcoming words to some higher or lower extent can be composed differently but are typically all evaluated using cloze-probability (Rayner & Well, 1996). Less predicted words were found to correlate with more negative N400 (e.g., Frank et al., 2015; Kutas & Hillyard, 1984) and longer reading times (Rayner & Well, 1996; Smith & Levy, 2013). Recently, however, it has been suggested that predictability, as in cloze-probability, is only one influence on processing cost (e.g., DeLong et al., 2014). As DeLong et al. show, differences in plausibility of words with similar cloze-probability also affect processing of such words, reflected in different ERP components. This hints at a difference between frequency-based and deeper semantic processing. Moreover, a relatively novel measure, the Index of Cognitive Activity (ICA) capturing pupil jitter, has been linked to cognitive load and predictability (Demberg et al., 2013).

• CUNY2016     (0.06MB)
• https://www.coli.uni-saarland.de/~mirjana/papers/CUNY2016.pdf

@inproceedings{CUNY2016_A5,
title = {Low Predictability: An Empirical Comparison of Paradigms Used for Sentence Comprehension},
author = {Maria Staudte},
url = {https://www.coli.uni-saarland.de/~mirjana/papers/CUNY2016.pdf},
year = {2016},
date = {2016},
booktitle = {29th Annual Conference on Human Sentence Processing (CUNY)},
address = {Gainesville, FL},
abstract = {Contexts that constrain upcoming words to some higher or lower extent can be composed differently but are typically all evaluated using cloze-probability (Rayner & Well, 1996). Less predicted words were found to correlate with more negative N400 (e.g., Frank et al., 2015; Kutas & Hillyard, 1984) and longer reading times (Rayner & Well, 1996; Smith & Levy, 2013). Recently, however, it has been suggested that predictability, as in cloze-probability, is only one influence on processing cost (e.g., DeLong et al., 2014). As DeLong et al. show, differences in plausibility of words with similar cloze-probability also affect processing of such words, reflected in different ERP components. This hints at a difference between frequency-based and deeper semantic processing. Moreover, a relatively novel measure, the Index of Cognitive Activity (ICA) capturing pupil jitter, has been linked to cognitive load and predictability (Demberg et al., 2013).},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard