Ankener, Christine; Drenhaus, Heiner; Crocker, Matthew W.; Staudte, Maria
Multimodal Surprisal in the N400 and the Index of Cognitive Activity
Proceedings of the 40th Annual Cognitive Science Society Meeting Proceedings of the 40th Annual Cognitive Science Society Meeting , The Cognitive Science Society, pp. 94-100, Madison, Wisconsin, 2018.
A word’s predictability or surprisal, as determined by cloze probabilities or language models (e.g. Frank, Otten, Galli, & Vigliocco, 2015) is related to processing effort, in that less expected words take more effort to process (e.g. Hale, 2001). A words surprisal, however, may also be influenced by the non-linguistic context, such as visual cues: In the visual world paradigm (VWP), for example, anticipatory eye movements suggest that comprehenders exploit the scene to predict what will be mentioned next (Altmann & Kamide, 1999).
How visual context affects word surprisal and processing effort, however, remains unclear. Here, we present evidence that visually-determined probabilistic expectations for a spoken target word predict graded processing effort for that word, in both pupillometric (ICA) and ERP (N400) measures. These findings demonstrate that the non-linguistic context can immediately influence both lexical expectations, and surprisal-based processing effort.