Sekicki, Mirjana; Ankener, Christine; Staudte, Maria
Language Processing: Cognitive Load with(out) Visual Context
Proceedings at the 22nd Annual Conference on Architectures and Mechanisms for Language Processing (AMLaP), Bilbao, Spain, 2016.
We investigated the effect of visual context on cognitive load (CL) that is induced by prediction forming during sentence processing, using a novel measure of CL: the Index of Cognitive Activity. We conducted two experiments, one including only linguistic stimuli (LING) and one with the additional visual context of four potential target objects (VIS). Noun predictability was modulated by verb constraint (ironable vs. describable objects) and thematic fit; and further by visual competitors (two ironable vs. four describable objects).
“The woman (1) irons / (2) describes soon the (a) t-shirt / (b) sock.“
We found lower CL on the noun in (1a) compared to (1b) in both studies, suggesting that after “iron“, “t-shirt“ was more predictable, and hence easier to process, than “sock“. More importantly, VIS findings show higher CL on “iron“ compared to “describe“, suggesting that visual context allowed for active exclusion of two non-ironable targets. Conversely, CL on nouns was lower when following “iron“ than “describe“, due to only one ironable competitor compared to three describable competitors. These findings suggest that the presence of visual context alters the distribution of CL during sentence processing. Future work includes gaze cues as additional information, potentially further affecting CL distribution.