Language models as models of human language processing? Evidence, from priming and information density in dialogue to language acquisition. - Speaker: David Reitter
Penn State College of IST
The theory of predictive coding maintains that simple, implicit expectations about what happens next in our environment help us perceive and disambiguate a complex world. This principle is no stranger to language processing. But how do we arrive at these predictions, and how do they influence our linguistic behavior? In this talk, I review some recent results from our lab that adopt predictive language models as they are commonly found in natural-language processing systems.
Generative cognitive models of language production, such as the ACT-R model of syntactic priming , can explain and predict empirical effects related to priming. By contrast, today’s neural-network (“deep learning”) language models do not represent syntactic structure symbolically, but can achieve high performance capturing distributions in naturalistic data by predicting words given their lexical contexts .
In this talk, I will show some applications of language models in computational cognitive science, which suggest that (a) people show systematic sensitivity to the model’s predictions, and (b) that naturalistic, multi-modal learning settings improve model performance. In a first study of human dialogue , we quantify information density in language via an entropy-like measure: input that is surprising to the language model also carries more information. We discover that speakers systematically converge in their information density, revealing a topic structure. Second, I examine a language model that learns language in an ecological setting, that is, with access to a visual sensory information : the model indeed shows an improved match to unseen data. Are language models a good starting point for understanding a cognitive process?
 Reitter, Keller, & Moore (2011), Cognitive Science 35(4)
 Ororbia, Mikolov, & Reitter (2017), Neural Computation 29(12)
 Xu & Reitter (2018), Cognition (170)
 Ororbia, Mali, Kelly, & Reitter (submitted – ACL)
If you would like to meet with the speaker please contact Vera Demberg.