Language models as models of human language processing?
Evidence, from priming and information density in dialogue to language acquisition.

David Reitter
Penn State College of IST

The theory of predictive coding maintains that simple, implicit expectations about what happens next in our environment help us perceive and disambiguate a complex world. This principle is no stranger to language processing. But how do we arrive at these predictions, and how do they influence our linguistic behavior? In this talk, I review some recent results from our lab that adopt predictive language models as they are commonly found in natural-language processing systems.
Generative cognitive models of language production, such as the ACT-R model of syntactic priming [1], can explain and predict empirical effects related to priming. By contrast, today’s neural-network (“deep learning”) language models do not represent syntactic structure symbolically, but can achieve high performance capturing distributions in naturalistic data by predicting words given their lexical contexts [2].
In this talk, I will show some applications of language models in computational cognitive science, which suggest that (a) people show systematic sensitivity to the model’s predictions, and (b) that naturalistic, multi-modal learning settings improve model performance. In a first study of human dialogue [3], we quantify information density in language via an entropy-like measure: input that is surprising to the language model also carries more information. We discover that speakers systematically converge in their information density, revealing a topic structure. Second, I examine a language model that learns language in an ecological setting, that is, with access to a visual sensory information [4]: the model indeed shows an improved match to unseen data. Are language models a good starting point for understanding a cognitive process?

[1] Reitter, Keller, & Moore (2011), Cognitive Science 35(4)

 

[2] Ororbia, Mikolov, & Reitter (2017), Neural Computation 29(12)

 

[3] Xu & Reitter (2018), Cognition (170)

 

[4] Ororbia, Mali, Kelly, & Reitter (submitted – ACL)

 

If you would like to meet with the speaker please contact Vera Demberg.