Publications

We introduce a large dataset of narrative texts and questions about these texts, intended to be used in a machine comprehension task that requires reasoning using commonsense knowledge. Our dataset complements similar datasets in that we focus on stories about everyday activities, such as going to the movies or working in the garden, and that the questions require commonsense knowledge, or more specifically, script knowledge, to be answered. We show that our mode of data collection via crowdsourcing results in a substantial amount of such inference questions. The dataset forms the basis of a shared task on commonsense and script knowledge organized at SemEval 2018 and provides challenging test cases for the broader natural language understanding community

• https://aclanthology.org/L18-1564"

@inproceedings{MCScript,
title = {MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge},
author = {Michael Roth and Stefan Thater andSimon Ostermann and Ashutosh Modi and Manfred Pinkal},
url = {https://aclanthology.org/L18-1564"},
year = {2018},
date = {2018},
booktitle = {Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018)},
publisher = {European Language Resources Association (ELRA)},
address = {Miyazaki, Japan},
abstract = {We introduce a large dataset of narrative texts and questions about these texts, intended to be used in a machine comprehension task that requires reasoning using commonsense knowledge. Our dataset complements similar datasets in that we focus on stories about everyday activities, such as going to the movies or working in the garden, and that the questions require commonsense knowledge, or more specifically, script knowledge, to be answered. We show that our mode of data collection via crowdsourcing results in a substantial amount of such inference questions. The dataset forms the basis of a shared task on commonsense and script knowledge organized at SemEval 2018 and provides challenging test cases for the broader natural language understanding community},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

This report summarizes the results of the SemEval 2018 task on machine comprehension using commonsense knowledge. For this machine comprehension task, we created a new corpus, MCScript. It contains a high number of questions that require commonsense knowledge for finding the correct answer. 11 teams from 4 different countries participated in this shared task, most of them used neural approaches. The best performing system achieves an accuracy of 83.95%, outperforming the baselines by a large margin, but still far from the human upper bound, which was found to be at 98%.

• https://aclanthology.org/S18-1119
• https://doi.org/10.18653/v1/S18-1119

@inproceedings{SemEval2018Task11,
title = {SemEval-2018 Task 11: Machine Comprehension using Commonsense Knowledge},
author = {Michael Roth and Stefan Thater andSimon Ostermann and Ashutosh Modi and Manfred Pinkal},
url = {https://aclanthology.org/S18-1119},
doi = {https://doi.org/10.18653/v1/S18-1119},
year = {2018},
date = {2018},
booktitle = {Proceedings of the 12th International Workshop on Semantic Evaluation},
pages = {747-757},
publisher = {Association for Computational Linguistics},
address = {New Orleans, Louisiana},
abstract = {This report summarizes the results of the SemEval 2018 task on machine comprehension using commonsense knowledge. For this machine comprehension task, we created a new corpus, MCScript. It contains a high number of questions that require commonsense knowledge for finding the correct answer. 11 teams from 4 different countries participated in this shared task, most of them used neural approaches. The best performing system achieves an accuracy of 83.95%, outperforming the baselines by a large margin, but still far from the human upper bound, which was found to be at 98%.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

Script knowledge plays a central role in text understanding and is relevant for a variety of downstream tasks. In this paper, we consider two recent datasets which provide a rich and general representation of script events in terms of paraphrase sets.

We introduce the task of mapping event mentions in narrative texts to such script event types, and present a model for this task that exploits rich linguistic representations as well as information on temporal ordering. The results of our experiments demonstrate that this complex task is indeed feasible.

• http://www.aclweb.org/anthology/S17-1016

@inproceedings{ostermann-EtAl:2017:starSEM,
title = {Aligning Script Events with Narrative Texts},
author = {Michael Roth and Stefan Thater andSimon Ostermann and Manfred Pinkal},
url = {http://www.aclweb.org/anthology/S17-1016},
year = {2017},
date = {2017-10-17},
booktitle = {Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017)},
publisher = {Association for Computational Linguistics},
address = {Vancouver, Canada},
abstract = {Script knowledge plays a central role in text understanding and is relevant for a variety of downstream tasks. In this paper, we consider two recent datasets which provide a rich and general representation of script events in terms of paraphrase sets. We introduce the task of mapping event mentions in narrative texts to such script event types, and present a model for this task that exploits rich linguistic representations as well as information on temporal ordering. The results of our experiments demonstrate that this complex task is indeed feasible.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

Word embeddings are now a standard technique for inducing meaning representations for words. For getting good representations, it is important to take into account different senses of a word. In this paper, we propose a mixture model for learning multi-sense word embeddings.

Our model generalizes the previous works in that it allows to induce different weights of different senses of a word. The experimental results show that our model outperforms previous models on standard evaluation tasks.

• http://www.aclweb.org/anthology/S17-1015

@inproceedings{nguyen-EtAl:2017:starSEM,
title = {A Mixture Model for Learning Multi-Sense Word Embeddings},
author = {Dai Quoc Nguyen and Dat Quoc Nguyen and Ashutosh Modi and Stefan Thater and Manfred Pinkal},
url = {http://www.aclweb.org/anthology/S17-1015},
year = {2017},
date = {2017},
pages = {121-127},
publisher = {Association for Computational Linguistics},
address = {Vancouver, Canada},
abstract = {Word embeddings are now a standard technique for inducing meaning representations for words. For getting good representations, it is important to take into account different senses of a word. In this paper, we propose a mixture model for learning multi-sense word embeddings. Our model generalizes the previous works in that it allows to induce different weights of different senses of a word. The experimental results show that our model outperforms previous models on standard evaluation tasks.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

This paper presents an approach to the task of predicting an event description from a preceding sentence in a text. Our approach explores sequence-to-sequence learning using a bidirectional multi-layer recurrent neural network. Our approach substantially outperforms previous work in terms of the BLEU score on two datasets derived from WikiHow and DeScript respectively.

Since the BLEU score is not easy to interpret as a measure of event prediction, we complement our study with a second evaluation that exploits the rich linguistic annotation of gold paraphrase sets of events.

• I17-2007     (0.16MB)
• http://www.aclweb.org/anthology/I17-2007

@inproceedings{nguyen-EtAl:2017:I17-2,
title = {Sequence to Sequence Learning for Event Prediction},
author = {Dai Quoc Nguyen and Dat Quoc Nguyen and Cuong Xuan Chu and Stefan Thater and Manfred Pinkal},
url = {http://www.aclweb.org/anthology/I17-2007},
year = {2017},
date = {2017-10-17},
booktitle = {Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)},
pages = {37-42},
publisher = {Asian Federation of Natural Language Processing},
address = {Taipei, Taiwan},
abstract = {This paper presents an approach to the task of predicting an event description from a preceding sentence in a text. Our approach explores sequence-to-sequence learning using a bidirectional multi-layer recurrent neural network. Our approach substantially outperforms previous work in terms of the BLEU score on two datasets derived from WikiHow and DeScript respectively. Since the BLEU score is not easy to interpret as a measure of event prediction, we complement our study with a second evaluation that exploits the rich linguistic annotation of gold paraphrase sets of events.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

• http://www.frontiersin.org/language_sciences/10.3389/fpsyg.2016.00844/abstract
• https://doi.org/10.3389/fpsyg.2016.00844

@article{Zarcone2016,
title = {Salience and attention in surprisal-based accounts of language processing},
author = {Alessandra Zarcone and Marten van Schijndel and Jorrig Vogels and Vera Demberg},
url = {http://www.frontiersin.org/language_sciences/10.3389/fpsyg.2016.00844/abstract},
doi = {https://doi.org/10.3389/fpsyg.2016.00844},
year = {2016},
date = {2016},
journal = {Frontiers in Psychology},
volume = {7},
number = {844},
abstract = {

Copy BibTeX to Clipboard

A common problem in cognitive modelling is lack of access to accurate broad-coverage models of event-level surprisal. As shown in,
e.g., Bicknell et al. (2010), event-level knowledge does affect human expectations for verbal arguments. For example, the model should be able to predict that mechanics are likely to check tires, while journalists are more likely to check typos. Similarly, we would like to predict what locations are likely for playing football or playing flute in order to estimate the surprisal of actually-encountered locations. Furthermore, such a model can be used to provide a probability distribution over fillers for a thematic role which is not mentioned in the text at all.

To this end, we train two neural network models (an incremental one and a non-incremental one) on large amounts of automatically rolelabelled text. Our models are probabilistic and can handle several roles at once, which also enables them to learn interactions between different role fillers. Evaluation shows a drastic improvement over current state-of-the-art systems on modelling human thematic fit judgements, and we demonstrate via a sentence similarity task that the system learns highly useful embeddings.

• https://www.semanticscholar.org/paper/Event-participant-modelling-with-neural-networks-Tilk-Demberg/d08d663d7795c76bb008f539b1ac7caf8a9ef26c

@inproceedings{Tilk2016,
title = {Event participation modelling with neural networks},
author = {Ottokar Tilk and Vera Demberg and Asad Sayeed and Dietrich Klakow and Stefan Thater},
url = {https://www.semanticscholar.org/paper/Event-participant-modelling-with-neural-networks-Tilk-Demberg/d08d663d7795c76bb008f539b1ac7caf8a9ef26c},
year = {2016},
date = {2016},
publisher = {Conference on Empirical Methods in Natural Language Processing},
address = {Austin, Texas, USA},
abstract = {A common problem in cognitive modelling is lack of access to accurate broad-coverage models of event-level surprisal. As shown in, e.g., Bicknell et al. (2010), event-level knowledge does affect human expectations for verbal arguments. For example, the model should be able to predict that mechanics are likely to check tires, while journalists are more likely to check typos. Similarly, we would like to predict what locations are likely for playing football or playing flute in order to estimate the surprisal of actually-encountered locations. Furthermore, such a model can be used to provide a probability distribution over fillers for a thematic role which is not mentioned in the text at all. To this end, we train two neural network models (an incremental one and a non-incremental one) on large amounts of automatically rolelabelled text. Our models are probabilistic and can handle several roles at once, which also enables them to learn interactions between different role fillers. Evaluation shows a drastic improvement over current state-of-the-art systems on modelling human thematic fit judgements, and we demonstrate via a sentence similarity task that the system learns highly useful embeddings.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

Recent research in psycholinguistics has provided increasing evidence that humans predict upcoming content. Prediction also affects perception and might be a key to robustness in human language processing. In this paper, we investigate the factors that affect human prediction by building a computational model that can predict upcoming discourse referents based on linguistic knowledge alone vs. linguistic knowledge jointly with common-sense knowledge in the form of scripts. We find that script knowledge significantly improves model estimates of human predictions. In a second study, we test the highly controversial hypothesis that predictability influences referring expression type but do not find evidence for such an effect.

• https://aclanthology.org/Q17-1003
• https://doi.org/10.1162/tacl_a_00044

@article{ashutoshTacl2016,
title = {Modeling Semantic Expectations: Using Script Knowledge for Referent Prediction},
author = {Ashutosh Modi and Ivan Titov and Vera Demberg and Asad Sayeed and Manfred Pinkal},
url = {https://aclanthology.org/Q17-1003},
doi = {https://doi.org/10.1162/tacl_a_00044},
year = {2016},
date = {2016},
journal = {Transactions of the Association for Computational Linguistics},
pages = {31-44},
publisher = {MIT Press},
address = {Cambridge, MA},
abstract = {Recent research in psycholinguistics has provided increasing evidence that humans predict upcoming content. Prediction also affects perception and might be a key to robustness in human language processing. In this paper, we investigate the factors that affect human prediction by building a computational model that can predict upcoming discourse referents based on linguistic knowledge alone vs. linguistic knowledge jointly with common-sense knowledge in the form of scripts. We find that script knowledge significantly improves model estimates of human predictions. In a second study, we test the highly controversial hypothesis that predictability influences referring expression type but do not find evidence for such an effect.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

Semantic scripts is a conceptual representation which defines how events are organized into higher level activities. Practically all the previous approaches to inducing script knowledge from text relied on count-based techniques (e.g., generative models) and have not attempted to compositionally model events. In this work, we introduce a neural network model which relies on distributed compositional representations of events. The model captures statistical dependencies between events in a scenario, overcomes some of the shortcomings of previous approaches (e.g., by more effectively dealing with data sparsity) and outperforms count-based counterparts on the narrative cloze task.

• https://www.researchgate.net/publication/306093411_Event_Embeddings_for_Semantic_Script_Modeling

@inproceedings{modi:CONLL2016,
title = {Event Embeddings for Semantic Script Modeling},
author = {Ashutosh Modi},
url = {https://www.researchgate.net/publication/306093411_Event_Embeddings_for_Semantic_Script_Modeling},
year = {2016},
date = {2016-10-17},
booktitle = {Proceedings of the Conference on Computational Natural Language Learning (CoNLL)},
address = {Berlin, Germany},
abstract = {Semantic scripts is a conceptual representation which defines how events are organized into higher level activities. Practically all the previous approaches to inducing script knowledge from text relied on count-based techniques (e.g., generative models) and have not attempted to compositionally model events. In this work, we introduce a neural network model which relies on distributed compositional representations of events. The model captures statistical dependencies between events in a scenario, overcomes some of the shortcomings of previous approaches (e.g., by more effectively dealing with data sparsity) and outperforms count-based counterparts on the narrative cloze task.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

This paper presents the InScript corpus (Narrative Texts Instantiating Script structure). InScript is a corpus of 1,000 stories centered around 10 different scenarios. Verbs and noun phrases are annotated with event and participant types, respectively. Additionally, the text is annotated with coreference information. The corpus shows rich lexical variation and will serve as a unique resource for the study of the role of script knowledge in natural language processing.

• 352_Paper     (0.62MB)
• https://aclanthology.org/L16-1555

@inproceedings{MODI16.352,
title = {InScript: Narrative texts annotated with script information},
author = {Ashutosh Modi and Tatjana Anikina and Simon Ostermann and Manfred Pinkal},
editor = {Nicoletta Calzolari and Khalid Choukri and Thierry Declerck and Marko Grobelnik and Bente Maegaard and Joseph Mariani and Asuncion Moreno and Jan Odijk and Stelios Piperidis},
url = {https://aclanthology.org/L16-1555},
year = {2016},
date = {2016-10-17},
booktitle = {Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016)},
isbn = {978-2-9517408-9-1},
pages = {3485-3493},
publisher = {European Language Resources Association (ELRA)},
address = {Portoro{\v{z}, Slovenia},
abstract = {This paper presents the InScript corpus (Narrative Texts Instantiating Script structure). InScript is a corpus of 1,000 stories centered around 10 different scenarios. Verbs and noun phrases are annotated with event and participant types, respectively. Additionally, the text is annotated with coreference information. The corpus shows rich lexical variation and will serve as a unique resource for the study of the role of script knowledge in natural language processing.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

• paper19     (0.8MB)
• https://www.researchgate.net/publication/282740292_Same_same_but_different_Type_and_typicality_in_a_distributional_model_of_complement_coercion

@inproceedings{zarcone2015same,
title = {Same Same but Different: Type and Typicality in a Distributional Model of Complement Coercion},
author = {Alessandra Zarcone and Sebastian Padó and Alessandro Lenci},
url = {https://www.researchgate.net/publication/282740292_Same_same_but_different_Type_and_typicality_in_a_distributional_model_of_complement_coercion},
year = {2015},
date = {2015},
booktitle = {Word Structure and Word Usage. Proceedings of the NetWordS Final Conference. Pisa, March 30-April 1, 2015},
pages = {91-94},
address = {Pisa, Italy},
abstract = {

Copy BibTeX to Clipboard

• ki16     (1.52MB)
• https://link.springer.com/article/10.1007/s13218-015-0409-5

@article{demberg:hoffmann:ki-2015,
title = {Search Challenges in Natural Language Generation with Complex Optimization Objectives},
author = {Vera Demberg and J{\"o}rg Hoffmann and David M. Howcroft and Dietrich Klakow and {\'A}lvaro Torralba},
url = {https://link.springer.com/article/10.1007/s13218-015-0409-5},
year = {2015},
date = {2015},
journal = {KI - K{\"u}nstliche Intelligenz, Special Issue on Companion Technologies},
publisher = {Springer Berlin Heidelberg},
abstract = {

Copy BibTeX to Clipboard

• https://www.semanticscholar.org/paper/Semantically-underinformative-utterances-trigger-Kravtchenko-Demberg/33256a5fca918eef5de8998db5a695d9bced5975

@proceeding{kravtchenko:demberg,
title = {Semantically Underinformative Utterances Trigger Pragmatic Inferences},
author = {Ekaterina Kravtchenko and Vera Demberg},
url = {https://www.semanticscholar.org/paper/Semantically-underinformative-utterances-trigger-Kravtchenko-Demberg/33256a5fca918eef5de8998db5a695d9bced5975},
year = {2015},
date = {2015-10-17},
booktitle = {Annual Conference of the Cognitive Science Society (CogSci)},
address = {Mind, Technology, and Society Pasadena Convention Center},
abstract = {

Copy BibTeX to Clipboard

• https://brill.com/display/book/edcoll/9789004292772/B9789004292772-s010.xml

@incollection{batiukova2015identifying,
title = {Identifying Actional Features Through Semantic Priming: Cross-Romance Comparison},
author = {Olga Batiukova and Pier Marco Bertinetto and Alessandro Lenci and Alessandra Zarcone},
url = {https://brill.com/display/book/edcoll/9789004292772/B9789004292772-s010.xml},
year = {2015},
date = {2015},
booktitle = {Taming the TAME systems. Cahiers Chronos 27},
pages = {161-187},
publisher = {Rodopi},
address = {Amsterdam/Philadelphia},
abstract = {

Copy BibTeX to Clipboard

The automatic induction of scripts (Schank and Abelson, 1977) has been the focus of many recent works. In this paper, we employ a variety of these methods to learn Schank and Abelson’s canonical restaurant script, using a novel dataset of restaurant narratives we have compiled from a website called “Dinners from Hell.” Our models learn narrative chains, script-like structures that we evaluate with the “narrative cloze” task (Chambers and Jurafsky, 2008).

• http://www.aclweb.org/anthology/S15-1024

@article{rudinger2015learning,
title = {Learning to Predict Script Events from Domain-Specific Text},
author = {Rachel Rudinger and Vera Demberg and Ashutosh Modi and Benjamin Van Durme and Manfred Pinkal},
url = {http://www.aclweb.org/anthology/S15-1024},
year = {2015},
date = {2015},
journal = {Lexical and Computational Semantics (* SEM 2015)},
pages = {205-210},
abstract = {The automatic induction of scripts (Schank and Abelson, 1977) has been the focus of many recent works. In this paper, we employ a variety of these methods to learn Schank and Abelson’s canonical restaurant script, using a novel dataset of restaurant narratives we have compiled from a website called “Dinners from Hell.” Our models learn narrative chains, script-like structures that we evaluate with the “narrative cloze” task (Chambers and Jurafsky, 2008).},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

We describe an algorithm for inducing clause-combining rules for use in a traditional natural language generation architecture. An experiment pairing lexicalized text plans from the SPaRKy Restaurant Corpus with logical forms obtained by parsing the corresponding sentences demonstrates that the approach is able to learn clause-combining operations which have essentially the same coverage as those used in the SPaRKy Restaurant Corpus. This paper fills a gap in the literature, showing that it is possible to learn microplanning rules for both aggregation and discourse connective insertion, an important step towards ameliorating the knowledge acquisition bottleneck for NLG systems that produce texts with rich discourse structures using traditional architectures.

• http://www.aclweb.org/anthology/W15-4704

@inproceedings{white:howcroft:enlg-2015,
title = {Inducing Clause-Combining Rules: A Case Study with the SPaRKy Restaurant Corpus},
author = {Michael White and David M. Howcroft},
url = {http://www.aclweb.org/anthology/W15-4704},
year = {2015},
date = {2015},
booktitle = {Proc. of the 15th European Workshop on Natural Language Generation},
publisher = {Association for Computational Linguistics},
address = {Brighton, England, UK},
abstract = {We describe an algorithm for inducing clause-combining rules for use in a traditional natural language generation architecture. An experiment pairing lexicalized text plans from the SPaRKy Restaurant Corpus with logical forms obtained by parsing the corresponding sentences demonstrates that the approach is able to learn clause-combining operations which have essentially the same coverage as those used in the SPaRKy Restaurant Corpus. This paper fills a gap in the literature, showing that it is possible to learn microplanning rules for both aggregation and discourse connective insertion, an important step towards ameliorating the knowledge acquisition bottleneck for NLG systems that produce texts with rich discourse structures using traditional architectures.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

• http://link.springer.com/article/10.1007/s13218-015-0391-y/fulltext.html
• https://doi.org/10.1007/s13218-015-0391-y

@article{crocker:demberg:teich,
title = {Information Density and Linguistic Encoding (IDeaL)},
author = {Matthew W. Crocker and Vera Demberg and Elke Teich},
url = {http://link.springer.com/article/10.1007/s13218-015-0391-y/fulltext.html},
doi = {https://doi.org/10.1007/s13218-015-0391-y},
year = {2015},
date = {2015},
journal = {KI - K{\"u}nstliche Intelligenz},
pages = {77-81},
volume = {30},
number = {1},
abstract = {

Copy BibTeX to Clipboard