Publications

The availability of corpora annotated for discourse relations is limited and discourse relation classification performance varies greatly depending on both language and domain. This is a problem for downstream applications that are intended for a language (i.e., not English) or a domain (i.e., not financial news) with comparatively low coverage for discourse annotations. In this paper, we experiment with a state-of-the-art model for discourse relation classification, originally developed for English, extend it to a multi-lingual setting (testing on Italian, Portuguese and Turkish), and employ a simple, yet effective method to mark out-of-domain training instances. By doing so, we aim to contribute to better generalization and more robust discourse relation classification performance across both language and domain.

• 2024.sigdial-1.47     (0.39MB)
• https://aclanthology.org/2024.sigdial-1.47/
• https://doi.org/10.18653/v1/2024.sigdial-1.47

@inproceedings{bourgonje-demberg-2024-generalizing,
title = {Generalizing across Languages and Domains for Discourse Relation Classification},
author = {Peter Bourgonje and Vera Demberg},
editor = {Tatsuya Kawahara and Vera Demberg and Stefan Ultes and Koji Inoue and Shikib Mehri and David Howcroft and Kazunori Komatani},
url = {https://aclanthology.org/2024.sigdial-1.47/},
doi = {https://doi.org/10.18653/v1/2024.sigdial-1.47},
year = {2024},
date = {2024},
booktitle = {Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
pages = {554-565},
publisher = {Association for Computational Linguistics},
address = {Kyoto, Japan},
abstract = {The availability of corpora annotated for discourse relations is limited and discourse relation classification performance varies greatly depending on both language and domain. This is a problem for downstream applications that are intended for a language (i.e., not English) or a domain (i.e., not financial news) with comparatively low coverage for discourse annotations. In this paper, we experiment with a state-of-the-art model for discourse relation classification, originally developed for English, extend it to a multi-lingual setting (testing on Italian, Portuguese and Turkish), and employ a simple, yet effective method to mark out-of-domain training instances. By doing so, we aim to contribute to better generalization and more robust discourse relation classification performance across both language and domain.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

Connectives like ‘because’ are referred to as ‘processing instructions’ as they facilitate processing of linguistic material directly following the connective. In an expectation-driven account of discourse processing, this can be attributed to predictions that readers make about the upcoming discourse relation, but also to predictions about up-coming discourse content. By modeling these two accounts, termed the relation prediction account and the content prediction account respectively, we show that they make different predictions about when the presence of a connective is most beneficial. In a self-paced reading study, we replicate the facilitative effect of the connective on processing, but do not find any evidence that this effect can be explained by a strong or weak version of either of the two accounts. This suggests that the role of the connective goes above and beyond informing the reader about the upcoming relation and content and possibly triggers a different processing strategy.

• eScholarship UC item 2sc1k7pf     (1.73MB)
• https://escholarship.org/uc/item/2sc1k7pf

@inproceedings{marchal-etal-2024,
title = {What processing instructions do connectives provide? Modeling the facilitative effect of the connective},
author = {Marian Marchal and Merel Scholman and Ted J. M. Sanders and Vera Demberg},
url = {https://escholarship.org/uc/item/2sc1k7pf},
year = {2024},
date = {2024},
booktitle = {Proceedings of the Annual Meeting of the Cognitive Science Society},
pages = {3435-3441},
abstract = {Connectives like ‘because’ are referred to as ‘processing instructions’ as they facilitate processing of linguistic material directly following the connective. In an expectation-driven account of discourse processing, this can be attributed to predictions that readers make about the upcoming discourse relation, but also to predictions about up-coming discourse content. By modeling these two accounts, termed the relation prediction account and the content prediction account respectively, we show that they make different predictions about when the presence of a connective is most beneficial. In a self-paced reading study, we replicate the facilitative effect of the connective on processing, but do not find any evidence that this effect can be explained by a strong or weak version of either of the two accounts. This suggests that the role of the connective goes above and beyond informing the reader about the upcoming relation and content and possibly triggers a different processing strategy.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

We present a pipeline for multi-lingual Shallow Discourse Parsing. The pipeline exploits Machine Translation and Word Alignment, by translating any incoming non-English input text into English, applying an English discourse parser, and projecting the found relations onto the original input text through word alignments. While the purpose of the pipeline is to provide rudimentary discourse relation annotations for low-resource languages, in order to get an idea of performance, we evaluate it on the sub-task of discourse connective identification for several languages for which gold data are available. We experiment with different setups of our modular pipeline architecture and analyze intermediate results. Our code is made available on GitHub.

• 2024.codi-1.4     (0.17MB)
• https://aclanthology.org/2024.codi-1.4

@inproceedings{bourgonje-lin-2024-projecting,
title = {Projecting Annotations for Discourse Relations: Connective Identification for Low-Resource Languages},
author = {Peter Bourgonje and Pin-Jie Lin},
editor = {Michael Strube and Chloe Braud and Christian Hardmeier and Junyi Jessy Li and Sharid Loaiciga and Amir Zeldes and Chuyuan Li},
url = {https://aclanthology.org/2024.codi-1.4},
year = {2024},
date = {2024},
booktitle = {Proceedings of the 5th Workshop on Computational Approaches to Discourse (CODI 2024)},
pages = {39-49},
publisher = {Association for Computational Linguistics},
address = {St. Julians, Malta},
abstract = {We present a pipeline for multi-lingual Shallow Discourse Parsing. The pipeline exploits Machine Translation and Word Alignment, by translating any incoming non-English input text into English, applying an English discourse parser, and projecting the found relations onto the original input text through word alignments. While the purpose of the pipeline is to provide rudimentary discourse relation annotations for low-resource languages, in order to get an idea of performance, we evaluate it on the sub-task of discourse connective identification for several languages for which gold data are available. We experiment with different setups of our modular pipeline architecture and analyze intermediate results. Our code is made available on GitHub.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

Pre-trained large language models, such as ChatGPT, archive outstanding performance in various reasoning tasks without supervised training and were found to have outperformed crowdsourcing workers. Nonetheless, ChatGPT’s performance in the task of implicit discourse relation classification, prompted by a standard multiple-choice question, is still far from satisfactory and considerably inferior to state-of-the-art supervised approaches. This work investigates several proven prompting techniques to improve ChatGPT’s recognition of discourse relations. In particular, we experimented with breaking down the classification task that involves numerous abstract labels into smaller subtasks. Nonetheless, experiment results show that the inference accuracy hardly changes even with sophisticated prompt engineering, suggesting that implicit discourse relation classification is not yet resolvable under zero-shot or few-shot settings.

• 2402.04918     (0.44MB)
• https://arxiv.org/abs/2402.04918

@inproceedings{yung-etal-2024-prompting,
title = {Prompting Implicit Discourse Relation Annotation},
author = {Frances Pik Yu Yung and Mansoor Ahmad and Merel Scholman and Vera Demberg},
url = {https://arxiv.org/abs/2402.04918},
year = {2024},
date = {2024},
booktitle = {Proceedings of Linguistic Annotation Workshop of European Chapter of the Association for Computational Linguistics},
abstract = {Pre-trained large language models, such as ChatGPT, archive outstanding performance in various reasoning tasks without supervised training and were found to have outperformed crowdsourcing workers. Nonetheless, ChatGPT's performance in the task of implicit discourse relation classification, prompted by a standard multiple-choice question, is still far from satisfactory and considerably inferior to state-of-the-art supervised approaches. This work investigates several proven prompting techniques to improve ChatGPT's recognition of discourse relations. In particular, we experimented with breaking down the classification task that involves numerous abstract labels into smaller subtasks. Nonetheless, experiment results show that the inference accuracy hardly changes even with sophisticated prompt engineering, suggesting that implicit discourse relation classification is not yet resolvable under zero-shot or few-shot settings.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

We present DiscoGeM 2.0, a crowdsourced, parallel corpus of 12,834 implicit discourse relations, with English, German, French and Czech data. We propose and validate a new single-step crowdsourcing annotation method and apply it to collect new annotations in German, French and Czech. The corpus was constructed by having crowdsourced annotators choose a suitable discourse connective for each relation from a set of unambiguous candidates. Every instance was annotated by 10 workers. Our corpus hence represents the first multi-lingual resource that contains distributions of discourse interpretations for implicit relations. The results show that the connective insertion method of discourse annotation can be reliably extended to other languages. The resulting multi-lingual annotations also reveal that implicit relations inferred in one language may differ from those inferred in the translation, meaning the annotations are not always directly transferable. DiscoGem 2.0 promotes the investigation of cross-linguistic differences in discourse marking and could improve automatic discourse parsing applications. It is openly downloadable here: https://github.com/merelscholman/DiscoGeM.

• 2024.lrec-main.443     (1.26MB)
• https://aclanthology.org/2024.lrec-main.443

@inproceedings{yung-etal-2024-discogem-2,
title = {DiscoGeM 2.0: A parallel corpus of English, German, French and Czech implicit discourse relations},
author = {Frances Pik Yu Yung and Merel Scholman and Sarka Zikanova and Vera Demberg},
editor = {Nicoletta Calzolari and Min-Yen Kan and Veronique Hoste and Alessandro Lenci and Sakriani Sakti and Nianwen Xue},
url = {https://aclanthology.org/2024.lrec-main.443},
year = {2024},
date = {2024},
booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
pages = {4940-4956},
publisher = {ELRA and ICCL},
address = {Torino, Italia},
abstract = {We present DiscoGeM 2.0, a crowdsourced, parallel corpus of 12,834 implicit discourse relations, with English, German, French and Czech data. We propose and validate a new single-step crowdsourcing annotation method and apply it to collect new annotations in German, French and Czech. The corpus was constructed by having crowdsourced annotators choose a suitable discourse connective for each relation from a set of unambiguous candidates. Every instance was annotated by 10 workers. Our corpus hence represents the first multi-lingual resource that contains distributions of discourse interpretations for implicit relations. The results show that the connective insertion method of discourse annotation can be reliably extended to other languages. The resulting multi-lingual annotations also reveal that implicit relations inferred in one language may differ from those inferred in the translation, meaning the annotations are not always directly transferable. DiscoGem 2.0 promotes the investigation of cross-linguistic differences in discourse marking and could improve automatic discourse parsing applications. It is openly downloadable here: https://github.com/merelscholman/DiscoGeM.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

Nigerian Pidgin is an English-derived contact language and is traditionally an oral language, spoken by approximately 100 million people. No orthographic standard has yet been adopted, and thus the few available Pidgin datasets that exist are characterised by noise in the form of orthographic variations. This contributes to under-performance of models in critical NLP tasks. The current work is the first to describe various types of orthographic variations commonly found in Nigerian Pidgin texts, and model this orthographic variation. The variations identified in the dataset form the basis of a phonetic-theoretic framework for word editing, which is used to generate orthographic variations to augment training data. We test the effect of this data augmentation on two critical NLP tasks: machine translation and sentiment analysis. The proposed variation generation framework augments the training data with new orthographic variants which are relevant for the test set but did not occur in the training set originally. Our results demonstrate the positive effect of augmenting the training data with a combination of real texts from other corpora as well as synthesized orthographic variation, resulting in performance improvements of 2.1 points in sentiment analysis and 1.4 BLEU points in translation to English.

• 2024.lrec-main.1006     (0.35MB)
• https://aclanthology.org/2024.lrec-main.1006

@inproceedings{lin-etal-2024-modeling-orthographic,
title = {Modeling Orthographic Variation Improves NLP Performance for Nigerian Pidgin},
author = {Pin-Jie Lin and Merel Scholman and Muhammed Saeed and Vera Demberg},
editor = {Nicoletta Calzolari and Min-Yen Kan and Veronique Hoste and Alessandro Lenci and Sakriani Sakti and Nianwen Xue},
url = {https://aclanthology.org/2024.lrec-main.1006},
year = {2024},
date = {2024},
booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
pages = {11510-11522},
publisher = {ELRA and ICCL},
address = {Torino, Italia},
abstract = {Nigerian Pidgin is an English-derived contact language and is traditionally an oral language, spoken by approximately 100 million people. No orthographic standard has yet been adopted, and thus the few available Pidgin datasets that exist are characterised by noise in the form of orthographic variations. This contributes to under-performance of models in critical NLP tasks. The current work is the first to describe various types of orthographic variations commonly found in Nigerian Pidgin texts, and model this orthographic variation. The variations identified in the dataset form the basis of a phonetic-theoretic framework for word editing, which is used to generate orthographic variations to augment training data. We test the effect of this data augmentation on two critical NLP tasks: machine translation and sentiment analysis. The proposed variation generation framework augments the training data with new orthographic variants which are relevant for the test set but did not occur in the training set originally. Our results demonstrate the positive effect of augmenting the training data with a combination of real texts from other corpora as well as synthesized orthographic variation, resulting in performance improvements of 2.1 points in sentiment analysis and 1.4 BLEU points in translation to English.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

• https://www.tandfonline.com/doi/full/10.1080/0163853X.2024.2325262
• https://doi.org/10.1080/0163853X.2024.2325262

@article{Scholman_etal_2024,
title = {Connective comprehension in adults: The influence of lexical transparency, frequency, and individual differences},
author = {Merel Scholman and Marian Marchal and Vera Demberg},
url = {https://www.tandfonline.com/doi/full/10.1080/0163853X.2024.2325262},
doi = {https://doi.org/10.1080/0163853X.2024.2325262},
year = {2024},
date = {2024},
journal = {Discourse Processes},
abstract = {

Copy BibTeX to Clipboard

In discourse relation recognition, the classification labels are typically represented as one-hot vectors. However, the categories are in fact not all independent of one another on the contrary, there are several frameworks that describe the labels‘ similarities (by e.g. sorting them into a hierarchy or describing them interms of features (Sanders et al., 2021)). Recently, several methods for representing the similarities between labels have been proposed (Zhang et al., 2018; Wang et al., 2018; Xiong et al., 2021). We here explore and extend the Label Confusion Model (Guo et al., 2021) for learning a representation for discourse relation labels. We explore alternative ways of informing the model about the similarities between relations, by representing relations in terms of their names (and parent category), their typical markers, or in terms of CCR features that describe the relations. Experimental results show that exploiting label similarity improves classification results.

• https://doi.org/10.18653/v1/2023.codi-1.13
• https://doi.org/10.18653/v1/2023.codi-1.13

@inproceedings{varghese-etal-2023-exploiting,
title = {Exploiting Knowledge about Discourse Relations for Implicit Discourse Relation Classification},
author = {Nobel Varghese and Frances Pik Yu Yung and Kaveri Anuranjana and Vera Demberg},
editor = {Michael Strube and Chloe Braud and Christian Hardmeier and Junyi Jessy Li and Sharid Loaiciga and Amir Zeldes},
url = {https://doi.org/10.18653/v1/2023.codi-1.13},
doi = {https://doi.org/10.18653/v1/2023.codi-1.13},
year = {2023},
date = {2023},
booktitle = {Proceedings of the 4th Workshop on Computational Approaches to Discourse (CODI 2023)},
pages = {99-105},
publisher = {Association for Computational Linguistics},
address = {Toronto, Canada},
abstract = {In discourse relation recognition, the classification labels are typically represented as one-hot vectors. However, the categories are in fact not all independent of one another on the contrary, there are several frameworks that describe the labels' similarities (by e.g. sorting them into a hierarchy or describing them interms of features (Sanders et al., 2021)). Recently, several methods for representing the similarities between labels have been proposed (Zhang et al., 2018; Wang et al., 2018; Xiong et al., 2021). We here explore and extend the Label Confusion Model (Guo et al., 2021) for learning a representation for discourse relation labels. We explore alternative ways of informing the model about the similarities between relations, by representing relations in terms of their names (and parent category), their typical markers, or in terms of CCR features that describe the relations. Experimental results show that exploiting label similarity improves classification results.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

Developing effective spoken language processing systems for low-resource languages poses several challenges due to the lack of parallel data and limited resources for fine-tuning models. In this work, we target on improving upon both text classification and translation of Nigerian Pidgin (Naija) by collecting a large-scale parallel English-Pidgin corpus and further propose a framework of cross-lingual adaptive training that includes both continual and task adaptive training so as to adapt a base pre-trained model to low-resource languages. Our studies show that English pre-trained language models serve as a stronger prior than multilingual language models on English-Pidgin tasks with up to 2.38 BLEU improvements; and demonstrate that augmenting orthographic data and using task adaptive training with back-translation can have a significant impact on model performance.

• https://arxiv.org/abs/2307.00382

@inproceedings{lin-et-al-2023,
title = {Low-Resource Cross-Lingual Adaptive Training for Nigerian Pidgin},
author = {Pin-Jie Lin and Muhammed Saeed and Ernie Chang and Merel Scholman},
url = {https://arxiv.org/abs/2307.00382},
year = {2023},
date = {2023},
booktitle = {Proceedings of the 24th INTERSPEECH conference},
abstract = {Developing effective spoken language processing systems for low-resource languages poses several challenges due to the lack of parallel data and limited resources for fine-tuning models. In this work, we target on improving upon both text classification and translation of Nigerian Pidgin (Naija) by collecting a large-scale parallel English-Pidgin corpus and further propose a framework of cross-lingual adaptive training that includes both continual and task adaptive training so as to adapt a base pre-trained model to low-resource languages. Our studies show that English pre-trained language models serve as a stronger prior than multilingual language models on English-Pidgin tasks with up to 2.38 BLEU improvements; and demonstrate that augmenting orthographic data and using task adaptive training with back-translation can have a significant impact on model performance.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

• https://benjamins.com/catalog/slcs.231.06hoe

@inbook{hoek-scholman-2023,
title = {Expressing non-volitional causality in English},
author = {Jet Hoek and Merel Scholman},
editor = {Łukasz Jędrzejowski and Constanze Fleczoreck},
url = {https://benjamins.com/catalog/slcs.231.06hoe},
year = {2023},
date = {2023},
booktitle = {Micro- and Macro-variation of Causal Clauses: Synchronic and Diachronic Insights},
pages = {167–183},
publisher = {John Benjamins Publishing Company},
address = {Amsterdam},
abstract = {

Copy BibTeX to Clipboard

• https://doi.org/10.1037/xlm0001270

@article{marchal-etal-2023,
title = {How Statistical Correlations Influence Discourse-Level Processing: Clause Type as a Cue for Discourse Relations},
author = {Marian Marchal and Merel Scholman and Vera Demberg},
url = {https://doi.org/10.1037/xlm0001270},
year = {2023},
date = {2023},
journal = {Journal of Experimental Psychology: Learning, Memory, and Cognition},
publisher = {Advance online publication},
abstract = {

Copy BibTeX to Clipboard

Discourse relations have different patterns of marking across different languages. As a result, discourse connectives are often added, omitted, or rephrased in translation. Prior work has shown a tendency for explicitation of discourse connectives, but such work was conducted using restricted sample sizes due to difficulty of connective identification and alignment. The current study exploits automatic methods to facilitate a large-scale study of connectives in English and German parallel texts. Our results based on over 300 types and 18000 instances of aligned connectives and an empirical approach to compare the cross-lingual specificity gap provide strong evidence of the Explicitation Hypothesis. We conclude that discourse relations are indeed more explicit in translation than texts written originally in the same language. Automatic annotations allow us to carry out translation studies of discourse relations on a large scale. Our methodology using relative entropy to study the specificity of connectives also provides more fine-grained insights into translation patterns.

• 14_Paper     (0.22MB)
• https://aclanthology.org/2023.sigdial-1.2
• https://doi.org/10.18653/v1/2023.sigdial-1.2

@inproceedings{yung-etal-2023-investigating,
title = {Investigating Explicitation of Discourse Connectives in Translation Using Automatic Annotations},
author = {Frances Pik Yu Yung and Merel Scholman and Ekaterina Lapshinova-Koltunski and Christina Pollkl{\"a}sener and Vera Demberg},
editor = {Svetlana Stoyanchev and Shafiq Joty and David Schlangen and Ondrej Dusek and Casey Kennington and Malihe Alikhani},
url = {https://aclanthology.org/2023.sigdial-1.2},
doi = {https://doi.org/10.18653/v1/2023.sigdial-1.2},
year = {2023},
date = {2023},
booktitle = {Proceedings of the 24th Meeting of Special Interest Group on Discourse and Dialogue (SIGDAIL)},
pages = {21-30},
publisher = {Association for Computational Linguistics},
address = {Prague, Czechia},
abstract = {Discourse relations have different patterns of marking across different languages. As a result, discourse connectives are often added, omitted, or rephrased in translation. Prior work has shown a tendency for explicitation of discourse connectives, but such work was conducted using restricted sample sizes due to difficulty of connective identification and alignment. The current study exploits automatic methods to facilitate a large-scale study of connectives in English and German parallel texts. Our results based on over 300 types and 18000 instances of aligned connectives and an empirical approach to compare the cross-lingual specificity gap provide strong evidence of the Explicitation Hypothesis. We conclude that discourse relations are indeed more explicit in translation than texts written originally in the same language. Automatic annotations allow us to carry out translation studies of discourse relations on a large scale. Our methodology using relative entropy to study the specificity of connectives also provides more fine-grained insights into translation patterns.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

Disagreement in natural language annotation has mostly been studied from a perspective of biases introduced by the annotators and the annotation frameworks. Here, we propose to analyze another source of bias: task design bias, which has a particularly strong impact on crowdsourced linguistic annotations where natural language is used to elicit the interpretation of laymen annotators. For this purpose we look at implicit discourse relation annotation, a task that has repeatedly been shown to be difficult due to the relations‘ ambiguity. We compare the annotations of 1,200 discourse relations obtained using two distinct annotation tasks and quantify the biases of both methods across four different domains. Both methods are natural language annotation tasks designed for crowdsourcing. We show that the task design can push annotators towards certain relations and that some discourse relations senses can be better elicited with one or the other annotation approach. We also conclude that this type of bias should be taken into account when training and testing models.

• 2304.00815     (0.44MB)
• https://arxiv.org/abs/2304.00815

@article{Pyatkinetal.,
title = {Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases introduced by Task Design},
author = {Valentina Pyatkin and Frances Pik Yu Yung and Merel Scholman and Reut Tsarfaty and Ido Dagan and Vera Demberg},
url = {https://arxiv.org/abs/2304.00815},
year = {2023},
date = {2023},
journal = {Transactions of the Association for Computational Linguistics (TACL)},
abstract = {Disagreement in natural language annotation has mostly been studied from a perspective of biases introduced by the annotators and the annotation frameworks. Here, we propose to analyze another source of bias: task design bias, which has a particularly strong impact on crowdsourced linguistic annotations where natural language is used to elicit the interpretation of laymen annotators. For this purpose we look at implicit discourse relation annotation, a task that has repeatedly been shown to be difficult due to the relations' ambiguity. We compare the annotations of 1,200 discourse relations obtained using two distinct annotation tasks and quantify the biases of both methods across four different domains. Both methods are natural language annotation tasks designed for crowdsourcing. We show that the task design can push annotators towards certain relations and that some discourse relations senses can be better elicited with one or the other annotation approach. We also conclude that this type of bias should be taken into account when training and testing models.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

In an eye-tracking-while-reading study, we investigated adult monolinguals’ (N=80) processing of two-clause sentences embedded in short narratives. Three principles theorized to guide comprehension of complex sentences were contrasted: one operating at the clause level, namely clause structure (main clause – subordinate clause or vice versa), and two operating at the discourse-level, namely givenness (given-new vs. new-given) and event order (chronological vs. reverse order). The results indicate that clause structure mainly affects early stages of processing, whereas the two principles operating at the discourse level are more important during later stages and for reading times of the entire sentence. Event order was found to operate relatively independently of the other principles. Givenness was found to overrule clause structure, a phenomenon that can be related to the grounding function of preposed subordinate clauses. We propose a new principle to reflect this interaction effect: the grounding principle.

• https://www.tandfonline.com/doi/full/10.1080/23273798.2022.2077971
• https://doi.org/10.1080/23273798.2022.2077971

@article{Merel_Rules_2022,
title = {Discourse Rules:The Effects of Clause Order Principles on the Reading Process},
author = {Merel Scholman and Liam Blything and Kate Cain and Jacqueline Evers-Vermeul},
url = {https://www.tandfonline.com/doi/full/10.1080/23273798.2022.2077971},
doi = {https://doi.org/10.1080/23273798.2022.2077971},
year = {2022},
date = {2022},
journal = {Language, Cognition and Neuroscience},
pages = {1277-1291},
volume = {37(10)},
abstract = {In an eye-tracking-while-reading study, we investigated adult monolinguals’ (N=80) processing of two-clause sentences embedded in short narratives. Three principles theorized to guide comprehension of complex sentences were contrasted: one operating at the clause level, namely clause structure (main clause - subordinate clause or vice versa), and two operating at the discourse-level, namely givenness (given-new vs. new-given) and event order (chronological vs. reverse order). The results indicate that clause structure mainly affects early stages of processing, whereas the two principles operating at the discourse level are more important during later stages and for reading times of the entire sentence. Event order was found to operate relatively independently of the other principles. Givenness was found to overrule clause structure, a phenomenon that can be related to the grounding function of preposed subordinate clauses. We propose a new principle to reflect this interaction effect: the grounding principle.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

Experimental pragmatics tells us that a metaphor conveys salient features of a vehicle and that highly typical featurestend to be salient. But can highly atypical features also be salient? When asking if John is loyal and hearing “John is afox”, will the hearer conclude that John is disloyal because loyalty is saliently atypical for a fox? This prediction followsfrom our RSA-based model of metaphor understanding which relies on gradient salience. Our behavioral experimentscorroborate the model’s predictions, providing evidence that high and low typicality are salient and result in high in-terpretation confidence and agreement, while average typicality is not salient and makes a metaphor confusing. Ourmodel implements the idea that other features of a vehicle, along with possible alternative vehicles, influence metaphorinterpretation. It produces a significantly better fit compared to an existing RSA model of metaphor understanding,supporting our predictions about the factors at play.

• eScholarship UC item 7kq207zs     (0.07MB)
• https://escholarship.org/uc/item/7kq207zs

@inproceedings{Mayn_2022_of,
title = {Pragmatics of Metaphor Revisited: Modeling the Role of Degree and Salience in Metaphor Understanding},
author = {Alexandra Mayn and Vera Demberg},
url = {https://escholarship.org/uc/item/7kq207zs},
year = {2022},
date = {2022},
booktitle = {Proceedings of the Annual Meeting of the Cognitive Science Society, 43(43)},
pages = {3178ff.},
publisher = {CogSci2022},
abstract = {Experimental pragmatics tells us that a metaphor conveys salient features of a vehicle and that highly typical featurestend to be salient. But can highly atypical features also be salient? When asking if John is loyal and hearing “John is afox”, will the hearer conclude that John is disloyal because loyalty is saliently atypical for a fox? This prediction followsfrom our RSA-based model of metaphor understanding which relies on gradient salience. Our behavioral experimentscorroborate the model’s predictions, providing evidence that high and low typicality are salient and result in high in-terpretation confidence and agreement, while average typicality is not salient and makes a metaphor confusing. Ourmodel implements the idea that other features of a vehicle, along with possible alternative vehicles, influence metaphorinterpretation. It produces a significantly better fit compared to an existing RSA model of metaphor understanding,supporting our predictions about the factors at play.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

Obtaining linguistic annotation from novice crowdworkers is far from trivial. A case in point is the annotation of discourse relations, which is a complicated task. Recent methods have obtained promising results by extracting relation labels from either discourse connectives (DCs) or question-answer (QA) pairs that participants provide. The current contribution studies the effect of worker selection and training on the agreement on implicit relation labels between workers and gold labels, for both the DC and the QA method. In Study 1, workers were not specifically selected or trained, and the results show that there is much room for improvement. Study 2 shows that a combination of selection and training does lead to improved results, but the method is cost- and time-intensive. Study 3 shows that a selection-only approach is a viable alternative; it results in annotations of comparable quality compared to annotations from trained participants. The results generalized over both the DC and QA method and therefore indicate that a selection-only approach could also be effective for other crowdsourced discourse annotation tasks.

• https://aclanthology.org/2022.lrec-1.231/

@inproceedings{ Scholmanet-al22-3,
title = {Design Choices in Crowdsourcing Discourse Relation Annotations: The Effect of Worker Selection and Training},
author = {Merel Scholman and Valentina Pyatkin and Frances Pik Yu Yung and Ido Dagan and Reut Tsarfaty and Vera Demberg},
url = {https://aclanthology.org/2022.lrec-1.231/},
year = {2022},
date = {2022},
booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference, Marseille, France},
pages = {2148–2156},
publisher = {European Language Resources Association},
abstract = {Obtaining linguistic annotation from novice crowdworkers is far from trivial. A case in point is the annotation of discourse relations, which is a complicated task. Recent methods have obtained promising results by extracting relation labels from either discourse connectives (DCs) or question-answer (QA) pairs that participants provide. The current contribution studies the effect of worker selection and training on the agreement on implicit relation labels between workers and gold labels, for both the DC and the QA method. In Study 1, workers were not specifically selected or trained, and the results show that there is much room for improvement. Study 2 shows that a combination of selection and training does lead to improved results, but the method is cost- and time-intensive. Study 3 shows that a selection-only approach is a viable alternative; it results in annotations of comparable quality compared to annotations from trained participants. The results generalized over both the DC and QA method and therefore indicate that a selection-only approach could also be effective for other crowdsourced discourse annotation tasks.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

We present DiscoGeM, a crowdsourced corpus of 6,505 implicit discourse relations from three genres: political speech, literature, and encyclopedic texts. Each instance was annotated by 10 crowd workers. Various label aggregation methods were explored to evaluate how to obtain a label that best captures the meaning inferred by the crowd annotators. The results show that a significant proportion of discourse relations in DiscoGeM are ambiguous and can express multiple relation senses. Probability distribution labels better capture these interpretations than single labels. Further, the results emphasize that text genre crucially affects the distribution of discourse relations, suggesting that genre should be included as a factor in automatic relation classification. We make available the newly created DiscoGeM corpus, as well as the dataset with all annotator-level labels. Both the corpus and the dataset can facilitate a multitude of applications and research purposes, for example to function as training data to improve the performance of automatic discourse relation parsers, as well as facilitate research into non-connective signals of discourse relations.

• https://aclanthology.org/2022.lrec-1.351/

@article{Scholman_et-al22.2,
title = {DiscoGeM: A Crowdsourced Corpus of Genre-Mixed Implicit Discourse Relations},
author = {Merel Scholman and Tianai Dong and Frances Pik Yu Yung and Vera Demberg},
url = {https://aclanthology.org/2022.lrec-1.351/},
year = {2022},
date = {2022},
journal = {Proceedings of the 13th International Conference on Language Resources and Evaluation (LREC 22), Marseille, France},
pages = {3281-3290},
abstract = {We present DiscoGeM, a crowdsourced corpus of 6,505 implicit discourse relations from three genres: political speech, literature, and encyclopedic texts. Each instance was annotated by 10 crowd workers. Various label aggregation methods were explored to evaluate how to obtain a label that best captures the meaning inferred by the crowd annotators. The results show that a significant proportion of discourse relations in DiscoGeM are ambiguous and can express multiple relation senses. Probability distribution labels better capture these interpretations than single labels. Further, the results emphasize that text genre crucially affects the distribution of discourse relations, suggesting that genre should be included as a factor in automatic relation classification. We make available the newly created DiscoGeM corpus, as well as the dataset with all annotator-level labels. Both the corpus and the dataset can facilitate a multitude of applications and research purposes, for example to function as training data to improve the performance of automatic discourse relation parsers, as well as facilitate research into non-connective signals of discourse relations.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

A central issue in linguistics concerns the relationship between theories and evidence in data. We investigate this issue in the field of discourse coherence, and particularly the study of coherence relations such as causal and contrastive. Proposed inventories of coherence relations differ greatly in the type and number of proposed relations. Such proposals are often validated by focusing on either the descriptive adequacy (researcher’s intuitions on textual interpretations) or the cognitive plausibility of distinctions (empirical research on cognition). We argue that both are important, and note that the concept of cognitive plausibility is in need of a concrete definition and quantifiable operationalization. This contribution focuses on how the criterion of cognitive plausibility can be operationalized and presents a systematic validation approach to evaluate discourse frameworks. This is done by detailing how various sources of evidence can be used to support or falsify distinctions between coherence relational labels. Finally, we present methodological issues regarding verification and falsification that are of importance to all discourse researchers studying the relationship between theory and data.

• https://journals.openedition.org/discours/12075

@article{Scholman_etal22,
title = {Descriptively adequate and cognitively plausible? Validating distinctions between types of coherence relations},
author = {Merel Scholman and Vera Demberg and Ted J. M. Sanders},
url = {https://journals.openedition.org/discours/12075},
year = {2022},
date = {2022},
journal = {Discours},
pages = {1-30a},
volume = {30},
abstract = {A central issue in linguistics concerns the relationship between theories and evidence in data. We investigate this issue in the field of discourse coherence, and particularly the study of coherence relations such as causal and contrastive. Proposed inventories of coherence relations differ greatly in the type and number of proposed relations. Such proposals are often validated by focusing on either the descriptive adequacy (researcher’s intuitions on textual interpretations) or the cognitive plausibility of distinctions (empirical research on cognition). We argue that both are important, and note that the concept of cognitive plausibility is in need of a concrete definition and quantifiable operationalization. This contribution focuses on how the criterion of cognitive plausibility can be operationalized and presents a systematic validation approach to evaluate discourse frameworks. This is done by detailing how various sources of evidence can be used to support or falsify distinctions between coherence relational labels. Finally, we present methodological issues regarding verification and falsification that are of importance to all discourse researchers studying the relationship between theory and data.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard

In many linguistic fields requiring annotated data, multiple interpretations of a single item are possible. Multi-label annotations more accurately reflect this possibility. However, allowing for multi-label annotations also affects the chance that two coders agree with each other. Calculating inter-coder agreement for multi-label datasets is therefore not trivial. In the current contribution, we evaluate different metrics for calculating agreement on multi-label annotations: agreement on the intersection of annotated labels, an augmented version of Cohen’s Kappa, and precision, recall and F1. We propose a bootstrapping method to obtain chance agreement for each measure, which allows us to obtain an adjusted agreement coefficient that is more interpretable. We demonstrate how various measures affect estimates of agreement on simulated datasets and present a case study of discourse relation annotations. We also show how the proportion of double labels, and the entropy of the label distribution, influences the measures outlined above and how a bootstrapped adjusted agreement can make agreement measures more comparable across datasets in multi-label scenarios.

• https://aclanthology.org/2022.coling-1.322/

@inproceedings{Marchaletal22-2,
title = {Establishing annotation quality in multi-label annotations},
author = {Marian Marchal and Merel Scholman and Frances Pik Yu Yung and Vera Demberg},
url = {https://aclanthology.org/2022.coling-1.322/},
year = {2022},
date = {2022},
booktitle = {Proceedings of the 29th International Conference on Computational Linguistic (COLING)},
pages = {3659–3668},
abstract = {In many linguistic fields requiring annotated data, multiple interpretations of a single item are possible. Multi-label annotations more accurately reflect this possibility. However, allowing for multi-label annotations also affects the chance that two coders agree with each other. Calculating inter-coder agreement for multi-label datasets is therefore not trivial. In the current contribution, we evaluate different metrics for calculating agreement on multi-label annotations: agreement on the intersection of annotated labels, an augmented version of Cohen’s Kappa, and precision, recall and F1. We propose a bootstrapping method to obtain chance agreement for each measure, which allows us to obtain an adjusted agreement coefficient that is more interpretable. We demonstrate how various measures affect estimates of agreement on simulated datasets and present a case study of discourse relation annotations. We also show how the proportion of double labels, and the entropy of the label distribution, influences the measures outlined above and how a bootstrapped adjusted agreement can make agreement measures more comparable across datasets in multi-label scenarios.},
pubstate = {published},
type = {inproceedings}
}

Copy BibTeX to Clipboard

It is generally assumed that readers draw on their background knowledge to make inferences about information that is left implicit in the text. However, readers may differ in how much background knowledge they have, which may impact their text understanding. The present study investigates the role of domain knowledge in discourse relation interpretation, in order to examine how readers with high vs. low domain knowledge differ in their discourse relation inferences. We compare interpretations of experts from the field of economics and biomedical sciences in scientific biomedical texts as well as more easily accessible economic texts. The results show that high-knowledge readers from the biomedical domain are better at inferring the correct relation interpretation in biomedical texts compared to low-knowledge readers, but such an effect was not found for the economic domain. The results also suggest that, in the absence of domain knowledge, readers exploit linguistic signals other than connectives to infer the discourse relation, but domain knowledge is sometimes required to exploit these cues. The study provides insight into the impact of domain knowledge on discourse relation inferencing and how readers interpret discourse relations when they lack the required domain knowledge.

• DD_MarchalScholmanDemberg_2109     (0.25MB)
• https://journals.uic.edu/ojs/index.php/dad/article/view/12343/10711

@article{Marchaletal22,
title = {The effect of domain knowledge on discourse relation inferences: Relation marking and interpretation strategies},
author = {Marian Marchal and Merel Scholman and Vera Demberg},
url = {https://journals.uic.edu/ojs/index.php/dad/article/view/12343/10711},
year = {2022},
date = {2022},
journal = {Dialogue & Discourse},
pages = {49-78},
volume = {13},
number = {(2)},
abstract = {It is generally assumed that readers draw on their background knowledge to make inferences about information that is left implicit in the text. However, readers may differ in how much background knowledge they have, which may impact their text understanding. The present study investigates the role of domain knowledge in discourse relation interpretation, in order to examine how readers with high vs. low domain knowledge differ in their discourse relation inferences. We compare interpretations of experts from the field of economics and biomedical sciences in scientific biomedical texts as well as more easily accessible economic texts. The results show that high-knowledge readers from the biomedical domain are better at inferring the correct relation interpretation in biomedical texts compared to low-knowledge readers, but such an effect was not found for the economic domain. The results also suggest that, in the absence of domain knowledge, readers exploit linguistic signals other than connectives to infer the discourse relation, but domain knowledge is sometimes required to exploit these cues. The study provides insight into the impact of domain knowledge on discourse relation inferencing and how readers interpret discourse relations when they lack the required domain knowledge.},
pubstate = {published},
type = {article}
}

Copy BibTeX to Clipboard