Roth, Michael; Thater, Stefan; Ostermann, Simon; Modi, Ashutosh; Pinkal, Manfred
SemEval-2018 Task 11: Machine Comprehension using Commonsense Knowledge
Proceedings of the 12th International Workshop on Semantic Evaluation, Association for Computational Linguistics, pp. 747-757, New Orleans, Louisiana, 2018.
This report summarizes the results of the SemEval 2018 task on machine comprehension using commonsense knowledge. For this machine comprehension task, we created a new corpus, MCScript. It contains a high number of questions that require commonsense knowledge for finding the correct answer. 11 teams from 4 different countries participated in this shared task, most of them used neural approaches. The best performing system achieves an accuracy of 83.95%, outperforming the baselines by a large margin, but still far from the human upper bound, which was found to be at 98%.