Multi-Sentence Reading Comprehension
MultiRC (Multi-Sentence Reading Comprehension) is a dataset of short paragraphs and multi-sentence questions, i.e., questions that can be answered by combining information from multiple sentences of the paragraph.
The dataset was designed with three key challenges in mind:
* The number of correct answer-options for each question is not pre-specified. This removes the over-reliance on answer-options and forces them to decide on the correctness of each candidate answer independently of others. In other words, the task is not to simply identify the best answer-option, but to evaluate the correctness of each answer-option individually.
* The correct answer(s) is not required to be a span in the text.
* The paragraphs in the dataset have diverse provenance by being extracted from 7 different domains such as news, fiction, historical text etc., and hence are expected to be more diverse in their contents as compared to single-domain datasets.
The entire corpus consists of around 10K questions (including about 6K multiple-sentence questions). The 60% of the data is released as training and development data. The rest of the data is saved for evaluation and every few months a new unseen additional data is included for evaluation to prevent unintentional overfitting over time.
Source: https://cogcomp.seas.upenn.edu/multirc/
Image Source: https://paperswithcode.com/paper/looking-beyond-the-surface-a-challenge-set/
Variants: MultiRC
This dataset is used in 1 benchmark:
Recent papers with results on this dataset: