Conversational Question Answering Challenge
CoQA is a large-scale dataset for building Conversational Question Answering systems. The goal of the CoQA challenge is to measure the ability of machines to understand a text passage and answer a series of interconnected questions that appear in a conversation.
CoQA contains 127,000+ questions with answers collected from 8000+ conversations. Each conversation is collected by pairing two crowdworkers to chat about a passage in the form of questions and answers. The unique features of CoQA include 1) the questions are conversational; 2) the answers can be free-form text; 3) each answer also comes with an evidence subsequence highlighted in the passage; and 4) the passages are collected from seven diverse domains. CoQA has a lot of challenging phenomena not present in existing reading comprehension datasets, e.g., coreference and pragmatic reasoning.
Source: https://stanfordnlp.github.io/coqa/
Image Source: https://stanfordnlp.github.io/coqa/
Variants: CoQA
This dataset is used in 1 benchmark:
| Task | Model | Paper | Date |
|---|---|---|---|
| Question Answering | GPT-3 175B (few-shot, k=32) | Language Models are Few-Shot Learners | 2020-05-28 |
| Question Answering | SDNet (ensemble) | SDNet: Contextualized Attention-based Deep Network … | 2018-12-10 |
| Question Answering | SDNet (single model) | SDNet: Contextualized Attention-based Deep Network … | 2018-12-10 |
| Question Answering | BERT Large Augmented (single model) | BERT: Pre-training of Deep Bidirectional … | 2018-10-11 |
| Question Answering | BERT-base finetune (single model) | BERT: Pre-training of Deep Bidirectional … | 2018-10-11 |
| Question Answering | FlowQA (single model) | FlowQA: Grasping Flow in History … | 2018-10-06 |
| Question Answering | BiDAF++ (single model) | A Qualitative Comparison of CoQA, … | 2018-09-27 |
| Question Answering | Vanilla DrQA (single model) | CoQA: A Conversational Question Answering … | 2018-08-21 |
| Question Answering | DrQA + seq2seq with copy attention (single model) | CoQA: A Conversational Question Answering … | 2018-08-21 |
Recent papers with results on this dataset: