RWSD

A Winograd schema is a pair of sentences that differ in only one or two words and that contain an ambiguity that is resolved in opposite ways in the two sentences and requires the use of world knowledge and reasoning for its resolution. The schema takes its name from a well-known example by Terry Winograd.

The set would then be presented as a challenge for AI programs, along the lines of the Turing test. The strengths of the challenge are that it is clear-cut, in that the answer to each schema is a binary choice; vivid, in that it is obvious to non-experts that a program that fails to get the right answers clearly has serious gaps in its understanding; and difficult, in that it is far beyond the current state of the art.

Task Type

Logic and Reasoning, World knowledge. Binary Classification: true/false

Example

{
  "text": "Кубок не помещается в коричневый чемодан, потому что он слишком большой."
  "label": false,
  "idx": 5,
  "target": {
    "span1_text": "чемодан",
    "span2_text": "он слишком большой",
    "span1_index": 5,
    "span2_index": 8
  },
}

How did we collect data?

All text examples were collected manually translating and adapting original Winograd dataset for Russian. Human assessment was carried out on Yandex.Toloka.