CrowS-Pairs has 1508 examples that cover stereotypes dealing with nine types of bias, like race, religion, and age. In CrowS-Pairs a model is presented with two sentences: one that is more stereotyping and another that is less stereotyping. The data focuses on stereotypes about historically disadvantaged groups and contrasts them with advantaged groups.
Source: CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
Variants: CrowS-Pairs
This dataset is used in 1 benchmark:
| Task | Model | Paper | Date |
|---|---|---|---|
| Stereotypical Bias Analysis | LLaMA 65B | LLaMA: Open and Efficient Foundation … | 2023-02-27 |
| Stereotypical Bias Analysis | GAL 120B | Galactica: A Large Language Model … | 2022-11-16 |
| Stereotypical Bias Analysis | GPT-3 | OPT: Open Pre-trained Transformer Language … | 2022-05-02 |
| Stereotypical Bias Analysis | OPT-175B | OPT: Open Pre-trained Transformer Language … | 2022-05-02 |
Recent papers with results on this dataset: