BIG-bench

Overview edit

Variants: BBH-nlp, BBH-alg, Big-bench Hard, BIG-bench (Logical Sequence), BIG-bench (Logical Fallacy Detection), BIG-bench (Known Unknowns), BIG-bench (Hindu Knowledge), BIG-bench (Novel Concepts), BIG-bench (StrategyQA), BIG-bench (Winowhy), BIG-bench (Logic Grid Puzzle), BIG-bench (Anachronisms), BIG-bench (Temporal Sequences), BIG-bench (Sports Understanding), BIG-bench (SNARKS), BIG-bench (Ruin Names), BIG-bench (Reasoning About Colored Objects), BIG-bench (Penguins In A Table), BIG-bench (Navigate), BIG-bench (Movie Recommendation), BIG-bench (Hyperbaton), BIG-bench (Formal Fallacies Syllogisms Negation), BIG-bench (Disambiguation QA), BIG-bench (Date Understanding), BIG-bench (Causal Judgment), BIG-bench-lite, Big-bench Lite, BIG-bench

Associated Benchmarks

This dataset is used in 41 benchmarks:

• Analogical Similarity - Metrics: Accuracy
• Identify Odd Metapor - Metrics: Accuracy
• Odd One Out - Metrics: Accuracy
• General Knowledge - Metrics: Accuracy
• High School European History - Metrics: Accuracy
• High School US History - Metrics: Accuracy
• High School World History - Metrics: Accuracy
• International Law - Metrics: Accuracy
• Jurisprudence - Metrics: Accuracy
• Logical Fallacies - Metrics: Accuracy
• Management - Metrics: Accuracy
• Marketing - Metrics: Accuracy
• Philosophy - Metrics: Accuracy
• Prehistory - Metrics: Accuracy
• Professional Law - Metrics: Accuracy
• World Religions - Metrics: Accuracy
• Anatomy - Metrics: Accuracy
• Clinical Knowledge - Metrics: Accuracy
• College Medicine - Metrics: Accuracy
• Human Aging - Metrics: Accuracy
• Human Organs Senses Multiple Choice - Metrics: Accuracy
• Medical Genetics - Metrics: Accuracy
• Nutrition - Metrics: Accuracy
• Professional Medicine - Metrics: Accuracy
• Virology - Metrics: Accuracy
• Econometrics - Metrics: Accuracy
• High School Geography - Metrics: Accuracy
• High School Government and Politics - Metrics: Accuracy
• High School Macroeconomics - Metrics: Accuracy
• High School Microeconomics - Metrics: Accuracy
• High School Psychology - Metrics: Accuracy
• Human Sexuality - Metrics: Accuracy
• Professional Psychology - Metrics: Accuracy
• Public Relations - Metrics: Accuracy
• Security Studies - Metrics: Accuracy
• Sociology - Metrics: Accuracy
• US Foreign Policy - Metrics: Accuracy
• Intent Recognition - Metrics: Accuracy
• BIG-bench Machine Learning - Metrics: Accuracy
• Astronomy - Metrics: Accuracy
• Computer Security - Metrics: Accuracy

Recent Benchmark Submissions

Research Papers

Recent papers with results on this dataset:

• Training Compute-Optimal Large Language Models (2022) -
• Scaling Language Models: Methods, Analysis & Insights from Training Gopher (2021) -