GRIT

General Robust Image Task Benchmark

Dataset Information
Modalities
Images, Texts
Languages
English
Introduced
2022
License
Apache License 2.0
Homepage
Official Website
Contents

Overview

The General Robust Image Task (GRIT) Benchmark is an evaluation-only benchmark for evaluating the performance and robustness of vision systems across multiple image prediction tasks, concepts, and data sources. GRIT hopes to encourage our research community to pursue the following research directions:

  1. General purpose vision models - GRIT facilitates the evaluation of unified and general-purpose vision models that demonstrate a wide range of skills across a diverse set of concepts.
  2. Robust specialized models - GRIT simplifies and unifies quantification of misinformation, calibration, and generalization under distribution shifts due to novel concepts, novel data sources or image distortions for 7 standard vision and vision-language tasks.
  3. Efficient learning - GRIT includes a restricted and an unrestricted track. The restrictedtrack constrains the allowed training data to a selected but rich set of data sources that allows more scientific and meaningful comparison between models. This is meant to encourage resource constrained researchers to participate in the GRIT challenge and to spur interest in efficient learning methods as opposed to the dominant paradigm of training larger models on ever increasing amounts of training data. The unrestricted track allows much more flexibility in training data selection to test the capability of vision models trained with massive data and compute.

Variants: GRIT

Associated Benchmarks

This dataset is used in 5 benchmarks:

Recent Benchmark Submissions

Task Model Paper Date
Object Segmentation Unified-IOXL Unified-IO: A Unified Model for … 2022-06-17
Visual Question Answering (VQA) Unified-IOXL Unified-IO: A Unified Model for … 2022-06-17
Object Localization Unified-IOXL Unified-IO: A Unified Model for … 2022-06-17
Object Categorization Unified-IOXL Unified-IO: A Unified Model for … 2022-06-17
Visual Question Answering OFA OFA: Unifying Architectures, Tasks, and … 2022-02-07
Object Categorization OFA_Large OFA: Unifying Architectures, Tasks, and … 2022-02-07
Object Localization GPV-2 Webly Supervised Concept Expansion for … 2022-02-04
Object Categorization GPV-2 Webly Supervised Concept Expansion for … 2022-02-04
Visual Question Answering (VQA) GPV-2 Webly Supervised Concept Expansion for … 2022-02-04
Object Categorization CLIP Learning Transferable Visual Models From … 2021-02-26
Object Segmentation Mask R-CNN Mask R-CNN 2017-03-20
Object Localization Mask R-CNN Mask R-CNN 2017-03-20

Research Papers

Recent papers with results on this dataset:

External Links: