ChartQA

Dataset Information
Introduced
2022
License
Unknown
Homepage

Overview

Charts are very popular for analyzing data. When exploring charts, people often ask a variety of complex reasoning questions that involve several logical and arithmetic operations. They also commonly refer to visual features of a chart in their questions. However, most existing datasets do not focus on such complex reasoning questions as their questions are template-based and answers come from a fixed-vocabulary. In this work, we present a large-scale benchmark covering 9.6K human-written questions as well as 23.1K questions generated from human-written chart summaries. To address the unique challenges in our benchmark involving visual and logical reasoning over charts, we present two transformer-based models that combine visual features and the data table of the chart in a unified way to answer questions. While our models achieve the state-of-the-art results on the previous datasets as well as on our benchmark, the evaluation also reveals several challenges in answering complex reasoning questions.

Variants: ChartQA

Associated Benchmarks

This dataset is used in 1 benchmark:

Recent Benchmark Submissions

Task Model Paper Date
Chart Question Answering ChartPaLI-5B Chart-based Reasoning: Transferring Capabilities from … 2024-03-19
Chart Question Answering ChartPaLI-5B + PaLM 2-S Chart-based Reasoning: Transferring Capabilities from … 2024-03-19
Chart Question Answering ScreenAI 5B (4.62 B params, w/ OCR) ScreenAI: A Vision-Language Model for … 2024-02-07
Chart Question Answering MatCha4096 + LaMenDa Synthesize Step-by-Step: Tools, Templates and … 2024-01-01
Chart Question Answering Gemini Ultra Gemini: A Family of Highly … 2023-12-19
Chart Question Answering SMoLA-PaLI-X Specialist Model Omni-SMoLA: Boosting Generalist Multimodal Models … 2023-12-01
Chart Question Answering SMoLA-PaLI-X Generalist Model Omni-SMoLA: Boosting Generalist Multimodal Models … 2023-12-01
Chart Question Answering PaLI-3 PaLI-3 Vision Language Models: Smaller, … 2023-10-13
Chart Question Answering PaLI-3 (w/ OCR) PaLI-3 Vision Language Models: Smaller, … 2023-10-13
Chart Question Answering StructChart+GPT3.5 (STR ChartQA+SimChart9K) StructChart: On the Schema, Metric, … 2023-09-20
Chart Question Answering StructChart+GPT3.5 (STR) StructChart: On the Schema, Metric, … 2023-09-20
Chart Question Answering Qwen-VL Qwen-VL: A Versatile Vision-Language Model … 2023-08-24
Chart Question Answering Qwen-VL-Chat Qwen-VL: A Versatile Vision-Language Model … 2023-08-24
Chart Question Answering PaLI-X (Single-task FT w/ OCR) PaLI-X: On Scaling up a … 2023-05-29
Chart Question Answering PaLI-X (Single-task FT) PaLI-X: On Scaling up a … 2023-05-29
Chart Question Answering PaLI-X (Multi-task FT) PaLI-X: On Scaling up a … 2023-05-29
Chart Question Answering UniChart UniChart: A Universal Vision-language Pretrained … 2023-05-24
Chart Question Answering DePlot+GPT3 (CoT) DePlot: One-shot visual language reasoning … 2022-12-20
Chart Question Answering DePlot+FlanPaLM+Codex (PoT Self-Consistency) DePlot: One-shot visual language reasoning … 2022-12-20
Chart Question Answering DePlot+Codex (PoT Self-Consistency) DePlot: One-shot visual language reasoning … 2022-12-20

Research Papers

Recent papers with results on this dataset: