ML Research Wiki / Benchmarks / Question Answering / PubMedQA

PubMedQA

Question Answering Benchmark

Performance Over Time

📊 Showing 26 results | 📏 Metric: Accuracy

Top Performing Models

Rank	Model	Paper	Accuracy	Date	Code
1	Meditron-70B (CoT + SC)	MEDITRON-70B: Scaling Medical Pretraining for Large Language Models	81.60	2023-11-27	📦 epfllm/meditron
2	BioGPT-Large(1.5B)	BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining	81.00	2022-10-19	📦 huggingface/transformers 📦 microsoft/biogpt 📦 2024-MindSpore-1/Code2 📦 TaoQin/taoqin.github.io
3	RankRAG-llama3-70B (Zero-Shot)	RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs	79.80	2024-07-02	-
4	Med-PaLM 2 (5-shot)	Towards Expert-Level Medical Question Answering with Large Language Models	79.20	2023-05-16	📦 m42-health/med42
5	Flan-PaLM (540B, Few-shot)	Large Language Models Encode Clinical Knowledge	79.00	2022-12-26	📦 dmis-lab/olaph
6	BioGPT(345M)	BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining	78.20	2022-10-19	📦 huggingface/transformers 📦 microsoft/biogpt 📦 2024-MindSpore-1/Code2 📦 TaoQin/taoqin.github.io
7	Codex 5-shot CoT	Can large language models reason about medical questions?	78.20	2022-07-17	📦 vlievin/medical-reasoning
8	Human Performance (single annotator)	PubMedQA: A Dataset for Biomedical Research Question Answering	78.00	2019-09-13	📦 open-dataflow/rare 📦 okanvk/Medical-Specific-Electra-Med-Electra- 📦 okanvk/Medical-Electra 📦 okanvk/Question-Answering-Project 📦 8023looker/med-rr
9	MetaGen Blended RAG (zero-shot)	MetaGen Blended RAG: Higher Accuracy for Domain-Specific Q&A Without Fine-Tuning	77.90	2025-05-23	📦 ibm-self-serve-assets/metagen-blended-rag
10	GAL 120B (zero-shot)	Galactica: A Large Language Model for Science	77.60	2022-11-16	📦 paperswithcode/galai

All Papers (26)

MEDITRON-70B: Scaling Medical Pretraining for Large Language Models

2023

Meditron-70B (CoT + SC)

epfllm/meditron

BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining

2022

BioGPT-Large(1.5B)

huggingface/transformers microsoft/biogpt

RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs

2024

RankRAG-llama3-70B (Zero-Shot)

Towards Expert-Level Medical Question Answering with Large Language Models

2023

Med-PaLM 2 (5-shot)

m42-health/med42

Large Language Models Encode Clinical Knowledge

2022

Flan-PaLM (540B, Few-shot)

dmis-lab/olaph

BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining

2022

BioGPT(345M)

huggingface/transformers microsoft/biogpt

Can large language models reason about medical questions?

2022

Codex 5-shot CoT

vlievin/medical-reasoning

PubMedQA: A Dataset for Biomedical Research Question Answering

2019

Human Performance (single annotator)

open-dataflow/rare okanvk/Medical-Specific-Electra-Med-Electra-

MetaGen Blended RAG: Higher Accuracy for Domain-Specific Q&A Without Fine-Tuning

2025

MetaGen Blended RAG (zero-shot)

ibm-self-serve-assets/metagen-blended-rag

Galactica: A Large Language Model for Science

2022

GAL 120B (zero-shot)

paperswithcode/galai

Large Language Models Encode Clinical Knowledge

2022

Flan-PaLM (62B, Few-shot)

dmis-lab/olaph

MediSwift: Efficient Sparse Pre-trained Biomedical Language Models

2024

MediSwift-XL

BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicine

2023

BioMedGPT-10B

pharmolix/openbiomed

Large Language Models Encode Clinical Knowledge

2022

Flan-PaLM (540B, SC)

dmis-lab/olaph

Towards Expert-Level Medical Question Answering with Large Language Models

2023

Med-PaLM 2 (ER)

m42-health/med42

Towards Expert-Level Medical Question Answering with Large Language Models

2023

Med-PaLM 2 (CoT + SC)

m42-health/med42

Galactica: A Large Language Model for Science

2022

BLOOM (zero-shot)

paperswithcode/galai

The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning

2023

CoT-T5-11B (1024 Shot)

kaistai/cot-collection kaist-lklab/cot-collection

LinkBERT: Pretraining Language Models with Document Links

2022

BioLinkBERT (large)

michiyasunaga/LinkBERT

LinkBERT: Pretraining Language Models with Document Links

2022

BioLinkBERT (base)

michiyasunaga/LinkBERT

Galactica: A Large Language Model for Science

2022

OPT (zero-shot)

paperswithcode/galai

Large Language Models Encode Clinical Knowledge

2022

Flan-PaLM (8B, Few-shot)

dmis-lab/olaph

Large Language Models Encode Clinical Knowledge

2022

PaLM (62B, Few-shot)

dmis-lab/olaph

Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

2020

PubMedBERT uncased

bionlu-coling2024/biomed-ner-intent_detection rohanshad/cmr_transformer

Large Language Models Encode Clinical Knowledge

2022

PaLM (540B, Few-shot)

dmis-lab/olaph

Large Language Models Encode Clinical Knowledge

2022

PaLM (8B, Few-shot)

dmis-lab/olaph

PubMedQA

Performance Over Time

Edit Benchmark Results

Edit Result

Top Performing Models

All Papers (26)

MEDITRON-70B: Scaling Medical Pretraining for Large Language Models

BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining

RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs

Towards Expert-Level Medical Question Answering with Large Language Models

Large Language Models Encode Clinical Knowledge

BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining

Can large language models reason about medical questions?

PubMedQA: A Dataset for Biomedical Research Question Answering

MetaGen Blended RAG: Higher Accuracy for Domain-Specific Q&A Without Fine-Tuning

Galactica: A Large Language Model for Science

Large Language Models Encode Clinical Knowledge

MediSwift: Efficient Sparse Pre-trained Biomedical Language Models

BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicine

Large Language Models Encode Clinical Knowledge

Towards Expert-Level Medical Question Answering with Large Language Models

Towards Expert-Level Medical Question Answering with Large Language Models

Galactica: A Large Language Model for Science

The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning

LinkBERT: Pretraining Language Models with Document Links

LinkBERT: Pretraining Language Models with Document Links

Galactica: A Large Language Model for Science

Large Language Models Encode Clinical Knowledge

Large Language Models Encode Clinical Knowledge

Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

Large Language Models Encode Clinical Knowledge

Large Language Models Encode Clinical Knowledge

Model	Paper	Accuracy	Date
Meditron-70B (CoT + SC)	MEDITRON-70B: Scaling Medical Pretraining for Lar…	81.60	2023-11-27
BioGPT-Large(1.5B)	BioGPT: Generative Pre-trained Transformer for Bi…	81.00	2022-10-19
RankRAG-llama3-70B (Zero-Shot)	RankRAG: Unifying Context Ranking with Retrieval-…	79.80	2024-07-02
Med-PaLM 2 (5-shot)	Towards Expert-Level Medical Question Answering w…	79.20	2023-05-16
Flan-PaLM (540B, Few-shot)	Large Language Models Encode Clinical Knowledge	79.00	2022-12-26
BioGPT(345M)	BioGPT: Generative Pre-trained Transformer for Bi…	78.20	2022-10-19
Codex 5-shot CoT	Can large language models reason about medical qu…	78.20	2022-07-17
Human Performance (single annotator)	PubMedQA: A Dataset for Biomedical Research Quest…	78.00	2019-09-13
MetaGen Blended RAG (zero-shot)	MetaGen Blended RAG: Higher Accuracy for Domain-S…	77.90	2025-05-23
GAL 120B (zero-shot)	Galactica: A Large Language Model for Science	77.60	2022-11-16
Flan-PaLM (62B, Few-shot)	Large Language Models Encode Clinical Knowledge	77.20	2022-12-26
MediSwift-XL	MediSwift: Efficient Sparse Pre-trained Biomedica…	76.80	2024-03-01
BioMedGPT-10B	BioMedGPT: Open Multimodal Generative Pre-trained…	76.10	2023-08-18
Flan-PaLM (540B, SC)	Large Language Models Encode Clinical Knowledge	75.20	2022-12-26
Med-PaLM 2 (ER)	Towards Expert-Level Medical Question Answering w…	75.00	2023-05-16
Med-PaLM 2 (CoT + SC)	Towards Expert-Level Medical Question Answering w…	74.00	2023-05-16
BLOOM (zero-shot)	Galactica: A Large Language Model for Science	73.60	2022-11-16
CoT-T5-11B (1024 Shot)	The CoT Collection: Improving Zero-shot and Few-s…	73.42	2023-05-23
BioLinkBERT (large)	LinkBERT: Pretraining Language Models with Docume…	72.20	2022-03-29
BioLinkBERT (base)	LinkBERT: Pretraining Language Models with Docume…	70.20	2022-03-29
OPT (zero-shot)	Galactica: A Large Language Model for Science	70.20	2022-11-16
Flan-PaLM (8B, Few-shot)	Large Language Models Encode Clinical Knowledge	67.60	2022-12-26
PaLM (62B, Few-shot)	Large Language Models Encode Clinical Knowledge	57.80	2022-12-26
PubMedBERT uncased	Domain-Specific Language Model Pretraining for Bi…	55.84	2020-07-31
PaLM (540B, Few-shot)	Large Language Models Encode Clinical Knowledge	55.00	2022-12-26
PaLM (8B, Few-shot)	Large Language Models Encode Clinical Knowledge	34.00	2022-12-26