ML Research Wiki / Benchmarks / Meme Classification / Hateful Memes

Hateful Memes

Meme Classification Benchmark

Performance Over Time

📊 Showing 17 results | 📏 Metric: ROC-AUC

Top Performing Models

Rank	Model	Paper	ROC-AUC	Date	Code
1	PaLI-X-VPD	Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models	0.89	2023-12-05	-
2	Flamingo (fine-tuned)	Flamingo: a Visual Language Model for Few-Shot Learning	0.87	2022-04-29	📦 mlfoundations/open_flamingo 📦 lucidrains/flamingo-pytorch 📦 unispac/visual-adversarial-examples-jailbreak-large-language-models 📦 doc-doc/NExT-OE 📦 happen2me/cross-gnn
3	Hate-CLIPper - Align	Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Features	0.86	2022-10-12	📦 gokulkarthik/hateclipper
4	ISSUES	Mapping Memes to Words for Multimodal Hateful Meme Classification	0.86	2023-10-12	📦 miccunifi/issues
5	Human	The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes	0.85	2020-05-10	📦 facebookresearch/mmf 📦 rizavelioglu/hateful_memes-hate_detectron 📦 holman57/Hateful-Memes 📦 SebKleiner/Hateful_Memes
6	RA-HMD (Qwen2-VL-7B)	Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection	0.82	2025-02-18	📦 JingbiaoMei/RGCL
7	RA-HMD (LLaVA-1.5-7B)	Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection	0.81	2025-02-18	📦 JingbiaoMei/RGCL
8	RA-HMD (Qwen2-VL-2B)	Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection	0.79	2025-02-18	📦 JingbiaoMei/RGCL
9	RGCL (CLIP)	Improving Hateful Meme Detection through Retrieval-Guided Contrastive Learning	0.79	2023-11-14	📦 JingbiaoMei/RGCL
10	HateDetectron27 📚	Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge	0.77	2020-12-23	📦 rizavelioglu/hateful_memes-hate_detectron

All Papers (17)

Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models

2023

PaLI-X-VPD

Flamingo: a Visual Language Model for Few-Shot Learning

2022

Flamingo (fine-tuned)

mlfoundations/open_flamingo lucidrains/flamingo-pytorch

Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Features

2022

Hate-CLIPper - Align

gokulkarthik/hateclipper

Mapping Memes to Words for Multimodal Hateful Meme Classification

2023

ISSUES

miccunifi/issues

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes

2020

Human

facebookresearch/mmf rizavelioglu/hateful_memes-hate_detectron

Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection

2025

RA-HMD (Qwen2-VL-7B)

JingbiaoMei/RGCL

Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection

2025

RA-HMD (LLaVA-1.5-7B)

JingbiaoMei/RGCL

Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection

2025

RA-HMD (Qwen2-VL-2B)

JingbiaoMei/RGCL

Improving Hateful Meme Detection through Retrieval-Guided Contrastive Learning

2023

RGCL (CLIP)

JingbiaoMei/RGCL

Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge

2020

HateDetectron27

rizavelioglu/hateful_memes-hate_detectron

Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervision

2022

SEER (RegNet10B)

facebookresearch/vissl

Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solution

2020

Ron Zhu

himario/hatefulmemeschallenge

Pro-Cap: Leveraging a Frozen Vision-Language Model for Hateful Meme Detection

2023

Pro-Cap

social-ai-studio/pro-cap abril4416/kgen_vqa

Flamingo: a Visual Language Model for Few-Shot Learning

2022

Flamingo (few-shot:32)

mlfoundations/open_flamingo lucidrains/flamingo-pytorch

Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memes

2020

Vilio

Muennighoff/vilio

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes

2020

Visual BERT COCO

facebookresearch/mmf rizavelioglu/hateful_memes-hate_detectron

Learning Transferable Visual Models From Natural Language Supervision

2021

CLIP (zero-shot)

openai/CLIP mlfoundations/open_clip

Hateful Memes

Performance Over Time

Edit Benchmark Results

Edit Result

Top Performing Models

All Papers (17)

Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models

Flamingo: a Visual Language Model for Few-Shot Learning

Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Features

Mapping Memes to Words for Multimodal Hateful Meme Classification

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes

Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection

Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection

Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection

Improving Hateful Meme Detection through Retrieval-Guided Contrastive Learning

Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge

Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervision

Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solution

Pro-Cap: Leveraging a Frozen Vision-Language Model for Hateful Meme Detection

Flamingo: a Visual Language Model for Few-Shot Learning

Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memes

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes

Learning Transferable Visual Models From Natural Language Supervision

Model	Paper	ROC-AUC	Date
PaLI-X-VPD	Visual Program Distillation: Distilling Tools and…	0.89	2023-12-05
Flamingo (fine-tuned)	Flamingo: a Visual Language Model for Few-Shot Le…	0.87	2022-04-29
Hate-CLIPper - Align	Hate-CLIPper: Multimodal Hateful Meme Classificat…	0.86	2022-10-12
ISSUES	Mapping Memes to Words for Multimodal Hateful Mem…	0.86	2023-10-12
Human	The Hateful Memes Challenge: Detecting Hate Speec…	0.85	2020-05-10
RA-HMD (Qwen2-VL-7B)	Robust Adaptation of Large Multimodal Models for …	0.82	2025-02-18
RA-HMD (LLaVA-1.5-7B)	Robust Adaptation of Large Multimodal Models for …	0.81	2025-02-18
RA-HMD (Qwen2-VL-2B)	Robust Adaptation of Large Multimodal Models for …	0.79	2025-02-18
RGCL (CLIP)	Improving Hateful Meme Detection through Retrieva…	0.79	2023-11-14
HateDetectron27	Detecting Hate Speech in Memes Using Multimodal D…	0.77	2020-12-23
SEER (RegNet10B)	Vision Models Are More Robust And Fair When Pretr…	0.73	2022-02-16
Ron Zhu	Enhance Multimodal Transformer With External Labe…	0.73	2020-12-15
Pro-Cap	Pro-Cap: Leveraging a Frozen Vision-Language Mode…	0.72	2023-08-16
Flamingo (few-shot:32)	Flamingo: a Visual Language Model for Few-Shot Le…	0.70	2022-04-29
Vilio	Vilio: State-of-the-art Visio-Linguistic Models a…	0.70	2020-12-14
Visual BERT COCO	The Hateful Memes Challenge: Detecting Hate Speec…	0.70	2020-05-10
CLIP (zero-shot)	Learning Transferable Visual Models From Natural …	0.66	2021-02-26