ML Research Wiki / Benchmarks / Cross-Modal Retrieval / Flickr30k

Flickr30k

Cross-Modal Retrieval Benchmark

Performance Over Time

📊 Showing 23 results | 📏 Metric: Image-to-text R@1

Top Performing Models

Rank	Model	Paper	Image-to-text R@1	Date	Code
1	ERNIE-ViL 2.0 📚	ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-training	93.30	2022-09-30	📦 PaddlePaddle/ERNIE
2	X2-VLM (large) 📚	X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks	91.80	2022-11-22	📦 zengyan-97/x-vlm 📦 zengyan-97/x2-vlm
3	VAST 📚	VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset	91.00	2023-05-29	📦 TXH-mercury/VALOR 📦 txh-mercury/vast
4	X2-VLM (base) 📚	X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks	90.40	2022-11-22	📦 zengyan-97/x-vlm 📦 zengyan-97/x2-vlm
5	BEiT-3 📚	Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks	90.30	2022-08-22	📦 microsoft/unilm 📦 lyan62/data-curation
6	OmniVL (14M) 📚	OmniVL:One Foundation Model for Image-Language and Video-Language Tasks	87.90	2022-09-15	-
7	X-VLM (base) 📚	Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts	86.90	2021-11-16	📦 zengyan-97/x-vlm
8	VSE-Gradient 📚	Dissecting Deep Metric Learning Losses for Image-Text Retrieval	86.30	2022-10-21	📦 microsoft/VSE_Gradient 📦 littleredxh/vse-gradient
9	ALIGN 📚	Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision	84.90	2021-02-11	📦 facebookresearch/metaclip 📦 kakaobrain/coyo-dataset 📦 MicPie/clasp 📦 willard-yuan/video-text-retrieval-papers 📦 pwc-1/Paper-8
10	IAIS 📚	Learning Relation Alignment for Calibrated Cross-modal Retrieval	76.86	2021-05-28	📦 lancopku/IAIS

All Papers (23)

ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-training

2022

ERNIE-ViL 2.0

PaddlePaddle/ERNIE

X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks

2022

X2-VLM (large)

zengyan-97/x-vlm zengyan-97/x2-vlm

VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset

2023

VAST

TXH-mercury/VALOR txh-mercury/vast

X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks

2022

X2-VLM (base)

zengyan-97/x-vlm zengyan-97/x2-vlm

Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

2022

BEiT-3

microsoft/unilm lyan62/data-curation

OmniVL:One Foundation Model for Image-Language and Video-Language Tasks

2022

OmniVL (14M)

Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts

2021

X-VLM (base)

zengyan-97/x-vlm

Dissecting Deep Metric Learning Losses for Image-Text Retrieval

2022

VSE-Gradient

microsoft/VSE_Gradient littleredxh/vse-gradient

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

2021

ALIGN

facebookresearch/metaclip kakaobrain/coyo-dataset

Learning Relation Alignment for Calibrated Cross-modal Retrieval

2021

IAIS

lancopku/IAIS

ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrieval

2022

ViSTA

3SHNet: Boosting Image-Sentence Retrieval via Visual Semantic-Spatial Self-Highlighting

2024

3SHNet

xurige1995/3shnet

Dynamic Self-adaptive Multiscale Distillation from Pre-trained Multimodal Large Model for Efficient Cross-modal Representation Learning

2024

DSMD

chrisx599/dsmd

ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision

2021

ViLT-B/32

huggingface/transformers dandelin/vilt

Plug-and-Play Regulators for Image-Text Matching

2023

RCAR

paranioar/rcar

Similarity Reasoning and Filtration for Image-Text Matching

2021

SGRAF

Paranioar/SGRAF

Graph Structured Network for Image-Text Matching

2020

GSMN

CrossmodalGroup/GSMN

Dual-Path Convolutional Image-Text Embeddings with Instance Loss

2017

Dual-Path (ResNet)

layumi/Image-Text-Embedding pshroff04/Dual_Path_CNN

IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrieval

2020

IMRAM

HuiChen24/IMRAM

Stacked Cross Attention for Image-Text Matching

2018

SCAN

kuanghuei/SCAN MysteryVaibhav/SCAN

Learning Semantic Concepts and Order for Image and Sentence Matching

2017

SCO (ResNet)

VSE++: Improving Visual-Semantic Embeddings with Hard Negatives

2017

VSE++ (ResNet)

fartashf/vsepp cshizhe/hgr_v2t

Dual-Path Convolutional Image-Text Embeddings with Instance Loss

2017

Dual-Path (ResNet)

layumi/Image-Text-Embedding pshroff04/Dual_Path_CNN

Flickr30k

Performance Over Time

Edit Benchmark Results

Edit Result

Top Performing Models

All Papers (23)

ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-training

X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks

VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset

X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks

Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

OmniVL:One Foundation Model for Image-Language and Video-Language Tasks

Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts

Dissecting Deep Metric Learning Losses for Image-Text Retrieval

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

Learning Relation Alignment for Calibrated Cross-modal Retrieval

ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrieval

3SHNet: Boosting Image-Sentence Retrieval via Visual Semantic-Spatial Self-Highlighting

Dynamic Self-adaptive Multiscale Distillation from Pre-trained Multimodal Large Model for Efficient Cross-modal Representation Learning

ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision

Plug-and-Play Regulators for Image-Text Matching

Similarity Reasoning and Filtration for Image-Text Matching

Graph Structured Network for Image-Text Matching

Dual-Path Convolutional Image-Text Embeddings with Instance Loss

IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrieval

Stacked Cross Attention for Image-Text Matching

Learning Semantic Concepts and Order for Image and Sentence Matching

VSE++: Improving Visual-Semantic Embeddings with Hard Negatives

Dual-Path Convolutional Image-Text Embeddings with Instance Loss

Model	Paper	Image-to-text R@1	Date
ERNIE-ViL 2.0	ERNIE-ViL 2.0: Multi-view Contrastive Learning fo…	93.30	2022-09-30
X2-VLM (large)	X$^2$-VLM: All-In-One Pre-trained Model For Visio…	91.80	2022-11-22
VAST	VAST: A Vision-Audio-Subtitle-Text Omni-Modality …	91.00	2023-05-29
X2-VLM (base)	X$^2$-VLM: All-In-One Pre-trained Model For Visio…	90.40	2022-11-22
BEiT-3	Image as a Foreign Language: BEiT Pretraining for…	90.30	2022-08-22
OmniVL (14M)	OmniVL:One Foundation Model for Image-Language an…	87.90	2022-09-15
X-VLM (base)	Multi-Grained Vision Language Pre-Training: Align…	86.90	2021-11-16
VSE-Gradient	Dissecting Deep Metric Learning Losses for Image-…	86.30	2022-10-21
ALIGN	Scaling Up Visual and Vision-Language Representat…	84.90	2021-02-11
IAIS	Learning Relation Alignment for Calibrated Cross-…	76.86	2021-05-28
ViSTA	ViSTA: Vision and Scene Text Aggregation for Cros…	75.80	2022-03-31
3SHNet	3SHNet: Boosting Image-Sentence Retrieval via Vis…	69.50	2024-04-26
DSMD	Dynamic Self-adaptive Multiscale Distillation fro…	68.40	2024-04-16
ViLT-B/32	ViLT: Vision-and-Language Transformer Without Con…	64.40	2021-02-05
RCAR	Plug-and-Play Regulators for Image-Text Matching	62.60	2023-03-23
SGRAF	Similarity Reasoning and Filtration for Image-Tex…	58.50	2021-01-05
GSMN	Graph Structured Network for Image-Text Matching	57.40	2020-04-01
Dual-Path (ResNet)	Dual-Path Convolutional Image-Text Embeddings wit…	55.60	2017-11-15
IMRAM	IMRAM: Iterative Matching with Recurrent Attentio…	53.90	2020-03-08
SCAN	Stacked Cross Attention for Image-Text Matching	48.60	2018-03-21
SCO (ResNet)	Learning Semantic Concepts and Order for Image an…	41.10	2017-12-06
VSE++ (ResNet)	VSE++: Improving Visual-Semantic Embeddings with …	39.60	2017-07-18
Dual-Path (ResNet)	Dual-Path Convolutional Image-Text Embeddings wit…	39.10	2017-11-15