ML Research Wiki / Benchmarks / Semantic Textual Similarity / STS Benchmark

STS Benchmark

Semantic Textual Similarity Benchmark

Performance Over Time

📊 Showing 62 results | 📏 Metric: Pearson Correlation

Top Performing Models

Rank	Model	Paper	Pearson Correlation	Date	Code
1	SMARTRoBERTa	SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization	92.80	2019-11-08	📦 namisan/mt-dnn 📦 microsoft/MT-DNN 📦 archinetai/smart-pytorch
2	DeBERTa (large)	DeBERTa: Decoding-enhanced BERT with Disentangled Attention	92.50	2020-06-05	📦 huggingface/transformers 📦 microsoft/DeBERTa 📦 osu-nlp-group/mind2web
3	SMART-BERT	SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization	90.00	2019-11-08	📦 namisan/mt-dnn 📦 microsoft/MT-DNN 📦 archinetai/smart-pytorch
4	MT-DNN-SMART	SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization	0.93	2019-11-08	📦 namisan/mt-dnn 📦 microsoft/MT-DNN 📦 archinetai/smart-pytorch
5	StructBERTRoBERTa ensemble	StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding	0.93	2019-08-13	-
6	Mnet-Sim	MNet-Sim: A Multi-layered Semantic Similarity Network to Evaluate Sentence Similarity	0.93	2021-11-09	-
7	T5-11B	Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer	0.93	2019-10-23	📦 huggingface/transformers 📦 PaddlePaddle/PaddleNLP 📦 google-research/text-to-text-transfer-transformer
8	ALBERT 📚	ALBERT: A Lite BERT for Self-supervised Learning of Language Representations	0.93	2019-09-26	📦 huggingface/transformers 📦 tensorflow/models 📦 PaddlePaddle/PaddleNLP
9	XLNet (single model)	XLNet: Generalized Autoregressive Pretraining for Language Understanding	0.93	2019-06-19	📦 huggingface/transformers 📦 PaddlePaddle/PaddleNLP 📦 zihangdai/xlnet
10	RoBERTa	RoBERTa: A Robustly Optimized BERT Pretraining Approach	0.92	2019-07-26	📦 huggingface/transformers 📦 pytorch/fairseq 📦 PaddlePaddle/PaddleNLP

All Papers (62)

SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization

2019

SMARTRoBERTa

namisan/mt-dnn microsoft/MT-DNN

DeBERTa: Decoding-enhanced BERT with Disentangled Attention

2020

DeBERTa (large)

huggingface/transformers microsoft/DeBERTa

SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization

2019

SMART-BERT

namisan/mt-dnn microsoft/MT-DNN

SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization

2019

MT-DNN-SMART

namisan/mt-dnn microsoft/MT-DNN

StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding

2019

StructBERTRoBERTa ensemble

MNet-Sim: A Multi-layered Semantic Similarity Network to Evaluate Sentence Similarity

2021

Mnet-Sim

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

2019

T5-11B

huggingface/transformers PaddlePaddle/PaddleNLP

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

2019

ALBERT

huggingface/transformers tensorflow/models

XLNet: Generalized Autoregressive Pretraining for Language Understanding

2019

XLNet (single model)

huggingface/transformers PaddlePaddle/PaddleNLP

RoBERTa: A Robustly Optimized BERT Pretraining Approach

2019

RoBERTa

huggingface/transformers pytorch/fairseq

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

2022

RoBERTa-large 355M (MLP quantized vector-wise, fine-tuned)

timdettmers/bitsandbytes huggingface/transformers-bloom-inference

A Statistical Framework for Low-bitwidth Training of Deep Neural Networks

2020

PSQ (Chen et al., 2020)

cjf00000/StatQuant gaochang-bjtu/1-bit-fqt

Entailment as Few-Shot Learner

2021

RoBERTa-large 355M + Entailment as Few-shot Learner

PaddlePaddle/PaddleNLP sunyilgdx/prompts4keras cactilab/hateguard

ERNIE 2.0: A Continual Pre-training Framework for Language Understanding

2019

ERNIE 2.0 Large

PaddlePaddle/PaddleNLP PaddlePaddle/ERNIE DataScienceNigeria/ERNIE-2.0-from-Baidu-Inc.

Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT

2019

Q-BERT (Shen et al., 2020)

Q8BERT: Quantized 8Bit BERT

2019

Q8BERT (Zafrir et al., 2019)

NervanaSystems/nlp-architect intellabs/model-compression-research-package

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

2019

DistilBERT 66M

huggingface/transformers PaddlePaddle/PaddleNLP

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

2019

T5-3B

huggingface/transformers PaddlePaddle/PaddleNLP

CLEAR: Contrastive Learning for Sentence Representation

2020

MLM+ del-word

RealFormer: Transformer Likes Residual Attention

2020

RealFormer

google-research/google-research cloneofsimo/RealFormer-pytorch

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

2019

T5-Large

huggingface/transformers PaddlePaddle/PaddleNLP

SpanBERT: Improving Pre-training by Representing and Predicting Spans

2019

SpanBERT

facebookresearch/SpanBERT mandarjoshi90/coref

AnglE-optimized Text Embeddings

2023

AnglE-LLaMA-13B

SeanLee97/AnglE 4ai/bellm

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

2019

T5-Base

huggingface/transformers PaddlePaddle/PaddleNLP

Adversarial Self-Attention for Language Understanding

2022

ASA + RoBERTa

gingasan/adversarialsa

Scaling Sentence Embeddings with Large Language Models

2023

PromptEOL+CSE+LLaMA-30B

kongds/scaling_sentemb

AnglE-optimized Text Embeddings

2023

AnglE-LLaMA-7B

SeanLee97/AnglE 4ai/bellm

AnglE-optimized Text Embeddings

2023

AnglE-LLaMA-7B-v2

SeanLee97/AnglE 4ai/bellm

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

2019

T5-Large 770M

huggingface/transformers PaddlePaddle/PaddleNLP

Scaling Sentence Embeddings with Large Language Models

2023

PromptEOL+CSE+OPT-13B

kongds/scaling_sentemb

Scaling Sentence Embeddings with Large Language Models

2023

PromptEOL+CSE+OPT-2.7B

kongds/scaling_sentemb

Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learning

2022

PromCSE-RoBERTa-large (0.355B)

yjiangcm/promcse

Big Bird: Transformers for Longer Sequences

2020

BigBird

huggingface/transformers tensorflow/models

ERNIE 2.0: A Continual Pre-training Framework for Language Understanding

2019

ERNIE 2.0 Base

PaddlePaddle/PaddleNLP PaddlePaddle/ERNIE DataScienceNigeria/ERNIE-2.0-from-Baidu-Inc.

Charformer: Fast Character Transformers via Gradient-based Subword Tokenization

2021

Charformer-Tall

google-research/google-research lucidrains/charformer-pytorch

SimCSE: Simple Contrastive Learning of Sentence Embeddings

2021

SimCSE-RoBERTalarge

PaddlePaddle/PaddleNLP princeton-nlp/SimCSE

Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations

2021

Trans-Encoder-RoBERTa-large-cross (unsup.)

amzn/trans-encoder

Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations

2021

Trans-Encoder-RoBERTa-large-bi (unsup.)

amzn/trans-encoder

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

2018

BERT-LARGE

huggingface/transformers tensorflow/models

Adversarial Self-Attention for Language Understanding

2022

ASA + BERT-base

gingasan/adversarialsa

Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations

2021

Trans-Encoder-BERT-large-bi (unsup.)

amzn/trans-encoder

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

2019

SRoBERTa-NLI-STSb-large

UKPLab/sentence-transformers PaddlePaddle/PaddleNLP

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

2019

T5-Small

huggingface/transformers PaddlePaddle/PaddleNLP

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

2019

SBERT-STSb-base

UKPLab/sentence-transformers PaddlePaddle/PaddleNLP

Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations

2021

Trans-Encoder-RoBERTa-base-cross (unsup.)

amzn/trans-encoder

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

2019

SBERT-STSb-large

UKPLab/sentence-transformers PaddlePaddle/PaddleNLP

FNet: Mixing Tokens with Fourier Transforms

2021

FNet-Large

labmlai/annotated_deep_learning_paper_implementations google-research/google-research

Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations

2021

Trans-Encoder-BERT-base-bi (unsup.)

amzn/trans-encoder

ERNIE: Enhanced Language Representation with Informative Entities

2019

ERNIE

thunlp/ERNIE Mind23-2/MindCode-136

How to Train BERT with an Academic Budget

2021

24hBERT

peteriz/academic-budget-bert IntelLabs/academic-budget-bert

TinyBERT: Distilling BERT for Natural Language Understanding

2019

TinyBERT-4 14.5M

PaddlePaddle/PaddleNLP huawei-noah/Pretrained-Language-Model

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

2019

SBERT-NLI-large

UKPLab/sentence-transformers PaddlePaddle/PaddleNLP

Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders

2021

Mirror-RoBERTa-base (unsup.)

cambridgeltl/mirror-bert

Universal Sentence Encoder

2018

USE_T

facebookresearch/InferSent facebookresearch/SentEval

Generating Datasets with Pretrained Language Models

2021

Dino (STSb/̄🦕)

timoschick/dino yipingnus/scratchplot-story-generation

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

2019

SRoBERTa-NLI-base

UKPLab/sentence-transformers PaddlePaddle/PaddleNLP

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

2019

SBERT-NLI-base

UKPLab/sentence-transformers PaddlePaddle/PaddleNLP

Generating Datasets with Pretrained Language Models

2021

Dino (STS/̄🦕)

timoschick/dino yipingnus/scratchplot-story-generation

Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders

2021

Mirror-BERT-base (unsup.)

cambridgeltl/mirror-bert

On the Sentence Embeddings from Pre-trained Language Models

2020

BERTlarge-flow (target)

InsaneLife/dssm bohanli/BERT-flow sleepthroughdifficulties/kernelwhitening

An Unsupervised Sentence Embedding Method by Mutual Information Maximization

2020

IS-BERT-NLI

yanzhangnlp/IS-BERT

Rematch: Robust and Efficient Matching of Local Knowledge Graphs to Improve Structural and Semantic Similarity

2024

Rematch

osome-iu/Rematch-RARE

Model	Paper	Pearson Correlation	Date
SMARTRoBERTa	SMART: Robust and Efficient Fine-Tuning for Pre-t…	92.80	2019-11-08
DeBERTa (large)	DeBERTa: Decoding-enhanced BERT with Disentangled…	92.50	2020-06-05
SMART-BERT	SMART: Robust and Efficient Fine-Tuning for Pre-t…	90.00	2019-11-08
MT-DNN-SMART	SMART: Robust and Efficient Fine-Tuning for Pre-t…	0.93	2019-11-08
StructBERTRoBERTa ensemble	StructBERT: Incorporating Language Structures int…	0.93	2019-08-13
Mnet-Sim	MNet-Sim: A Multi-layered Semantic Similarity Net…	0.93	2021-11-09
T5-11B	Exploring the Limits of Transfer Learning with a …	0.93	2019-10-23
ALBERT	ALBERT: A Lite BERT for Self-supervised Learning …	0.93	2019-09-26
XLNet (single model)	XLNet: Generalized Autoregressive Pretraining for…	0.93	2019-06-19
RoBERTa	RoBERTa: A Robustly Optimized BERT Pretraining Ap…	0.92	2019-07-26
RoBERTa-large 355M (MLP quantized vector-wise, fine-tuned)	LLM.int8(): 8-bit Matrix Multiplication for Trans…	0.92	2022-08-15
PSQ (Chen et al., 2020)	A Statistical Framework for Low-bitwidth Training…	0.92	2020-10-27
RoBERTa-large 355M + Entailment as Few-shot Learner	Entailment as Few-Shot Learner	0.92	2021-04-29
ERNIE 2.0 Large	ERNIE 2.0: A Continual Pre-training Framework for…	0.91	2019-07-29
Q-BERT (Shen et al., 2020)	Q-BERT: Hessian Based Ultra Low Precision Quantiz…	0.91	2019-09-12
Q8BERT (Zafrir et al., 2019)	Q8BERT: Quantized 8Bit BERT	0.91	2019-10-14
DistilBERT 66M	DistilBERT, a distilled version of BERT: smaller,…	0.91	2019-10-02
T5-3B	Exploring the Limits of Transfer Learning with a …	0.91	2019-10-23
MLM+ del-word	CLEAR: Contrastive Learning for Sentence Represen…	0.91	2020-12-31
RealFormer	RealFormer: Transformer Likes Residual Attention	0.90	2020-12-21
T5-Large	Exploring the Limits of Transfer Learning with a …	0.90	2019-10-23
SpanBERT	SpanBERT: Improving Pre-training by Representing …	0.90	2019-07-24
AnglE-LLaMA-13B	AnglE-optimized Text Embeddings	0.90	2023-09-22
T5-Base	Exploring the Limits of Transfer Learning with a …	0.89	2019-10-23
ASA + RoBERTa	Adversarial Self-Attention for Language Understan…	0.89	2022-06-25
PromptEOL+CSE+LLaMA-30B	Scaling Sentence Embeddings with Large Language M…	0.89	2023-07-31
AnglE-LLaMA-7B	AnglE-optimized Text Embeddings	0.89	2023-09-22
AnglE-LLaMA-7B-v2	AnglE-optimized Text Embeddings	0.89	2023-09-22
T5-Large 770M	Exploring the Limits of Transfer Learning with a …	0.89	2019-10-23
PromptEOL+CSE+OPT-13B	Scaling Sentence Embeddings with Large Language M…	0.89	2023-07-31
PromptEOL+CSE+OPT-2.7B	Scaling Sentence Embeddings with Large Language M…	0.88	2023-07-31
PromCSE-RoBERTa-large (0.355B)	Improved Universal Sentence Embeddings with Promp…	0.88	2022-03-14
BigBird	Big Bird: Transformers for Longer Sequences	0.88	2020-07-28
ERNIE 2.0 Base	ERNIE 2.0: A Continual Pre-training Framework for…	0.88	2019-07-29
Charformer-Tall	Charformer: Fast Character Transformers via Gradi…	0.87	2021-06-23
SimCSE-RoBERTalarge	SimCSE: Simple Contrastive Learning of Sentence E…	0.87	2021-04-18
Trans-Encoder-RoBERTa-large-cross (unsup.)	Trans-Encoder: Unsupervised sentence-pair modelli…	0.87	2021-09-27
Trans-Encoder-RoBERTa-large-bi (unsup.)	Trans-Encoder: Unsupervised sentence-pair modelli…	0.87	2021-09-27
BERT-LARGE	BERT: Pre-training of Deep Bidirectional Transfor…	0.87	2018-10-11
ASA + BERT-base	Adversarial Self-Attention for Language Understan…	0.87	2022-06-25
Trans-Encoder-BERT-large-bi (unsup.)	Trans-Encoder: Unsupervised sentence-pair modelli…	0.86	2021-09-27
SRoBERTa-NLI-STSb-large	Sentence-BERT: Sentence Embeddings using Siamese …	0.86	2019-08-27
T5-Small	Exploring the Limits of Transfer Learning with a …	0.86	2019-10-23
SBERT-STSb-base	Sentence-BERT: Sentence Embeddings using Siamese …	0.85	2019-08-27
Trans-Encoder-RoBERTa-base-cross (unsup.)	Trans-Encoder: Unsupervised sentence-pair modelli…	0.85	2021-09-27
SBERT-STSb-large	Sentence-BERT: Sentence Embeddings using Siamese …	0.84	2019-08-27
FNet-Large	FNet: Mixing Tokens with Fourier Transforms	0.84	2021-05-09
Trans-Encoder-BERT-base-bi (unsup.)	Trans-Encoder: Unsupervised sentence-pair modelli…	0.84	2021-09-27
ERNIE	ERNIE: Enhanced Language Representation with Info…	0.83	2019-05-17
24hBERT	How to Train BERT with an Academic Budget	0.82	2021-04-15
TinyBERT-4 14.5M	TinyBERT: Distilling BERT for Natural Language Un…	0.80	2019-09-23
SBERT-NLI-large	Sentence-BERT: Sentence Embeddings using Siamese …	0.79	2019-08-27
Mirror-RoBERTa-base (unsup.)	Fast, Effective, and Self-Supervised: Transformin…	0.79	2021-04-16
USE_T	Universal Sentence Encoder	0.78	2018-03-29
Dino (STSb/̄🦕)	Generating Datasets with Pretrained Language Mode…	0.78	2021-04-15
SRoBERTa-NLI-base	Sentence-BERT: Sentence Embeddings using Siamese …	0.78	2019-08-27
SBERT-NLI-base	Sentence-BERT: Sentence Embeddings using Siamese …	0.77	2019-08-27
Dino (STS/̄🦕)	Generating Datasets with Pretrained Language Mode…	0.77	2021-04-15
Mirror-BERT-base (unsup.)	Fast, Effective, and Self-Supervised: Transformin…	0.76	2021-04-16
BERTlarge-flow (target)	On the Sentence Embeddings from Pre-trained Langu…	0.72	2020-11-02
IS-BERT-NLI	An Unsupervised Sentence Embedding Method by Mutu…	0.69	2020-09-25
Rematch	Rematch: Robust and Efficient Matching of Local K…	0.67	2024-04-02

STS Benchmark

Performance Over Time

Edit Benchmark Results

Edit Result

Top Performing Models

All Papers (62)