← ML Research Wiki / 2010.11929

AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE

Alexey Dosovitskiy [email protected], Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby [email protected] (2020)

Paper Information
arXiv ID
2010.11929
Venue
International Conference on Learning Representations
Domain
artificial intelligence, deep learning, computer vision
SOTA Claim
Yes
Reproducibility
8/10
Contents

Abstract

While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited.In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place.We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks.When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train. 1

Summary

This paper presents the Vision Transformer (ViT), a model that applies the Transformer architecture directly to image classification tasks without relying on convolutional neural networks (CNNs). By treating image patches as sequences akin to words in text processing, the ViT achieves competitive results on various benchmarks, particularly when pre-trained on large datasets such as ImageNet-21k and JFT-300M. The authors demonstrate that while Transformers traditionally underperform on smaller datasets due to a lack of inductive bias inherent in CNNs, they excel when scaled to larger datasets. The experimental results show that the ViT achieves impressive accuracies, reaching 88.55% on ImageNet and outperforming state-of-the-art CNNs in terms of computational efficiency. The paper emphasizes the potential of Transformers to transform image recognition approaches, especially at scale, while also pointing to future challenges in adapting the model for other vision tasks and enhancing self-supervised learning techniques.

Methods

This paper employs the following methods:

  • Transformer
  • Vision Transformer

Models Used

  • Vision Transformer (ViT)

Datasets

The following datasets were used in this research:

  • ImageNet
  • ImageNet-21k
  • JFT-300M
  • CIFAR-100
  • VTAB

Evaluation Metrics

  • Accuracy

Results

  • ViT achieves 88.55% accuracy on ImageNet
  • ViT reaches 90.72% on ImageNet-ReaL
  • ViT obtains 94.55% accuracy on CIFAR-100
  • ViT performs 77.63% on VTAB suite

Limitations

The authors identified the following limitations:

  • The model lacks some inductive biases found in CNNs, affecting performance on smaller datasets
  • There is a performance gap between self-supervised and large-scale supervised pre-training.

Technical Requirements

  • Number of GPUs: 8
  • GPU Type: TPU v3

Keywords

Transformers Vision Transformer ViT self-attention image classification large-scale training

Papers Using Similar Methods

External Resources

Funding: Not specified
References: 65
Influential Citations: 4885