← ML Research Wiki / 1912.01703

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Adam Paszke [email protected], Sam Gross [email protected], Francisco Massa [email protected], Adam Lerer [email protected], James Bradbury Google, Gregory Chanan [email protected], Trevor Killeen [email protected], Zeming Lin [email protected], Natalia Gimelshein [email protected], Luca Antiga [email protected], Alban Desmaison, Andreas Köpf [email protected], Edward Yang [email protected], Zach Devito [email protected], Martin Raison Nabla, Alykhan Tejani, Sasank Chilamkurthy [email protected], Qure Ai, Benoit Steiner [email protected], Lu Fang Facebook, Junjie Bai Facebook, Soumith Chintala [email protected], Facebook AI Research Facebook AI Research Facebook AI Research Facebook AI Research Self Employed Facebook AI Research University of Warsaw NVIDIA Orobix, Facebook AI Research Facebook AI Research Facebook AI Research Facebook AI Research Oxford University (2019)

Paper Information
arXiv ID
1912.01703
Venue
Neural Information Processing Systems
Domain
Computer science / Artificial intelligence
SOTA Claim
Yes
Code
Available
Reproducibility
8/10
Contents

Abstract

Deep learning frameworks have often focused on either usability or speed, but not both. PyTorch is a machine learning library that shows that these two goals are in fact compatible: it provides an imperative and Pythonic programming style that supports code as a model, makes debugging easy and is consistent with other popular scientific computing libraries, while remaining efficient and supporting hardware accelerators such as GPUs. In this paper, we detail the principles that drove the implementation of PyTorch and how they are reflected in its architecture. We emphasize that every aspect of PyTorch is a regular Python program under the full control of its user. We also explain how the careful and pragmatic implementation of the key components of its runtime enables them to work together to achieve compelling performance. We demonstrate the efficiency of individual subsystems, as well as the overall speed of PyTorch on several common benchmarks.

Summary

This paper presents PyTorch, a high-performance deep learning library that combines an imperative programming style with efficient computation. The authors argue that usability and speed can coexist, providing a Pythonic interface that simplifies debugging and model deployment. Key design principles emphasize the importance of being user-friendly for researchers, while balancing performance with simplicity. The paper discusses various system optimizations that address Python's performance limitations, especially in GPU execution, memory management, and automatic differentiation mechanics. The evaluation showcases PyTorch's competitive performance against other frameworks using common benchmarks. By adopting a user-centric approach and leveraging the strengths of Python’s ecosystem, PyTorch has gained significant adoption in the deep learning community and aims to further enhance its capabilities.

Methods

This paper employs the following methods:

  • Automatic Differentiation
  • Dynamic Eager Execution
  • GPU Acceleration

Models Used

  • None specified

Datasets

The following datasets were used in this research:

  • None specified

Evaluation Metrics

  • None specified

Results

  • Competitive performance against other deep learning frameworks
  • Popularity in the research community with numerous mentions in papers

Limitations

The authors identified the following limitations:

  • None specified

Technical Requirements

  • Number of GPUs: 1
  • GPU Type: NVIDIA Quadro GP100

Keywords

PyTorch deep learning autograd GPU acceleration imperative programming

Papers Using Similar Methods

External Resources

Funding: Facebook AI Research
References: 39
Influential Citations: 4128