Distilling the Knowledge in a Neural Network

Abstract edit

Summary

The paper discusses a method called 'distillation' for transferring knowledge from large, cumbersome neural network models or ensembles of models to smaller, more efficient models. The authors, Hinton, Vinyals, and Dean, illustrate this approach by referencing existing work on model compression and elaborating on their technique which employs soft targets derived from the cumbersome models to train smaller models. They demonstrate the effectiveness of distillation on datasets such as MNIST and in applications like automatic speech recognition, achieving significant improvements in model performance while being computationally efficient. They also introduce specialist models that can work in conjunction with a generalist model to better handle large datasets with many classes, thus improving efficiency and accuracy while mitigating overfitting through soft targets. The paper concludes by emphasizing the potential for distillation to bridge the gap between training complex models and deploying simpler ones effectively.

Methods add

This paper employs the following methods:

• Distillation
• Softmax
• Stochastic Gradient Descent

Models Used add

• Neural Networks

Datasets add

The following datasets were used in this research:

• MNIST
• JFT

Evaluation Metrics add

• Frame Classification Accuracy
• Word Error Rate (WER)

Results add

• Distillation improves performance significantly on MNIST and speech recognition tasks.
• Over 80% of the improvement in frame classification accuracy from an ensemble of models is transferred to the distilled model.

Technical Requirements edit

• Number of GPUs: None specified
• GPU Type: None specified

Keywords add

Related Papers

Papers with similar methods or datasets:

• Fast R-CNN (2015) - 24186 citations
• SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation (2015) - 15261 citations
• Caffe: Convolutional Architecture for Fast Feature Embedding * (2014) - 14659 citations
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (2017) - 11356 citations
• Perceptual Losses for Real-Time Style Transfer and Super-Resolution (2016) - 9974 citations
• ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION (2014) - 146110 citations
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (2015) - 42400 citations
• Generative Adversarial Networks (2021) - 30040 citations
• Learning Transferable Visual Models From Natural Language Supervision (2021) - 25508 citations
• Published as a conference paper at ICLR 2015 EXPLAINING AND HARNESSING ADVERSARIAL EXAMPLES (2014) - 18317 citations

Papers Using Similar Methods

• Fast R-CNN (2015)
• SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation (2015)
• Caffe: Convolutional Architecture for Fast Feature Embedding * (2014)
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (2017)
• Perceptual Losses for Real-Time Style Transfer and Super-Resolution (2016)

External Resources