To realize the promise of ubiquitous embedded deep network inference, it is essential to seek limits of energy and area efficiency.   To this end, low-precision networks offer tremendous promise because both energy and area scale down quadratically with the reduction in precision.   Here, for the first time, we demonstrate ResNet-18, ResNet-34, ResNet-50, ResNet-152, Inception-v3, densenet-161, and VGG-16bn networks on the ImageNet classification benchmark that, at 8-bit precision exceed the accuracy of the full-precision baseline networks after one epoch of finetuning, thereby leveraging the availability of pretrained models.
 We also demonstrate ResNet-18, ResNet-34, and ResNet-50 4-bit models that match the accuracy of the full-precision baseline networks -- the highest scores to date. Surprisingly, the weights of the low-precision networks are very close (in cosine similarity) to the weights of the corresponding baseline networks, making training from scratch unnecessary.

 We find that gradient noise due to quantization during training increases with reduced precision, and seek ways to overcome this noise. The number of iterations required by stochastic gradient descent to achieve a given training error is related to the square of (a) the distance of the initial solution from the final plus (b) the maximum variance of the gradient estimates.   By drawing inspiration from this observation, we (a) reduce solution distance by starting with pretrained fp32 precision baseline networks and fine-tuning, and (b) combat noise introduced by quantizing weights and activations during training, by using larger batches along with matched learning rate annealing.   Sensitivity analysis indicates that these techniques, coupled with proper activation function range calibration, offer a promising heuristic to discover low-precision networks, if they exist, close to fp32 precision baseline networks.

Finetuning after quantization matches or exceeds full-precision state-of-the-art networks at both 8- and 4-bit quantization. This paper proposes to improve the performance of low-precision models by doing quantization on pre-trained models, using large batches size, and using proper learning rate annealing with longer training time. A method for low bit quantization to enable inference on efficient hardware that achieves full accuracy on ResNet50 with 4-bit weights and activations, based on observations that fine-tuning at low precision introduces noise in the gradient.