Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: 8-Bit Approximations for Parallelism in Deep Learning   The creation of practical deep learning data-products often requires
parallelization across processors and computers to make deep learning feasible
on large data sets, but bottlenecks in communication bandwidth make it
difficult to attain good speedups through parallelism. Here we develop and test
8-bit approximation algorithms which make better use of the available bandwidth
by compressing 32-bit gradients and nonlinear activations to 8-bit
approximations. We show that these approximations do not decrease predictive
performance on MNIST, CIFAR10, and ImageNet for both model and data parallelism
and provide a data transfer speedup of 2x relative to 32-bit parallelism. We
build a predictive model for speedups based on our experimental data, verify
its validity on known speedup data, and show that we can obtain a speedup of
50x and more on a system of 96 GPUs compared to a speedup of 23x for 32-bit. We
compare our data types with other methods and show that 8-bit approximations
achieve state-of-the-art speedups for model parallelism. Thus 8-bit
approximation is an efficient method to parallelize convolutional networks on
very large systems of GPUs.

2
The categories of this paper are: math
28
The categories of this paper are: alg-geom
15
The categories of this paper are: nucl-ex
32
The categories of this paper are: atom-ph
37
The categories of this paper are: supr-con
6
The categories of this paper are: astro-ph
13
The categories of this paper are: quant-ph
9
The categories of this paper are: nlin
4
The categories of this paper are: cond-mat
27
The categories of this paper are: comp-gas
12
The categories of this paper are: nucl-th
20
The categories of this paper are: acc-phys
3
The categories of this paper are: physics
36
The categories of this paper are: cmp-lg
35
The categories of this paper are: mtrl-th
23
The categories of this paper are: solv-int