Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Communication-Efficient Distributed Blockwise Momentum SGD with
  Error-Feedback   Communication overhead is a major bottleneck hampering the scalability of
distributed machine learning systems. Recently, there has been a surge of
interest in using gradient compression to improve the communication efficiency
of distributed neural network training. Using 1-bit quantization, signSGD with
majority vote achieves a 32x reduction on communication cost. However, its
convergence is based on unrealistic assumptions and can diverge in practice. In
this paper, we propose a general distributed compressed SGD with Nesterov's
momentum. We consider two-way compression, which compresses the gradients both
to and from workers. Convergence analysis on nonconvex problems for general
gradient compressors is provided. By partitioning the gradient into blocks, a
blockwise compressor is introduced such that each gradient block is compressed
and transmitted in 1-bit format with a scaling factor, leading to a nearly 32x
reduction on communication. Experimental results show that the proposed method
converges as fast as full-precision distributed momentum SGD and achieves the
same testing accuracy. In particular, on distributed ResNet training with 7
workers on the ImageNet, the proposed algorithm achieves the same testing
accuracy as momentum SGD using full-precision gradients, but with $46\%$ less
wall clock time.

34
The categories of this paper are: chem-ph
33
The categories of this paper are: plasm-ph
13
The categories of this paper are: quant-ph
23
The categories of this paper are: solv-int
19
The categories of this paper are: eess
7
The categories of this paper are: hep-th
26
The categories of this paper are: bayes-an
4
The categories of this paper are: cond-mat
32
The categories of this paper are: atom-ph
21
The categories of this paper are: adap-org
18
The categories of this paper are: econ
20
The categories of this paper are: acc-phys
36
The categories of this paper are: cmp-lg
28
The categories of this paper are: alg-geom
14
The categories of this paper are: math-ph
22
The categories of this paper are: chao-dyn