Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: A Hybrid-Order Distributed SGD Method for Non-Convex Optimization to
  Balance Communication Overhead, Computational Complexity, and Convergence
  Rate   In this paper, we propose a method of distributed stochastic gradient descent
(SGD), with low communication load and computational complexity, and still fast
convergence. To reduce the communication load, at each iteration of the
algorithm, the worker nodes calculate and communicate some scalers, that are
the directional derivatives of the sample functions in some \emph{pre-shared
directions}. However, to maintain accuracy, after every specific number of
iterations, they communicate the vectors of stochastic gradients. To reduce the
computational complexity in each iteration, the worker nodes approximate the
directional derivatives with zeroth-order stochastic gradient estimation, by
performing just two function evaluations rather than computing a first-order
gradient vector. The proposed method highly improves the convergence rate of
the zeroth-order methods, guaranteeing order-wise faster convergence. Moreover,
compared to the famous communication-efficient methods of model averaging (that
perform local model updates and periodic communication of the gradients to
synchronize the local models), we prove that for the general class of
non-convex stochastic problems and with reasonable choice of parameters, the
proposed method guarantees the same orders of communication load and
convergence rate, while having order-wise less computational complexity.
Experimental results on various learning problems in neural networks
applications demonstrate the effectiveness of the proposed approach compared to
various state-of-the-art distributed SGD methods.

7
The categories of this paper are: hep-th
8
The categories of this paper are: hep-ex
10
The categories of this paper are: q-bio
30
The categories of this paper are: q-alg
11
The categories of this paper are: hep-lat
13
The categories of this paper are: quant-ph
29
The categories of this paper are: funct-an
14
The categories of this paper are: math-ph
28
The categories of this paper are: alg-geom
5
The categories of this paper are: gr-qc
37
The categories of this paper are: supr-con
32
The categories of this paper are: atom-ph
9
The categories of this paper are: nlin
24
The categories of this paper are: dg-ga
23
The categories of this paper are: solv-int
25
The categories of this paper are: patt-sol