Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Dynamic of Stochastic Gradient Descent with State-Dependent Noise   Stochastic gradient descent (SGD) and its variants are mainstream methods to
train deep neural networks. Since neural networks are non-convex, more and more
works study the dynamic behavior of SGD and the impact to its generalization,
especially the escaping efficiency from local minima. However, these works take
the over-simplified assumption that the covariance of the noise in SGD is (or
can be upper bounded by) constant, although it is actually state-dependent. In
this work, we conduct a formal study on the dynamic behavior of SGD with
state-dependent noise. Specifically, we show that the covariance of the noise
of SGD in the local region of the local minima is a quadratic function of the
state. Thus, we propose a novel power-law dynamic with state-dependent
diffusion to approximate the dynamic of SGD. We prove that, power-law dynamic
can escape from sharp minima exponentially faster than flat minima, while the
previous dynamics can only escape sharp minima polynomially faster than flat
minima. Our experiments well verified our theoretical results. Inspired by our
theory, we propose to add additional state-dependent noise into (large-batch)
SGD to further improve its generalization ability. Experiments verify that our
method is effective.

14
The categories of this paper are: math-ph
5
The categories of this paper are: gr-qc
25
The categories of this paper are: patt-sol
18
The categories of this paper are: econ
7
The categories of this paper are: hep-th
2
The categories of this paper are: math
21
The categories of this paper are: adap-org
31
The categories of this paper are: ao-sci
36
The categories of this paper are: cmp-lg
15
The categories of this paper are: nucl-ex
30
The categories of this paper are: q-alg
29
The categories of this paper are: funct-an
3
The categories of this paper are: physics
9
The categories of this paper are: nlin
34
The categories of this paper are: chem-ph
4
The categories of this paper are: cond-mat