Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: A Convergence Analysis of Nesterov's Accelerated Gradient Method in
  Training Deep Linear Neural Networks   Momentum methods, including heavy-ball~(HB) and Nesterov's accelerated
gradient~(NAG), are widely used in training neural networks for their fast
convergence. However, there is a lack of theoretical guarantees for their
convergence and acceleration since the optimization landscape of the neural
network is non-convex. Nowadays, some works make progress towards understanding
the convergence of momentum methods in an over-parameterized regime, where the
number of the parameters exceeds that of the training instances. Nonetheless,
current results mainly focus on the two-layer neural network, which are far
from explaining the remarkable success of the momentum methods in training deep
neural networks. Motivated by this, we investigate the convergence of NAG with
constant learning rate and momentum parameter in training two architectures of
deep linear networks: deep fully-connected linear neural networks and deep
linear ResNets. Based on the over-parameterization regime, we first analyze the
residual dynamics induced by the training trajectory of NAG for a deep
fully-connected linear neural network under the random Gaussian initialization.
Our results show that NAG can converge to the global minimum at a $(1 -
\mathcal{O}(1/\sqrt{\kappa}))^t$ rate, where $t$ is the iteration number and
$\kappa > 1$ is a constant depending on the condition number of the feature
matrix. Compared to the $(1 - \mathcal{O}(1/{\kappa}))^t$ rate of GD, NAG
achieves an acceleration over GD. To the best of our knowledge, this is the
first theoretical guarantee for the convergence of NAG to the global minimum in
training deep neural networks. Furthermore, we extend our analysis to deep
linear ResNets and derive a similar convergence result.

27
The categories of this paper are: comp-gas
37
The categories of this paper are: supr-con
25
The categories of this paper are: patt-sol
14
The categories of this paper are: math-ph
13
The categories of this paper are: quant-ph
3
The categories of this paper are: physics
31
The categories of this paper are: ao-sci
29
The categories of this paper are: funct-an
22
The categories of this paper are: chao-dyn
6
The categories of this paper are: astro-ph
9
The categories of this paper are: nlin
34
The categories of this paper are: chem-ph
15
The categories of this paper are: nucl-ex
16
The categories of this paper are: stat
33
The categories of this paper are: plasm-ph
5
The categories of this paper are: gr-qc