Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: On the Convergence of A Class of Adam-Type Algorithms for Non-Convex
  Optimization   This paper studies a class of adaptive gradient based momentum algorithms
that update the search directions and learning rates simultaneously using past
gradients. This class, which we refer to as the "Adam-type", includes the
popular algorithms such as the Adam, AMSGrad and AdaGrad. Despite their
popularity in training deep neural networks, the convergence of these
algorithms for solving nonconvex problems remains an open question. This paper
provides a set of mild sufficient conditions that guarantee the convergence for
the Adam-type methods. We prove that under our derived conditions, these
methods can achieve the convergence rate of order $O(\log{T}/\sqrt{T})$ for
nonconvex stochastic optimization. We show the conditions are essential in the
sense that violating them may make the algorithm diverge. Moreover, we propose
and analyze a class of (deterministic) incremental adaptive gradient
algorithms, which has the same $O(\log{T}/\sqrt{T})$ convergence rate. Our
study could also be extended to a broader class of adaptive gradient methods in
machine learning and optimization.

10
The categories of this paper are: q-bio
36
The categories of this paper are: cmp-lg
21
The categories of this paper are: adap-org
12
The categories of this paper are: nucl-th
31
The categories of this paper are: ao-sci
32
The categories of this paper are: atom-ph
5
The categories of this paper are: gr-qc
18
The categories of this paper are: econ
34
The categories of this paper are: chem-ph
4
The categories of this paper are: cond-mat
35
The categories of this paper are: mtrl-th
7
The categories of this paper are: hep-th
9
The categories of this paper are: nlin
22
The categories of this paper are: chao-dyn
19
The categories of this paper are: eess
3
The categories of this paper are: physics