Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Convergence Theory of Learning Over-parameterized ResNet: A Full
  Characterization   ResNet structure has achieved great empirical success since its debut. Recent
work established the convergence of learning over-parameterized ResNet with a
scaling factor $\tau=1/L$ on the residual branch where $L$ is the network
depth. However, it is not clear how learning ResNet behaves for other values of
$\tau$. In this paper, we fully characterize the convergence theory of gradient
descent for learning over-parameterized ResNet with different values of $\tau$.
Specifically, with hiding logarithmic factor and constant coefficients, we show
that for $\tau\le 1/\sqrt{L}$ gradient descent is guaranteed to converge to the
global minma, and especially when $\tau\le 1/L$ the convergence is irrelevant
of the network depth. Conversely, we show that for $\tau>L^{-\frac{1}{2}+c}$,
the forward output grows at least with rate $L^c$ in expectation and then the
learning fails because of gradient explosion for large $L$. This means the
bound $\tau\le 1/\sqrt{L}$ is sharp for learning ResNet with arbitrary depth.
To the best of our knowledge, this is the first work that studies learning
ResNet with full range of $\tau$.

20
The categories of this paper are: acc-phys
0
The categories of this paper are: hep-ph
18
The categories of this paper are: econ
17
The categories of this paper are: q-fin
11
The categories of this paper are: hep-lat
37
The categories of this paper are: supr-con
23
The categories of this paper are: solv-int
19
The categories of this paper are: eess
25
The categories of this paper are: patt-sol
10
The categories of this paper are: q-bio
12
The categories of this paper are: nucl-th
3
The categories of this paper are: physics
22
The categories of this paper are: chao-dyn
13
The categories of this paper are: quant-ph
30
The categories of this paper are: q-alg
4
The categories of this paper are: cond-mat