We study the convergence of gradient descent (GD) and stochastic gradient descent (SGD) for training $L$-hidden-layer linear residual networks (ResNets). We prove that for training deep residual networks with certain linear transformations at input and output layers, which are fixed throughout training, both GD and SGD with zero initialization on all hidden weights can converge to the global minimum of the training loss. Moreover, when specializing to appropriate Gaussian random linear transformations, GD and SGD provably optimize wide enough deep linear ResNets. Compared with the global convergence result of GD for training standard deep linear networks \citep{du2019width}, our condition on the neural network width is sharper by a factor of $O(\kappa L)$, where $\kappa$ denotes the condition number of the covariance matrix of the training data. In addition, for the first time we establish the global convergence of SGD for training deep linear ResNets and prove a linear convergence rate when the global minimum is $0$.
Under certain condition on the input and output linear transformations, both GD and SGD can achieve global convergence for training deep linear ResNets. The authors study the convergence of gradient descent in training deep linear residual networks, and establish a global convergence of GD/SGD and linear convergence rates of SG/SGD. Study of convergence properties of GD and SGD on deep linear resnets, and proof that under certain conditions on the input and output transformations and with zero initialization, GD and SGD converges to global minima.