Normalization layers are a staple in state-of-the-art deep neural network architectures. They are widely believed to stabilize training, enable higher learning rate, accelerate convergence and improve generalization, though the reason for their effectiveness is still an active research topic. In this work, we challenge the commonly-held beliefs by showing that none of the perceived benefits is unique to normalization. Specifically, we propose fixed-update initialization (Fixup), an initialization motivated by solving the exploding and vanishing gradient problem at the beginning of training via properly rescaling a standard initialization. We find training residual networks with Fixup to be as stable as training with normalization -- even for networks with 10,000 layers. Furthermore, with proper regularization, Fixup enables residual networks without normalization to achieve state-of-the-art performance in image classification and machine translation.
All you need to train deep residual networks is a good initialization; normalization layers are not necessary. A method is presented for initialization and normalization of deep residual networks. This is based on observations of forward and backward explosion in such networks. The method performance is on par with the best results obtained by other networks with more explicit normalization. The authors propose a novel way to initialize residual networks, which is motivated by the need to avoid exploding/vanishing gradients. Proposes a new initialization method used to train very deep RedNets without using batch-norm.