We develop a framework for understanding and improving recurrent neural networks (RNNs) using max-affine spline operators (MASOs). We prove that RNNs using piecewise affine and convex nonlinearities can be written as a simple piecewise affine spline operator. The resulting representation provides several new perspectives for analyzing RNNs, three of which we study in this paper. First, we show that an RNN internally partitions the input space during training and that it builds up the partition through time. Second, we show that the affine slope parameter of an RNN corresponds to an input-specific template, from which we can interpret an RNN as performing a simple template matching (matched filtering) given the input. Third, by carefully examining the MASO RNN affine mapping, we prove that using a random initial hidden state corresponds to an explicit L2 regularization of the affine parameters, which can mollify exploding gradients and improve generalization. Extensive experiments on several datasets of various modalities demonstrate and validate each of the above conclusions. In particular, using a random initial hidden states elevates simple RNNs to near state-of-the-art performers on these datasets.
We provide new insights and interpretations of RNNs from a max-affine spline operators perspective. Rewrites equations of Elman RNN in terms of so-called max-affine spline operators Provide a novel approach toward understanding RNNs using max-affline spline operators (MASO) by rewriting them with piecewise affine and convex activations MASOs The authors build upon max-affine spline operator interpetation of a substantial class of deep networks, focusing on Recurrent Neural Networks using noise in initial hidden state acts as regularization