Recurrent Neural Networks (RNNs) continue to show  outstanding performance in sequence modeling tasks. However, training RNNs on long sequences often face challenges like slow inference, vanishing gradients and difficulty in capturing long term dependencies. In backpropagation through time settings, these issues are tightly coupled with the large, sequential computational graph resulting from unfolding the RNN in time. We introduce the Skip RNN model which extends existing RNN models by learning to skip state updates and shortens the effective size of the computational graph. This model can also be encouraged to perform fewer state updates through a budget constraint. We evaluate the proposed model on various tasks and show how it can reduce the number of required RNN updates while preserving, and sometimes even improving, the performance of the baseline RNN models. Source code is publicly available at https://imatge-upc.github.io/skiprnn-2017-telecombcn/.
A modification for existing RNN architectures which allows them to skip state updates while preserving the performance of the original architectures. Proposes the Skip RNN model which allows a recurrent network to selectively skip updating its hidden state for some inputs, leading to reduced computation at test-time. Proposes a novel RNN model where both the input and the state update of the recurrent cells are skipped adaptively for some time steps.