We propose the dense RNN, which has the fully connections from each hidden state to multiple preceding hidden states of all layers directly. As the density of the connection increases, the number of paths through which the gradient flows can be increased. It increases the magnitude of gradients, which help to prevent the vanishing gradient problem in time. Larger gradients, however, can also cause exploding gradient problem. To complement the trade-off between two problems, we propose an attention gate, which controls the amounts of gradient flows. We describe the relation between the attention gate and the gradient flows by approximation. The experiment on the language modeling using Penn Treebank corpus shows dense connections with the attention gate improve the model’s performance.
Dense RNN that has fully connections from each hidden state to multiple preceding hidden states of all layers directly. Proposes a new RNN architecture that models long-term dependencies better, can learn multiscale representation of sequential data, and sidestep the gradients problem by using parametrized gating units. This paper proposes a fully connected dense RNN architecture with gated connections to every layer and preceding layer connections, and it's results on PTB charcter-level modelling task.