Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Improving the Gating Mechanism of Recurrent Neural Networks   Gating mechanisms are widely used in neural network models, where they allow
gradients to backpropagate more easily through depth or time. However, their
saturation property introduces problems of its own. For example, in recurrent
models these gates need to have outputs near 1 to propagate information over
long time-delays, which requires them to operate in their saturation regime and
hinders gradient-based learning of the gate mechanism. We address this problem
by deriving two synergistic modifications to the standard gating mechanism that
are easy to implement, introduce no additional hyperparameters, and improve
learnability of the gates when they are close to saturation. We show how these
changes are related to and improve on alternative recently proposed gating
mechanisms such as chrono initialization and Ordered Neurons. Empirically, our
simple gating mechanisms robustly improve the performance of recurrent models
on a range of applications, including synthetic memorization tasks, sequential
image classification, language modeling, and reinforcement learning,
particularly when long-term dependencies are involved.

18
The categories of this paper are: econ
5
The categories of this paper are: gr-qc
0
The categories of this paper are: hep-ph
2
The categories of this paper are: math
22
The categories of this paper are: chao-dyn
19
The categories of this paper are: eess
31
The categories of this paper are: ao-sci
25
The categories of this paper are: patt-sol
9
The categories of this paper are: nlin
36
The categories of this paper are: cmp-lg
20
The categories of this paper are: acc-phys
15
The categories of this paper are: nucl-ex
28
The categories of this paper are: alg-geom
32
The categories of this paper are: atom-ph
37
The categories of this paper are: supr-con
24
The categories of this paper are: dg-ga