Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: A Practical Sparse Approximation for Real Time Recurrent Learning   Current methods for training recurrent neural networks are based on
backpropagation through time, which requires storing a complete history of
network states, and prohibits updating the weights `online' (after every
timestep). Real Time Recurrent Learning (RTRL) eliminates the need for history
storage and allows for online weight updates, but does so at the expense of
computational costs that are quartic in the state size. This renders RTRL
training intractable for all but the smallest networks, even ones that are made
highly sparse.
  We introduce the Sparse n-step Approximation (SnAp) to the RTRL influence
matrix, which only keeps entries that are nonzero within n steps of the
recurrent core. SnAp with n=1 is no more expensive than backpropagation, and we
find that it substantially outperforms other RTRL approximations with
comparable costs such as Unbiased Online Recurrent Optimization. For highly
sparse networks, SnAp with n=2 remains tractable and can outperform
backpropagation through time in terms of learning speed when updates are done
online. SnAp becomes equivalent to RTRL when n is large.

27
The categories of this paper are: comp-gas
23
The categories of this paper are: solv-int
30
The categories of this paper are: q-alg
6
The categories of this paper are: astro-ph
2
The categories of this paper are: math
29
The categories of this paper are: funct-an
12
The categories of this paper are: nucl-th
37
The categories of this paper are: supr-con
32
The categories of this paper are: atom-ph
24
The categories of this paper are: dg-ga
36
The categories of this paper are: cmp-lg
11
The categories of this paper are: hep-lat
3
The categories of this paper are: physics
22
The categories of this paper are: chao-dyn
7
The categories of this paper are: hep-th
18
The categories of this paper are: econ