Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Unbiasing Truncated Backpropagation Through Time   Truncated Backpropagation Through Time (truncated BPTT) is a widespread
method for learning recurrent computational graphs. Truncated BPTT keeps the
computational benefits of Backpropagation Through Time (BPTT) while relieving
the need for a complete backtrack through the whole data sequence at every
step. However, truncation favors short-term dependencies: the gradient estimate
of truncated BPTT is biased, so that it does not benefit from the convergence
guarantees from stochastic gradient theory. We introduce Anticipated Reweighted
Truncated Backpropagation (ARTBP), an algorithm that keeps the computational
benefits of truncated BPTT, while providing unbiasedness. ARTBP works by using
variable truncation lengths together with carefully chosen compensation factors
in the backpropagation equation. We check the viability of ARTBP on two tasks.
First, a simple synthetic task where careful balancing of temporal dependencies
at different scales is needed: truncated BPTT displays unreliable performance,
and in worst case scenarios, divergence, while ARTBP converges reliably.
Second, on Penn Treebank character-level language modelling, ARTBP slightly
outperforms truncated BPTT.

17
The categories of this paper are: q-fin
23
The categories of this paper are: solv-int
28
The categories of this paper are: alg-geom
34
The categories of this paper are: chem-ph
22
The categories of this paper are: chao-dyn
37
The categories of this paper are: supr-con
19
The categories of this paper are: eess
36
The categories of this paper are: cmp-lg
33
The categories of this paper are: plasm-ph
5
The categories of this paper are: gr-qc
0
The categories of this paper are: hep-ph
18
The categories of this paper are: econ
31
The categories of this paper are: ao-sci
8
The categories of this paper are: hep-ex
2
The categories of this paper are: math
14
The categories of this paper are: math-ph