Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Staircase Attention for Recurrent Processing of Sequences   Attention mechanisms have become a standard tool for sequence modeling tasks,
in particular by stacking self-attention layers over the entire input sequence
as in the Transformer architecture. In this work we introduce a novel attention
procedure called staircase attention that, unlike self-attention, operates
across the sequence (in time) recurrently processing the input by adding
another step of processing. A step in the staircase comprises of backward
tokens (encoding the sequence so far seen) and forward tokens (ingesting a new
part of the sequence), or an extreme Ladder version with a forward step of zero
that simply repeats the Transformer on each step of the ladder, sharing the
weights. We thus describe a family of such models that can trade off
performance and compute, by either increasing the amount of recurrence through
time, the amount of sequential processing via recurrence in depth, or both.
Staircase attention is shown to be able to solve tasks that involve tracking
that conventional Transformers cannot, due to this recurrence. Further, it is
shown to provide improved modeling power for the same size model (number of
parameters) compared to self-attentive Transformers on large language modeling
and dialogue tasks, yielding significant perplexity gains.

37
The categories of this paper are: supr-con
23
The categories of this paper are: solv-int
9
The categories of this paper are: nlin
7
The categories of this paper are: hep-th
35
The categories of this paper are: mtrl-th
27
The categories of this paper are: comp-gas
25
The categories of this paper are: patt-sol
19
The categories of this paper are: eess
18
The categories of this paper are: econ
28
The categories of this paper are: alg-geom
31
The categories of this paper are: ao-sci
21
The categories of this paper are: adap-org
10
The categories of this paper are: q-bio
36
The categories of this paper are: cmp-lg
13
The categories of this paper are: quant-ph
14
The categories of this paper are: math-ph