Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: ABC: Attention with Bounded-memory Control   Transformer architectures have achieved state-of-the-art results on a variety
of sequence modeling tasks. However, their attention mechanism comes with a
quadratic complexity in sequence lengths, making the computational overhead
prohibitive, especially for long sequences. Attention context can be seen as a
random-access memory with each token taking a slot. Under this perspective, the
memory size grows linearly with the sequence length, and so does the overhead
of reading from it. One way to improve the efficiency is to bound the memory
size. We show that disparate approaches can be subsumed into one abstraction,
attention with bounded-memory control (ABC), and they vary in their
organization of the memory. ABC reveals new, unexplored possibilities. First,
it connects several efficient attention variants that would otherwise seem
apart. Second, this abstraction gives new insights--an established approach
(Wang et al., 2020b) previously thought to be not applicable in causal
attention, actually is. Last, we present a new instance of ABC, which draws
inspiration from existing ABC approaches, but replaces their heuristic
memory-organizing functions with a learned, contextualized one. Our experiments
on language modeling, machine translation, and masked language model finetuning
show that our approach outperforms previous efficient attention models;
compared to the strong transformer baselines, it significantly improves the
inference time and space efficiency with no or negligible accuracy loss.

14
The categories of this paper are: math-ph
32
The categories of this paper are: atom-ph
22
The categories of this paper are: chao-dyn
18
The categories of this paper are: econ
37
The categories of this paper are: supr-con
29
The categories of this paper are: funct-an
33
The categories of this paper are: plasm-ph
11
The categories of this paper are: hep-lat
4
The categories of this paper are: cond-mat
5
The categories of this paper are: gr-qc
10
The categories of this paper are: q-bio
28
The categories of this paper are: alg-geom
16
The categories of this paper are: stat
9
The categories of this paper are: nlin
26
The categories of this paper are: bayes-an
0
The categories of this paper are: hep-ph