Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Improve Transformer Models with Better Relative Position Embeddings   Transformer architectures rely on explicit position encodings in order to
preserve a notion of word order. In this paper, we argue that existing work
does not fully utilize position information. For example, the initial proposal
of a sinusoid embedding is fixed and not learnable. In this paper, we first
review absolute position embeddings and existing methods for relative position
embeddings. We then propose new techniques that encourage increased interaction
between query, key and relative position embeddings in the self-attention
mechanism. Our most promising approach is a generalization of the absolute
position embedding, improving results on SQuAD1.1 compared to previous position
embeddings approaches. In addition, we address the inductive property of
whether a position embedding can be robust enough to handle long sequences. We
demonstrate empirically that our relative position embedding method is
reasonably generalized and robust from the inductive perspective. Finally, we
show that our proposed method can be adopted as a near drop-in replacement for
improving the accuracy of large models with a small computational budget.

22
The categories of this paper are: chao-dyn
5
The categories of this paper are: gr-qc
15
The categories of this paper are: nucl-ex
19
The categories of this paper are: eess
34
The categories of this paper are: chem-ph
33
The categories of this paper are: plasm-ph
12
The categories of this paper are: nucl-th
2
The categories of this paper are: math
36
The categories of this paper are: cmp-lg
14
The categories of this paper are: math-ph
32
The categories of this paper are: atom-ph
4
The categories of this paper are: cond-mat
8
The categories of this paper are: hep-ex
28
The categories of this paper are: alg-geom
9
The categories of this paper are: nlin
6
The categories of this paper are: astro-ph