Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Learning to Encode Position for Transformer with Continuous Dynamical
  Model   We introduce a new way of learning to encode position information for
non-recurrent models, such as Transformer models. Unlike RNN and LSTM, which
contain inductive bias by loading the input tokens sequentially, non-recurrent
models are less sensitive to position. The main reason is that position
information among input units is not inherently encoded, i.e., the models are
permutation equivalent; this problem justifies why all of the existing models
are accompanied by a sinusoidal encoding/embedding layer at the input. However,
this solution has clear limitations: the sinusoidal encoding is not flexible
enough as it is manually designed and does not contain any learnable
parameters, whereas the position embedding restricts the maximum length of
input sequences. It is thus desirable to design a new position layer that
contains learnable parameters to adjust to different datasets and different
architectures. At the same time, we would also like the encodings to
extrapolate in accordance with the variable length of inputs. In our proposed
solution, we borrow from the recent Neural ODE approach, which may be viewed as
a versatile continuous version of a ResNet. This model is capable of modeling
many kinds of dynamical systems. We model the evolution of encoded results
along position index by such a dynamical system, thereby overcoming the above
limitations of existing methods. We evaluate our new position layers on a
variety of neural machine translation and language understanding tasks, the
experimental results show consistent improvements over the baselines.

18
The categories of this paper are: econ
34
The categories of this paper are: chem-ph
4
The categories of this paper are: cond-mat
11
The categories of this paper are: hep-lat
10
The categories of this paper are: q-bio
35
The categories of this paper are: mtrl-th
33
The categories of this paper are: plasm-ph
24
The categories of this paper are: dg-ga
36
The categories of this paper are: cmp-lg
37
The categories of this paper are: supr-con
17
The categories of this paper are: q-fin
6
The categories of this paper are: astro-ph
14
The categories of this paper are: math-ph
20
The categories of this paper are: acc-phys
9
The categories of this paper are: nlin
31
The categories of this paper are: ao-sci