Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Exploring Unsupervised Pretraining Objectives for Machine Translation   Unsupervised cross-lingual pretraining has achieved strong results in neural
machine translation (NMT), by drastically reducing the need for large parallel
data. Most approaches adapt masked-language modeling (MLM) to
sequence-to-sequence architectures, by masking parts of the input and
reconstructing them in the decoder. In this work, we systematically compare
masking with alternative objectives that produce inputs resembling real (full)
sentences, by reordering and replacing words based on their context. We
pretrain models with different methods on English$\leftrightarrow$German,
English$\leftrightarrow$Nepali and English$\leftrightarrow$Sinhala monolingual
data, and evaluate them on NMT. In (semi-) supervised NMT, varying the
pretraining objective leads to surprisingly small differences in the finetuned
performance, whereas unsupervised NMT is much more sensitive to it. To
understand these results, we thoroughly study the pretrained models using a
series of probes and verify that they encode and use information in different
ways. We conclude that finetuning on parallel data is mostly sensitive to few
properties that are shared by most models, such as a strong decoder, in
contrast to unsupervised NMT that also requires models with strong
cross-lingual abilities.

30
The categories of this paper are: q-alg
31
The categories of this paper are: ao-sci
9
The categories of this paper are: nlin
19
The categories of this paper are: eess
27
The categories of this paper are: comp-gas
15
The categories of this paper are: nucl-ex
14
The categories of this paper are: math-ph
26
The categories of this paper are: bayes-an
2
The categories of this paper are: math
35
The categories of this paper are: mtrl-th
4
The categories of this paper are: cond-mat
34
The categories of this paper are: chem-ph
12
The categories of this paper are: nucl-th
10
The categories of this paper are: q-bio
23
The categories of this paper are: solv-int
32
The categories of this paper are: atom-ph