Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Explicit Cross-lingual Pre-training for Unsupervised Machine Translation   Pre-training has proven to be effective in unsupervised machine translation
due to its ability to model deep context information in cross-lingual
scenarios. However, the cross-lingual information obtained from shared BPE
spaces is inexplicit and limited. In this paper, we propose a novel
cross-lingual pre-training method for unsupervised machine translation by
incorporating explicit cross-lingual training signals. Specifically, we first
calculate cross-lingual n-gram embeddings and infer an n-gram translation table
from them. With those n-gram translation pairs, we propose a new pre-training
model called Cross-lingual Masked Language Model (CMLM), which randomly chooses
source n-grams in the input text stream and predicts their translation
candidates at each time step. Experiments show that our method can incorporate
beneficial cross-lingual information into pre-trained models. Taking
pre-trained CMLM models as the encoder and decoder, we significantly improve
the performance of unsupervised machine translation.

26
The categories of this paper are: bayes-an
20
The categories of this paper are: acc-phys
31
The categories of this paper are: ao-sci
18
The categories of this paper are: econ
34
The categories of this paper are: chem-ph
27
The categories of this paper are: comp-gas
19
The categories of this paper are: eess
15
The categories of this paper are: nucl-ex
32
The categories of this paper are: atom-ph
0
The categories of this paper are: hep-ph
2
The categories of this paper are: math
28
The categories of this paper are: alg-geom
6
The categories of this paper are: astro-ph
21
The categories of this paper are: adap-org
25
The categories of this paper are: patt-sol
9
The categories of this paper are: nlin