Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Unsupervised Pretraining for Neural Machine Translation Using Elastic
  Weight Consolidation   This work presents our ongoing research of unsupervised pretraining in neural
machine translation (NMT). In our method, we initialize the weights of the
encoder and decoder with two language models that are trained with monolingual
data and then fine-tune the model on parallel data using Elastic Weight
Consolidation (EWC) to avoid forgetting of the original language modeling
tasks. We compare the regularization by EWC with the previous work that focuses
on regularization by language modeling objectives. The positive result is that
using EWC with the decoder achieves BLEU scores similar to the previous work.
However, the model converges 2-3 times faster and does not require the original
unlabeled training data during the fine-tuning stage. In contrast, the
regularization using EWC is less effective if the original and new tasks are
not closely related. We show that initializing the bidirectional NMT encoder
with a left-to-right language model and forcing the model to remember the
original left-to-right language modeling task limits the learning capacity of
the encoder for the whole bidirectional context.

11
The categories of this paper are: hep-lat
34
The categories of this paper are: chem-ph
32
The categories of this paper are: atom-ph
28
The categories of this paper are: alg-geom
22
The categories of this paper are: chao-dyn
10
The categories of this paper are: q-bio
19
The categories of this paper are: eess
4
The categories of this paper are: cond-mat
27
The categories of this paper are: comp-gas
29
The categories of this paper are: funct-an
16
The categories of this paper are: stat
6
The categories of this paper are: astro-ph
12
The categories of this paper are: nucl-th
36
The categories of this paper are: cmp-lg
25
The categories of this paper are: patt-sol
15
The categories of this paper are: nucl-ex