Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language
  Generation   A well-known limitation in pretrain-finetune paradigm lies in its
inflexibility caused by the one-size-fits-all vocabulary. This potentially
weakens the effect when applying pretrained models into natural language
generation (NLG) tasks, especially for the subword distributions between
upstream and downstream tasks with significant discrepancy. Towards approaching
this problem, we extend the vanilla pretrain-finetune pipeline with an extra
embedding transfer step. Specifically, a plug-and-play embedding generator is
introduced to produce the representation of any input token, according to
pre-trained embeddings of its morphologically similar ones. Thus, embeddings of
mismatch tokens in downstream tasks can also be efficiently initialized. We
conduct experiments on a variety of NLG tasks under the pretrain-finetune
fashion. Experimental results and extensive analyses show that the proposed
strategy offers us opportunities to feel free to transfer the vocabulary,
leading to more efficient and better performed downstream NLG models.

36
The categories of this paper are: cmp-lg
21
The categories of this paper are: adap-org
16
The categories of this paper are: stat
32
The categories of this paper are: atom-ph
13
The categories of this paper are: quant-ph
23
The categories of this paper are: solv-int
19
The categories of this paper are: eess
24
The categories of this paper are: dg-ga
0
The categories of this paper are: hep-ph
26
The categories of this paper are: bayes-an
2
The categories of this paper are: math
25
The categories of this paper are: patt-sol
6
The categories of this paper are: astro-ph
14
The categories of this paper are: math-ph
33
The categories of this paper are: plasm-ph
12
The categories of this paper are: nucl-th