Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Pre-training Universal Language Representation   Despite the well-developed cut-edge representation learning for language,
most language representation models usually focus on specific levels of
linguistic units. This work introduces universal language representation
learning, i.e., embeddings of different levels of linguistic units or text with
quite diverse lengths in a uniform vector space. We propose the training
objective MiSAD that utilizes meaningful n-grams extracted from large unlabeled
corpus by a simple but effective algorithm for pre-trained language models.
Then we empirically verify that well designed pre-training scheme may
effectively yield universal language representation, which will bring great
convenience when handling multiple layers of linguistic objects in a unified
way. Especially, our model achieves the highest accuracy on analogy tasks in
different language levels and significantly improves the performance on
downstream tasks in the GLUE benchmark and a question answering dataset.

30
The categories of this paper are: q-alg
16
The categories of this paper are: stat
3
The categories of this paper are: physics
13
The categories of this paper are: quant-ph
2
The categories of this paper are: math
19
The categories of this paper are: eess
10
The categories of this paper are: q-bio
33
The categories of this paper are: plasm-ph
27
The categories of this paper are: comp-gas
8
The categories of this paper are: hep-ex
6
The categories of this paper are: astro-ph
28
The categories of this paper are: alg-geom
21
The categories of this paper are: adap-org
25
The categories of this paper are: patt-sol
15
The categories of this paper are: nucl-ex
23
The categories of this paper are: solv-int