Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: XLNet: Generalized Autoregressive Pretraining for Language Understanding   With the capability of modeling bidirectional contexts, denoising
autoencoding based pretraining like BERT achieves better performance than
pretraining approaches based on autoregressive language modeling. However,
relying on corrupting the input with masks, BERT neglects dependency between
the masked positions and suffers from a pretrain-finetune discrepancy. In light
of these pros and cons, we propose XLNet, a generalized autoregressive
pretraining method that (1) enables learning bidirectional contexts by
maximizing the expected likelihood over all permutations of the factorization
order and (2) overcomes the limitations of BERT thanks to its autoregressive
formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the
state-of-the-art autoregressive model, into pretraining. Empirically, under
comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a
large margin, including question answering, natural language inference,
sentiment analysis, and document ranking.

0
The categories of this paper are: hep-ph
32
The categories of this paper are: atom-ph
15
The categories of this paper are: nucl-ex
16
The categories of this paper are: stat
26
The categories of this paper are: bayes-an
14
The categories of this paper are: math-ph
2
The categories of this paper are: math
21
The categories of this paper are: adap-org
22
The categories of this paper are: chao-dyn
35
The categories of this paper are: mtrl-th
34
The categories of this paper are: chem-ph
31
The categories of this paper are: ao-sci
11
The categories of this paper are: hep-lat
13
The categories of this paper are: quant-ph
30
The categories of this paper are: q-alg
12
The categories of this paper are: nucl-th