Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: To Tune or Not To Tune? How About the Best of Both Worlds?   The introduction of pre-trained language models has revolutionized natural
language research communities. However, researchers still know relatively
little regarding their theoretical and empirical properties. In this regard,
Peters et al. perform several experiments which demonstrate that it is better
to adapt BERT with a light-weight task-specific head, rather than building a
complex one on top of the pre-trained language model, and freeze the parameters
in the said language model. However, there is another option to adopt. In this
paper, we propose a new adaptation method which we first train the task model
with the BERT parameters frozen and then fine-tune the entire model together.
Our experimental results show that our model adaptation method can achieve 4.7%
accuracy improvement in semantic similarity task, 0.99% accuracy improvement in
sequence labeling task and 0.72% accuracy improvement in the text
classification task.

2
The categories of this paper are: math
6
The categories of this paper are: astro-ph
28
The categories of this paper are: alg-geom
4
The categories of this paper are: cond-mat
11
The categories of this paper are: hep-lat
25
The categories of this paper are: patt-sol
16
The categories of this paper are: stat
26
The categories of this paper are: bayes-an
7
The categories of this paper are: hep-th
17
The categories of this paper are: q-fin
14
The categories of this paper are: math-ph
29
The categories of this paper are: funct-an
24
The categories of this paper are: dg-ga
30
The categories of this paper are: q-alg
5
The categories of this paper are: gr-qc
13
The categories of this paper are: quant-ph