Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: ElitePLM: An Empirical Study on General Language Ability Evaluation of
  Pretrained Language Models   Nowadays, pretrained language models (PLMs) have dominated the majority of
NLP tasks. While, little research has been conducted on systematically
evaluating the language abilities of PLMs. In this paper, we present a
large-scale empirical study on general language ability evaluation of PLMs
(ElitePLM). In our study, we design four evaluation dimensions, i.e. memory,
comprehension, reasoning, and composition, to measure ten widely-used PLMs
within five categories. Our empirical results demonstrate that: (1) PLMs with
varying training objectives and strategies are good at different ability tests;
(2) fine-tuning PLMs in downstream tasks is usually sensitive to the data size
and distribution; (3) PLMs have excellent transferability between similar
tasks. Moreover, the prediction results of PLMs in our experiments are released
as an open resource for more deep and detailed analysis on the language
abilities of PLMs. This paper can guide the future work to select, apply, and
design PLMs for specific tasks. We have made all the details of experiments
publicly available at https://github.com/RUCAIBox/ElitePLM.

16
The categories of this paper are: stat
32
The categories of this paper are: atom-ph
21
The categories of this paper are: adap-org
8
The categories of this paper are: hep-ex
11
The categories of this paper are: hep-lat
31
The categories of this paper are: ao-sci
6
The categories of this paper are: astro-ph
27
The categories of this paper are: comp-gas
13
The categories of this paper are: quant-ph
18
The categories of this paper are: econ
7
The categories of this paper are: hep-th
29
The categories of this paper are: funct-an
3
The categories of this paper are: physics
5
The categories of this paper are: gr-qc
2
The categories of this paper are: math
22
The categories of this paper are: chao-dyn