Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Pyramid-BERT: Reducing Complexity via Successive Core-set based Token
  Selection   Transformer-based language models such as BERT have achieved the
state-of-the-art performance on various NLP tasks, but are computationally
prohibitive. A recent line of works use various heuristics to successively
shorten sequence length while transforming tokens through encoders, in tasks
such as classification and ranking that require a single token embedding for
prediction. We present a novel solution to this problem, called Pyramid-BERT
where we replace previously used heuristics with a {\em core-set} based token
selection method justified by theoretical results. The core-set based token
selection technique allows us to avoid expensive pre-training, gives a
space-efficient fine tuning, and thus makes it suitable to handle longer
sequence lengths. We provide extensive experiments establishing advantages of
pyramid BERT over several baselines and existing works on the GLUE benchmarks
and Long Range Arena datasets.

10
The categories of this paper are: q-bio
36
The categories of this paper are: cmp-lg
19
The categories of this paper are: eess
37
The categories of this paper are: supr-con
14
The categories of this paper are: math-ph
34
The categories of this paper are: chem-ph
28
The categories of this paper are: alg-geom
16
The categories of this paper are: stat
20
The categories of this paper are: acc-phys
33
The categories of this paper are: plasm-ph
9
The categories of this paper are: nlin
32
The categories of this paper are: atom-ph
30
The categories of this paper are: q-alg
18
The categories of this paper are: econ
22
The categories of this paper are: chao-dyn
35
The categories of this paper are: mtrl-th