Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Improving Question Answering Performance Using Knowledge Distillation
  and Active Learning   Contemporary question answering (QA) systems, including transformer-based
architectures, suffer from increasing computational and model complexity which
render them inefficient for real-world applications with limited resources.
Further, training or even fine-tuning such models requires a vast amount of
labeled data which is often not available for the task at hand. In this
manuscript, we conduct a comprehensive analysis of the mentioned challenges and
introduce suitable countermeasures. We propose a novel knowledge distillation
(KD) approach to reduce the parameter and model complexity of a pre-trained
BERT system and utilize multiple active learning (AL) strategies for immense
reduction in annotation efforts. In particular, we demonstrate that our model
achieves the performance of a 6-layer TinyBERT and DistilBERT, whilst using
only 2% of their total parameters. Finally, by the integration of our AL
approaches into the BERT framework, we show that state-of-the-art results on
the SQuAD dataset can be achieved when we only use 20% of the training data.

32
The categories of this paper are: atom-ph
15
The categories of this paper are: nucl-ex
14
The categories of this paper are: math-ph
21
The categories of this paper are: adap-org
28
The categories of this paper are: alg-geom
23
The categories of this paper are: solv-int
19
The categories of this paper are: eess
10
The categories of this paper are: q-bio
18
The categories of this paper are: econ
27
The categories of this paper are: comp-gas
3
The categories of this paper are: physics
34
The categories of this paper are: chem-ph
37
The categories of this paper are: supr-con
31
The categories of this paper are: ao-sci
11
The categories of this paper are: hep-lat
33
The categories of this paper are: plasm-ph