Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: A Short Study on Compressing Decoder-Based Language Models   Pre-trained Language Models (PLMs) have been successful for a wide range of
natural language processing (NLP) tasks. The state-of-the-art of PLMs, however,
are extremely large to be used on edge devices. As a result, the topic of model
compression has attracted increasing attention in the NLP community. Most of
the existing works focus on compressing encoder-based models (tiny-BERT,
distilBERT, distilRoBERTa, etc), however, to the best of our knowledge, the
compression of decoder-based models (such as GPT-2) has not been investigated
much. Our paper aims to fill this gap. Specifically, we explore two directions:
1) we employ current state-of-the-art knowledge distillation techniques to
improve fine-tuning of DistilGPT-2. 2) we pre-train a compressed GPT-2 model
using layer truncation and compare it against the distillation-based method
(DistilGPT2). The training time of our compressed model is significantly less
than DistilGPT-2, but it can achieve better performance when fine-tuned on
downstream tasks. We also demonstrate the impact of data cleaning on model
performance.

23
The categories of this paper are: solv-int
30
The categories of this paper are: q-alg
21
The categories of this paper are: adap-org
29
The categories of this paper are: funct-an
37
The categories of this paper are: supr-con
0
The categories of this paper are: hep-ph
24
The categories of this paper are: dg-ga
14
The categories of this paper are: math-ph
22
The categories of this paper are: chao-dyn
7
The categories of this paper are: hep-th
27
The categories of this paper are: comp-gas
26
The categories of this paper are: bayes-an
16
The categories of this paper are: stat
11
The categories of this paper are: hep-lat
13
The categories of this paper are: quant-ph
5
The categories of this paper are: gr-qc