Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: MergeDistill: Merging Pre-trained Language Models using Distillation   Pre-trained multilingual language models (LMs) have achieved state-of-the-art
results in cross-lingual transfer, but they often lead to an inequitable
representation of languages due to limited capacity, skewed pre-training data,
and sub-optimal vocabularies. This has prompted the creation of an ever-growing
pre-trained model universe, where each model is trained on large amounts of
language or domain specific data with a carefully curated, linguistically
informed vocabulary. However, doing so brings us back full circle and prevents
one from leveraging the benefits of multilinguality. To address the gaps at
both ends of the spectrum, we propose MergeDistill, a framework to merge
pre-trained LMs in a way that can best leverage their assets with minimal
dependencies, using task-agnostic knowledge distillation. We demonstrate the
applicability of our framework in a practical setting by leveraging
pre-existing teacher LMs and training student LMs that perform competitively
with or even outperform teacher LMs trained on several orders of magnitude more
data and with a fixed model capacity. We also highlight the importance of
teacher selection and its impact on student model performance.

32
The categories of this paper are: atom-ph
7
The categories of this paper are: hep-th
9
The categories of this paper are: nlin
3
The categories of this paper are: physics
11
The categories of this paper are: hep-lat
34
The categories of this paper are: chem-ph
5
The categories of this paper are: gr-qc
17
The categories of this paper are: q-fin
19
The categories of this paper are: eess
28
The categories of this paper are: alg-geom
2
The categories of this paper are: math
25
The categories of this paper are: patt-sol
6
The categories of this paper are: astro-ph
23
The categories of this paper are: solv-int
29
The categories of this paper are: funct-an
24
The categories of this paper are: dg-ga