Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: KALA: Knowledge-Augmented Language Model Adaptation   Pre-trained language models (PLMs) have achieved remarkable success on
various natural language understanding tasks. Simple fine-tuning of PLMs, on
the other hand, might be suboptimal for domain-specific tasks because they
cannot possibly cover knowledge from all domains. While adaptive pre-training
of PLMs can help them obtain domain-specific knowledge, it requires a large
training cost. Moreover, adaptive pre-training can harm the PLM's performance
on the downstream task by causing catastrophic forgetting of its general
knowledge. To overcome such limitations of adaptive pre-training for PLM
adaption, we propose a novel domain adaption framework for PLMs coined as
Knowledge-Augmented Language model Adaptation (KALA), which modulates the
intermediate hidden representations of PLMs with domain knowledge, consisting
of entities and their relational facts. We validate the performance of our KALA
on question answering and named entity recognition tasks on multiple datasets
across various domains. The results show that, despite being computationally
efficient, our KALA largely outperforms adaptive pre-training. Code is
available at: https://github.com/Nardien/KALA/.

14
The categories of this paper are: math-ph
12
The categories of this paper are: nucl-th
31
The categories of this paper are: ao-sci
16
The categories of this paper are: stat
29
The categories of this paper are: funct-an
27
The categories of this paper are: comp-gas
5
The categories of this paper are: gr-qc
9
The categories of this paper are: nlin
30
The categories of this paper are: q-alg
19
The categories of this paper are: eess
32
The categories of this paper are: atom-ph
26
The categories of this paper are: bayes-an
2
The categories of this paper are: math
4
The categories of this paper are: cond-mat
10
The categories of this paper are: q-bio
22
The categories of this paper are: chao-dyn