Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Model Compression with Multi-Task Knowledge Distillation for Web-scale
  Question Answering System   Deep pre-training and fine-tuning models (like BERT, OpenAI GPT) have
demonstrated excellent results in question answering areas. However, due to the
sheer amount of model parameters, the inference speed of these models is very
slow. How to apply these complex models to real business scenarios becomes a
challenging but practical problem. Previous works often leverage model
compression approaches to resolve this problem. However, these methods usually
induce information loss during the model compression procedure, leading to
incomparable results between compressed model and the original model. To tackle
this challenge, we propose a Multi-task Knowledge Distillation Model (MKDM for
short) for web-scale Question Answering system, by distilling knowledge from
multiple teacher models to a light-weight student model. In this way, more
generalized knowledge can be transferred. The experiment results show that our
method can significantly outperform the baseline methods and even achieve
comparable results with the original teacher models, along with significant
speedup of model inference.

22
The categories of this paper are: chao-dyn
9
The categories of this paper are: nlin
11
The categories of this paper are: hep-lat
33
The categories of this paper are: plasm-ph
13
The categories of this paper are: quant-ph
4
The categories of this paper are: cond-mat
20
The categories of this paper are: acc-phys
27
The categories of this paper are: comp-gas
18
The categories of this paper are: econ
0
The categories of this paper are: hep-ph
32
The categories of this paper are: atom-ph
28
The categories of this paper are: alg-geom
8
The categories of this paper are: hep-ex
17
The categories of this paper are: q-fin
29
The categories of this paper are: funct-an
7
The categories of this paper are: hep-th