Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Towards Understanding Knowledge Distillation   Knowledge distillation, i.e., one classifier being trained on the outputs of
another classifier, is an empirically very successful technique for knowledge
transfer between classifiers. It has even been observed that classifiers learn
much faster and more reliably if trained with the outputs of another classifier
as soft labels, instead of from ground truth data. So far, however, there is no
satisfactory theoretical explanation of this phenomenon. In this work, we
provide the first insights into the working mechanisms of distillation by
studying the special case of linear and deep linear classifiers. Specifically,
we prove a generalization bound that establishes fast convergence of the
expected risk of a distillation-trained linear classifier. From the bound and
its proof we extract three key factors that determine the success of
distillation: * data geometry -- geometric properties of the data distribution,
in particular class separation, has a direct influence on the convergence speed
of the risk; * optimization bias -- gradient descent optimization finds a very
favorable minimum of the distillation objective; and * strong monotonicity --
the expected risk of the student classifier always decreases when the size of
the training set grows.

35
The categories of this paper are: mtrl-th
2
The categories of this paper are: math
5
The categories of this paper are: gr-qc
9
The categories of this paper are: nlin
10
The categories of this paper are: q-bio
31
The categories of this paper are: ao-sci
33
The categories of this paper are: plasm-ph
20
The categories of this paper are: acc-phys
21
The categories of this paper are: adap-org
34
The categories of this paper are: chem-ph
13
The categories of this paper are: quant-ph
7
The categories of this paper are: hep-th
32
The categories of this paper are: atom-ph
30
The categories of this paper are: q-alg
17
The categories of this paper are: q-fin
6
The categories of this paper are: astro-ph