Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Controlling the Quality of Distillation in Response-Based Network
  Compression   The performance of a distillation-based compressed network is governed by the
quality of distillation. The reason for the suboptimal distillation of a large
network (teacher) to a smaller network (student) is largely attributed to the
gap in the learning capacities of given teacher-student pair. While it is hard
to distill all the knowledge of a teacher, the quality of distillation can be
controlled to a large extent to achieve better performance. Our experiments
show that the quality of distillation is largely governed by the quality of
teacher's response, which in turn is heavily affected by the presence of
similarity information in its response. A well-trained large capacity teacher
loses similarity information between classes in the process of learning
fine-grained discriminative properties for classification. The absence of
similarity information causes the distillation process to be reduced from one
example-many class learning to one example-one class learning, thereby
throttling the flow of diverse knowledge from the teacher. With the implicit
assumption that only the instilled knowledge can be distilled, instead of
focusing only on the knowledge distilling process, we scrutinize the knowledge
inculcation process. We argue that for a given teacher-student pair, the
quality of distillation can be improved by finding the sweet spot between batch
size and number of epochs while training the teacher. We discuss the steps to
find this sweet spot for better distillation. We also propose the distillation
hypothesis to differentiate the behavior of the distillation process between
knowledge distillation and regularization effect. We conduct all our
experiments on three different datasets.

21
The categories of this paper are: adap-org
15
The categories of this paper are: nucl-ex
8
The categories of this paper are: hep-ex
32
The categories of this paper are: atom-ph
29
The categories of this paper are: funct-an
24
The categories of this paper are: dg-ga
22
The categories of this paper are: chao-dyn
10
The categories of this paper are: q-bio
30
The categories of this paper are: q-alg
35
The categories of this paper are: mtrl-th
37
The categories of this paper are: supr-con
17
The categories of this paper are: q-fin
0
The categories of this paper are: hep-ph
14
The categories of this paper are: math-ph
7
The categories of this paper are: hep-th
5
The categories of this paper are: gr-qc