Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Zero-Shot Knowledge Distillation in Deep Networks   Knowledge distillation deals with the problem of training a smaller model
(Student) from a high capacity source model (Teacher) so as to retain most of
its performance. Existing approaches use either the training data or meta-data
extracted from it in order to train the Student. However, accessing the dataset
on which the Teacher has been trained may not always be feasible if the dataset
is very large or it poses privacy or safety concerns (e.g., bio-metric or
medical data). Hence, in this paper, we propose a novel data-free method to
train the Student from the Teacher. Without even using any meta-data, we
synthesize the Data Impressions from the complex Teacher model and utilize
these as surrogates for the original training data samples to transfer its
learning to Student via knowledge distillation. We, therefore, dub our method
"Zero-Shot Knowledge Distillation" and demonstrate that our framework results
in competitive generalization performance as achieved by distillation using the
actual training data samples on multiple benchmark datasets.

10
The categories of this paper are: q-bio
28
The categories of this paper are: alg-geom
13
The categories of this paper are: quant-ph
19
The categories of this paper are: eess
23
The categories of this paper are: solv-int
24
The categories of this paper are: dg-ga
35
The categories of this paper are: mtrl-th
8
The categories of this paper are: hep-ex
14
The categories of this paper are: math-ph
26
The categories of this paper are: bayes-an
11
The categories of this paper are: hep-lat
3
The categories of this paper are: physics
5
The categories of this paper are: gr-qc
25
The categories of this paper are: patt-sol
32
The categories of this paper are: atom-ph
12
The categories of this paper are: nucl-th