Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Show, Attend and Distill:Knowledge Distillation via Attention-based
  Feature Matching   Knowledge distillation extracts general knowledge from a pre-trained teacher
network and provides guidance to a target student network. Most studies
manually tie intermediate features of the teacher and student, and transfer
knowledge through pre-defined links. However, manual selection often constructs
ineffective links that limit the improvement from the distillation. There has
been an attempt to address the problem, but it is still challenging to identify
effective links under practical scenarios. In this paper, we introduce an
effective and efficient feature distillation method utilizing all the feature
levels of the teacher without manually selecting the links. Specifically, our
method utilizes an attention-based meta-network that learns relative
similarities between features, and applies identified similarities to control
distillation intensities of all possible pairs. As a result, our method
determines competent links more efficiently than the previous approach and
provides better performance on model compression and transfer learning tasks.
Further qualitative analyses and ablative studies describe how our method
contributes to better distillation. The implementation code is available at
github.com/clovaai/attention-feature-distillation.

18
The categories of this paper are: econ
2
The categories of this paper are: math
26
The categories of this paper are: bayes-an
31
The categories of this paper are: ao-sci
13
The categories of this paper are: quant-ph
25
The categories of this paper are: patt-sol
14
The categories of this paper are: math-ph
7
The categories of this paper are: hep-th
37
The categories of this paper are: supr-con
17
The categories of this paper are: q-fin
32
The categories of this paper are: atom-ph
36
The categories of this paper are: cmp-lg
10
The categories of this paper are: q-bio
5
The categories of this paper are: gr-qc
8
The categories of this paper are: hep-ex
19
The categories of this paper are: eess