Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Meta Knowledge Distillation   Recent studies pointed out that knowledge distillation (KD) suffers from two
degradation problems, the teacher-student gap and the incompatibility with
strong data augmentations, making it not applicable to training
state-of-the-art models, which are trained with advanced augmentations.
However, we observe that a key factor, i.e., the temperatures in the softmax
functions for generating probabilities of both the teacher and student models,
was mostly overlooked in previous methods. With properly tuned temperatures,
such degradation problems of KD can be much mitigated. However, instead of
relying on a naive grid search, which shows poor transferability, we propose
Meta Knowledge Distillation (MKD) to meta-learn the distillation with learnable
meta temperature parameters. The meta parameters are adaptively adjusted during
training according to the gradients of the learning objective. We validate that
MKD is robust to different dataset scales, different teacher/student
architectures, and different types of data augmentation. With MKD, we achieve
the best performance with popular ViT architectures among compared methods that
use only ImageNet-1K as training data, ranging from tiny to large models. With
ViT-L, we achieve 86.5% with 600 epochs of training, 0.6% better than MAE that
trains for 1,650 epochs.

2
The categories of this paper are: math
10
The categories of this paper are: q-bio
14
The categories of this paper are: math-ph
32
The categories of this paper are: atom-ph
25
The categories of this paper are: patt-sol
9
The categories of this paper are: nlin
5
The categories of this paper are: gr-qc
20
The categories of this paper are: acc-phys
16
The categories of this paper are: stat
27
The categories of this paper are: comp-gas
17
The categories of this paper are: q-fin
35
The categories of this paper are: mtrl-th
6
The categories of this paper are: astro-ph
19
The categories of this paper are: eess
3
The categories of this paper are: physics
23
The categories of this paper are: solv-int