Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain
  Analysis: From Theory to Practice   Vision Transformer (ViT) has recently demonstrated promise in computer vision
problems. However, unlike Convolutional Neural Networks (CNN), it is known that
the performance of ViT saturates quickly with depth increasing, due to the
observed attention collapse or patch uniformity. Despite a couple of empirical
solutions, a rigorous framework studying on this scalability issue remains
elusive. In this paper, we first establish a rigorous theory framework to
analyze ViT features from the Fourier spectrum domain. We show that the
self-attention mechanism inherently amounts to a low-pass filter, which
indicates when ViT scales up its depth, excessive low-pass filtering will cause
feature maps to only preserve their Direct-Current (DC) component. We then
propose two straightforward yet effective techniques to mitigate the
undesirable low-pass limitation. The first technique, termed AttnScale,
decomposes a self-attention block into low-pass and high-pass components, then
rescales and combines these two filters to produce an all-pass self-attention
matrix. The second technique, termed FeatScale, re-weights feature maps on
separate frequency bands to amplify the high-frequency signals. Both techniques
are efficient and hyperparameter-free, while effectively overcoming relevant
ViT training artifacts such as attention collapse and patch uniformity. By
seamlessly plugging in our techniques to multiple ViT variants, we demonstrate
that they consistently help ViTs benefit from deeper architectures, bringing up
to 1.1% performance gains "for free" (e.g., with little parameter overhead). We
publicly release our codes and pre-trained models at
https://github.com/VITA-Group/ViT-Anti-Oversmoothing.

24
The categories of this paper are: dg-ga
4
The categories of this paper are: cond-mat
8
The categories of this paper are: hep-ex
2
The categories of this paper are: math
17
The categories of this paper are: q-fin
33
The categories of this paper are: plasm-ph
10
The categories of this paper are: q-bio
35
The categories of this paper are: mtrl-th
22
The categories of this paper are: chao-dyn
23
The categories of this paper are: solv-int
19
The categories of this paper are: eess
13
The categories of this paper are: quant-ph
20
The categories of this paper are: acc-phys
37
The categories of this paper are: supr-con
26
The categories of this paper are: bayes-an
16
The categories of this paper are: stat