Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: RePre: Improving Self-Supervised Vision Transformer with Reconstructive
  Pre-training   Recently, self-supervised vision transformers have attracted unprecedented
attention for their impressive representation learning ability. However, the
dominant method, contrastive learning, mainly relies on an instance
discrimination pretext task, which learns a global understanding of the image.
This paper incorporates local feature learning into self-supervised vision
transformers via Reconstructive Pre-training (RePre). Our RePre extends
contrastive frameworks by adding a branch for reconstructing raw image pixels
in parallel with the existing contrastive objective. RePre is equipped with a
lightweight convolution-based decoder that fuses the multi-hierarchy features
from the transformer encoder. The multi-hierarchy features provide rich
supervisions from low to high semantic information, which are crucial for our
RePre. Our RePre brings decent improvements on various contrastive frameworks
with different vision transformer architectures. Transfer performance in
downstream tasks outperforms supervised pre-training and state-of-the-art
(SOTA) self-supervised counterparts.

7
The categories of this paper are: hep-th
2
The categories of this paper are: math
6
The categories of this paper are: astro-ph
5
The categories of this paper are: gr-qc
35
The categories of this paper are: mtrl-th
21
The categories of this paper are: adap-org
34
The categories of this paper are: chem-ph
30
The categories of this paper are: q-alg
12
The categories of this paper are: nucl-th
22
The categories of this paper are: chao-dyn
0
The categories of this paper are: hep-ph
32
The categories of this paper are: atom-ph
15
The categories of this paper are: nucl-ex
29
The categories of this paper are: funct-an
17
The categories of this paper are: q-fin
14
The categories of this paper are: math-ph