Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Align and Prompt: Video-and-Language Pre-training with Entity Prompts   Video-and-language pre-training has shown promising improvements on various
downstream tasks. Most previous methods capture cross-modal interactions with a
transformer-based multimodal encoder, not fully addressing the misalignment
between unimodal video and text features. Besides, learning fine-grained
visual-language alignment usually requires off-the-shelf object detectors to
provide object information, which is bottlenecked by the detector's limited
vocabulary and expensive computation cost.
  We propose Align and Prompt: an efficient and effective video-and-language
pre-training framework with better cross-modal alignment. First, we introduce a
video-text contrastive (VTC) loss to align unimodal video-text features at the
instance level, which eases the modeling of cross-modal interactions. Then, we
propose a new visually-grounded pre-training task, prompting entity modeling
(PEM), which aims to learn fine-grained region-entity alignment. To achieve
this, we first introduce an entity prompter module, which is trained with VTC
to produce the similarity between a video crop and text prompts instantiated
with entity names. The PEM task then asks the model to predict the entity
pseudo-labels (i.e~normalized similarity scores) for randomly-selected video
crops. The resulting pre-trained model achieves state-of-the-art performance on
both text-video retrieval and videoQA, outperforming prior work by a
substantial margin. Our code and pre-trained models are available at
https://github.com/salesforce/ALPRO.

35
The categories of this paper are: mtrl-th
32
The categories of this paper are: atom-ph
10
The categories of this paper are: q-bio
3
The categories of this paper are: physics
16
The categories of this paper are: stat
13
The categories of this paper are: quant-ph
20
The categories of this paper are: acc-phys
23
The categories of this paper are: solv-int
24
The categories of this paper are: dg-ga
15
The categories of this paper are: nucl-ex
9
The categories of this paper are: nlin
18
The categories of this paper are: econ
30
The categories of this paper are: q-alg
14
The categories of this paper are: math-ph
17
The categories of this paper are: q-fin
19
The categories of this paper are: eess