Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Decoupling the Role of Data, Attention, and Losses in Multimodal
  Transformers   Recently multimodal transformer models have gained popularity because their
performance on language and vision tasks suggest they learn rich
visual-linguistic representations. Focusing on zero-shot image retrieval tasks,
we study three important factors which can impact the quality of learned
representations: pretraining data, the attention mechanism, and loss functions.
By pretraining models on six datasets, we observe that dataset noise and
language similarity to our downstream task are important indicators of model
performance. Through architectural analysis, we learn that models with a
multimodal attention mechanism can outperform deeper models with modality
specific attention mechanisms. Finally, we show that successful contrastive
losses used in the self-supervised learning literature do not yield similar
performance gains when used in multimodal transformers

11
The categories of this paper are: hep-lat
6
The categories of this paper are: astro-ph
36
The categories of this paper are: cmp-lg
14
The categories of this paper are: math-ph
29
The categories of this paper are: funct-an
27
The categories of this paper are: comp-gas
32
The categories of this paper are: atom-ph
13
The categories of this paper are: quant-ph
25
The categories of this paper are: patt-sol
19
The categories of this paper are: eess
31
The categories of this paper are: ao-sci
7
The categories of this paper are: hep-th
4
The categories of this paper are: cond-mat
28
The categories of this paper are: alg-geom
24
The categories of this paper are: dg-ga
3
The categories of this paper are: physics