Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Is An Image Worth Five Sentences? A New Look into Semantics for
  Image-Text Matching   The task of image-text matching aims to map representations from different
modalities into a common joint visual-textual embedding. However, the most
widely used datasets for this task, MSCOCO and Flickr30K, are actually image
captioning datasets that offer a very limited set of relationships between
images and sentences in their ground-truth annotations. This limited ground
truth information forces us to use evaluation metrics based on binary
relevance: given a sentence query we consider only one image as relevant.
However, many other relevant images or captions may be present in the dataset.
In this work, we propose two metrics that evaluate the degree of semantic
relevance of retrieved items, independently of their annotated binary
relevance. Additionally, we incorporate a novel strategy that uses an image
captioning metric, CIDEr, to define a Semantic Adaptive Margin (SAM) to be
optimized in a standard triplet loss. By incorporating our formulation to
existing models, a \emph{large} improvement is obtained in scenarios where
available training data is limited. We also demonstrate that the performance on
the annotated image-caption pairs is maintained while improving on other
non-annotated relevant items when employing the full training set. Code with
our metrics and adaptive margin formulation will be made public.

10
The categories of this paper are: q-bio
30
The categories of this paper are: q-alg
16
The categories of this paper are: stat
21
The categories of this paper are: adap-org
23
The categories of this paper are: solv-int
6
The categories of this paper are: astro-ph
15
The categories of this paper are: nucl-ex
36
The categories of this paper are: cmp-lg
13
The categories of this paper are: quant-ph
7
The categories of this paper are: hep-th
4
The categories of this paper are: cond-mat
34
The categories of this paper are: chem-ph
17
The categories of this paper are: q-fin
9
The categories of this paper are: nlin
18
The categories of this paper are: econ
2
The categories of this paper are: math