Andrej Karpathy Li Fei-Fei
Department of Computer Science, Stanford University
_{_ karpathy,feifeili _}_ @cs.stanford.edu

Deep Visual-Semantic Alignments for Generating Image Descriptions


_We present a model that generates natural language de-_
_scriptions of images and their regions. Our approach lever-_
_ages datasets of images and their sentence descriptions to_
_learn about the inter-modal correspondences between lan-_
_guage and visual data. Our alignment model is based on a_
_novel combination of Convolutional Neural Networks over_
_image regions, bidirectional Recurrent Neural Networks_
_over sentences, and a structured objective that aligns the_
_two modalities through a multimodal embedding. We then_
_describe a Multimodal Recurrent Neural Network architec-_
_ture that uses the inferred alignments to learn to generate_
_novel descriptions of image regions. We demonstrate that_
_our alignment model produces state of the art results in re-_
_trieval experiments on Flickr8K, Flickr30K and MSCOCO_
_datasets. We then show that the generated descriptions sig-_
_nificantly outperform retrieval baselines on both full images_
_and on a new dataset of region-level annotations._


A quick glance at an image is sufficient for a human to
point out and describe an immense amount of details about
the visual scene [14]. However, this remarkable ability has
proven to be an elusive task for our visual recognition models. The majority of previous work in visual recognition
has focused on labeling images with a fixed set of visual
categories and great progress has been achieved in these endeavors [45, 11]. However, while closed vocabularies of visual concepts constitute a convenient modeling assumption,
they are vastly restrictive when compared to the enormous
amount of rich descriptions that a human can compose.
Some pioneering approaches that address the challenge of
generating image descriptions have been developed [29,
13]. However, these models often rely on hard-coded visual
concepts and sentence templates, which imposes limits on
their variety. Moreover, the focus of these works has been
on reducing complex visual scenes into a single sentence,
which we consider to be an unnecessary restriction.
In this work, we strive to take a step towards the goal of
Figure 1. Motivation/Concept Figure: Our model treats language
as a rich label space and generates descriptions of image regions.
generating dense descriptions of images (Figure 1). The
primary challenge towards this goal is in the design of a
model that is rich enough to simultaneously reason about
contents of images and their representation in the domain
of natural language. Additionally, the model should be free
of assumptions about specific hard-coded templates, rules
or categories and instead rely on learning from the training
data. The second, practical challenge is that datasets of image captions are available in large quantities on the internet
[21, 58, 37], but these descriptions multiplex mentions of
several entities whose locations in the images are unknown.
Our core insight is that we can leverage these large imagesentence datasets by treating the sentences as weak labels,
in which contiguous segments of words correspond to some
particular, but unknown location in the image. Our approach is to infer these alignments and use them to learn
a generative model of descriptions. Concretely, our contributions are twofold:
_•_ We develop a deep neural network model that infers the latent alignment between segments of sentences and the region of the image that they describe.
Our model associates the two modalities through a
common, multimodal embedding space and a structured objective. We validate the effectiveness of this
approach on image-sentence retrieval experiments in
which we surpass the state-of-the-art.
_•_ We introduce a multimodal Recurrent Neural Network
architecture that takes an input image and generates
its description in text. Our experiments show that the
generated sentences significantly outperform retrievalbased baselines, and produce sensible qualitative predictions. We then train the model on the inferred correspondences and evaluate its performance on a new
dataset of region-level annotations.
We make code, data and annotations publicly available. [1]


**Dense image annotations.** Our work shares the high-level
goal of densely annotating the contents of images with
many works before us. Barnard et al. [2] and Socher et
al. [48] studied the multimodal correspondence between
words and images to annotate segments of images. Several works [34, 18, 15, 33] studied the problem of holistic
scene understanding in which the scene type, objects and
their spatial support in the image is inferred. However, the
focus of these works is on correctly labeling scenes, objects
and regions with a fixed set of categories, while our focus is
on richer and higher-level descriptions of regions.
**Generating descriptions.** The task of describing images
with sentences has also been explored. A number of approaches pose the task as a retrieval problem, where the
most compatible annotation in the training set is transferred
to a test image [21, 49, 13, 43, 23], or where training annotations are broken up and stitched together [30, 35, 31].
Several approaches generate image captions based on fixed
templates that are filled based on the content of the image
[19, 29, 13, 55, 56, 9, 1] or generative grammars [42, 57],
but this approach limits the variety of possible outputs.
Most closely related to us, Kiros et al. [26] developed a logbilinear model that can generate full sentence descriptions
for images, but their model uses a fixed window context
while our Recurrent Neural Network (RNN) model conditions the probability distribution over the next word in a sentence on all previously generated words. Multiple closely
related preprints appeared on Arxiv during the submission
of this work, some of which also use RNNs to generate image descriptions [38, 54, 8, 25, 12, 5]. Our RNN is simpler
than most of these approaches but also suffers in performance. We quantify this comparison in our experiments.
**Grounding natural language in images.** A number of approaches have been developed for grounding text in the vi
1cs.stanford.edu/people/karpathy/deepimagesent
sual domain [27, 39, 60, 36]. Our approach is inspired by
Frome et al. [16] who associate words and images through
a semantic embedding. More closely related is the work of
Karpathy et al. [24], who decompose images and sentences
into fragments and infer their inter-modal alignment using a
ranking objective. In contrast to their model which is based
on grounding dependency tree relations, our model aligns
contiguous segments of sentences which are more meaningful, interpretable, and not fixed in length.
**Neural networks in visual and language domains.** Multiple approaches have been developed for representing images and words in higher-level representations. On the image side, Convolutional Neural Networks (CNNs) [32, 28]
have recently emerged as a powerful class of models for
image classification and object detection [45]. On the sentence side, our work takes advantage of pretrained word
vectors [41, 22, 3] to obtain low-dimensional representations of words. Finally, Recurrent Neural Networks have
been previously used in language modeling [40, 50], but we
additionally condition these models on images.


**Overview** . The ultimate goal of our model is to generate
descriptions of image regions. During training, the input
to our model is a set of images and their corresponding
sentence descriptions (Figure 2). We first present a model
that aligns sentence snippets to the visual regions that they
describe through a multimodal embedding. We then treat
these correspondences as training data for a second, multimodal Recurrent Neural Network model that learns to generate the snippets.


Our alignment model assumes an input dataset of images
and their sentence descriptions. Our key insight is that sentences written by people make frequent references to some
particular, but unknown location in the image. For example, in Figure 2, the words _“Tabby cat is leaning”_ refer to
the cat, the words _“wooden table”_ refer to the table, etc.
We would like to infer these latent correspondences, with
the eventual goal of later learning to generate these snippets
from image regions. We build on the approach of Karpathy
et al. [24], who learn to ground dependency tree relations
to image regions with a ranking objective. Our contribution is in the use of bidirectional recurrent neural network
to compute word representations in the sentence, dispensing of the need to compute dependency trees and allowing
unbounded interactions of words and their context in the
sentence. We also substantially simplify their objective and
show that both modifications improve ranking performance.
We first describe neural networks that map words and image
regions into a common, multimodal embedding. Then we
introduce our novel objective, which learns the embedding
Figure 2. Overview of our approach. A dataset of images and their sentence descriptions is the input to our model (left). Our model first
infers the correspondences (middle, Section 3.1) and then learns to generate novel descriptions (right, Section 3.2).
representations so that semantically similar concepts across
the two modalities occupy nearby regions of the space.

3.1. Learning to align visual and language data


Following prior work [29, 24], we observe that sentence descriptions make frequent references to objects and their attributes. Thus, we follow the method of Girshick et al. [17]
to detect objects in every image with a Region Convolutional Neural Network (RCNN). The CNN is pre-trained on
ImageNet [6] and finetuned on the 200 classes of the ImageNet Detection Challenge [45]. Following Karpathy et al.
[24], we use the top 19 detected locations in addition to the
whole image and compute the representations based on the
pixels _Ib_ inside each bounding box as follows:
_v_ = _Wm_ [ _CNNθc_ ( _Ib_ )] + _bm,_ (1)
where _CNN_ ( _Ib_ ) transforms the pixels inside bounding box
_Ib_ into 4096-dimensional activations of the fully connected
layer immediately before the classifier. The CNN parameters _θc_ contain approximately 60 million parameters. The
matrix _Wm_ has dimensions _h ×_ 4096, where _h_ is the size
of the multimodal embedding space ( _h_ ranges from 10001600 in our experiments). Every image is thus represented
as a set of _h_ -dimensional vectors _{vi | i_ = 1 _. . ._ 20 _}_ .


To establish the inter-modal relationships, we would like
to represent the words in the sentence in the same _h_ dimensional embedding space that the image regions occupy. The simplest approach might be to project every individual word directly into this embedding. However, this
approach does not consider any ordering and word context
information in the sentence. An extension to this idea is
to use word bigrams, or dependency tree relations as previously proposed [24]. However, this still imposes an arbitrary maximum size of the context window and requires
the use of Dependency Tree Parsers that might be trained on
unrelated text corpora.
To address these concerns, we propose to use a Bidirectional Recurrent Neural Network (BRNN) [46] to compute
the word representations. The BRNN takes a sequence of
_N_ words (encoded in a 1-of-k representation) and transforms each one into an _h_ -dimensional vector. However, the
representation of each word is enriched by a variably-sized
context around that word. Using the index _t_ = 1 _. . . N_ to
denote the position of a word in a sentence, the precise form
of the BRNN is as follows:
_xt_ = _Ww_ I _t_ (2)
_et_ = _f_ ( _Wext_ + _be_ ) (3)
_h_ _[f]_ _t_ [=] _[ f]_ [(] _[e][t]_ [+] _[ W][f]_ _[h][f]_ _t−_ 1 [+] _[ b][f]_ [)] (4)
_h_ _[b]_ _t_ [=] _[ f]_ [(] _[e][t]_ [+] _[ W][b][h][b]_ _t_ +1 [+] _[ b][b]_ [)] (5)
_st_ = _f_ ( _Wd_ ( _h_ _[f]_ _t_ [+] _[ h]_ _t_ _[b]_ [) +] _[ b][d]_ [)] _[.]_ (6)
Here, I _t_ is an indicator column vector that has a single one
at the index of the _t_ -th word in a word vocabulary. The
weights _Ww_ specify a word embedding matrix that we initialize with 300-dimensional word2vec [41] weights and
keep fixed due to overfitting concerns. However, in practice we find little change in final performance when these
vectors are trained, even from random initialization. Note
that the BRNN consists of two independent streams of processing, one moving left to right ( _h_ _[f]_ _t_ [) and the other right to]
left ( _h_ _[b]_ _t_ [) (see Figure][ 3][ for diagram). The final] _[ h]_ [-dimensional]
representation _st_ for the _t_ -th word is a function of both the
word at that location and also its surrounding context in the
sentence. Technically, every _st_ is a function of all words in
the entire sentence, but our empirical finding is that the final
word representations ( _st_ ) align most strongly to the visual
concept of the word at that location ( I _t_ ).
We learn the parameters _We, Wf_ _, Wb, Wd_ and the respective biases _be, bf_ _, bb, bd_ . A typical size of the hidden representation in our experiments ranges between 300-600 dimensions. We set the activation function _f_ to the rectified
linear unit (ReLU), which computes _f_ : _x �→_ _max_ (0 _, x_ ).


We have described the transformations that map every image and sentence into a set of vectors in a common _h_ dimensional space. Since the supervision is at the level of
entire images and sentences, our strategy is to formulate an
image-sentence score as a function of the individual regionword scores. Intuitively, a sentence-image pair should have
a high matching score if its words have a confident support
in the image. The model of Karpathy et a. [24] interprets the
dot product _vi_ _[T]_ _[s][t]_ [ between the] _[ i]_ [-th region and] _[ t]_ [-th word as a]
measure of similarity and use it to define the score between
image _k_ and sentence _l_ as:
_Skl_ =
_t∈gl_
- _max_ (0 _, vi_ _[T]_ _[s][t]_ [)] _[.]_ (7)
_i∈gk_
Here, _gk_ is the set of image fragments in image _k_ and _gl_
is the set of sentence fragments in sentence _l_ . The indices
_k, l_ range over the images and sentences in the training set.
Together with their additional Multiple Instance Learning
objective, this score carries the interpretation that a sentence
fragment aligns to a subset of the image regions whenever
the dot product is positive. We found that the following
reformulation simplifies the model and alleviates the need
for additional objectives and their hyperparameters:
_Skl_ =        - _maxi∈gk_ _vi_ _[T]_ _[s][t][.]_ (8)
_t∈gl_
Here, every word _st_ aligns to the single best image region.
As we show in the experiments, this simplified model also
leads to improvements in the final ranking performance.
Assuming that _k_ = _l_ denotes a corresponding image and
sentence pair, the final max-margin, structured loss remains:
_E_ ( **a** ) =
- _ψj_ _[U]_ [(] _[a][j]_ [) +]
_j_ =1 _...N_ _j_ =1 _...N_
Figure 3. Diagram for evaluating the image-sentence score _Skl_ .
Object regions are embedded with a CNN (left). Words (enriched
by their context) are embedded in the same multimodal space with
a BRNN (right). Pairwise similarities are computed with inner
products (magnitudes shown in grayscale) and finally reduced to
image-sentence score with Equation 8.
alignment to the same region. Concretely, given a sentence
with _N_ words and an image with _M_ bounding boxes, we
introduce the latent alignment variables _aj ∈{_ 1 _. . . M_ _}_ for
_j_ = 1 _. . . N_ and formulate an MRF in a chain structure
along the sentence as follows:
 - _ψj_ _[B]_ [(] _[a][j][, a][j]_ [+1][)] (10)
_j_ =1 _...N_ _−_ 1
��
(9)
_C_ ( _θ_ ) =
_max_ (0 _, Skl −_ _Skk_ + 1)
_l_
_ψj_ _[U]_ [(] _[a][j]_ [=] _[ t]_ [) =] _[ v]_ _i_ _[T]_ _[s][t]_ (11)
_ψj_ _[B]_ [(] _[a][j][, a][j]_ [+1][) =] _[ β]_ [1] [[] _[a][j]_ [=] _[ a][j]_ [+1][]] _[.]_ (12)
Here, _β_ is a hyperparameter that controls the affinity towards longer word phrases. This parameter allows us to
interpolate between single-word alignments ( _β_ = 0) and
aligning the entire sentence to a single, maximally scoring
region when _β_ is large. We minimize the energy to find the
best alignments **a** using dynamic programming. The output
of this process is a set of image regions annotated with segments of text. We now describe an approach for generating
novel phrases based on these correspondences.


In this section we assume an input set of images and their
textual descriptions. These could be full images and their
sentence descriptions, or regions and text snippets, as inferred in the previous section. The key challenge is in the
design of a model that can predict a variable-sized sequence
of outputs given an image. In previously developed language models based on Recurrent Neural Networks (RNNs)
[40, 50, 10], this is achieved by defining a probability distribution of the next word in a sequence given the current word
and context from previous time steps. We explore a simple
_k_
~~�~~ - ~~�~~ rank images
+ - _max_ (0 _, Slk −_ _Skk_ + 1)
_l_
~~�~~ ~~�~~  - ~~�~~
rank sentences
_._
This objective encourages aligned image-sentences pairs to
have a higher score than misaligned pairs, by a margin.


Consider an image from the training set and its corresponding sentence. We can interpret the quantity _vi_ _[T]_ _[s][t]_ [ as the un-]
normalized log probability of the _t_ -th word describing any
of the bounding boxes in the image. However, since we are
ultimately interested in generating snippets of text instead
of single words, we would like to align extended, contiguous sequences of words to a single bounding box. Note that
the na¨ıve solution that assigns each word independently to
the highest-scoring region is insufficient because it leads to
words getting scattered inconsistently to different regions.
To address this issue, we treat the true alignments as latent
variables in a Markov Random Field (MRF) where the binary interactions between neighboring words encourage an
but effective extension that additionally conditions the generative process on the content of an input image. More formally, during training our Multimodal RNN takes the image
pixels _I_ and a sequence of input vectors ( _x_ 1 _, . . ., xT_ ). It
then computes a sequence of hidden states ( _h_ 1 _, . . ., ht_ ) and
a sequence of outputs ( _y_ 1 _, . . ., yt_ ) by iterating the following
recurrence relation for _t_ = 1 to _T_ :
_bv_ = _Whi_ [ _CNNθc_ ( _I_ )] (13)
_ht_ = _f_ ( _Whxxt_ + _Whhht−_ 1 + _bh_ + 1 ( _t_ = 1) _⊙_ _bv_ ) (14)
_yt_ = _softmax_ ( _Wohht_ + _bo_ ) _._ (15)
In the equations above, _Whi, Whx, Whh, Woh, xi_ and _bh, bo_
are learnable parameters, and _CNNθc_ ( _I_ ) is the last layer of
a CNN. The output vector _yt_ holds the (unnormalized) log
probabilities of words in the dictionary and one additional
dimension for a special END token. Note that we provide
the image context vector _bv_ to the RNN only at the first
iteration, which we found to work better than at each time
step. In practice we also found that it can help to also pass
both _bv,_ ( _Whxxt_ ) through the activation function. A typical
size of the hidden layer of the RNN is 512 neurons.
**RNN training.** The RNN is trained to combine a word ( _xt_ ),
the previous context ( _ht−_ 1) to predict the next word ( _yt_ ).
We condition the RNN’s predictions on the image information ( _bv_ ) via bias interactions on the first step. The training
proceeds as follows (refer to Figure 4): We set _h_ 0 = _[⃗]_ 0, _x_ 1 to
a special START vector, and the desired label _y_ 1 as the first
word in the sequence. Analogously, we set _x_ 2 to the word
vector of the first word and expect the network to predict
the second word, etc. Finally, on the last step when _xT_ represents the last word, the target label is set to a special END
token. The cost function is to maximize the log probability
assigned to the target labels (i.e. Softmax classifier).
**RNN at test time.** To predict a sentence, we compute the
image representation _bv_, set _h_ 0 = 0, _x_ 1 to the START vector and compute the distribution over the first word _y_ 1. We
sample a word from the distribution (or pick the argmax),
set its embedding vector as _x_ 2, and repeat this process until
the END token is generated. In practice we found that beam
search (e.g. beam size 7) can improve results.

3.1.4 Decoding text segment alignments to images


We use SGD with mini-batches of 100 image-sentence pairs
and momentum of 0.9 to optimize the alignment model. We
cross-validate the learning rate and the weight decay. We
also use dropout regularization in all layers except in the
recurrent layers [59] and clip gradients elementwise at 5
(important). The generative RNN is more difficult to optimize, party due to the word frequency disparity between
rare words and common words (e.g. ”a” or the END token).
We achieved the best results using RMSprop [52], which is
an adaptive step size method that scales the update of each
weight by a running average of its gradient norm.
Figure 4. Diagram of our multimodal Recurrent Neural Network
generative model. The RNN takes a word, the context from previous time steps and defines a distribution over the next word in the
sentence. The RNN is conditioned on the image information at the
first time step. START and END are special tokens.


**Datasets.** We use the Flickr8K [21], Flickr30K [58] and
MSCOCO [37] datasets in our experiments. These datasets
contain 8,000, 31,000 and 123,000 images respectively
and each is annotated with 5 sentences using Amazon
Mechanical Turk. For Flickr8K and Flickr30K, we use
1,000 images for validation, 1,000 for testing and the rest
for training (consistent with [21, 24]). For MSCOCO we
use 5,000 images for both validation and testing.
**Data Preprocessing.** We convert all sentences to lowercase, discard non-alphanumeric characters. We filter words
to those that occur at least 5 times in the training set,
which results in 2538, 7414, and 8791 words for Flickr8k,
Flickr30K, and MSCOCO datasets respectively.


We first investigate the quality of the inferred text and image
alignments with ranking experiments. We consider a withheld set of images and sentences and retrieve items in one
modality given a query from the other by sorting based on
the image-sentence score _Skl_ (Section 3.1.3). We report the
median rank of the closest ground truth result in the list and
Recall@K, which measures the fraction of times a correct
item was found among the top K results. The result of these
experiments can be found in Table 1, and example retrievals
in Figure 5. We now highlight some of the takeaways.
**Our full model outperforms previous work.** First, our
full model (“Our model: BRNN”) outperforms Socher et
al. [49] who trained with a similar loss but used a single
image representation and a Recursive Neural Network over
the sentence. A similar loss was adopted by Kiros et al.
[25], who use an LSTM [20] to encode sentences. We list
their performance with a CNN that is equivalent in power
(AlexNet [28]) to the one used in this work, though similar to [54] they outperform our model with a more powerful
CNN (VGGNet [47], GoogLeNet [51]). “DeFrag” are the
results reported by Karpathy et al. [24]. Since we use different word vectors, dropout for regularization and different
cross-validation ranges and larger embedding sizes, we reimplemented their loss for a fair comparison (“Our imple
Image Annotation Image Search
**Model** **R@1** **R@5** **R@10** **Med** _r_ **R@1** **R@5** **R@10** **Med** _r_

4.1. Image-Sentence Alignment Evaluation


|SDT-RNN (Socher et al. [49])
Kiros et al. [25]
Mao et al. [38]
Donahue et al. [8]
DeFrag (Karpathy et al. [24])
Our implementation of DeFrag [24]
Our model: DepTree edges
Our model: BRNN|9.6 29.8 41.1 16
14.8 39.2 50.9 10
18.4 40.2 50.9 10
17.5 40.3 50.8 9
14.2 37.7 51.3 10
19.2 44.5 58.0 6.0
20.0 46.6 59.4 5.4
22.2 48.2 61.4 4.8|8.9 29.8 41.1 16
11.8 34.0 46.3 13
12.6 31.2 41.5 16
- - - -
10.2 30.8 44.2 14
12.9 35.4 47.5 10.8
15.0 36.5 48.2 10.4
15.2 37.7 50.5 9.2|
|---|---|---|
|Vinyals et al. [54] (more powerful CNN)|23
-
63
5|17
-
57
8|


Our model: 1K test images 38.4 69.9 80.5 1.0 27.4 60.2 74.8 3.0
Our model: 5K test images 16.5 39.2 52.0 9.0 10.7 29.6 42.2 14.0
Table 1. Image-Sentence ranking experiment results. **R@K** is Recall@K (high is good). **Med** _r_ is the median rank (low is good). In the
results for our models, we take the top 5 validation set models, evaluate each independently on the test set and then report the average
performance. The standard deviations on the recall values range from approximately 0.5 to 1.0.
Figure 5. Example alignments predicted by our model. For every test image above, we retrieve the most compatible test sentence and
visualize the highest-scoring region for each word (before MRF smoothing described in Section 3.1.4) and the associated scores ( _vi_ _[T]_ _[s][t]_ [).]
We hide the alignments of low-scoring words to reduce clutter. We assign each region an arbitrary color.
mentation of DeFrag”). Compared to other work that uses
AlexNets, our full model shows consistent improvement.
**Our simpler cost function improves performance.** We
strive to better understand the source of our performance.
First, we removed the BRNN and used dependency tree relations exactly as described in Karpathy et al. [24] (“Our
model: DepTree edges”). The only difference between this
model and “Our reimplementation of DeFrag” is the new,
simpler cost function introduced in Section 3.1.3. We see
that our formulation shows consistent improvements.
**BRNN outperforms dependency tree relations** . Furthermore, when we replace the dependency tree relations with
the BRNN we observe additional performance improvements. Since the dependency relations were shown to work
better than single words and bigrams [24], this suggests that
the BRNN is taking advantage of contexts longer than two
words. Furthermore, our method does not rely on extracting
a Dependency Tree and instead uses the raw words directly.
**MSCOCO results for future comparisons.** We are not
aware of other published ranking results on MSCOCO.
Therefore, we report results on a subset of 1,000 images
and the full set of 5,000 test images for future comparisons.
Note that the 5000 images numbers are lower since Recall@K is a function of test set size.
**Qualitative.** As can be seen from example groundings in
Figure 5, the model discovers interpretable visual-semantic
correspondences, even for small or relatively rare objects
such as an _“accordion”_ . These would be likely missed by
models that only reason about full images.
**Learned region and word vector magnitudes.** An appealing feature of our model is that it learns to modulate
the magnitude of the region and word embeddings. Due
to their inner product interaction, we observe that representations of visually discriminative words such as _“kayak-_
_ing, pumpkins“_ have embedding vectors with higher magnitudes, which in turn translates to a higher influence on
the image-sentence score. Conversely, stop words such as
_“now, simply, actually, but”_ are mapped near the origin,
which reduces their influence. See more analysis in supplementary material.
|Col1|Flickr8K Flickr30K MSCOCO 2014|Col3|Col4|
|---|---|---|---|
|**Model**|B-1
B-2
B-3
B-4|B-1
B-2
B-3
B-4|B-1
B-2
B-3
B-4
METEOR
CIDEr|
|Nearest Neighbor
Mao et al. [38]
Google NIC [54]
LRCN [8]
MS Research [12]
Chen and Zitnick [5]
Our model|—
—
—
—
58
28
23
—
63
41
27
—
—
—
—
—
—
—
—
—
—
—
—
14.1
57.9
38.3
24.5
16.0|—
—
—
—
55
24
20
—
66.3
42.3
27.7
18.3
58.8
39.1
25.1
16.5
—
—
—
—
—
—
—
12.6
57.3
36.9
24.0
15.7|48.0
28.1
16.6
10.0
15.7
38.3
—
—
—
—
—
—
66.6
46.1
32.9
24.6
—
—
62.8
44.2
30.4
—
—
—
—
—
—
21.1
20.7
—
—
—
—
19.0
20.4
—
62.5
45.0
32.1
23.0
19.5
66.0|
Table 2. Evaluation of full image predictions on 1,000 test images. **B-n** is BLEU score that uses up to n-grams. High is good in all columns.
For future comparisons, our METEOR/CIDEr Flickr8K scores are 16.7/31.8 and the Flickr30K scores are 15.3/24.7.
Figure 6. Example sentences generated by the multimodal RNN for test images. We provide many more examples on our project page.


We now evaluate the ability of our RNN model to describe
images and regions. We first trained our Multimodal RNN
to generate sentences on full images with the goal of verifying that the model is rich enough to support the mapping
from image data to sequences of words. For these full image experiments we use the more powerful VGGNet image
features [47]. We report the BLEU [44], METEOR [7] and
CIDEr [53] scores computed with the coco-caption
code [4] [2] . Each method evaluates a _candidate_ sentence
by measuring how well it matches a set of five _reference_
sentences written by humans.
**Qualitative.** The model generates sensible descriptions of
images (see Figure 6), although we consider the last two
images failure cases. The first prediction _“man in black_
_shirt is playing a guitar”_ does not appear in the training set.
However, there are 20 occurrences of “man in black shirt”
and 60 occurrences of “is paying guitar”, which the model
may have composed to describe the first image. In general,
we find that a relatively large portion of generated sentences
(60% with beam size 7) can be found in the training data.
This fraction decreases with lower beam size; For instance,
with beam size 1 this falls to 25%, but the performance also
deteriorates (e.g. from 0.66 to 0.61 CIDEr).
**Multimodal RNN outperforms retrieval baseline.** Our
first comparison is to a nearest neighbor retrieval baseline.
2https://github.com/tylin/coco-caption
Here, we annotate each test image with a sentence of the
most similar training set image as determined by L2 norm
over VGGNet [47] fc7 features. Table 2 shows that the Multimodal RNN confidently outperforms this retrieval method.
Hence, even with 113,000 train set images in MSCOCO
the retrieval approach is inadequate. Additionally, the RNN
takes only a fraction of a second to evaluate per image.
**Comparison to other work.** Several related models have
been proposed in Arxiv preprints since the original submission of this work. We also include these in Table 2 for comparison. Most similar to our model is Vinyals et al. [54].
Unlike this work where the image information is communicated through a bias term on the first step, they incorporate it as a first word, they use a more powerful but more
complex sequence learner (LSTM [20]), a different CNN
(GoogLeNet [51]), and report results of a model ensemble.
Donahue et al. [8] use a 2-layer factored LSTM (similar
in structure to the RNN in Mao et al. [38]). Both models
appear to work worse than ours, but this is likely in large
part due to their use of the less powerful AlexNet [28] features. Compared to these approaches, our model prioritizes
simplicity and speed at a slight cost in performance.

4.2. Generated Descriptions: Fulframe evaluation


We now train the Multimodal RNN on the correspondences
between image regions and snippets of text, as inferred by
the alignment model. To support the evaluation, we used
Amazon Mechanical Turk (AMT) to collect a new dataset
Figure 7. Example region predictions. We use our region-level multimodal RNN to generate text (shown on the right of each image) for
some of the bounding boxes in each image. The lines are grounded to centers of bounding boxes and the colors are chosen arbitrarily.
of region-level annotations that we only use at test time. The
labeling interface displayed a single image and asked annotators (we used nine per image) to draw five bounding boxes
and annotate each with text. In total, we collected 9,000 text
snippets for 200 images in our MSCOCO test split (i.e. 45
snippets per image). The snippets have an average length of
2.3 words. Example annotations include _“sports car”, “el-_
_derly couple sitting”, “construction site”, “three dogs on_
_leashes”, “chocolate cake”_ . We noticed that asking annotators for grounded text snippets induces language statistics
different from those in full image captions. Our region annotations are more comprehensive and feature elements of
scenes that would rarely be considered salient enough to be
included in a single sentence sentence about the full image,
such as _“heating vent”, “belt buckle”, and “chimney”_ .
**Qualitative** . We show example region model predictions
in Figure 7. To reiterate the difficulty of the task, consider
for example the phrase _“table with wine glasses”_ that is
generated on the image on the right in Figure 7. This phrase
only occurs in the training set 30 times. Each time it may
have a different appearance and each time it may occupy a
few (or none) of our object bounding boxes. To generate
this string for the region, the model had to first correctly
learn to ground the string and then also learn to generate it.

4.3. Generated Descriptions: Region evaluation


**ing baseline** . Similar to the full image description task, we
evaluate this data as a prediction task from a 2D array of
pixels (one image region) to a sequence of words and record
the BLEU score. The ranking baseline retrieves training
sentence substrings most compatible with each region as
judged by the BRNN model. Table 3 shows that the region
RNN model produces descriptions most consistent with our
collected data. Note that the fullframe model was trained
only on full images, so feeding it smaller image regions
deteriorates its performance. However, its sentences are
also longer than the region model sentences, which likely
negatively impacts the BLEU score. The sentence length
is non-trivial to control for with an RNN, but we note that
the region model also outperforms the fullframe model on
all other metrics: CIDEr 61.6/20.3, METEOR 15.8/13.3,
ROUGE 35.1/21.0 for region/fullframe respectively.
|Model|B-1 B-2 B-3 B-4|
|---|---|
|Human agreement|61.5
45.2
30.1
22.0|
|Nearest Neighbor
RNN: Fullframe model
RNN: Region level model|22.9
10.5
0.0
0.0
14.2
6.0
2.2
0.0
**35.2**
**23.0**
**16.1**
**14.8**|
Table 3. BLEU score evaluation of image region annotations.

Region model outperforms full frame model and rank-


Although our results are encouraging, the Multimodal RNN
model is subject to multiple limitations. First, the model can
only generate a description of one input array of pixels at a
fixed resolution. A more sensible approach might be to use
multiple saccades around the image to identify all entities,
their mutual interactions and wider context before generating a description. Additionally, the RNN receives the image
information only through additive bias interactions, which
are known to be less expressive than more complicated multiplicative interactions [50, 20]. Lastly, our approach consists of two separate models. Going directly from an imagesentence dataset to region-level annotations as part of a single model trained end-to-end remains an open problem.


We introduced a model that generates natural language descriptions of image regions based on weak labels in form of
a dataset of images and sentences, and with very few hardcoded assumptions. Our approach features a novel ranking
model that aligned parts of visual and language modalities
through a common, multimodal embedding. We showed
that this model provides state of the art performance on
image-sentence ranking experiments. Second, we described
a Multimodal Recurrent Neural Network architecture that
generates descriptions of visual data. We evaluated its performance on both fullframe and region-level experiments
and showed that in both cases the Multimodal RNN outperforms retrieval baselines.
**Acknowledgements.**
We thank Justin Johnson and Jon Krause for helpful comments and discussions. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the GPUs
used for this research. This research is partially supported
by an ONR MURI grant, and NSF ISS-1115313.


An appealing feature of our alignment model is that it learns
to modulate the importance of words and regions by scaling
the magnitude of their corresponding embedding vectors.
To see this, recall that we compute the image-sentence similarity between image _k_ and sentence _l_ as follows:
_Skl_ =        - _maxi∈gk_ _vi_ _[T]_ _[s][t][.]_ (16)
_t∈gl_
**Disciminative words.** As a result of this formulation,
we observe that representations of visually discriminative
words such as _“kayaking, pumpkins“_ tend to have higher
magnitude in the embedding space, which translates to a
higher influence on the final image-sentence scores due to
the inner product. Conversely, the model learns to map stop
words such as _“now, simply, actually, but”_ near the origin, which reduces their influence. Table 4 show the top
40 words with highest and lowest magnitudes _∥st∥_ .
**Disciminative regions.** Similarly, image regions that contain discriminative entities are assigned vectors of higher
magnitudes by our model. This can be be interpreted as a
measure of visual saliency, since these regions would produced large scores if their textual description was present in
a corresponding sentence. We show the regions with high
magnitudes in Figure 8. Notice the common occurrence of
often described regions such as balls, bikes, helmets.
Figure 8. Flickr30K test set regions with high vector magnitude.
|Magnitude|Word|Magnitude|Word|
|---|---|---|---|
|0.42
0.42
0.43
0.44
0.44
0.45
0.45
0.46
0.47
0.47
0.47
0.47
0.47
0.47
0.48
0.48
0.48
0.48
0.48
0.48
0.48
0.49
0.49
0.50
0.50
0.50
0.50
0.50
0.50
0.51
0.51
0.51
0.51
0.51
0.51
0.51
0.51
0.51
0.52
0.52|now
simply
actually
but
neither
then
still
obviously
that
which
felt
not
might
because
appeared
therefore
been
if
also
only
so
would
yet
be
had
revealed
never
very
without
they
either
could
feel
otherwise
when
already
being
else
just
ones|2.61
2.59
2.59
2.58
2.56
2.54
2.54
2.54
2.52
2.52
2.51
2.51
2.50
2.50
2.50
2.48
2.48
2.48
2.47
2.47
2.46
2.46
2.46
2.46
2.46
2.46
2.46
2.46
2.45
2.43
2.43
2.43
2.42
2.42
2.42
2.42
2.41
2.41
2.40
2.40|kayaking
trampoline
pumpkins
windsurﬁng
wakeboard
acrobatics
sousaphone
skydivers
wakeboarders
skateboard
snowboarder
wakeboarder
skydiving
guitar
snowboard
kitchen
paraglider
ollie
ﬁretruck
gymnastics
waterfalls
motorboat
fryer
skateboarding
dulcimer
waterfall
backﬂips
unicyclist
kayak
costumes
wakeboarding
trike
dancers
cupcakes
tuba
skijoring
ﬁrewood
elevators
cranes
bassoon|
Table 4. This table shows the top magnitudes of vectors ( _∥st∥_ ) for
words in Flickr30K. Since the magnitude of individual words in
our model is also a function of their surrounding context in the
sentence, we report the average magnitude.


**Learned appearance of text snippets** . We can query our
alignment model with a piece of text and retrieve individual
image regions that have the highest score with that snippet. We show examples of such queries in Figure 9 and
Figure 10. Notice that the model is sensitive to compound
words and modifiers. For example, _“red bus”_ and _“yel-_
_low bus”_ give very different results. Similarly, _“bird flying_
_in the sky”_ and _“bird on a tree branch”_ give different results. Additionally, it can be seen that the quality of the
results deteriorates for less frequently occurring concepts,
such as _“roof”_ or _“straw hat”_ . However, we emphasize that
the model learned these visual appearances of text snippets
from raw data of full images and sentences, without any explicit correspondences.
**Additional alignment visualizations** . See additional examples of inferred alignments between image regions and
words in Figure 11. Note that one limitation of our model is
that it does not explicitly handle or support counting. For instance, the last example we show contains the phrase _“three_
_people”_ . These words should align to the three people in
the image, but our model puts the bounding box around two
of the people. In doing so, the model may be taking advantage of the BRNN structure to modify the “people” vector
to preferentially align to regions that contain multiple people. However, this is still unsatisfying because such spurious detections only exist as a result of an error in the RCNN
inference process, which presumably failed to localize the
individual people.
**Web demo** . We have published a web demo that displays
our alignments for all images in the test set [3] .
**Additional Flickr8K experiments** . We omitted ranking
experiment results from our paper due to space constraints,
but these can be found in Table 5
**Counting** . We experimented with losses that perform probabilistic inference in the forward pass that explicitly tried
to localize exactly three distinct people in the image. However, this worked poorly because while the RCNN is good
at finding people, it is not very good at localizing them. For
instance, a single person can easily yield multiple detections
(the head, the torso, or the full body, for example). We were
not able to come up with a simple approach to collapsing
these into a single detection (non-maxim suppression by itself was not sufficient in our experiments). Note that this
ambiguity is partly an artifact of the training data. For example, torsos of people can often be labeled alone if the
body is occluded. We are therefore lead to believe that this
additional modeling step is highly non-trivial and a worthy
subject of future work.
3 http://cs.stanford.edu/people/karpathy/deepimagesent/rankingdemo/


**toolkits.** Before adopting the BRNN approach, we also
tried to use Natural Language Processing toolkits to process
the input sentences into graphs of noun phrases and their binary relations. For instance, in the sentence _“a brown dog is_
_chasing a young child”_, the toolkit would infer that there are
two noun phrases ( _“a brown dog”, “young child”_ ), joined
by a binary relationship of _“chasing”_ . We then developed
a CRF that inferred the grounding of these noun phrases to
the detection bounding boxes in the image with a unary appearance model and a spatial binary model. However, this
endeavor proved fruitless. First, performing CRF-like inference during the forward pass of a Neural Network proved
to be extremely slow. Second, we found that there is surprisingly little information in the relative spatial positions
between bounding boxes. For instance, almost any two
bounding boxes in the image could correspond to the action of _“chasing”_ due to huge amount of possibly camera
views of a scene. Hence, we were unable to extract enough
signal from the binary relations in the coordinate system
of the image and suspect that more complex 3-dimensional
reasoning may be required. Lastly, we found that NLP tools
(when used out of the box) introduce a large amount of mistakes in the extracted parse trees, dependency trees and parts
of speech tags. We tried to fix these with complex rules and
exceptions, but ultimately decided to abandon the idea. We
believe that part of the problem is that these tools are usually
trained on different text corpora (e.g. news articles), so image captions are outside of their domain of competence. In
our experience, adopting the BRNN model instead of this
approach provided immediate performance improvements
and produced significant reductions in code complexity.

Plug and play use of Natural Language Processing


Additional examples of generated captions on the full image level can be found in Figure 12 (and our website). The
model often gets the right gist of the scene, but sometimes
guesses specific fine-grained words incorrectly. We expect
that reasoning not only the global level of the image but also
on the level of objects will significantly improve these results. We find the last example ( _“woman in bikini is jumping_
_over hurdle”_ ) to be especially illuminating. This sentence
does not occur in the training data. Our general qualitative
impression of the model is that it learns certain templates,
e.g. _“<noun>in <noun>is <verb>in <noun>”_, and then
fills these in based on textures in the image. In this particular case, the volleyball net has the visual appearance of a
hurdle, which may have caused the model to insert it as a
noun (along with the woman) into one of its learned sentence templates.

6.3. Additional examples: Image annotation


Additional examples of region annotations can be found
in Figure 13. Note that we annotate regions based on the
content of each image region alone, which can cause erroneous predictions when not enough context is available in
the bounding box (e.g. a generated description that says
“container” detected on the back of a dog’s head in the image on the right, in the second row). We found that one effective way of using the contextual information and improving the predictions is to concatenate the fullframe feature
CNN vector to the vector of the region of interest, giving
8192-dimensional input vector the to RNN. However, we
chose to omit these experiments in our paper to preserve the
simplicity of the mode, and because we believe that cleaner
and more principled approaches to this challenge can be developed.

6.4. Additional examples: Region annotation


There are a few tricks needed to get the Multimodal RNN to
train efficiently. We found that **clipping the gradients** (we
only experimented with simple per-element clipping) at an
appropriate value consistently gave better results and helped
on the validation data. As mentioned in our paper, we experimented with SGD, SGD+Momentum, Adadelta, Adagrad,
but found **RMSProp** to give best results. However, some
SGD checkpoints usually also converged to nearby validation performance vicinity. Moreover, the distribution of the
words in English language are highly non-uniform. Therefore, the model spends the first few iterations mostly learning the biases for the Softmax classifier such that it is predicting every word at random with the appropriate dataset
frequency. We found that we could obtain faster convergence early in the training (and nicer loss curves) by explicitly **initializing the biases** of all words in the dictionary (in
the Softmax classifier) to log probability of their occurrence
in the training data. Therefore, with small weights and biases set appropriately the model right away predicts word
at random according to their chance distribution. After submission of our original paper we performed additional experiments with comparing an RNN to an LSTM and found
that **LSTMs** consistently produced better results, but took
longer to train. Lastly, we initially used word2vec vectors
as our word representations _xi_, but found that it was sufficient to train these vectors from random initialization without changes in the final performance. Moreover, we found
that the word2vec vectors have some unappealing properties
when used in multimodal language-visual tasks. For example, all colors (e.g. red, blue, green) are clustered nearby
in the word2vec representation because they are relatively
interchangeable in most language contexts. However, their
visual instantiations are very different.
“glass of wine”
“yellow bus”
“closeup of zebra”
“sprinkled donut”
“shiny laptop”
Figure 9. Examples of highest scoring regions for queried snippets of text, on 5,000 images of our MSCOCO test set.
“bird flying in the sky”
“bird sitting on roof”
“closeup of fruit”
“man riding a horse”
Figure 10. Examples of highest scoring regions for queried snippets of text, on 5,000 images of our MSCOCO test set.
Image Annotation Image Search
**Model** **R@1** **R@5** **R@10** **Med** _r_ **R@1** **R@5** **R@10** **Med** _r_


DeViSE (Frome et al. [16]) 4.5 18.1 29.2 26 6.7 21.9 32.7 25
SDT-RNN (Socher et al. [49]) 9.6 29.8 41.1 16 8.9 29.8 41.1 16
Kiros et al. [25] 13.5 36.2 45.7 13 10.4 31.0 43.7 14
Mao et al. [38] 14.5 37.2 48.5 11 11.5 31.0 42.4 15
DeFrag (Karpathy et al. [24]) 12.6 32.9 44.0 14 9.7 29.6 42.5 15
Our implementation of DeFrag [24] 13.8 35.8 48.2 10.4 9.5 28.2 40.3 15.6
Our model: DepTree edges 14.8 37.9 50.0 9.4 11.6 31.4 43.8 13.2
Our model: BRNN **16.5** **40.6** **54.2** **7.6** **11.8** **32.1** **44.7** **12.4**
Table 5. Ranking experiment results for the Flickr8K dataset.
Figure 11. Additional examples of alignments. For each query test image above we retrieve the most compatible sentence from the test set
and show the alignments.
Figure 12. Additional examples of captions on the level of full images. Green: Human ground truth. Red: Top-scoring sentence from
training set. Blue: Generated sentence.
Figure 13. Additional examples of region captions on the test set of Flickr30K.


Graduate School of System Design,
Tokyo Metropolitan University, Japan
matsuo-junki@ed.tmu.ac.jp,
komachi@tmu.ac.jp

Word-Alignment-Based Segment-Level Machine Translation Evaluation using
  Word Embeddings

Junki Matsuo
Mamoru Komachi
Katsuhito Sudoh


One of the most important problems in
machine translation (MT) evaluation is to
evaluate the similarity between translation
hypotheses with different surface forms
from the reference, especially at the segment level. We propose to use word
embeddings to perform word alignment
for segment-level MT evaluation. We
performed experiments with three types
of alignment methods using word embeddings. We evaluated our proposed
methods with various translation datasets.
Experimental results show that our proposed methods outperform previous word
embeddings-based methods.


Automatic evaluation of machine translation (MT)
systems without human intervention has gained
importance. For example, BLEU (Papineni et al.,
2002) has improved the MT research in the last
decade. However, BLEU has little correlation
with human judgment on the segment level since
it is originally proposed for system-level evaluation. Segment-level evaluation is crucial for analyzing MT outputs to improve the system accuracy, but there are few studies addressing the issue
of segment-level evaluation of MT outputs.
Another issue in MT evaluation is to evaluate MT hypotheses that are semantically equivalent with different surfaces from the reference.
For instance, BLEU does not consider any words
that do not match the reference at the surface level. METEOR-Universal (Denkowski and
Lavie, 2014) handles word similarities better,
_∗_ The last author is currently affiliated with Nara Institute
of Science and Technology, Japan.


NTT Communication Science
Laboratories, Japan
sudoh@is.naist.jp _[∗]_
but it uses external resources that require timeconsuming annotations. It is also not as simple
as BLEU and its score is difficult to interpret.
DREEM (Chen and Guo, 2015), another metric
that addresses the issue of word similarity, does
not require human annotations and uses distributed
representations for MT evaluation. It shows higher
accuracy than popular metrics such as BLEU and
METEOR.
Therefore, we follow the approach of DREEM
to propose a lightweight MT evaluation measure
that employs only a raw corpus as an external resource. We adopt sentence similarity measures
proposed by Song and Roth (2015) for a Semantic
Textual Similarity (STS) task. They use word embeddings to align words so that the sentence similarity score takes near-synonymous expressions
into account and propose three types of heuristics using m:n (average), 1:n (maximum) and 1:1
(Hungarian) alignments. It has been reported that
sentence similarity calculated with a word alignment based on word embeddings shows high accuracy on STS tasks.
We evaluated the word-alignment-based sentence similarity for MT evaluation to use the
WMT12, WMT13, and WMT15 datasets of
European–English translation and WAT2015 and
NTCIR8 datasets of Japanese–English translation.
Experimental results confirmed that the maximum alignment similarity outperforms previous
word embeddings-based methods in European–
English translation tasks and the average alignment similarity has the highest human correlation
in Japanese–English translation tasks.


Several studies have examined automatic evaluation of MT systems. The de facto standard automatic MT evaluation metrics BLEU
(Papineni et al., 2002) may assign inappropriate score to a translation hypothesis that uses
similar but different words because it considers only word n-gram precision (Callison-Burch
et al., 2006). METEOR-Universal (Denkowski
and Lavie, 2014) alleviates the problem of surface mismatch by using a thesaurus and a stemmer
but it needs external resources, such as WordNet.
In this work, we used a distributed word representation to evaluate semantic relatedness between
the hypothesis and reference sentences. This approach has the advantage that it can be implemented only with only a raw monolingual corpus.
To address the problem of word n-gram precision, Wang and Merlo (2016) propose to smooth
it by word embeddings. They also employ maximum alignment between n-grams of hypothesis and reference sentences and a threshold to
cut off n-gram embeddings with low similarity.
Their work is similar to our maximum alignment
similarity method, but they only experimented
in European–English datasets, where maximum
alignment works better than average alignment.
The previous method most similar to ours is
DREEM (Chen and Guo, 2015). It has shown
to achieve state-of-the-art accuracy compared with
popular metrics such as BLEU and METEOR. It
uses various types of representations such as word
and sentence representations. Word representations are trained with a neural network and sentence representations are trained with a recursive
auto-encoder, respectively. DREEM uses cosine
similarity between distributed representations of
hypothesis and reference as a translation evaluation score. Both their and our methods employ
word embeddings to compute sentence similarity
score, but our method differs in the use of alignment and length penalty. As for alignment, we set
a threshold to remove noisy alignments, whereas
they use a hyper-parameter to down-weight overall sentence similarity. As for length penalty,
we compared average, maximum, and Hungarian
alignments to compensate for the difference between the lengths of translation hypothesis and
reference, whereas they use an exponential penalty
to normalize the length.
Another way to improve the robustness of MT
evaluation is to use a character-based model.
CHRF (Popovi´c, 2015) is one such metric that
uses character n-grams. It is a harmonic mean
of character n-gram precision and recall. It works
MASasym( _a, b_ ) = [1]
_|a|_
well for morphologically rich languages. We, instead, adopt a word-based approach because our
target language, English, is morphologically simple but etymologically complex.


In this section, we introduce word-alignmentbased sentence similarity (Song and Roth, 2015)
applied as an MT evaluation metrics. Song and
Roth (2015) propose to use word embeddings to
align words in a pair of sentences. Their approach
shows promising results in STS tasks.
In MT evaluation, a word in the source language aligns to either a word or a phrase in the target language; therefore, it is not likely for a word
to align with the whole sentence. Thus, we use
several heuristics to constrain word alignment between the hypothesis and reference sentences.
In the following subsections, we present three
sentence similarity measures. All of them use cosine similarity to calculate word similarity. To
avoid alignment between unrelated words, we cut
off word alignment whose similarity is less than a
threshold value.


First, the average alignment similarity (AAS)
heuristic aligns a word with multiple words in a
sentence pair. Similarity of words between a hypothesis sentence and a reference sentence is calculated. AAS is given by averaging word similarity scores of all combinations of words in _|x||y|_ .
_|y|_
- _φ_ ( _xi, yj_ ) (1)
_j_ =1
AAS( _x, y_ ) =
_|x||y|_
_|x|_
_i_ =1
Here, _x_ is a hypothesis and _y_ is a reference; and _xi_
and _yj_ represent words in each sentence.


Second, we propose the maximum alignment similarity (MAS) heuristic averaging only the word
that has the maximum similarity score of each
aligned word pair. By definition, MAS itself is an
asymmetric score so we symmetrize it by averaging the score in both directions.
_|a|_
- max _φ_ ( _ai, bj_ ) (2)
_j_
_i_ =1
tence _y_ by the Hungarian method (Kuhn, 1955).
HAS( _x, y_ ) =
min( _|x|, |y|_ )


_|x|_
- _φ_ ( _xi, h_ ( _xi_ )) (4)
_i_ =1
Figure 1: Correlation of each word-alignmentbased method with varying the threshold for WMT
datasets.
Figure 2: Correlation of each word-alignmentbased method with varying the threshold for
WAT2015 and NTCIR8 datasets.
MAS( _x, y_ ) = [1]
2 [(MAS][asym][(] _[x, y]_ [)+MAS][asym][(] _[y, x]_ [))]
We report the results of MT evaluation in a
European–English translation task of the WMT12,
WMT13, and WMT15 datasets and Japanese–
English task of WAT2015 and NTCIR8 datasets.
For the WMT datasets, we compared our metrics
with BLEU and DREEM taken from the official
score of the WMT15 metric task (Stanojevi´c et al.,
2015). For WAT2015 and NTCIR8 datasets, the
three types of proposed methods are compared.


We used the WMT12, WMT13, and WMT15
datasets containing a total of 137,007 sentences
in French, Finnish, German, Czech, and Russian
translated to English. As Japanese–English translation datasets, WAT2015 includes 600 sentences
and NTCIR8 includes 1,200 sentences. We measured correlation between human adequacy score
and each of the evaluation metrics. We used
Kendall’s _τ_ for segment-level evaluation. We used
a pre-trained model of word2vec using the Google
News corpus for calculating word similarity using
our proposed methods. [1]


Table 1 shows a breakdown of correlation scores
for each language pair in WMT15. MAS shows
the best accuracy among all the proposed metrics
for all language pairs. Its accuracy is better than
that of DREEM for all language pairs except for
Czech–English. This result shows that removal of
noisy word embeddings by either using a threshold or 1:n alignment is important for European–
English datasets.
Figure 1 shows correlation of word-alignmentbased methods for WMT datasets with varying threshold values. For the WMT datasets,
MAS has the highest correlation scores among the
three word-alignment-based methods. A threshold value of 0.2 gives the maximum correlation for
MAS for all WMT datasets.
Figure 2 shows correlation of word-alignmentbased methods for the two Japanese–English
[1https://code.google.com/archive/p/](https://code.google.com/archive/p/word2vec/)
[word2vec/](https://code.google.com/archive/p/word2vec/)
(3)
Here, _a_ and _b_ are words in a hypothesis and a reference sentence, respectively.


Third, we introduce the Hungarian alignment similarity (HAS) to restrict word alignment to 1:1.
HAS formulates the task of word alignment as bipartite graph matching where the words in a hypothesis and a reference are represented as nodes
whose edges have weight _φ_ ( _xi, yi_ ). One-to-one
word alignment is achieved by calculating maximum alignment of the perfect bipartite graph. For
each word _xi_ included in a hypothesis sentence,
HAS chooses the word _h_ ( _xi_ ) in a reference sen
|Evaluation Metrics|Fr-En|Fi-En|De-En|Cs-En|Ru-En|Average|
|---|---|---|---|---|---|---|
|Average Alignment Similarity
Maximum Alignment Similarity
Hungarian Alignment Similarity|0.324
**0.368**
0.223|0.247
**0.355**
0.211|0.304
**0.392**
0.259|0.288
0.400
0.251|0.273
**0.349**
0.239|0.287
**0.373**
0.237|
|BLEU (Stanojevi´c et al., 2015)
DREEM (Chen and Guo, 2015)|0.358
0.362|0.308
0.340|0.360
0.368|0.391
**0.423**|0.329
0.348|0.349
0.368|
Table 1: Kendall’s _τ_ correlations of automatic evaluation metrics and official human judgements for the
WMT15 dataset. (Fr: French, Fi: Finnish, De: German, Cs: Czech, Ru: Russian, En: English)
|Evaluation Metrics|WMT12|WMT13|WMT15|WAT2015|NTCIR8|
|---|---|---|---|---|---|
|Average Alignment Similariy
Maximum Alignment Similarity
Hungarian Alignment Similarity|0.211
**0.353**
0.106|0.312
**0.381**
0.272|0.287
**0.373**
0.237|**0.332**
0.235
0.092|**0.343**
0.171
0.075|
Table 2: Kendall’s _τ_ correlations of word-alignment-based methods and the official human judgements
for each dataset. (WMT12, WMT13, and WMT15: European–English datasets, and WAT2015 and
NTCIR8: Japanese–English datasets)
datasets with a varying threshold. Although MAS
has the highest correlation for the WMT datasets,
AAS has the highest correlation for the WAT2015
and NTCIR8 datasets.
Table 2 describes segment-level correlation results for WMT, WAT2015, and NTCIR8 datasets.
MAS has the highest correlation score for the
WMT datasets, whereas AAS has the highest correlation score for WAT2015 and NTCIR8 datasets.


Figure 1 demonstrated that MAS and AAS are
more stable than HAS for European–English
datasets. This may be because it is relatively
easy for the AAS and MAS to perform word
alignment using word embeddings in translation
pairs of similar languages, but HAS suffers from
alignment sparsity more than the other methods.
In European–English translation, all the wordalignment-based methods perform poorly when
using no word embeddings.
Unlike the European–English translation task,
the Japanese–English translation task exhibits a
different tendency. Figure 2 shows the comparison between three types of word-alignment-based
methods for each threshold. This is partly because
word embeddings help evaluating lexically similar
word pairs but fail to model syntactic variations.
Also, we note that in Japanese–English datasets,
AAS achieved the highest correlation. We suppose
that this is because in Japanese–English transla
tion, it is difficult to cover all the source information in the target language, resulting in misalignment of inadequate words by HAS and MAS.
Table 2 shows that MAS performs stably on the
WMT datasets. In particular, Kendall’s _τ_ score of
HAS in WMT12 exhibits very low correlation. It
seems that the 1:1 alignment is too strict to calculate sentence similarity in MT evaluation, while
the 1:m (MAS) alignment performs well, possibly
because of the removal of noisy word alignment.
On the other hand, AAS is more stable than MAS
and HAS for WAT2015 and NTCIR8 datasets. As
a rule of thumb, AAS with high threshold values
(0.6–0.9) shows stable high correlation across all
language pairs, but if it is possible to use development data to tune the parameters, MAS with different values of thresholds should be considered.


In this paper, we presented word-alignment-based
MT evaluation metrics using distributed word representations. In our experiments, MAS showed
higher correlation with human evaluation than
other automatic MT metrics such as BLEU and
DREEM for European–English datasets. On the
other hand, for Japanese–English datasets, AAS
showed higher correlation with human evaluation
than other metrics. These results indicate that appropriate word alignment using word embeddings
is helpful in evaluating the MT output.


```
 {dengyuntian@seas,yoonkim@seas,justinchiu@g,dguo@college,srush@seas}.harvard.edu
```
School of Engineering and Applied Sciences
Harvard University
Cambridge, MA, USA

Latent Alignment and Variational Attention

Yuntian Deng _[∗]_ Yoon Kim _[∗]_ Justin Chiu Demi Guo Alexander M. Rush

Yuntian Deng
Yoon Kim
Justin Chiu
Demi Guo
Alexander M. Rush


Neural attention has become central to many state-of-the-art models in natural
language processing and related domains. Attention networks are an easy-to-train
and effective method for softly simulating alignment; however, the approach does
not marginalize over latent alignments in a probabilistic sense. This property makes
it difficult to compare attention to other alignment approaches, to compose it with
probabilistic models, and to perform posterior inference conditioned on observed
data. A related latent approach, hard attention, fixes these issues, but is generally
harder to train and less accurate. This work considers _variational attention_ networks, alternatives to soft and hard attention for learning latent variable alignment
models, with tighter approximation bounds based on amortized variational inference. We further propose methods for reducing the variance of gradients to make
these approaches computationally feasible. Experiments show that for machine
translation and visual question answering, inefficient exact latent variable models
outperform standard neural attention, but these gains go away when using hard
attention based training. On the other hand, variational attention retains most of
the performance gain but with training speed comparable to neural attention.


Attention networks [6] have quickly become the foundation for state-of-the-art models in natural
language understanding, question answering, speech recognition, image captioning, and more [15, 81,
16, 14, 63, 80, 71, 62]. Alongside components such as residual blocks and long-short term memory
networks, soft attention provides a rich neural network building block for controlling gradient flow
and encoding inductive biases. However, more so than these other components, which are often
treated as black-boxes, researchers use intermediate attention decisions directly as a tool for model
interpretability [43, 1] or as a factor in final predictions [25, 68]. From this perspective, attention
plays the role of a latent alignment variable [10, 37]. An alternative approach, hard attention [80],
makes this connection explicit by introducing a latent variable for alignment and then optimizing a
bound on the log marginal likelihood using policy gradients. This approach generally performs worse
(aside from a few exceptions such as [80]) and is used less frequently than its soft counterpart.
Still the latent alignment approach remains appealing for several reasons: (a) latent variables facilitate
reasoning about dependencies in a probabilistically principled way, e.g. allowing composition with
other models, (b) posterior inference provides a better basis for model analysis and partial predictions
than strictly feed-forward models, which have been shown to underperform on alignment in machine
translation [38], and finally (c) directly maximizing marginal likelihood may lead to better results.
_∗_ Equal contribution.
32nd Conference on Neural Information Processing Systems (NIPS 2018), Montréal, Canada.
The aim of this work is to quantify the issues with attention and propose alternatives based on recent
developments in variational inference. While the connection between variational inference and hard
attention has been noted in the literature [4, 41], the space of possible bounds and optimization
methods has not been fully explored and is growing quickly. These tools allow us to better quantify
whether the general underperformance of hard attention models is due to modeling issues (i.e. soft
attention imbues a better inductive bias) or optimization issues.
Our main contribution is a _variational attention_
approach that can effectively fit latent alignments while remaining tractable to train. We
consider two variants of variational attention:
_categorical_ and _relaxed_ . The categorical method
is fit with amortized variational inference using
a learned inference network and policy gradient
with a soft attention variance reduction baseline.
With an appropriate inference network (which
conditions on the entire source/target), it can be
used at training time as a drop-in replacement
for hard attention. The relaxed version assumes
that the alignment is sampled from a Dirichlet
distribution and hence allows attention over multiple source elements.
Figure 1: Sketch of variational attention applied to
machine translation. Two alignment distributions are
shown, the blue prior _p_, and the red variational posterior
_q_ taking into account future observations. Our aim is to
use _q_ to improve estimates of _p_ and to support improved
inference of _z_ .
_x_ 1: _T_
_x_ ˜ _y_ 3
Experiments describe how to implement this
_q_ taking into account future observations. Our aim is to
approach for two major attention-based models:
use _q_ to improve estimates of _p_ and to support improved
neural machine translation and visual question
inference of _z_ .
answering (Figure 1 gives an overview of our
approach for machine translation). We first show
that maximizing exact marginal likelihood can increase performance over soft attention. We further
show that with variational (categorical) attention, alignment variables significantly surpass both
soft and hard attention results without requiring much more difficult training. We further explore
the impact of posterior inference on alignment decisions, and how latent variable models might be
employed. Our code is available at `[https://github.com/harvardnlp/var-attn/](https://github.com/harvardnlp/var-attn/)` .
**Related Work** Latent alignment has long been a core problem in NLP, starting with the seminal IBM
models [11], HMM-based alignment models [75], and a fast log-linear reparameterization of the IBM
2 model [20]. Neural soft attention models were originally introduced as an alternative approach
for neural machine translation [6], and have subsequently been successful on a wide range of tasks
(see [15] for a review of applications). Recent work has combined neural attention with traditional
alignment [18, 72] and induced structure/sparsity [48, 33, 44, 85, 54, 55, 49], which can be combined
with the variational approaches outlined in this paper.
In contrast to soft attention models, hard attention [80, 3] approaches use a single sample at training
time instead of a distribution. These models have proven much more difficult to train, and existing
works typically treat hard attention as a black-box reinforcement learning problem with log-likelihood
as the reward [80, 3, 53, 26, 19]. Two notable exceptions are [4, 41]: both utilize amortized variational
inference to learn a sampling distribution which is used obtain importance-sampled estimates of the
log marginal likelihood [12]. Our method uses uses different estimators and targets the single sample
approach for efficiency, allowing the method to be employed for NMT and VQA applications.
There has also been significant work in using variational autoencoders for language and translation
application. Of particular interest are those that augment an RNN with latent variables (typically
Gaussian) at each time step [17, 22, 66, 23, 40] and those that incorporate latent variables into
sequence-to-sequence models [84, 7, 70, 64]. Our work differs by modeling an explicit model
component (alignment) as a latent variable instead of auxiliary latent variables (e.g. topics). The
term "variational attention" has been used to refer to a different component the output from attention
(commonly called the context vector) as a latent variable [7], or to model both the memory and the
alignment as a latent variable [9]. Finally, there is some parallel work [78, 67] which also performs
exact/approximate marginalization over latent alignments for sequence-to-sequence learning.


We begin by introducing notation for latent alignment, and then show how it relates to neural attention.
For clarity, we are careful to use _alignment_ to refer to this probabilistic model (Section 2.1), and _soft_
and _hard_ attention to refer to two particular inference approaches used in the literature to estimate
alignment models (Section 2.2).

2 Background: Latent Alignment and Neural Attention


Figure 2(a) shows a latent alignment model. Let _x_ be an observed set with associated members
_{x_ 1 _, . . ., xi, . . ., xT }_ . Assume these are vector-valued (i.e. _xi ∈_ R _[d]_ ) and can be stacked to form a
matrix _X ∈_ R _[d][×][T]_ . Let the observed ˜ _x_ be an arbitrary “query”. These generate a discrete output
variable _y ∈Y_ . This process is mediated through a latent alignment variable _z_, which indicates
which member (or mixture of members) of _x_ generates _y_ . The generative process we consider is:
_z ∼D_ ( _a_ ( _x,_ ˜ _x_ ; _θ_ )) _y ∼_ _f_ ( _x, z_ ; _θ_ )
where _a_ produces the parameters for an alignment distribution _D_ . The function _f_ gives a distribution
over the output, e.g. an exponential family. To fit this model to data, we set the model parameters _θ_
by maximizing the log marginal likelihood of training examples ( _x,_ ˜ _x,_ ˆ _y_ ): [2]
max log _p_ ( _y_ = ˆ _y | x,_ ˜ _x_ ) = max log E _z_ [ _f_ ( _x, z_ ; _θ_ ) _y_ ˆ]
_θ_ _θ_
Directly maximizing this log marginal likelihood in the presence of the latent variable _z_
is often difficult due to the expectation (though
tractable in certain cases).
For this to represent an alignment, we restrict
the variable _z_ to be in the simplex ∆ _[T][ −]_ [1] over
source indices _{_ 1 _, . . ., T_ _}_ . We consider two distributions for this variable: first, let _D_ be a _cat-_
_egorical_ where _z_ is a one-hot vector with _zi_ = 1
if _xi_ is selected. For example, _f_ ( _x, z_ ) could use
_z_ to pick from _x_ and apply a softmax layer to
predict _y_, i.e. _f_ ( _x, z_ ) = softmax( **W** _Xz_ ) and
**W** _∈_ R _[|Y|×][d]_,
(a)
(b)
Figure 2: Models over observed set _x_, query ˜ _x_, and
alignment _z_ . (a) Latent alignment model, (b) Soft attention with _z_ absorbed into prediction network.
log _p_ ( _y_ = ˆ _y | x,_ ˜ _x_ ) = log
_T_
_p_ ( _zi_ = 1 _| x,_ ˜ _x_ ) _p_ ( _y_ = ˆ _y | x, zi_ = 1) = log E _z_ [softmax( **W** _Xz_ ) _y_ ˆ]
_i_ =1
This computation requires a factor of _O_ ( _T_ ) additional runtime, and introduces a major computational
factor into already expensive deep learning models. [3]
Second we consider a _relaxed_ alignment where _z_ is a mixture taken from the interior of the simplex by
letting _D_ be a Dirichlet. This objective looks similar to the categorical case, i.e. log _p_ ( _y_ = ˆ _y | x,_ ˜ _x_ ) =
log E _z_ [softmax( **W** _Xz_ ) _y_ ˆ], but the resulting expectation is intractable to compute exactly.


When training deep learning models with gradient methods, it can be difficult to use latent alignment
directly. As such, two alignment-like approaches are popular: _soft attention_ replaces the probabilistic
model with a deterministic soft function and _hard attention_ trains a latent alignment model by
maximizing a lower bound on the log marginal likelihood (obtained from Jensen’s inequality) with
policy gradient-style training. We briefly describe how these methods fit into this notation.
2When clear from context, the random variable is dropped from E[ _·_ ]. We also interchangeably use _p_ (ˆ _y | x,_ ˜ _x_ )
and _f_ ( _x, z_ ; _θ_ ) _y_ ˆ to denote _p_ ( _y_ = ˆ _y | x,_ ˜ _x_ ).
3Although not our main focus, explicit marginalization is sometimes tractable with efficient matrix operations
on modern hardware, and we compare the variational approach to explicit enumeration in the experiments. In
some cases it is also possible to efficiently perform exact marginalization with dynamic programming if one
imposes additional constraints (e.g. monotonicity) on the alignment distribution [83, 82, 58].
**Soft Attention** Soft attention networks use an altered model shown in Figure 2b. Instead of using a
latent variable, they employ a deterministic network to compute an expectation over the alignment
variable. We can write this model using the same functions _f_ and _a_ from above,
log _p_ soft( _y | x,_ ˜ _x_ ) = log _f_ ( _x,_ E _z_ [ _z_ ]; _θ_ ) = log softmax( **W** _X_ E _z_ [ _z_ ])
A major benefit of soft attention is efficiency. Instead of paying a multiplicative penalty of _O_ ( _T_ )
or requiring integration, the soft attention model can compute the expectation before _f_ . While
formally a different model, soft attention has been described as an approximation of alignment [80].
Since E[ _z_ ] _∈_ ∆ _[T][ −]_ [1], soft attention uses a convex combination of the input representations _X_ E[ _z_ ]
(the _context vector_ ) to obtain a distribution over the output. While also a “relaxed” decision, this
expression differs from both the latent alignment models above. Depending on _f_, the gap between
E[ _f_ ( _x, z_ )] and _f_ ( _x,_ E[ _z_ ]) may be large.
However there are some important special cases. In the case where _p_ ( _z | x,_ ˜ _x_ ) is deterministic, we
have E[ _f_ ( _x, z_ )] = _f_ ( _x,_ E[ _z_ ]), and _p_ ( _y | x,_ ˜ _x_ ) = _p_ soft( _y | x,_ ˜ _x_ ). In general we can bound the absolute
difference based on the maximum curvature of _f_, as shown by the following proposition.
**Proposition 1.** _Define gx,y_ ˆ : ∆ _[T][ −]_ [1] _�→_ [0 _,_ 1] _to be the function given by gx,y_ ˆ( _z_ ) = _f_ ( _x, z_ ) _y_ ˆ _(i.e._
_gx,y_ ˆ( _z_ ) = _p_ ( _y_ = ˆ _y | x,_ ˜ _x, z_ )) _for a twice differentiable function f_ _. Let Hgx,y_ ˆ( _z_ ) _be the Hessian of_
_gx,y_ ˆ( _z_ ) _evaluated at z, and further suppose ∥Hgx,y_ ˆ( _z_ ) _∥_ 2 _≤_ _c for all z ∈_ ∆ _[T][ −]_ [1] _,_ ˆ _y ∈Y, and x, where_
_∥· ∥_ 2 _is the spectral norm. Then for all_ ˆ _y ∈Y,_
_| p_ ( _y_ = ˆ _y | x,_ ˜ _x_ ) _−_ _p_ soft( _y_ = ˆ _y | x,_ ˜ _x_ ) _| ≤_ _c_
The proof is given in Appendix A. [4] Empirically the soft approximation works remarkably well, and
often moves towards a sharper distribution with training. Alignment distributions learned this way
often correlate with human intuition (e.g. word alignment in machine translation) [38]. [5]
**Hard Attention** Hard attention is an approximate inference approach for latent alignment (Figure 2a) [80, 4, 53, 26]. Hard attention takes a single hard sample of _z_ (as opposed to a soft mixture)
and then backpropagates through the model. The approach is derived by two choices: First apply Jensen’s inequality to get a lower bound on the log marginal likelihood, log E _z_ [ _p_ ( _y | x, z_ )] _≥_
E _z_ [log _p_ ( _y | x, z_ )], then maximize this lower-bound with policy gradients/REINFORCE [76] to obtain
unbiased gradient estimates,
_∇θ_ E _z_ [log _f_ ( _x, z_ ))] = E _z_ [ _∇θ_ log _f_ ( _x, z_ ) + (log _f_ ( _x, z_ ) _−_ _B_ ) _∇θ_ log _p_ ( _z | x,_ ˜ _x_ )] _,_
where _B_ is a baseline that can be used to reduce the variance of this estimator. To implement this
approach efficiently, hard attention uses Monte Carlo sampling to estimate the expectation in the
gradient computation. For efficiency, a single sample from _p_ ( _z | x,_ ˜ _x_ ) is used, in conjunction with
other tricks to reduce the variance of the gradient estimator (discussed more below) [80, 50, 51].


Amortized variational inference (AVI, closely related to variational auto-encoders) [36, 61, 50] is a
class of methods to efficiently approximate latent variable inference, using learned inference networks.
In this section we explore this technique for deep latent alignment models, and propose methods for
_variational attention_ that combine the benefits of soft and hard attention.
First note that the key approximation step in hard attention is to optimize a lower bound derived from
Jensen’s inequality. This gap could be quite large, contributing to poor performance. [6] Variational
4It is also possible to study the gap in finer detail by considering distributions over the inputs of _f_ that have
high probability under approximately linear regions of _f_, leading to the notion of _approximately expectation-_
_linear_ functions, which was originally proposed and studied in the context of dropout [46].
5Another way of viewing soft attention is as simply a non-probabilistic learned function. While it is possible
that such models encode better inductive biases, our experiments show that when properly optimized, latent
alignment attention with explicit latent variables do outperform soft attention.
6Prior works on hard attention have generally approached the problem as a black-box reinforcement learning
problem where the rewards are given by log _f_ ( _x, z_ ). Ba et al. (2015) [4] and Lawson et al. (2017) [41] are
the notable exceptions, and both works utilize the framework from [51] which obtains multiple samples from a
learned sampling distribution to optimize the IWAE bound [12] or a reweighted wake-sleep objective.
**Algorithm 1** Variational Attention
_λ ←_ enc( _x,_ ˜ _x, y_ ; _φ_ ) _▷_ _Compute var. params_
_z ∼_ _q_ ( _z_ ; _λ_ ) _▷_ _Sample var. attention_
log _f_ ( _x, z_ ) _▷_ Compute output dist
_z_ _[′]_ _←_ E _p_ ( _z′ | x,x_ ˜)[ _z_ _[′]_ ] _▷_ Compute soft atten.
_B_ = log _f_ ( _x, z_ _[′]_ ) _▷_ Compute baseline dist
Backprop _∇θ_ and _∇φ_ based on eq. 1 and KL
**Algorithm 2** Variational Relaxed Attention
max _θ_ E _z∼p_ [log _p_ ( _y | x, z_ )] _▷_ _Pretrain fixed θ_
_. . ._
_u ∼U_ _▷_ _Sample unparam._
_z ←_ _gφ_ ( _u_ ) _▷_ _Reparam sample_
log _f_ ( _x, z_ ) _▷_ Compute output dist
Backprop _∇θ_ and _∇φ_, reparam and KL
inference methods directly aim to tighten this gap. In particular, the _evidence lower bound_ (ELBO)
is a parameterized bound over a family of distributions _q_ ( _z_ ) _∈Q_ (with the constraint that the
supp _q_ ( _z_ ) _⊆_ supp _p_ ( _z | x,_ ˜ _x, y_ )),
log E _z∼p_ ( _z | x,x_ ˜)[ _p_ ( _y | x, z_ )] _≥_ E _z∼q_ ( _z_ )[log _p_ ( _y | x, z_ )] _−_ KL[ _q_ ( _z_ ) _∥_ _p_ ( _z | x,_ ˜ _x_ )]
This allows us to search over variational distributions _q_ to improve the bound. It is tight when the
variational distribution is equal to the posterior, i.e. _q_ ( _z_ ) = _p_ ( _z | x,_ ˜ _x, y_ ). Hard attention is a special
case of the ELBO with _q_ ( _z_ ) = _p_ ( _z | x,_ ˜ _x_ ).
There are many ways to optimize the evidence lower bound; an effective choice for deep learning
applications is to use _amortized variational inference_ . AVI uses an _inference network_ to produce the
parameters of the variational distribution _q_ ( _z_ ; _λ_ ). The inference network takes in the input, query,
and the output, i.e. _λ_ = _enc_ ( _x,_ ˜ _x, y_ ; _φ_ ). The objective aims to reduce the gap with the inference
network _φ_ while also training the generative model _θ_,
max _φ,θ_ [E] _[z][∼][q]_ [(] _[z]_ [;] _[λ]_ [)][[log] _[ p]_ [(] _[y][ |][ x, z]_ [)]] _[ −]_ [KL[] _[q]_ [(] _[z]_ [;] _[ λ]_ [)] _[ ∥]_ _[p]_ [(] _[z][ |][ x,]_ [ ˜] _[x]_ [)]]
With the right choice of optimization strategy and inference network this form of variational attention
can provide a general method for learning latent alignment models. In the rest of this section, we
consider strategies for accurately and efficiently computing this objective; in the next section, we
describe instantiations of _enc_ for specific domains.
**Algorithm 1: Categorical Alignments** First consider the case where _D_, the alignment distribution,
and _Q_, the variational family, are categorical distributions. Here the generative assumption is that
_y_ is generated from a single index of _x_ . Under this setup, a low-variance estimator of _∇θ_ ELBO, is
easily obtained through a single sample from _q_ ( _z_ ). For _∇φ_ ELBO, the gradient with respect to the
KL portion is easily computable, but there is an optimization issue with the gradient with respect to
the first term E _z∼q_ ( _z_ )[log _f_ ( _x, z_ ))].
Many recent methods target this issue, including neural estimates of baselines [50, 51], RaoBlackwellization [59], reparameterizable relaxations [31, 47], and a mix of various techniques
[73, 24]. We found that an approach using REINFORCE [76] along with a specialized baseline was
effective. However, note that REINFORCE is only one of the inference choices we can select, and
as we will show later, alternative approaches such as reparameterizable relaxations work as well.
Formally, we first apply the likelihood-ratio trick to obtain an expression for the gradient with respect
to the inference network parameters _φ_,
_∇φ_ E _z∼q_ ( _z_ )[log _p_ ( _y | x, z_ )] = E _z∼q_ ( _z_ )[(log _f_ ( _x, z_ ) _−_ _B_ ) _∇φ_ log _q_ ( _z_ )]
As with hard attention, we take a single Monte Carlo sample (now drawn from the variational
distribution). Variance reduction of this estimate falls to the baseline term _B_ . The ideal (and intuitive)
baseline would be E _z∼q_ ( _z_ )[log _f_ ( _x, z_ )], analogous to the value function in reinforcement learning.
While this term cannot be easily computed, there is a natural, cheap approximation: soft attention (i.e.
log _f_ ( _x,_ E[ _z_ ])). Then the gradient is
- _∇φ_ log _q_ ( _z | x,_ ˜ _x_ ) (1)
E _z∼q_ ( _z_ )
�� _f_ ( _x, z_ )
log
_f_ ( _x,_ E _z′∼p_ ( _z′ | x,x_ ˜)[ _z_ _[′]_ ])
Effectively this weights gradients to _q_ based on the ratio of the inference network alignment approach
to a soft attention baseline. Notably the expectation in the soft attention is over _p_ (and not over _q_ ),
and therefore the baseline is constant with respect to _φ_ . Note that a similar baseline can also be used
for hard attention, and we apply it to both variational/hard attention models in our experiments.
**Algorithm 2: Relaxed Alignments** Next consider treating both _D_ and _Q_ as Dirichlets, where _z_
represents a mixture of indices. This model is in some sense closer to the soft attention formulation
which assigns mass to multiple indices, though fundamentally different in that we still formally treat
alignment as a latent variable. Again the aim is to find a low variance gradient estimator. Instead of
using REINFORCE, certain continuous distributions allow the use reparameterization [36], where
sampling _z ∼_ _q_ ( _z_ ) can be done by first sampling from a simple unparameterized distribution _U_, and
then applying a transformation _gφ_ ( _·_ ), yielding an unbiased estimator,
E _u∼U_ [ _∇φ_ log _p_ ( _y|x, gφ_ ( _u_ ))] _−∇φ_ KL [ _q_ ( _z_ ) _∥_ _p_ ( _z | x,_ ˜ _x_ )]
The Dirichlet distribution is not directly reparameterizable. While transforming the standard uniform
distribution with the inverse CDF of Dirichlet would result in a Dirichlet distribution, the inverse
CDF does not have an analytical solution. However, we can use rejection based sampling to get a
sample, and employ implicit differentiation to estimate the gradient of the CDF [32].
Empirically, we found the random initialization would result in convergence to uniform Dirichlet
parameters for _λ_ . (We suspect that it is easier to find low KL local optima towards the center of the
simplex). In experiments, we therefore initialize the latent alignment model by first minimizing the
Jensen bound, E _z∼p_ ( _z | x,x_ ˜)[log _p_ ( _y | x, z_ )], and then introducing the inference network.

3 Variational Attention for Latent Alignment Models


We experiment with variational attention in two different domains where attention-based models are
essential and widely-used: neural machine translation and visual question answering.
**Neural Machine Translation** Neural machine translation (NMT) takes in a source sentence and
predicts each word of a target sentence _yj_ in an auto-regressive manner. The model first contextually
embeds each source word using a bidirectional LSTM to produce the vectors _x_ 1 _. . . xT_ . The query
_x_ ˜ consists of an LSTM-based representation of the previous target words _y_ 1: _j−_ 1. Attention is used
to identify which source positions should be used to predict the target. The parameters of _D_ are
generated from an MLP between the query and source [6], and _f_ concatenates the selected _xi_ with
the query ˜ _x_ and passes it to an MLP to produce the distribution over the next target word _yj_ .
For variational attention, the inference network applies a bidirectional LSTM over the source and
the target to obtain the hidden states _x_ 1 _, . . ., xT_ and _h_ 1 _, . . ., hS_, and produces the alignment scores
at the _j_ -th time step via a bilinear map, _s_ [(] _i_ _[j]_ [)] = exp( _h_ _[⊤]_ _j_ **[U]** _[x][i]_ [)][. For the categorical case, the scores]
are normalized, _q_ ( _zi_ [(] _[j]_ [)] = 1) _∝_ _s_ [(] _i_ _[j]_ [)][; in the relaxed case the parameters of the Dirichlet are] _[ α]_ _i_ [(] _[j]_ [)] =
_si_ [(] _[j]_ [)][. Note, the inference network sees the entire target (through bidirectional LSTMs). The word]
embeddings are shared between the generative/inference networks, but other parameters are separate.
**Visual Question Answering** Visual question answering (VQA) uses attention to locate the parts of
an image that are necessary to answer a textual question. We follow the recently-proposed “bottom-up
top-down” attention approach [2], which uses Faster R-CNN [60] to obtain object bounding boxes
and performs mean-pooling over the convolutional features (from a pretrained ResNet-101 [27]) in
each bounding box to obtain object representations _x_ 1 _, . . ., xT_ . The query ˜ _x_ is obtained by running
an LSTM over the question, the attention function _a_ passes the query and the object representation
through an MLP. The prediction function _f_ is also similar to the NMT case: we concatenate the
chosen _xi_ with the query ˜ _x_ to use as input to an MLP which produces a distribution over the output.
The inference network _enc_ uses the answer embedding _hy_ and combines it with _xi_ and ˜ _x_ to produce
the variational (categorical) distribution,
_q_ ( _zi_ = 1) _∝_ exp( _u_ _[⊤]_ tanh( **U** 1( _xi ⊙_ ReLU( **V** 1 _hy_ )) + **U** 2(˜ _x ⊙_ ReLU( **V** 2 _hy_ ))))
where _⊙_ is the element-wise product. This parameterization worked better than alternatives. We did
not experiment with the relaxed case in VQA, as the object bounding boxes already give us the ability
to attend to larger portions of the image.
**Inference Alternatives** For categorical alignments we described maximizing a particular variational lower bound with REINFORCE. Note that other alternatives exist, and we briefly discuss them
here: 1) instead of the single-sample variational bound we can use a multiple-sample importance
sampling based approach such as Reweighted Wake-Sleep (RWS) [4] or VIMCO [52]; 2) instead of
REINFORCE we can approximate sampling from the discrete categorical distribution with GumbelSoftmax [30]; 3) instead of using an inference network we can directly apply Stochastic Variational
Inference (SVI) [28] to learn the local variational parameters in the posterior.
**Predictive Inference** At test time, we need to marginalize out the latent variables, i.e.
E _z_ [ _p_ ( _y | x,_ ˜ _x, z_ )] using _p_ ( _z | x,_ ˜ _x_ ). In the categorical case, if speed is not an issue then enumerating alignments is preferable, which incurs a multiplicative cost of _O_ ( _T_ ) (but the enumeration is
parallelizable). Alternatively we experimented with a _K_ -max renormalization, where we only take
the top- _K_ attention scores to approximate the attention distribution (by re-normalizing). This makes
the multiplicative cost constant with respect to _T_ . For the relaxed case, sampling is necessary.