**Setup** For NMT we mainly use the IWSLT dataset [13]. This dataset is relatively small, but has
become a standard benchmark for experimental NMT models. We follow the same preprocessing as
in [21] with the same Byte Pair Encoding vocabulary of 14k tokens [65]. To show that variational
attention scales to large datasets, we also experiment on the WMT 2017 English-German dataset [8],
following the preprocessing in [74] except that we use newstest2017 as our test set. For VQA, we use
the VQA 2.0 dataset. As we are interested in intrinsic evaluation (i.e. log-likelihood) in addition to
the standard VQA metric, we randomly select half of the standard validation set as the test set (since
we need access to the actual labels). [7] (Therefore the numbers provided are not strictly comparable to
existing work.) While the preprocessing is the same as [2], our numbers are worse than previously
reported as we do not apply any of the commonly-utilized techniques to improve performance on
VQA such as data augmentation and label smoothing.
Experiments vary three components of the systems: (a) training objective and model, (b) training
approximations, comparing enumeration or sampling, [8] (c) test inference. All neural models have the
same architecture and the exact same number of parameters _θ_ (the inference network parameters _φ_
vary, but are not used at test). When training hard and variational attention with sampling both use
the same baseline, i.e the output from soft attention. The full architectures/hyperparameters for both
NMT and VQA are given in Appendix B.
**Results and Discussion** Table 1 shows the main results. We first note that hard attention underperforms soft attention, even when its expectation is enumerated. This indicates that Jensen’s inequality
alone is a poor bound. On the other hand, on both experiments, exact marginal likelihood outperforms
soft attention, indicating that when possible it is better to have latent alignments.
For NMT, on the IWSLT 2014 German-English task, variational attention with enumeration and
sampling performs comparably to optimizing the log marginal likelihood, despite the fact that it is
optimizing a lower bound. We believe that this is due to the use of _q_ ( _z_ ), which conditions on the
entire source/target and therefore potentially provides better training signal to _p_ ( _z | x,_ ˜ _x_ ) through the
KL term. Note that it is also possible to have _q_ ( _z_ ) come from a pretrained external model, such as
a traditional alignment model [20]. Table 3 (left) shows these results in context compared to the
best reported values for this task. Even with sampling, our system improves on the state-of-the-art.
On the larger WMT 2017 English-German task, the superior performance of variational attention
persists: our baseline soft attention reaches 24.10 BLEU score, while variational attention reaches
24.98. Note that this only reflects a reasonable setting without exhaustive tuning, yet we show that
we can train variational attention at scale. For VQA the trend is largely similar, and results for NLL
with variational attention improve on soft attention and hard attention. However the task-specific
evaluation metrics are slightly worse.
Table 2 (left) considers test inference for variational attention, comparing enumeration to _K_ -max with
_K_ = 5. For all methods exact enumeration is better, however _K_ -max is a reasonable approximation.
7 VQA eval metric is defined as min _{_ # humans that said answer3 _,_ 1 _}_ . Also note that since there are sometimes
multiple answers for a given question, in such cases we sample (where the sampling probability is proportional
to the number of humans that said the answer) to get a single label.
8Note that enumeration does not imply exact if we are enumerating an expectation on a lower bound.
NMT VQA
Model Objective E PPL BLEU NLL Eval
Soft Attention log _p_ ( _y |_ E[ _z_ ])    - 7.17 32.77 1.76 58.93
Marginal Likelihood log E[ _p_ ] Enum 6.34 33.29 1.69 60.33
Hard Attention E _p_ [log _p_ ] Enum 7.37 31.40 1.78 57.60
Hard Attention E _p_ [log _p_ ] Sample 7.38 31.00 1.82 56.30
Variational Relaxed Attention E _q_ [log _p_ ] _−_ KL Sample 7.58 30.05    -    Variational Attention E _q_ [log _p_ ] _−_ KL Enum 6.08 33.68 1.69 58.44
Variational Attention E _q_ [log _p_ ] _−_ KL Sample 6.17 33.30 1.75 57.52
Table 1: Evaluation on NMT and VQA for the various models. E column indicates whether the expectation
is calculated via enumeration (Enum) or a single sample (Sample) during training. For NMT we evaluate
intrinsically on perplexity (PPL) (lower is better) and extrinsically on BLEU (higher is better), where for BLEU
we perform beam search with beam size 10 and length penalty (see Appendix B for further details). For VQA
we evaluate intrinsically on negative log-likelihood (NLL) (lower is better) and extrinsically on VQA evaluation
PPL BLEU
Model Exact _K_ -Max Exact _K_ -Max
Marginal Likelihood 6.34 6.90 33.29 33.31
Hard + Enum 7.37 7.37 31.40 31.37
Hard + Sample 7.38 7.38 31.00 31.04
Variational + Enum 6.08 6.42 33.68 33.69
Variational + Sample 6.17 6.51 33.30 33.27
Table 2: (Left) Performance change on NMT from exact decoding to _K_ -Max decoding with _K_ = 5. (see section
5 for definition of K-max decoding). (Right) Test perplexity of different approaches while varying _K_ to estimate
E _z_ [ _p_ ( _y|x,_ ˜ _x_ )]. Dotted lines compare soft baseline and variational with full enumeration.
Table 2 (right) shows the PPL of different models as we increase _K_ . Good performance requires
_K >_ 1, but we only get marginal benefits for _K >_ 5. Finally, we observe that it is possible to _train_
with soft attention and _test_ using _K_ -Max with a small performance drop ( `Soft KMax` in Table 2
(right)). This possibly indicates that soft attention models are approximating latent alignment models.
On the other hand, training with latent alignments and testing with soft attention performed badly.
Table 3 (lower right) looks at the entropy of the prior distribution learned by the different models.
Note that hard attention has very low entropy (high certainty) whereas soft attention is quite high.
The variational attention model falls in between. Figure 3 (left) illustrates the difference in practice.
Table 3 (upper right) compares inference alternatives for variational attention. RWS reaches a
comparable performance as REINFORCE, but at a higher memory cost as it requires multiple
samples. Gumbel-Softmax reaches nearly the same performance and seems like a viable alternative;
although we found its performance is sensitive to its temperature parameter. We also trained a
non-amortized SVI model, but found that at similar runtime it was not able to produce satisfactory
results, likely due to insufficient updates of the local variational parameters. A hybrid method such as
semi-amortized inference [39, 34] might be a potential future direction worth exploring.
Despite extensive experiments, we found that variational relaxed attention performed worse than other
methods. In particular we found that when training with a Dirichlet KL, it is hard to reach low-entropy
regions of the simplex, and the attentions are more uniform than either soft or variational categorical
attention. Table 3 (lower right) quantifies this issue. We experimented with other distributions such
as Logistic-Normal and Gumbel-Softmax [31, 47] but neither fixed this issue. Others have also noted
difficulty in training Dirichlet models with amortized inference [69].
Besides performance, an advantage of these models is the ability to perform posterior inference, since
the _q_ function can be used directly to obtain posterior alignments. Contrast this with hard attention
where _q_ = _p_ ( _z | x,_ ˜ _x_ ), i.e. the variational posterior is independent of the future information. Figure 3
shows the alignments of _p_ and _q_ for variational attention over a fixed sentence (see Appendix C for
more examples). We see that _q_ is able to use future information to correct alignments. We note that
the inability of soft and hard attention to produce good alignments has been noted as a major issue
in NMT [38]. While _q_ is not used directly in left-to-right NMT decoding, it could be employed for
other applications such as in an iterative refinement approach [56, 42].
Figure 3: (Left) An example demonstrating the difference between the prior alignment (red) and the variational
posterior (blue) when translating from DE-EN (left-to-right). Note the improved blue alignments for `actually`
and `violent` which benefit from seeing the next word. (Right) Comparison of soft attention (green) with the _p_
of variational attention (red). Both models imply a similar alignment, but variational attention has lower entropy.
Inference Method #Samples PPL BLEU
IWSLT
Model BLEU
Beam Search Optimization [77] 26.36
Actor-Critic [5] 28.53
Neural PBMT + LM [29] 30.08
Minimum Risk Training [21] 32.84
Soft Attention 32.77
Marginal Likelihood 33.29
Hard Attention + Enum 31.40
Hard Attention + Sample 30.42
Variational Relaxed Attention 30.05
Variational Attention + Enum 33.69
Variational Attention + Sample 33.30
REINFORCE 1 6.17 33.30
RWS 5 6.41 32.96
Gumbel-Softmax 1 6.51 33.08
Entropy
Model NMT VQA
Soft Attention 1.24 2.70
Marginal Likelihood 0.82 2.66
Hard Attention + Enum 0.05 0.73
Hard Attention + Sample 0.07 0.58
Variational Relaxed Attention 2.02 Variational Attention + Enum 0.54 2.07
Variational Attention + Sample 0.52 2.44
Table 3: (Left) Comparison against the best prior work for NMT on the IWSLT 2014 German-English test set.
(Upper Right) Comparison of inference alternatives of variational attention on IWSLT 2014. (Lower Right)
Comparison of different models in terms of implied discrete entropy (lower = more certain alignment).
**Potential Limitations** While this technique is a promising alternative to soft attention, there are
some practical limitations: (a) Variational/hard attention needs a good baseline estimator in the form
of soft attention. We found this to be a necessary component for adequately training the system. This
may prevent this technique from working when _T_ is intractably large and soft attention is not an
option. (b) For some applications, the model relies heavily on having a good posterior estimator. In
VQA we had to utilize domain structure for the inference network construction. (c) Recent models
such as the Transformer [74], utilize many repeated attention models. For instance the current best
translation models have the equivalent of 150 different attention queries per word translated. It is
unclear if this approach can be used at that scale as predictive inference becomes combinatorial.


Attention methods are ubiquitous tool for areas like natural language processing; however they
are difficult to use as latent variable models. This work explores alternative approaches to latent
alignment, through variational attention with promising result. Future work will experiment with
scaling the method on larger-scale tasks and in more complex models, such as multi-hop attention
models, transformer models, and structured models, as well as utilizing these latent variables for
interpretability and as a way to incorporate prior knowledge.

Latent Alignment and Variational Attention

Yuntian Deng
Yoon Kim
Justin Chiu
Demi Guo
Alexander M. Rush


We are grateful to Sam Wiseman and Rachit Singh for insightful comments and discussion, as well as
Christian Puhrsch for help with translations. This project was supported by a Facebook Research
Award (Low Resource NMT). YK is supported by a Google AI PhD Fellowship. YD is supported by
a Bloomberg Research Award. AMR gratefully acknowledges the support of NSF CCF-1704834 and
an Amazon AWS Research award.


**Proposition.** _Define gx,y_ ˆ : ∆ _[T][ −]_ [1] _�→_ [0 _,_ 1] _to be the function given by gx,y_ ˆ( _z_ ) = _f_ ( _x, z_ ) _y_ ˆ _(i.e._
_gx,y_ ˆ( _z_ ) = _p_ ( _y_ = ˆ _y | x,_ ˜ _x, z_ )) _for a twice differentiable function f_ _. Let Hgx,y_ ˆ( _z_ ) _be the Hessian of_
_gx,y_ ˆ( _z_ ) _evaluated at z, and further suppose ∥Hgx,y_ ˆ( _z_ ) _∥_ 2 _≤_ _c for all z ∈_ ∆ _[T][ −]_ [1] _,_ ˆ _y ∈Y, and x, where_
_∥· ∥_ 2 _is the spectral norm. Then for all_ ˆ _y ∈Y,_
_| p_ ( _y_ = ˆ _y | x,_ ˜ _x_ ) _−_ _p_ soft( _y_ = ˆ _y | x,_ ˜ _x_ ) _| ≤_ _c_
_Proof._ We begin by performing Taylor’s expansion of _gx,y_ ˆ at E[ _z_ ]:
        -         E[ _gx,y_ ˆ( _z_ )] = E _gx,y_ ˆ(E[ _z_ ]) + ( _z −_ E[ _z_ ]) _[⊤]_ _∇gx,y_ ˆ(E[ _z_ ]) + [1]
2 [(] _[z][ −]_ [E][[] _[z]_ [])] _[⊤][H][g][x,][y]_ [ˆ][(ˆ] _[z]_ [)(] _[z][ −]_ [E][[] _[z]_ [])]
= _gx,y_ ˆ(E[ _z_ ]) + [1]
2 [E][[(] _[z][ −]_ [E][[] _[z]_ [])] _[⊤][H][g][x,][y]_ [ˆ][(ˆ] _[z]_ [)(] _[z][ −]_ [E][[] _[z]_ [])]]
for some ˆ _z_ = _λz_ + (1 _−_ _λ_ )E[ _z_ ] _, λ ∈_ [0 _,_ 1]. Then letting _u_ = _z −_ E[ _z_ ], we have
_u_ _[⊤]_ _u_
_|_ ( _z −_ E[ _z_ ]) _[⊤]_ _Hgx,y_ ˆ(ˆ _z_ )( _z −_ E[ _z_ ]) _|_ = _| ∥u∥_ [2] 2 _∥u∥_ 2 _Hgx,y_ ˆ(ˆ _z_ ) _∥u∥_ 2 _|_
_≤∥u∥_ [2] 2 _[c]_
where _c_ = max _{|λ_ max _|, |λ_ min _|}_ is the largest absolute eigenvalue of _Hgx,y_ ˆ(ˆ _z_ ). (Here _λ_ max and _λ_ min
are maximum/minimum eigenvalues of _HgX,q_ (ˆ _z_ )). Note that _c_ is also equal to the spectral norm
_∥HgX,q_ (ˆ _z_ ) _∥_ 2 since the Hessian is symmetric.
Then,
_|_ E[( _z −_ E[ _z_ ]) _[⊤]_ _Hgx,y_ ˆ(ˆ _z_ )( _z −_ E[ _z_ ])] _| ≤_ E[ _|_ ( _z −_ E[ _z_ ]) _[⊤]_ _Hgx,y_ ˆ(ˆ _z_ )( _z −_ E[ _z_ ]) _|_ ]
_≤_ E[ _∥u∥_ [2] 2 _[c]_ []]
_≤_ 2 _c_
Here the first inequality follows due to the convexity of the absolute value function and the last
inequality follows since
_∥u∥_ [2] 2 [= (] _[z][ −]_ [E][[] _[z]_ [])] _[⊤]_ [(] _[z][ −]_ [E][[] _[z]_ [])]
= _z_ _[⊤]_ _z_ + E[ _z_ ] _[⊤]_ E[ _z_ ] _−_ 2E[ _z_ ] _[⊤]_ _z_
_≤_ _z_ _[⊤]_ _z_ + E[ _z_ ] _[⊤]_ E[ _z_ ]
_≤_ 2
where the last two inequalities are due to the fact that _z,_ E[ _z_ ] _∈_ ∆ _[T][ −]_ [1] . Then putting it all together
we have,
_| p_ ( _y_ = ˆ _y | x,_ ˜ _x_ ) _−_ _p_ soft( _y_ = ˆ _y | x,_ ˜ _x_ ) _|_ = _|_ E[ _gx,y_ ˆ( _z_ )] _−_ _gx,y_ ˆ(E[ _z_ ]) _|_
= [1]
2 _[|]_ [ E][[(] _[z][ −]_ [E][[] _[z]_ [])] _[⊤][H][g][x,][y]_ [ˆ][(ˆ] _[z]_ [)(] _[z][ −]_ [E][[] _[z]_ [])]] _[ |]_
_≤_ _c_


For data processing we closely follow the setup in [21], which uses Byte Pair Encoding over the
combined source/target training set to obtain a vocabulary size of 14,000 tokens. However, different
from [21] which uses maximum sequence length of 175, for faster training we only train on sequences
of length up to 125.
The encoder is a two-layer bi-directional LSTM with 512 units in each direction, and the decoder as
a two-layer LSTM with with 768 units. For the decoder, the convex combination of source hidden
states at each time step from the attention distribution is used as additional input at the next time step.
Word embedding is 512-dimensional.
The inference network consists of two bi-directional LSTMs (also two-layer and 512-dimensional
each) which is run over the source/target to obtain the hidden states at each time step. These hidden
states are combined using bilinear attention [45] to produce the variational parameters. (In contrast
the generative model uses MLP attention from [6], though we saw little difference between the two
parameterizations). Only the word embedding is shared between the inference network and the
generative model.
Other training details include: batch size of 6, dropout rate of 0.3, parameter initialization over a
uniform distribution _U_ [ _−_ 0 _._ 1 _,_ 0 _._ 1], gradient norm clipping at 5, and training for 30 epochs with Adam
(learning rate = 0.0003, _β_ 1 = 0.9, _β_ 2 = 0.999) [35] with a learning rate decay schedule which starts
halving the learning rate if validation perplexity does not improve. Most models converged well
before 30 epochs.
For decoding we use beam search with beam size 10 and length penalty _α_ = 1, from [79]. The length
penalty added about 0.5 BLEU points across all the models.


The model first obtains object features by mean-pooling the pretrained ResNet-101 features [27]
(which are 2048-dimensional) over object regions given by Faster R-CNN [60].The ResNet features
are kept fixed and not fine-tuned during training. We fix the maximum number of possible regions to
be 36. For the question embedding we use a one-layer LSTM with 1024 units over word embeddings.
The word embeddings are 300-dimensional and initialized with GloVe [57]. The generative model
produces a distribution over the possible objects via applying MLP attention, i.e.
_p_ ( _zi_ = 1 _| x,_ ˜ _x_ ) _∝_ exp( _w_ _[⊤]_ tanh( **W** 1 _xi_ + **W** 2 _x_ ˜))
The selected image region is concatenated with the question embedding and fed to a one-layer MLP
with ReLU non-linearity and 1024 hidden units.
The inference network produces a categorical distribution over the image regions by interacting
the answer embedding _hy_ (which are 256-dimensional and initialized randomly) with the question
embedding ˜ _x_ and the image regions _xi_,
_q_ ( _zi_ = 1) _∝_ exp( _u_ _[⊤]_ tanh( **U** 1( _xi ⊙_ ReLU( **V** 1 _hy_ )) + **U** 2(˜ _x ⊙_ ReLU( **V** 2 _hy_ ))))
where _⊙_ denotes element-wise multiplication. The generative/inference attention MLPs have 1024
hidden units each (i.e. _w, u ∈_ R [1024] ).
Other training details include: batch size of 512, dropout rate of 0.5 on the penultimate layer (i.e.
before affine transformation into answer vocabulary), and training for 50 epochs with with Adam
(learning rate = 0.0005, _β_ 1 = 0.9, _β_ 2 = 0.999) [35].
In cases where there is more than one answer for a given question/image pair, we randomly sample
the answer, where the sampling probability is proportional to the number of humans who gave the
answer.


(a) (b)
(c) (d)
(e) (f)
Figure 4: (Left Column) Further examples highlighting the difference between the prior alignment (red) and
the variational posterior (blue) when translating from DE-EN (left-to-right). The variational posterior is able to
better handle reordering; in (a) the variational posterior successfully aligns ‘turning’ to ‘verwandelt’, in (c) we
see a similar pattern with the alignment of the clause ‘that’s my brand’ to ‘das ist meine marke’. In (e) the prior
and posterior both are confused by the ‘-ial’ in ‘territor-ial’, however the posterior still remains more accurate
overall and correctly aligns the rest of ‘revierverhalten’ to ‘territorial behaviour’. (Right Column) Additional
comparisons between soft attention (green) and the prior alignments of variational attention (red). Alignments
from both models are similar, but variational attention is lower entropy. Both soft and variational attention rely
on aligning the inserted English word ‘orientation’ to the comma in (b) since a direct translation does not appear
in the German source.


Computer Science Division
University of California, Berkeley
_{_ stevencao,kitaev,klein _}_ @berkeley.edu
ABSTRACT
We propose procedures for evaluating and strengthening contextual embedding
alignment and show that they are useful in analyzing and improving multilingual
BERT. In particular, after our proposed alignment procedure, BERT exhibits significantly improved zero-shot performance on XNLI compared to the base model,
remarkably matching pseudo-fully-supervised translate-train models for Bulgarian and Greek. Further, to measure the degree of alignment, we introduce a contextual version of word retrieval and show that it correlates well with downstream
zero-shot transfer. Using this word retrieval task, we also analyze BERT and
find that it exhibits systematic deficiencies, e.g. worse alignment for open-class
parts-of-speech and word pairs written in different scripts, that are corrected by
the alignment procedure. These results support contextual alignment as a useful
concept for understanding large multilingual pre-trained models.
1 INTRODUCTION
Figure 1: t-SNE (Maaten & Hinton, 2008) visualization of the embedding space of multilingual
BERT for English-German word pairs (left: pre-alignment, right: post-alignment). Each point is a
different instance of the word in the Europarl corpus. This figure suggests that BERT begins already
somewhat aligned out-of-the-box but becomes much more aligned after our proposed procedure.
Embedding alignment was originally studied for word vectors with the goal of enabling cross-lingual
transfer, where the embeddings for two languages are in alignment if word translations, e.g. _cat_ and
_Katze_, have similar representations (Mikolov et al., 2013a; Smith et al., 2017). Recently, large pretrained models have largely subsumed word vectors based on their accuracy on downstream tasks,
partly due to the fact that their word representations are context-dependent, allowing them to more
richly capture the meaning of a word (Peters et al., 2018; Howard & Ruder, 2018; Radford et al.,
2018; Devlin et al., 2018). Therefore, with the same goal of cross-lingual transfer but for these more
complex models, we might consider contextual embedding alignment, where we observe whether
word pairs within parallel sentences, e.g. _cat_ in _“The cat sits”_ and _Katze_ in _“Die Katze sitzt,”_ have
similar representations.
Published as a conference paper at ICLR 2020
One model relevant to these questions is multilingual BERT, a version of BERT pre-trained on 104
languages that achieves remarkable transfer on downstream tasks. For example, after the model is
fine-tuned on the English MultiNLI training set, it achieves 74.3% accuracy on the test set in Spanish, which is only 7.1% lower than the English accuracy (Devlin et al., 2018; Conneau et al., 2018b).
Furthermore, while the model transfers better to languages similar to English, it still achieves reasonable accuracies even on languages with different scripts.
However, given the way that multilingual BERT was pre-trained, it is unclear why we should expect
such high zero-shot performance. Compared to monolingual BERT which exhibits no zero-shot
transfer, multilingual BERT differs only in that (1) during pre-training (i.e. masked word prediction),
each batch contains sentences from all of the languages, and (2) it uses a single shared vocabulary,
formed by WordPiece on the concatenated monolingual corpora (Devlin et al., 2019). Therefore,
we might wonder: (1) How can we better understand BERT’s multilingualism? (2) Can we further
improve BERT’s cross-lingual transfer?
In this paper, we show that contextual embedding alignment is a useful concept for addressing
these questions. First, we propose a contextual version of word retrieval to evaluate the degree
of alignment, where a model is presented with two parallel corpora, and given a word within a
sentence in one corpus, it must find the correct word and sentence in the other. Using this metric
of alignment, we show that multilingual BERT achieves zero-shot transfer because its embeddings
are partially aligned, as depicted in Figure 1, with the degree of alignment predicting the degree of
downstream transfer.
Next, using between 10K and 250K sentences per language from the Europarl corpus as parallel
data (Koehn, 2005), we propose a fine-tuning-based alignment procedure and show that it significantly improves BERT as a multilingual model. Specifically, on zero-shot XNLI, where the model
is trained on English MultiNLI and tested on other languages (Conneau et al., 2018b), the aligned
model improves accuracies by 2.78% on average over the base model, and it remarkably matches
translate-train models for Bulgarian and Greek, which approximate the fully-supervised setting.
To put our results in the context of past work, we also use word retrieval to compare our finetuning procedure to two alternatives: (1) fastText augmented with sentence and aligned using rotations (Bojanowski et al., 2017; R¨uckl´e et al., 2018; Artetxe et al., 2018), and (2) BERT aligned using
rotations (Aldarmaki & Diab, 2019; Schuster et al., 2019; Wang et al., 2019). We find that when
there are multiple occurences per word, fine-tuned BERT outperforms fastText, which outperforms
rotation-aligned BERT. This result supports the intuition that contextual alignment is more difficult
than its non-contextual counterpart, given that a rotation, at least when applied naively, is no longer
sufficient to produce strong alignments. In addition, when there is only one occurrence per word,
fine-tuned BERT matches the performance of fastText. Given that context disambiguation is no
longer necessary, this result suggests that our fine-tuning procedure is able to align BERT at the type
level to a degree that matches non-contextual approaches.
Finally, we use the contextual word retrieval task to conduct finer-grained analysis of multilingual
BERT, with the goal of better understanding its strengths and shortcomings. Specifically, we find
that base BERT has trouble aligning open-class compared to closed-class parts-of-speech, as well
as word pairs that have large differences in usage frequency, suggesting insight into the pre-training
procedure that we explore in Section 5. Together, these experiments support contextual alignment
as an important task that provides useful insight into large multilingual pre-trained models.
2 RELATED WORK
**Word vector alignment.** There has been a long line of works that learn aligned word vectors
from varying levels of supervision (Ruder et al., 2019). One popular family of methods starts with
word vectors learned independently for each language (using a method like skip-gram with negative
sampling (Mikolov et al., 2013b)), and it learns a mapping from source language vectors to target
language vectors with a bilingual dictionary as supervision (Mikolov et al., 2013a; Smith et al.,
2017; Artetxe et al., 2017). When the mapping is constrained to be an orthogonal linear transformation, the optimal mapping that minimizes distances between word pairs can be solved in closed
form (Artetxe et al., 2016; Schonemann, 1966). Alignment is evaluated using bilingual lexicon induction, so these papers also propose ways to mitigate the hubness problem in nearest neighbors,
Published as a conference paper at ICLR 2020
e.g. by using alternate similarity functions like CSLS (Conneau et al., 2018a). A recent set of works
has also shown that the mapping can be learned with minimal to no supervision by starting with
some minimal seed dictionary and alternating between learning the linear map and inducing the dictionary (Artetxe et al., 2018; Conneau et al., 2018a; Hoshen & Wolf, 2018; Xu et al., 2018; Chen &
Cardie, 2018).
**Incorporating context into alignment.** One key challenge in making alignment context aware is
that the embeddings are now different across multiple occurrences of the same word. Past papers
have handled this issue by removing context and aligning the “average sense” of a word. In one
such study, Schuster et al. (2019) learn a rotation to align contextual ELMo embeddings (Peters
et al., 2018) with the goal of improving zero-shot multilingual dependency parsing, and they handle
context by taking the average embedding for a word in all of its contexts. In another paper, Aldarmaki & Diab (2019) learn a rotation on sentence vectors, produced by taking the average word
vector over the sentence, and they show that the resulting alignment also works well for word-level
tasks. In a contemporaneous work, Wang et al. (2019) align not only the word but also the context
by learning a linear transformation using word-aligned parallel data to align multilingual BERT,
with the goal of improving zero-shot dependency parsing numbers. In this paper, we similarly align
not only the word but also the context, and we also depart from these past works by using more
expressive alignment methods than rotation.
**Incorporating parallel texts into pre-training.** Instead of performing alignment post-hoc, another line of works proposes contextual pre-training procedures that are more cross-lingually-aware.
Wieting et al. (2019) pre-train sentence embeddings using parallel texts by maximizing similarity between sentence pairs while minimizing similarity with negative examples. Lample & Conneau (2019) propose a cross-lingual pre-training objective that incorporates parallel data in addition to monolingual corpora, leading to improved downstream cross-lingual transfer. In contrast,
our method uses less parallel data and aligns existing pre-trained models rather than requiring pretraining from scratch.
**Analyzing multilingual BERT.** Pires et al. (2019) present a series of probing experiments to better
understand multilingual BERT, and they find that transfer is possible even between dissimilar languages, but that it works better between languages that are typologically similar. They conclude that
BERT is remarkably multilingual but falls short for certain language pairs.
3 METHODS
3.1 MULTILINGUAL PRE-TRAINING
We first briefly describe multilingual BERT (Devlin et al., 2018). Like monolingual BERT, multilingual BERT is pre-trained on sentences from Wikipedia to perform two tasks: masked word
prediction, where it must predict words that are masked within a sentence, and next sentence prediction, where it must predict whether the second sentence follows the first one. The model is trained
on 104 languages, with each batch containing training sentences from each language, and it uses a
shared vocabulary formed by WordPiece on the 104 Wikipedias concatenated (Wu et al., 2016).
3.2 DEFINING AND EVALUATING CONTEXTUAL ALIGNMENT
In the following sections, we describe how to define, evaluate, and improve contextual alignment. Given two languages, a model is in _contextual alignment_ if it has similar representations
for word pairs within parallel sentences. More precisely, suppose we have _N_ parallel sentences
_C_ = _{_ ( **s** [1] _,_ **t** [1] ) _, ...,_ ( **s** _[N]_ _,_ **t** _[N]_ ) _}_, where ( **s** _,_ **t** ) is a source-target sentence pair. Also, let each sentence
pair ( **s** _,_ **t** ) have word pairs, denoted _a_ ( **s** _,_ **t** ) = _{_ ( _i_ 1 _, j_ 1) _, ...,_ ( _im, jm_ ) _}_, containing position tuples
( _i, j_ ) such that the words **s** _i_ and **t** _j_ are translations of each other. [1] We will use _f_ to represent a
pre-trained model such that _f_ ( _i,_ **s** ) is the contextual embedding for the _i_ th word in **s** .
1These pairs are called word alignments in the machine translation community, but we use the term “word
pairs” to avoid confusion with embedding alignment. Also, because BERT operates on subwords while the
corpus is aligned at the word level, we keep only the BERT vector for the last subword of each word.
Published as a conference paper at ICLR 2020
As an example, we might have the following sentence pair:
0 1 2 3 4 0 1 2 3 4 5
**s** = _{I_ _ate_ _the_ _apple_ _.}_ **t** = _{Ich_ _habe_ _den_ _Apfel_ _gegessen_ _.}_
_a_ ( **s** _,_ **t** ) = _{_ (0 _,_ 0) _,_ (1 _,_ 4) _,_ (2 _,_ 2) _,_ (3 _,_ 3) _,_ (4 _,_ 5) _}_
Then, using the parallel corpus _C_, we can measure the contextual alignment of the model _f_ using its
accuracy in _contextual word retrieval_ . In this task, the model is presented with two parallel corpora,
and given a word within a sentence in one corpus, it must find the correct word and sentence in the
other. Specifically, we can define a nearest neighbor retrieval function
neighbor( _i,_ **s** ; _f, C_ ) = argmax sim( _f_ ( _i,_ **s** ) _, f_ ( _j,_ **t** )) _,_
**t** _∈C,_ 0 _≤j≤_ len( **t** )
where _i_ and _j_ denote the position within a sentence and sim is a similarity function. The accuracy
is then given by the percentage of exact matches over the entire corpus, or
_A_ ( _f_ ; _C_ ) = [1]
_N_
( **s** _,_ **t** ) _∈C_
I(neighbor( _i,_ **s** ; _f, C_ ) = ( _j,_ **t** )) _,_
( _i,j_ ) _∈a_ ( **s** _,_ **t** )
where I represents the indicator function. We can perform the same procedure in the other direction,
where we pull target words given source words, so we report the average between the two directions.
As our similarity function, we use CSLS, a modified version of cosine similarity that mitigates
the hubness problem, with neighborhood size 10 (Conneau et al., 2018a). One additional point is
that this procedure can be made more or less contextual based on the corpus: a corpus with more
occurrences for each word type requires better representations of context. Therefore, we also test
non-contextual word retrieval by removing all but the first occurrence of each word type.
Given parallel data, these word pairs can be procured in an unsupervised fashion using standard
techniques developed by the machine translation community (Brown et al., 1993). While these
methods can be noisy, by running the algorithm in both the source-target and target-source directions
and only keeping word pairs in their intersection, we can trade-off coverage for accuracy, producing
a reasonably high-precision dataset (Och & Ney, 2003).
3.3 ALIGNING PRE-TRAINED CONTEXTUAL EMBEDDINGS
To improve the alignment of the model _f_ with respect to the corpus _C_, we can encapsulate alignment
in the loss function
_L_ ( _f_ ; _C_ ) = _−_
( **s** _,_ **t** ) _∈C_
 - sim( _f_ ( _i,_ **s** ) _, f_ ( _j,_ **t** )) _,_
( _i,j_ ) _∈a_ ( **s** _,_ **t** )
where we sum the similarities between word pairs. Because the CSLS metric is not easily optimized,
we instead use the squared error loss, or sim( _f_ ( _i,_ **s** ) _, f_ ( _j,_ **t** )) = _−||f_ ( _i,_ **s** ) _−_ _f_ ( _j,_ **t** ) _||_ [2] 2 [.]
However, note that this loss function does not account for the informativity of _f_ ; for example, it is
zero if _f_ is constant. Therefore, at a high level, we would like to minimize _L_ ( _f_ ; _C_ ) while maintaining some aspect of _f_ that makes it useful, e.g. its high accuracy when fine-tuned on downstream
tasks. Letting _f_ 0 denote the initial pre-trained model before alignment, we achieve this goal by
defining a regularization term
_R_ ( _f_ ; _C_ ) =
**t** _∈C_
len( **t** )
- _||f_ ( _j,_ **t** ) _−_ _f_ 0( _j,_ **t** ) _||_ [2] 2 _[,]_
_i_ =1
which imposes a penalty if the target language embeddings stray from their initialization. Then,
we sample minibatches _B ⊂_ _C_ and take gradient steps of the function _L_ ( _f_ ; _B_ ) + _λR_ ( _f_ ; _B_ ) directly on the weights of _f_, which moves the source embeddings toward the target embeddings while
preventing the latter from drifting too far. In our experiments, we set _λ_ = 1.
In the multilingual case, suppose we have _k_ parallel corpora _C_ [1] _, ..., C_ _[k]_, where each corpus has a
different source language with the target language as English. Then, we sample equal-sized batches
_B_ _[i]_ _⊂_ _C_ _[i]_ from each corpus and take gradient steps on [�] _i_ _[k]_ =1 _[L]_ [(] _[f]_ [;] _[ B][i]_ [) +] _[ λR]_ [(] _[f]_ [;] _[ B][i]_ [)][, which moves]
all of the non-English embeddings toward English.
Published as a conference paper at ICLR 2020
Note that this alignment method departs from prior work, in which each non-English language is
rotated to match the English embedding space through individual learned matrices. Specifically, the
most widely used post-hoc alignment method learns a rotation _W_ applied to the source vectors to
minimize the distance between parallel word pairs, or
 - _||Wf_ ( _i,_ **s** ) _−_ _f_ ( _j,_ **t** ) _||_ [2] 2 _s.t._ _W_ _[⊤]_ _W_ = _I._ (1)
( _i,j_ ) _∈a_ ( **s** _,_ **t** )
min
_W_
( **s** _,_ **t** ) _∈C_
This problem is known as the Procrustes problem and can be solved in closed form (Schonemann,
1966). This approach has the nice property that the vectors are only rotated, preserving distances
and therefore the semantic information captured by the embeddings (Artetxe et al., 2016). However,
rotation requires the strong assumption that the embedding spaces are roughly isometric (Søgaard
et al., 2018), an assumption that may not hold for contextual pre-trained models because they represent more aspects of a word than just its type, i.e. context and syntax, which are less likely to
be isomorphic between languages. Given that past work has also found independent alignment per
language pair to be inferior to joint training (Heyman et al., 2019), another advantage of our method
is that the alignment for all languages is done simultaneously.
As our dataset, we use the Europarl corpora for English paired with Bulgarian, German, Greek,
Spanish, and French, the languages represented in both Europarl and XNLI (Koehn, 2005). After
tokenization (Koehn et al., 2007), we produce word pairs using fastAlign and keep the one-to-one
pairs in the intersection (Dyer et al., 2013). We use the most recent 1024 sentences as the test set, the
previous 1024 sentences as the development set, and the following 250K sentences as the training
set. Furthermore, we modify the test set accuracy calculation to only include word pairs not seen in
the training set. We also remove any exact matches, e.g. punctuation and numbers, because BERT is
already aligned for these pairs due to its shared vocabulary. Given that parallel data may be limited
for low-resource language pairs, we also report numbers for 10K and 50K parallel sentences.
3.4 SENTENCE-AUGMENTED NON-CONTEXTUAL BASELINE
Given that there has been a long line of work on word vector alignment (Artetxe et al., 2018; Conneau et al., 2018a; Smith et al., 2017, _inter alia_ ), we also compare BERT to a sentence-augmented
fastText baseline (Bojanowski et al., 2017). Following Artetxe et al. (2018), we first normalize, then
mean-center, then normalize the word vectors, and we then learn a rotation with the same parallel
data as in the contextual case, as described in Equation 1. We also strengthen this baseline by including sentence information: specifically, during word retrieval, we concatenate each word vector
with a vector representing its sentence. Following R¨uckl´e et al. (2018), we compute the sentence
vector by concatenating the average, maximum, and minimum vector over all of the words in the
sentence, a method that was shown to be state-of-the-art for a suite of cross-lingual tasks. We also
experimented with other methods, such as first retrieving the sentence and then the word, but found
this method resulted in the highest accuracy. As a result, the fastText vectors are 1200-dimensional,
while the BERT vectors are 768-dimensional.
3.5 TESTING ZERO-SHOT TRANSFER
The next step is to determine whether better alignment improves cross-lingual transfer. As our
downstream task, we use the XNLI dataset, where the English MultiNLI development and test sets
are human-translated into multiple languages (Conneau et al., 2018b; Williams et al., 2018). Given
a pair of sentences, the task is to predict whether the first sentence implies the second, where there
are three labels: entailment, neutral, or contradiction. Starting from either the base or aligned multilingual BERT, we train on English and evaluate on Bulgarian, German, Greek, Spanish, and French,
the XNLI languages represented in Europarl.
As our architecture, following Devlin et al. (2018), we apply a linear layer followed by softmax
on the [CLS] embedding of the sentence pair, producing scores for each of the three labels. The
model is trained using cross-entropy loss and selected based on its development set accuracy averaged across all of the languages. As a fully-supervised ceiling, we also compare to models trained
and tested on the same language, where for the non-English training data, we use the machine translations of the English MultiNLI training data provided by Conneau et al. (2018b). While the quality
of the training data is affected by the quality of the MT system, this comparison nevertheless serves
as a good approximation for the fully-supervised setting.
Published as a conference paper at ICLR 2020
English Bulgarian German Greek Spanish French Average
Translate-Train
Base BERT 81.9 73.6 75.9 71.6 77.8 76.8 76.3
Zero-Shot _[a]_
Base BERT 80.4 68.7 70.4 67.0 74.5 73.4 72.4
Sentence-aligned BERT (rotation) **81.1** 68.9 71.2 66.7 74.9 73.5 72.7
Word-aligned BERT (rotation) 78.8 69.0 71.3 67.1 74.3 73.0 72.2
Word-aligned BERT (fine-tuned) 80.1 **73.4** **73.1** **71.4** **75.5** **74.5** **74.7**
XLM (MLM + TLM) 85.0 77.4 77.8 76.6 78.9 78.7 79.1
Table 1: Accuracy on the XNLI test set, where we compare to base BERT (Devlin et al., 2018)
and two rotation-based methods, sentence alignment (Aldarmaki & Diab, 2019) and word alignment (Wang et al., 2019). We also include the current state-of-the-art zero-shot achieved by
XLM (Lample & Conneau, 2019). Rotation-based methods provide small gains on some languages
but not others. On the other hand, after fine-tuning-based alignment, Bulgarian and Greek match the
translate-train ceiling, while German, Spanish, and French close roughly one-third of the gap.
_a_ Note that the zero-shot Base BERT numbers are slightly different from those reported in Devlin et al.
(2019) because we select a single model using the average accuracy across the six languages. This selection
method also accounts for the varying English accuracies across the zero-shot methods.
Sentences English Bulgarian German Greek Spanish French Average
None 80.4 68.7 70.4 67.0 74.5 73.4 72.4
10K 79.2 71.0 71.8 67.5 75.3 74.1 73.2
50K **81.1** 73 72.6 69.6 75 **74.5** 74.3
250K 80.1 **73.4** **73.1** **71.4** **75.5** **74.5** **74.7**
Table 2: Zero-shot accuracy on the XNLI test set, where we align BERT with varying amounts of
parallel data. The method scales with the amount of data but achieves a large fraction of the gains
with 50K sentences per language pair.
4 RESULTS
4.1 ZERO-SHOT XNLI TRANSFER
First, we test whether alignment improves multilingual BERT by applying the models to zero-shot
XNLI, as displayed in Table 1. We see that our alignment procedure greatly improves accuracies,
with all languages seeing a gain of at least 1%. In particular, the Bulgarian and Greek zero-shot
numbers are boosted by almost 5% each and match the translate-train numbers, suggesting that the
alignment procedure is especially effective for languages that are initially difficult for BERT. We
also run alignment for more distant language pairs (Chinese, Arabic, Urdu) and find similar results,
which we report in the appendix.
Comparing to rotation-based methods (Aldarmaki & Diab, 2019; Wang et al., 2019), we find that a
rotation produces small gains for some languages, namely Bulgarian, German, and Spanish, but is
sub-optimal overall, providing evidence that the increased expressivity of our proposed procedure is
beneficial for contextual alignment. We explore this comparison more in Section 5.1.
4.2 ALIGNMENT WITH LESS DATA
Given that our goal is zero-shot transfer, we cannot expect to always have large amounts of parallel data. Therefore, we also characterize the performance of our alignment method with varying
amounts of data, as displayed in Table 2. We find that it improves transfer with as little as 10K
sentences per language, making it a promising approach for low-resource languages.
Published as a conference paper at ICLR 2020
bg-en de-en el-en es-en fr-en Average
Contextual
Aligned fastText + sentence 44.0 46.4 42.0 48.6 44.5 45.1
Base BERT 19.5 26.1 13.9 32.5 28.3 24.1
Word-aligned BERT (rotation) 29.8 31.6 20.8 36.8 31.0 30.0
Word-aligned BERT (fine-tuned) **50.7** **51.3** **49.8** **51.0** **48.6** **50.3**
Non-Contextual
Aligned fastText + sentence 61.3 **65.4** 61.6 **71.1** 64.8 64.8
Base BERT 29.1 37.0 22.3 46.5 41.8 35.3
Word-aligned BERT (rotation) 39.6 43.6 32.4 51.4 46.1 42.6
Word-aligned BERT (fine-tuned) **62.8** 64.3 **67.5** 68.4 **66.3** **65.9**
Table 3: Word retrieval accuracy for the aligned sentence-augmented fastText baseline and BERT
pre- and post-alignment. Across languages, base BERT has variable accuracy while fine-tuningaligned BERT is consistently effective. Fine-tuned BERT also matches fastText in a version of the
task where context is not necessary, suggesting that our method matches the type-level alignment of
fastText while also aligning context.
5 ANALYSIS
5.1 WORD RETRIEVAL
In the following sections, we present word retrieval results to both compare our method to past work
and better understand the strengths and weaknesses of multilingual BERT. Table 3 displays the word
retrieval accuracies for the aligned sentence-augmented fastText baseline and BERT pre- and postalignment. First, we find that in contextual retrieval, fine-tuned BERT outperforms fastText, which
outperforms rotation-aligned BERT. This result supports the intuition that aligning large pre-trained
models is more difficult than aligning word vectors, given that a rotation, at least when applied
naively, produces sub-par alignments. In addition, fine-tuned BERT matches the performance of
fastText in non-contextual retrieval, suggesting that our alignment procedure overcomes these challenges and achieves type-level alignment that matches non-contextual approaches. In the appendix,
we also provide examples of aligned BERT disambiguating between different meanings of a word,
giving qualitative evidence of the benefit of context alignment.
We also find that before alignment, BERT’s performance varies greatly between languages, while
after alignment it is consistently effective. In particular, Bulgarian and Greek initially have very
low accuracies. This phenomenon is also reflected in the XNLI numbers (Table 1), where Bulgarian
and Greek receive the largest boosts from alignment. Examining the connection between alignment
and zero-shot more closely, we find that the word retrieval accuracies are highly correlated with
downstream zero-shot performance (Figure 2), supporting our evaluation measure as predictive of
cross-lingual transfer.
The language discrepancies are also consistent with a hypothesis by Pires et al. (2019) to explain
BERT’s multilingualism. They posit that due to the shared vocabulary, shared words between languages, e.g. numbers and names, are forced to have the same representation. Then, due to the
masked word prediction task, other words that co-occur with these shared words also receive similar
representations. If this hypothesis is true, then languages with higher lexical overlap with English are
likely to experience higher transfer. As an extreme form of this phenomenon, Bulgarian and Greek
have completely different scripts and should experience worse transfer than the common-script languages, an intuition that is confirmed by the word retrieval and XNLI accuracies. The fact that all
languages are equally aligned with English post-alignment suggests that the pre-training procedure
is suboptimal for these languages.
Published as a conference paper at ICLR 2020
Lexical Overlap Numeral Punctuation Proper Noun Average
Base BERT 0.90 0.88 0.80 0.86
Aligned BERT 0.97 0.96 0.95 0.96
Closed-Class Determiner Preposition Conjunction Pronoun Auxiliary Average
Base BERT 0.76 0.72 0.71 0.70 0.61 0.70
Aligned BERT 0.91 0.86 0.89 0.89 0.84 0.88
Open-Class Noun Adverb Adjective Verb Average
Base BERT 0.61 0.57 0.50 0.49 0.54
Aligned BERT 0.90 0.88 0.90 0.89 0.89
Table 4: Accuracy by part-of-speech tag for non-contextual word retrieval. To achieve better
word type coverage, we do not remove word pairs seen in the training set. The tags are grouped into
lexically overlapping, closed-class, and open-class groups. The “Particle,” “Symbol,” “Interjection,”
and “Other” tags are omitted.
15 20 25 30
Contextual word retrieval accuracy
Figure 2: XNLI zero-shot versus word retrieval accuracy for base BERT, where each
point is a language paired with English.
This plot suggests that alignment correlates
well with cross-lingual transfer.
|1.00<br>0.95<br>0.90<br>0.85<br>0.80<br>0.75<br>0.70<br>0.65<br>0.60|Aligned BERT<br>Base BERT|
|---|---|
|0.60<br>0.65<br>0.70<br>0.75<br>0.80<br>0.85<br>0.90<br>0.95<br>1.00|100<br>101<br>102<br>103<br>104|
Figure 3: Contextual word retrieval accuracy plotted
against difference in frequency rank between source
and target. The accuracy of base BERT plummets for
larger differences, suggesting that its alignment depends on word pairs having similar usage statistics.
5.2 WORD RETRIEVAL PART-OF-SPEECH ANALYSIS
Next, to gain insight into the multilingual pre-training procedure, we analyze the accuracy broken
down by part-of-speech using the Universal Part-of-Speech Tagset (Petrov et al., 2012), annotated
using polyglot (Al-Rfou et al., 2013) for Bulgarian and spaCy (Honnibal & Montani, 2017) for the
other languages, as displayed in Table 4. Unsurprisingly, multilingual BERT has high alignment
out-of-the-box for groups with high lexical overlap, e.g. numerals, punctuation, and proper nouns,
due to its shared vocabulary. We further divide the remaining tags into closed-class and open-class,
where closed-class parts-of-speech correspond to fixed sets of words serving grammatical functions
(e.g. determiner, preposition, conjunction, pronoun, and auxiliary), while open-class parts-of-speech
correspond to lexical words (e.g. noun, adverb, adjective, verb). Interestingly, we see that base BERT
has consistently lower accuracy for closed-class versus open-class categories (0 _._ 70 vs 0 _._ 54), but that
this discrepancy disappears after alignment (0 _._ 88 vs 0 _._ 89).
5.3 USAGE HYPOTHESIS FOR ALIGNMENT
From this closed-class vs open-class difference, we hypothesize that BERT’s alignment of a particular word pair is influenced by the similarity of their usage statistics. Specifically, given that
BERT is trained through masked word prediction, its embeddings are in large part determined by
Published as a conference paper at ICLR 2020
the co-occurrences between words. Therefore, two words that are used in similar contexts should be
better aligned. This hypothesis provides an explanation of the closed-class vs open-class difference:
closed-class words are typically grammatical, so they are used in similar ways across typologically
similar languages. Furthermore, these words cannot be substituted for one another due to their
grammatical function. Therefore, their usage statistics are a strong signature that can be used for
alignment. On the other hand, open-class words can be substituted for one another: for example, in
most sentences, the noun tokens could be replaced by a wide range of semantically dissimilar nouns
with the sentence remaining syntactically well-formed. By this effect, many nouns have similar
co-occurrences, making them difficult to align through masked word prediction alone.
To further test this hypothesis, we plot the word retrieval accuracy versus the difference between the
frequency rank of the target and source word, where this difference measures discrepancies in usage,
as depicted in Figure 3. We see that accuracy drops off significantly as the source-target difference
increases, supporting our hypothesis. Furthermore, this shortcoming is remedied by alignment,
revealing another systematic deficiency of multilingual pre-training.
6 CONCLUSION
Given that the degree of alignment is causally predictive of downstream cross-lingual transfer, contextual alignment proves to be a useful concept for understanding and improving multilingual pretrained models. Given small amounts of parallel data, our alignment procedure improves multilingual BERT and corrects many of its systematic deficiencies. Contextual word retrieval also provides
useful new insights into the pre-training procedure, opening up new avenues for analysis.
REFERENCES
Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. Polyglot: Distributed word representations for
multilingual nlp. In _Proceedings of the Seventeenth Conference on Computational Natural Lan-_
_guage Learning_, pp. 183–192, Sofia, Bulgaria, August 2013. Association for Computational Lin[guistics. URL http://www.aclweb.org/anthology/W13-3520.](http://www.aclweb.org/anthology/W13-3520)
Hanan Aldarmaki and Mona Diab. Context-aware cross-lingual mapping. In _Proceedings of the 2019_
_Conference of the North American Chapter of the Association for Computational Linguistics:_
_Human Language Technologies, Volume 1 (Long and Short Papers)_, pp. 3906–3911, Minneapolis,
Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1391.
[URL https://www.aclweb.org/anthology/N19-1391.](https://www.aclweb.org/anthology/N19-1391)
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Learning principled bilingual mappings of word
embeddings while preserving monolingual invariance. In _Proceedings of the 2016 Conference on_
_Empirical Methods in Natural Language Processing_, pp. 2289–2294, Austin, Texas, November
[2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1250. URL https:](https://www.aclweb.org/anthology/D16-1250)
[//www.aclweb.org/anthology/D16-1250.](https://www.aclweb.org/anthology/D16-1250)
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Learning bilingual word embeddings with (almost)
no bilingual data. In _Proceedings of the 55th Annual Meeting of the Association for Computational_
_Linguistics (Volume 1: Long Papers)_, pp. 451–462, Vancouver, Canada, July 2017. Association
[for Computational Linguistics. doi: 10.18653/v1/P17-1042. URL https://www.aclweb.](https://www.aclweb.org/anthology/P17-1042)
[org/anthology/P17-1042.](https://www.aclweb.org/anthology/P17-1042)
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In _Proceedings of the 56th Annual Meet-_
_ing of the Association for Computational Linguistics (Volume 1: Long Papers)_, pp. 789–798,
Melbourne, Australia, July 2018. Association for Computational Linguistics. [URL https:](https://www.aclweb.org/anthology/P18-1073)
[//www.aclweb.org/anthology/P18-1073.](https://www.aclweb.org/anthology/P18-1073)
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors
with subword information. _Transactions of the Association for Computational Linguistics_, 5:135–
146, 2017. doi: 10.1162/tacl ~~a 0~~ [0051. URL https://www.aclweb.org/anthology/](https://www.aclweb.org/anthology/Q17-1010)
[Q17-1010.](https://www.aclweb.org/anthology/Q17-1010)
Published as a conference paper at ICLR 2020
Peter F. Brown, Vincent J. Della Pietra, Stephen A. Della Pietra, and Robert L. Mercer. The
mathematics of statistical machine translation: Parameter estimation. _Comput. Linguist._, 19(2):
[263–311, June 1993. ISSN 0891-2017. URL http://dl.acm.org/citation.cfm?id=](http://dl.acm.org/citation.cfm?id=972470.972474)
[972470.972474.](http://dl.acm.org/citation.cfm?id=972470.972474)
Xilun Chen and Claire Cardie. Unsupervised multilingual word embeddings. In _Proceedings of the_
_2018 Conference on Empirical Methods in Natural Language Processing_, pp. 261–270, Brussels,
Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/
[v1/D18-1024. URL https://www.aclweb.org/anthology/D18-1024.](https://www.aclweb.org/anthology/D18-1024)
Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Herve J’egou.
Word translation without parallel data. In _Proceedings of the 6th International Conference on_
_Learning Representations (ICLR 2018)_ [, 2018a. URL https://arxiv.org/pdf/1710.](https://arxiv.org/pdf/1710.04087.pdf)
[04087.pdf.](https://arxiv.org/pdf/1710.04087.pdf)
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger
Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross-lingual sentence representations. In
_Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pp.
2475–2485, Brussels, Belgium, October-November 2018b. Association for Computational Lin[guistics. doi: 10.18653/v1/D18-1269. URL https://www.aclweb.org/anthology/](https://www.aclweb.org/anthology/D18-1269)
[D18-1269.](https://www.aclweb.org/anthology/D18-1269)
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep
bidirectional transformers for language understanding. _
[URL http://arxiv.org/abs/1810.04805.](http://arxiv.org/abs/1810.04805)
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training
of deep bidirectional transformers for language understanding. [https://github.com/](https://github.com/google-research/bert/blob/master/multilingual.md)
[google-research/bert/blob/master/multilingual.md, 2019.](https://github.com/google-research/bert/blob/master/multilingual.md)
Chris Dyer, Victor Chahuneau, and Noah A. Smith. A simple, fast, and effective reparameterization
of IBM model 2. In _Proceedings of the 2013 Conference of the North American Chapter of_
_the Association for Computational Linguistics: Human Language Technologies_, pp. 644–648,
[Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL https://www.](https://www.aclweb.org/anthology/N13-1073)
[aclweb.org/anthology/N13-1073.](https://www.aclweb.org/anthology/N13-1073)
Andreas Eisele and Yu Chen. MultiUN: A multilingual corpus from united nation documents.
In _Proceedings of the Seventh International Conference on Language Resources and Eval-_
_uation (LREC’10)_, Valletta, Malta, May 2010. European Language Resources Association
[(ELRA). URL http://www.lrec-conf.org/proceedings/lrec2010/pdf/686_](http://www.lrec-conf.org/proceedings/lrec2010/pdf/686_Paper.pdf)
[Paper.pdf.](http://www.lrec-conf.org/proceedings/lrec2010/pdf/686_Paper.pdf)
Geert Heyman, Bregt Verreet, Ivan Vuli´c, and Marie-Francine Moens. Learning unsupervised multilingual word embeddings with incremental multilingual hubs. In _Proceedings of the 2019 Con-_
_ference of the North American Chapter of the Association for Computational Linguistics: Human_
_Language Technologies, Volume 1 (Long and Short Papers)_, pp. 1890–1902, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1188. URL
[https://www.aclweb.org/anthology/N19-1188.](https://www.aclweb.org/anthology/N19-1188)
Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 2017.
Yedid Hoshen and Lior Wolf. Non-adversarial unsupervised word translation. In _Proceedings of the_
_2018 Conference on Empirical Methods in Natural Language Processing_, pp. 469–478, Brussels,
Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/
[v1/D18-1043. URL https://www.aclweb.org/anthology/D18-1043.](https://www.aclweb.org/anthology/D18-1043)
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification.
In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_
_(Volume 1: Long Papers)_, pp. 328–339. Association for Computational Linguistics, 2018. URL
[http://aclweb.org/anthology/P18-1031.](http://aclweb.org/anthology/P18-1031)
Published as a conference paper at ICLR 2020
Philipp Koehn. Europarl: A parallel corpus for statistical machine translation. In _Conference Pro-_
_ceedings: The Tenth Machine Translation Summit_, pp. 79–86, Phuket, Thailand, 2005. AAMT.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola
Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondˇrej Bojar, Alexandra Constantin, and Evan Herbst. Moses: Open source toolkit for statistical machine translation. In _Proceedings of the 45th Annual Meeting of the Association for Com-_
_putational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions_, pp.
177–180, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL
[https://www.aclweb.org/anthology/P07-2045.](https://www.aclweb.org/anthology/P07-2045)
Guillame Lample and Alexis Conneau. Cross-lingual language model pretraining. 2019. URL
[https://arxiv.org/pdf/1901.07291.pdf.](https://arxiv.org/pdf/1901.07291.pdf)
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. _Journal of Ma-_
_chine Learning Research_ [, 9:2579–2605, 2008. URL http://www.jmlr.org/papers/v9/](http://www.jmlr.org/papers/v9/vandermaaten08a.html)
[vandermaaten08a.html.](http://www.jmlr.org/papers/v9/vandermaaten08a.html)
Tomas Mikolov, Quoc V Le, and Ilya Sutskever. Exploiting similarities among languages for ma[chine translation. 2013a. URL https://arxiv.org/pdf/1309.4168.pdf.](https://arxiv.org/pdf/1309.4168.pdf)
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In _Proceedings of the 26th International_
_Conference on Neural Information Processing Systems - Volume 2_, NIPS’13, pp. 3111–3119,
USA, 2013b. Curran Associates Inc. [URL http://dl.acm.org/citation.cfm?id=](http://dl.acm.org/citation.cfm?id=2999792.2999959)
[2999792.2999959.](http://dl.acm.org/citation.cfm?id=2999792.2999959)
Franz Josef Och and Hermann Ney. A systematic comparison of various statistical alignment
models. _Comput. Linguist._, 29(1):19–51, March 2003. ISSN 0891-2017. doi: 10.1162/
[089120103321337421. URL http://dx.doi.org/10.1162/089120103321337421.](http://dx.doi.org/10.1162/089120103321337421)
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and
Luke Zettlemoyer. Deep contextualized word representations. In _Proceedings of the 2018 Con-_
_ference of the North American Chapter of the Association for Computational Linguistics: Hu-_
_man Language Technologies, Volume 1 (Long Papers)_, pp. 2227–2237, New Orleans, Louisiana,
June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-1202. URL
[https://www.aclweb.org/anthology/N18-1202.](https://www.aclweb.org/anthology/N18-1202)
Slav Petrov, Dipanjan Das, and Ryan McDonald. A universal part-of-speech tagset. In _Proceed-_
_ings of the Eighth International Conference on Language Resources and Evaluation (LREC-_
_2012)_, pp. 2089–2096, Istanbul, Turkey, May 2012. European Languages Resources Association
[(ELRA). URL http://www.lrec-conf.org/proceedings/lrec2012/pdf/274_](http://www.lrec-conf.org/proceedings/lrec2012/pdf/274_Paper.pdf)
[Paper.pdf.](http://www.lrec-conf.org/proceedings/lrec2012/pdf/274_Paper.pdf)
Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual BERT? In
_Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_,
pp. 4996–5001, Florence, Italy, July 2019. Association for Computational Linguistics. URL
[https://www.aclweb.org/anthology/P19-1493.](https://www.aclweb.org/anthology/P19-1493)
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. URL [https:](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf)
[//s3-us-west-2.amazonaws.com/openai-assets/research-covers/](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf)
[language-unsupervised/language_understanding_paper.pdf.](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf)
Andreas R¨uckl´e, Steffen Eger, Maxime Peyrard, and Iryna Gurevych. Concatenated p-mean word
embeddings as universal cross-lingual sentence representations. _
[URL http://arxiv.org/abs/1803.01400.](http://arxiv.org/abs/1803.01400)
Sebastian Ruder, Ivan Vuli´c, and Anders Søgaard. A survey of cross-lingual word embedding models. _J. Artif. Int. Res._, 65(1):569–630, May 2019. ISSN 1076-9757. doi: 10.1613/jair.1.11640.
[URL https://doi.org/10.1613/jair.1.11640.](https://doi.org/10.1613/jair.1.11640)
Published as a conference paper at ICLR 2020
Peter H. Schonemann. A generalized solution of the orthogonal procrustes problem. _Psychometrika_,
31(1):1–10, 1966.
Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. Cross-lingual alignment of contextual
word embeddings, with applications to zero-shot dependency parsing. In _Proceedings of the 2019_
_Conference of the North American Chapter of the Association for Computational Linguistics:_
_Human Language Technologies, Volume 1 (Long and Short Papers)_, pp. 1599–1613, Minneapolis,
Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1162.
[URL https://www.aclweb.org/anthology/N19-1162.](https://www.aclweb.org/anthology/N19-1162)
Samuel L. Smith, David H. P. Turban, Steven Hamblin, and Nils Y. Hammerla. Offline bilingual
word vectors, orthogonal transformations and the inverted softmax. In _Proceedings of the 5th_
_International Conference on Learning Representations (ICLR 2017)_ [, 2017. URL https://](https://openreview.net/pdf?id=r1Aab85gg)
[openreview.net/pdf?id=r1Aab85gg.](https://openreview.net/pdf?id=r1Aab85gg)
Anders Søgaard, Sebastian Ruder, and Ivan Vuli´c. On the limitations of unsupervised bilingual dictionary induction. In _Proceedings of the 56th Annual Meeting of the Association for_
_Computational Linguistics (Volume 1: Long Papers)_, pp. 778–788, Melbourne, Australia, July
[2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-1072. URL https:](https://www.aclweb.org/anthology/P18-1072)
[//www.aclweb.org/anthology/P18-1072.](https://www.aclweb.org/anthology/P18-1072)
J¨org Tiedemann. Parallel data, tools and interfaces in OPUS. In _Proceedings of the Eighth In-_
_ternational Conference on Language Resources and Evaluation (LREC’12)_, pp. 2214–2218, Is[tanbul, Turkey, May 2012. European Language Resources Association (ELRA). URL http:](http://www.lrec-conf.org/proceedings/lrec2012/pdf/463_Paper.pdf)
[//www.lrec-conf.org/proceedings/lrec2012/pdf/463_Paper.pdf.](http://www.lrec-conf.org/proceedings/lrec2012/pdf/463_Paper.pdf)
Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu, and Ting Liu. Cross-lingual BERT transformation for zero-shot dependency parsing. In _Proceedings of the 2019 Conference on Em-_
_pirical Methods in Natural Language Processing and the 9th International Joint Conference on_
_Natural Language Processing (EMNLP-IJCNLP)_, pp. 5725–5731, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1575. URL
[https://www.aclweb.org/anthology/D19-1575.](https://www.aclweb.org/anthology/D19-1575)
John Wieting, Kevin Gimpel, Graham Neubig, and Taylor Berg-Kirkpatrick. Simple and effective paraphrastic similarity from parallel translations. In _Proceedings of the 57th Annual_
_Meeting of the Association for Computational Linguistics_, pp. 4602–4608, Florence, Italy, July
[2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1453. URL https:](https://www.aclweb.org/anthology/P19-1453)
[//www.aclweb.org/anthology/P19-1453.](https://www.aclweb.org/anthology/P19-1453)
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In _Proceedings of the 2018 Conference of the North_
_American Chapter of the Association for Computational Linguistics: Human Language Technolo-_
_gies, Volume 1 (Long Papers)_, pp. 1112–1122, New Orleans, Louisiana, June 2018. Association
[for Computational Linguistics. doi: 10.18653/v1/N18-1101. URL https://www.aclweb.](https://www.aclweb.org/anthology/N18-1101)
[org/anthology/N18-1101.](https://www.aclweb.org/anthology/N18-1101)
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey,
Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa,
Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa,
Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google’s
neural machine translation system: Bridging the gap between human and machine translation.
_
Ruochen Xu, Yiming Yang, Naoki Otani, and Yuexin Wu. Unsupervised cross-lingual transfer of
word embedding spaces. In _Proceedings of the 2018 Conference on Empirical Methods in Natural_
_Language Processing_, pp. 2465–2474, Brussels, Belgium, October-November 2018. Association
[for Computational Linguistics. doi: 10.18653/v1/D18-1268. URL https://www.aclweb.](https://www.aclweb.org/anthology/D18-1268)
[org/anthology/D18-1268.](https://www.aclweb.org/anthology/D18-1268)
Published as a conference paper at ICLR 2020
English Bulgarian German Greek Spanish French Arabic Chinese Urdu Average
Translate-Train
Base BERT 81.9 73.6 75.9 71.6 77.8 76.8 70.7 76.6 61.6 74.1
Zero-Shot
Base BERT 80.4 68.7 70.4 67.0 74.5 73.4 65.6 70.6 60.3 70.1
Aligned BERT (20K sent) **80.8** **71.6** **72.5** **68.1** **74.7** **73.6** **66.3** **71.5** **61.1** **71.1**
Table 5: Zero-shot accuracy on the XNLI test set with more languages, where we use 20K parallel
sentences for each language paired with English. This result confirms that the alignment method
works for distant languages and a variety of parallel corpora, including Europarl, MultiUN, and
Tanzil, which contains sentences from the Quran (Koehn, 2005; Eisele & Chen, 2010; Tiedemann,
2012).
A APPENDIX
A.1 OPTIMIZATION HYPERPARAMETERS
For both alignment and XNLI optimization, we use a learning rate of 5 _×_ 10 _[−]_ [5] with Adam hyperparameters _β_ = (0 _._ 9 _,_ 0 _._ 98), _ϵ_ = 10 _[−]_ [9] and linear learning rate warmup for the first 10% of the training
data. For alignment, the model is trained for one epoch, with each batch containing 2 sentence pairs
per language. For XNLI, each model is trained for 3 epochs with 32 examples per batch, and 10%
dropout is applied to the BERT embeddings.
A.2 ALIGNMENT OF CHINESE, ARABIC, AND URDU
In Table 5, we report numbers for additional languages, where we align a single BERT model for all
eight languages and then fine-tune on XNLI. We use 20K sentences per language, where we use the
MultiUN corpus for Arabic and Chinese (Eisele & Chen, 2010), the Tanzil corpus for Urdu (Tiedemann, 2012), and the Europarl corpus for the other five languages (Koehn, 2005). This result confirms that the alignment method works for a variety of languages and corpora. Furthermore, the
Tanzil corpus consists of sentences from the Quran, suggesting that the method works even when
the parallel corpus and downstream task contain sentences from entirely different domains.
A.3 EXAMPLES OF CONTEXT-AWARE RETRIEVAL
In this section, we qualitatively show that aligned BERT is able to disambiguate between different
occurences of a word.
First, we find two meanings of the word “like” occurring in the English-German Europarl test set.
Note also that in the second and third example, the two senses of “like” occur in the same sentence.
_•_ This empire did not look for colonies far from home or overseas, **like** most Western European States, but close by.
Dieses Reich suchte seine Kolonien nicht weit von zu Hause und in bersee **wie** die meisten
westeurop¨aischen Staaten, sondern in der unmittelbaren Umgebung.
_•_ **Like** other speakers, I would like to support the call for the arms embargo to remain.
**Wie** andere Sprecher, so m¨ochte auch ich den Aufruf zur Aufrechterhaltung des Waffenembargos untersttzen.
_•_ Like other speakers, I would **like** to support the call for the arms embargo to remain.
Wie andere Sprecher, so **m¨ochte** auch ich den Aufruf zur Aufrechterhaltung des Waffenembargos untersttzen.
_•_ I would also **like**, although they are absent, to mention the Commission and the Council.
Ich **m¨ochte** mir sogar erlauben, die Kommission und den Rat zu nennen, auch wenn sie
nicht anwesend sind.
Published as a conference paper at ICLR 2020
Multiple meanings of “order”:
_•_ Moreover, the national political elite had to make a detour in Ambon in **order** to reach the
civil governor’s residence by warship.
In Ambon mußte die politische Spitze des Landes auch noch einen Umweg machen, **um**
mit einem Kriegsschiff die Residenz des Provinzgouverneurs zu erreichen.
_•_ Although the European Union has an interest in being surrounded by large, stable regions,
the tools it has available in **order** to achieve this are still very limited.
Der Europ¨aischen Union ist zwar an großen stabilen Regionen in ihrer Umgebung gelegen,
aber sie verfgt nach wie vor nur ber recht begrenzte Instrumente, **um** das zu erreichen.
_•_ We could reasonably expect the new Indonesian government to take action in three fundamental areas: restoring public **order**, prosecuting and punishing those who have blood on
their hands and entering into a political dialogue with the opposition.
Von der neuen indonesischen Regierung darf man mit Fug und Recht drei elementare Maßnahmen erwarten: die Wiederherstellung der ¨offentlichen **Ordnung**, die Verfolgung und
Bestrafung derjenigen, an deren H¨anden Blut klebt, und die Aufnahme des politischen Dialogs mit den Gegnern.
_•_ Firstly, I might mention the fact that the army needs to be reformed, secondly that a stable
system of law and **order** needs to be introduced.
Ich nenne hier an erster Stelle die notwendige Reform der Armee, ferner die Einfhrung
eines stabilen Systems rechtsstaatlicher **Ordnung** .
Multiple meanings of “support”:
_•_ Financial **support** is needed to enable poor countries to take part in these court activities.
Arme L¨ander m¨ussen finanziell **unterst¨utzt** werden, damit auch sie sich an der Arbeit des
Gerichtshofs beteiligen k¨onnen.
_•_ We must help them and ensure that a proper action plan is implemented to **support** their
work.
Es gilt einen wirklichen Aktionsplan auf den Weg zu bringen, um die Arbeit dieser Organisationen zu **unterst¨utzen** .
_•_ So I hope that you will all **support** this resolution condemning the abominable conditions
of prisoners and civilians in Djibouti.
Ich hoffe daher, daß Sie alle diese Entschließung **bef¨urworten**, die die entsetzlichen Bedingungen von Inhaftierten und Zivilpersonen in Dschibuti verurteilt.
_•_ It would be difficult to **support** a subsidy scheme that channelled most of the aid to the
large farms in the best agricultural regions.
Es w¨are auch problematisch, ein Beihilfesystem zu **bef¨urworten**, das die meisten Beihilfen
in die großen Betriebe in den besten landwirtschaftlichen Gebieten lenkt.
Multiple meanings of “close”:
_•_ This empire did not look for colonies far from home or overseas, like most Western European States, but **close** by.
Dieses Reich suchte seine Kolonien nicht weit von zu Hause und in bersee wie die meisten
westeurop¨aischen Staaten, sondern in der unmittelbaren **Umgebung** .
_•_ In addition, if we are to shut down or refuse investment from every company which may
have an association with the arms industry, then we would have to **close** virtually every
American and Japanese software company on the island of Ireland with catastrophic consequences.
Wenn wir zudem jedes Unternehmen, das auf irgendeine Weise mit der Rstungsindustrie
verbunden ist, schließen oder Investitionen dieser Unternehmen unterbinden, dann mßten
wir so ziemlich alle amerikanischen und japanischen Softwareunternehmen auf der irischen
Insel **schließen**, was katastrophale Auswirkungen h¨atte.
Published as a conference paper at ICLR 2020
_•_ On the other hand, the deployment of resources left over in the Structural Funds from the
programme planning period 1994 to 1999 is hardly worth considering as the available funds
have already been allocated to specific measures, in this case in **close** collaboration with
the relevant French authorities.
Die Verwendung verbliebener Mittel der Strukturfonds aus dem Programmplanungszeitraum 1994 bis 1999 ist dagegen kaum in Erw¨agung zu ziehen, da die verfgbaren
Mittel bereits bestimmten Maßnahmen zugewiesen sind, und zwar im konkreten Fall im
**engen** Zusammenwirken mit den zust¨andigen franz¨osischen Beh¨orden.
_•_ This is particularly justified given that, as already stated, many Member States have very
**close** relations with Djibouti.
Zumal, wie erw¨ahnt, viele Mitgliedstaaten sehr **enge** Beziehungen zu Dschibuti unterhalten.
_•_ Mr President, it is regrettable that, at the **close** of the 20th century, a century symbolised so
positively by the peaceful women’s revolution, there are still countries, such as Kuwait and
Afghanistan, where half the population, women that is, is still denied fundamental human
rights.
Herr Pr¨asident! Es ist wirklich bedauerlich, daß es am **Ende** des 20. Jahrhunderts, eines
so positiv von der friedlichen Revolution der Frauen gepr¨agten Jahrhunderts, noch immer
L¨ander wie Kuwait und Afghanistan gibt, in denen der H¨alfte der Bev¨olkerung, den Frauen,
die elementaren Menschenrechte verweigert werden.

Multilingual Alignment of Contextual Word Representations


**Fangyu Liu** _[♣]_ **, Ehsan Shareghi** _[♦][,][♣]_ **, Zaiqiao Meng** _[♣]_ **, Marco Basaldella** _[♥][∗]_ **, Nigel Collier** _[♣]_
_♣_ Language Technology Lab, TAL, University of Cambridge
_♦_ Department of Data Science & AI, Monash University _♥_ Amazon Alexa
_♣_ {fl399, zm324, nhc30}@cam.ac.uk
_♦_ ehsan.shareghi@monash.edu _♥_ mbbasald@amazon.co.uk

Self-Alignment Pretraining for Biomedical Entity Representations

Fangyu Liu
Ehsan Shareghi
Zaiqiao Meng
Marco Basaldella
Nigel Collier


main, we achieve SOTA even without taskspecific supervision. With substantial improvement over various domain-specific pretrained
MLMs such as BIOBERT, SCIBERT and PUBMEDBERT, our pretraining scheme proves to
be both effective and robust. [1]


Biomedical entity [2] representation is the foundation for a plethora of text mining systems in the
medical domain, facilitating applications such as
literature search (Lee et al., 2016), clinical decision
making (Roberts et al., 2015) and relational knowledge discovery (e.g. chemical-disease, drug-drug
and protein-protein relations, Wang et al. 2018).
The heterogeneous naming of biomedical concepts
_∗_ Work conducted prior to joining Amazon.
[1For code and pretrained models, please visit: https:](https://github.com/cambridgeltl/sapbert)
[//github.com/cambridgeltl/sapbert.](https://github.com/cambridgeltl/sapbert)
2In this work, _biomedical entity_ refers to the surface forms
of biomedical concepts, which can be a single word (e.g.
_fever_ ), a compound (e.g. _sars-cov-2_ ) or a short phrase (e.g.
_abnormal retinal vascular development_ ).
poses a major challenge to representation learning.
For instance, the medication _Hydroxychloroquine_
is often referred to as _Oxichlorochine_ (alternative
name), _HCQ_ (in social media) and _Plaquenil_ (brand
name).
MEL addresses this problem by framing it as
a task of mapping entity mentions to unified concepts in a medical knowledge graph. [3] The main
bottleneck of MEL is the quality of the entity representations (Basaldella et al., 2020). Prior works
in this domain have adopted very sophisticated
text pre-processing heuristics (D’Souza and Ng,
2015; Kim et al., 2019; Ji et al., 2020; Sung et al.,
2020) which can hardly cover all the variations
of biomedical names. In parallel, self-supervised
learning has shown tremendous success in NLP via
leveraging the masked language modelling (MLM)
3Note that we consider only the biomedical entities themselves and not their contexts, also known as medical concept
normalisation/disambiguation in the BioNLP community.
TRON, Shin et al. 2020) have made much progress
in biomedical text mining tasks. Nonetheless, representing medical entities with the existing SOTA
pretrained MLMs (e.g. PUBMEDBERT, Gu et al.
2020) as suggested in Fig. 1 (left) does not lead to
a well-separated representation space.
To address the aforementioned issue, we propose
to pretrain a Transformer-based language model on
the biomedical knowledge graph of UMLS (Bodenreider, 2004), the largest interlingua of biomedical
ontologies. UMLS contains a comprehensive collection of biomedical synonyms in various forms
(UMLS 2020AA has 4M+ concepts and 10M+ synonyms which stem from over 150 controlled vocabularies including MeSH, SNOMED CT, RxNorm,
Gene Ontology and OMIM). [4] We design a selfalignment objective that clusters synonyms of the
same concept. To cope with the immense size of
UMLS, we sample hard training pairs from the
knowledge base and use a scalable metric learning
loss. We name our model as **S** elf- **a** ligning **p** retrained **BERT** (SAPBERT).
Being both simple and powerful, SAPBERT obtains new SOTA performances across all six MEL
benchmark datasets. In contrast with the current
systems which adopt complex pipelines and hybrid
components (Xu et al., 2020; Ji et al., 2020; Sung
et al., 2020), SAPBERT applies a much simpler
training procedure without requiring any pre- or
post-processing steps. At test time, a simple nearest
neighbour’s search is sufficient for making a prediction. When compared with other domain-specific
pretrained language models (e.g. BIOBERT and
SCIBERT), SAPBERT also brings substantial improvement by up to 20% on accuracy across all
tasks. The effectiveness of the pretraining in SAPBERT is especially highlighted in the scientific language domain where SAPBERT outperforms previous SOTA even without fine-tuning on any MEL
datasets. We also provide insights on pretraining’s
impact across domains and explore pretraining with
fewer model parameters by using a recently introduced ADAPTER module in our training scheme.
[https://www.nlm.nih.gov/research/umls/knowledge_](https://www.nlm.nih.gov/research/umls/knowledge_sources/metathesaurus/release/statistics.html)
[sources/metathesaurus/release/statistics.html](https://www.nlm.nih.gov/research/umls/knowledge_sources/metathesaurus/release/statistics.html)
Figure 2: The distribution of similarity scores for
all sampled PUBMEDBERT representations in a minibatch. The left graph shows the distribution of **+** and **-**
pairs which are easy and already well-separated. The
right graph illustrates larger overlap between the two
groups generated by the online mining step, making
them harder and more informative for learning.


We design a metric learning framework that learns
to self-align synonymous biomedical entities. The
framework can be used as both pretraining on
UMLS, and fine-tuning on task-specific datasets.
We use an existing BERT model as our starting
point. In the following, we introduce the key components of our framework.
**Formal Definition.** Let ( _x, y_ ) _∈X × Y_ denote a tuple of a name and its categorical label.
For the self-alignment pretraining step, _X × Y_
is the set of all (name, CUI [5] ) pairs in UMLS,
e.g. ( _Remdesivir_, C4726677); while for the finetuning step, it is formed as an entity mention
and its corresponding mapping from the ontology, e.g. ( _scratchy throat_, 102618009). Given
any pair of tuples ( _xi, yi_ ) _,_ ( _xj, yj_ ) _∈X × Y_, the
goal of the self-alignment is to learn a function
_f_ ( _·_ ; _θ_ ) : _X →_ R _[d]_ parameterised by _θ_ . Then, the
similarity _⟨f_ ( _xi_ ) _, f_ ( _xj_ ) _⟩_ (in this work we use cosine similarity) can be used to estimate the resemblance of _xi_ and _xj_ (i.e., high if _xi, xj_ are synonyms and low otherwise). We model _f_ by a BERT
model with its output [CLS] token regarded as the
representation of the input. [6] During the learning,
a sampling procedure selects the informative pairs
of training samples and uses them in the pairwise
metric learning loss function (introduced shortly).
**Online Hard Pairs Mining.** We use an online
hard triplet mining condition to find the most
5In UMLS, CUI is the **C** oncept **U** nique **I** dentifier.
6We tried multiple strategies including first-token, meanpooling, [CLS] and also NOSPEC (recommended by Vuli´c
et al. 2020) but found no consistent best strategy (optimal
strategy varies on different *BERTs).
informative training examples (i.e. hard positive/negative pairs) within a mini-batch for efficient
training, Fig. 2. For biomedical entities, this step
can be particularly useful as most examples can
be easily classified while a small set of very hard
ones cause the most challenge to representation
learning. [7] We start from constructing all possible
triplets for all names within the mini-batch where
each triplet is in the form of ( _xa, xp, xn_ ). Here
_xa_ is called _anchor_, an arbitrary name in the minibatch; _xp_ a positive match of _xa_ (i.e. _ya_ = _yp_ ) and
_xn_ a negative match of _xa_ (i.e. _ya ̸_ = _yn_ ). Among
the constructed triplets, we select out all triplets
that violate the following condition:
_∥f_ ( _xa_ ) _−_ _f_ ( _xp_ ) _∥_ 2 _< ∥f_ ( _xa_ ) _−_ _f_ ( _xn_ ) _∥_ 2 + _λ,_ (1)
where _λ_ is a pre-set margin. In other words, we
only consider triplets with the negative sample
closer to the positive sample by a margin of _λ_ .
These are the hard triplets as their original representations were very far from correct. Every hard
triplet contributes one hard positive pair ( _xa, xp_ )
and one hard negative pair ( _xa, xn_ ). We collect
all such positive & negative pairs and denote them
as _P, N_ . A similar but not identical triplet mining condition was used by Schroff et al. (2015) for
face recognition to select hard negative samples.
Switching-off this mining process, causes a drastic
performance drop (see Tab. 2).
**Loss Function.** We compute the pairwise cosine
similarity of all the BERT-produced name representations and obtain a similarity matrix **S** _∈_
R _[|X][b][|×|X][b][|]_ where each entry **S** _ij_ corresponds to the
cosine similarity between the _i_ -th and _j_ -th names in
the mini-batch _b_ . We adapted the Multi-Similarity
loss (MS loss, Wang et al. 2019), a SOTA metric
learning objective on visual recognition, for learning from the positive and negative pairs:
_e_ _[−][β]_ [(] **[S]** _[ip][−][ϵ]_ [)][��]
_p∈Pi_
_,_
(2)
_L_ =
_|Xb|_
_|Xb|_
_i_ =1
+ [1]
1  -  1 + _e_ _[α]_ [(] **[S]** _[in][−][ϵ]_ [)][�]
_α_ [log]
_n∈Ni_
1  -  1 +
_α_ [log]
While the first term in Eq. 2 pushes negative
pairs away from each other, the second term pulls
positive pairs together. This dynamic allows for
a re-calibration of the alignment space using the
semantic biases of synonymy relations. The MS
loss leverages similarities among and between positive and negative pairs to re-weight the importance
of the samples. The most informative pairs will
receive more gradient signals during training and
thus can better use the information stored in data.


**ing.** We download the full release of UMLS
2020AA version. [9] We then extract all English
entries from the MRCONSO.RFF raw file and
convert all entity names into lowercase (duplicates are removed). Besides synonyms defined
in MRCONSO.RFF, we also include tradenames of
drugs as synonyms (extracted from MRREL.RRF).
After pre-processing, a list of 9,712,959 (name,
CUI) entries is obtained. However, random batching on this list can lead to very few (if not none)
positive pairs within a mini-batch. To ensure sufficient positives present in each mini-batch, we generate offline positive pairs in the format of (name1,
name2, CUI) where name1 and name2 have the
same CUI label. This can be achieved by enumerating all possible combinations of synonym pairs
with common CUIs. For balanced training, any
concepts with more than 50 positive pairs are randomly trimmed to 50 pairs. In the end we obtain a
training list with 11,792,953 pairwise entries.
**UMLS Pretraining Details.** During training, we
use AdamW (Loshchilov and Hutter, 2018) with
a learning rate of 2e-5 and weight decay rate of
1e-2. Models are trained on the prepared pairwise
UMLS data for 1 epoch (approximately 50k iterations) with a batch size of 512 (i.e., 256 pairs per
mini-batch). We train with Automatic Mixed Precision (AMP) [10] provided in PyTorch 1.7.0. This
takes approximately 5 hours on our machine (configurations specified in App. SectionB.4). For other hyper
(Oord et al., 2018), NCA loss (Goldberger et al., 2005),
simple cosine loss (Phan et al., 2019), max-margin triplet
loss (Basaldella et al., 2020) but found our choice is empirically better. See App. SectionB.2 for comparison.
[https://download.nlm.nih.gov/umls/kss/2020AA/](https://download.nlm.nih.gov/umls/kss/2020AA/umls-2020AA-full.zip)
[umls-2020AA-full.zip](https://download.nlm.nih.gov/umls/kss/2020AA/umls-2020AA-full.zip)
[https://pytorch.org/docs/stable/amp.html](https://pytorch.org/docs/stable/amp.html)
  -   1 +
_β_ [log]
where _α, β_ are temperature scales; _ϵ_ is an offset
applied on the similarity matrix; _Pi, Ni_ are indices
of positive and negative samples of the _anchor i_ . [8]
7Most of _Hydroxychloroquine_ ’s variants are easy: _Hydrox-_
_ychlorochin_, _Hydroxychloroquine (substance)_, _Hidroxicloro-_
_quina_, but a few can be very hard: _Plaquenil_ and _HCQ_ .
8We explored several loss functions such as InfoNCE
scientific language social media language
NCBI BC5CDR-d BC5CDR-c MedMentions AskAPatient COMETA
model
@1 @5 @1 @5 @1 @5 @1 @5 @1 @5 @1 @5
vanilla BERT (Devlin et al., 2019) 67.6 77.0 81.4 89.1 79.8 91.2 39.6 60.2 38.2 43.3 40.4 47.7
BIOBERT (Lee et al., 2020) 71.3 84.1 79.8 92.3 74.0 90.0 24.2 38.5 41.4 51.5 35.9 46.1
BLUEBERT (Peng et al., 2019) 75.7 87.2 83.2 91.0 87.7 94.1 41.6 61.9 41.5 48.5 42.9 52.9
CLINICALBERT (Alsentzer et al., 2019) 72.1 84.5 82.7 91.6 75.9 88.5 43.9 54.3 43.1 51.8 40.6 61.8
SCIBERT (Beltagy et al., 2019) 85.1 88.4 89.3 92.8 94.2 95.5 42.3 51.9 48.0 54.8 45.8 66.8
UMLSBERT (Michalopoulos et al., 2020) 77.0 85.4 85.5 92.5 88.9 94.1 36.1 55.8 44.4 54.5 44.6 53.0
PUBMEDBERT (Gu et al., 2020) 77.8 86.9 89.0 93.8 93.0 94.6 43.9 64.7 42.5 49.6 46.8 53.2
+ SAPBERT 92.0 95.6 93.5 96.0 96.5 98.2 50.8 74.4 70.5 88.9 65.9 77.9
Table 1: **Top** : Comparison of 7 BERT-based models before and after SAPBERT pretraining (+ SAPBERT). All
results in this section are from unsupervised learning (not fine-tuned on task data). The gradient of green indicates
the improvement comparing to the base model (the deeper the more). **Bottom** : SAPBERT vs. SOTA results. Blue
and red denote unsupervised and supervised models. **Bold** and underline denote the best and second best results
in the column. “ _[†]_ ” denotes statistically significant better than supervised SOTA (T-test, _ρ <_ 0 _._ 05). On COMETA,
the results inside the parentheses added the supervised SOTA’s dictionary back-off technique (Basaldella et al.,
2020). “-”: not reported in the SOTA paper. “OOM”: out-of-memory (192GB+).
parameters used, please view App. SectionC.2.
**Evaluation Data and Protocol.** We experiment
on 6 different English MEL datasets: 4 in the scientific domain (NCBI, Do˘gan et al. 2014; BC5CDR-c
and BC5CDR-d, Li et al. 2016; MedMentions, Mohan and Li 2018) and 2 in the social media domain
(COMETA, Basaldella et al. 2020 and AskAPatient, Limsopatham and Collier 2016). Descriptions of the datasets and their statistics are provided
in App. SectionA. We report Acc@1 and Acc@5 (denoted
as @1 and @5) for evaluating performance. In all
experiments, SAPBERT denotes further pretraining
with our self-alignment method on UMLS. At the
test phase, for all SAPBERT models we use nearest neighbour search without further fine-tuning on
task data (unless stated otherwise). Except for numbers reported in previous papers, all results are the
average of five runs with different random seeds.
**Fine-Tuning on Task Data.** The red rows in Tab. 1
are results of models (further) fine-tuned on the
training sets of the six MEL datasets. Similar to
pretraining, a positive pair list is generated through
traversing the combinations of mention and all
ground truth synonyms where mentions are from
the training set and ground truth synonyms are from
the reference ontology. We use the same optimiser
and learning rates but train with a batch size of
256 (to accommodate the memory of 1 GPU). On
scientific language datasets, we train for 3 epochs
while on AskAPatient and COMETA we train for
15 and 10 epochs respectively. For BIOSYN on social media language datasets, we empirically found
that 10 epochs work the best. Other configurations
are the same as the original BIOSYN paper.

Data Preparation Details for UMLS Pretrain-


***BERT + SAPBERT (Tab. 1, top).** We illustrate
the impact of SAPBERT pretraining over 7 existing BERT-based models (*BERT = {BIOBERT,
PUBMEDBERT, ...}). SAPBERT obtains consistent improvement over all *BERT models across all
datasets, with larger gains (by up to 31.0% absolute
Acc@1 increase) observed in the social media domain. While SCIBERT is the leading model before
applying SAPBERT, PUBMEDBERT+SAPBERT
performs the best afterwards.
**SAPBERT vs. SOTA (Tab. 1, bottom).** We take
PUBMEDBERT+SAPBERT (w/wo fine-tuning) and
compare against various published SOTA results
(see App. SectionC.1 for a full listing of 10 baselines)
which all require task supervision. For the scientific language domain, the SOTA is BIOSYN (Sung
et al., 2020). For the social media domain, the
SOTA are Basaldella et al. (2020) and GENRANK (Xu et al., 2020) on COMETA and AskAPatient respectively. All these SOTA methods combine BERT with heuristic modules such as tf-idf,
string matching and information retrieval system
(i.e. Apache Lucene) in a multi-stage manner.
Measured by Acc@1, SAPBERT achieves new
SOTA with statistical significance on 5 of the 6
datasets and for the dataset (BC5CDR-c) where
SAPBERT is not significantly better, it performs on
par with SOTA (96.5 vs. 96.6). Interestingly, on scientific language datasets, SAPBERT outperforms
SOTA without any task supervision (fine-tuning
mostly leads to overfitting and performance drops).
On social media language datasets, unsupervised
SAPBERT lags behind supervised SOTA by large
margins, highlighting the well-documented complex nature of social media language (Baldwin
et al., 2013; Limsopatham and Collier, 2015, 2016;
Basaldella et al., 2020; Tutubalina et al., 2020).
However, after fine-tuning on the social media
datasets (using the MS loss introduced earlier),
SAPBERT outperforms SOTA significantly, indicating that knowledge acquired during the selfaligning pretraining can be adapted to a shifted
domain without much effort.
**The ADAPTER Variant.** As an option for parameter efficient pretraining, we explore a variant of
SAPBERT using a recently introduced training module named ADAPTER (Houlsby et al., 2019). While
maintaining the same pretraining scheme with the
same SAPBERT online mining + MS loss, instead
of training from the full model of PUBMEDBERT,
we insert new ADAPTER layers between Transformer layers of the fixed PUBMEDBERT, and only
train the weights of these ADAPTER layers. In our
experiments, we use the enhanced ADAPTER configuration by Pfeiffer et al. (2020). We include two
variants where trained parameters are 13.22% and
1.09% of the full SAPBERT variant. The ADAPTER
variant of SAPBERT achieves comparable performance to full-model-tuning in scientific datasets
but lags behind in social media datasets, Tab. 1. The
results indicate that more parameters are needed
in pretraining for knowledge transfer to a shifted
domain, in our case, the social media datasets.
**The Impact of Online Mining (Eq. (1)).** As sug
gested in Tab. 2, switching off the online hard pairs
mining procedure causes a large performance drop
in @1 and a smaller but still significant drop in @5.
This is due to the presence of many easy and already well-separated samples in the mini-batches.
These uninformative training examples dominated
the gradients and harmed the learning process.
configuration @1 @5
Mining switched-on **67.2** **80.3**
Mining switched-off 52.3 _↓_ 14 _._ 9 76.1 _↓_ 4 _._ 2
Table 2: This table compares PUBMEDBERT+SAPBERT’s performance with and without
online hard mining on COMETA (zeroshot general).
**Integrating SAPBERT in Existing Systems.**
SAPBERT can be easily inserted into existing
BERT-based MEL systems by initialising the systems with SAPBERT pretrained weights. We use
the SOTA scientific language system, BIOSYN
(originally initialised with BIOBERT weights), as
an example and show the performance is boosted
across all datasets (last two rows, Tab. 1).


We present SAPBERT, a self-alignment pretraining
scheme for learning biomedical entity representations. We highlight the consistent performance
boost achieved by SAPBERT, obtaining new SOTA
in all six widely used MEL benchmarking datasets.
Strikingly, without any fine-tuning on task-specific
labelled data, SAPBERT already outperforms the
previous supervised SOTA (sophisticated hybrid entity linking systems) on multiple datasets in the scientific language domain. Our work opens new avenues to explore for general domain self-alignment
(e.g. by leveraging knowledge graphs such as DBpedia). We plan to incorporate other types of relations (i.e., hypernymy and hyponymy) and extend
our model to sentence-level representation learning.
In particular, our ongoing work using a combination of SAPBERT and ADAPTER is a promising
direction for tackling sentence-level tasks.


We thank the three reviewers and the Area Chair
for their insightful comments and suggestions. FL
is supported by Grace & Thomas C.H. Chan Cambridge Scholarship. NC and MB would like to
acknowledge funding from Health Data Research
UK as part of the National Text Analytics project.


We divide our experimental datasets into two categories (1) scientific language datasests where the
data is extracted from scientific papers and (2) social media language datasets where the data is coming from social media forums like Reddit.com.
For an overview of the key statistics, see Tab. 3.


**NCBI disease (Do˘gan et al., 2014)** is a corpus
containing 793 fully annotated PubMed abstracts
and 6,881 mentions. The mentions are mapped
into the MEDIC dictionary (Davis et al., 2012). We
denote this dataset as “NCBI” in our experiments.
**BC5CDR (Li et al., 2016)** consists of 1,500
PubMed articles with 4,409 annotated chemicals,
5,818 diseases and 3,116 chemical-disease interactions. The disease mentions are mapped into the
MEDIC dictionary like the NCBI disease corpus.
The chemical mentions are mapped into the Comparative Toxicogenomics Database (CTD) (Davis
et al., 2019) chemical dictionary. We denote the
disease and chemical mention sets as “BC5CDRd” and “BC5CDR-c” respectively. For NCBI and
BC5CDR we use the same data and evaluation protocol by Sung et al. (2020). [11]
**MedMentions (Mohan and Li, 2018)** is a verylarge-scale entity linking dataset containing over
4,000 abstracts and over 350,000 mentions linked
to UMLS 2017AA. According to Mohan and Li
(2018), training TAGGERONE (Leaman and Lu,
2016), a very popular MEL system, on a subset
of MedMentions require >900 GB of RAM. Its
massive number of mentions and more importantly
the used reference ontology (UMLS 2017AA has
3M+ concepts) make the application of most MEL
systems infeasible. However, through our metric
learning formulation, SAPBERT can be applied on
MedMentions with minimal effort.


includes 17,324 adverse drug reaction (ADR) annotations collected from askapatient.com blog
posts. The mentions are mapped to 1,036 medical
concepts grounded onto SNOMED-CT (Donnelly,
2006) and AMT (the Australian Medicines Terminology). For this dataset, we follow the 10-fold
evaluation protocol stated in the original paper. [12]
**COMETA (Basaldella et al., 2020)** is a recently
released large-scale MEL dataset that specifically
focuses on MEL in the social media domain, containing around 20k medical mentions extracted
from health-related discussions on reddit.com.
Mentions are mapped to SNOMED-CT. We use the
“stratified (general)” split and follow the evaluation
protocol of the original paper. [13]

AskAPatient (Limsopatham and Collier, 2016)