We list all the versions of BERT models used in
this study, linking to the specific versions in Tab. 5.
Note that we exhaustively tried all official variants
of the selected models and the best performing ones
are chosen. All BERT models refer to the BERTBase
architecture in this paper.
[11https://github.com/dmis-lab/BioSyn](https://github.com/dmis-lab/BioSyn)
[12https://zenodo.org/record/55013](https://zenodo.org/record/55013)
[13https://www.siphs.org/corpus](https://www.siphs.org/corpus)
dataset NCBI BC5CDR-d BC5CDR-c MedMentions AskAPAtient COMETA (s.g.) COMETA (z.g.)
Ontology MEDIC MEDIC CTD UMLS 2017AA SNOMED & AMT SNOMED SNOMED
_C_ searched ⊊ _C_ ontology?
_|C_ searched _|_ 11,915 11,915 171,203 3,415,665 1,036 350,830 350,830
_|S_ searched _|_ 71,923 71,923 407,247 14,815,318 1,036 910,823 910,823
_|M_ train _|_ 5,134 4,182 5,203 282,091 15,665.2 13,489 14,062
_|M_ validation _|_ 787 4,244 5,347 71,062 792.6 2,176 1,958
_|M_ test _|_ 960 4,424 5,385 70,405 866.2 4,350 3,995
Table 3: This table contains basic statistics of the MEL datasets used in the study. _C_ denotes the set of concepts;
_S_ denotes the set of all surface forms / synonyms of all concepts in _C_ ; _M_ denotes the set of mentions / queries.
COMETA (s.g.) and (z.g.) are the stratified (general) and zeroshot (general) split respectively.
NCBI BC5CDR-d BC5CDR-c MedMentions AskAPatient COMETA
model
@1 @5 @1 @5 @1 @5 @1 @5 @1 @5 @1 @5
SIEVE-BASED (D’Souza and Ng, 2015) 84.7 - 84.1 - 90.7 - - WORDCNN (Limsopatham and Collier, 2016) - - - - - - - - 81.4 - - WORDGRU+TF-IDF (Tutubalina et al., 2018) - - - - - - - - 85.7 - - TAGGERONE (Leaman and Lu, 2016) 87.7 - 88.9 - 94.1 - OOM OOM - - - NORMCO (Wright et al., 2019) 87.8 - 88.0 - - - - - - - - BNE (Phan et al., 2019) 87.7 - 90.6 - 95.8 - - - - - - BERTRANK (Ji et al., 2020) 89.1 - - - - - - - - - - GEN-RANK (Xu et al., 2020) - - - - - - - - **87.5** - - BIOSYN (Sung et al., 2020) **91.1 93.9 93.2 96.0** **96.6 97.2** OOM OOM 82.6 _[∗]_ 87.0 _[∗]_ 71.3 _[∗]_ 77.8 _[∗]_
DICT+SOILOS+NEURAL (Basaldella et al., 2020) - - - - - - - - - - **79.0** supervised SOTA 91.1 93.9 93.2 96.0 96.6 97.2 OOM OOM 87.5 - 79.0
Table 4: A list of baselines on the 6 different MEL datasets, including both scientific and social media language ones. The last
row collects reported numbers from the best performing models. “ _∗_ ” denotes results produced using official released code. “-”
denotes results not reported in the cited paper. “OOM” means out-of-memoery.


We use COMETA (zeroshot general) as a benchmark for selecting learning objectives. Note
that this split of COMETA is different from the
stratified-general split used in Tab. 4. It is very
challenging (so easy to see the difference of the
performance) and also does not directly affect the
model’s performance on other datasets. The results
are listed in Tab. 6. Note that online mining is
switched on for all models here.
loss @1 @5
cosine loss (Phan et al., 2019) 55.1 64.6
max-margin triplet loss (Basaldella et al., 2020) 64.6 74.6
NCA loss (Goldberger et al., 2005) 65.2 77.0
Lifted-Structure loss (Oh Song et al., 2016) 62.0 72.1
InfoNCE (Oord et al., 2018; He et al., 2020) 63.3 74.2
Circle loss (Sun et al., 2020) 66.7 78.7
Multi-Similarity loss (Wang et al., 2019) **67.2 80.3**
Table 6: This table compares loss functions used
for SAPBERT pretraining. Numbers reported are on
COMETA (zeroshot general).
The cosine loss was used by Phan et al. (2019)
for learning UMLS synonyms for LSTM models.
The max-margin triplet loss was used by Basaldella
et al. (2020) for training MEL models. A very
similar (though not identical) hinge-loss was used
by Schumacher et al. (2020) for clinical concept
linking. InfoNCE has been very popular in selfsupervised learning and contrastive learning (Oord
et al., 2018; He et al., 2020). Lifted-Structure loss
(Oh Song et al., 2016) and NCA loss (Goldberger
et al., 2005) are two very classic metric learning objectives. Multi-Similarity loss (Wang et al., 2019)
and Circle loss (Sun et al., 2020) are two recently
proposed metric learning objectives and have been
considered as SOTA on large-scale visual recognition benchmarks.

Self-Alignment Pretraining for Biomedical Entity Representations

Fangyu Liu
Ehsan Shareghi
Zaiqiao Meng
Marco Basaldella
Nigel Collier


In Tab. 7 we list number of parameters trained in
the three ADAPTER variants along with full-modeltuning for easy comparison.
model URL
vanilla BERT (Devlin et al., 2019) [https://huggingface.co/bert-base-uncased](https://huggingface.co/bert-base-uncased)
BIOBERT (Lee et al., 2020) [https://huggingface.co/dmis-lab/biobert-v1.1](https://huggingface.co/dmis-lab/biobert-v1.1)
BLUEBERT (Peng et al., 2019) [https://huggingface.co/bionlp/bluebert_pubmed_mimic_uncased_L-12_H-768_A-12](https://huggingface.co/bionlp/bluebert_pubmed_mimic_uncased_L-12_H-768_A-12)
CLINICALBERT (Alsentzer et al., 2019) [https://huggingface.co/emilyalsentzer/Bio_ClinicalBERT](https://huggingface.co/emilyalsentzer/Bio_ClinicalBERT)
SCIBERT (Beltagy et al., 2019) [https://huggingface.co/allenai/scibert_scivocab_uncased](https://huggingface.co/allenai/scibert_scivocab_uncased)
[UMLSBERT (Michalopoulos et al., 2020) https://www.dropbox.com/s/qaoq5gfen69xdcc/umlsbert.tar.xz?dl=0](https://www.dropbox.com/s/qaoq5gfen69xdcc/umlsbert.tar.xz?dl=0)
PUBMEDBERT (Gu et al., 2020) [https://huggingface.co/microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext](https://huggingface.co/microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext)
Table 5: This table lists the URL of models used in this study.
#params
method reduction rate #params #params in BERT
ADAPTER13% 1 14.47M 13.22%
ADAPTER1% 16 0.60M 1.09%
full-model-tuning  - 109.48M 100%
Table 7: This table compares number of parameters trained in ADAPTER variants and also full-modeltuning.


All our experiments are conducted on a server with
specifications listed in Tab. 8.
hardware specification
RAM 192 GB
CPU Intel Xeon W-2255 @3.70GHz, 10-core 20-threads
GPU NVIDIA GeForce RTX 2080 Ti (11 GB) _×_ 4
Table 8: Hardware specifications of the used machine.


The full table of supervised baseline models is provided in Tab. 4.


Tab. 9 lists hyper-parameter search space for obtaining the set of used numbers. Note that the
chosen hyper-parameters yield the overall best performance but might be sub-optimal on any single
dataset. Also, we balanced the memory limit and
model performance.


We show a clearer version of t-SNE embedding
visualisation in Fig. 3.


**Chi Chen** [1] _[,]_ [3] _[,]_ [4] **, Maosong Sun** [1] _[,]_ [3] _[,]_ [4] _[,]_ [5] **, Yang Liu** _[∗]_ [1] _[,]_ [2] _[,]_ [3] _[,]_ [4] _[,]_ [5]
1Department of Computer Science and Technology, Tsinghua University, Beijing, China
2Institute for AI Industry Research, Tsinghua University, Beijing, China
3Institute for Artificial Intelligence, Tsinghua University, Beijing, China
4Beijing National Research Center for Information Science and Technology
5Beijing Academy of Artificial Intelligence

Mask-Align: Self-Supervised Neural Word Alignment

MASK-ALIGN: Self-Supervised Neural Word Alignment


Word alignment, which aims to align translationally equivalent words between source and
target sentences, plays an important role in
many natural language processing tasks. Current unsupervised neural alignment methods
focus on inducing alignments from neural machine translation models, which does not leverage the full context in the target sequence. In
this paper, we propose MASK-ALIGN, a selfsupervised word alignment model that takes
advantage of the full context on the target side.
Our model parallelly masks out each target token and predicts it conditioned on both source
and the remaining target tokens. This two-step
process is based on the assumption that the
source token contributing most to recovering
the masked target token should be aligned.
We also introduce an attention variant called
_leaky attention_, which alleviates the problem
of high cross-attention weights on specific tokens such as periods. Experiments on four language pairs show that our model outperforms
previous unsupervised neural aligners and obtains new state-of-the-art results.


Word alignment is an important task of finding
the correspondence between words in a sentence
pair (Brown et al., 1993) and used to be a key
component of statistical machine translation (SMT)
(Koehn et al., 2003; Dyer et al., 2013). Although
word alignment is no longer explicitly modeled in
neural machine translation (NMT) (Bahdanau et al.,
2015; Vaswani et al., 2017), it is often leveraged to
analyze NMT models (Tu et al., 2016; Ding et al.,
2017). Word alignment is also used in many other
scenarios such as imposing lexical constraints on
the decoding process (Arthur et al., 2016; Hasler
et al., 2018), improving automatic post-editing (Pal
_∗_ Corresponding author


Induced alignment link: **Tokio - Tokyo**
Figure 1: An example of inducing an alignment link for
target token “Tokyo” in MASK-ALIGN. First, we mask
out “Tokyo” and predict it with source and other target
tokens. Then, the source token “Tokio” that contributes
most to recovering the masked word (highlighted in
red) is chosen to be aligned to “Tokyo”.
et al., 2017), and providing guidance for translators
in computer-aided translation (Dagan et al., 1993).
Compared with statistical methods, neural methods can learn representations end-to-end from raw
data and have been successfully applied to supervised word alignment (Yang et al., 2013; Tamura
et al., 2014). For unsupervised word alignment,
however, previous neural methods fail to significantly exceed their statistical counterparts such
as FAST-ALIGN (Dyer et al., 2013) and GIZA++
(Och and Ney, 2003). Recently, there is a surge of
interest in NMT-based alignment methods which
take alignments as a by-product of NMT systems
(Li et al., 2019; Garg et al., 2019; Zenkel et al.,
2019, 2020; Chen et al., 2020). Using attention
weights or feature importance measures to induce
alignments for to-be-predicted target tokens, these
methods outperform unsupervised statistical aligners like GIZA++ on a variety of language pairs.
Although NMT-based unsupervised aligners
have proven to be effective, they suffer from two
major limitations. First, due to the autoregressive
property of NMT systems (Sutskever et al., 2014),
Alignment Attention Weights - ��� ���� ����
Leaky Attention
|t1|Col2|Col3|Col4|Col5|
|---|---|---|---|---|
|t**2**|||||
|t**3**|||||
|t**4**|||||
h **1** h **2** h **3** h **4**
|Col1|Col2|Col3|Col4|
|---|---|---|---|
|t|t|t|t|
|t||||
|t||||
Feed Forward
✕ L
L ✕
��� ���� ������ ��� - ��� ���� ����
Figure 2: The architecture of MASK-ALIGN.
they only leverage part of the target context. This
inevitably brings noisy alignments when the prediction is ambiguous. Consider the target sentence
in Figure 1. When predicting “Tokyo”, an NMT
system may generate “1968” because future context is not observed, leading to a wrong alignment
link (“1968”, “Tokyo”). Second, they have to incorporate an additional guided alignment loss (Chen
et al., 2016) to outperform GIZA++. This loss requires pseudo alignments of the full training data
to guide the training of the model. Although these
pseudo alignments can be utilized to partially alleviate the problem of ignoring future context, they
are computationally expensive to obtain.
In this paper, we propose a self-supervised
model specifically designed for the word alignment
task, namely MASK-ALIGN. Our model parallelly
masks out each target token and recovers it conditioned on the source and other target tokens. Figure 1 shows an example where the target token
“Tokyo” is masked out and re-predicted. Intuitively,
as all source tokens except “Tokio” can find their
counterparts on the target side, “Tokio” should be
aligned to the masked token. Based on this intuition, we assume that the source token contributing
most to recovering a masked target token should be
aligned to that target token. Compared with NMTbased methods, MASK-ALIGN is able to take full
advantage of bidirectional context on the target side
and hopefully achieves higher alignment quality.
We also introduce an attention variant called _leaky_
_attention_ to reduce the high attention weights on
specific tokens such as periods. By encouraging
agreement between two directional models both
for training and inference, our method consistently
outperforms the state-of-the-art on four language
pairs without using guided alignment loss.


Figure 2 shows the architecture of our model. The
model predicts each target token conditioned on the
source and other target tokens and generates alignments from the attention weights between source
and target (Section 2.1). Specifically, our approach
introduces two attention variants, _static-KV atten-_
_tion_ and _leaky attention_, to efficiently obtain attention weights for word alignment. To better utilize
attention weights from two directions, we encourage agreement between two unidirectional models
during both training (Section 2.2) and inference
(Section 2.3).


Conventional unsupervised neural aligners are
based on NMT models (Peter et al., 2017; Garg
et al., 2019). Given a source sentence **x** =
_x_ 1 _, . . ., xJ_ and a target sentence **y** = _y_ 1 _, . . ., yI_,
NMT models the probability of the target sentence
conditioned on the source sentence:
where **y** _<i_ is a partial translation. One problem of
this type of approaches is that they fail to exploit the
future context on the target side, which is probably
helpful for word alignment.
To address this problem, we model the same
conditional probability but predict each target token
_yi_ conditioned on the source sentence **x** and the
_P_ ( **y** _|_ **x** ; _θ_ ) =
_I_
_P_ ( _yi|_ **y** _<i,_ **x** ; _θ_ ) (1)
_i_ =1
remaining target tokens **y** _\yi_ :
Alignment
_P_ ( **y** _|_ **x** ; _θ_ ) =
_I_
_P_ ( _yi|_ **y** _\yi,_ **x** ; _θ_ ) (2)
_i_ =1
This equals to masking out each _yi_ and then recovering it. We build our model on top of Transformer
(Vaswani et al., 2017) which is the state-of-the-art
sequence-to-sequence architecture. Next, we will
discuss in detail the implementation of our model.


As self-attention is fully-connected, directly computing [�] _i_ _[I]_ =1 _[P]_ [(] _[y][i][|]_ **[y]** _[\][y][i][,]_ **[ x]** [;] _[ θ]_ [)][ with a vanilla Trans-]
former requires _I_ separate forward passes, in each
of which only one target token is masked out
and predicted. This is costly and time-consuming.
Therefore, how to parallelly mask out and predict
all target tokens in a single pass is important.
To do so, a major challenge is to avoid the representation of a masked token getting involved in
the prediction process of itself. Inspired by Kasai
et al. (2020), we modify the self-attention in the
Transformer decoder to perform the forward passes
concurrently. Given the word embedding **w** _i_ and
position embedding **p** _i_ for target token _yi_, we first
separate the query inputs **q** _i_ from key **k** _i_ and value
inputs **v** _i_ to prevent the to-be-predicted token itself
from participating in the prediction:
**q** _i_ = **p** _i_ **W** _[Q]_ (3)
**k** _i_ = ( **w** _i_ + **p** _i_ ) **W** _[K]_ (4)
**v** _i_ = ( **w** _i_ + **p** _i_ ) **W** _[V]_ (5)
where **W** _[Q]_, **W** _[K]_ and **W** _[V]_ are parameter matrices.
The hidden representation **h** _i_ for _yi_ is computed by
attending to keys and values, **K** = _i_ and **V** = _i_, that
correspond to the remaining tokens **y** _\yi_ :
**h** _i_ = Attention( **q** _i,_ **K** = _i,_ **V** = _i_ ) (6)
**K** = _i_ = Concat( _{_ **k** _m|m ̸_ = _i}_ ) (7)
**V** = _i_ = Concat( _{_ **v** _m|m ̸_ = _i}_ ) (8)
In this way, we ensure that **h** _i_ is isolated from the
word embedding **w** _i_ in a single decoder layer. However, there exists a problem of information leakage
if we update the key and value inputs for each position across decoder layers since they will contain
the representation of each position from previous
layers. Therefore, we keep the key and value inputs unchanged and only update the query inputs
Attention Weights
Figure 3: An example of inducing alignments from attention weights where the source token “.” has high attention weights. The two “in”s in the target sentence
are wrongly aligned to “.” because of the high attention
weights on it.
to avoid information leakage:
**h** _[l]_ _i_ [= Attention(] **[q]** _i_ _[l][,]_ **[ K]** [=] _[i][,]_ **[ V]** [=] _[i]_ [)] (9)
**q** _[l]_ _i_ [=] **[ h]** _i_ _[l][−]_ [1] **W** _[Q]_ (10)
where **q** _[l]_ _i_ [and] **[ h]** _i_ _[l]_ [denote the query inputs and hidden]
states for _yi_ in the _l_ -th layer, respectively. **h** [0] _i_ [is ini-]
tialized with **p** _i_ . We name this variant of attention
the **static-KV attention** . By static-KV, we mean
the keys and values are unchanged across different
layers in our approach. Our model replaces all selfattention in the decoder with static-KV attention.


Extracting alignments from vanilla cross-attention
often suffers from the high attention weights on
some specific source tokens such as periods, [EOS],
or other high frequency tokens (see Figure 3). This
is similar to the “garbage collectors” effect (Moore,
2004) in statistical aligners, where a source token
is aligned to too many target tokens. Hereinafter,
we will refer to these tokens as _collectors_ . As a
result of such effect, many target tokens (e.g., the
not 1.0


true
0.5 0.5
not true
true
1.0 falsch
falsch
|0.4|0.6|
|---|---|
|0.2|0.8|
falsch
falsch 0.2 0.4 0.4
[NULL]
[NULL]
not true
Figure 4: An illustrative example of the attention
weights from two directional models using vanilla and
leaky attention. Leaky attention provides a leak position “[NULL]” to collect extra attention weights.
two “in”s in Figure 3) will be incorrectly aligned
to the collectors according to the attention weights.
This phenomenon has been studied in previous
works (Clark et al., 2019; Kobayashi et al., 2020).
Kobayashi et al. (2020) show that the norms of the
value vectors for the collectors are usually small,
making their influence on attention outputs actually
limited. We conjecture that this phenomenon is due
to the incapability of NMT-based aligners to deal
with tokens that have no counterparts on the other
side because there is no empty (NULL) token that
is widely used in statistical aligners (Brown et al.,
1993; Och and Ney, 2003).
We propose to explicitly model the NULL token with an attention variant, namely **leaky atten-**
**tion** . As shown in Figure 4, when calculating crossattention weights, leaky attention provides an extra
“leak” position in addition to the encoder outputs.
Acting as the NULL token, this leak position is expected to address the biased attention weight problem. To be specific, we parameterize the key and
value vectors as **k** NULL and **v** NULL for the leak position in the cross-attention, and concatenate them
with the transformed vectors of the encoder outputs.
The attention output **z** _i_ is computed as follows:
**z** _i_ = Attention( **h** _[L]_ _i_ **[W]** _[Q][,]_ **[ K]** _[,]_ **[ V]** [)] (11)
**K** = Concat( **k** NULL _,_ **H** enc **W** _[K]_ ) (12)
**V** = Concat( **v** NULL _,_ **H** enc **W** _[V]_ ) (13)
where **H** enc denotes encoder outputs. [1] We use a
normal distribution with a mean of 0 and a small
1A similar attention implementation can be found
in [https://github.com/pytorch/fairseq/blob/master/fairseq/](https://github.com/pytorch/fairseq/blob/master/fairseq/modules/multihead_attention.py)
[modules/multihead](https://github.com/pytorch/fairseq/blob/master/fairseq/modules/multihead_attention.py) ~~a~~ ttention.py.
deviation to initialize **k** NULL and **v** NULL to ensure
that their initial norms are rather small. When extracting alignments, we only consider the attention
matrix without the leak position.
Note that leaky attention is different from adding
a special token in the source sequence, which will
share the same high attention weights with the existing collector instead of calibrating it (Vig and Belinkov, 2019). Our parameterized method is more
flexible than Leaky-Softmax (Sabour et al., 2017)
which adds an extra dimension with the value of
zero to the routing logits. In Section 2.2, we will
show that leaky attention is also helpful for applying agreement-based training on two directional
models.
We remove the cross-attention in all but the last
decoder layer. This makes the interaction between
the source and target restricted in the last layer.
Our experiments demonstrate that this modification improves alignment results with fewer model
parameters.


To better utilize the attention weights from two directions, we apply an agreement loss in the training
process to improve the symmetry of our model,
which has proven effective in statistical alignment
models (Liang et al., 2006; Liu et al., 2015). Given
a parallel sentence pair _⟨_ **x** _,_ **y** _⟩_, we can obtain the
attention weights from two different directions, denoted as _**W**_ **x** _→_ **y** and _**W**_ **y** _→_ **x** . As alignment is bijective, _**W**_ **x** _→_ **y** is supposed to be equal to the transpose of _**W**_ **y** _→_ **x** . We encourage this kind of symmetry through an agreement loss:
_La_ = MSE    - _**W**_ **x** _→_ **y** _,_ _**W**_ _[⊤]_ **y** _→_ **x**    - (14)
where MSE represents the mean squared error.
For vanilla attention, _La_ is hardly small because
of the normalization constraint. As shown in Figure
4, due to the use of softmax activation, the minimal
value of _La_ is 0 _._ 25 for vanilla attention. Using
leaky attention, our approach can achieve a lower
agreement loss ( _La_ = 0.1) by adjusting the weights
on the leak position.
However, our model may converge to a degenerate case of zero agreement loss where attention
weights are all zero except for the leak position.
We circumvent this case by introducing an entropy
loss on the attention weights:
_J_
- _W_ ˜ **x** _[ij]_ _→_ **y** [log ˜] _[W][ij]_ (15)
_j_ =1
(Zenkel et al., 2019, 2020) and used the preprocessing scripts from Zenkel et al. (2019) [3] . Following
Ding et al. (2019), we take the last 1000 sentences
of the training data for these three datasets as validation sets. We used a joint source and target Byte
Pair Encoding (BPE) (Sennrich et al., 2016) with
40k merge operations. During training, we filtered
out sentences with the length of 1 to ensure the
validity of the masking process.


We implemented our model based on the Transforemr architecture (Vaswani et al., 2017). The encoder consists of 6 standard Transformer encoder
layers. The decoder is composed of 6 layers, each
of which contains static-KV attention while only
the last layer is equipped with leaky attention. We
set the embedding size to 512, the hidden size to
1024, and attention heads to 4. The input and output
embeddings are shared for the decoder.
We trained the models with a batch size of 36K
tokens. We used early stopping based on the prediction accuracy on the validation sets. We tuned
the hyperparameters via grid search on the ChineseEnglish validation set as it contains gold word alignments. In all of our experiments, we set _λ_ = 0 _._ 05
(Eq. (16)), _α_ = 5, _β_ = 1 (Eq. (17)) and _τ_ = 0 _._ 2
(Eq. (19)). The evaluation metric is Alignment Error Rate (AER) (Och and Ney, 2000).


We introduce the following unsupervised neural
baselines besides two statistical baselines FASTALIGN and GIZA++:
  - NAIVE-ATT (Garg et al., 2019): a method
that induces alignments from cross-attention
weights of the best (usually penultimate) decoder layer in a vanilla Tranformer.
  - NAIVE-ATT-LAST: same as NAIVE-ATT except that only the last decoder layer performs
cross-attention.
  - ADDSGD (Zenkel et al., 2019): a method that
adds an extra alignment layer to repredict the
to-be-aligned target token.
  - MTL-FULLC (Garg et al., 2019): a method
that supervises an attention head with symmetrized NAIVE-ATT alignments in a multitask learning framework.
[3https://github.com/lilt/alignment-scripts](https://github.com/lilt/alignment-scripts)
_Le,_ **x** _→_ **y** = _−_ [1]
_I_
_I_
_i_ =1
_W_ **x** _[ij]_ _→_ **y** [+] _[ λ]_
_W_ ˜ **x** _[ij]_ _→_ **y** [=] ~~�~~ (16)
_j_ [(] _[W][ ij]_ **x** _→_ **y** [+] _[ λ]_ [)]
where _W_ [˜] **x** _[ij]_ _→_ **y** [is the renormalized attention weights]
and _λ_ is a smoothing hyperparamter. Similarly, we
have _Le,_ **y** _→_ **x** for the inverse direction.
We jointly train two directional models using the
following loss:
_L_ = _L_ **x** _→_ **y** + _L_ **y** _→_ **x** + _αLa_ +
(17)
_β_ ( _Le,_ **x** _→_ **y** + _Le,_ **y** _→_ **x** )
where _L_ **x** _→_ **y** and _L_ **y** _→_ **x** are NLL losses, _α_ and _β_
are hyperparameters.


When extracting alignments, we compute an alignment score _Sij_ for _yi_ and _xj_ as the harmonic mean
of attention weights _W_ **x** _[ij]_ _→_ **y** [and] _[ W][ ji]_ **y** _→_ **x** [from two]
directional models:


_Sij_ = [2] _[ W][ ij]_ (18)
_W_ **x** _[ij]_ _→_ **y** [+] _[ W][ ji]_ **y** _→_ **x**
We use the harmonic mean because we assume a
large _Sij_ requires both _W_ **x** _[ij]_ _→_ **y** [and] _[ W][ ji]_ **y** _→_ **x** [to be]
large. Word alignments can be induced from the
alignment score matrix as follows:
     - 1 if _Sij ≥_ _τ_
_Aij_ = (19)
0 otherwise
where _τ_ is a threshold.


We conducted our experiments on four public
datasets: German-English (De-En), English-French
(En-Fr), Romanian-English (Ro-En) and ChineseEnglish (Zh-En). The Chinese-English training set
is from the LDC corpus that consists of 1.2M sentence pairs. For validation and testing, we used the
Chinese-English alignment dataset from Liu et al.
(2005) [2], which contains 450 sentence pairs for validation and 450 for testing. For other three language pairs, we followed the experimental setup in
[2http://nlp.csai.tsinghua.edu.cn/](http://nlp.csai.tsinghua.edu.cn/~ly/systems/TsinghuaAligner/TsinghuaAligner.html) _∼_ ly/systems/
[TsinghuaAligner/TsinghuaAligner.html](http://nlp.csai.tsinghua.edu.cn/~ly/systems/TsinghuaAligner/TsinghuaAligner.html)


FAST-ALIGN (Dyer et al., 2013) N 25.7 12.1 31.8     GIZA++ (Och and Ney, 2003) N 17.8 6.1 26.0 18.5
NAIVE-ATT (Garg et al., 2019) N 31.9 18.5 32.9 28.9
NAIVE-ATT-LAST N 28.4 17.7 32.4 26.4
ADDSGD (Zenkel et al., 2019) N 21.2 10.0 27.6     MTL-FULLC (Garg et al., 2019) N 20.2 7.7 26.0     BAO (Zenkel et al., 2020) N 17.9 8.4 24.1     SHIFT-ATT (Chen et al., 2020) N 17.9 6.6 23.9 20.2
MTL-FULLC-GZ (Garg et al., 2019) Y 16.0 4.6 23.1     BAO-GUIDED (Zenkel et al., 2020) Y 16.3 5.0 23.4     SHIFT-AET (Chen et al., 2020) Y 15.4 4.7 21.2 17.2
MASK-ALIGN N **14.4** **4.4** **19.5** **13.8**
Table 1: Alignment Error Rate (AER) scores on four datasets for different alignment methods. The lower AER, the
better. “Guided” denotes whether the guided alignment loss is used during training. All results are symmetrized.
We highlight the best results for each language pair in bold.
  - BAO (Zenkel et al., 2020): an improved version of ADDSGD that extracts alignments
with Bidirectional Attention Optimization.
  - SHIFT-ATT (Chen et al., 2020): a method that
induces alignments when the to-be-aligned
tatget token is the decoder input instead of the
output.
We also included three additional baselines with
guided training: (1) MTL-FULLC-GZ (Garg et al.,
2019) which replaces the alignment labels in MTLFULLC with GIZA++ results, (2) BAO-GUIDED
(Zenkel et al., 2020) which uses alignments from
BAO for guided alignment training, (3) SHIFTAET (Chen et al., 2020) which trains an additional alignment module with supervision from
symmetrized SHIFT-ATT alignments.


Table 1 shows the results on four datasets. Our
approach significantly outperforms all statistical
and neural baselines. Specifically, it improves over
GIZA++ by 1.7-6.5 AER points across different
language pairs without using any guided alignment
loss, making it a good substitute to this commonly
used statistical alignment tool. Compared to SHIFTATT, the best neural methods without guided training, our approach achieves a gain of 2.2-6.4 AER
points with fewer parameters (as we remove some
cross-attention sublayers in the decoder).
When compared with baselines using guided
training, we find MASK-ALIGN still achieves sub
Masked Leaky Agree AER
_×_ _×_ _×_ 28.4
✓ _×_ _×_ 27.2
_×_ ✓ _×_ 28.3
_×_ _×_ ✓ 26.6
_×_ ✓ ✓ 23.4
✓ _×_ ✓ 17.6
✓ ✓ _×_ 17.2
✓ ✓ ✓ **14.4**
Table 2: Ablation study on the German-English dataset.
We use “Masked” to denote the masked modeling with
static-KV attention in Section 2.1, “Leaky” to denote
the leaky attention in Section 2.1 and “Agree” to denote
the agreement-based training and inference in Sections
2.2 and 2.3.
stantial improvements over all methods. For example, on the Romanian-English dataset, it improves over SHIFT-AET by 1.7 AER points. Recall
that our method is fully end-to-end, which does
not require a time-consuming process of obtaining
pseudo alignments for full training data.


Table 2 shows the ablation results on the GermanEnglish dataset. As we can see, masked modeling
seems to play a critical role since removing it will
deteriorate the performance by at least 9.0 AER.
We also find that leaky attention and agreementbased training and inference are both important.
Removing any of them will significantly diminish
human
rights
throughout
the
world
in
human
rights
throughout
the
world
in
(a) Vanilla Attention
(b) Leaky Attention
Figure 5: Attention weights from vanilla and leaky attention. “MR” is short for “menschenrechte”, which means
“human rights” in English. We use “[NULL]” to denote the leak position.
source sentence [NULL] MR in der welt 1995 _\_ 1996
vanilla attention    - 21.1 11.7 **5.2** 15.0 21.2 17.7 21.8
leaky attention **1.9** 28.5 17.2 18.1 20.2 24.2 21.4 23.8
Table 3: Norms of the transformed value vectors of different source tokens in Figure 5. We mark the minimum
norm for each variant of attention with boldface.
the performance.


Figure 5 shows the attention weights from vanilla
and leaky attention and Table 3 presents the norms
of the transformed value vectors of each source token for two types of attention. For vanilla attention,
we can see large weights on the high frequency
token “der” and the small norm of its transformed
value vector. As a result, the target token “in” will
be wrongly aligned to “der”. While for leaky attention, we observe a similar phenomenon on the leak
position “[NULL]”, and “in” will not be aligned to
any source tokens since the weights on all source tokens are small. This example shows leaky attention
can effectively prevent the collector phenomenon.


**Removing End Punctuation** To further investigate the performance of leaky attention, we tested
an extraction method that excludes the attention
weights on the end punctuation of a source sentence.
The reason behind this is that when the source sentence contains the end punctuation, it will act as the
collector in most cases. Therefore removing it will
Method w/ punc. w/o punc.
vanilla attention 27.2 17.7
leaky attention 17.2 17.4
Table 4: Comparison of AER with and without considering the attention weights on end punctuation.
alleviate the effect of collectors to a certain extent.
Table 4 shows the comparison results. For vanilla attention, removing end punctuation obtains a gain of
7.7 AER points. For leaky attention, however, such
extraction method brings no improvement on alignment quality. This suggests that leaky attention can
effectively alleviate the problem of collectors.
**Case Study** Figure 6 shows the attention weights
from four different models for the example in Figure 1. As we have discussed in Section 1, in this
example, NMT-based methods might fail to resolve
ambiguity when predicting the target token “tokyo”.
From the attention weight matrices, we can see that
NMT-based methods (Figures 6(b) and 6(c)) indeed put high weights wrongly on “1968” in the
source sentence. As for MASK-ALIGN, we can see
i
was
born
in
tokyo
in
.
(d) MASK-ALIGN
(a) Reference
(b) NAIVE-ATT-LAST
(c) SHIFT-ATT
Figure 6: Attention weights from different models for the example in Figure 1. Gold alignment is shown in (a). For
target token “tokyo”, NMT-based methods NAIVE-ATT-LAST (b) and SHIFT-ATT (c) assign high weights to the
wrongly aligned source token “1968”, while MASK-ALIGN (d) focuses on the correct source token “tokio”.
|7 cPcA
wPcA
6 cPwA
wPwA
5
4
3
2
1
0
Naive-Att Naive-Att-Last Shift-Att Mask-Align|cPcA
wPcA
cPwA
wPwA|Col3|Col4|
|---|---|---|---|
|Naive~~-~~Att
Naive~~-~~Att~~-~~Last
Shift~~-~~Att
Mask~~-~~Align
0
1
2
3
4
5
6
7
cPcA
wPcA
cPwA
wPwA|cPcA
wPcA
cPwA
wPwA|ift~~-~~Att
Ma|k~~-~~Align|
Figure 7: Relations between prediction and alignment
for different methods.
that the attention weights are highly consistent with
the gold alignment, showing that our method can
generate sparse and accurate attention weights.
**Prediction and Alignment** We analyzed the relevance between the correctness of word-level prediction and alignment. We regard a word as correctly
predicted if any of its subwords are correct and as
correctly aligned if one of its possible alignment
is matched. Figure 7 shows the results. We divide
target tokens into four categories:
1. cPcA: correct prediction & correct alignment;
2. wPcA: wrong prediction & correct alignment;
3. cPwA: correct prediction & wrong alignment;
4. wPwA: wrong prediction & wrong alignment.
Compared with other methods, MASK-ALIGN
significantly reduces the alignment errors caused by
wrong predictions (wPwA). In addition, the number of the tokens with correct prediction but wrong
alignment (cPwA) maintains at a low level, indicating that our model does not degenerate into a
target masked language model despite the use of
bidirectional target context.


Our work is closely related to unsupervised neural
word alignment. While early unsupervised neural
aligners (Tamura et al., 2014; Alkhouli et al., 2016;
Peter et al., 2017) failed to outperform their statistical counterparts such as FAST-ALIGN (Dyer et al.,
2013) and GIZA++ (Och and Ney, 2003), recent
studies have made significant progress by inducing
alignments from NMT models (Garg et al., 2019;
Zenkel et al., 2019, 2020; Chen et al., 2020). Our
work differs from prior studies in that we design a
novel self-supervised model that is capable of utilizing more target context than NMT-based models
to generate high quality alignments without using
guided training.
Our work is also inspired by the success of
conditional masked language models (CMLMs)
(Ghazvininejad et al., 2019), which have been applied to non-autoregressive machine translation.
The CMLM can leverage both previous and future
context on the target side for sequence-to-sequence
tasks with the masking mechanism. Kasai et al.
(2020) extend it with a disentangled context Transformer that predicts every target token conditioned
on arbitrary context. By taking the characteristics
of word alignment into consideration, we propose
to use static-KV attention to achieve masking and
aligning in parallel. To the best of our knowledge,
this is the first work that incorporates a CMLM into
alignment models.


We have presented a self-supervised neural alignment model MASK-ALIGN. Our model parallelly
masks out and predicts each target token. We
propose static-KV attention and leaky attention
to achieve parallel computation and address the
“garbage collectors” problem, respectively. Experiments show that MASK-ALIGN achieves new stateof-the-art results without using the guided alignment loss. In the future, we plan to extend our
method to directly generate symmetrized alignments without leveraging the agreement between
two unidirectional models.


Language Technologies Institute, Carnegie Mellon University
_{_ zdou,gneubig _}_ @cs.cmu.edu

Word Alignment by Fine-tuning Embeddings on Parallel Corpora


Word alignment over parallel corpora has a
wide variety of applications, including learning translation lexicons, cross-lingual transfer of language processing tools, and automatic evaluation or analysis of translation outputs. The great majority of past work on
word alignment has worked by performing
unsupervised learning on parallel text. Recently, however, other work has demonstrated
that pre-trained contextualized word embeddings derived from multilingually trained language models (LMs) prove an attractive alternative, achieving competitive results on the
word alignment task even in the absence of
explicit training on parallel data. In this paper, we examine methods to marry the two approaches: leveraging pre-trained LMs but finetuning them on parallel text with objectives
designed to improve alignment quality, and
proposing methods to effectively extract alignments from these fine-tuned models. We perform experiments on five language pairs and
demonstrate that our model can consistently
outperform previous state-of-the-art models of
all varieties. In addition, we demonstrate that
we are able to train multilingual word aligners that can obtain robust performance on different language pairs. Our aligner, **AWE-**
**SOME** ( **A** ligning **W** ord **E** mbedding **S** paces
**O** f **M** ultilingual **E** ncoders), with pre-trained
models is available at [https://github.](https://github.com/neulab/awesome-align)
[com/neulab/awesome-align.](https://github.com/neulab/awesome-align)


Word alignment is a useful tool to tackle a variety of
natural language processing (NLP) tasks, including
learning translation lexicons (Ammar et al., 2016;
Cao et al., 2019), cross-lingual transfer of language
processing tools (Yarowsky et al., 2001; Pado and´
Lapata, 2009; Tiedemann, 2014; Agic et al.´, 2016;
Mayhew et al., 2017; Nicolai and Yarowsky, 2019),
semantic parsing (Herzig and Berant, 2018) and
|0.89 0.82 0.78 0.80 0.77 0.77 0.78 0.79 envorwirdrungendeEine
0.83 0.85 0.82 0.81 0.78 0.77 0.79 0.79
0.78 0.80 0.84 0.84 0.77 0.74 0.78 0.80
0.82 0.80 0.78 0.80 0.88 0.86 0.81 0.81
0.78 0.78 0.79 0.78 0.81 0.82 0.82 0.77
0.76 0.77 0.76 0.77 0.78 0.81 0.84 0.77
.rdenm
0.76 0.73 0.74 0.75 0.77 0.80 0.80 0.78
0.79 0.75 0.76 0.75 0.76 0.73 0.77 0.95
nec ce o #s rs r #T i a e o h r ct e y n will be m ade .
Before Fine-tuning|0.89|0.82 0|.78 0.80|0.77|0.77 0.78|0.79|
|---|---|---|---|---|---|---|
|The
necessary
correct
##ion
will
be
made
.
Before Fine~~-~~tuning
Eine
ende
rung
wird
vor
men
rden
.
0.89
0.82
0.78
0.80
0.77
0.77
0.78
0.79
0.83
0.85
0.82
0.81
0.78
0.77
0.79
0.79
0.78
0.80
0.84
0.84
0.77
0.74
0.78
0.80
0.82
0.80
0.78
0.80
0.88
0.86
0.81
0.81
0.78
0.78
0.79
0.78
0.81
0.82
0.82
0.77
0.76
0.77
0.76
0.77
0.78
0.81
0.84
0.77
0.76
0.73
0.74
0.75
0.77
0.80
0.80
0.78
0.79
0.75
0.76
0.75
0.76
0.73
0.77
0.95|0.83|0.85
0|.82
0.81|0.78
|0.77
0.79|0.79|
|The
necessary
correct
##ion
will
be
made
.
Before Fine~~-~~tuning
Eine
ende
rung
wird
vor
men
rden
.
0.89
0.82
0.78
0.80
0.77
0.77
0.78
0.79
0.83
0.85
0.82
0.81
0.78
0.77
0.79
0.79
0.78
0.80
0.84
0.84
0.77
0.74
0.78
0.80
0.82
0.80
0.78
0.80
0.88
0.86
0.81
0.81
0.78
0.78
0.79
0.78
0.81
0.82
0.82
0.77
0.76
0.77
0.76
0.77
0.78
0.81
0.84
0.77
0.76
0.73
0.74
0.75
0.77
0.80
0.80
0.78
0.79
0.75
0.76
0.75
0.76
0.73
0.77
0.95|0.78|0.80
0|.84
0.84|0.77
|0.74
0.78|0.80|
|The
necessary
correct
##ion
will
be
made
.
Before Fine~~-~~tuning
Eine
ende
rung
wird
vor
men
rden
.
0.89
0.82
0.78
0.80
0.77
0.77
0.78
0.79
0.83
0.85
0.82
0.81
0.78
0.77
0.79
0.79
0.78
0.80
0.84
0.84
0.77
0.74
0.78
0.80
0.82
0.80
0.78
0.80
0.88
0.86
0.81
0.81
0.78
0.78
0.79
0.78
0.81
0.82
0.82
0.77
0.76
0.77
0.76
0.77
0.78
0.81
0.84
0.77
0.76
0.73
0.74
0.75
0.77
0.80
0.80
0.78
0.79
0.75
0.76
0.75
0.76
0.73
0.77
0.95|0.82|0.80
0|.78
0.80|0.88
|0.86
0.81|0.81|
|The
necessary
correct
##ion
will
be
made
.
Before Fine~~-~~tuning
Eine
ende
rung
wird
vor
men
rden
.
0.89
0.82
0.78
0.80
0.77
0.77
0.78
0.79
0.83
0.85
0.82
0.81
0.78
0.77
0.79
0.79
0.78
0.80
0.84
0.84
0.77
0.74
0.78
0.80
0.82
0.80
0.78
0.80
0.88
0.86
0.81
0.81
0.78
0.78
0.79
0.78
0.81
0.82
0.82
0.77
0.76
0.77
0.76
0.77
0.78
0.81
0.84
0.77
0.76
0.73
0.74
0.75
0.77
0.80
0.80
0.78
0.79
0.75
0.76
0.75
0.76
0.73
0.77
0.95|0.78|0.78
0|.79
0.78|0.81
|0.82
0.82|0.77|
|The
necessary
correct
##ion
will
be
made
.
Before Fine~~-~~tuning
Eine
ende
rung
wird
vor
men
rden
.
0.89
0.82
0.78
0.80
0.77
0.77
0.78
0.79
0.83
0.85
0.82
0.81
0.78
0.77
0.79
0.79
0.78
0.80
0.84
0.84
0.77
0.74
0.78
0.80
0.82
0.80
0.78
0.80
0.88
0.86
0.81
0.81
0.78
0.78
0.79
0.78
0.81
0.82
0.82
0.77
0.76
0.77
0.76
0.77
0.78
0.81
0.84
0.77
0.76
0.73
0.74
0.75
0.77
0.80
0.80
0.78
0.79
0.75
0.76
0.75
0.76
0.73
0.77
0.95|0.76|0.77
0|.76
0.77|0.78
|0.81
0.84|0.77|
|The
necessary
correct
##ion
will
be
made
.
Before Fine~~-~~tuning
Eine
ende
rung
wird
vor
men
rden
.
0.89
0.82
0.78
0.80
0.77
0.77
0.78
0.79
0.83
0.85
0.82
0.81
0.78
0.77
0.79
0.79
0.78
0.80
0.84
0.84
0.77
0.74
0.78
0.80
0.82
0.80
0.78
0.80
0.88
0.86
0.81
0.81
0.78
0.78
0.79
0.78
0.81
0.82
0.82
0.77
0.76
0.77
0.76
0.77
0.78
0.81
0.84
0.77
0.76
0.73
0.74
0.75
0.77
0.80
0.80
0.78
0.79
0.75
0.76
0.75
0.76
0.73
0.77
0.95|0.76|0.73
0|.74
0.75|0.77
|0.80
0.80|0.78|
|The
necessary
correct
##ion
will
be
made
.
Before Fine~~-~~tuning
Eine
ende
rung
wird
vor
men
rden
.
0.89
0.82
0.78
0.80
0.77
0.77
0.78
0.79
0.83
0.85
0.82
0.81
0.78
0.77
0.79
0.79
0.78
0.80
0.84
0.84
0.77
0.74
0.78
0.80
0.82
0.80
0.78
0.80
0.88
0.86
0.81
0.81
0.78
0.78
0.79
0.78
0.81
0.82
0.82
0.77
0.76
0.77
0.76
0.77
0.78
0.81
0.84
0.77
0.76
0.73
0.74
0.75
0.77
0.80
0.80
0.78
0.79
0.75
0.76
0.75
0.76
0.73
0.77
0.95|0.79|0.75
0|.76
0.75|0.76
|0.73
0.77|0.95|
|0.88|0.77|0.75 0.77|0.76|0.77 0.76|0.76|
|---|---|---|---|---|---|
|0.77|0.83
|0.77
0.76|0.75|0.72
0.75|0.75|
|0.73|0.71
|0.83
0.85|0.74|0.68
0.73|0.75|
|0.79|0.74
|0.72
0.74|0.90|0.87
0.77|0.76|
|0.73|0.75
|0.73
0.73|0.77|0.79
0.86|0.73|
|0.73|0.75
|0.72
0.74|0.76|0.80
0.88|0.73|
|0.77|0.72
|0.71
0.73|0.81|0.88
0.79|0.75|
|0.76|0.72
|0.75
0.75|0.75|0.71
0.73|0.97|
|The
necessary
correct
##ion
will
be
made
.
After Fine~~-~~tuning|The
necessary
correct
##ion
will
be
made
.
After Fine~~-~~tuning|The
necessary
correct
##ion
will
be
made
.
After Fine~~-~~tuning|The
necessary
correct
##ion
will
be
made
.
After Fine~~-~~tuning|The
necessary
correct
##ion
will
be
made
.
After Fine~~-~~tuning|The
necessary
correct
##ion
will
be
made
.
After Fine~~-~~tuning|
Figure 1: Cosine similarities between subword representations in a parallel sentence pair before and after
fine-tuning. Red boxes indicate the gold alignments.
speech recognition (Xu et al., 2019). In particular,
word alignment plays a crucial role in many machine translation (MT) related methods, including
guiding learned attention (Liu et al., 2016), incorporating lexicons during decoding (Arthur et al.,
2016), domain adaptation (Hu et al., 2019), unsupervised MT (Ren et al., 2020) and automatic
evaluation or analysis of translation models (Bau
et al., 2018; Stanovsky et al., 2019; Neubig et al.,
2019; Wang et al., 2020). However, with neural networks advancing the state of the arts in almost every
field of NLP, tools developed based on the 30-yearold IBM word-based translation models (Brown
et al., 1993), such as GIZA++ (Och and Ney, 2003)
or fast-align (Dyer et al., 2013), remain popular
choices for word alignment tasks.
One alternative to using statistical word-based
translation models to learn alignments would be to
instead train state-of-the-art neural machine translation (NMT) models on parallel corpora, and extract
alignments therefrom, as examined by Luong et al.
(2015); Garg et al. (2019); Zenkel et al. (2020).
However, these methods have two disadvantages
(also shared with more traditional alignment methods): (1) they are directional and the source and target side are treated differently and (2) they cannot
easily take advantage of large-scale contextualized
word embeddings derived from language models
(LMs) multilingually trained on monolingual corpora (Devlin et al., 2019; Lample and Conneau,
2019; Conneau et al., 2020), which have proven
useful in other cross-lingual transfer settings (Libovicky et al.`, 2019; Hu et al., 2020b). In the field
of word alignment, Sabet et al. (2020) have recently
proposed methods to align words using multilingual contextualized embeddings and achieve good
performance even in the absence of explicit training on parallel data, suggesting that these are an
attractive alternative for neural word alignment.
In this paper, we investigate if we can combine
the best of the two lines of approaches. Concretely,
we leverage pre-trained LMs and fine-tune them on
parallel text with not only LM-based objectives, but
also unsupervised objectives over the parallel corpus designed to improve alignment quality. Specifically, we propose a self-training objective, which
encourages aligned words to have further closer
contextualized representations, and a parallel sentence identification objective, which enables the
model to bring parallel sentences’ representations
closer to each other. In addition, we propose to effectively extract alignments from these fine-tuned
models using probability thresholding or optimal
transport.
We perform experiments on five different language pairs and demonstrate that our model can
achieve state-of-the-art performance on all of them.
In analysis, we find that these approaches also generate more aligned contextualized representations
after fine-tuning (see Figure 1 as an example) and
we can incorporate supervised signals within our
paradigm. Importantly, we show that it is possible
to train multilingual word aligners that can obtain
robust performance even in zero-shot settings, making them a valuable tool that can be used out-ofthe-box with good performance over a wide variety
of language pairs.


Formally, the task of word alignment can be defined as: given a sentence **x** = _⟨x_ 1 _, · · ·, xn⟩_ in
the source language and its corresponding parallel
sentence **y** = _⟨y_ 1 _, · · ·, ym⟩_ in the target language,
a word aligner needs to find a set of pairs of source
and target words:
_A_ = _{⟨xi, yj⟩_ : _xi ∈_ **x** _, yj ∈_ **y** _},_
where for each word pair _⟨xi, yj⟩_, _xi_ and _yj_ are semantically similar to each other within the context
of the sentence.
In the following paragraphs, we will first illustrate how we extract alignments from contextualized word embeddings, then describe our objectives
designed to improve alignment quality.


Contextualized word embedding models such as
BERT (Devlin et al., 2019) and RoBERTa (Liu
et al., 2019) represent words using continuous vectors calculated in context, and have achieved impressive performance on a diverse array of NLP
tasks. Multilingually trained word embedding models such as multilingual BERT can generate contextualized embeddings across different languages.
These models can be used to extract contextualized word embeddings _h_ **x** = _⟨hx_ 1 _, · · ·, hxn⟩_ and
_h_ **y** = _⟨hy_ 1 _, · · ·, hym⟩_ for each pair of parallel sentences **x** and **y** . Specifically, this is done by extracting the hidden states of the _i_ -th layer of the model,
where _i_ is an empirically-chosen hyper-parameter.
Given these contextualized word embeddings, we
propose two methods to calculate unidirectional
alignment scores based on probability simplexes
and optimal transport. We then turn these alignment scores into alignment matrices and reconcile
alignments in the forward and backward directions.
**Probability Thresholding.** In this method, for
each word in the source/target sentence, we calculate a value on the probability simplex for each
word in the aligned target/source sentence, and then
select all values that exceed a particular threshold
as “aligned” words. Concretely, taking inspiration
from attention mechanisms (Bahdanau et al., 2015;
Vaswani et al., 2017), we take the contextualized
embeddings _h_ **x** and _h_ **y** and compute the dot products between them and get the similarity matrix:
_S_ = _h_ **x** _h_ _[T]_ **y** _[.]_
Then, we apply a normalization function _N_ to
convert the similarity matrix into values on the
probability simplex _S_ **xy** = _N_ ( _S_ ), and treat _S_ **xy**
as the source-to-target alignment matrix. In this
paper, we propose to use _softmax_ and a sparse variant _α_ -entmax (Peters et al., 2019) to do the normalization. Compared with the _softmax_ function,
_α_ -entmax can produce sparse alignments for any
_α >_ 1 and assign non-zero probability to a short
Figure 2: Extracting word alignments from multilingual BERT using probability thresholding ( _softmax_ ). Red
boxes denote the gold alignments.
list of plausible word pairs, where a higher _α_ will
lead to a more sparse alignment.
**Optimal Transport.** The goal of optimal transport (Monge, 1781; Cuturi, 2013) is to find a mapping that moves probability from one distribution
to another, which can be used to find an optimal matching of similar words between two sequences (Kusner et al., 2015). Formally, in a discrete optimal transport problem, we are given two
point sets _{xi}_ _[n]_ _i_ =1 [and] _[ {][y][j][}]_ _j_ _[m]_ =1 [associated with]
their probability distributions _p_ **x** and _p_ **y** where
_i_ _[p][x]_ _i_ [= 1][ and][ �] _j_ _[p][y]_ _j_ [= 1][. Also, a function]
_C_ ( _xi, yj_ ) defines the cost of moving point _xi_ to
_yj_ . The goal of optimal transport is to find a mapping that moves probability mass from _{xi}_ _[n]_ _i_ =1 [to]
_{yj}_ _[m]_ _j_ =1 [and the total cost of moving the mass be-]
tween points is minimized. In other words, it finds
the transition matrix _S_ **xy** that minimizes:
_C_ ( _xi, yj_ ) _S_ **xy** _ij,_ (1)
_i,j_
where _S_ **xy1** _m_ = _p_ **x** and _S_ **xy** _[T]_ **[1]** _[n]_ [=] _[ p]_ **[y]** [. The re-]
sulting transition matrix is self-normalized and
sparse (Swanson et al., 2020), making it appealing alternative towards extracting alignments from
word embeddings.
In this paper, we propose to adapt optimal transport techniques to the task of word alignment. Concretely, we treat the parallel sentences **x** and **y** as
two point sets and assume each word is uniformly
distributed. The cost function is obtained by computing the pairwise distance (e.g. cosine distance)
between _h_ **x** and _h_ **y**, and all the distance values are
scaled to [0, 1] with min-max normalization. The
optimal transition matrix _S_ **xy** to Equation 1 can be
calculated using the Sinkhorn-Knopp matrix scaling algorithm (Sinkhorn and Knopp, 1967). If the
value of _S_ **xy** _ij_ is high, _xi_ and _yj_ are likely to have
similar semantics and values that exceed a particular threshold will be considered as “aligned”.
**Extracting** **Bidirectional** **Alignments.** After
we obtain both the source-to-target and target-tosource alignment probability matrices _S_ **xy** and _S_ **yx**
using the previous methods, we can deduce the final alignment matrix by taking the intersection of
the two matrices:
_A_ = ( _S_ **xy** _> c_ ) _∗_ ( _S_ **yx** _[T]_ _[> c]_ [)] _[,]_
where _c_ is a threshold and _Aij_ = 1 means _xi_ and
_yj_ are aligned.
Note that growing heuristics such as _grow-diag-_
_final_ (Och and Ney, 2000; Koehn et al., 2005) that
are popular in statistical word aligners can also be
applied in our alignment extraction algorithms, and
we will demonstrate the effect of these heuristics
in the experiment section.
**Handling Subwords.** Subword segmentation
techniques (Sennrich et al., 2016; Kudo and
Richardson, 2018) are widely used in training LMs,
thus the above alignment extraction methods can
only produce alignments on the subword level. To
convert them to word alignments, we follow previous work (Sabet et al., 2020; Zenkel et al., 2020)
and consider two words to be aligned if any of their
subwords are aligned. Figure 2 shows a concrete
example of how we extract word-level alignments
from a pre-trained embedding model.

2.1 Extracting Alignments from Embeddings


While language models can be used to produce
reasonable word alignments even without any finetuning (Sabet et al., 2020), we propose objectives
that further improve their alignment ability if we
have access to parallel data.
**Masked Language Modeling (MLM).** Gururangan et al. (2020) suggest that we can gain improvements in downstream tasks by further pretraining LMs on the task datasets. Therefore, we
propose to fine-tune the LMs with a masked language modeling objective on both the source and
target side of parallel corpora. Specifically, given a
pair of parallel sentences **x** and **y**, we choose 15%
of the token positions randomly for both **x** and **y**,
and for each chosen token, we replace it with (1)
the [MASK] token 80% of the time (2) a random
token 10% of the time and (3) unchanged 10% of
the time. The model is trained to reconstruct the
original tokens given the masked sentences **x** _[mask]_
and **y** _[mask]_ :
_LMLM_ = log _p_ ( **x** _|_ **x** _[mask]_ )+log _p_ ( **y** _|_ **y** _[mask]_ ) _._ (2)
**Translation Language Modeling (TLM).** The
MLM objective only requires monolingual data and
the model cannot make direct connections between
parallel sentences. To solve the issue, similarly
to Lample and Conneau (2019), we concatenate
parallel sentences **x** and **y** and perform MLM on
the concatenated data. Compared with MLM, the
translation language modeling (TLM) objective enable the model to align the source and target representations. Different from Lample and Conneau
(2019), we feed source and target sentences twice
in different orders instead of resetting the positions
of target sentences:
_LTLM_ = log _p_ ([ **x** ; **y** ] _|_ [ **x** _[mask]_ ; **y** _[mask]_ ])
(3)
+ log _p_ ([ **y** ; **x** ] _|_ [ **y** _[mask]_ ; **x** _[mask]_ ]) _._
**Self-training Objective (SO).** We also propose
a self-training objective for fine-tuning LMs which
is similar to the EM algorithm used in the IBM
models and the agreement constraints in Tamura
et al. (2014). Specifically, at each training step,
we first use our alignment extraction methods (described in Section 2.1) to extract the alignment _A_
for **x** and **y**, then maximize the following objective:
_Aij_ 1 _S_ **xy** _ij_
2 [(] _n_
_i,j_
 _LSO_ =
_ij_ + _S_ **yx** [T] _ij_
_n_ _m_
(4)
_m_ [)] _[.]_


#Train Sents. 1.9M 1.1M 450K 444K 40K
#Test Sents. 508 447 248 582 450
Table 1: Statistics of datasets.
spurious alignments and encourage the source-totarget and target-to-source alignments to be symmetrical to each other by exploiting their agreement (Liang et al., 2006).
**Parallel Sentence Identification (PSI).** We also
propose a contrastive parallel sentence identification loss that attempts to make parallel sentences
more similar than mismatched sentence pairs (Liu
and Sun, 2015; Legrand et al., 2016). This encourages the overall alignments of embeddings on both
word and sentence level to be closer together. Concretely, we randomly select a pair of parallel or
non-parallel sentences _⟨_ **x** _[′]_ _,_ **y** _[′]_ _⟩_ from the training
data with equal probability. Then, the model is
required to predict whether the two sampled sentences are parallel or not. The representation of the
first [CLS] token is fed into a multi-layer perceptron to output a prediction score _s_ ( **x** _[′]_ _,_ **y** _[′]_ ). Denoting the binary label as _l_, the objective function can
be written as:
_LPSI_ = _l_ log _s_ ( **x** _[′]_ _,_ **y** _[′]_ )+(1 _−_ _l_ ) log(1 _−_ _s_ ( **x** _[′]_ _,_ **y** _[′]_ )) _._
(5)
**Consistency Optimization (CO).** While the
self-training objective can potentially improve the
symmetricity between forward and backward alignments, following previous work on machine translation and multilingual representation learning (Cohn
et al., 2016; Zhang et al., 2019; Hu et al., 2020a),
we use an objective to explicitly encourage the
consistency between the two alignment matrices.
Specifically, we maximize the trace of _S_ **xy** [T] _[S]_ **[yx]** [:]
**xy** _[S]_ **[yx]** [)]
_LCO_ = [trace][(] _[S]_ [T] _._ (6)
min( _m, n_ )
**Our Final Objective.** In summary, our training
objective is a combination of the proposed objectives and we train the model with them jointly at
each training step:
_L_ = _LMLM_ + _LTLM_ + _LSO_ + _LPSI_ + _βLCO,_
where _β_ is set to 0 or 1 in our experiments.
Intuitively, this objective encourages words
aligned in the first pass of alignment to have further
closer contextualized representations. In addition,
because of the intersection operation during extraction, the self-training objective can ideally reduce


_Baseline_
SimAlign _w/o fine-tuning_ 18.8 7.6 27.2 46.6 21.6
fast ~~a~~ lign _bilingual_ 27.0 10.5 32.1 51.1 38.1
eflomal _bilingual_ 22.6 8.2 25.1 47.5 28.7
GIZA++ _bilingual_ 20.6 5.9 26.4 48.0 35.1
Zenkel et al. (2020) _bilingual_ 16.0 5.0 23.4 - Chen et al. (2020) _bilingual_ 15.4 4.7 21.2 -
_Ours_
_α_ -entmax
_softmax_
_w/o fine-tuning_ 18.1 5.6 29.0 46.3 18.4
_bilingual_ 16.1 _**4.1**_ 23.4 38.6 15.4
_multilingual (β = 0)_ 15.4 _**4.1**_ 22.9 _**37.4**_ **13.9**
_multilingual (β = 1)_ _**15.0**_ 4.5 **20.8** 38.7 14.5
_zero-shot_ 16.0 4.3 28.4 44.0 **13.9**
_w/o fine-tuning_ 17.4 5.6 27.9 45.6 18.1
_bilingual_ 15.6 **4.4** 23.0 38.4 15.3
_multilingual (β = 0)_ 15.3 **4.4** 22.6 **37.9** _**13.6**_
_multilingual (β = 1)_ **15.1** 4.5 _**20.7**_ 38.4 14.5
_zero-shot_ 15.7 4.6 27.2 43.7 14.0
Table 2: Performance (AER) of our models in bilingual, multilingual and zero-shot settings. The best scores for
each alignment extraction method are in **bold** and the overall best scores are in _**italicized bold**_ .

Model Setting De-En Fr-En Ro-En Ja-En Zh-En


In this section, we first present our main results,
then conduct several ablation studies and analyses
of our models.


**Datasets.** We perform experiments on five different language pairs, namely German-English (DeEn), French-English (Fr-En), Romanian-English
(Ro-En), Japanese-English (Ja-En) and ChineseEnglish (Zh-En). For the De-En, Fr-En, Ro-En
datasets, we follow the experimental setting of previous work (Zenkel et al., 2019; Garg et al., 2019;
Zenkel et al., 2020). The training and test data
for Ro-En and Fr-En are provided by Mihalcea
and Pedersen (2003). The Ro-En training data are
also augmented by the Europarl v8 corpus (Koehn,
2005). For the De-En data, the Europarl v7 corpus
is used as training data and the gold alignments
are provided by Vilar et al. (2006). The Ja-En
dataset is obtained from the Kyoto Free Translation
Task (KFTT) word alignment data (Neubig, 2011),
and the Japanese sentences are tokenized with the
KyTea tokenizer (Neubig et al., 2011). The ZhEn dataset is obtained from the TsinghuaAligner
website [1] . We treat their evaluation set as the training data and use the test set in Liu and Sun (2015)
[1http://nlp.csai.tsinghua.edu.cn/˜ly/](http://nlp.csai.tsinghua.edu.cn/~ly/systems/TsinghuaAligner/TsinghuaAligner.html)
[systems/TsinghuaAligner/TsinghuaAligner.](http://nlp.csai.tsinghua.edu.cn/~ly/systems/TsinghuaAligner/TsinghuaAligner.html)
[html](http://nlp.csai.tsinghua.edu.cn/~ly/systems/TsinghuaAligner/TsinghuaAligner.html)
ignoring possible alignments. The De-En, En-Fr
datasets contain the distinction between sure and
possible alignment links. The statistics of these
datasets are shown in Table 1. We use the Ja-En
development set to tune the hyper-parameters.
**Baselines.** We compare our models with:
  - fast ~~a~~ lign (Dyer et al., 2013): a popular statistical word aligner which is a simple, fast
reparameterization of IBM Model 2.
  - eflomal (Ostling and Tiedemann [¨], 2016): an
efficient statistical word aligner using a
Bayesian model with Markov Chain Monte
Carlo (MCMC) inference.
  - GIZA++ (Och and Ney, 2003; Gao and Vogel,
2008): an implementation of IBM models.
Following previous work (Zenkel et al., 2020),
we use five iterations each for Model 1, the
HMM model, Model 3 and Model 4.
  - SimAlign (Sabet et al., 2020): a BERT-based
word aligner that is not fine-tuned on any parallel data. The authors propose three alignment extraction methods and we implement
their IterMax model with default parameters.
  - Zenkel et al. (2020) and Chen et al. (2020):
two state-of-the-art neural word aligners
based on MT models.
**Implementation Details.** Our main results are
obtained by using the probability thresholding
method on the contextualized embeddings in the
8-th layer of multilingual BERT-Base (mBERT; Devlin et al. (2019)) and we will discuss this choice
in our ablation studies. We use the AdamW optimizer (Loshchilov and Hutter, 2019) with a learning rate of 2e-5 and the batch size is set to 8. Following Peters et al. (2019), we set _α_ to 1.5 for
_α_ -entmax. The threshold _c_ is set to 0 for _α_ -entmax
and 0.001 for _softmax_ and optimal transport. Unless otherwise stated, _β_ is set to 0. We mainly
evaluate the model performance using Alignment
Error Rate (AER).


We first train our model on each individual language pair, then investigate if it is possible to train
multilingual word aligners.
**Bilingual Model Performance.** From Table 2,
we can see that our _softmax_ model can achieve
consistent improvements over the baseline models,
demonstrating the effectiveness of our proposed
method. Surprisingly, directly extracting alignments from mBERT (the _w/o fine-tuning_ setting)
can already achieve better performance than the
popular statistical word aligner GIZA++ on 4 out
of 5 settings, especially in the Zh-En setting where
the size of parallel data is small.
**Multilingual Model Performance.** We also randomly sample 200k parallel sentence pairs from
each language pair (except for Zh-En where we
take all of its 40k parallel sentences) and concatenate them together to train multilingual word aligners. As shown in Table 2, the multilingually trained
word aligners can achieve further improvements
and they consistently outperform our bilingual
word aligners and all the baselines even though the
size of training data for each individual language
pair is smaller. The results demonstrate that we can
indeed obtain a neural word aligner that has stateof-the-art and robust performance across different
language pairs. We also test the performance of our
consistency optimization objective in this setting.
We can see that incorporating this objective ( _β_ =1)
can significantly improve the model performance
on Ro-En, while it also deteriorates the Ja-En and
Zh-En performance by a non-negligible margin.
We find that this is because the CO objective can
significantly improve the alignment recall while
sacrificing the precisions, and our Ro-En dataset


_softmax_ **17.4** **5.6** **27.9** **45.6** **18.1** **33.22**
Prob.
_α_ -entmax 18.1 **5.6** 29.0 46.3 18.4 32.36
Cosine 24.4 15.7 33.7 54.0 31.1 3.36
OT Dot Product 25.4 17.1 34.1 54.2 30.9 3.82
Euclidean 20.7 15.1 33.3 53.2 29.8 3.05
Table 3: Comparisons of probability thresholding
(Prob.) and optimal transport (OT) for alignment extraction. We try both _softmax_ and _α_ -entmax for probability thresholding and different cost functions for optimal transport. We measure both the extraction speed
(#sentences/seconds) and the alignment quality (AER)
on five language pairs, namely German-English (DeEn), French-English (Fr-En), Romanian-English (RoEn), Japanese-English (Ja-En), and Chinese-English
(Zh-En). The best scores are in **bold** .
tends to favor models with high recall and the Ja-En
and Zh-En datasets have an opposite tendency.
**Zero-Shot Performance.** In this paragraph, we
want to find out how our models perform on language pairs that it has never seen during training.
To this end, for each language pair, we train our
model with data of all the other language pairs and
test its performance on the target language pair.
Results in Table 2 demonstrate that training our
models with parallel data on _other_ language pairs
can still improve the model performance on the
target language pair. This is a very important result, as it indicates that our model can be used as a
off-the-shelf tool for multilingual word alignment
for any language supported by the underlying embeddings, _regardless of whether parallel data has_
_been used for training or not_ .

Component De-En Fr-En Ro-En Ja-En Zh-En Speed


In this part, we compare the performance of different alignment extraction methods, pre-trained
embedding models and training objectives.
**Alignment Extraction Methods.** We first compare the performance of our two proposed alignment extraction methods, namely the probability
thresholding and optimal transport techniques. We
use the representations of the 8-th layer of mBERT
following Sabet et al. (2020).
As shown in Table 3, probability thresholding
methods can consistently outperform optimal transport by a large margin on the five language pairs.
In addition, probability thresholding methods are
much faster than optimal transport. _softmax_ is
marginally better than _α_ -entmax, yet one advantage
of _α_ -entmax is that we do not need to manually set


7 18.7 6.1 19.1
mBERT 8 **17.4** **5.6** _**18.1**_
9 18.8 6.1 20.1
4 21.1 6.8 **25.3**
XLM-15 (MLM) 5 **20.4** **6.1** 26.1
6 23.2 7.7 33.3
4 16.4 4.9 **18.6**
XLM-15 (MLM+TLM) 5 _**16.2**_ _**4.7**_ 23.7
6 18.8 5.7 26.2
7 20.5 8.5 30.8
XLM-100 (MLM) 8 **19.8** **8.2** **28.6**
9 19.9 8.8 29.3
7 24.4 10.3 33.2
XLM-R 8 **23.1** **9.2** 30.7
9 24.7 11.5 **28.1**
Table 4: Comparisons of different LMs in terms of
AER. We extract alignments using _softmax_ and take
representations from different layers of LMs. The best
scores for each individual model are in **bold** and the
overall best scores are in _**italicized bold**_ .
the threshold. Therefore, we use both _softmax_ and
_α_ -entmax to obtain the main results.
**Pre-trained Embedding Models.** In this paragraph, we investigate the performance of three
different types of pre-trained embedding models,
including mBERT, XLM (Lample and Conneau,
2019) and XLM-R (Conneau et al., 2020). For
XLM, we have tried its three released models: 1)
XLM-15 (MLM) pre-trained with MLM and supports 15 languages; 2) XLM-15 (MLM+TLM) pretrained with both the MLM and TLM objectives
and supports 15 languages; 3) XLM-100 (MLM)
pre-trained with MLM and supports 100 languages.
We use _softmax_ to extract the alignments.
Because XLM-15 does not support Japanese or
Romanian, we only report the performance on the
three other language pairs in Table 4. We take representations from different layers and report the performance of the best three layers. We can see that
while XLM-15 (MLM+TLM) can achieve the best
performance on De-En and Fr-En, the best layer is
not consistent across language pairs. On the other
hand, the optimal configurations for mBERT are
consistent across language pairs. In addition, considering mBERT supports many more languages
than XLM-15 (MLM+TLM), we will use mBERT
in the following sections.
**Training Objectives.** We also conduct ablation
studies on each of our training objectives. We can
see from Table 5 that the self-training objective
can best improve the model performance. Also,
the translation language modeling and parallel sentence identification objectives can marginally benefit the model. The masked language modeling objective, on the other hand, cannot always improve
the model and can sometimes even deteriorate the
model performance, possibly because the TLM objective already provides the model with sufficient
supervision signals.


We conduct several analyses to better understand
our models. Unless otherwise stated, we perform
experiments on the _softmax_ model using mBERT.
**Incorporating Supervised Signals.** We investigate if our models can benefit from supervised signals. If we have access to word-level gold labels
for word alignment, we can simply utilize them in
our self-training objectives. Specifically, we can set
_Aij_ in Equation 4 to 1 if and only if they are aligned.
In our experimental settings, we have gold labels
for all the Zh-En sentences and 653 sentences from
the Ja-En development set. Table 6 demonstrates
that training our models with as few as 653 labeled
sentences can dramatically improve the alignment
quality, and combining labeled and unlabeled parallel data can further improve the model performance. This analysis demonstrate the generality
of our models as they can also be applied in semisupervised settings.
**Growing Heuristics.** As stated in Section 2.1,
because our alignment extraction methods essentially take the intersection of forward and backward alignments, growing heuristics can also be
applied in our settings. The main motivation of
growing heuristics is to improve the recall of the
resulting alignments. While effective in statistical
word aligners, as shown in Table 7, the growing
heuristics only improve our alignment extraction
method on the vanilla mBERT model in the Ro-En
setting while degrading the model performance on
all the other language pairs. After fine-tuning, the
growing heuristics can only hurt the model performance, possibly because the self-training objective
encourages the forward and backward alignments
to be symmetrical. Based on these results, we do
not adopt the growing heuristics in our models.
**Annotation Projection.** Word alignment has
been a useful tool in cross-lingual annotation
projection (Yarowsky et al., 2001; Nicolai and
Yarowsky, 2019). Therefore, it would be inter


_softmax_
_All_ 15.3 4.4 22.6 37.9 13.6
_All w/o MLM_ 15.3 4.4 22.8 38.6 13.7
_All w/o TLM_ 15.5 4.7 22.9 39.7 14.0
_All w/o SO_ 16.9 4.8 23.0 39.1 15.4
_All w/o PSI_ 15.4 4.4 22.7 37.9 13.8
Table 5: Ablation studies on our training objectives in multilingual settings.
Figure 3: An example of extracting alignments from our fine-tuned model using _softmax_ . Red boxes indicate the
gold alignments. The fine-tuned model can generate more accurate alignments then vanilla mBERT (Figure 2).
**Lang.** **Unsup.** **Sup.** **Semi-Sup.**
Zh-En 15.3 12.5   Ja-En 38.4 31.6 30.0
Table 6: Incorporating supervised word alignment signals into our model can further improve the model performance in terms of AER.

Model Objective De-En Fr-En Ro-En Ja-En Zh-En


mBERT
Ours-Multi.
X-En 24.7 14.4 31.9 54.7 27.4
En-X 22.6 12.2 32.0 52.7 29.9
_softmax_ 17.4 5.6 27.9 45.6 18.1
gd 18.7 9.2 27.0 48.5 23.4
gd-fnal 18.6 9.3 26.9 48.7 23.2
X-En 20.2 12.9 25.4 42.1 19.3
En-X 18.1 9.3 25.9 41.7 23.5
_softmax_ 15.3 4.4 22.6 37.9 13.6
gd 16.3 8.1 23.1 38.2 18.3
gd-final 16.5 8.3 23.2 38.7 18.5
Table 7: The _grow-diag-final_ heuristic can only
improve our alignment extraction method in the
Romanian-English setting without fine-tuning. “gd”
refers to grow-diag.
esting to see if our model can be beneficial in these
settings. To this end, we evaluate our model and
baselines on cross-lingual named entity recognition
(NER). We train a BERT-based NER model on the
CoNLL 2003 English data (Tjong Kim Sang and
De Meulder, 2003) and test it on the CoNLL 2002
Spanish data (Tjong Kim Sang, 2002). We use
Google Translate to translate Spanish test set into
English, predict the labels using the NER model,
then project the labels from English to Spanish us

Model Ext. De-En Fr-En Ro-En Ja-En Zh-En


BERT-En (zero-shot) 53.1 54.3 52.7
fast align 51.5 59.8 55.2
GIZA++ 56.5 64.1 60.0
SimAlign 59.9 67.6 63.5
Ours **60.6** **68.5** **64.3**
Table 8: Our model is also effective in an annotation
projection setting where we train a BERT-based NER
model on English data and test it on Spanish data. The
best scores are in **bold** .
ing word aligners. From Table 8, we can see that
our model is also better than baselines in this setting, demonstrating its usefulness in cross-lingual
annotation projection.
**Sentence-Level Representation Transfer.** We
also test if the aligned representations are beneficial for sentence-level cross-lingual transfer. In
doing so, we perform experiments on XNLI (Conneau et al., 2018), which evaluates cross-lingual
sentence representations in 15 languages on the
task of natural language inference (NLI). We train
our models with the provided 10k parallel data on
the 15 languages, fine-tune our model on the English NLI data, then test its performance on other
languages. As shown in Table 9, our model can outperform the baseline, indicating the aligned word
representations can also be helpful for sentencelevel cross-lingual transfer.
**Model** **En** **Fr** **Es** **De** **El** **Bg** **Ru** **Tr** **Ar** **Vi** **Th** **Zh** **Hi** **Sw** **Ur** **Ave.**
mBERT 81.3 73.4 74.3 70.5 66.9 68.2 68.5 59.5 64.3 **70.6** 50.7 68.8 59.3 49.4 57.5 65.5
Ours **81.5** **74.1*** **74.9*** **71.2*** **67.1** **68.7*** **68.6** **61.0*** **66.2*** 70.5 **53.8*** **69.1** **59.8*** **50.6*** **58.6*** **66.4***
Table 9: Results of mBERT and our fine-tuned model on XNLI (Conneau et al., 2018). Our objectives can improve
the model cross-lingual transfer ability. “*” denotes significant differences using paired bootstrapping (p _<_ 0.05) .
**Alignment Examples.** We also conduct qualitative analyses as shown in Figure 1, 2 and 3. After
fine-tuning, the learned contextualized representations are more aligned, as the cosine distances
between semantically similar words become closer,
and the extracted alignments are more accurate.
More examples are shown in Appendix B.


Based on the IBM translation models (Brown et al.,
1993), many statistical word aligners have been
proposed (Vogel et al., 1996; Ostling and Tiede- [¨]
mann, 2016), including the current most popular
tools GIZA++ (Och and Ney, 2000, 2003; Gao and
Vogel, 2008) and fast ~~a~~ lign (Dyer et al., 2013).
Recently, there is a resurgence of interest in neural word alignment (Tamura et al., 2014; Alkhouli
et al., 2018). Based on NMT models trained on
parallel corpora, researchers have proposed several methods to extract alignments from them (Luong et al., 2015; Zenkel et al., 2019; Garg et al.,
2019; Li et al., 2019) and successfully build an
end-to-end neural model that can outperform statistical tools (Zenkel et al., 2020). However, there
is an inherent discrepancy between translation and
word alignment: translation models are directional
and the source and target side are treated differently, while word alignment is a non-directional
task. Therefore, certain adaptations are required
for translation models to perform word alignment.
Another disadvantage of MT-based word aligners is that they cannot easily utilize contextualized
embeddings. Using learned representations to improve word alignment have been investigated (Sabet et al., 2016; Pourdamghani et al., 2018). Recently, pre-trained LMs (Peters et al., 2018; Devlin
et al., 2019; Brown et al., 2020) have proven to be
useful in cross-lingual transfer (Libovicky et al.`,
2019; Hu et al., 2020b). In word alignment, Sabet
et al. (2020) propose effective methods to extract
alignments from multilingual LMs without explicit
training on parallel data. In this work, we propose
better alignment extraction methods and combine
the best of the two worlds by fine-tuning contextualized embeddings on parallel data.
There are also work on supervised neural word
alignment (Stengel-Eskin et al., 2019; Nagata et al.,
2020). However, supervised data are not always
accessible, making their methods inapplicable in
many scenarios. In this paper, we demonstrate that
our model can incorporate supervised signals if
available and perform semi-supervised learning,
which is a more realistic and general setting.
Some work on bilingual lexicon induction also
share similar general ideas with ours. For example, Zhang et al. (2017) minimize the earth mover’s
distance to match the embedding distributions from
different languages. Similarly, Grave et al. (2019)
present an algorithm to align point clouds with
Procrustes (Schonemann¨, 1966) in Wasserstein distance for unsupervised embedding alignment.


We present a neural word aligner that achieves stateof-the-art performance on five diverse language
pairs and obtains robust performance in zero-shot
settings. We propose to fine-tune multilingual embeddings with objectives suitable for word alignment and develop two alignment extraction methods. We also demonstrate its applications in semisupervised settings. We hope our word aligner can
be a tool that can be used out-of-the-box with good
performance over various language pairs. Future
directions include designing better training objectives and experimenting on more language pairs.
Also, note that we mainly evaluate our word
aligners using AER following previous work,
which has certain limitations. For example, it
may not be well-correlated with statistical machine
translation performance Fraser and Marcu (2007)
and different types of alignments can be suitable for
different tasks or conditions (Lambert et al., 2012;
Stymne et al., 2014). Although we have evaluated
models in annotation projection and cross-lingual
transfer settings, alternative metrics (Tiedemann,
2005; Søgaard and Wu, 2009; Ahrenberg, 2010)
are also worth considering in the future.


_Baseline_
Yao et al. (2013a) 91.3 82.0 86.4
Yao et al. (2013b) 90.4 81.9 85.9
Sultan et al. (2014) **93.5** 82.6 87.6
_Ours_
mBERT 87.0 89.0 88.0
Ours-Multilingual 87.0 89.3 88.1
Ours-Supervised 87.2 **89.8** **88.5**
Table 10: Our model is also effective in monolingual
alignment settings.


We use the AdamW optimizer (Loshchilov and
Hutter, 2019) with a learning rate of 2e-5 and the
batch size is set to 8. Following Peters et al. (2019),
we set _α_ to 1.5 for _α_ -entmax. The threshold _c_ is
set to 0 for _α_ -entmax and 0.001 for _softmax_ and
optimal transport. We train our models on one 2080
Ti for one epoch and it takes 3 to 24 hours for the
model to converge depending on the size of the
dataset. We evaluate the model performance using
Alignment Error Rate (AER).


In this section, we conduct more analyses of our
models.
**Monolingual Alignment.** We also investigate
how our models perform in monolingual alignment
settings. Previous methods (MacCartney et al.,
2008; Yao et al., 2013a,b; Sultan et al., 2014) typically exploit external resources such as WordNet to
tackle the problem. As shown in Table 10, mBERT
can outperform previous methods in terms of recall and F1 without any fine-tuning. Our multilingually fine-tuned model can achieve better recall
and slightly better F1 score than the vanilla mBERT
model, and fine-tuning our model with supervised
signals can achieve further improvements.
**Sensitivity Analysis.** We also conduct a sensitivity analysis on the threshold _c_ for our _softmax_ alignment extraction method. As shown in Table 11, our
method is relatively robust to this threshold. In particular, after fine-tuning, the AERs change within
0.5% when varying the threshold.
**Comparisons with IterMax.** IterMax is the best
alignment extraction method in SimAlign (Sabet
et al., 2020). The results in the main paper have
demonstrated that our alignment extraction methods are able to outperform IterMax. In Figure 4, we


mBERT
Ours-Multilingual
1e-6 17.3 6.0 27.2 45.2 18.9
1e-5 17.3 5.9 27.4 45.1 18.6
1e-4 17.3 5.7 27.6 45.3 18.3
1e-3 17.4 5.6 27.9 45.6 18.1
1e-2 17.7 5.6 28.4 45.8 18.2
1e-1 18.1 5.6 28.9 46.3 18.3
5e-1 18.4 5.6 29.5 47.0 18.7
1e-6 15.4 4.6 22.7 38.2 14.1
1e-5 15.4 4.5 22.7 38.1 14.0
1e-4 15.3 4.5 22.6 37.9 13.9
1e-3 15.3 4.4 22.6 37.9 13.8
1e-2 15.3 4.3 22.7 37.9 13.8
1e-1 15.4 4.3 22.8 38.0 13.8
5e-1 15.4 4.2 23.0 38.2 13.9
Table 11: Our _softmax_ alignment extraction method is
relatively robust to the threshold _c_ .
can see that the IterMax algorithm tends to sacrifice
precision for a small improvements in recall, while
our model can generate more accurate alignments.
**Ablation Studies on Training Objectives.** Table 12 presents more ablation studies on our training objectives. We can see that the self training
objective is the most effective one, with the translation language modeling objective being the second
and the parallel sentence identification objective
being the third. The masked language modeling objective can sometimes hurt the model performance,
possibly because of the translation language modeling objective.
**Experiments on More Language Pairs.** We
also test our alignment extraction methods on other
language pairs following the setting of Sabet et al.
(2020) without fine-tuning as shown in Table 13. [2]
**More Qualitative Examples.** In addition to the
examples provided in the main text, we also present
some randomly sampled samples in Figure 5. We
can clearly see that our model learns more aligned
representations than the baseline model.
2Their English-Persian dataset is unavailable at the time
of writing the paper.


_Ours-Bilingual_
_α_ -entmax
_All_ 16.1 4.1 23.4 38.6 15.4
_All w/o MLM_ 15.6 4.2 23.3 38.8 15.1
_All w/o TLM_ 16.4 4.3 23.7 40.1 15.3
_All w/o SO_ 17.8 4.7 23.9 39.4 16.3
_All w/o PSI_ 16.5 4.2 23.1 38.5 15.4
_All_ 15.6 4.4 23.0 38.4 15.3
_All w/o MLM_ 15.5 4.2 23.2 38.9 14.9
_All w/o TLM_ 15.9 4.5 23.7 40.1 15.1
_softmax_
_All w/o SO_ 17.4 4.7 23.2 38.6 16.3
_All w/o PSI_ 15.6 4.3 23.1 38.8 15.4
_Ours-Multilingual_
_α_ -entmax
_softmax_
_All_ 15.4 4.1 22.9 37.4 13.9
_All w/o MLM_ 15.1 4.2 22.8 37.8 13.7
_All w/o TLM_ 16.4 4.4 23.3 39.7 14.4
_All w/o SO_ 17.5 4.6 23.6 40.0 15.6
_All w/o PSI_ 15.5 3.9 23.0 38.2 14.1
_All_ 15.3 4.4 22.6 37.9 13.6
_All w/o MLM_ 15.3 4.4 22.8 38.6 13.7
_All w/o TLM_ 15.5 4.7 22.9 39.7 14.0
_All w/o SO_ 16.9 4.8 23.0 39.1 15.4
_All w/o PSI_ 15.4 4.4 22.7 37.9 13.8
Table 12: Ablation studies on training objectives.


GIZA++ 18.2 51.8
SimAlign 13.4 40.2
Ours ( _softmax_, _c_ =1e-3) **12.3** 41.2
Ours ( _softmax_, _c_ =1e-5) 12.7 39.5
Ours ( _softmax_, _c_ =1e-7) 13.3 **39.2**
Table 13: Performance on more language pairs.
(a) mBERT Itermax
(b) mBERT _softmax_
(c) Fine-tuned IterMax
(d) Fine-tuned _softmax_
Figure 4: Extracting alignments from our model using IterMax(Sabet et al., 2020) and our _softmax_ method from
the vanilla and fine-tuned mBERT models.
|.86 0.80 0.80 0.78 0.77|0.73 0.71 0.7|3 0.74 0.7|5 0.81|
|---|---|---|---|
|.80
0.81
0.75
0.74
0.72
.79
0.81
0.87
0.75
0.73|0.70
0.67
0.7
0.73
0.68
0.7|0
0.71
0.7
2
0.73
0.7|0
0.74
4
0.78|
|.76
0.77
0.76
0.78
0.83

|0.76
0.77
0.7

|6
0.77
0.7

|8
0.77

|
|.74
0.73
0.73
0.71
0.76
.70
0.71
0.69
0.71
0.74|0.93
0.76
0.8
0.81
0.79
0.7|7
0.85
0.8
7
0.81
0.7|1
0.78
7
0.72|
|.72
0.73
0.72
0.73
0.76

|0.77
0.95
0.7

|9
0.80
0.8

|0
0.75

|
|.74
0.74
0.75
0.73
0.76

|0.87
0.79
0.9

|6
0.89
0.8

|2
0.78

|
|.74
0.73
0.73
0.72
0.75

|0.85
0.79
0.8

|9
0.91
0.8

|0
0.77

|
|.69
0.69
0.68
0.69
0.71

|0.75
0.76
0.7

|4
0.79
0.7

|4
0.70

|
|.74
0.73
0.75
0.73
0.77

|0.79
0.81
0.7

|9
0.81
0.8

|5
0.78

|
|.77
0.74
0.76
0.75
0.80

|0.77
0.73
0.7

|7
0.76
0.8

|2
0.78

|
|.74
0.72
0.74
0.74
0.79

|0.74
0.72
0.7

|6
0.75
0.8

|1
0.75

|
|.80
0.77
0.78
0.78
0.80

|0.76
0.73
0.7

|4
0.77
0.8

|2
0.80

|
|s
e
st
to
e
ic
d
n
n
n
.
.81
0.79
0.79
0.77
0.77
0.78
0.73
0.76
0.78
0.81
0.96|s
e
st
to
e
ic
d
n
n
n
.
.81
0.79
0.79
0.77
0.77
0.78
0.73
0.76
0.78
0.81
0.96|s
e
st
to
e
ic
d
n
n
n
.
.81
0.79
0.79
0.77
0.77
0.78
0.73
0.76
0.78
0.81
0.96|s
e
st
to
e
ic
d
n
n
n
.
.81
0.79
0.79
0.77
0.77
0.78
0.73
0.76
0.78
0.81
0.96|
|w
th
fir

tak
mat
an
uma
taria
actio|w
th
fir

tak
mat
an
uma
taria
actio|w
th
fir

tak
mat
an
uma
taria
actio|w
th
fir

tak
mat
an
uma
taria
actio|
|3 0.76 0.74 0.68 0.69|0.68 0.67 0.7|1 0.7|
|---|---|---|
|3
0.76
0.70
0.67
0.69

0
0.70
0.69
0.69
0.65
|0.66
0.66
0.
0.67
0.66
0.|8
0.7
7
0.7|
|1
0.80
0.79
0.72
0.77

|0.69
0.69
0.7

|5
0.7

|
|9
0.68
0.72
0.93
0.73

8
0.69
0.71
0.83
0.76
|0.80
0.78
0.
0.74
0.78
0.|5
0.7
3
0.7|
|7
0.71
0.71
0.74
0.97

|0.74
0.73
0.7

|4
0.7

|
|9
0.67
0.68
0.78
0.74

|0.97
0.88
0.

|4
0.7

|
|6
0.66
0.68
0.78
0.73

|0.88
0.92
0.

|2
0.7

|
|6
0.68
0.67
0.72
0.72

|0.76
0.83
0.

|0
0.6

|
|0
0.68
0.74
0.72
0.75

|0.70
0.72
0.

|1
0.7

|
|0
0.71
0.83
0.72
0.70

|0.70
0.69
0.

|1
0.7

|
|9
0.71
0.83
0.70
0.68

|0.69
0.68
0.

|1
0.7

|
|2
0.73
0.85
0.72
0.70

|0.69
0.71
0.

|1
0.7

|
|to
e
ic
d
n
n
n
.
2
0.74
0.74
0.75
0.73
0.72
0.73
0.76
0.9|to
e
ic
d
n
n
n
.
2
0.74
0.74
0.75
0.73
0.72
0.73
0.76
0.9|to
e
ic
d
n
n
n
.
2
0.74
0.74
0.75
0.73
0.72
0.73
0.76
0.9|
|tak
mat
an
uma
taria
actio|tak
mat
an
uma
taria
actio|tak
mat
an
uma
taria
actio|
|0.82 0.78|0.76|0.69 0.72|0.75 0.74|0.8|
|---|---|---|---|---|
|0.87
0.82|0.76
|0.71
0.75|0.75
0.76|0.7|
|0.82
0.78|0.74
|0.66
0.71|0.73
0.72|0.7|
|0.80
0.81|0.76
|0.69
0.73|0.75
0.75|0.7|
|0.73
0.77|0.73
|0.67
0.71|0.73
0.73|0.7|
|0.72
0.72|0.70
|0.67
0.68|0.71
0.70|0.6|
|0.79
0.82|0.76
|0.68
0.74|0.75
0.76|0.7|
|0.79
0.79|0.88
|0.75
0.79|0.81
0.76|0.8|
|0.76
0.76|0.82
|0.79
0.84|0.80
0.75|0.7|
|0.77
0.76|0.80
|0.76
0.82|0.81
0.77|0.7|
|0.74
0.75|0.78
|0.74
0.77|0.82
0.78|0.7|
|0.68
0.68|0.69
|0.68
0.70|0.73
0.73|0.7|
|0.75
0.74|0.74
|0.69
0.74|0.77
0.80|0.7|
|0.74
0.77|0.80
|0.71
0.79|0.80
0.78|0.7|
|0.76
0.78|0.76
|.69
0.74|0.74
0.73|0.9|
|5 0.75 0.72 0.71|0.76|0.74|0.74|
|---|---|---|---|
|5
0.69
0.71
0.74|0.69|0.74|0.73|
|5
0.71
0.69
0.71|0.70|0.73|0.72|
|1
0.69
0.65
0.67|0.70|0.74|0.72|
|5
0.71
0.65
0.70|0.70|0.76|0.72|
|3
0.70
0.68
0.70|0.71|0.75|0.71|
|6
0.71
0.67
0.70|0.70|0.77|0.74|
|3
0.90
0.76
0.76|0.81|0.74|0.75|
|9
0.74
0.82
0.87|0.72|0.71|0.71|
|0
0.73
0.71
0.77|0.73|0.72|0.71|
|5
0.75
0.71
0.74|0.81|0.85|0.73|
|2
0.70
0.68
0.71|0.74|0.83|0.70|
|5
0.71
0.67
0.72|0.75|0.87|0.73|
|3
0.81
0.75
0.80|0.87|0.80|0.74|
|4
0.72
0.69
0.72|0.71|0.72|0.96|
Figure 5: Cosine similarities between subword representations in a parallel sentence pair before and after finetuning. Red boxes indicate the gold alignments.


Word alignment is essential for the downstream cross-lingual language understanding
and generation tasks. Recently, the performance of the neural word alignment models (Garg et al., 2019; Ding et al., 2019;
Zenkel et al., 2020) has exceeded that of statistical models. However, they heavily rely
on sophisticated translation models. In this
study, we propose a super lightweight unsupervised word alignment model named _Mirro-_
_rAlign_, in which a bidirectional symmetric attention trained with a contrastive learning objective is introduced, and an agreement loss
is employed to bind the attention maps, such
that the alignments follow mirror-like symmetry hypothesis. Experimental results on several public benchmarks demonstrate that our
model achieves competitive, if not better, performance compared to the state of the art
in word alignment while significantly reducing the training and decoding time on average. Further ablation analysis and case studies show the superiority of our proposed MirrorAlign. Notably, we recognize our model as
a pioneer attempt to unify bilingual word embedding and word alignments. Encouragingly,
our approach achieves _16.4× speedup_ against
GIZA++, and _50× parameter compression_
compared with the Transformer-based alignment methods. We release our code to facilitate the community [1] .

MirrorAlign: A Super Lightweight Unsupervised Word Alignment Model via
  Cross-Lingual Contrastive Learning


Word alignment, aiming to find the word-level correspondence between a pair of parallel sentences,
is a core component of the statistical machine translation (Brown et al., 1993, SMT). It also has benefited several downstream tasks, _e.g._, computeraided translation (Dagan et al., 1993), semantic
role labeling (Kozhevnikov and Titov, 2013), crosslingual dataset creation (Yarowsky et al., 2001),
[1https://github.com/moore3930/MirrorAlign](https://github.com/moore3930/MirrorAlign)


The University of Sydney, Australia
liangding.liam@gmail.com


Independent Researcher, China
liamlmy@163.com
Figure 1: Two examples of word alignment. The upper
and bottom cases are the Chinese and Japanese references, respectively.
cross-lingual modeling (Ding et al., 2020a), and
cross-lingual text generation (Zan et al., 2022).
Recently, in the era of neural machine translation (Bahdanau et al., 2015; Vaswani et al., 2017,
NMT), the attention mechanism plays the role of
the alignment model in translation system. Unfortunately, Koehn and Knowles (2017) show that
attention mechanism may in fact dramatically diverge with word alignment. The works of Ghader
and Monz (2017); Li et al. (2019) also confirm this
finding.
Although there are some studies attempt to mitigate this problem, most of them are rely on a sophisticated translation architecture (Garg et al., 2019;
Zenkel et al., 2020). These methods are trained
with a translation objective, which computes the
probability of each target token conditioned on
source tokens and previous target tokens. This will
bring tremendous parameters and noisy alignments.
Most recent work avoids the noisy alignment of
translation models but employed too much expensive human-annotated alignments (Stengel-Eskin
et al., 2019). Given these disadvantages, simple
statistical alignment tools, _e.g.,_ FastAlign (Dyer
et al., 2013) and GIZA++ (Och and Ney, 2003) [2],
are still the most representative solutions due to
their efficiency and unsupervised fashion. We argue that the word alignment task, intuitively, is
2GIZA++ employs the IBM Model 4 as default setting.
much simpler than translation, and thus should be
performed before translation rather than inducing
alignment matrix with heavy neural machine translation models. For example, the IBM word alignment model, _e.g._, FastAlign, is the prerequisite of
SMT. _However, related research about lightweight_
_neural word alignment without NMT is currently_
_very scarce._
Inspired by cross-lingual word embeddings (Luong et al., 2015b, CLWEs), we propose to implement a super lightweight unsupervised word
alignment model in Section 3, named MirrorAlign,
which encourages the embedding distance between
aligned words to be closer. We also provide the
theoretical justification from mutual information
perspective for our proposed contrastive learning
objective in Section 3.4, demonstrating the reasonableness of our method. Figure 1 shows an
English sentence, and its corresponding Chinese
and Japanese sentences, and their word alignments.
The links indicate the correspondence between
English _⇔_ Chinese and English _⇔_ Japanese words.
If the Chinese word “ `举行` ” can be aligned to English word “held”, the reverse mapping should also
hold. Specifically, a bidirectional attention mechanism with contrastive estimation is proposed to
capture the alignment between parallel sentences.
In addition, we employ an agreement loss to constrain the attention maps such that the alignments
follow symmetry hypothesis (Liang et al., 2006).
Our contributions can be summarized as follows:
  - We propose a super lightweight unsupervised
alignment model (MirrorAlign), even merely
updating the embedding matrices, achieves
better alignment quality on several public
benchmark datasets compare to baseline models while preserving comparable training efficiency with FastAlign.
  - To boost the performance of our model, we
design a theoretically and empirically proved
bidirectional symmetric attention with contrastive learning objective for word alignment
task, in which we introduce extra objective to
follow the mirror-like symmetry hypothesis.
  - Further analysis show that the by-product of
our model in training phase has the ability
to learn bilingual word representations, which
endows the possibility to unify these two tasks
in the future.


Word alignment studies can be divided into two
classes:
**Statistical Models** Statistical alignment models
directly build on the lexical translation models of
(Brown et al., 1993), also known as IBM models.
The most popular implementation of this statistical alignment model is FastAlign (Dyer et al.,
2013) and GIZA++ (Och and Ney, 2000, 2003).
For optimal performance, the training pipeline
of GIZA++ relies on multiple iterations of IBM
Model 1, Model 3, Model 4 and the HMM alignment model (Vogel et al., 1996). Initialized with
parameters from previous models, each subsequent
model adds more assumptions about word alignments. Model 2 introduces non-uniform distortion,
and Model 3 introduces fertility. Model 4 and the
HMM alignment model introduce relative distortion, where the likelihood of the position of each
alignment link is conditioned on the position of
the previous alignment link. FastAlign (Dyer et al.,
2013), which is based on a reparametrization of
IBM Model 2, is almost the existing fastest word
aligner, while keeping the quality of alignment.
In contrast to GIZA++, our model achieves
nearly 15 _×_ speedup during training, while achieving the comparable performance. Encouragingly,
our model is at least 1.5 _×_ faster to train than FastAlign and consistently outperforms it.
**Neural Models** Most neural alignment approaches in the literature, such as Alkhouli et al.
2018, rely on alignments generated by statistical
systems that are used as supervision for training the
neural systems. These approaches tend to learn to
copy the alignment errors from the supervising statistical models. Zenkel et al. (2019) use attention
to extract alignments from a dedicated alignment
layer of a neural model without using any output
from a statistical aligner, but fail to match the quality of GIZA++. Garg et al. (2019) represents the
current state of the art in word alignment, outperforming GIZA++ by training a single model that
is able to both translate and align. This model is
supervised with a guided alignment loss, and existing word alignments must be provided to the
model during training. Garg et al. (2019) can produce alignments using an end-to-end neural training pipeline guided by attention activations, but
this approach underperforms GIZA++. The performance of GIZA++ is only surpassed by training
Figure 2: Illustration of MirrorAlign, where a pair of sentences are given as example. Each _xi_ and _yj_ are the
representation of words in source and target part respectively. Given _yj_, we can calculate context vector in source
part. The NCE training objective is encouraging the dot product of this context vector and _yj_ to be large. The
process in the other direction is consistent. By stacking all of the soft weights, two attention maps _As→t_ and _At→s_
can be produced, which will be bound by an agreement loss to encourage symmetry.
the guided alignment loss using GIZA++ output.
Stengel-Eskin et al. (2019) introduce a discriminative neural alignment model that uses a dot-productbased distance measure between learned source and
target representation to predict if a given sourcetarget pair should be aligned. Alignment decisions
are conditioned on the neighboring decisions using
convolution. The model is trained using gold alignments. Zenkel et al. (2020) uses guided alignment
training, but with large number of modules and parameters, they can surpass the alignment quality of
GIZA++.
They either use translation models for alignment
task, which introduces a extremely huge number of
parameters (compared to ours), making the training and deployment of the model cumbersome. Or
they train the model with the alignment supervision,
however, these alignment data is scarce in practice
especially for low resource languages. These settings make above approaches less versatile.
Instead, our approach is fully unsupervised at
word level, that is, it does not require gold alignments generated by human annotators during training. Moreover, our model achieves comparable
performance and is at least 50 times smaller than
theirs, i.e., #Parameters: 4M (ours) vs. 200M
(above).


Our model trains in an unsupervised fashion, where
the word level alignments are not provided. Therefore, we need to leverage sentence-level supervision of the parallel corpus. To achieve this, we in
troduce negative sampling strategy with contrastive
learning to fully exploit the corpus. Besides, inspired by the concept of cross-lingual word embedding, we design the model under the following
assumption: _If a target token can be aligned to a_
_source token, then the dot product of their embed-_
_ding vectors should be large._ Figure 2 shows the
schema of our approach **MirrorAlign** .


For a given source-target sentence pair ( **s** _,_ **t** ),
_si, tj ∈_ R _[d]_ represent the _i_ -th and _j_ -th word embeddings for the source and target sentences, respectively. Luong et al. (2015a); Ding et al. (2020b) illustrate that modelling the neighbour words within
the local window helps to understand the current
words. Inspired by this, we perform a extremely
simple but effective mean pooling operation with
the representations of its surrounding words to capture the contextualized information. Padding operation is used to ensure the sequence length. As
a result, the final representation of each word can
be calculated by element-wisely adding the mean
pooling embedding and its original embedding:
_xi_ =MEANPOOL([ _si_ ] _[win]_ ) + _si,_ (1)
where _win_ is the pooling window size. We can
therefore derive the sentence level representations
( _x_ 1 _, x_ 2 _, ..., x|s|_ ) _,_ ( _y_ 1 _, y_ 2 _, ..., y|t|_ ) for **s** and **t** . In addition to modeling words, modeling structured information (such as syntactic information) may be
helpful to enhance the sentence representation (Li
et al., 2017; Marcheggiani and Titov, 2017; Ding
and Tao, 2019), thus improving the word alignment.
We leave this exploration for future work.


Bidirectional symmetric attention is the basic component of our proposed model. The aim of this
module is to generate the source-to-target ( _aka._
s2t) and target-to-source ( _aka._ t2s) soft attention
maps. The details of the attention mechanism:
given a source side word representation _xi_ as query
_qi ∈_ R _[d]_ and pack all the target tokens together into
a matrix _Vt ∈_ R _[|][t][|×][d]_ . The attention context can be
calculated as:
ATTENTION ( _qi, Vt, Vt_ ) = ( _a_ _[i]_ _t_ _[·][ V][t]_ [)][⊺] _[,]_ (2)
where the vector _a_ _[i]_ _t_ _[∈]_ [R][1] _[×|][t][|]_ [ represents the atten-]
tion probabilities for _qi_ in source sentence over all
the target tokens, in which each element signifies
the relevance to the query, and can be derived from:
_a_ _[i]_ _t_ [=][ S][OFTMAX][ (] _[V][t]_ _[·][ q][i]_ [)][⊺] _[.]_ (3)
For simplicity, we denote the attention context of
_qi_ in the target side as _attt_ ( _qi_ ). s2t attention map
_As,t ∈_ R _[|][s][|×|][t][|]_ is constructed by stacking the probability vectors _a_ _[i]_ _t_ [corresponding to all the source]
tokens.
Reversely, we can obtain t2s attention map _At,s_
in a symmetric way. Then, these two attention
matrices _As,t_ and _At,s_ will be used to decode alignment links. Take s2t for example, given a target
token, the source token with the highest attention
weight is viewed as the aligned word.


Intuitively, the two attention matrices _As,t_ and _A_ _[T]_ _t,s_
should be very close. However, the attention mechanism suffers from symmetry error in different direction (Koehn and Knowles, 2017).
To bridge this discrepancy, we introduce agreement mechanism (Liang et al., 2006), acting like a
mirror that precisely reflects the matching degree
between _As,t_ and _At,s_, which is also empirically
confirmed in machine translation (Levinboim et al.,
2015). In particular, we use an agreement loss to
bind above two matrices:
symmetric constraint, demonstrating the effectiveness of this component.