**Attention** was introduced in “Neural Machine Translation by Jointly Learning to Align
and Translate” (Bahdanau, Cho, and Bengio 2015). The authors propose attention as a
method for aligning words between the input and output sentences. They term these
“soft-alignments” because each target word has a set of real-valued weights, one for
each source word. In contrast, word alignments have source and target words being
either aligned or not (0 or 1). Still, the intuition remains that the general concept of
aligning words between source and target sentences can also inform the decision of
neural MT systems. In fact, Bahdanau, Cho, and Bengio (2015) mention “attention” only
3 times, but mention “alignment” over 20 times.
Let us momentarily pause to describe the motivation for attention. The basic
encoder-decoder model has a glaring issue, in that the encoder representing sequences
of all lengths into a fixed-length vector. For especially long sequences, the encoder
would generate a very dense representation, and the decoder will likely have difficulty
decoding all pieces of information.
Attention solves this issue within the encoder-decoder paradigm. An attentional
model will keep all encoded vector representations of input tokens, and reference these
in the decoding step. At each decode time step _t_, then, we have access to each part of the
input (as well as the prior decoded tokens). Intuitively, the decoder has to pay attention
to different parts of the input when making each decision.
_Attention Explained._ We now provide the mathematical formulation of the original attentional model Bahdanau, Cho, and Bengio (2015), which extends an RNN-based encoderdecoder model. As in prior notation, let _f_ represent the source sentence and _e_ represent
the target sentence. Let _fj_ be the source word at index _j_, and and _ei_ be the target word
at index _i_ .
_Encoder._ To encode the source sentence, we use two RNNs, one in the forward (left-toright) direction, and one in the backward (right-to-level) direction:
_−→h_ ( _jf_ ) = _RNN_ ( _embed_ ( _fj_ ) _,_ _[−→]_ _h_ [(] _j_ _[f]_ _−_ [)] 1 [)] (8)
_←−h_ ( _jf_ ) = _RNN_ ( _embed_ ( _fj_ ) _,_ _[←−]_ _h_ [(] _j_ _[f]_ +1 [)] [)] (9)
where _[−→]_ _h_ [(] _j_ _[f]_ [)] and _[←−]_ _h_ [(] _j_ _[f]_ [)] are the RNN hidden states.
Each source word is then represented bidirectionally as the concatenation of forward and backward vectors
_h_ [(] _j_ _[f]_ [)] = [ _[←−]_ _h_ [(] _j_ _[f]_ [)] ; _[−→]_ _h_ [(] _j_ _[f]_ [)] ] _._ (10)
The source is now encoded as the set of vectors _h_ [(] 0 _[f]_ [)] _[, ..., h]_ _n_ [(] _[f]_ [)]
_Decoder._ Recall Equation 7:
_Pr_ ( _y_ ) =
_j_
- _Pr_ ( _yt|{y_ 1 _, ..., yt−_ 1 _}, h_ [(] _[x]_ [)] )
_t_ =1
In the original model, the same vector _h_ [(] _[x]_ [)] is used at all time steps. In contrast, the
attentional model uses a different context vector _ct_ at each time step _t_ . Maintaining the
prior notation, and noting that _y_ is equivalent to _f_,
_Pr_ ( _y_ ) =
_j_
- _Pr_ ( _yt|{y_ 1 _, ..., yt−_ 1 _}, ct_ ) _._ (11)
_t_ =1
_ct_ is defined as the sum of encoder hidden states weighted by alignment scores _αt_
_ct_ =
_m_
- _αt,j,_ (12)
_j_ =1
where _ct_ is of dimension _m_, and thus defines a weight distribution over the input words.
_αt_ is the attention vector.
Now we have seen where _αt_ is used. To explain how _αt_ is calculated, let us work
forwards from the decoder hidden state, specified by
_h_ [(] _t_ _[e]_ [)] = _Enc_ ([ _embed_ ( _et−_ 1); _ct−_ 1] _, h_ [(] _t−_ _[e]_ [)] 1 [)] _[.]_ (13)
We see that the current decoder state takes in the prior context vector _ct−_ 1 and the prior
decoder hidden state _h_ [(] _t−_ _[e]_ [)] 1
A Tutorial on Word Alignment in the Age of Deep Learning
Figure 6: An illustration of the attentional NMT model, when generating the target word
_yt_ given the source sentence _x_ 1 _, x_ 2 _, x_ 3 _, ..., xT_ . Reproduced from (Bahdanau, Cho, and
Bengio 2015).
We then calculate an attention score _at_, where each of its element _at,j_ is
_at,j_ = Attn( _h_ [(] _j_ _[f]_ [)] _, h_ [(] _t_ _[e]_ [)][)] _[.]_ (14)
Attn( _·_ ) is an arbitrary function that takes in two vectors and outputs a weight for how
much we should focus on a particular input encoding at the current decode time step.
Bahdanau, Cho, and Bengio (2015) use a simple RNN:
Attn( _h_ [(] _j_ _[f]_ [)] _, h_ [(] _t_ _[e]_ [)][) =] _[ v]_ _a_ _[⊤]_ [tanh (] _[W][a]_ [[] _[h]_ _j_ [(] _[f]_ [)] ; _h_ [(] _t_ _[e]_ [)][])] (15)
Where _Wa_ is the weight matrix of the first layer and _va_ _[⊤]_ [is the vector of the second layer.]
Finally, to use the attention vector as a probability distribution, we apply softmax
_αt_ = soft max( _at_ ) _,_ (16)
so that the weights sum to 1. This is then used in Equation 12 from above.
For each time step _t_, we now have a representation consisting of a context vector _ct_
and a decoder hidden state _h_ [(] _j_ _[e]_ [)][. We can use these, for instance, to calculate a softmax]
distribution over the next word(s):
_Pr_ ( _et_ ) = soft max( _Whs_ [ _vj_ [(] _[e]_ [)][;] _[ c][t]_ [] +] _[ b][s]_ [)] (17)
We can then use the sampling algorithms described in Section 4.1 to decode the target
language tokens. The attentional NMT model is shown in Figure 6.
**5.1.1 Attention Functions and Transformers.** It turns out that attention is a highly
effective and influential approach, beyond machine translation and to the NLP field
as a whole. We now summarize follow-up work in the field.
Luong, Pham, and Manning (2015) proposes several modifications to the original
attention mechanism. First, they use a unidirectional encoder, instead of bidirectional.
Second, to calculate attention, they consider the decoder state at _t_, instead of _t −_ 1,
allowing us to directly use _h_ [(] _t_ _[e]_ [)] without the extra RNN layer. Finally, they introduce
several different attention functions, the simplest (and most effective) of which is dotproduct attention:
Attn( _h_ [(] _j_ _[f]_ [)] _, h_ [(] _t_ _[e]_ [)][) =] _[ h]_ _j_ [(] _[f]_ [)]
_⊤_ _· h_ ( _te_ ) (18)
The main takeaway here is that regardless of differences in formulation, attention is
quite powerful as a general mechanism.
Attention solves one issue with RNN-based encoder-decoder models, but another
major issue remains. RNNs are slow, during both training and inference, because
generating a hidden state at time step _t_ requires us to have generated all prior time
steps 0 _, ..., t −_ 1. Vaswani et al. (2017) propose the _transformer_, which removes recurrence
entirely from the language model, in favor of only using attention [7] .
The key idea of the transformer is the idea of _self-attention_, which turns attention in
on itself. The attention described above relates words between two different sequences.
Self-attention operates in much the same way, except we now consider a single sequence
_s_, learning correlations between each word _si_ with the prior words _s_ 0 _, ..., si−_ 1. A transformer thus uses three instances of attention: source self-attention, target self-attention,
and source-target attention. Further contributions of the transformer can be found in
Appendix B.2.
Today, the transformer architecture is the dominant approach to NLP tasks. Notable
follow-up work includes BERT (Devlin et al. 2019) and GPT (Radford et al. 2018), among
many others.


A major advantage of attention, aside from its improvements to language models’
peformance, is in its interpretability. We now briefly summarize the body of work
investigating the correspondence of attention with “traditional” word alignment.
First, we turn to Bahdanau, Cho, and Bengio (2015), who provide us a visualization
of alignment weights — not unlike the word alignment visualization of Figure 1 —
as reproduced in Figure 7. Inspecting the attention diagram, we see that many of the
alignments are indeed quite reasonable. Bahdanau, Cho, and Bengio (2015) provide as a
qualitative example the aligment mapping of source phrase “the man” to target phrase
“l’ homme”. A word alignment would have instead two alignments “the” to “l”’ and
“man” to “homme”. They argue that the alignment suggested by attention is better here
because it captures the gender inflection inherent to French.
However in this same diagram, we observe a negative qualitative example. Using
attention weights, the target phrase “said” maps to the source words ““, a dit .” In fact,
the magnitude of the weights for “dit” and “.” are nearly equal, which is intuitively not
the case. This is an example of a spurious alignment, indicating that attention captures
more than just word alignments. While models like GIZA++ also make spurious alignment, we will see in later sections that their error rate is much lower than this vanilla
7 For Attn( _·_ ), Vaswani et al. (2017) use scaled dot-product attention, which adds a scaling factor _[√]_ ~~_n_~~ to
dot-product attention.
A Tutorial on Word Alignment in the Age of Deep Learning
Figure 7: A matrix visualization of alignments produced by an attentional NMT model,
reproduced from Bahdanau, Cho, and Bengio (2015). English is the source, and French
is the target. Lighter colored boxes indicate higher attention scores (black: 0, white: 1).
attentional model. More problematic is the fact that because attention alignments are
soft, it is impossible to collect ground-truth labels—we cannot ask humans to perform
the imprecise task of assigning real-valued weights between possible aligned words.
Ghader and Monz (2017) performs an analysis comparing attention and word
alignment, using the attentional model of Luong, Pham, and Manning (2015). Their
motivating hypothesis is: if attention corresponds to to word alignment, then better consistency between the two should lead to better translations (as Callison-Burch, Talbot,
and Osborne (2004) showed for SMT). First, they define an _attention loss_ between the
gold word alignments [8] and the attention weights. Next, they define a _word prediction_
_loss_ . They find that the correlation between these two losses depends on the part-ofspeech tag of the target word. Nouns have a high correlation, whereas verbs have a
lower correlation. Despite this, verbs are translated more accurately on average. This
shows that when the model translates verbs, it pays attention to more than just the
aligned target verb.
Ferrando and Costa-jussà (2021) performs a similar analysis, but on a transformer
model. They focus on the encoder-decoder attention mechanism and come to a similar conclusion: that attention only _sometimes_ correlates with alignments, but it _largely_
explains model predictions. They use a previously proposed method (Kobayashi et al.
2020) (which is described in Section 6) for inducing word alignments from a Transformer
8 The hard word alignments are converted to soft alignments beforehand. The conversion is simple:
suppose target word _yt_ is aligned to the set of source words _Ayt_ . Then each alignment has weight
1 _/|Ayt_ _|._
model. This method makes more alignment errors than GIZA++. They analyze these
alignment errors by considering the relative contributions of both parts of the decoder
input, at time step _t_ : the encoder embeddings (or _source sequence_ ), and the decoder
embeddings of the prior time steps (or _prefix sequence_ ). Contributions are calculated
through an input perturbation method of either the source or prefix sequence, while
keeping the other component unchanged. This allows them to calculate the proportion
of contribution of either sequence.
Intuitively, when generating a target word, if source contribution is high, then the
source sequence encodes useful, perhaps alignment-like, information. This happens, for
example, when the model is predicting a named entity, and just has to copy it from the
source. Otherwise, if the target contribution is high, then the source sequence is less
informative. This happens, for example, when the model is in the middle of generating
an idiom such as “ladies and gentlemen”. In this case, these target tokens are most
often incorrectly aligned to _finalizing tokens_, which are either end-of-sentence (</s>) or
closing punctuation (.). Ferrando and Costa-jussà (2021) conjecture that these wrong
alignments occur because the model learns to use these tokens as throwaways, using
them as signals to skip attention, and thus skip alignment.
To summarize these findings, in some cases, attention weights correspond well to
human intuition on what source and target phrases are aligned. In many other cases
they do not, which suggests attention captures other information. To conclude this
section, we have shown that attention does not replace the explicit word alignment
task.

Word Alignment in the Era of Deep Learning: A Tutorial


In this section, we survey the literature on neural word alignment, and classify them
into three broad approaches: **induction**, **unsupervised**, and **guided** . We first consider
models that operate on the attentional NMT models covered in the prior section. We
furthermore describe works that obtain alignments from _multilingual_, encoder-only
language models, and also classify them into the three broad approaches.
We focus on those papers that report alignment error rate (AER), the standard
metric for evaluating the quality of predicted word alignments respect to gold-standard
ones (lower is better). AER is given mathematically as
AER = 1 _−_ _[|][A][ ∩]_ _[S][|]_ [ +] _[ |][A][ ∩]_ _[P]_ _[|]_ _,_ (19)
_|A|_ + _|S|_
where _A_ is the set of hypothesized links, and _S_ and _P_ are from a manually annotated
set of links [9]
In the main text, we use AER as shorthand for AER on the RWTH German-English
(de-en) word alignment test set [10] (500 sentence pairs). We cover work that does not
report AER for de-en in Appendix D.1.
Table 1 summarizes AER results for all aligners covered. Here we report AER for
both de-en, and the Hansards English-French (fr-en) test set [11] (447 sentence pairs).
9 _S_ ure and _P_ ossible respectively. For ease of calculation, _P_ is often collapsed into _S_ .
[10 https://www-i6.informatik.rwth-aachen.de/goldAlignment/index.php](https://www-i6.informatik.rwth-aachen.de/goldAlignment/index.php)
[11 https://web.eecs.umich.edu/~mihalcea/wpt/](https://web.eecs.umich.edu/~mihalcea/wpt/)
A Tutorial on Word Alignment in the Age of Deep Learning
Unfortunately, the literature is not consistent with reporting statistical system baselines;
we therefore use the most recently reported scores for these.
Model Approach De-En Fr-En
Luong, Pham, and Manning (2015) induction 34.0   Li et al. (2019), 1 induction 42.8    Garg et al. (2019), 1 induction 32.6 17.0
Ding, Xu, and Koehn (2019) induction 22.3 8.5
Kobayashi et al. (2020) induction 25.0    Ferrando and Costa-jussà (2021) induction 22.1    Chen et al. (2020), 1 induction **17.9** **6.6**
Zenkel, Wuebker, and DeNero (2019) unsupervised 21.2 10.0
Zenkel, Wuebker, and DeNero (2020), 1 unsupervised 17.9 8.4
Chen, Sun, and Liu (2021) unsupervised **14.4** **4.4**
Li et al. (2019), 2 guided 39.3    Peter, Nix, and Ney (2017) guided 19.0    Garg et al. (2019), 2 guided 16.0 **4.6**
Zenkel, Wuebker, and DeNero (2020), 2 guided 16.3 5.0
Chen et al. (2020), 2 guided **15.8** 4.7
Jalili Sabet et al. (2020) multilingual (i) 18.8 7.6
Dou and Neubig (2021), 2 multilingual (u) 15.0 4.1
Nagata, Chousa, and Nishino (2020) multilingual (g) 11.4 4.0
Berkeley Aligner statistical 20.5    fast_align statistical 27.0 10.5
GIZA++ statistical 18.7 5.5
Table 1: Alignment Error Rates (AER) for several word alignment models, on the
German-English and French-English datasets. We report bidirectional (i.e., symmetrized
alignment) results where possible, and otherwise unidirectional from source to English.
The last 2 rows are statistical models, whereas the rest are neural. For papers with
multiple models of interest, we append 1 or 2 to the citation. The multilingual models
are further subclassified into (i)nduction, (u)nsupervised, and (g)uided.

6. Neural Approaches toward Word Alignment


In this section, we cover approaches that directly induce word alignments without
additional finetuning. We begin by describing an initial hurdle. Word alignments operates on sequences _e, f_ that are translations of each other. However, an attentional
NMT model will generate a predicted translation _f_ [ˆ] that most likely differs from _f_ . We
cannot directly compare word alignments between _e, f_ and _e,_ _f_ [ˆ] . The typical approach is
to “force decode” the attentional models: at each time step, the gold token _fi_ is selected,
and thus _f_ [ˆ] = _f_ .
Luong, Pham, and Manning (2015) use force decoding on their global attention
model (described in Section 5.1.1), and extract alignments by considering, for each target
word, the source word with the highest attention weight (we refer to this as the _argmax_
approach). They do the same for their local attention models, which looks at a subset of
source words at each time step. They find the best model achieves 34 AER.
The following works perform word alignment induction on transformer NMT models. Garg et al. (2019) create a similar, “naive” attention baseline by layer-wise averaging
of attention probabilities, finding the best AER of 32.6 on layer 5 of a 6-layer transformer.
Kobayashi et al. (2020) propose a simple refinement. Instead of using the attention
weight _α_ directly, they used the norm of the weighted projected vector _∥αf_ ( _x_ ) _∥_, where
_f_ ( _x_ ) is the transformed input vector. They modify the argmax approach by selecting
the source word _sj_ that gains the highest weight when _inputting ti_ (instead of when
_outputting ti_ ). This method with norm-based induction achieves 25.0 AER.
Ferrando and Costa-jussà (2021) extend this method by masking out final tokens,
and performing a weighted average across attention heads, by a calculated head importance score. Using weight-based induction they achieve 22.1 AER, vs. 18.7 for GIZA++.
Chen et al. (2020), similarly to Kobayashi et al. (2020), induce alignment on a
transformer when inputting _ti_, then average attention weights across heads. Their
implementation achieves 17.9 AER, finally beating GIZA++ in 2020.
An alternate line of work uses input perturbation for induction word alignments.
Li et al. (2019) measure the prediction difference of a target word if a source word is
removed, aligning those words with the highest difference. This method achieves 42.8
AER. Ding, Xu, and Koehn (2019) introduce a word saliency method inspired by visual
saliency from computer vision. Afterwards they apply smoothing through random
sampling. which This approach allows induction of word alignments from any NMT
model. Applied to a transformer it achieves 36.4 AER, whil to a convolutional NMT
model achieves 27.3 AER.

6.1 Word Alignments through Induction on Attention


This section discusses _unsupervised_ neural approaches to word alignment; i.e., those that
do not train on gold word alignments. Our definition of unsupervised means that we
exclude those methods that rely on any given word alignments – even if they come
from another system such as GIZA++. In this way, the task of these neural aligners is
the same as the statistical ones discussed in Section 3.
Zenkel, Wuebker, and DeNero (2019) extends the transformer architecture with a
separate _alignment layer_ on top of the decoder sub-network. This layer differs from the
decoder in that there is no self-attention nor skip connections. This essentially forces the
alignment layer to rely only on the context vector _ct_ . This model thus predicts target
words twice: once for the alignment layer, and once for the decoder layers. To initialize
the attention weights, they perform a forward pass of the transformer, then take the
weights from the alignment layer. This model achieves 26.6 AER. Zenkel, Wuebker, and
DeNero (2020) build on this by incorporating an auxiliary loss function to encourage
contiguous attention matrices, achieving 17.9 AER.
Chen, Sun, and Liu (2021) design a self-supervised word alignment model, which in
parallel masks out target tokens and recovers it conditioned on the source tokens, and
other target tokens. This model uses two variants of attention, which they term _static-KV_
_attention_, which enables masking over target words in parallel, and _leaky attention_, which
minimizes incorrect alignments to finalizing tokens. They also perform agreementbased training, and their best model achieves 14.4 AER.
A Tutorial on Word Alignment in the Age of Deep Learning


This section discusses _guided_ neural approaches to word alignment, i.e., those that
utilize training data with word alignments. We use “guided” instead of “supervised”
to convey that these models, instead of using gold-annotated alignments, usually are
guided silver word alignments generated by GIZA++.
Peter, Nix, and Ney (2017) extend an attentional NMT model with _target foresight_ .
In the original attention mechanism, the attention head only knows the prior predicted
target words. Target foresight introduces knowledge of the current target word into the
calculation of attention. They further train with supervised alignments from GIZA++,
achieving a 19.0 AER.
Garg et al. (2019) propose a single transformer-based model to jointly translate and
align. They choose one attention head to supervise with a _guided attention loss_ . The best
model further considers the entire target context, instead of only the current target word,
or prior target words. The self-supervised approach using induced alignments achieves
20.2 AER, whereas supervising with GIZA++ output achieves 16.0 AER.
Zenkel, Wuebker, and DeNero (2020) further refine their unsupervised model (see
Section 6.2), which had 17.9 AER, using the guided alignment loss of (Garg et al. 2019).
The silver alignments come from the first-pass model instead of GIZA++, and after
further training, the final model achieves 16.3 AER.
Chen et al. (2020) similarly use silver alignments from the predictions of their
induction-based model (see Section 6.1) to train an additional alignment module,
achieving 15.8 AER.


Transformers were first developed with the machine translation task in mind, and
started out as encoder-decoder, bi-lingual language models. However, follow-up work
has shown the effectiveness of encoder-only, _multilingual_ LMs for various NLP tasks.
Examples include mBERT (Devlin et al. 2019) and XLM-roBERTa (Conneau et al. 2020).
Because they are encoder-only, they do not have source-target attention (only selfattention), and are not trained on any parallel data (only on monolingual datasets in
many languages). Still, a line of work has found that high-quality word alignments can
be extracted from these models.
_Induction._ Jalili Sabet et al. (2020) propose to extract alignments from similarity matrices
induced from parallel sentence embeddings. Each cell of this similarity matrix measures
some similarity function between source word _xi_ and target word _yj_ . They first define
a method, _Argmax_, to align words _xi_ and _yj_ if and only if they are most similar to each
other. Then they propose _Itermax_, to iteratively apply _Argmax_, zeroing out similarities at
each iteration, until all words are aligned. This method achieves 18.8 AER for our de-en
dataset, and beats or is competitive with GIZA++ AER for other language pairs. This
suggests that surprisingly, multilingual encoder-only LMs, in the process of training on
their masked language modeling tasks, come to an internal notion of word alignments
as well.
_Unsupervised._ Dou and Neubig (2021) propose to combine the above multilingual LM
induction approach with parallel text fine-tuning on two word alignment objectives.
The first encourages closer contextualized representations at the aligned word-level,
whereas the second does so for representations of parallel sentences. They further
(a) The source sentence has NER annotations
(Mary, PERSON) and (Paris, CITY).
These are projected to the target sentence,
resulting in annotations ( `玛` `丽`, PERSON)
and ( `巴黎`, CITY).
(b) First, the part-of-speech annotations are
projected. Also projected are the dependency
relations represented by the arcs, which relate
two words — a head, and a dependent.
Figure 8: Examples of annotation projection for (a) named entity recognition (NER), (b)
dependency parsing.
introduce a softmax-based alignment between the for the similarity matrices. Their
multilingual model achieves 15.0 AER for our de-en task, and similarly high scores for
other tasks.
_Guided._ Nagata, Chousa, and Nishino (2020) frame word alignment as a _question-_
_answering_ (QA) task, where the context is the target sentence, the question is the source
span highlighted in context, and the answer is the aligned target span. They then then
train in a standard extractive QA paradigm, achieving 11.4 AER.

6.4 Word Alignments from Multilingual Language Models


Our literature review has shown that researchers in the deep learning era have slowly
but steadily been hammering away at the word alignment task. After nearly 2 decades
of GIZA++ at the top, researchers have finally surpassed it in the late 2010s. Still, in
contrast to other NLP tasks, which have grown rapidly in interest, word alignment has
remained fairly niche. Skeptical readers may ask: other than as another leaderboard
task, why should I care about the word alignment task today?
Broadly speaking, the main application of word alignments is in cross-lingual settings, or settings that go across languages. A prototypical cross-lingual task is, of course,
machine translation. The core property that makes word alignments so useful in crosslingual settings is that alignments can be learned in an unsupervised manner. This was
true in the statistical MT era, and remains true today.
In this final section, we broadly survey the literature for applications of word alignment. We identify three main use cases: annotation projection, improving translation,
and monolingual word alignments. As applications of the task are agnostic to the
modeling approach used (neural, statistical, etc.), we consider works across the decades.
A Tutorial on Word Alignment in the Age of Deep Learning


Despite advances in unsupervised and self-supervised learning, the performance of supervised learning approaches still dominates for most NLP tasks (as seen in Section 6).
Even a minimal amount of supervision goes a long way. How do we apply our stateof-the-art systems to low-resource languages which have little to no annotated data?
Annotation projection is a way to provide supervision in such settings. It has been a
major use-case of word alignment historically and to the present.
Annotation projection is the process of transferring annotations on text in a highresource language (most commonly English) to a low-resource language. It is assumed
that a parallel corpus exists, with word-aligned parallel sentences. In this way, we can
first train a model using available annotated data for some task in the source language.
We use this model to generate annotations on the source side of the parallel bitext.
Then we use the word alignments to project source labels to target labels. We now have
labeled examples in the target language, which we can use to train a new model on the
task. Example [12] annotation projections are shown in Figure 8a and 8b.
_Formalizing Word Alignment._ Annotation projection addresses the following setting (with
reference to Rasooli (2019)). Suppose we want a system which can solve a task for
some given sentences _{t_ [(] _[i]_ [)] _}_ _[m]_ _i_ =1 [in a low-resource language] _[ L][t]_ [. For each sentence we]
have source language sentences _{s_ [(] _[i]_ [)] _}_ _[m]_ _i_ =1 [, where] _[ s]_ [(] _[i]_ [)][ and] _[ t]_ [(] _[i]_ [)][ are translations for all] _[ i]_ [.]
Furthermore, we have word alignments between these sentences.
We first have supervised model _Msup_ learn a task from labeled source language
examples _D_ = _{_ ( _xj, yj_ ) _}_ _[n]_ _j_ =1 [. We apply the trained model to the source-side translated]
text _{si}_ _[m]_ _i_ =1 [to obtain predictions] _[ {][y][s]_ _i_ _[}][m]_ _i_ =1 [. Now, we project each source prediction to]
obtain the target labels _yti_ .
The annotation projection process gives us labeled target language examples _Dt_ =
_{_ ( _t_ [(] _[i]_ [)] _, yti_ ) _}_ _[m]_ _i_ =1 [, from which we can train a model] _[ M][proj]_ [. We can now use system] _[ M][proj]_
to solve the task in the target language.
The key insight is that reasonable word alignments can be learned in an unsupervised manner. Acquiring human-labeled data for a specific NLP task in a low-resource
language can be difficult and expensive. Acquiring translations from this low-resource
language to a high-resource one is less so, and after running a unsupervised word
alignment system over these translations, we can use annotation projection to create
models for any NLP task in the low-resource language. The projected annotations are
imperfect, of course, given both a) errors in the projection processes, such as incorrect
word alignments, and b) differences in how languages represent (or do not represent)
meaning and features. But the technique remains a reasonable, effective start for addressing various tasks and domains in low-resource languages or domains.
**7.1.1 Annotation Projection for NLP Tasks.** Hwa et al. (2005) apply annotation projection to parsing, the task of predicting a tree relating the words of a sentence. They first
directly project English trees to Spanish and Chinese, finding the projected trees are
quite lacking. They then perform post-projection transformation of these trees based
on linguistic knowledge of the target languages. Finally, they train target language
syntactic parsers on these transformed trees, finding the process allows for reasonable
bootstrapping of syntactic parsers for these languages. Follow-up work improves syn
[12 The top half of these diagrams are generated using https://corenlp.run/.](https://corenlp.run/)
tactic parsing performance by incorporating posterior regularization (Ganchev, Gillenwater, and Taskar 2009), by using parallel guidance and entropy regularization (Ma and
Xia 2014), and by iteratively learning over from more dense to less dense dependency
structures (Rasooli and Collins 2015, 2017).
The annotation projection process itself is generally kept constant; research instead
involves changing the model, the representation of the task, or the refinements to the
projection models. As those do not involve word alignments, and are thus beyond the
scope of the current tutorial, we omit further details. We refer interested readers to
several papers by Mohammad Sadegh Rasooli, Maryam Aminian, and collaborators:
annotation projection for semantic role labeling (Aminian, Rasooli, and Diab 2017,
2019), for sentiment analysis (Rasooli et al. 2018), and for broad-coverage semantic
dependency parsing (Aminian, Rasooli, and Diab 2020). We discuss non-NLP use cases
of annotation projection, and word alignment in general, in the following sections.
**7.1.2 Annotation Projection for Medical Terms.** Deléger, Merkel, and Zweigenbaum
(2009) uses annotation projection for the medical domain. They note that the increasing internationalization of medicine necessitates translation of medical terminology.
Because of the specific, technical nature of the domain, this is difficult even for human
translators. Therefore, they propose a human-in-the-loop system in which humans are
given candidate aligned medical terms, which they can then inspect, then correct and
filter out if needed.
We highlight the major steps of their proposed system here. First, they collect a
parallel corpus, using a Canadian health website which has documents in both English
and French. Then, they align sentence translations within these relatively parallel documents using an existing system. They perform automatic word alignment on these
sentence pairs, then refine the alignments using human annotators. Finally, they aggregate similar alignments together into a relational database, and have human annotators
select and refine them, arriving at a final set of translated medical terminologies.
They find their methodology creates English-French medical dictionaries with reasonably high-precision entries. From 50k sentence pairs, they obtain 15k filtered alignments. A manual review of 2127 terms finds a 79% precision (vs. 41% before filtering).
Nyström et al. (2006) use the same a similar method for the English-Swedish pair, but
use aligned terms instead of aligned sentences, resulting in more limited coverage.

7.1 Annotation Projection through Word Alignment


Let us now return to machine translation, the task which initially motivated word
alignments. For statistical MT, word alignment allows for decomposition of translating
an entire sentence to translating parts of a sentence. But what about for neural MT? We
have seen that the attention mechanism, at the core of modern day transformer models,
captures some notion of alignment. We have also seen that high-quality alignments can
be induced from both encoder-decoder models and encoder-only models. Still, followup work has shown that word alignment can additionally inform neural MT models.
A major issue with NMT systems is that they are prone to mistranslating lowfrequency words, especially without proper sampling algorithms. This is a side-effect
of their treating vocabulary words as embeddings, or real-valued vectors. This representation allows embedding notions of similarity between related words. However, a
resulting drawback is that NMT systems are more likely to mistranslate words that
make sense in a given context, but actually do not reflect the source sentence. It is often
the case that low-frequency words are often the main content words, so effort should be
A Tutorial on Word Alignment in the Age of Deep Learning
made to properly translate them. In contrast, SMT systems avoid this problem due to
their usage of explicit, discrete translation tables.
Arthur, Neubig, and Nakamura (2016) give an example for English-to-Japanese
translation, in which the system mistakenly translates the word for “Tunisia” into the
word for “Norway”. This is likely because “Norway” was seen more often in the training distribution. With this motivating example, Arthur, Neubig, and Nakamura (2016)
address this by incorporating an external, discrete lexicon—collected through word
alignments—into NMT models. They first transform the translation table probabilities
into a next-word prediction probability, then incorporate this probability into NMT. We
now summarize this incorporation process. Given an input sentence _F_, they construct a
matrix _LF_ of shape _|Ve| × |F_ _|_, where _Ve_ is the target vocabulary. Each cell then specifies
_Pr_ ( _ei_ = _n|fj_ ); these probabilities come from the t-tables learned by word alignment.
They then repurpose the attention vector _αt_ to weight each column of _Lf_, such that
_Pr_ lex( _ei|F, e_ _[i]_ 1 _[−]_ [1] ) = _LF αt,_ (20)
where _e_ _[i]_ 1 _[−]_ [1] = _e_ 1 _, ..., ei−_ 1.
_αt_ is computed in much the same way, differing in that it is over the target vocabulary, instead of the source tokens. They now need to incorporate this next-word
probability with that of the NMT model. They find the best overall method is by using
it as a bias within the NMT model’s probability distribution,
_Pr_ bias( _ei|F, e_ _[i]_ 1 _[−]_ [1] ) = soft max( _Wsηi_ + _bs_ + log _Pr_ lex( _ei|F, e_ _[i]_ 1 _[−]_ [1] ) + _ϵ_ )) _,_ (21)
where _Wsηi_ + _bs_ is from the original NMT model equation, and _ϵ_ is a hyper-parameter
that biases towards using the lexicon probabilities when smaller.
Arthur, Neubig, and Nakamura (2016) compare a baseline attentional NMT model
to one that incorporates the modified probability. On a English-to-Japanese translation
task, they find increases for both translation accuracy (23.2 vs. 20.9 BLEU [13] ), recall
of low-frequency words (19.3 vs. 17.7). They further find that their proposed model
achieves higher BLEU scores in earlier training steps, suggesting the model can bootstrap learning from the lexical information provided by word alignments. Mi, Wang,
and Ittycheriah (2016) take a similar, but simpler approach, leveraging word alignment
information to limit the vocabulary size predicted by each sentence.
Cohn et al. (2016) likewise seek to incorporate information from the word alignment
process into attentional NMT. Instead of using word alignments directly, they take the
structural biases used in prior statistical, unsupervised word alignment, and incorporate them into NMT models. First, they incorporate absolute position from IBM Model
2, which adds in a bias towards aligning source and target words at similar relative
positions within their sentences. Second, they incorporate the Markov condition from
HMM aligners (Vogel, Ney, and Tillmann 1996), which captures locality in the source
language. This means the assumption that if two target words are contiguous, it is likely
that their aligned words are contiguous. Third, they model fertility, which captures the
number of aligned source words to a target word. Fourth, inspired by Liang, Taskar, and
Klein (2006) they jointly train two models in parallel, one in each direction. On several
language pairs in low-resource settings, they find their model trained with these biases
13 BLEU (Papineni et al. 2002) is an evaluation metric for MT, based on modified n-gram precision.
achieves BLEU increases over a baseline attentional NMT model. For example ChineseEnglish BLEU is 44.1 vs. 41.2.
Raganato et al. (2021) proposes to supervise existing zero-shot [14] multilingual NMT
systems with word alignment information. They follow the joint training of translation
and alignment from (Garg et al. 2019), and produce word alignments using (Dou and
Neubig 2021). They compare their model to a transformer baseline, and find that while
average BLEU scores are the same across seen language pairs en-xx and xx-en (where xx
is a non-English language), the word alignment supervised models achieve a 1.9 BLEU
increase for unseen pairs (11.9 vs. 10.0).
We next consider the dictionary-guided MT task, where the input includes a dictionary of suggested translations. This is used in the case of technical domains, such as
medical or information technology, in which users would like to enforce translations
of given domain-specific terms. Chatterjee et al. (2017) extend a NMT decoder with
a guidance mechanism, which uses supervision from learned word alignments, to
constrain decoding. This increases en-de BLEU (25.5 vs 21.7). Alkhouli, Bretschner, and
Ney (2018) augment a transformer NMT model with an alignment head, which models
its source context as a binary-valued vector; 1 if there is an alignment, 0 otherwise. This
is concatenated to the attention head, and the model increases en-ro BLEU (31.0 vs.
29.7). Song et al. (2020) instead use a dedicated attention head to emulate the external
alignments, and calculate a separate attention loss, finding similar improvements.

7.2 Word Alignments for Improving Neural Machine Translation


A related field to machine translation is computer-assisted translation (CAT). Instead
of using systems to directly translate a source text, CAT instead develops systems to
assist human translators. CAT involves translation for translators (who are necessarily
bilingual), whereas MT involves translation for more general end-users (who are likely
monolingual). CAT is commonly used when translating technical documents, which
contain HTML tags and other markup. These tags are not natural language, and must
be handled separately by any MT and CAT systems. An example of markup is that
if a source term is bolded, the corresponding target term should remain bolded after
translation. Many authors have explored using annotation projection as part of the CAT
processing pipeline, to suggest possible tags for translators (Tezcan and Vandeghinste
2011; Joanis et al. 2013; Arcan et al. 2017; Müller 2017).
Other work has used word alignments to improve the translation memory (TM)
component of CAT systems. TMs are databases that store previously translated segments (sentences, paragraphs, etc.) that human translators can refer to; they are used to
assist in nearly all CAT tasks. Wu et al. (2005) propose to enhance TMs using word
alignment information at several level of matching. For the sentence-level and subsentential level matching metrics, they incorporate into each an alignment confidence
score between source and target sentence. For pattern-based machine translation, they
parse the source sentences into phrases, then use the word alignment phrase tables to
extract suggested translations. They find their system improves translation quality and
saves 20% translation time for humans.
Koehn and Senellart (2010) integrate concepts from both TM and SMT systems to
improve CAT systems. The proposed method first finds a fuzzy-matched TM sentence
14 Zero-shot means to perform the task without having seen relevant training data – here the language pairs
of interest.
A Tutorial on Word Alignment in the Age of Deep Learning
to the source sentence, then identifies the differing words between the two (they do
the latter by edit distance). They use the target sentence from the TM for the common
words, and use an SMT system to translate the differing words. Finally, they use word
alignments on the TM example to project from the source to the target span, and replace
that span with the SMT translation. Their results show that when the fuzzy match scores
are _≥_ 80%, the combined SMT+TM approach achieves higher BLEU performance than
either alone.
Moving out of CAT, He et al. (2021) propose a model which incorporates translation
memories into NMT. They extend a transformer model with a TM component of a
single sentence pair. They consider three ways of encoding the TM, the best of which
weights the sentence similarity scores by an external word alignment (obtained from
fast_align), so that the model pays more attention to aligned words when considering
the TM. Compared to NMT systems which have larger TMs, i.e., they retrieve multiple
sentences, their system is faster and more accurate, showing that using word alignment
information assists TMs used for NMT as well as CAT.

7.3 Word alignment for Computer-Assisted Translation (CAT)


Finally, we discuss other use cases for word alignment.
**7.4.1 Monolingual Word Alignment.** Thus far we have considered only the word
alignment task in cross-lingual settings. But just as translation is one instance of a
sequence-to-sequence task, so too cross-lingual word alignment is one instance of word
alignment. _Monolingual word alignment_ is the task of aligning words between two related
sentences in the same language.
Early work on monolingual word alignment addresses natural language inference
(NLI) (MacCartney, Galley, and Manning 2008), the task of determining if a natural
language hypothesis _H_ can be entailed by a given premise _P_ . Despite the difference
between the MT and NLI tasks [15], they train GIZA++ and other supervised (MT) word
aligners on NLI sentence pairs, achieving 74.1% F1 on a human word-aligned NLI test
set. Their proposed system, however, which more effectively leverages aspects of the
NLI problem by techniques such as using external semantic relatedness information,
and a phrase-based representation of alignment, achieves 85.5% F1.
Follow-up work improves this model by using an integer linear programming (ILP)
based exact decoding technique (Thadani and McKeown 2011). Other work extends
monolingual alignment to paraphrase alignment (Thadani, Martin, and White 2012),
question answering (Yao 2014), and semantic textual similarity (Li and Srikumar 2016).
More recently, Lan, Jiang, and Xu (2021) introduce a human-annotated benchmark
for monolingual word alignment. They further propose a neural semi-Markov CRF
alignment model, which unifies word-level and phrase-level alignments, captures semantic similarity between source and target spans, and incorporates Markov transition
probabilities. Their proposed model outperforms prior models on their dataset, as well
as on three prior datasets. They further show that monolingual word alignments have
useful downstream applications for text simplification tasks, building on Jiang et al.
(2020), and for several sentence pair classification tasks, following Lan and Xu (2018).
15 Other than the obvious monolinguality, NLI also differs in the word length asymmetry, given _P_ is
usually much longer than _H_
**7.4.2 Improving MT Evaluation.** We now discuss a specific use case of monolingual
word alignment: in improving metrics for machine translation evaluation.
We have seen the standard automated evaluation metric for word alignment is AER,
and briefly touched on a metric for machine translation, BLEU (Papineni et al. 2002).
BLEU is a precision-based metric that is easy to calculate and to understand. However,
the simplicity of BLEU leads to several issues. Perhaps most notably is that it entirely
fails to capture semantic similarity. For example, suppose the MT system generates “a
big dog”, and the gold reference is “one large canine”. BLEU gives a score of 0, even
though the phrases are essentially semantically equivalent.
METEOR (Lavie and Agarwal 2007) is an automated evaluation metric for evaluating machine translation that addresses these shortcomings of BLEU. It performs
scoring by aligning hypotheses to one or more reference translations. A monolingual
word aligner aligns words and phrases in successive stages: exact match, by stem, by
synonym, and by paraphrase. METEOR then scores the computed alignment using
unigram precision, recall, and a measure of explicit ordering. METEOR correlates well
with human judgment at the sentence-level, and is another commonly-used metric for
evaluating machine translation.


Let us conclude by recapping what we have covered in this tutorial. We began by
formalizing the tasks of word alignment and statistical machine translation, finding
that they are intrinsically linked. We the turned to neural machine translation, taking
a detour to describe a basic encoder-decoder NMT model. We got back on course by
formalizing the attention mechanism, which introduces a concept of alignment between
tokens in the source and target sentences. We found that attention captures some
notion of alignment, but also other linguistic information. We then moved to a survey
approach. We performed a comprehensive literature review of neural word aligners,
finding that the task remains a niche, yet underexplored task in the deep learning era.
Finally, we surveyed the applications of word alignment, from annotation projection to
improving translation.
We hope that our tutorial has instilled in readers not only a past and present
understanding of the word alignment task, but also an interest to future word alignment
research ahead.


We now provide further details on the Moses machine translation system [16] . Moses
learns both _phrase-based_ and _linguistic information_ about translations. The former involves leveraging co-occurences of phrases, or continuous sequences of words, whereas
the latter is to be added externally by knowledgable users. SMT methods which use
both of these two are known as **factored translation** .
The Moses software package consists of a **training pipeline** and a **decoder** . The
training pipeline is a collection of scripts that takes as input raw data, and outputs a
trained MT model. The decoder takes as input a source sentence, and gives as output a
ranked list of candidate target translations.
**A.1.1 Moses Training Pipeline.** The training pipeline can further be deconstructed into
preprocessing text, obtaining phrase translation tables, and training language models.
_Preprocessing text._ In the first step, we convert the parallel corpus into a format suitable
for machine use. We read over the bitext to obtain vocabulary files for both source
and target languages, which map words to indices. Indices descend from index 1, the
most common word, to T the _T_ -th most common word, where _T_ is the vocabulary size.
Words less common than _T_ are assigned the token UNK and the index 0, as described
in Section 2.3. We then use these vocabulary files to create a new bitext, replacing
surface-form tokens with their indices. This bitext is then easily and efficiently machinereadable.
16 The full documentation, on which we base this section on, can be found at
[https://www.statmt.org/moses/](https://www.statmt.org/moses/)
As an additional preprocessing step, we calculate _word classes_ for each language—
given a word, place it into a set of words with similar usage.
_Word alignment and translation tables._ Word alignment provides the phrase-based translation tables, which are key to the Moses phrase-based translation approach. The aligner
used is GIZA++ (Och and Ney 2003), which we further describe in Section 3.1.
First, GIZA++ is trained, in an unsupervised fashion, on a preprocessed parallel
corpus. This model then returns word alignments in each direction _F →_ _E_ and _E →_ _F_ .
Note that for unsupervised word alignment there are no train/validation/test splits.
Second, we need to combine the _F →_ _E_ and _E →_ _F_ word alignments. This is done
heuristically, the default of which is grow-diag-final. At a high-level, this heuristic
begins with the alignment points from the intersection, then adds alignment points from
the union if and only if both source and target words for that point were unaligned.
Third, we obtain lexical translation tables by collecting all aligned pairs from the
word alignments. The translation tables enumerate words and for each gives its possible
translations and associated probability.
Fourth, we extract all phrases into one file, where each line contains the source
phrase, translated phrase, and the alignment points.
Fifth, we now build a phrase translation table in the forward direction ( _F_ to _E_ )
using the outputs of the prior two steps. Each line of this table estimates the transition
probability _φ_ ( _e|f_ ) of phrases. Each line of this table contains a phrase, its candidate
translations, and several translation scores. We also build a phrase translation table from
_E_ to _F_ similarly, except inverting the phrase translation direction.
_Language model._ The language model (LM) is a stage of the training pipeline, which
models the likelihood of sentences of a given language, in this case the target language.
This allows us to score the different translations of a source sentence.
Moses supports several LMs, using KenLM (Heafield 2011). by default. In brief,
these LMs are all n-gram models, which learn from a large monolingual corpus the
probability of sequences of words. Typically, 4-grams are used, which means that sentences are decomposed into 4-grams and scored. For more on n-gram language models,
see Heafield (2011).
Related to the language model is the _reordering model_ . Reordering of phrases is
necessary as word order, of course, differs across languages. Details for how this re[ordering model is learned can be found at http://www2.statmt.org/moses/?n=](http://www2.statmt.org/moses/?n=FactoredTraining.BuildReorderingModel)
[FactoredTraining.BuildReorderingModel.](http://www2.statmt.org/moses/?n=FactoredTraining.BuildReorderingModel)
**A.1.2 Moses Decoder.** The Moses decoder is the key step in performing translations.
It requires the translation tables and language model from the training pipeline step.
We describe its mechanism in brief. The decoder segments source sentences into source
phrases, then translates these phrases into target phrases. The main source of knowledge is the phrase translation tables. There are several possible segmentations, which
result in several sets of translated phrases. For each, Moses uses the reordering model to
obtain candidate sentence translations. These translations are scored for target language
fluency by the language models. As a final heuristic, we impose a word length penalty,
to ensure translations are not too much longer or shorter than the source sentences. The
intuition is that if too long, a translation likely contains extra information or repetitions,
and if too short, a translation likely is missing information.
A Tutorial on Word Alignment in the Age of Deep Learning

A.1 Moses: A Statistical Machine Translation System


The fertility-based word aligners of IBM Models 3 to 6 are very slow to train due to their
complexity. This is even the case with the computing resources available in the 2020s,
given that GIZA++ implementations run only on CPU thread(s). Researchers therefore
often turn to these other two word aligners, which are simpler, yet still fairly accurate.
_fast_align._ fast_align (Dyer, Chahuneau, and Smith 2013) is a software package which
implements a reparameterized version of IBM Model 2. Recall Model 1, which has a
uniform alignment prior (see Equation 6). Model 2 additionally incorporates an alignment probability _Pr_ ( _aj|j, m, n_ ), with the equation
_Pr_ ( _F, a|E_ ) = _Pr_ ( _m|n_ ) _·_
_m_
- _Pr_ ( _aj|j, n, m_ ) _· Pr_ ( _Fj|Eaj_ ) _,_ (A.1)
_j_ =1
fast_align thus trains ten times faster than GIZA++. In practice, fast_align and GIZA++
have similar performance for language pairs with less reordering (i.e. French and
English, both are SOV), and worse performance for those with more reordering (e.g.
German is SOV, English is SVO).
_Berkeley Aligner._ Berkeley Aligner is the software package implementing the HMMbased model of Liang, Taskar, and Klein (2006). This work jointly trains two simple
HMM models (HMM aligners were described in Section 3.1), one in either direction, to
agree on their alignments. They report improvements in AER over IBM Model 1 and 2
baselines. In practice, Berkeley Aligner and GIZA++ have similar performance.

A.2 Other Statistical Word Alignment Methods


A **neural network** is a machine learning algorithm based on a network of nodes (or
neurons) arranged in layers. Each node receives as input _signals_ from prior nodes, and
compute an output based on a weighted sum of its inputs. The associated _weight_ for
each node modifies the strength of each signal. A non-linear function is then applied to
the output, which then becomes the input signal for the next node(s).
Figure 1 shows a simple neural network, which consists of 3 layers. Each node can
be thought of as performing its own linear regression. Another representation of neural
networks is as a series of matrix multiplications, with added non-linearity between
steps.
Because of the non-linearity used by neural networks, they are extremely powerful.
A neural network with just 3 layers – input, 1 hidden, output – can theoretically model
any possible function, if the hidden layer is large enough. In practice, many hidden
layers are used to learn better representations of problems, especially for the complex
domains of images and language.
Furthermore, neural networks are effectively trained, and thus useful, because they
model their complicated functions as a chain of simpler functions. All these simpler
functions are differentiable. The entire learning algorithm for a neural network is called
_automatic differentiation_, which consists of two steps:
[Figure 1: A 3-layer neural network, reproduced from http://en.wikipedia.org/](http://en.wikipedia.org/wiki/Neural_network)
[wiki/Neural_network.](http://en.wikipedia.org/wiki/Neural_network)
 - _Forward calculation_, passing the input through the neural network, which
computes the model output
 - _Backpropagation_, traversing the neural network backwards, computing the
gradient of the loss function (which compares the actual output vs the
expected output) with respect to the weights of the network
Automatic differentiation, as the name implies, is an efficient way to calculate the
gradient. After one step of automatic differentiation, we update the weights of the
network with _gradient descent_ methods, taking a step towards lower (training) loss, and
repeating until convergence. The final neural network, and its associated set of weights,
serve as an approximate solution the given task.


Due to the representational power of neural networks, they can almost perfectly fit
any set of training examples. As we want models to be able to generalize to unseen
examples, researchers use several regularization techniques to avoid overfitting to the
training data. One such technique is _early stopping_, which involves evaluating the loss
of a in-progress model on a held-out validation set, and stopping training when the
loss fails to improve after some threshold of steps. Another technique is _dropout_, which
randomly zeros out nodes of a neural network during training.
The basic neural network we described above is called a _feedforward neural network_
(or multilayer perceptron). Of course, researchers these days use more advanced neural
networks. These are especially important to handle temporal sequences of variable
length – such as sequences of words! Two classes of neural networks to handle sequences are _recurrent neural networks_ (RNNs) and transformers. RNNs allows hidden
states to reference the outputs of previous hidden states. Transformers use the (self
A Tutorial on Word Alignment in the Age of Deep Learning
Figure 2: Two views of an RNN. Left: the compressed RNN, Right: the unrolled
[RNN. Reproduced from http://en.wikipedia.org/wiki/Recurrent_neural_](http://en.wikipedia.org/wiki/Recurrent_neural_network)
[network.](http://en.wikipedia.org/wiki/Recurrent_neural_network)
) **attention** mechanism, which allow the model to weight previous parts of the input at
each step. We further describe attention, and how it correlates to word alignment, in
Section 5.
Modern-day neural networks can be very deep (hence the moniker deep learning).
This is done by stacking smaller neural networks, and connecting their inputs and
outputs. Therefore it is intuitive in diagrams to represent these smaller neural networks
as blocks. For example, consider Diagram 2, a simple recurrent neural network.


Here we summarize the contributions of the transformer (Vaswani et al. 2017). First,
the use of self-attention as described in Section 5.1.1. Second, it uses _multi-head attention_ ;
instead of computing attention once, we compute attention multiple times in parallel.
Vaswani et al. (2017) find that this approach “allows the model to jointly attend to
information from different representation subspaces at different positions.” Third, they
introduce positional encodings, which are summed with word embeddings to preserve
positional information in the sentence. Fourth, they stack these self-attention modules,
i.e. the encoder consists of stacked self-attention networks, as does the decoder.
Because the transformer removes recurrence, training is highly parallelizable. This
allows transformer-based language models to be efficiently trained on on huge amounts
of data. To see why this is, let us first consider an RNN. Because the input to each step
requires the output of the previous step, we must wait for the previous step to finish.
This step requires its previous step, and so on. Each word, therefore, depends on the
output of the RNN function, which is highly nonlinear.
In contrast, a transformer uses self-attention. At each decode step, then, we have
direct access to the the information to the prior steps—recall the attention mechanism
keeps a set of prior hidden states. To generate output at a given time step _t_, we still need
to consider the prior _t −_ 1 steps. But each can be looked up in constant time, meaning
we now have only a linear dependence [17] .
17 See Vaswani et al. (2017) for further explanation on how self-attention can be parallelized through matrix
multiplications.
In sum, RNNs require sequential training, given the nonlinear recurrence function.
Transformers do not have to be trained sequentially, since attention we can linearly look
up the context for any input position. This makes training parallelizable, allowing for
efficient training of language models on huge amounts of data.


In this section, we cover a few neural word aligners that were not included in the main
text, as they did not report AER for the German-English word alignment dataset.
_Unsupervised._ Cheng et al. (2016) propose an approach to jointly train an attentional
NMT model to agree on source-to-target and target-to source attention matrices, inspired by Liang, Taskar, and Klein (2006) (we refer to this as agreement-based training).
They report results for a Chinese-English word alignment test set (Liu, Liu, and Lin
2005) (we refer to the Chinese to English direction here on as zh-en) of 500 manually
aligned sentence pairs, finding the jointly trained model achieves 47.5 AER vs. 52.5 AER
for an independently trained model.
Zhang et al. (2017) explicitly incorporate word reordering knowledge into attentional NMT. Inspired by statistical alignment approaches (Och and Ney 2003), they add
a word reordering penalty to the calculation of attention. The best model achieves 46.9
AER vs 49.7 on zh-en.
_Guided._ Liu et al. (2016) propose to supervise attentional NMT models with alignments
from statistical aligners. They convert these (predicted) hard alignments to soft alignments (See Footnote 8). At training time, they jointly supervise across translation and
attention, whereas inference time proceeds as before. Their model achieves 43.4 AER (vs
30.6 GIZA, 50.6 baseline) on zh-en.
Stengel-Eskin et al. (2019) introduce a discriminative neural alignment model,
which measures the dot-product distance between learned source and target vectors.
From this distance, the model learns if source-target pairs should be aligned. They
further use convolution to consider neighboring decisions. This approach works for any
NMT model, but they find best results with a transformer. Trained on a small amount of
labeled data, the model achieves 73.4 F1 for a Chinese-English test set [18] of 636 aligned
sentences, vs 62.0 F1 for fast_align.
[18 http://catalog.ldc.upenn.edu/LDC2015T06](http://catalog.ldc.upenn.edu/LDC2015T06)


Proceedings of Machine Learning Research vol 236:1–23, 2024 3rd Conference on Causal Learning and Reasoning

Finding Alignments Between Interpretable Causal Variables and
  Distributed Neural Representations

Atticus Geiger
Zhengxuan Wu
Christopher Potts
Thomas Icard
Noah D. Goodman


**Atticus Geiger** [˚] _[♢]_ **, Zhengxuan Wu** [*] **, Christopher Potts, Thomas Icard, and Noah D. Goodman**

Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations


**Editors:** Francesco Locatello and Vanessa Didelez

{atticusg, wuzhengx, cgpotts, icard, ngoodman}@stanford.edu


Causal abstraction is a promising theoretical framework for explainable artificial intelligence that
defines when an interpretable high-level causal model is a faithful simplification of a low-level deep
learning system. However, existing causal abstraction methods have two major limitations: they
require a brute-force search over alignments between the high-level model and the low-level one, and
they presuppose that variables in the high-level model will align with disjoint sets of neurons in the
low-level one. In this paper, we present _distributed alignment search_ (DAS), which overcomes these
limitations. In DAS, we find the alignment between high-level and low-level models using gradient
descent rather than conducting a brute-force search, and we allow individual neurons to play multiple
distinct roles by analyzing representations in non-standard bases— _distributed_ representations. Our
experiments show that DAS can discover internal structure that prior approaches miss. Overall, DAS
removes previous obstacles to uncovering conceptual structure in trained neural nets.


Can an interpretable symbolic algorithm be used to faithfully explain a complex neural network
model? This is a key question for interpretability; a positive answer can provide guarantees about
how the model will behave, and a negative answer could lead to fundamental concerns about whether
the model will be safe and trustworthy.
Causal abstraction provides a mathematical framework for precisely characterizing what it means
for any complex causal system (e.g., a deep learning model) to implement a simpler causal system
(e.g., a symbolic algorithm) (Rubenstein et al., 2017; Beckers et al., 2019; Massidda et al., 2023).
For modern AI models, the fundamental operation for assessing whether this relationship holds in
practice has been the _interchange intervention_ (also known as activation patching), in which a neural
network is provided a ‘base’ input, and sets of neurons are forced to take on the values they would
have if different ‘source’ inputs were processed (Geiger et al., 2020; Vig et al., 2020; Finlayson et al.,
2021; Meng et al., 2022). The counterfactuals that these interventions create are the basis for causal
inferences about model behavior.
Geiger et al. (2021) show that the relevant causal abstraction relation obtains when interchange
interventions on aligned high-level variables and low-level variables have equivalent effects. This
ideal relationship rarely obtains in practice, but the proportion of interchange interventions with the
same effect ( _interchange intervention accuracy_ ; IIA) provides a graded notion, and Geiger et al.
(2023) formally ground this metric in the theory of approximate causal abstraction. Geiger et al. also
use causal abstraction theory as a unified framework for a wide range of recent intervention-based
analysis methods (Vig et al., 2020; Csordás et al., 2021; Feder et al., 2021; Ravfogel et al., 2020;
  - Equal contribution.
© 2024 .
Elazar et al., 2020; De Cao et al., 2021; Abraham et al., 2022; Olah et al., 2020; Olsson et al., 2022;
Chan et al., 2022).
Causal abstraction techniques have been applied to diverse problems (Geiger et al., 2019, 2020;
Li et al., 2021; Huang et al., 2022). However, previous applications have faced two central challenges.
First, causal abstraction requires a computationally intensive brute-force search process to find
optimal alignments between the variables in the high-level model and the states of the low-level one.
Where exhaustive search is intractable, we risk missing the best alignment entirely. Second, these
prior methods are _localist_ : they artificially limit the space of possible alignments by presupposing
that high-level causal variables will be aligned with disjoint groups of neurons. There is no reason to
assume this a priori, and indeed much recent work in model explanation (see especially Ravfogel
et al. 2020, 2022; Elazar et al. 2020; Olah et al. 2020; Olsson et al. 2022) is converging on the insight
of Smolensky (1986), Rumelhart et al. (1986), and McClelland et al. (1986) that individual neurons
can play multiple conceptual roles. Smolensky (1986) identified _distributed neural representations_
as “patterns” consisting of linear combinations of unit vectors.
In the current paper, we propose distributed alignment search (DAS), which overcomes the above
limitations of prior causal abstraction work. In DAS, we find the best alignment via _gradient descent_
rather than conducting a brute-force search. In addition, we use _distributed interchange interventions_,
which are “soft” interventions in which the causal mechanisms of a group of neurons are edited
such that (1) their values are rotated with a change-of-basis matrix, (2) the targeted dimensions
of the rotated neural representation are fixed to be the corresponding values in the rotated neural
representation created for the source inputs, and (3) the representation is rotated back to the standard
neuron-aligned basis. The key insight is that viewing a neural representation through an alternative
basis that is not aligned with individual neurons can reveal interpretable dimensions (Smolensky,
1986).
In our experiments, we evaluate the capabilities of DAS to provide faithful and interpretable
explanations with two tasks that have obvious interpretable high-level algorithmic solutions with two
intermediate variables. In both tasks, the distributed alignment learned by DAS is as good or better
than both the closest localist alignment and the best localist alignment in a brute-force search.
In our first set of experiments, we focus on a hierarchical equality task that has been used
extensively in developmental and cognitive psychology as a test of relational reasoning (Premack,
1983; Thompson et al., 1997; Geiger et al., 2022a): the inputs are sequences r _w,_ _x,_ _y,_ _z_ s, and the
label is given by p _w_ “ _x_ q “ p _y_ “ _z_ q. We train a simple feed-forward neural network on this task and
show that it perfectly solves the task. Our key question: does this model implement a program that
computes _w_ “ _x_ and _y_ “ _z_ as intermediate values, as we might hypothesize humans do? Using DAS,
we find a distributed alignment with 100% IIA. In other words, the network is perfectly abstracted by
the high-level model; the distinction between the learned neural model and the symbolic algorithm is
thus one of implementation.
Our second task models a natural language inference dataset (Geiger et al., 2020) where the
inputs are premise and hypothesis sentences p _p,_ _h_ q that are identical but for the words _wp_ and _wh_ ;
the label is either _entails_ ( _p_ makes _h_ true) or _contradicts_ / _neutral_ ( _p_ makes _h_ false). We fine-tune a
pretrained language model to perfectly solve the task. With DAS, we find a perfect alignment (100%
IIA) to a causal model with a binary variable for the entailment relation between the words _wp_ and
_wh_ (e.g., _dog_ entails _mammal_ ).
In both our sets of experiments, the DAS analyses reveal perfect abstraction relations. However,
we also identify an important difference between them. In the NLI case, the entailment relation can
FINDING DISTRIBUTED ALIGNMENTS
be decomposed into representations of _wp_ and _wh_ . What appears to be a representation of lexical
entailment is, in this case, a “data structure” containing two representations of word identity, rather
than an encoding of their entailment relation. By contrast, the hierarchical equality models learn
representations of _w_ “ _x_ and _y_ “ _z_ that cannot be decomposed into representations of _w_, _x_, _y_ and _z_ .
In other words, these relations are entirely abstracted from the entities participating in the relation;
DAS reveals that the neural network truly implements a symbolic, tree-structured algorithm.


A theory of _causal abstraction_ specifies exactly when a ‘high-level causal model’ can be seen as
an abstract characterization of some ‘low-level causal model’ (Iwasaki and Simon, 1994; Chalupka
et al., 2017; Rubenstein et al., 2017; Beckers et al., 2019). The basic idea is that high-level variables
are associated with (potentially overlapping) sets of low-level variables that summarize their causal
mechanisms with respect to a set of hard or soft interventions (Massidda et al., 2023). In practice, a
graded notion of _approximate_ causal abstraction is often more useful (Beckers et al., 2019; Rischel
and Weichwald, 2021; Geiger et al., 2023).
Geiger et al. (2023) argue that causal abstraction is a generic theoretical framework for providing
_faithful_ (Jacovi and Goldberg, 2020; Lyu et al., 2022) and _interpretable_ (Lipton, 2018) explanations of
AI models and show that LIME (Ribeiro et al., 2016), causal effect estimation (Abraham et al., 2022;
Feder et al., 2021), causal mediation analysis (Vig et al., 2020; Csordás et al., 2021; De Cao et al.,
2021), iterated nullspace projection (Ravfogel et al., 2020; Elazar et al., 2020), and circuit-based
explanations (Olah et al., 2020; Olsson et al., 2022; Wang et al., 2022; Chan et al., 2022) can all be
understood as causal abstraction analysis.
Interchange intervention training (IIT) objectives are minimized when a high-level causal model
is an abstraction of a neural network under a given alignment (Geiger et al., 2022b; Wu et al., 2022;
Huang et al., 2022). In this paper, we use IIT objectives to learn an alignment between a high-level
causal model and a deep learning model.


We focus on acyclic causal models (Pearl, 2001; Spirtes et al., 2000) and seek to provide an intuitive
overview of our method. An **acyclic causal model** consists of input, intermediate, and output
**variables**, where each variable has an associated set of **values** it can take on and a **causal mechanism**
that determine the value of the variable based on the value of its causal parents. For a simple running
example, we modify the boolean conjunction models of Geiger et al. (2022b) to reveal key properties
of DAS. A causal model _B_ for this problem can be defined as below, where the inputs and outputs
are booleans T and F. Alongside _B_, we also define a causal model _N_ of a linear feed-forward neural
network that solves the task. Here we show _B_, _N_, and the parameters of _N_ :
_WW_ 21 “ “ ““ sincospp2020 [˝][˝] qq ´cossinpp2020 [˝][˝] qq ‰‰ **w** “ _b_ “ ´“ 11 _._ 81 ‰
The model _N_ predicts T if _O_ ą 0 and F otherwise. This network solves the boolean conjunction
problem perfectly in that all pairs of input boolean values are mapped to the intended output.
An input **x** of a model _M_ determines a unique total setting _M_ p **x** q of all the variables in the
model. The inputs are fixed to be **x** and the causal mechanisms of the model determine the values of
the remaining variables. We denote the values that _M_ p **x** q assigns to the variable or variables **Z** as
GETVALUES **Z** p _M_ p **x** qq. For example, GETVALUES _V_ 3p _B_ prT _,_ Fsqq “ F.


Interventions are a fundamental building block of causal models, and of causal abstraction analysis
in particular. An intervention **I** Ð **i** is a setting **i** of variables **I** . Together, an intervention and an input
setting **x** of a model _M_ determine a unique total setting that we denote as _M_ **I** Ð **i** p **x** q. The inputs are
fixed to be **x**, and the causal mechanisms of the model determine the values of the non-intervened
variables, with the intervened variables **I** being fixed to **i** .
We can define interventions on both our causal model _B_ and our neural model _N_ . For example,
_BV_ 1ÐTprF _,_ Tsq is our boolean model when it processes input rF _,_ Ts but with variable _V_ 1 set to T. This
has the effect of changing the output value to T. Similarly, whereas _N_ pr0 _,_ 1sq leads to an intermediate
values _h_ 1 “ ´0 _._ 34 and _h_ 2 “ 0 _._ 94 and output value ´1 _._ 2, if we compute _Nh_ 1Ð1 _._ 34pr0 _,_ 1sq, then the
output value is 0 _._ 48. This has the effect of changing the predicted value to T, because 0 _._ 48 ą 0.


In causal abstraction analysis, we ask whether a specific low-level model like _N_ implements a
high-level algorithm like _B_ . This is always relative to a specific _alignment_ of variables between
the two models. An alignment Π “ ptΠ _X_ u _X_ _,_ t _τX_ u _X_ q assigns to each high-level variable _X_ a set of
low-level variables Π _X_ and a function _τX_ that maps from values of the low-level variables in Π _X_ to
values of the aligned high-level variable _X_ . One possible alignment between _B_ and _N_ is shown in
the diagram above: Π is depicted by the dashed lines connecting _B_ and _N_ .
We immediately know what the functions for high-level input and output variables are. For the
inputs, T is encoded as 1 and F is encoded as 0, meaning _τP_ p1q “ _τQ_ p1q “ T and _τP_ p0q “ _τQ_ p0q “ F.
For the output, the network only predicts T if _y_ ą 0, meaning _τV_ 3p _x_ q “ T if _x_ ą 0, else F. This is
simply a consequence of how a neural network is used and trained. The functions for high-level
intermediate variables _τV_ 1p _x_ q and _τV_ 2p _x_ q must be discovered and verified experimentally.


Relative to an alignment like this, we can define abstraction:
**Definition 1** (Constructive Causal Abstraction) _A high-level causal model H is a constructive_
_abstraction of a low-level causal model L under alignment_ Π _exactly when the following holds for_
_every low-level input setting_ **x** _and low-level intervention_ **I** Ð **i** _:_
_τ_ p _L_ **I** Ð **i** p **x** q˘ “ _Hτ_ p **I** Ð **i** qp _τ_ p **x** qq
_H_ being a causal abstraction of _L_ under Π guarantees that the causal mechanism for each high-level
variable _X_ is a faithful rendering of the causal mechanisms for the low-level variables in Π _X_ .
To assess the degree to which a high-level model is a constructive causal abstraction of a low-level
model, we perform interchange interventions:
**Definition 2** (Interchange Interventions) _Given source input settings_ t **s** _j_ u _[k]_ 1 _[, and non-overlapping]_
_sets of intermediate variables_ t **X** _j_ u _[k]_ 1 _[for model][ M][, define the interchange intervention as the model]_
IIp _M,_ t **s** _j_ u _[k]_ 1 _[,]_ [t] **[X]** _[ j]_ [u] _[k]_ 1 [q “] _[ M]_ [Ź] _[k]_
_j_ “1 [x] **[X]** _[j]_ [Ð][GetVals] **[X]** _j_ [p] _[M]_ [ p] _[s]_ _[j]_ [qqy]
_where_ [Ź] _[k]_ _j_ “1 [x¨y] _[ concatenates a set of interventions.]_
FINDING DISTRIBUTED ALIGNMENTS
A _base_ input setting can be fed into the resulting model to compute the counterfactual output value.
Consider the following interchange intervention:
IIp _B,_ trT _,_ Tsu _,_ tt _V_ 1uuq “ _B_ t _V_ 1uÐGetValst _V_ 1up _B_ prT _,_ Tsqq
We process a base input and a source input, and then we intervene on a target variable, replacing it
with the value obtained by processing the source. Our causal model is fully known, and so we know
ahead of time that this interchange intervention yields T. For our neural network, the corresponding
behavior is not known ahead of time. The interchange intervention corresponding to the above
(according to the alignment we are exploring) is as follows
IIp _N,_ tr1 _,_ 1su _,_ tt _H_ 1uuq “ _N_ t _V_ 1u Ð GetValst _H_ 1up _N_ pr1 _,_ 1sqq
And, indeed, the counterfactual behavior of the model and the network _N_ are unequal:
F
|V3 “ T V3 “ T
V1 “ T V2 “ T V1 “ T V2 “ T
F T T T|Col2|Col3|
|---|---|---|
|_V_1 “ T|_V_2 “ T|_V_2 “ T|
||||
|F|T|T|
|O “ ´0.26 O “ 0.08
H1 “ 0.6 H2 “ 0.94 H1 “ 0.6 H2 “ 1.28
0 1 1 1|Col2|Col3|
|---|---|---|
|_H_1 “ 0_._6|_H_2 “ 1_._28|_H_2 “ 1_._28|
||||
|0|1|1|
Under the given alignment, the interchange interventions at the low and high level have different
effects. Thus, we have a counterexample to constructive abstraction as given in Definition 1.
Although _N_ has perfect behavioral accuracy, its accuracy under the counterfactuals created by our
interventions is not perfect, and thus _B_ is not a constructive abstraction of _N_ under this alignment.


The above conclusion is based on the kind of localist causal abstraction explored in the literature to
date. As noted in Section 1, there are two risks associated with this conclusion: (1) we may have
chosen a suboptimal alignment, and (2) we may be wrong to assume that the relevant structure will
be encoded in the standard basis we have implicitly assumed throughout.
If we simply rotate the representation r _H_ 1 _,_ _H_ 2s by ´20 [˝] to get a new representation r _Y_ 1 _,Y_ 2s, then
the resulting network has perfect behavioral and counterfactual accuracy when we align _V_ 1 and _V_ 2
with _Y_ 1 and _Y_ 2. What this reveals is that there is an alignment, but not in the basis we chose. Since
the choice of basis was arbitrary, our negative conclusion about the causal abstraction relation was
spurious.
This rotation localizes the information about the first and second argument into separate dimensions. To understand this, observe that the weight matrix of the linear network rotates a two
dimensional vector by 20 [˝] and the rotation matrix rotates the representation by 340 [˝] . The two
matrices are inverses. Because this network is linear, there is no activation function and so rotating
the hidden representation “undoes” the transformation of the input by the weight matrix. Under this
non-standard basis, the first hidden dimension is equal to the first input argument and the second
hidden dimension is equal to the second input argument.
This reveals an essential aspect of distributed neural representations: there is a many-to-many
mapping between neurons and concepts, and thus multiple high-level causal variables might be
encoded in structures from overlapping groups of neurons (Rumelhart et al., 1986; McClelland et al.,
1986). In particular, Smolensky (1986) proposes that viewing a neural representation under a basis
**Y** 2
Figure 1: A generic multi-source distributed interchange intervention. The base input and two source
inputs create three total settings of a model. The top left (green) and right (blue) total
model settings are determined by two source inputs and the middle total model setting
(red) is determined by the base input. Three hidden units from each total setting are rotated
with an orthogonal matrix **R** : **X** Ñ **Y** . Then we intervene on the rotated representation for
the base input and fix two dimensions to be the value they take on for each source input,
respectively. Then we unrotate the representation with **R** [´][1] and compute a counterfactual
total model setting for the base input. In DAS, the orthogonal matrix is found with gradient
descent using a high-level causal model to guide the search process.
that is not aligned with individual neurons can reveal the interpretable distributed structure of the
neural representations.
To make good on this intuition we define a distributed intervention, which first transforms a set
of variables to a vector space, then does interchange on orthogonal sub-spaces, before transforming
back to the original representation space.
**Definition 3** Distributed Interchange Interventions _We begin with a causal model M with input_
_variables_ **S** _and source input settings_ t **s** _j_ u _[k]_ _j_ “1 _[. Let]_ **[ N]** _[ be a subset of variables in][ M][, the]_ [ target]
variables _. Let_ **Y** _be a vector space with subspaces_ t **Y** _j_ u _[k]_ 0 _[that form an orthogonal decomposition, i.e.,]_
**Y** “ [À] _[k]_ _j_ “0 **[Y]** _[j][. Let]_ **[ R]** _[ be an invertible function]_ **[ R]** [ :] **[ N]** [ Ñ] **[ Y]** _[. Write]_ [ Proj] **Y** _j_ _[for the orthogonal projection]_
_operator of a vector in_ **Y** _onto subspace_ **Y** _j._ [1] _A_ _**distributed interchange intervention**_ _yields a new_
_model_ DIIp _M,_ **R** _,_ t **s** _j_ u _[k]_ 1 _[,]_ [t] **[Y]** _[j]_ [u] _[k]_ 0 [q] _[ which is identical to][ M][ except that the mechanisms][ F]_ **[N]** _[ (which yield]_
1. Thus, Proj generalizes GetVals to arbitrary vector spaces.
FINDING DISTRIBUTED ALIGNMENTS
_values of_ **N** _from a total setting) are replaced by:_
ˆ
´
` ˘ [¯]
_F_ **N** [˚][p] **[v]** [q “] **[ R]** [´][1] Proj **Y** 0 **R** _F_ **N** p **v** q `
ÿ _k_ ´ ` ˘ [¯˙]
Proj **Y** _j_ **R** _F_ **N** p _M_ p **sj** qq _._
_j_ “1
Notice that in this definition the base setting is partially preserved through the intervention (in
subspace **Y** 0) and hence this is a _soft_ intervention on **N** that rewrites causal mechanisms while
maintaining a causal dependence between parent and child.
Under this new alignment, the high-level interchange intervention IIp _B,_ trT _,_ Tsu _,_ tt _V_ 1uuq “
_B_ t _V_ 1uÐGetValst _V_ 1up _B_ prT _,_ Tsqq is aligned with the low-level distributed interchange intervention
ff
_,_ tr1 _,_ 1su _,_ tt _Y_ 1uuq
DIIp _N,_
«
cosp´20 [˝] q ´ sinp´20 [˝] q
sinp´20 [˝] q cosp´20 [˝] q
and the counterfactual output behavior of _B_ and _N_ are equal:
T
ff
«
cosp20 [˝] q ´ sinp20 [˝] q
|Col1|Col2|O “ 0.08|Col4|Col5|
|---|---|---|---|---|
||_H_1 “ 0_._6|_H_1 “ 0_._6|_H_2 “ 1_._28||
_O_ “ 0 _._ 08
|Col1|H1 “ ´0.34|H2 “ 0.94|Col4|
|---|---|---|---|
||0|1|1|
|Col1|H1 “ 0.6|Col3|H2 “ 1.28|Col5|
|---|---|---|---|---|
||1||1||
In what follows we will assume that **X** are already vector spaces (which is true for neural nets)
and the functions **R** are rotation operators. In this case, the subspaces **Y** _j_ can be identified without
loss of generality with those spanned by the first | **Y** 0| basis vectors for **Y** 0, the next | **Y** 1| basis vectors
for **Y** 1, and so on. (The following methods would be well-defined for non-linear transformations, as
long as they were invertible and differentiable, but efficient implementation becomes harder.)


The question then arises of how to find good rotations. As we discussed above, previous causal
abstraction analyses of neural networks have performed brute-force search through a discrete space
of hand-picked alignments. In distributed alignment search (DAS), we find an alignment between one
or more high-level variables and disjoint sub-spaces (but not necessarily subsets) of a large neural
representation. We define a distributed interchange intervention training objective, use differentiable
parameterizations for the space of orthogonal matrices (such as provided by PyTorch), and then
optimize the objective with stochastic gradient descent. Crucially, the low-level and high-level
models are frozen during learning so we are only changing the alignment.
In the following definition we assume that a neural network specifies an output _distribution_ for a
given input, which can then be pushed forward to a distribution on output values of the high-level
model via an alignment function _τ_ . We may similarly interpret even a deterministic high-level
model as defining a (e.g., delta) distribution on output values. We make use of these distributions,
after interchange intervention, to define a differentiable loss for the rotation matrix which aligns
intermediate variables.
**Definition 4** Distributed Interchange Intervention Training Objective _Begin with a low-level neural_
_network L, with low-level input settings_ **Inputs** _L, a high-level algorithm H, with high-level output_
_settings_ **Out** _H, and an alignment τ for their input and output variables. Suppose we want to align_
_intermediate high level variables X_ _j_ P **Vars** _H with rotated subspaces_ **Y** _j of a neural representation_
**N** Ă **Vars** _L with learned rotation matrix_ **R** _[θ]_ : **N** Ñ **Y** _._
_In general, we can define a training objective using any differentiable loss function_ Loss _that_
_quantifies the distance between two total high-level settings._
ˆ ˙
ÿ
Loss DIIp _L,_ **R** _[θ]_ _,_ t **s** _j_ u _[k]_ 1 _[,]_ [t] **[Y]** _[ j]_ [u] _[k]_ 0 [qp] **[b]** [q] _[,]_ [II][p] _[H][,]_ [t] _[τ]_ [p] **[s]** _[j]_ [qu] _[k]_ 1 _[,]_ [t] **[X]** _[ j]_ [u] _[k]_ 1 [qp] _[τ]_ [p] **[b]** [qq]
**b** _,_ **s** 1 _,...,_ **s** _k_ P **Inputs** _L_
_For our experiments, we compute the cross entropy loss_ CEp¨ _,_ ¨q _between the high-level output_
_distribution_ Pp **out** _H_ | _H_ p _τ_ p **b** qqq _and the push-forward under τ of the low-level output distribution_
P _[τ]_ p **out** _H_ | _L_ p **b** qq _. The overall objective is:_
ˆ ˙
ÿ
CE Pp **out** _H_ |IIp _H,_ t _τ_ p **s** _j_ qu _[k]_ 1 _[,]_ [t] **[X]** _[ j]_ [u] _[k]_ 1 [qqp] _[τ]_ [p] **[b]** [qq] _[,]_ [P] _[τ]_ [p] **[out]** _[H]_ [|][DII][p] _[L][,]_ **[R]** _[θ]_ _[,]_ [t] **[s]** _[j]_ [u] _[k]_ 1 _[,]_ [t] **[Y]** _[j]_ [u] _[k]_ 0 [qp] **[b]** [qq]
**b** _,_ **s** 1 _,...,_ **s** _k_ P **Inputs** _L_
While we still have discrete hyperparameters p **N** _,_ | **Y** 0| _,...,_ | **Y** _k_ |q—the target population and the
dimensionality of the sub-spaces used for each high-level variable—we may use stochastic gradient
descent to determine the rotation that minimizes loss, thus yielding the best distributed alignment
between _L_ and _H_ .


Perfect causal abstraction relationships are unlikely to arise for neural networks trained to solve
complex empirical tasks. We use a graded notion of accuracy:
**Definition 5** Distributed Interchange Intervention Accuracy _Given low-level and high-level causal_
_models L and H with alignment_ pΠ _,_ _τ_ q _, rotation_ **R** : **N** Ñ **Y** _, and orthogonal decomposition_ t **Y** _j_ u _[k]_ 0 _[.]_
_If we let_ **Inputs** _L be low-level input settings and_ t **X** _j_ u _[k]_ 1 _[be high-level intermediate variables the]_
_**interchange intervention accuracy (IIA)**_ _is as follows_
ÿ
**b** _,_ **s** 1 _,...,_ **s** _k_ P **Inputs** _L_
| **Inputs** _L_ | _[k]_ [`][1]
” ı
_τ_ `DIIp _L,_ **R** _[θ]_ _,_ t **s** _j_ u _[k]_ 1 _[,]_ [t] **[Y]** _[ j]_ [u] _[k]_ 0 [qp] **[b]** [q] ˘ “ IIp _H,_ t _τ_ p **s** _j_ qu _[k]_ 1 _[,]_ [t] **[X]** _[ j]_ [u] _[k]_ 1 [qp] _[τ]_ [p] **[b]** [qq]
IIA is the proportion of aligned interchange interventions that have equivalent high-level and lowlevel effects. In our example with _N_ and _A_, IIA is 100% and the high-level model is a perfect
abstraction of the low-level model (Def. 1). When IIA is _α_ ă100%, we rely on the graded notion of
_α-on-average_ approximate causal abstraction (Geiger et al., 2023), which coincides with IIA.


We illustrate the value of DAS by analyzing feed-forward networks trained on a hierarchical equality
and pretrained Transformer-based language models (Vaswani et al., 2017) fine-tuned on a natural
language inference task. Our evaluation paradigm is as follows:
1. Train the neural network _N_ to solve the task. In all experiments, the neural models achieve
perfect accuracy on both training and testing data.
2. Create interchange intervention training datasets using a high-level causal model. Each
example consists of a base input, one or more source inputs, high-level causal variables
FINDING DISTRIBUTED ALIGNMENTS
targetted for intervention, and a counterfactual gold label that will be output by the network
if the interchange intervention has the hypothesized effect on model behavior. This gold
label is a counterfactual output of the high-level model we will align with the network. (See
Appendix A.1 for details)
3. Optimize an orthogonal matrix to learn a distributed alignment for each high-level model that
maximizes IIA using the training objective in Def. 4. We experiment with different hidden
dimension sizes for our low-level model and different intervention site sizes (dimensionality of low-level subspaces) and locations (the layer where the intervention happens). (See
Appendix A.2 for details)
4. Evaluate a baseline that brute-force searches through a discrete space of alignments and selects
the alignment with the highest IIA. We search the space of alignments by aligning each highlevel variable with groups of neurons in disjoint sliding windows. (See Appendix A.3 for
details)
5. Evaluate the localist alignment “closest” to the learned distributed alignment. The rotation matrix for the localist alignment will be axis-aligned with the standard basis, possibly permuting
and reflecting unit axes. (See Appendix A.4 for details)
6. Determine whether each distributed representation aligned with high-level variables can be
decomposed into multiple representations that encode the identity of the input values to the
variable’s causal mechanism. We do this by learning a second rotation matrix that decomposes
learned distributed representation, holding the first rotation matrix fixed. (See Appendix A.5
for details)
The codebase used to run these experiments is at [2] . We have replicated the hierarchical equality
experiment using the Pyvene library at [3] .


We now illustrate the power of DAS for analyzing networks designed to solve a hierarchical equality
task. We concentrate on analyzing a trained feed-forward network.
A _basic_ equality task is to determine whether a pair of objects are the same ( _x_ “ _y_ ). A _hierarchical_
equality task is to determine whether a pair of pairs of objects have identical relations: p _w_ “ _x_ q “
p _y_ “ _z_ q. Specifically, the input to the task is two pairs of objects and the output is True if both pairs
are equal or both pairs are unequal and False otherwise. For example, p _A,_ _A,_ _B,_ _B_ q and p _A,_ _B,C,_ _D_ q
are both assigned True while p _A,_ _B,C,C_ q is assigned False.


We train a three-layer feed-forward network with ReLU activations to perform the hierarchical
equality task. Each input object is represented by a randomly initialized vector. Specifically, our
model has the following architecture where _k_ is the number of layers.
_h_ 1 “ ReLUpr _x_ 1; _x_ 2; _x_ 3; _x_ 4s _W_ 1 ` _b_ 1q _h_ _j_ ´1 “ ReLUp _h_ _jWj_ ` _b_ _j_ q _y_ “ **softmax** p _hkWk_ ` _bk_ q
The input vectors are in R _[n]_, the biases are in R [4] _[n]_, and the weights are in R [4] _[n]_ [ˆ][4] _[n]_ . We evaluate our
model on held-out random vectors unseen during training, as in Geiger et al. 2022a.
[2. https://github.com/atticusg/InterchangeInterventions/tree/zen](https://github.com/atticusg/InterchangeInterventions/tree/zen)
[3. https://github.com/stanfordnlp/pyvene/blob/main/tutorials/advanced_tutorials/DAS_Main_](https://github.com/stanfordnlp/pyvene/blob/main/tutorials/advanced_tutorials/DAS_Main_Introduction.ipynb)
[Introduction.ipynb](https://github.com/stanfordnlp/pyvene/blob/main/tutorials/advanced_tutorials/DAS_Main_Introduction.ipynb)
Both Equality Relations Left Equality Relation Identity of First Argument Identity Subspace of
Left Equality
Hidden size Intervention size **Layer 1** **Layer 2** **Layer 3** **Layer 1** **Layer 2** **Layer 3** **Layer 1** **Layer 2** **Layer 3** **Layer 1**
| **N** | “ 16 1 0.88 0.51 0.50 0.85 0.54 0.50 0.51 0.52 0.50 0.51
| **N** | “ 16 2 0.97 0.54 0.50 0.85 0.55 0.50 0.50 0.52 0.51 0.50
| **N** | “ 16 8 1.00 0.57 0.50 0.90 0.56 0.50 0.52 0.53 0.51 0.51
| **N** | “ 32 2 0.93 0.63 0.49 0.92 0.65 0.50 0.52 0.55 0.52 0.50
| **N** | “ 32 4 0.97 0.63 0.49 0.94 0.65 0.50 0.51 0.55 0.52 0.51
| **N** | “ 32 16 0.99 0.67 0.53 0.99 0.65 0.50 0.49 0.55 0.52 0.51
Brute-Force Search 0.60 0.56 0.52 0.64 0.64 0.57 0.50 0.51 0.54    Localist Alignment 0.73 0.56 0.48 0.60 0.50 0.49 0.46 0.47 0.48
Table 1: Hierarchical equality alignment learning results. The table can be read as follows: **Layer**
**1**, **Layer 2**, and **Layer 3** indicate which layer of neurons is targeted, | **N** | is the number
of neurons in a layer, _k_ is the number of neurons aligned with each intermediate variable
(red) where our subspace model occupies _[k]_
2 [with rounding up to the closest integer, and]
the values in each cell are interchange intervention accuracies for the learned alignment
on training data. We report the best results from three runs with distinct random seeds for
training the rotation matrix (the same frozen low-level model is used for each seed).


We use DAS to evaluate whether trained neural networks have achieved the natural solution to the
hierarchical equality task where the left and right equality relations are computed and then used to
predict the final label (Figure 2).
However, evaluating this high-level model alone is
insufficient, as there are obviously many other high-level
models of this task. To further contextualize our results,
we also consider two alternatives: a high-level model
where only the equality relation of the first pair is represented and a high-level model where the lone intermediate
variable encodes the identity of the first input object (leaving all computation for the final step). These alternative
high-level models also solve the task perfectly.


Figure 2: A causal model that computes
the hierarchical equality task.
The IIA results achieved by the best alignment for each high-level model can be seen in Table 1. The
best alignments found are with the ‘Both Equality Relations’ model that is widely assumed in the
cognitive science literature. For all causal models, DAS learns a more faithful alignment (higher
IIA) than a brute-force search through localist alignments. This result is most pronounced for ‘Both
Equality Relations’, where DAS learns perfect or near-perfect alignments under a number of settings,
whereas the best brute-force alignment achieves only 0.60 and the best localist alignment achieves
only 0.73. Finally, the distributed representation of left equality could not be decomposed into a
representation of the first argument identity. We see this in the very low performance of the ‘Identity
Subspace of Left Equality’ results. This indicates that models are truly learning to encode an abstract
equality relation, rather than merely storing the identities of the inputs.
FINDING DISTRIBUTED ALIGNMENTS


_premise_ : A man is talking to someone in a taxi. _entails_
_hypothesis_ : A man is talking to someone in a car.
_premise_ : The people are **not** playing sitars. _neutral_
_hypothesis_ : The people are **not** playing instruments.
( _a_ ) Two MoNLI examples.
MONLIp **p** _,_ **h** q
1 _lexrel_ Ð GET-LEXRELp **p** _,_ **h** q
2 _neg_ Ð CONTAINS-NOTp **p** _,_ **h** q
3 **if** _neg_ :
4 **return** REVERSEp _lexrel_ q
5 **return** _lexrel_
( _b_ ) A simple program that
solves MoNLI.
Figure 4: Monotonicity NLI task examples and high-level model.


To calibrate intuitions about our method, we evaluate the
ability of DAS to optimize for interchange intervention
accuracy on a frozen randomly initialized networks that
achieves chance accuracy (50%) on the hierarchical equality task. This investigates the degree to which random
causal structures can be used to systematically manipulate
the counterfactual behavior of the network. We evaluate
networks with different hidden representation sizes while
holding the four input vectors fixed at 4 dimensions, under the hypothesis that more hidden neurons create more
random structure that DAS can search through. These results are summarized in Table 4.4. Observe that, in small
networks, there is no ability to increase interchange intervention accuracy. However, as we increase the size of
the hidden representation to be orders of magnitude larger
than the input dimension of 16, the interchange intervention accuracy increases. This confirms our hypothesis and
serves as a check that demonstrates DAS cannot construct
entirely new behaviors from random structure.

4.4. Analyzing a Randomly Initialized Network