[17] proposes non-autoregressive neural machine translation, which achieves significant decoding
speedup by generating target words simultaneously. NAT breaks the dependency among target tokens
and factorizes the joint probability of target words in the following form:
_p_ ( _y|x, θ_ ) =
_T_
- _pt_ ( _yt|x, θ_ ) _,_ (1)
_t_ =1
where _x_ is the source sentence belonging to the input space _X_ _, y_ = _{y_ 1 _, ..., yT }_ is the target sentence
belonging to the output space _Y_, and _pt_ ( _yt|x, θ_ ) indicates the translation probability in position _t_ .
The vanilla-NAT has a length predictor that takes encoder states as input to predict the target length.
During the training, the target length is set to the golden length, and the vanilla-NAT is trained with
the cross-entropy loss, which explicitly aligns model outputs to target words:
_LCE_ ( _θ_ ) = _−_
_T_
- log( _pt_ ( _yt|x, θ_ )) _._ (2)
_t_ =1
' = (ε, A, A, ε, A, B, B, C, ε, D)
(A, A, B, C, D) (A, A, A, B, B, C, D)
CTC: y = # [−1] (')
SCTC: y = #) [−1] (')
Figure 2: An example of target sentences obtained from collapsing functions of CTC and SCTC.
The collapsing function of SCTC is _βs_ _[−]_ [1][, which only removeS blanks in the alignment] _[ a]_ [, where the]
collapsing function of CTC removes repetitions first and then removes blanks.

2.1 Non-Autoregressive Machine Translation


The vanilla-NAT suffers from two major limitations. The first limitation is the explicit alignment
required by the cross-entropy loss, which cannot be guaranteed due to the multi-modality problem
and therefore leads to the inaccuracy of the loss. The second limitation is the requirement of target
length prediction. The predicted length may not be optimal and cannot be changed dynamically, so it
is often required to use multiple length candidates and re-score them to produce the final translation.
The two limitations can be addressed by using CTC-based latent alignment models [15, 29, 36],
which extend the output space _Y_ with a blank token _ϵ_ that means ‘output nothing’. We define the
extended output space as _Y_ _[∗]_ . Following prior works [14, 36], we refer to the elements _a ∈Y_ _[∗]_ as
alignments, since the location of the blank tokens determines an alignment between the extended
output and target sentence. Assume the generated alignments have length _Ta_, we define a function
_β_ ( _y_ ) which returns a subset of _Y_ _[∗]_ including all possible alignments of length _Ta_ for _y_, where
_β_ _[−]_ [1] : _Y_ _[∗]_ _�→Y_ is the collapsing function that first collapses all consecutive repeated words in
_a_ and then removes all blank tokens to obtain the target sentence. As illustrated in Figure 2, the
alignment _a_ = ( _ϵ, A, A, ϵ, A, B, B, C, ϵ, D_ ) is collapsed to the target sentence _y_ = ( _A, A, B, C, D_ )
with a monotonic mapping from target positions to alignment positions. During the training, latent
alignment models marginalize all alignments with the CTC loss [15]:
log _p_ ( _y|x, θ_ ) = log                - _p_ ( _a|x, θ_ ) _,_ (3)
_a∈β_ ( _y_ )
where the alignment probability _p_ ( _a|x, θ_ ) is modeled by a non-autoregressive Transformer:
_p_ ( _a|x, θ_ ) =
_Ta_
- _pt_ ( _at|x, θ_ ) _._ (4)
_t_ =1
NAT models with latent alignments achieve superior performance by overcoming the two major limitations of NAT [29, 36, 16, 25, 33, 49]. The weakness in handling non-monotonic latent alignments
is noticed in previous work [36, 16] but remains unsolved.

Non-Monotonic Latent Alignments for CTC-Based Non-Autoregressive Machine
  Translation


In this section, we explore non-monotonic latent alignments for NAT. We introduce a simplified CTC
loss in section 3.1, which has a simpler structure that helps further analysis and derivation under the
regular CTC loss. We explore non-monotonic alignments under the simplified CTC loss in section
3.2, and utilize the results to derive non-monotonic alignments under the regular CTC loss in section
3.3. Finally, we introduce the training strategy to combine monotonic and non-monotonic latent
alignments in section 3.4.


In CTC, the collapsing function _β_ _[−]_ [1] maps an alignment to a target sentence in two steps: (1) collapses
all consecutive repeated words, (2) removes all blank tokens. The two operations make the alignment
space _β_ ( _y_ ) complex. Therefore, we first introduce the simplified connectionist temporal classification
loss (SCTC), which has a simpler structure that helps the derivation of non-monotonic alignments,
and the results on SCTC are helpful for further analysis under the regular CTC loss.
To simplify the alignment structure, we consider only using one operation in the collapsing function.
One option is only collapsing all consecutive repeated words in the alignment. The concern is that it
will limit the expressive power of the model. For example, the target sentence _y_ = ( _A, A, B, C, D_ )
has probability 0 since it has repeated words that cannot be mapped from any alignment. Therefore,
we favor another option that only removes all blank tokens. In this way, the expressive power stays
the same and the alignment space becomes much simpler, where alignments of _y_ simply contain _T_
target words and _Ta −_ _T_ blank tokens.
We denote this simplified loss as SCTC, and illustrate the difference between CTC and SCTC in
Figure 2. The alignment space for the target sentence _y_ is defined as _βs_ ( _y_ ), and the collapsing
function is defined as _βs_ _[−]_ [1][. We can still obtain the translation probability] _[ p]_ [(] _[y][|][x, θ]_ [)][ with dynamic]
programming. We define the forward variable _αt_ ( _s_ ) to be the total probability of _y_ 1: _s_ considering all
possible alignments _a_ 1: _t_, which can be calculated recursively from _αt−_ 1( _s_ ) and _αt−_ 1( _s −_ 1):
_αt_ ( _s_ ) = _αt−_ 1( _s −_ 1) _pt_ ( _ys|x, θ_ ) + _αt−_ 1( _s_ ) _pt_ ( _ϵ|x, θ_ ) _._ (5)
Finally, the total translation probability _p_ ( _y|x, θ_ ) is given by the forward variable _αTa_ ( _T_ ), so we can
train the model with the cross-entropy loss _Lsctc_ ( _θ_ ) = _−_ log _αTa_ ( _T_ ).

3.1 Simplified Connectionist Temporal Classification


In this section, we explore non-monotonic alignments under the SCTC loss, which are helpful for
further analysis under the regular CTC loss. We first extend the alignment space from monotonic
alignments _βs_ ( _y_ ) to non-monotonic alignments _γs_ ( _y_ ) to allow for the global word reordering in
machine translation, where _γs_ ( _y_ ) is defined as:
_γs_ ( _y_ ) = _∪_ (6)
_y_ _[′]_ _∈P_ ( _y_ ) _[β][s]_ [(] _[y][′]_ [)] _[.]_
In the above definition, _P_ ( _y_ ) represents all permutations of _y_ . For example, _P_ (( _A, B_ )) =
_{_ ( _A, B_ ) _,_ ( _B, A_ ) _}_ . By enumerating _P_ ( _y_ ), we consider all possible word reorderings of the target sentence. Ideally, we want to traverse all alignments in _γs_ ( _y_ ) to calculate the log-likelihood loss:
_Lsum_ ( _θ_ ) = _−_ log            - _p_ ( _a|x, θ_ ) _._ (7)
_a∈γs_ ( _y_ )
However, without the monotonic structure, it becomes difficult to marginalize out all latent alignments
with dynamic programming. Alternatively, we can minimize the loss of the best alignment, which is
an upper bound of Equation 7:
_Lmax_ ( _θ_ ) = _−_ log max (8)
_a∈γs_ ( _y_ ) _[p]_ [(] _[a][|][x, θ]_ [)] _[.]_
Following prior work [46, 7, 9], we formulize finding the best alignment as a maximum bipartite
matching problem and solve it with the Hungarian algorithm [28]. Specifically, we observe that the
alignment space _γs_ ( _y_ ) is simply permutations of _T_ target words and _Ta −_ _T_ blank tokens. Therefore,
finding the best alignment is equivalent to finding the best bipartite matching between the _Ta_ model
predictions and the _T_ target words plus _Ta −_ _T_ blank tokens, where the two sets of nodes are
connected by edges with the prediction log-probability as weights.


Without the monotonic structure desired for dynamic programming, calculating Equation 7 becomes
difficult. This difficulty is also caused by the strict requirement of exact match for alignments, which
makes it intractable to simplify the summation of probabilities. In practice, it is not necessary to force
the exact match between alignments and the target sentence since a large overlap is also favorable.
Therefore, we further extend the alignment space by considering all alignments that overlap with the
target sentence. Specifically, we are interested in the overlap of n-grams, which is the core of some
evaluation metrics (e.g., BLEU).
We propose a non-monotonic and non-exclusive n-gram matching objective based on SCTC to
encourage the overlap of n-grams. Following the underlying idea of probabilistic matching [39], we
introduce the probabilistic variant of the n-gram count to make the objective differentiable.
Specifically, we use _Cg_ ( _y_ ) to denote the occurrence count of n-gram _g_ = ( _g_ 1 _, ..., gn_ ) in the target
sentence _y_, and use _Cg_ _[s]_ [(] _[θ]_ [)][ to denote the probabilistic count of n-gram] _[ g]_ [ for the model with input] _[ x]_
and parameter _θ_, where the superscript _s_ indicates SCTC. For simplicity, we omit the source sentence
_x_ in _Cg_ _[s]_ [(] _[θ]_ [)][ and the following notations. The probabilistic n-gram count is obtained by accumulating]
n-grams from all possible target sentences:
- _p_ ( _y|x, θ_ ) _Cg_ ( _y_ ) =
_y∈Y_ _a∈Y_
_Cg_ _[s]_ [(] _[θ]_ [) =]
- _p_ ( _a|x, θ_ ) _Cg_ ( _βs_ _[−]_ [1][(] _[a]_ [))] _[.]_ (9)
_a∈Y_ _[∗]_
The calculation of _Cg_ _[s]_ [(] _[θ]_ [)][ is the core of this method and will be described in detail later. We use]
_Mg_ _[s]_ [(] _[θ]_ [)][ to denote the match count of n-gram g between the latent alignment model and the target]
sentence _y_ (we omit _y_ in _Mg_ _[s]_ [(] _[θ]_ [)][ for simplicity), which is defined as follows:]
_Mg_ _[s]_ [(] _[θ]_ [) = min(] _[C][g]_ [(] _[y]_ [)] _[, C]_ _g_ _[s]_ [(] _[θ]_ [))] _[.]_ (10)
Our objective is to maximize the n-gram overlap between the target sentence and model output, so
we can train the model to maximize the precision or recall of n-gram matching:
_g∈Gn_ _[M][ s]_ _g_ [(] _[θ]_ [)]
(11)
~~�~~
_g∈Gn_ _[C][g]_ [(] _[y]_ [)] _[,]_
_Pn_ _[s]_ [(] _[θ]_ [) =]
_g∈Gn_ _[M][ s]_ _g_ [(] _[θ]_ [)]
~~�~~ _n_ [(] _[θ]_ [) =]
_g∈Gn_ _[C]_ _g_ _[s]_ [(] _[θ]_ [)] _[, R][s]_
where we use _Gn_ to denote the set of all n-grams. However, since latent alignment models have
the ability to control the translation length, both precision and recall cannot accurately reflect the
translation quality. The precision will encourage short translations to reduce the denominator, while
the recall prefers long translations. Therefore, we consider both of them and maximize the F1 score:
2 _·_ [�] _g∈Gn_ _[M][ s]_ _g_ [(] _[θ]_ [)]
~~�~~ _[s]_
2 _·_ [�]
_L_ _[s]_ _n_ [(] _[θ]_ [) =] _[ −]_ [F1] _[s]_ _n_ [(] _[θ]_ [) =] _[ −]_
_n_ (12)
_g∈Gn_ [(] _[C][g]_ [(] _[y]_ [) +] _[ C]_ _g_ _[s]_ [(] _[θ]_ [))] _[,]_
where we use _Gn_ to denote the set of all n-grams. In the numerator of Equation 12, _Mg_ _[s]_ [(] _[θ]_ [)][ is non-]
zero only if _Cg_ ( _y_ ) is non-zero, so we only need to calculate probabilistic n-gram counts _Cg_ _[s]_ [(] _[θ]_ [)][ for]
n-grams _g_ in the target sentence. In the denominator, [�] _g∈Gn_ _[C][g]_ [(] _[y]_ [)][ equals to the constant] _[ T][ −]_ _[n]_ [ + 1][.]
n-grams _g_ in the target sentence. In the denominator, [�] _g∈Gn_ _[C][g]_ [(] _[y]_ [)][ equals to the constant] _[ T][ −]_ _[n]_ [ + 1][.]
Therefore, there are only two tasks left: calculating _Cg_ _[s]_ [(] _[θ]_ [)][ and the summation][ �] _g∈Gn_ _[C]_ _g_ _[s]_ [(] _[θ]_ [)][.]
For the first task of calculating _Cg_ _[s]_ [(] _[θ]_ [)][, we first consider the simple case] _[ n]_ [ = 1][. The main difference]
between 1-gram matching and bipartite matching is that 1-gram matching is non-exclusive, which
allows us to marginalize out all alignments using the property of non-autoregressive generation. Due
to the space limit, we refer readers to Appendix.A for the derivation and directly give the probabilistic
1-gram count as follows:
_g∈Gn_ _[C]_ _g_ _[s]_ [(] _[θ]_ [)][.]
_Cg_ _[s]_ [(] _[θ]_ [) =]
_Ta_
- _pt_ ( _g_ 1 _|x, θ_ ) _, g ∈_ _G_ 1 _,_ (13)
_t_ =1
where _G_ 1 denotes the set of all 1-grams _g_ = ( _g_ 1). Both 1-gram matching and bipartite matching are
completely non-monotonic, which aim at generating correct target words regardless of the word order.
When the target sentence is “I ate pizza this afternoon”, they will think that “pizza ate I this afternoon”
is also a good translation. We consider 2-gram matching to overcome this limitation, which is also
non-monotonic but takes target dependency into consideration. We first introduce a transition matrix
_A_ of size _Ta × Ta_, where _Ai,j_ is the probability that all positions between _i_ and _j_ are blank tokens:
_Ai,j_ =


0 _j < i_ + 1
1 _j_ = _i_ + 1 _._ (14)
- _jt_ = _−i_ 1+1 _[p][t]_ [(] _[ϵ][|][x, θ]_ [)] _j > i_ + 1

Then we give the probabilistic 2-gram count for _g_ = ( _g_ 1 _, g_ 2) (see Appendix.A for derivation):
_Ta_
- _pi_ ( _g_ 1 _|x, θ_ ) _Ai,jpj_ ( _g_ 2 _|x, θ_ ) _, g ∈_ _G_ 2 _._ (15)
_j_ =1
_Cg_ _[s]_ [(] _[θ]_ [) =]
_Ta_
_i_ =1
For a larger _n_, the probabilistic n-gram count is:
_Ta_
_Ta_
- _pi_ 1( _g_ 1 _|x, θ_ ) _×_
_in_ =1
_Ta_
_n_
- _Aik−_ 1 _,ik_ _pik_ ( _gk|x, θ_ ) _, g ∈_ _Gn._ (16)
_k_ =2
_Cg_ _[s]_ [(] _[θ]_ [) =]
_Ta_
_i_ 1=1
_Ta_
- _..._
_i_ 2=1
Equation 15 can be efficiently calculated in matrix form with _O_ ( _Ta_ [2][)][ time complexity. For Equation]
16, we need _O_ ( _nTa_ _[n]_ [)][ operations to calculate it by brute force, which is impractical when] _[ n]_ [ is large.]
We refer readers to Appendix.B for the _O_ ( _nTa_ [2][)][ algorithm, which is similar to calculating the n-step]
transition probability of a Markov chain.
For the second task of efficiently calculating the summation of probabilistic n-gram counts
_g∈Gn_ _[C]_ _g_ _[s]_ [(] _[θ]_ [)][, we also consider the simple case] _[ n]_ [ = 1][ first, where the summation can be cal-]
culated by directly summing all probabilities:
- _Cg_ _[s]_ [(] _[θ]_ [) =]
_g_
_Ta_
_t_ =1
- _pt_ ( _g_ 1 _|x, θ_ ) _, g ∈_ _G_ 1 _._ (17)
_g_
For _n >_ 1, the search space is too big for the direct summation. Fortunately, the definition of _Cg_ _[s]_ [(] _[θ]_ [)]
in Equation 9 allows us to safely extend the result of _n_ = 1 to a larger _n_ since n-grams count in a
sentence is always _n −_ 1 less than 1-grams count.
     - _[s]_     - _[s]_
- _Cg_ _[s]_ [(] _[θ]_ [) =]
_g∈Gn_ _g∈G_
- _Cg_ _[s]_ [(] _[θ]_ [)] _[ −]_ _[n]_ [ + 1] _[.]_ (18)
_g∈G_ 1
In summary, with the above results (Equation 13-18), we are able to efficiently calculate the loss
defined in Equation 12 to maximize the F1 score of non-monotonic n-gram matching, which reflects
the translation quality to a certain extent.


In this section, we explore non-monotonic latent alignments under the regular CTC loss. Under
SCTC, we can use both the bipartite matching and n-gram matching to train the latent alignment
model. However, bipartite matching is infeasible under the regular CTC loss. Recall that the search
space _γs_ ( _y_ ) is simply permutations of target words and _Ta −_ _T_ blank tokens, which allows us to
find the best alignment with the Hungarian algorithm. However, as consecutive repeated words will
be collapsed in the regular CTC loss, alignments in _γ_ ( _y_ ) may contain different words, so the best
alignment cannot be found by the Hungarian algorithm.
Fortunately, for n-gram matching, we can utilize previous results of SCTC to help the analysis and
derivation under the regular CTC loss. As the output of the collapsing function _βs_ _[−]_ [1][(] _[a]_ [)][ simply]
contains more repeated words than _β_ _[−]_ [1] ( _a_ ), we can first identify how much influence these repeated
words have, and then remove this influence from the results of SCTC. We first formally define the
probabilistic n-gram count under the regular CTC:
_Cg_ ( _θ_ ) =            - _p_ ( _a|x, θ_ ) _Cg_ ( _β_ _[−]_ [1] ( _a_ )) _._ (19)
_a∈Y_ _[∗]_
We define _Rg_ ( _θ_ ) to be the gap bewteen _Cg_ ( _θ_ ) and _Cg_ _[s]_ [(] _[θ]_ [)][ when] _[ n]_ [ = 1][, which represents the number]
of repeated words _g_ 1 removed by the collapsing function. _Rg_ ( _θ_ ) can be efficiently calculated as
follows (see Appendix.C for derivation):
_Rg_ ( _θ_ ) = _Cg_ _[s]_ [(] _[θ]_ [)] _[ −]_ _[C][g]_ [(] _[θ]_ [) =]
_Ta−_ 1
- _pt_ ( _g_ 1 _|x, θ_ ) _pt_ +1( _g_ 1 _|x, θ_ ) _, g ∈_ _G_ 1 _._ (20)
_t_ =1
The above equation enables us to efficiently calculate _Cg_ ( _θ_ ) and [�] _g_ _[C][g]_ [(] _[θ]_ [)][ when] _[ g][ ∈]_ _[G]_ [1][. For]
2-grams _g_ = ( _g_ 1 _, g_ 2), we divide it into two cases _g_ 1 = _g_ 2 and _g_ 1 _̸_ = _g_ 2. The collapsing of consecutive
repeated words will only remove 2-grams of the first case, where each removed word corresponds
to a removed 2-gram. Taking Figure 2 as an example, the output of SCTC _βs_ _[−]_ [1][(] _[a]_ [)][ have two more]
2-grams _{_ ( _A, A_ ) _,_ ( _B, B_ ) _}_ of the _g_ 1 = _g_ 2 case than the output of CTC _β_ _[−]_ [1] ( _a_ ), and they contain
exactly the same 2-grams of the _g_ 1 _̸_ = _g_ 2 case. Therefore, we can directly reuse previous results of
SCTC and the number of repeated words _Rg_ ( _θ_ ) to calculate the probabilistic 2-gram count:
          - _Cgs_ [(] _[θ]_ [)] _[ −]_ _[R][g]_ [(] _[θ]_ [)] _[,]_ _g_ 1 = _g_ 2
_Cg_ ( _θ_ ) = _, g ∈_ _G_ 2 _._ (21)
_Cg_ _[s]_ [(] _[θ]_ [)] _[,]_ _g_ 1 _̸_ = _g_ 2
Unfortunately, for _n >_ 2, there is no such a clear relationship between _Cg_ ( _θ_ ) and _Cg_ _[s]_ [(] _[θ]_ [)][, where the]
CTC output can contain n-grams (for example, ( _A, B, C_ ) in Figure 2) that the SCTC output does not
have. If we directly formulize _Cg_ ( _θ_ ) like in Equation 16, it will be more complex and the _O_ ( _nTa_ [2][)]
algorithm is no longer applicable. Therefore, we do not use n-gram matching with _n >_ 2 under the
regular CTC loss. Finally, we define the match count _Mg_ ( _θ_ ) similarly as in Equation 10 and train the
CTC-based latent alignment model to maximize the F1 score:
2 _·_ [�]
_Ln_ ( _θ_ ) = _−_
2 _·_ [�] _g∈Gn_ _[M][g]_ [(] _[θ]_ [)]
~~�~~
_n_ _._ (22)
_g∈Gn_ ( _Cg_ ( _y_ )+ _Cg_ ( _θ_ ))


In the above, we propose non-monotonic training objectives for the latent alignment model. They need
to be combined with monotonic training objectives, otherwise, they may generate translations like
“pizza ate I this afternoon”. Therefore, we first pretrain the latent alignment model with monotonic
training objectives like SCTC or CTC. The model pretrained by SCTC can be finetuned with the
bipartite matching objective (Equation 8) or the n-gram matching objective (Equation 12). The model
pretrained by CTC can be finetuned with the n-gram matching objective (Equation 22), and the
choices of _n_ are limited to _{_ 1 _,_ 2 _}_ .


**Datasets** We evaluate our methods on the most widely used public benchmarks in previous NAT studies: WMT14 English _↔_ German (En _↔_ De, 4.5M sentence pairs) [5] and WMT16 English _↔_ Romanian
(En _↔_ Ro, 0.6M sentence pairs) [6]. For WMT14 En _↔_ De, the validation set is _newstest2013_ and the
test set is _newstest2014_ . For WMT16 En _↔_ Ro, the validation set is _newsdev-2016_ and the test set
is _newstest-2016_ . Following prior works, we use tokenized BLEU [32] to evaluate the translation
quality. Considering that BLEU might be biased as our method optimizes the n-gram overlap, we also
report METEOR [2], which is not directly based on n-gram overlap. We learn a joint BPE model [37]
with 32K operations to process the data and share the vocabulary for source and target languages.
**Knowledge Distillation** Following prior works [17], we apply sequence-level knowledge distillation
[26] to reduce the complexity of training data. We use Transformer-base setting for the teacher model
and train NAT on the distilled data.
**Implementation Details** We adopt Transformer-base [47] as our autoregressive baseline as well as
the teacher model. We uniformly copy encoder outputs to construct decoder inputs. For CTC-based
NAT models, the length for decoder inputs is 3 _×_ as long as the source length. On WMT14 En _↔_ De,
we use a dropout rate of 0.2 to train NAT models and use a dropout rate of 0.1 for finetuning. On
WMT16 En _↔_ Ro, the dropout rate is 0.3 for both the pretraining and finetuning. We use the batch
size 64K and train NAT models for 300K steps on WMT14 En _↔_ De and 150K steps on WMT16
En _↔_ Ro. During the finetuning, we train NAT models for 6K steps with the batch size 256K. All
models are optimized with Adam [27] with _β_ = (0 _._ 9 _,_ 0 _._ 98) and _ϵ_ = 10 _[−]_ [8] . The learning rate warms
up to 5 _·_ 10 _[−]_ [4] within 10K steps in the pretraining and warms up to _e ·_ 10 _[−]_ [4] within 500 steps in the
finetuning. We use the GeForce RTX 3090 GPU to train models and measure the translation latency.
We implement our models based on the open-source framework of `fairseq` [MIT License, 31].
**Decoding** For autoregressive models, we use beam search with beam size 5 for the decoding. For
Vanilla-NAT, we use argmax decoding to generate the translation. For CTC-based models, we can
also use argmax decoding to generate the alignment and then collapse it to obtain the translation.
Besides, following [29, 16], we can use beam search decoding combined with a 4-gram language
model [20] to find the translation that maximizes:
log _p_ ( _y|x, θ_ ) + _α_ log _pLM_ ( _y_ ) + _β_ log( _|y|_ ) _,_ (23)
where _α_ and _β_ are hyperparameters for the language model score and length bonus. We use a
fixed beam size 20 with grid-search _α_, _β_ . The beam search does not contain any neural network
computations and can be implemented efficiently in C++ [3] .
3https://github.com/parlance/ctcdecode
Table 1: Performance comparison between baselines and our methods on WMT14 En-De test set.


Transformer 27.54 54.38 1.0 _×_
Base
Vanilla-NAT 19.32 45.79 15.5 _×_
SCTC 25.26 52.06
SCTC
+bipartite 26.13 53.17
+1-gram 26.22 53.20
+2-gram **26.79** 53.54
+3-gram 26.66 53.48
+4-gram 26.62 53.43
14.7 _×_
CTC 26.34 53.15
CTC +1-gram 27.16 53.95 14.7 _×_
+2-gram **27.57** **54.50**
Table 2: Performance comparison between our models and existing methods. The speedup is
measured on WMT14 En-De test set with batch size 1. AT means autoregressive. **Iter.** denotes the
number of iterations at inference time. ‘–’ indicates that the result is not reported. ‘12-1’ means the
Transformer with 12 encoder layers and 1 decoder layer [24].


Transformer (teacher) N 1.0 _×_ 27.54 **31.57** **34.26** **33.87**
AT Transformer (12-1) N 2.6 _×_ 26.09 30.30 32.76 32.39
+ KD N 2.7 _×_ **27.61** 31.48 33.43 33.50
NAT-FT [17] 1 15.6 _×_ 17.69 21.47 27.29 29.06
NAT-REG [48] 1 27.6 _×_ 20.65 24.77         -         AXE [13] 1         - 23.53 27.90 30.75 31.54
GLAT [33] 1 15.3 _×_ 25.21 29.84 31.19 32.04
Non-CTC Seq-NAT [42] 1 15.6 _×_ 25.54 29.91 31.69 31.78
NAT AligNART [45] 1 13.4 _×_ 26.40 30.40 32.50 33.10
CMLM [12] 10        - 27.03 30.53 33.08 33.31
LevT [18] 2.05 4.0 _×_ 27.27          -          - 33.26
JM-NAT [19] 10         - 27.69 **32.24** 33.52 33.72
RewriteNAT [11] 2.70          - **27.83** 31.52 **33.63** **34.09**
CMLMC [22] 10        - 28.37 31.41 34.57 34.13
CTC [29] 1          - 16.56 18.64 19.54 24.67
Imputer [36] 1           - 25.80 28.40 32.30 31.70
GLAT+CTC [33] 1 14.6 _×_ 26.39 29.54 32.79 33.84
CTC-based REDER [49] 1 15.5 _×_ 26.70 30.68 33.10 33.23
NAT + beam&reranking 1 5.5 _×_ 27.36 31.10 33.60 34.03
DSLP [21] 1 14.8 _×_ 27.02 31.61 34.17 **34.60**
Fully-NAT [16] 1 16.8 _×_ 27.20 31.39 33.71 34.16
Imputer [36] 8           - **28.20** **31.80** **34.40** 34.10
Bag-of-ngrams [41] 1 15.5 _×_ 25.28 29.66 31.37 31.51
Our + rescore 5 candidates 1 9.5 _×_ 25.95 30.43 32.78 32.92
Implementation OAXE [9] 1 15.3 _×_ 24.70 29.30 31.06 31.37
+ rescore 5 candidates 1 14.0 _×_ 25.78 30.29 32.45 32.63
Our work
CTC w/o finetune 1 14.7 _×_ 26.34 29.58 33.45 33.32
CTC w/ NMLA 1 14.7 _×_ 27.57 31.28 33.86 33.94
+ beam&lm 1 5.0 _×_ **28.35** **32.27** **34.72** **34.95**
DDRS w/o finetune 1 14.7 _×_ 27.18 30.91 34.42 34.31
DDRS w/ NMLA 1 14.7 _×_ 28.02 31.80 34.73 34.76
+ beam&lm 1 5.0 _×_ **28.63** **32.65** **35.51** **35.85**


We first compare the performance of different non-monotonic training objectives for latent alignment
models. Baseline models include the Transformer, vanilla-NAT, SCTC, and CTC, and we finetune
latent alignment models with bipartite matching and n-gram matching objectives. In Table 1, we
report BLEU and METEOR scores along with the speedup on WMT14 En-De test set.
From Table 1, we have the following observations: (1) Non-monotonic training objectives can
effectively improve the performance of latent alignment models, which improves SCTC by 1.53
BLEU and 1.48 METEOR and improves CTC by 1.23 BLEU and 1.35 METEOR, illustrating the
importance of non-monotonic alignments. (2) Despite having the same expressive power, SCTC
underperforms CTC in the task of non-autoregressive translation. We speculate that it is because
of the difference in the distribution of the alignment space. Most target sentences do not contain
repeated words, in which case the alignment space of SCTC _βs_ _[−]_ [1][(] _[y]_ [)][ is a subset of] _[ β][−]_ [1][(] _[y]_ [)][, making]
the ability of SCTC relatively weak. (3) SCTC achieves the best performance with 2-gram finetuning,
and then the performance decreases with the increase of n due to the sparsity of higher rank n-grams.
It suggests that we do not need a large n and relieves the concern on CTC that we cannot calculate
n-gram matching for _n >_ 2. (4) The correlation between the reported BLEU and METEOR scores is
very strong, suggesting that BLEU is not biased when evaluating our methods. Therefore, we can
safely use BLEU to evaluate the translation quality in the following experiments.
We call the 2-gram matching objective NMLA since it represents our best non-monotonic training
method for latent alignment models. Besides CTC, we also apply NMLA finetuning on a stronger
baseline DDRS [43] [4], which trains the CTC model with multiple references. To extend our method
to multiple references, we calculate the n-gram counts for each reference and use their average
as the target n-gram count. We compare their performance with existing approaches in Table 2.
Notably, CTC with NMLA achieves comparable performance to autoregressive Transformer on all
benchmarks, demonstrating the effectiveness of our method. On the strong baseline DDRS, NMLA
finetuning also brings remarkable improvements and outperforms autoregressive baselines. With
the help of beam search and 4-gram language model, NMLA even outperforms other iterative NAT
models and the autoregressive Transformer with only one-iteration parallel decoding. Compared to
our implementations of prior works that explore non-monotonic matching for NAT [41, 9], we can see
that they underperform the CTC baseline even when rescoring 5 candidates, and their speedup after
rescoring also lays behind CTC. It demonstrates the importance of exploring non-monotonic matching
on latent alignment models, which is more desirable for its flexibility of generating predictions with
variable lengths.
As NMLA finetuning brings remarkable improvements over the strong baseline DDRS, we further
conduct experiments to verify whether the NMLA is orthogonal to other methods. In Table 3, we
reimplement GLAT+CTC (Fully-NAT) [16, 33] based on their open-source code [5] and finetune it with
the NMLA objective. Our implementation replaces the SoftCopy mechanism with UniformCopy,
which surprisingly improves 0.7 BLEU and achieves 27.07 BLEU on WMT14 En-De test set.
Finetuning with NMLA can further improve its performance to 27.75 BLEU, demonstrating the
extensibility of NMLA. In Table 4, we apply NMLA on the big version of DDRS to explore its
capability on large models. The results show that the performance of DDRS-big can be greatly
boosted by NMLA finetuning. Combined with beam search, it achieves 30.06 BLEU on WMT14
En-De, which narrows the performance gap between non-autoregressive and autoregressive models
to a new level.
Table 3: BLEU scores of GLAT+CTC and
NMLA finetuning on WMT14 En-De test set.


GLAT+CTC w/o finetune 27.07 14.7 _×_
GLAT+CTC w/ NMLA 27.75 14.7 _×_
+ beam&lm 28.46 5.0 _×_
4https://github.com/ictnlp/DDRS-NAT
5https://github.com/FLC777/GLAT
Table 4: BLEU scores of DDRS-big and
NMLA finetuning on WMT14 En-De test set.


DDRS-big w/o finetune 28.24 14.1 _×_
DDRS-big w/ NMLA 29.11 14.1 _×_
+ beam&lm 30.06 4.8 _×_


In this section, we will analyze the training cost and show that our method is cost-effective. The
additional training cost of our method only comes from the NMLA finetuning. In Table 5, we compare
the training cost of CTC and NMLA on WMT14 En-De. We measure the training speed by Words Per
Second (WPS) and find that the training speed of NMLA is slower than CTC, which can be attributed
to the large calculation cost of the NMLA objective. However, since we only finetune the model for
6K steps, the overall cost of the NMLA finetuning is much smaller than the CTC pretraining. On the
whole, the training cost of CTC with NMLA is less than 1 _._ 2 _×_ compared to the CTC baseline.
Table 5: The comparison of CTC pretraining and NMLA finetuning. ’WPS’ means words per second.
’Step’ means the number of training steps. ’Batch’ means batch size. ’Time’ means the training time
measured on 8 GeForce RTX 3090 GPUs.
WPS Step Batch Time
CTC pretraining 141K 300K 64K 26.4h
NMLA finetuning 60K 6K 256K 5.0h


[17] first proposes non-autoregressive machine translation to accelerate the decoding process of NMT.
The cross-entropy loss is applied to train the Vanilla-NAT, which requires the explicit alignment
between the model output and target sentence. The explicit alignment cannot be guaranteed due to
the multi-modality problem, and many efforts have been devoted to mitigating this issue.
One research direction is to enable explicit alignment by leaking part of the target-side information
during the training. [17] uses fertilities as a latent variable to specify the number of output words
corresponding to each input word. [23, 35, 4] apply vector quantization to train NAT with discrete
latent variables, and [30, 44, 16] apply variational inference to train NAT with continuous latent
variables. [34, 3, 45] also introduce latent variables that carry the reordering information. Besides
using latent variables, [12] directly feeds the partially masked target sentence as the decoder input,
and [33] applies a glancing sampling strategy to control the mask ratio.
Another research direction is to propose better training objectives for NAT. [48] introduces two
auxiliary regularization terms to reduce errors of repeated translations and incomplete translations.
[40] directly optimizes sequence-level objectives with reinforcement learning. [21] trains NAT with
deep supervision and feeds additional layer-wise predictions. [43] creates a dataset with multiple
references and dynamically selects an appropriate reference for the training. Prior to this work,
[41, 13, 9] have explored ways to establish an alignment between the model output and target
sentence, which can promote the training of NAT. Their limitation is that they are basically built on
the framework of Vanilla-NAT, which requires a target length predictor and cannot dynamically adjust
the output sequence length. CTC-based latent alignment models [29, 36, 16] are more desirable
due to their superior performance and the flexibility of generating predictions with variable lengths.
However, latent alignment models have a weakness in handling non-monotonic alignments, which is
noticed in previous work [36, 16] and solved in this work.


Latent alignment models cannot handle non-monotonic alignments during the training, which is
nonnegligible in machine translation. In this paper, we explore non-monotonic latent alignments
for NAT and propose two matching objectives named bipartite matching and n-gram matching.
Experiments show that our method achieves strong improvements on latent alignment models.


We thank the anonymous reviewers for their insightful comments. This work was supported by
National Key R&D Program of China (NO.2018AAA0102502).


1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they
were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation? [N/A]


Equation 13 and and Equation 15 are the special cases of Equation 16 when n is 1 or 2, so we directly
provide the proof of Equation 16:
_Ta_
- _pi_ 1( _g_ 1 _|x, θ_ ) _×_
_in_ =1
_Ta_
- _. . ._
_i_ 2=1
_Cg_ _[s]_ [(] _[θ]_ [) =]
_Ta_
_i_ 1=1
_n_
- _Aik−_ 1 _,ik_ _pik_ ( _gk|x, θ_ ) _, g ∈_ _Gn._
_k_ =2
We begin with the definition of _Cg_ _[s]_ [(] _[θ]_ [)][, which accumulates the n-gram] _[ g]_ [ from all alignments:]
_Cg_ _[s]_ [(] _[θ]_ [) =]            - _p_ ( _a|x, θ_ ) _Cg_ ( _βs_ _[−]_ [1][(] _[a]_ [))] _[,]_
_a∈Y_ _[∗]_
where _Cg_ ( _βs_ _[−]_ [1][(] _[a]_ [))][ is the occurrence count of n-gram g in the collapsed sentence] _[ β]_ _s_ _[−]_ [1][(] _[a]_ [)][. We use]
_i_ 1: _n_ = _{i_ 1 _, ..., in}_ to denote the possible position of _g_ = _{g_ 1 _, ..., gn} ∈_ _Gn_ . There is an n-gram
_g ∈_ _Gn_ in the position _i_ 1: _n_ if and only if _aik_ = _gk_ for all _k ∈{_ 1 _, ..., n}_ and all other words between
them are blank tokens. By traversing all possible positions, we can obtain the n-gram count as
follows:
_Ta_
1( _ai_ 1 = _g_ 1) _×_
_in_ =1
1( _ik−_ 1 _< ik, aik_ = _gk, aik−_ 1+1: _ik−_ 1 = _ϵ_ ) _, g ∈_ _Gn._
_k_ =2
_n_
_Cg_ ( _βs_ _[−]_ [1][(] _[a]_ [)) =]
_Ta_
_i_ 1=1
_Ta_
- _. . ._
_i_ 2=1
_Ta_
where 1( _·_ ) is the indicator function that takes value 1 when the inside condition holds, otherwise it
takes value 0. Recall that _Aij_ is the probability that all positions between _i_ and _j_ are blank tokens,
which satisfies the following equation:
_Aik−_ 1 _,ik_ = _p_ ( _ik−_ 1 _< ik, aik−_ 1+1: _ik−_ 1 = _ϵ|x, θ_ ) _._
Using the above equations, we can rewrite _Cg_ _[s]_ [(] _[θ]_ [)][ as:]
_Cg_ _[s]_ [(] _[θ]_ [) =] - _p_ ( _a|x, θ_ ) _Cg_ ( _βs_ _[−]_ [1][(] _[a]_ [))]
_a_
_Ta_
- _. . ._
_i_ 2=1
_Ta_
- _. . ._
_i_ 2=1
_Ta_
_in_ =1
_n_
- 1( _ik−_ 1 _< ik, aik_ = _gk, aik−_ 1+1: _ik−_ 1 = _ϵ_ )
_k_ =2
=
=
=
_Ta_
_i_ 1=1
_Ta_
_i_ 1=1
_Ta_
_i_ 1=1
- _p_ ( _a|x, θ_ )1( _ai_ 1 = _g_ 1) _×_
_a_
_n_
- _Aik−_ 1 _,ik_ _pik_ ( _gk|x, θ_ ) _, g ∈_ _Gn._
_k_ =2
_Ta_
- _. . ._
_i_ 2=1
_Ta_
- _p_ ( _ai_ 1 = _g_ 1 _|x, θ_ ) _×_
_in_ =1
_n_
- _p_ ( _aik_ = _gk|x, θ_ ) _p_ ( _ik−_ 1 _< ik, aik−_ 1+1: _ik−_ 1 = _ϵ_ )
_k_ =2
_Ta_
- _pi_ 1( _g_ 1 _|x, θ_ ) _×_
_in_ =1

A Proof of Equation 13, Equation 15, and Equation 16


Here we give the _O_ ( _nTa_ [2][)][ algorithm to calculate] _[ C]_ _g_ _[s]_ [(] _[θ]_ [)][ according to Equation 16. Notice that]
Equation 16 has a similar form with the n-step transition probability of a Markov chain, so we can
try a similar method as calculating the transition probability for a Markov chain. Specifically, we
introduce a set of state vectors _sk_, where _k ∈{_ 1 _,_ 2 _, ..., Ta}_ and the dimension of each vector _sk_ is
_Ta_ . We define the state vectors as follows:
_sk_ ( _ik_ ) =
_pik_ ( _gk|x, θ_ ) _,_ _k_ = 1
 - _Tika−_ 1=1 _[s][k][−]_ [1][(] _[i][k][−]_ [1][)] _[A][i]_ _k−_ 1 _[,i]_ _k_ _[p][i]_ _k_ [(] _[g][k][|][x, θ]_ [)] _[,]_ _k >_ 1 _[.]_
From the above equation, the time complexity to calculate _sk_ given _sk−_ 1 is _O_ ( _Ta_ [2][)][ and can be]
efficiently calculated in matrix form. Therefore, the time complexity to sequentially calculate _s_ 1: _n_ is
_O_ ( _nTa_ [2][)][. Finally, we can easily obtain] _[ C]_ _g_ _[s]_ [(] _[θ]_ [)][ by summing the last vector] _[ s][n]_ [:]
_Ta_
- _. . ._
_i_ 2=1
_Ta_
- _pi_ 1( _g_ 1 _|x, θ_ ) _×_
_in_ =1
_Cg_ _[s]_ [(] _[θ]_ [) =]
=
=
_Ta_
_i_ 1=1
_Ta_
- _. . ._
_in_ =1
_Ta_
- _. . ._
_in_ =1
_Ta_
_i_ 2=1
_Ta_
_i_ 3=1
_n_
- _Aik−_ 1 _,ik_ _pik_ ( _gk|x, θ_ )
_k_ =3
_n_
- _Aik−_ 1 _,ik_ _pik_ ( _gk|x, θ_ )
_k_ =2
_Ta_
- _s_ 1( _i_ 1)
_i_ 1=1
_Ta_
- _s_ 2( _i_ 2)
_i_ 2=1
_n_
- _Aik−_ 1 _,ik_ _pik_ ( _gk|x, θ_ )
_k_ =2
= _......_
_Ta_
- _sn−_ 1( _in−_ 1) _× Ain−_ 1 _,inpin_ ( _gn|x, θ_ )
_in−_ 1=1
=
=
_Ta_
_in_ =1
_Ta_
- _sn_ ( _in_ ) _, g ∈_ _Gn._
_in_ =1

B The _O_ ( _nTa_ [2][)] [ algorithm for Equation 16]


Here we provide the proof of Equation 20:
_Rg_ ( _θ_ ) = _Cg_ _[s]_ [(] _[θ]_ [)] _[ −]_ _[C][g]_ [(] _[θ]_ [) =]
_Ta−_ 1
- _pt_ ( _g_ 1 _|x, θ_ ) _pt_ +1( _g_ 1 _|x, θ_ ) _, g ∈_ _G_ 1 _,_
_t_ =1
which is equivalent to proving the following equation:
_Cg_ ( _θ_ ) = _p_ 1( _g_ 1 _|x, θ_ ) +
_Ta_
- _pt_ ( _g_ 1 _|x, θ_ )(1 _−_ _pt−_ 1( _g_ 1 _|x, θ_ )) _, g ∈_ _G_ 1 _._
_t_ =2
We begin with the definition of _Cg_ ( _θ_ ), which accumulates the n-gram _g_ from all alignments:
_Cg_ ( _θ_ ) =            - _p_ ( _a|x, θ_ ) _Cg_ ( _β_ _[−]_ [1] ( _a_ )) _._
_a∈Y_ _[∗]_
_Cg_ ( _β_ _[−]_ [1] ( _a_ )) is the occurrence count of n-gram _g_ in the collapsed sentence _β_ _[−]_ [1] ( _a_ ). Besides blank
tokens, repeated words will also be removed by _β_ _[−]_ [1] . Therefore, we call a word in the alignment a
valid word when it does not repeat the word before it. We can calculate the 1-gram count by collecting
all valid words, which will not be removed by the collapsing function:
_Cg_ ( _β_ _[−]_ [1] ( _a_ )) = 1( _a_ 1 = _g_ 1) +
_Ta_
- 1( _at_ = _g_ 1 _, at−_ 1 _̸_ = _g_ 1) _, g ∈_ _G_ 1 _._
_t_ =2
Using the above equation, we can rewrite _Cg_ ( _θ_ ) as:
_Cg_ ( _θ_ ) =   - _p_ ( _a|x, θ_ ) _Cg_ ( _β_ _[−]_ [1] ( _a_ )) = _p_ ( _a_ 1 = _g_ 1 _|x, θ_ ) +
_a_
_Ta_
- _p_ ( _at_ = _g_ 1 _, at−_ 1 _̸_ = _g_ 1 _|x, θ_ )
_t_ =2
= _p_ 1( _g_ 1 _|x, θ_ ) +


_Ta_
- _pt_ ( _g_ 1 _|x, θ_ )(1 _−_ _pt−_ 1( _g_ 1 _|x, θ_ )) _, g ∈_ _G_ 1 _._
_t_ =2
To better understand our method, we present a qualitative analysis to provide some insights into the
improvements of NMLA. We analyze the generated outputs of the WMT14 En-De test set in the
following experiments.
**NMLA Handles Long Sequences Better** We first investigate the performance of AT and NAT
models for different sequence lengths. We split the test set into different buckets based on target
sequence length and calculate BLEU for each bucket. We report the results in Table 6. The
first observation is that the performance of Vanilla-NAT drops drastically as the sequence length
increases. It is a well-known property of NAT and can be explained as the increased probability of
observing misalignment between model output and target as the target sequence becomes longer. The
misalignment in long sequences causes the inaccuracy of loss and therefore harms the translation
quality. We have a similar observation when comparing CTC with NMLA. The performance of CTC
is close to NMLA on short sequences, but the gap becomes larger as the sequence length increases.
Our explanation is that the longer the sequence, the more likely the monotonic assumption fails due
to the global word reordering, which causes the inaccuracy of loss and therefore harms the translation
quality. NMLA models non-monotonic alignments and alleviates the inaccuracy of loss on long
sequences, so it achieves strong improvements on long sequences and even outperforms AT by more
than 2 BLEU, which suffers from the error accumulation on long sequences.
Table 6: The performance of AT and NAT models with respect to the target sequence length. ‘Vanilla’
means Vanilla-NAT. ‘N’ denotes the target sequence length. ‘AT’ means autoregressive Transformer.


1 _≤_ _N <_ 20 21.27 24.83 25.68 25.55
20 _≤_ _N <_ 40 19.49 26.81 28.05 27.82
40 _≤_ _N <_ 60 16.21 26.54 26.71 27.74
60 _≤_ _N_ 10.69 26.69 26.44 28.89
All 19.32 26.34 27.54 27.57
**NMLA Improves Model Confidence** NAT suffers from the multi-modality problem, which makes
the model consider many possible translations at the same time and become less confident in
generating outputs. We investigate whether our method alleviates the multi-modality problem by
measuring how confident the model is during the decoding. We use the information entropy to
measure the confidence, which is calculated as _H_ ( _X_ ) = _−_ [�] _x∈X_ _[p]_ [(] _[x]_ [) log] _[ p]_ [(] _[x]_ [)][, and lower entropy]
indicates higher confidence. We calculate the entropy in every position when decoding the WMT14
En-De test set and use the average entropy to represent the model confidence. We do not report the
entropy of AT since it varies with the decoding algorithm. The average entropy scores of NAT models
are reported in Table 7. We can see that the Vanilla-NAT has low prediction confidence, which is
substantially improved by CTC. NMLA further reduces the average entropy to 0 _._ 061, showing that
the model confidence can be improved by modeling non-monotonic latent alignments.
Table 7: The average prediction entropy of NAT models. ‘Vanilla’ means Vanilla-NAT.


**Entropy** 2.098 0.260 0.061
**NMLA Improves Generation Fluency** Since NAT generates target words independently, generation
fluency is a main weakness of NAT. The multi-modality problem makes NAT consider many possible
translations at the same time and generate an inconsistent output, which further reduces the generation
fluency. We investigate whether our method improves the generation fluency with an n-gram language
model [20] trained on the target side of WMT14 En-De training data. We use the language model
to calculate the perplexity scores of model outputs and report the results in Table 8. We can see
that Vanilla-NAT and CTC have high perplexity which indicates low fluency. NMLA substantially
reduces the perplexity score, which is only 42.6 higher than the gold reference. For AT, its perplexity
score is even lower than the gold reference, which is consistent with prior works that autoregressive
NMT generally improves fluency at the cost of adequacy [8, 10].
Table 8: The perplexity scores of AT and NAT models on WMT14 En-De test set.


For CTC-based models including NMLA, a common concern is that they require a larger number
of calculations than the autoregressive model due to the longer decoder length. Though there is a
significant speedup when decoding sentence by sentence, this advantage may disappear when we use
a larger decoding batch. In response to this concern, we measure the decoding speedup on WMT14
En-De test set under different batch sizes and report both BLEU and speedup in Figure 3. Under the
memory limit, the maximum batch size is 400. From Figure 3, NMLA still maintains 2 _._ 4 _×_ speedup
under the maximum batch size while achieving comparable performance to the autoregressive model.
For NMLA+beam&lm, it simultaneously achieves superior translation quality and speed to the
autoregressive model, which addresses the concern on batch speedup for CTC-based models.
Figure 3: BLEU and speedup of NAT and AT models measured under different batch sizes on
WMT14 En-De test set.


In Table 1, we only finetune the baseline with different n-gram matching objectives respectively. Here
we combine different n-gram matching objectives to see whether the combination can improve the
model performance. First, we linearly combine 1-gram and 2-gram matching with a hyperparameter
_α_ :
_L_ ( _θ_ ) = _α · L_ 1( _θ_ ) + (1 _−_ _α_ ) _· L_ 2( _θ_ ) _,_ _L_ _[s]_ ( _θ_ ) = _α · L_ _[s]_ 1 [(] _[θ]_ [) + (1] _[ −]_ _[α]_ [)] _[ · L]_ 2 _[s]_ [(] _[θ]_ [)] _[.]_
We train CTC and SCTC baselines with different _α_ and report the results in Table 9, which shows
that the combination of 1-gram and 2-gram underperforms simply using 2-gram matching.
Table 9: BLEU scores of SCTC and CTC under different _α_ on WMT14 En-De validation set.
_α_ 0 0.25 0.5 0.75 1
SCTC 25.09 24.98 24.85 24.70 24.54
CTC 25.90 25.74 25.66 25.51 25.42
As _n >_ 2 is allowed in SCTC, we further try to combine higher rank n-grams. We consider
_n ∈{_ 1 _,_ 2 _,_ 3 _,_ 4 _}_ and combine them with arithmetic average and geometric average respectively. The
BLEU scores of arithmetic average and geometric average are respectively 24 _._ 94 BLEU and 24 _._ 88
BLEU on WMT14 En-De validation set, which also underperforms simply using 2-gram matching.
Therefore, we conclude that simply combining different n-gram matching objectives cannot improve
the model performance.


_Fernando L´opez_ [1] _[,]_ [2] _, Jordi Luque_ [1]
1Telef´onica I+D
2Universidad Aut´onoma de Madrid
wiliam.lopezgavilanez@telefonica.com, jordi.luque@telefonica.com

Iterative pseudo-forced alignment by acoustic CTC loss for
  self-supervised ASR domain adaptation


High-quality data labeling from specific domains is costly
and human time-consuming. In this work, we propose a selfsupervised domain adaptation method, based upon an iterative
pseudo-forced alignment algorithm. The produced alignments
are employed to customize an end-to-end Automatic Speech
Recognition (ASR) and iteratively refined. The algorithm is fed
with frame-wise character posteriors produced by a seed ASR,
trained with out-of-domain data, and optimized throughout a
Connectionist Temporal Classification (CTC) loss. The alignments are computed iteratively upon a corpus of broadcast TV.
The process is repeated by reducing the quantity of text to be
aligned or expanding the alignment window until finding the
best possible audio-text alignment. The starting timestamps,
or temporal anchors, are produced uniquely based on the confidence score of the last aligned utterance. This score is computed
with the paths of the CTC-alignment matrix. With this methodology, no human-revised text references are required. Alignments from long audio files with low-quality transcriptions, like
TV captions, are filtered out by confidence score and ready for
further ASR adaptation. The obtained results, on both the Spanish RTVE2022 and CommonVoice databases, underpin the feasibility of using CTC-based systems to perform: highly accurate
audio-text alignments, domain adaptation and semi-supervised
training of end-to-end ASR.
**Index Terms** : Forced-alignment, Iterative, Automatic Speech
Recognition, Connectionist Temporal Classification, Speech
Segmentation, Closed-Captions


The quality and quantity of data required to train ASR systems
are fundamental. Even more in end-to-end ASR systems, which
require more data than hybrid Deep Neural Network (DNN)
- Hidden Markov Models (HMM) to learn acoustic representations [1]. When it comes to medium and low-resource languages, the maximum exploitation of data is sought, given that
manual annotations are costly and time-consuming. A common
technique is to retrieve audio-to-text alignments from available
audio data that has text references from the internet or TV
broadcast (e.g. conferences, subtitles). Some alignment methods only work under strict conditions as human-revised transcriptions [2]. Nevertheless, in the major part of the data available, the transcription does not exactly match the spoken content, depending on the method that has been used to generate the
text (e.g. respeaking for subtitling). Therefore, other methods
have explored alignment retrieval with low-quality text references [3, 4, 5, 1]. Some of these systems use a post-filtering
process based on a confidence score to overcome mistaken references. Additionally, there is work with unaligned speech and
text but requires word-level segmentation [6].
More recently, the use of non-labeled data has been explored. Either in semi-supervised approaches, with pseudolabeling of unlabeled audio [7]. Or with self-supervised approaches, learning speech representations that are subsequently
used in a concrete task (supervised), requiring less quantity of
labeled data to achieve state-of-the-art results [8, 9, 10].
In this work, we train and adapt to the TV broadcast domain
an ASR in the Spanish language. Firstly, we fine-tune, only
acoustically by a CTC loss, a model that was formerly trained
with non-labeled data from several languages. Then, we use the
trained ASR to perform utterance-level alignments on a corpus
comprised of Spanish TV broadcast data. For the alignments,
we developed an anchor-based algorithm that generates pseudoforced alignments with an iterative approach. The aligned data
is used to continue training the model. Hence, we propose a
framework that consists of a multi-step training process that iteratively improves the former ASR performance and adapts it,
step by step, to the specific broadcast TV acoustics and language domain.


Recent tools and challenges have promoted the improvement of
Spanish ASR. To begin with, Hugging Face’s Robust Speech
Recognition Challenge sought improvement in ASR systems in
more than 50 languages. The participants used a fine-tuned version of Wav2Vec2-XLS-R (300M, 1B, or 2B of parameters)
with the CommonVoice database, and decoding based on an
n-gram language model (LM). Additionally, any pre and postprocessing system, such as denoising mechanisms or spelling
correctors, was allowed. The winner in the Spanish language
achieved a Word Error Rate (WER) of 8.82% in the test set
of CommonVoice v6.0, reducing it to 6.27% with an LM [11].
Moreover, the Speech-To-Text Albayzin Evaluation consists of
automatic transcription for Spanish TV shows. The generated
transcripts are compared with human revised transcriptions. In
the previous editions, the Machine Learning and Language Processing (MLLP) research group achieved the best results, in
2021 a 16.06% WER in the test split of RTVE2020DB [12].


Audio-to-text alignments enrich transcriptions with temporal
references to the spoken content. They are required in fields
such as closed-captioning [13], audiobooks or database generation for speech technologies [14, 2]. Traditionally, alignments
have been performed with HMM-based models and the Viterbi
algorithm [15, 3, 2, 14]. Frameworks as sphinx [16] and MAUS
[17] are based on these technologies. This approach presents
many inconveniences [15, 1, 13], such as the assumption that
the transcription contains the exactly spoken utterance, problems with out-of-vocabulary words, problems with long audio
tracks as the Viterbi algorithm fails to scale well with speech
length; and Viterbi losses may lead to a complete misaligned
file due to incapacity to recover.
Recently, in [1], a segmentation mechanism using CTCbased end-to-end networks was proposed. It shows better results for utterance-alignment in German, compared with different tools as MAUS (based in HTK [18]) or Gentle (based in
Kaldi [19]). Our work is based on this alignment mechanism so
it will be described in detail in section 3.1.


The algorithm proposed in [1] requires an end-to-end ASR
trained with already aligned data and optimized with a CTC
loss. Having a _T_ -length acoustic observation _**x**_ _t_, associated
with a text reference of length _M −_ 2. A special token is
added at the beginning of the sequence and the blank token
added at the end of the sequence. Let’s consider this _M_ -length
sequence of characters to be aligned, which indexes are represented by _j ∈_ [1 _, M_ ], and a discrete sequence of acoustic
frames _a_ = _{a_ 1 _, a_ 2 _, ..., aT }_ within the interval _t ∈_ [1 _, T_ ]. The
alignment task is defined as the assigning of the corresponding
character index _j_ to each temporal element _at_ within the _a_ sequence. The alignment works as follows:
1. The network generates frame-wise character posteriors
_P_ ( _c|_ _**x**_ _t_ ) that are used to calculate the maximum joint probabilities ( _kt,j_ ). A trellis diagram is drawn by taking the most
probable of two possible transitions: the next character in reference text or the blank symbol (∅). The transition cost for the
first character in text reference is set to zero, allowing arbitrary
start of the sequence in any frame.
2. Character-wise alignments ( _at_ ) are calculated with backtracking. The last character alignment, which is the start of the
algorithm, is selected by taking the most probable temporal position of the character. For the rest of the alignments, the most
probable transitions determine the alignments.
Going deeper, the joint probabilities are defined as follows:
fragment is represented by _l ∈_ [1 _, T/L_ ], and is obtained by:
_ml_ = [1]
_L_
( _l_ +1) _L_
- _ρt._ (3)
_t_ = _lL_
Finally, the utterance alignment confidence score _sseg_ is
given by selecting the worst fragment score, this inflicts a
penalty in mismatches between ASR results and text references.
_sseg_ = min _ml,_ (4)
_l_


The alignment method presented in section 3.1 cannot be used
directly in our case for several reasons. As we pretend to align
long audio files using a transformer-based ASR, we found the
necessity to process chunks instead of the whole audio. In addition, we noticed that even human-revised transcripts present
differences with the spoken content. Thus, we propose an iterative chunk processing, where several combinations of audio
and text are aligned until finding the best possible match. Our
proposal is implemented as follows:
1. Pre-process audio with a Voice Activity Detector (VAD), removing only non-speech segments longer than 30 seconds. Typically opening sequences of shows and sections.
2. Split the reference text in utterances with a maximum length
of 24 words, as this is the maximum word sequence in CommonVoice.
3. Calculate initial time references based on total text length
and total speech time. We assign an audio duration proportional
to text length. This assumes constant speech velocity but is only
used to have initial time references.
4. Read audio and utterances from the last temporal anchor
point. In the first step, the previous anchor point is defined
by the first voice event in the audio file. As the algorithm
progresses, new anchor points will be defined by the accepted
alignments.
5. Perform iterative alignments with a fixed quantity of audio
and a variable amount of text. First, get frame-wise character posteriors for the acoustic observation with the ASR. Then,
_kt,j_ =
max( _kt−_ 1 _,j · P_ (∅ _|_ _**x**_ _t_ ) _,_
 _kt−_ 1 _,j−_ 1 _· P_ ( _cj|_ _**x**_ _t_ )) if _t >_ 0 _∧_ _j >_ 0
0 _,_ if _t_ = 0 _∧_ _j >_ 0
1 _,_ if _t_ = _j_ = 0
(1)
Considering that the most probable temporal position of
character _M_ is _ψ_ = arg max _t kt,M_, alignments are obtained
with:
(2)
_M −_ 1 _,_ if _ψ ≥_ arg max _t kt,M_ _−_ 1
_at_ =


_at_ +1 _,_ if _kt,at_ +1 _−_ 1 _· P_ ( _cat_ +1 _−_ 1 _|_ _**x**_ _t_ +1)
_< kt,at_ +1 _· P_ (∅ _|_ _**x**_ _t_ +1)
_at_ +1 _−_ 1 _,_ else

The algorithm produces a posterior probability ( _ρt_ ) for every aligned frame, selecting the maximum probability between
the aligned char or the blank token. So, we capture the spikes in
the temporal _ρt_ sequence. Moreover, a fragment score is computed by splitting the _T_ -length aligned utterance in segments of
length _L_ . Thus, _ml_ is the fragment score where the index of the
Figure 1: _Simplified alignment algorithm. th represents the_
_alignment threshold, W stands for the audio window length,_
_N is the number of utterances to align, and max_ ~~_w_~~ _stands for_
_the maximum allowed window until recalculate time references._
_Iterations whenever a good alignment has been found are not_
_considered._
Table 1: _Number of hours and sentences for the different splits_
_of the Spanish CommonVoice datasets._


hours N hours N hours N
6.1 236.9 161.8k 25.2 15.1k 25.9 15.1k
7.0 291.2 196.0k 25.8 15.3k 26.4 15.3k
RTVE2018DB is a collection of shows from public Spanish
Television (RTVE) during the years 2015 to 2018. It has 569
hours of unaligned audio, partitioned into 4 different subsets:
train, dev1, dev2, and test. The train split consists of 460 hours
of audio with closed captions from TV shows. The dev1, dev2,
and test splits contain 57, 15, and 41 hours of human-revised
transcripts, respectively. Whereas RTVE2020DB is a collection
of shows from RTVE during the years 2019 to 2020. It includes
a test split with 55 hours of human transcribed audio.


A purely end-to-end acoustic model has been used. Based
on the large XLS-R architecture, which consist of a convolutional feature encoder, followed by a transformer with 24 blocks
with an inner dimension of 4096 and a model dimension of
1024. Concretely, we used the XLSR-53 model, which was
pre-trained with 56k hours of audio in 53 languages [9], and
then fine-tuned in the Spanish CommonVoice dataset (version
6.1). We added two linear layers randomly initialized to the
original Wav2Vec2 architecture. The resultant model counts for
more than 300M trainable parameters. Finally, the transcription
is obtained using simple Greedy decoding from the frame-wise
character posteriors that the model produces.
Two versions of the model were built. Firstly, a model
that classifies 87 characters without punctuation, as a result of
retrieving any unique char from the CommonVoice database.
Secondly, as the Spanish language has fewer characters, we removed other languages’ utterances, characters, e.g., from the
Cyrillic alphabet. Characters coming from different alphabets
used in proper nouns are also discarded. As a result, the model
classifies among 38 characters, including unaccented letters between a-z, the accented vowels ´a, ´e, ´ı, ´o, ´u, and the dieresis on
the vowel u(¨u). The unique difference between both models is
the number of output neurons.


The 87-char model was trained during 150 epochs with CommonVoice 6.1. While the 38-char model was trained during
15 epochs with CommonVoice 7.0. For the second model, we
used the trained Wav2Vec2 weights of the first model. In the
test split of the CommonVoice 7.0, the 87-char model achieved
8.8% WER, while the 38-char model achieved 8.17% WER.
These checkpoints are our baseline experiments. Only the 38char model was adapted to the TV domain, so, hereinafter it will
be simply referred as model or ASR.


Based on the CTC recipe from the SpeechBrain toolkit [22],
utterance audio length is limited from 4 to 10 seconds. Similarly, signals are ordered by length, using shorter ones in the
first batches of the epoch, and longer clips at the end. Regarding
data augmentation, the unique technique used is SpecAugment
[23]. Furthermore, the model is optimized minimizing only a
3b
|Anchor point ut ut Candidate Aligned Bad score ut|Col2|
|---|---|
|**ca-1**
**ca-2**
**ut-1**
**ut-2**
**ca-3**
**ca-4**
**ca-5**
**ut-1**
**ut-2**
**ca-3**
**ca-4**
**ut-1**
**ut-2**
**ut-3**
**ut-4**
**ca-5**
**ca-6**
**ca-7**
**ca-8**
**ut-1**
**ut-2**
**ut-3**
**ut-4**
**ca-5**
**ca-6**
**ca-7**
**ut-1**
**ut-2**
**ut-3**
**ut-4**
**ut-5**
**ut-6**
**ut-7**
**ca-8**
**ca-9**
**ca-10**
**ut-1**
**ut-2**
**ut-3**
**ut-4**
**ut-5**
**ut-6**
**ut-7**
**ut-8**
**ut-9**
**ut-10**|**ca-1**
**ca-2**
**ut-1**
**ut-2**
**ca-3**
**ca-4**
**ca-5**
**ut-1**
**ut-2**
**ca-3**
**ca-4**
**ut-1**
**ut-2**
**ut-3**
**ut-4**
**ca-5**
**ca-6**
**ca-7**
**ca-8**
**ut-1**
**ut-2**
**ut-3**
**ut-4**
**ca-5**
**ca-6**
**ca-7**
**ut-1**
**ut-2**
**ut-3**
**ut-4**
**ut-5**
**ut-6**
**ut-7**
**ca-8**
**ca-9**
**ca-10**
**ut-1**
**ut-2**
**ut-3**
**ut-4**
**ut-5**
**ut-6**
**ut-7**
**ut-8**
**ut-9**
**ut-10**|
|**ut-1**|**ut-2**|
Figure 2: _Iterations of pseudo-forced alignment algorithm._
_Same number represents the same audio window. Similarly, dif-_
_ferent letters represent a different set of utterances under align-_
_ment. In iterations 2b and 4a, the algorithm accepts alignments_
_with sseg < threshold between anchors. R is the resultant_
_alignment, with 7 utterances with sseg above the alignment_
_threshold and 3 utterances under the alignment threshold._
perform alignments while the number of utterances is bigger or
equal to one. The confidence score of the last aligned utterance
is the unique criterion used to accept or reject the alignments.
We use a fixed threshold, _sseg_ = 0 _._ 13 (-2.0 in log space), to
select anchor points. If the last aligned utterance provides a
_sseg < threshold_, we reject alignments. Therefore, we remove an utterance from the text reference and try again. In contrast, if _sseg >_ = _threshold_ we can accept the alignments, so
we store them. Nevertheless, we continue iterations by removing utterances from text, looking for a better confident score.
This process is repeated until finding a bad alignment or until
there is no improvement. That is to say, after finding a good
alignment if we find an even better alignment, then the alignment with the best confidence score is stored. If we exhaust the
number of utterances without any acceptable alignment, we go
back to the previous step to read more audio and text.
In addition, we do not allow short utterances to be anchors. Concretely, utterances with lengths inferior to or equal
to _L_ = 30 frames (0.6 seconds) cannot be anchors. We modify
the obtained _sseg_ by decreasing its value. This is done because
of the inability of the algorithm to provide a bad alignment confident score for short utterances. A simplified version of the
algorithm is presented in Figure 1. Furthermore, as our unique
criterion to accept the alignments of a set of utterances is the
confidence score of the last aligned utterance, the algorithm allows bad alignments between anchors, thus, we call it pseudoforced. This is especially useful for low-quality transcriptions.
Figure 2 shows an example of six iterations of the algorithm.


The proposal of this work is assessed with CommonVoice [20]
as out-of-domain database and RTVE2022 [21] as target domain database. Spanish CommonVoice dataset comprises more
than 200 hours of reading speech, which is utterance-level
aligned and validated by volunteers [20]. Table 1 presents the
amount of data of the two versions used in this work.
RTVE2022DB combines the databases used in previous
evaluations: 2018 and 2020 with released data in 2022 [21].
CTC loss and the learning rates (LR) are updated using the New
Bob scheduler. Additionally, we manually restarted the LR at
some points in training. Finally, the best checkpoint in terms of
WER is stored. The ASR has been trained using a batch size of
3, setting the starting LRs for the linear layers and Wav2Vec2
of 1.0 and 10 _[−]_ [5], respectively.


Aiming to evaluate the usefulness of our iterative pseudo-forced
alignment method, we partially aligned the RTVE2018 corpus. Afterward, we adapted an out-of-domain baseline ASR
by fine-tuning it through the aligned data. From this baseline
model, we continued training it by combining out-of-domain
data with the aligned RTVE domain data. Note that the development split comprises uniquely RTVE2018. Training and
development splits of RTVE2018 have been separated by show,
avoiding speaker repetition. Concretely, the development split
comprises four of the total seventeen shows available.
Performance is evaluated with WER. For CommonVoice,
it is computed with the SpeechBrain toolkit. For RTVE2020,
transcription is generated by processing the audio file with a
sliding window without overlap, and WER is computed with
the sclite tool of the NIST Speech Recognition Scoring Toolkit.


We evaluated three different methods to filter alignments. First,
discarding utterances with _sseg < −_ 1 _._ 0 (in log space). Secondly, using the Chebyshev’s inequality for filtering out the
15% worst aligned utterances. Lastly, normalizing the confidence score by considering the utterance length and then discard alignments with _sseg < −_ 1 _._ 5 (in log space). We heuristically selected a reference length in seconds ( _U_ = 8 _._ 0) that we
consider enough to detect mismatches between reference text
and ASR. Likewise, _S_ is the aligned segment length in seconds.
Thus, the normalized segment score is calculated as follows:
_s_ _[′]_ _seg_ [=] _[s][seg][ ·][ S]_ (5)
_U_
The best results are obtained while using the first method
commented above, so we used only that filtering method. In
Table 2 the quantity of aligned and filtered data is presented.
The first pass is performed using the baseline ASR. For the second pass, the model is already trained with domain data (23
hours) during 15 epochs. Similarly, Figure 3 depicts _sseg_ distribution between the two different pass alignments, showing the
improvement as the model is adapted.
Table 2: _Data recovery, in hours, from RTVE2018DB._

5.1. Filtering methods and data recovery


Aligned Filtered Aligned Filtered
train 460 0 0 17.5 14.7
dev1 55 45 18 53.6 29.9
dev2 15 8.5 5 14.3 9.1
test 39 0 0 33.1 20.9


The domain adaptation process is presented in Figure 4 and final results are presented in Table 3. Aligned data definitely con
|0.35
0.30
0.25
0.20 density
0.15
0.10
0.05
0.00|1st pass
2nd pass
s_seg=-1.5
s_seg=-4.0|Col3|Col4|
|---|---|---|---|
|~~10~~
~~8~~
~~6~~
~~4~~
~~2~~
~~0~~
segment score (log probabilities)
||||
0 15 35 55
Epochs
Figure 4: _WER across training epochs. In epoch 35, LR was_
_restarted to 1.0 for the linear layers and_ 10 _[−]_ [5] _for Wav2Vec2._
_W stands for the length in seconds of the segmentation window_
_used to get transcriptions._
Table 3: _Baseline and final WER %. DA stands for domain_
_adaptation._


RTVE2020DB 47.43 46.98 **30.37**
CommonVoice 6.1 8.51 8.13 **7.80**
CommonVoice 7.0 8.80 8.17 **7.83**


We propose an iterative pseudo-forced alignment algorithm to
produce audio-to-text alignments using the CTC paths. This allowed us to perform self-supervised domain adaptation on ASR
by aligning a Spanish speech corpora based on broadcast TV.
Our model is a purely acoustic end-to-end model that uses simple Greedy decoding. Without any LM, we improved ASR performance over generic and domain data. Our model achieved
a 35.4% of relative WER reduction from the baseline in TV
domain data. Similarly, we achieved a 4.2% relative WER reduction in out-of-domain data. This shows the potential of the
alignment method, even with low-quality text references.
Figure 3: _Alignment segment score (sseg) of dev1 split from_
_RTVE2018 across alignments. In the second pass, the confi-_
_dence increases notoriously. In sseg_ = _−_ 4 _._ 0 _, a small peak is_
_generated due to the penalty inflicted in short utterances._
tributes to the process. Best results are obtained while using the
biggest amount of domain data. In RTVE2020 test, we reduced
WER from 46.98% to 30.37%. Similarly, in CommonVoice 7.0
we reduced from 8.17% to 7.83%. This suggests that the use of
more data, even if it is from a different domain, contributes to
ASR performance in both used databases. To conclude, it is reasonable to think that there is still a margin for improvement by
using more complex decoding, using an LM, or aligning more
data, as only the 20% of RTVE2018 has been aligned.
WER(W=20) WER(W=30) RTVE2018 training hours
50,00
45,00
40,00
35,00
30,00
25,00
80,0
60,0
40,0
20,0
0,0


Bryan Li _[∗]_
The University of Pennsylvania
_The word alignment task, despite its prominence in the era of statistical machine translation_
_(SMT), is niche and under-explored today. In this two-part tutorial, we argue for the continued_
_relevance for word alignment. The first part provides a historical background to word alignment_
_as a core component of the traditional SMT pipeline. We zero-in on GIZA++, an unsupervised,_
_statistical word aligner with surprising longevity. Jumping forward to the era of neural ma-_
_chine translation (NMT), we show how insights from word alignment inspired the attention_
_mechanism fundamental to present-day NMT. The second part shifts to a survey approach. We_
_cover neural word aligners, showing the slow but steady progress towards surpassing GIZA++_
_performance. Finally, we cover the present-day applications of word alignment, from cross-_
_lingual annotation projection, to improving translation._

Word Alignment in the Era of Deep Learning: A Tutorial


Word alignment is the task of identifying words in a source sentence that correspond
to words in a target sentence, given that the sentences are translations of each other
(see Figure 1). It is a vital component of statistical machine translation (SMT) systems;
however, in the age of modern neural machine translation (NMT), it has fallen out of
prominence as an explicitly modeled task. Still, the concept of alignment motivated
the development of the _attention_ mechanism (Bahdanau, Cho, and Bengio 2015), which
allows neural models to learn soft alignments between source and target sentences.
Attention most prominently underlies the Transformer (Vaswani et al. 2017), the current
state-of-the-art neural approach to natural language processing (NLP) problems, which
extend the concept of alignment to within the same sequence – self-attention. Still, the
explicit word alignment task has become rather niche in the present-day literature.
The goal of our work is to argue for the importance, historical and continued, of the
word alignment task. In the SMT age, word alignment was an essential preprocessing
step which allowed decomposition of translating sentences into translating phrases.
In the modern NMT age, word alignment remains as a useful, yet under-explored
task. One of its main applications is for _annotation projection_, a technique to extend
datasets and tasks cross-lingually. Other applications are to improve translation, and as
_monolingual_ word alignments to reframe NLP tasks such as text simplification. Finally,
regardless of the era, (word) alignments are innately interpretable and thus useful in
downstream analysis of a system’s predictions.
_∗_ Philadelphia, PA. Email: bryanli@seas.upenn.edu
Submission Date to arXiv: 2022-11-30
Figure 1: Example word alignment between English and Chinese. Dotted cells represent
where an alignment occurs. For example, the top left cell shows that “Australia” and “ `澳`
`洲` ” are aligned.


This tutorial aims to be broadly useful to NLP researchers at different levels. The
primary audience are those researchers who have grown up in the deep learning era
- such as early-stage PhD students – and may not as keen on the word alignment task.
For more senior researchers, we provide an overview of the word alignment field from
the 1990s to the early 2020s. We further situate the task in the context of the attentionbased models so prevalent in modern-day NLP. In both covering the basic concepts and
providing intuition on the details, we hope that our tutorial is of general interest to
anyone interested in language technology.


As stated previously, **word alignment** is the task of identifying words in a source sentence that correspond to words in a target sentence, given the sentences are translations
of each other. **Machine translation** (MT) is the task of developing systems that can
translate source language text into target language text.
On the data side, **sentence pair** consists of a source sentence and a target sentence.
A dataset of sentence pairs is called a **bitext**, or a **parallel corpus** .
Word alignments can be a) one-to-one, b) one-to-many, c) many-to-one, or d) manyto-many. Figure 1 depicts an example of a word alignment between a Chinese and
A Tutorial on Word Alignment in the Age of Deep Learning
English sentence pair. It depicts one-to-one (“Australia” to “ `澳洲` ”) and many-to-one
(“diplomatic”, “relations” to “ `邦交` ”) alignments. We hope the observant reader can
imagine examples of the other cases.
The final key concept is the title phrase “the era of deep learning.” Let us decompose
this phrase into its parts. **Deep learning** refers to machine learning methods based on
multi-layered (hence “deep”) neural networks. **Neural networks** are computer systems
inspired by the operation of neurons in the human brain. While neural networks have
existed since the mid-20th century, it was only until the 2000s that the convergence of
developments in both computational power and neural modeling approaches allowed
their successful application to vision and language problems. Thus, **the era** of deep
learning refers to 2000s to the present. In this era deep learning is the go-to approach
towards tackling problems in artificial intelligence, and statistical methods are often
seen as antiquated. Crucially though, unsupervised word alignment retains a strong
statistical baseline, GIZA++ that remains competitive with neural aligners up to the late
2010s. It is in this context that we situate our current tutorial.


The sections of this tutorial follow in a roughly chronological order. We will provide
mathematical formulations and intuitive definitions of concepts throughout. We intend
that each section is fairly self-contained, and readers can read the sections in any order.
Broadly speaking, this tutorial is structured in two parts. Sections 2 to 5 take a more
in-depth tutorial approach, whereas Sections 6 to 7 take a more comprehensive literature
survey approach.
Section 2 introduces **statistical machine translation** (SMT). We first provide mathematical formulations of the SMT task, and the relate it to the word alignment task. We
then describe the operation of Moses, typical phrase-based SMT system, at a high-level.
This will elucidate the important role word alignment plays in this pipeline.
Section 3 describe **statistical approaches to word alignment** . These are generally
unsupervised approaches. We summarize the famous IBM family of aligners and related
work. Then we zero-in on GIZA++, a notable parameterization of the IBM models.
Section 4 introduces **neural machine translation** . We focus on encoder-decoder
models, one popular approach to NMT. We show that this end-to-end formulation of
MT does not concern itself with word alignments.
Section 5 describes **the attention mechanism**, which allows neural models to focus
on parts of the input sequence when generating its output. Attention extends encoderdecoder models to perform a soft-alignment between target and source tokens. To
understand how well attention correlates to, and differs, from word alignments, we
review a few works in this direction.
Section 6 performs a comprehensive survey on **neural approaches to word align-**
**ment** . We categorize these models into three broad approaches: induction from attention, unsupervised, and guided. We also discuss work that extracts alignments from
multilingual neural language models, and also classify them by these three approaches.
Section 7 performs a broadly scoped survey on the **applications of word alignment** .
We focus on approaches more relevant to modern-day NLP, covering a few key works
from the statistical era. We find that a major use-case of word alignment is in annotation
projection, extending tasks and datasets cross-lingually. We also discuss how word
alignment can improve NMT and computer-assisted translation.


Machine translation is the task of translating a source sentence _F_ into a target sentence
_E_, where
_F_ = _f_ 1 _, ..., fm_ _E_ = _e_ 1 _, ..., en_
As a mnemonic, suppose you know English but not French—you would want to
translate source _F_ rench into target _E_ nglish. **Statistical machine translation** systems
create a model for the probability of every target sentence _E_ given some source sentence
_F_ . They find _E_ [ˆ], the hypothesis which maximizes the probability
_E_ ˆ = arg max _Pr_ ( _E|F_ ; _θ_ ) (1)
_E_
where _θ_ are the (learned) parameters of the model and capture the probability
distribution. By applying Bayes’s theorem, we can decompose this equation:
_E_ ˆ = arg max _Pr_ ( _E|F_ ; _θ_ )
_E_
= [arg max] _[E][ Pr]_ [(] _[F]_ _[|][E]_ [;] _[ θ]_ [)] _[Pr]_ [(] _[E]_ [)]
_Pr_ ( _F_ )
_E_ ˆ _∝_ arg max _Pr_ ( _F_ _|E_ ; _θ_ ) _Pr_ ( _E_ ) (2)
_E_
_Pr_ ( _F_ _|E_ ; _θ_ ), or simply _Pr_ ( _F_ _|E_ ), is the **translation model** —given this target sentence, how likely is the source sentence . _Pr_ ( _E_ ) is the **language model** (LM)—given this
target sentence, how fluent is it in the target language. SMT systems require both **bitext**
data to learn the translation model, as well as **monolingual** data in the target language
to learn the language model.
Neubig (2017) specifies the three problems a good translation system must address:
1. _Modeling_ : How will we model _P_ ( _E|F_ ; _θ_ ), what are its parameters _θ_, and
how do the parameters specify a probability distribution?
2. _Learning_ : How will we learn the parameters _θ_ from the training data?
3. _Search_ : How will we find, or **decode**, the most probable sentence?

2.1 Formalizing Statistical Machine Translation


The word alignment task was directly motivated by the SMT task. An SMT system
aims to model _Pr_ ( _F_ _|E_ ), but to do so over all tokens _Pr_ ( _f_ 1 _, ..., fm|e_ 1 _, ..., en_ ) is difficult.
Word alignments were thus introduced as a set of hidden variables _a_ = _a_ 1 _, ...am_ to make
the problem more tractable. Suppose we have a given alignment _a_ . Then we have the
A Tutorial on Word Alignment in the Age of Deep Learning
Figure 2: Example Russian to English word alignment. The mapping and relations are
respectively
_A_ = _{_ (1 _,_ 1) _,_ (2 _,_ 2) _,_ (2 _,_ 3) _,_ (3 _,_ 4) _,_ (4 _,_ 5) _,_ (5 _,_ 6) _,_ (6 _,_ 6) _,_ (7 _,_ 7) _}_
_a_ : _a_ 1 = 1 _, a_ 2 = 2 _, a_ 2 = 3 _, a_ 3 = 4 _, a_ 4 = 5 _, a_ 5 = 6 _, a_ 6 = 6 _, a_ 7 = 7
problem
_Pr_ ( _f_ 1 _, . . ., fm, a_ 1 _, . . ., am|e_ 1 _, . . ., em_ ) = _Pr_ ( _F, a|E_ )
By summing over the possible alignments, we recover the original translation
model:
     - _Pr_ ( _F, a|E_ ) = _Pr_ ( _F_ _|E_ ) (3)
_a_
Essentially, word alignment decomposes the task of translating an entire sentence into
translating parts of it.
Och and Ney (2003) present the following formalization of word alignment. Given
a source string _F_ = _f_ 1 _, ..., fj, ..., fm_ and a target string _E_ = _e_ 1 _, ..., ei, ..., en_, an alignment
_A_ is a subset of the Cartesian product of word positions:
_A ⊆{_ ( _j, i_ ) : _j_ = 1 _, ..., m_ ; _i_ = 1 _, ..., n}_ (4)
_A_ is a mapping of individual alignment points _aj_, each of which maps a source
index _j_ to a target index _i_ . We can view _a_ as a relation where _aj_ = _i_ . For example, Figure 2
gives both the mapping and relation views of word alignment.
This complete, albeit general formulation allows for an exponential search space
of alignments. In practice, alignment models often impose additional constraints to
make the problem more tractable and produce better quality alignments. One common
restriction is to enforce an injection: each source word maps to exactly one target word.
A null word _e_ 0 is added to the set of target indices to maintain this property when some
source word has no alignment. Note that this restriction preventing one-to-many and
many-to-many mapping, but this issue is often addressed by running word alignments
in both directions and combining the two.
Recall Equation 3, which shows the relationship between the translation model and
word alignment. Given a a corpus of sentence pairs ( _F_ 1 _, E_ 1) _, ...,_ ( _Fs, Es_ ), this model is
tasked to approximate the model parameters _θ_ by maximizing the probability
- _Pr_ ( _Fi, a|Ei_ ; _θ_ ) (5)
_a_
_θ_ ˆ = arg max
_θ_
_s_
_i_ =1


Let us cover some additional key concepts before moving on. **Tokenization** is the
process of splitting a text into individual tokens. Most simply, a token could be a word;
other approaches operate at the subword-level, or even the character-level. This is a
preprocessing step for all machine translation systems, statistical or neural.
A **vocabulary** is the set of words (or in our case, tokens) in a language. For machine
translation, we will have both a source vocabulary and a target vocabulary. We define
the **vocabulary size** to be the number of tokens in a given vocabulary. This is generally
much smaller than the number of valid words in a natural language. For words that fall
outside the vocabulary, NLP systems generally map them to an unknown token _<unk>_ .


We use the Moses (Koehn et al. 2007) open-source toolkit as an exemplar of a typical
SMT system. Moses is a phrase-based machine translation system; as the name implies,
this works by translating source phrases, i.e., parts of a sentence, then recombining these
target phrases into fluent target sentences. As a translation system, Moses consists of 3
parts: the language model, the translation model, and the decoder. In this section, we
describe aspects of Moses relevant to word alignment—namely the translation model.
Further details are given in Appendix A.1.
The generation process of phrase-based MT occurs in three stages:
1. _Partition_ : Given a source sentence, create all of its possible partitions,
where each partition is a set of source phrases.
2. _Lexicalization_ : Given a partition, translate its source phrases into
hypothesized target phrases.
3. _Reordering_ : Given a set of target phrases, permute it into a grammatical
sentence in the target language. This involves rearranging words and
moving them across phrases boundaries.
Word alignment is evidently relevant in the partition and lexicalization stages. Let
us now describe word alignment’s role in SMT. As a preprocessing step, we first learn
word alignments from a large parallel corpus [1] . As for how they are learned, we describe
one such word aligner, GIZA++, in Section 3.1. We apply the trained aligner to the entire
training corpus, and for each source phrase, accumulate all the target phrases [2] it maps
to. Converting counts to probabilities, we have phrase translation tables (t-tables), such
as is shown in Figure 3.
We can use the source phrase entries to _partition_ a source sentence, and _lexicalize_
given this partition into the associated target phrase(s). For any given sentence, there is
a large state space of possible partitions multiplied by each possible target translation.
1 For example, the United Nations Parallel Corpus
[(https://conferences.unite.un.org/uncorpus) is a set of parallel corpora in six languages. The](https://conferences.unite.un.org/uncorpus)
fr-en corpus consists of 25 million sentence pairs.
2 To be specific, GIZA++ enforces an injection, so will only have one-to-one and many-to-one alignments,
where one is a word and many is a phrase. But by training aligners in both directions and combining the
two, we can have all 4 types of alignments, and thus align source and target phrases.
A Tutorial on Word Alignment in the Age of Deep Learning
Figure 3: An excerpted section of a phrase translation table, where source is English
(En), and target is German (De). The probabilities are _Pr_ (En _|_ De). In this excerpt, both
source and target phrases are of length 1, but in general, entries can be longer than 1
token.
Therefore, it is the job of the Moses MT system to efficiently choose the best, or n-best,
translations. These details go beyond the present tutorial, of course.
As further evidence for the key role of word alignment to statistical machine
translation, Callison-Burch, Talbot, and Osborne (2004) find that modifying the EM
algorithm for GIZA++ to incorporate a small amount of manual word alignments, in
addition to those automatically discovered word alignments, improves both translation
and alignment performance of the downstream SMT system.


Section 2.4 shows that word alignment occupies a key stage in SMT, in that it decomposes the harder task of translating entire source sentences to the simpler task of
translating source phrases. We now describe one statistical word alignment system.

3. Statistical Approaches toward Word Alignment


GIZA++ (Och and Ney 2003) is an unsupervised word alignment tool with surprising longevity, remaining the state-of-the-art well into the 2010s, and still being cited
today. GIZA++ serves as an implementation of the IBM family of word alignment
algorithms (Brown et al. 1993). These are numbered in order of increasing complexity.
Originally, there are (IBM) Models 1-5, and Och and Ney (2003) introduces Model
6 as the default GIZA++ model. Other popular statistical word alignment tools are
fast_align (Dyer, Chahuneau, and Smith 2013), and Berkeley Aligner (Liang, Taskar, and
Klein 2006). These are discussed briefly in Appendix A.2.
GIZA++ is an unsupervised word alignment tool, which works by leveraging
language-independent statistical methods. The approach of GIZA++, and the IBM
models in general, is to view word alignment as a hidden variable in the translation
process. Statistical estimation is used to compute the “optimal” model parameters, and
alignment search is performed to compute the best word alignment. Each IBM model
models _Pr_ ( _F, a|E_ ; _θ_ ) differently.
In the remainder of this section, we’ll first walk through IBM Model 1 in detail.
We’ll then cover, at a higher-level, the additional modeling assumptions made by the
GIZA++ model.
_“Unsupervised” Word Alignment._ As an aside, unsupervised in the context of word
alignment means learning this task without any labeled word alignment data. We do
need some supervision from human annotators in collecting parallel sentences. Still, this
is much less of an involved task that finding and training annotators to perform word
alignment. Parallel sentences can be found as artifacts from international organization
proceedings, or can be written by readily available human translators.

3.1 GIZA++: A Strong Baseline for Word Alignment


IBM Model 1 is the simplest model and makes the fewest assumptions. It has a uniform
prior over the possible alignments, which means every alignment is equally plausible.
The translation probability for a given alignment _a_ is specified by
_Pr_ ( _F, a|E_ ) = _[Pr]_ [(] _[m][|][n]_ [)]
( _n_ + 1) _[m]_ _[·]_
_m_
- _Pr_ ( _Fj|Eaj_ ) _,_ (6)
_j_ =1
where _m, n_ are the lengths of _F_ and _E_ respectively.
Let us explain the constituents of this formula. The numerator, _Pr_ ( _m|n_ ) is the
probability of choosing a source length given a target length. The denominator, ( _n_ + 1) _[m]_
specifies the uniform prior over all possible alignments; it is raised to the power _m_,
as we iterate over _m_ source tokens. For each source index _j_ we have _Pr_ ( _Fj|Eaj_ ), the
probability of source token _Fj_ given its predicted aligned target word _Eaj_ . We take
the product of these probabilities to translate all source phrases, and then multiply this
product by a normalizing factor.
A system implementing IBM Model 1 learns over a large parallel corpus. For
each sentence pair, it needs to learn both a) the best alignment, and b) the translation
probabilities. This is an unsupervised approach, so both must be learned in parallel,
leading to a very natural application of the EM algorithm. We refer interested readers
to Och and Ney (2003).


The GIZA++ model, IBM Model 6, is a combination of IBM Model 4, and the HMM
aligner of Vogel, Ney, and Tillmann (1996).
_HMM Aligner._ This decomposes the alignment probability into three different probabilities: length probability, prior alignment probability, and lexicon probability. Let us focus
on the prior alignment probability. The HMM aligner uses _locality in the source language_
- when aligning a source word _fi_, it considers prior alignments _f_ 0 _, ..., fi−_ 1. In fact, we
only need to consider _fi−_ 1 given the Markov assumption that the future is independent
of the past given the present.
_IBM Model 4._ This extends Model 1 with more assumptions, namely using _fertility_ and
_locality in the target language_ . The fertility _φ_ of a target word _ei_ is simply the number of
aligned source words. The model learns the probability that _ei_ is aligned to _φ_ words.
An example of the usefulness of fertility is the German word “übermorgen,” which
translates to the four English words “the day after tomorrow” [3] .
3 This example from Och and Ney (2003) has as the source language English and the target German.
A Tutorial on Word Alignment in the Age of Deep Learning
For locality in the target language, every word depends on the previous aligned
word. A further refinement of Model 4 is in considering the word classes [4] of the
surrounding words.
In sum, Model 6 is a log-linear combination of Model 4 and an HMM aligner, to
make use of locality in both the source and target languages. As with Model 1, Model 6
is also learned through the EM algorithm, but takes quite a bit longer to train due to its
additional complexity.
To close this section, we quote Zhao and Gildea (2010), “IBM Model 4 is so complex
that most researchers use GIZA++...and IBM Model 4 itself is treated as a black box.”
Still, we hope the quick overview we have presented gives readers a sense as to the
statistical effort placed behind GIZA++.


Sections 2 and 3 have shown how the word alignment task is intricately linked statistical
machine translation. These sections have also shown how such statistical approaches
require significant _feature engineering_, designed with a high-level of statistical rigor,
and further tuned by well-trained researchers. In contrast, **neural machine translation**
(NMT) utilizes neural networks to learn sophisticated functions for language modeling,
abstracting away the hard work of feature engineering (and introducing another line of
hard work!). For a review on how neural networks work, see Appendix B.1.
In the age of deep learning, neural machine translation has almost entirely supplanted statistical machine translation. Word alignment as an explicit task is no longer
necessary, and thus research using word alignments has become rather niche.
In this section, we first describe the encoder-decoder model paradigm commonly
used in present-day NMT. Next we cover the attention mechanism, which incorporates
into this paradigm a form of alignment. We then review some work analyzing the
correlation between attention weights and word alignment, concluding while they share
some similarities, they are not interchangeable.

4. The Age of Neural Machine Translation


Here we explain a basic encoder-decoder model (Sutskever, Vinyals, and Le 2014).
Let us generalize from the machine translation problem (from _F_ to _E_ ) to a sequenceto-sequence problem (from _x_ to _y_ ). A **encoder-decoder** model consists of two neural
networks. An example encoder-decoder model is shown in Figure 4.
The encoder takes in the input, and the decoder decodes the output. Most of the
time, the encoder and decoder share very similar model architectures. The difference is
in their input and output dimensions.
An _encoder_ takes as input _x_ 1 _, ..., xj_ [5] and outputs a hidden-state representation. In
the basic model, this is a fixed-length, real-valued vector _h_ [(] _[x]_ [)] .
4 Derivation of word classes, in the Moses pipeline, are described in Appendix A.1.
5 Neural models actually operate on embeddings, which are real-valued vector representations for words.
For MT, we have both source embeddings and target embeddings. We omit this detail for simplicity of
notation, but keep it in mind.
Figure 4: A encoder-decoder model for a seq2seq task, excerpted from Ippolito (2022).
Note the two inputs to the decoder at time step _t_ : a) the encoder output, and b) the past
decoder outputs.
The _decoder_ proceeds in time steps, decoding 1 word at a time. At each _t_, it takes two
inputs a) _h_ [(] _[x]_ [)] and b) the prior predicted words ˆ _y_ 1 _, ...,_ ˆ _yt−_ 1, and outputs an probability
distribution over the target vocabulary _Pr_ ( _yt_ = _i_ ) [6] .
In mathematical notation, the decoder defines the probability of an output sequence
_y_
_Pr_ ( _y_ ) =
_j_
- _Pr_ ( _yt|{y_ 1 _, ..., yt−_ 1 _}, h_ [(] _[x]_ [)] ) (7)
_t_ =1
Given these _t_ probabilities, we would like to convert them to target tokens. We do so
using a _sampling algorithm_ . Most obviously, we can greedily select the highest probability
vocabulary item at each time step _t_, and concatenate these together. A more informed
approach is to use **beam search** . At each time step, we consider the _b_ best hypotheses,
where _b_ is the size of the beam. At the _i_ th time step our hypotheses are of length _i_ . At
each successive step, we select the next _b_ best hypotheses (word length _i_ + 1) and prune
the rest. At the last step, we pick the best of the _b_ remaining hypotheses. Figure 5 shows
a visualization of the beam search process.
An encoder-decoder model is a general paradigm that can be applied to various text
to text tasks — as we do to translation in this tutorial. The underlying neural network
architecture can be recursive neural networks (RNNs), transformers, or anything else.


Whereas Sections 2 and 3 show how statistical MT and word alignment are intrinsically
linked, Section 4 shows how neural MT methods are can perform end-to-end translation
of full sentences, without any explicit word alignments.
In this section, we guide readers through the **attention** mechanism, a method
first developed to incorporate a notion of alignment back into NMT systems. After
formalizing attention with mathematical notation, we review several works analyzing
the correlation of attention weights with word alignments. We find attention implicitly
but imperfectly models alignments, underscoring our argument towards the continued
importance of the word alignment task today.
6 The decoder actually outputs another embedding. _Pr_ ( _yt_ = _i_ ) is the softmax of the output embedding
times the target embedding matrix — i.e., how similar is each target embedding to the output.
A Tutorial on Word Alignment in the Age of Deep Learning
Figure 5: A visualization of beam search with _b_ = 2, excerpted from Neubig (2017).
The numbers labeling arrows are log probabilities transitioning between nodes, while
numbers labeling nodes are the sum of log probabilities to this node. Red ‘X‘s indicates
the pruned hypotheses at each time step.

5. Attention: Implicit (Word) Alignments?