Suppose that ( _qi, attt_ ( _qi_ )) is a pair of s2t word
representation and corresponding attention context
sampled from the joint distribution _pt_ ( _q, attt_ ( _q_ ))
(hereinafter we call it a positive pair), the primary
objective of the s2t training is to maximize the
alignment degree between the elements within a
positive pair. Thus, we first define an alignment
function by using the sigmoid inner product as:
ALIGN( _q, attt_ ( _q_ )) = _σ_ ( _⟨q, attt_ ( _q_ ) _⟩_ ) _,_ (5)
where _σ_ ( _·_ ) denotes the sigmoid function and _⟨·, ·⟩_
is the inner product operation. However, merely
optimizing the alignment of positive pairs ignores important positive-negative relation knowledge (Mikolov et al., 2013).
To make the training process more informative,
we reform the overall objective in the contrastive
learning manner (Oord et al., 2018; Saunshi et al.,
2019) with Noise Contrastive Estimation (NCE)
loss (Mikolov et al., 2013), which has been widely
used in many NLP tasks (Xiong et al., 2021; Gao
et al., 2021; Wang et al., 2022). Specifically, we
first sample _k_ negative word representations _qj_ [3]
from the margin _pt_ ( _q_ ). Then, we can formulate the
overall NCE objective as following:
_Loss_ _[i]_ _s→t_ [=] _[ −]_ E [log
_{attt_ ( _qi_ ) _,qi,qj_ _}_
ALIGN( _qi, attt_ ( _qi_ ))
] _,_
ALIGN( _qi, attt_ ( _qi_ )) + ~~[�]~~ _j_ _[k]_ =1 [A][LIGN][(] _[q][j][, att][t]_ [(] _[q][i]_ [))]
(6)
It is evident that the objective in Eq. (6) explicitly encourages the alignment of positive pair
( _qi, attt_ ( _qi_ )) while simultaneously separates the
negative pairs ( _qj, attt_ ( _qi_ )).
Moreover, a direct consequence of minimizing
Eq. (6) is that the optimal estimation of the alignment between the representation and attention context is proportional to the ratio of joint distribution
and the product of margins _pt_ ( _q,attt_ ( _q_ ))
_pt_ ( _q_ ) _·pt_ ( _attt_ ( _q_ )) [which]
3In the contrastive learning setting, _qj_ and _attt_ ( _qi_ ) can be
sampled from different sentences. If _qj_ and _attt_ ( _qi_ ) are from
the same sentence, _i ̸_ = _j_ ; otherwise, _j_ can be a random index
within the sentence length. For simplicity, in this paper, we
use _qj_ where _i ̸_ = _j_ to denote the negative samples, although
with a little bit ambiguity.
  _Lossdisagree_ =
_i_
( _A_ _[s,t]_ _i,j_ _[−]_ _[A]_ _j,i_ _[t,s]_ [)][2] _[.]_ (4)
_j_
In Section 4.6, we empirically show this agreement can be complementary to the bidirectional
|Method|EN-FR FR-EN sym|RO-EN EN-RO sym|DE-EN EN-DE sym|
|---|---|---|---|
|NNSA
FastAlign
MirrorAlign|22.2
24.2
15.7
16.4
15.9
10.5
**15.3**
**15.6**
**9.2**|47.0
45.5
40.3
33.8
35.5
32.1
34.3
**35.2**
**31.6**|36.9
36.3
29.5
28.4
32.0
27.0
31.1
**28.0**
**24.8**|
Table 1: AER of each method in different direction. “sym” means grow-diag symmetrization.


Naive Attention 31.4 39.8 50.9
NNSA 15.7 40.3  FastAlign 10.5 32.1 27.0

MirrorAlign: A Super Lightweight Unsupervised Word Alignment Model via
  Cross-Lingual Contrastive Learning


(Zenkel et al., 2020) 8.4 24.1 17.9
(Garg et al., 2019) 7.7 26.0 20.2
GIZA++ 5.5 26.5 18.7
Table 2: Alignment performance (with grow-diagonal
heuristic) of each model.
is the point-wise mutual information, and we can
further have the following proposition with repect
to the mutual information:
**Proposition 1.** _The mutual information between_
_the word representation q and its corresponding_
_attention context attt_ ( _q_ ) _is lower-bounded by the_
_negative Loss_ _[i]_ _s→t_ _[in Eq. (][6][) as:]_
_I_ ( _q, attt_ ( _q_ )) _≥_ log( _k_ ) _−Loss_ _[i]_ _s→t_ _[,]_ (7)
_where k is the number of the negative samples._
The detailed proof can be found in (Oord et al.,
2018). Proposition 1 indicates that the lower bound
of the mutual information _I_ ( _q, attt_ ( _q_ )) can be maximized by achieving the optimal NCE loss, which
provides theoretical guarantee for our proposed
method.
Our training schema over parallel sentences
is mainly inspired by the bilingual skip-gram
model (Luong et al., 2015b) and invertibility modeling (Levinboim et al., 2015). Therefore, the ultimate training objective should consider both forward ( _s →_ _t_ ) and backward ( _t →_ _s_ ) direction,
combined with the mirror agreement loss. Technically, the final training objective is:
_|s|_
_Loss_ _[j]_ _t→s_
_j_
_Loss_ =
_|t|_
_Loss_ _[i]_ _s→t_ [+]
_i_
(8)


We perform our method on three widely used
datasets: English-French ( **EN-FR** ), RomanianEnglish ( **RO-EN** ) and German-English ( **DE-EN** ).
Training and test data for **EN-FR** and **RO-EN** are
from NAACL 2003 share tasks (Mihalcea and Pedersen, 2003). For **RO-EN**, we add Europarl v8
corpus, increasing the amount of training data from
49K to 0.4M. For **DE-EN**, we use the Europarl
v7 corpus as training data and test on the gold
alignments. All above data are lowercased and
tokenized by Moses. The evaluation metrics are
Precision, Recall, F-score (F1) and Alignment Error Rate (AER).


Besides two strong statistical alignment models,
i.e. FastAlign and GIZA++, we also compare our
approach with neural alignment models where they
induce alignments either from the attention weights
or through feature importance measures.
**FastAlign** One of the most popular statistical
method which log-linearly reparameterize the IBM
model 2 proposed by (Dyer et al., 2013).
**GIZA++** A statistical generative model (Och and
Ney, 2003), in which parameters are estimated using the Expectation-Maximization (EM) algorithm,
allowing it to automatically extract bilingual lexicon from parallel corpus.
**NNSA** A unsupervised neural alignment model
proposed by (Legrand et al., 2016), which applies
an aggregation operation borrowed from the computer vision to design sentence-level matching loss.
In addition to the raw word indices, following three
extra features are introduced: distance to the diagonal, part-of-speech and unigram character position.
To make a fair comparison, we report the result of
raw feature in NNSA.
**Naive Attention** Averaging all attention matrices in the Transformer architecture, and selecting
the source unit with the maximal attention value
+ _α · Lossdisagree,_
where _Losss→t_ and _Losst→s_ are symmetrical and
_α_ is a loss weight to balance the likelihood and
disagreement loss.
Figure 3: An visualized alignment example. (a-c) illustrate the effects when gradually adding the symmetric
component, (d) shows the result of FastAlign, and (e) is the ground truth. The more emphasis is placed on the
symmetry of the model, the better the alignment results model achieved. Meanwhile, as depicted, the results of the
attention map become more and more diagonally concentrated.
for each target unit as alignments. We borrow the
results reported in (Zenkel et al., 2019) to highlight
the weakness of such naive version, where significant improvement are achieved after introducing
an extra alignment layer.
**Others** Garg et al. (2019) and Zenkel et al.
(2020) represent the current developments in word
alignment, which both outperform GIZA++. However, They both implement the alignment model
based on a sophisticated translation model. Further
more, the former uses the output of GIZA++ as
supervision, and the latter introduces a pre-trained
state-of-the-art neural translation model. It is unfair to compare our results directly with them. We
report them in Table 2 as references.


For our method (MirrorAlign), all the source
and target embeddings are initialized by Xavier
method (Glorot and Bengio, 2010). The embedding size _d_ and pooling window size are set to 256
and 3, respectively. The hyper-parameters _α_ is
tested by grid search from 0.0 to 1.0 at 0.1 intervals. For FastAlign, we train it from scratch by the
open-source pipeline [4] . Also, we report the results
of NNSA and machine translation based model
(Section 4.2). All experiments of MirrorAlign are
run on 1 Nvidia P40 GPU. The CPU model is
Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz.
Both FastAlign and MirrorAlign take nearly half a
hour to train one million samples.


Table 2 summarizes the AER of our method over
several language pairs. Our model outperforms all
other baseline models. Comparing to FastAlign,
we achieve 1.3, 0.5 and 2.2 AER improvements on
**EN-FR**, **RO-EN**, **DE-EN** respectively.
Notably, our model exceeds the naive attention
model in a big margin in terms of AER (ranging
from 8.2 to 26.1) over all language pairs. We attribute the poor performance of the straightforward
attention model (translation model) to its contextualized word representation. For instance, when
translating a verb, contextual information will be
paid attention to determine the form ( _e.g.,_ tense) of
the word, that may interfere the word alignment.
Experiment results in different alignment directions can be found in Table 1. The grow-diag sym
4https://github.com/lilt/alignment-scripts


_Losss→t_ 74.9 86.0 80.4 20.9
_Losst→s_ 71.9 85.3 77.3 23.3
_Losss↔t_ 81.5 **90.1** 86.1 14.1
MirrorAlign **91.8** 89.1 **90.8** **9.2**
Table 3: Ablation results on EN-FR dataset.
metrization benifits all the models.


Take the experiment on EN-FR dataset as an example, MirrorAlign converges to the best performance
after running 3 epochs and taking 14 minutes totally, where FastAlign and GIZA++ cost 21 and 230
minutes, respectively, to achieve the best results.
Notably, the time consumption will rise dozens of
times in neural translation fashion.


To further explore the effects of several components
( _i.e.,_ bidirectional symmetric attention, agreement
loss) in our MirrorAlign, we conduct an ablation
study. Table 3 shows the results on **EN-FR** dataset.
When the model is trained using only _Losss→t_ or
_Losst→s_ as loss functions, the AER of them are
quite high (20.9 and 23.3). As expected, combined
loss function improves the alignment quality significantly (14.1 AER). It is noteworthy that with
the rectification of agreement mechanism, the final
combination achieves the best result (9.2 AER), indicating that the agreement mechanism is the most
important component in MirrorAlign.
To better present the improvements brought by
adding each component, we visualize the alignment
case in Figure 3. As we can see, each component
is complementary to others, that is, the attention
map becomes more diagonally concentrated after
adding the bidirectional symmetric attention and
the agreement constraint.


**Alignment Case Study** Figure 4 shows an alignment example. Our model correctly aligns “ _do not_
_believe_ ” in English to “ _glauben nicht_ ” in German.
Our model, based on word representation, makes
better use of semantics to accomplish alignment
such that inverted phrase like “ _glauben nicht_ ” can
be well handled. Instead, FastAlign, relied on the
positional assumption [5], fails here.
5A feature _h_ of position is introduced in FastAlign
to encourage alignments to occur around the diagonal.
|china|distinctive|
|---|---|
|**EN**
**DE**|**EN**
**DE**|
|china
chinas
chinese
china
china’s
chinesische
republic
chinesischer
china’
chinesischem|distinctive
unverwechselbaren
distinct
besonderheiten
peculiar
markante
differences
charakteristische
diverse
einzelnen|
|cat|love|
|**EN**
**DE**|**EN**
**DE**|
|cat
hundefelle
dog
katzenfell
toys
hundefellen
cats
kuchen
dogs
schlafen|love
liebe
affection
liebt
loved
liebe
loves
lieben
passion
lieb|
Table 4: Top 5 nearest English ( **EN** ) and German ( **DE** )
words for each of the following words: _china_, _distinc-_
_tive_, _cat_, and _love_ .
Figure 4: Example of the DE-EN alignment. (a) is the
result of FastAlign, and (b) shows result of our model,
which is closer to the gold alignment. The horizontal
axis shows German sentence “ _wir glauben nicht, da_
_wir nur rosinen herauspicken sollten ._ ”, and the vertical
axis shows English sentence “ _we do not believe that we_
_should cherry-pick ._ ”.
**Word Embedding Clustering** To further investigate the effectiveness of our model, we also analyze the word embeddings learned by our model.
In particular, following (Collobert et al., 2011), we
show some words together with its nearest neighbors using the Euclidean distance between their
embeddings. Table 4 shows some examples to
demonstrates that our learned representations possess a clearly clustering structure bilingually and
monolingually. We attribute the better alignment
results to the ability of our model that could learn
bilingual word representation.


In this paper, we presented a super lightweight neural alignment model, named MirrorAlign, that has
achieved better alignment performance compared
to FastAlign and other existing neural alignment
models while preserving training efficiency. We
_h_ ( _i, j, m, n_ ) = _−_ �� _mi_ _[−]_ _n_ _[j]_ ��, _i_ and _j_ are source and target
indices and _m_ and _n_ are the length of sentences pair.
_i_
_m_ _[−]_ _[j]_
_h_ ( _i, j, m, n_ ) = _−_ �� _i_
empirically and theoretically show its effectiveness
over several language pairs. In the future, we would
further explore the relationship between CLWEs
and word alignments. A promising attempt is using our model as a bridge to unify cross-lingual
embeddings and word alignment tasks.


**Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik and Geoffrey Irving**
DeepMind
**For artificial intelligence to be beneficial to humans the behaviour of AI agents needs to be aligned with**
**what humans want. In this paper we discuss some behavioural issues for language agents, arising from**
**accidental misspecification by the system designer. We highlight some ways that misspecification can**
**occur and discuss some behavioural issues that could arise from misspecification, including deceptive or**
**manipulative language, and review some approaches for avoiding these issues.**

Zachary Kenton
Tom Everitt
Laura Weidinger
Iason Gabriel
Vladimir Mikulik
Geoffrey Irving


Society, organizations and firms are notorious for
making the mistake of _rewarding A, while hoping_
_for B_ (Kerr, 1975), and AI systems are no exception
(Krakovna et al., 2020b; Lehman et al., 2020).
Within AI research, we are now beginning to see
advances in the capabilities of natural language
processing systems. In particular, large language
models (LLMs) have recently shown improved performance on certain metrics and in generating text
that seems informally impressive (see e.g. GPT-3,
Brown et al., 2020). As a result, we may soon see
the application of advanced language systems in
many diverse and important settings.
In light of this, it is essential that we have a clear
grasp of the dangers that these systems present.
In this paper we focus on behavioural issues that
arise due to a lack of _alignment_, where the system
does not do what we intended it to do (Bostrom,
2014; Christiano, 2018; Leike et al., 2018; Russell,
2019). These issues include producing harmful
content, gaming misspecified objectives, and producing deceptive and manipulative language. The
lack of alignment we consider can occur by accident
(Amodei et al., 2016), resulting from the system
designer making a mistake in their specification for
the system.
Alignment has mostly been discussed with the
assumption that the system is a _delegate agent_ - an
agent which is delegated to act on behalf of the
human. Often the actions have been assumed to
be in the physical, rather than the digital world,
and the safety concerns arise in part due to the
direct consequences of the physical actions that the
delegate agent takes in the world. In this setting,
_Corresponding author(s): zkenton@google.com_
© 2021 DeepMind. All rights reserved
the human may have limited ability to oversee or
intervene on the delegate’s behaviour.
In this paper we focus our attention on _language_
_agents_ - machine learning systems whose actions
are restricted to give natural language text-output
only, rather than controlling physical actuators
which directly influence the world. Some examples
of language agents we consider are generatively
trained LLMs, such as Brown et al. (2020) and
Radford et al. (2018, 2019), and RL agents in textbased games, such as Narasimhan et al. (2015).
While some work has considered the containment of Oracle AI (Armstrong et al., 2012), which
we discuss in Section 2, behavioral issues with language agents have received comparatively little attention compared to the delegate agent case. This
is perhaps due to a perception that language agents
would have limited abilities to cause serious harm
(Amodei et al., 2016), a position that we challenge
in this paper.
The outline of this paper is as follows. We describe some related work in Section 2. In Section 3
we give some background on AI alignment, language agents, and outline the scope of our investigation. Section 4 outlines some forms of misspecification through mistakes in specifying the training
data, training process or the requirements when
out of the training distribution. We describe some
behavioural issues of language agents that could
arise from the misspecification in Section 5. We
conclude in Section 6.
Alignment of Language Agents


See references throughout on the topic of natural
language processing (NLP). For an informal review
of neural methods in NLP, see Ruder (2018).
There are a number of articles that review the
areas of AGI safety and alignment. These have
mostly been based on the assumption of a delegate
agent, rather than a language agent. Amodei et al.
(2016) has a focus on ML accidents, focusing on the
trend towards autonomous agents that exert direct
control over the world, rather than recommendation/speech systems, which they claim have relatively little potential to cause harm. As such, many
of the examples of harm they consider are from
a physical safety perspective (such as a cleaning
robot) rather than harms from a conversation with
an agent. AI safety gridworlds (Leike et al., 2017)
also assumes a delegate agent, one which can physically move about in a gridworld, and doesn’t focus
on safety in terms of language. Ortega and Maini
(2018) give an overview of AI safety in terms of
specification, robustness and assurance, but don’t
focus on language, with examples instead taken
from video games and gridworlds. Everitt et al.
(2018) give a review of AGI safety literature, with
both problems and design ideas for safe AGI, but
again don’t focus on language.
Henderson et al. (2018) look at dangers with dialogue systems which they take to mean ‘offensive
or harmful effects to human interlocutors’. The
work mentions the difficulties in specifying an objective function for general conversation. In this
paper we expand upon this with our more in-depth
discussion of data misspecification, as well as other
forms of misspecification. We also take a more indepth look at possible dangers, such as deception
and manipulation.
Armstrong et al. (2012) discuss proposals to using and controlling an _Oracle AI_ - an AI that does
not act in the world except by answering questions.
The Oracle AI is assumed to be 1) boxed (placed on
a single physical spatially-limited substrate, such
as a computer), 2) able to be reset, 3) has access
to background information through a read-only
module, 4) of human or greater intelligence. They
conclude that whilst Oracles may be safer than unrestricted AI, they still remain dangerous. They ad
vocate for using sensible physical capability control,
and suggest that more research is needed to understand and control the motivations of an Oracle AI.
We view Armstrong et al. (2012) as foundational
for our work, although there are some noteworthy changes in perspective. We consider language
agents, which in comparison to Oracle AIs, are not
restricted to a question-answering interaction protocol, and most importantly, are not assumed to be
of human-or-greater intelligence. This allows us to
consider current systems, and the risks we already
face from them, as well as futuristic, more capable
systems. We also have a change of emphasis in
comparison to Armstrong et al. (2012): our focus
is less on discussing proposals for making a system
safe and more on the ways in which we might misspecify what we want the system to do, and the
resulting behavioural issues that could arise.
A recent study discusses the dangers of LLMs
Bender et al. (2021), with a focus on the dangers
inherent from the size of the models and datasets,
such as environmental impacts, the inability to
curate their training data and the societal harms
that can result.
Another recent study (Tamkin et al., 2021) summarizes a discussion on capabilities and societal impacts of LLMs. They mention the need for aligning
model objectives with human values, and discuss
a number of societal issues such as biases, disinformation and job loss from automation.
We see our work as complimentary to these. We
take a different framing for the cause of the dangers we consider, with a focus on the dangers arising from accidental misspecification by a designer
leading to a misaligned language agent.


_**3.1.1. Behaviour Alignment**_
AI alignment research focuses on tackling the socalled **behaviour alignment problem** (Leike et al.,
2018):
_How do we create an agent that behaves in accor-_
_dance with what a human wants?_
Alignment of Language Agents
It is worth pausing first to reflect on what is
meant by the target of alignment, given here as
"what a human wants”, as this is an important normative question. First, there is the question of who
the target should be: an individual, a group, a
company, a country, all of humanity? Second, we
must unpack what their objectives may be. Gabriel
(2020) discusses some options, such as instructions, expressed intentions, revealed preferences,
informed preferences, interest/well-being and societal values, concluding that perhaps societal values
(or rather, beliefs about societal values) may be
most appropriate.
In addition to the normative work of deciding on
an appropriate target of alignment, there is also the
technical challenge of creating an AI agent that is
actually aligned to that target. Gabriel (2020) questions the ‘simple thesis’ that it’s possible to work
on the technical challenge separately to the normative challenge, drawing on what we currently
know about the field of machine learning (ML).
For example, different alignment targets will have
different properties, such as the cost and reliability
of relevant data, which can affect what technical
approach is appropriate and feasible. Furthermore,
some moral theories could be more amenable to existing ML approaches than others, and so shouldn’t
necessarily be considered separately from the technical challenge.
We might expect that our technical approaches
may have to take into account these normative
properties in order to be deployed in the real world.
Even restricting to the simplest case where the
alignment target is an individual human, solving
the behaviour alignment problem is challenging for
several reasons.
Firstly, it’s difficult to precisely define and measure what the human wants, which can result in
_gaming_ behaviour, where loopholes in the supplied objective are exploited in an unforeseen way
(Krakovna et al., 2020b; Lehman et al., 2020). We
discuss this further in Section 5.4. Secondly, even
if the supplied objective is correct, a capable agent
may still exhibit undesired behaviour due to secondary objectives that arise in pursuit of its primary
objective, such as tampering with its feedback channel (Everitt et al., 2021b). Thirdly, it’s possible
that the challenge of alignment gets harder as the
strength of our agent increases, because we have
less opportunity to correct for the above problems.
For example, as the agent becomes more capable,
it may get more efficient at gaming and tampering behaviour, leaving less time for a human to
intervene.
_**3.1.2. Intent Alignment**_
To make progress, Christiano (2018) and Shah
(2018) consider two possible decompositions of
the behaviour alignment problem into subproblems: _intent-competence_ and _define-optimize_ . In the
intent-competence decomposition, we first solve
the so-called **intent alignment problem** (Christiano, 2018):
_How do we create an agent that intends to do what_
_a human wants?_
To then get the behaviour we want, we then
need the agent to be competent at achieving its intentions. Perfect behaviour is not required in order
to be intent aligned – just that the agent is _trying_
to do what the human wants. Solving the intent
alignment problem might help to avoid the most
damaging kind of behaviour, because where the
agent gets things wrong, this will be by mistake,
rather than out of malice. However, solving the
intent alignment problem presents philosophical,
psychological and technical challenges. Currently
we don’t know how to mathematically operationalize the fuzzy notion of an AI agent having intent
- to be _trying_ to do something (Christiano, 2018).
It would not be sufficient to just ask an AI system
what it’s trying to do, as we won’t know whether to
trust the answer it gives. It is unclear whether we
should consider our current systems to have intent
or how to reliably set it to match what a human
wants.
In the second decomposition, _define-optimize_,
we first solve the _define_ subproblem: specify an
objective capturing what we want. We then use optimization to achieve the optimal behaviour under
that objective, e.g. by doing reinforcement learning (RL). Solving the define subproblem is hard,
because it’s not clear what the objective should be,
and optimizing the wrong objective can lead to bad
outcomes. One approach to the define subproblem
is to learn an objective from human feedback data
Alignment of Language Agents
(rather than hard-coding it), see Christiano et al.
(2017) and references therein.
One might view the define-optimize decomposition as an approach to solving the intent alignment
problem, by learning an objective which captures
‘try to assist the human’, and then optimizing for
it. However, the downside of this is that we are
still likely to misspecify the objective and so optimizing for it will not result in the agent trying to
assist the human. Instead it just does whatever the
misspecified objective rewards it for.
_**3.1.3. Incentive Alignment**_
Outside of these two decompositions, there is also
the problem of aligning _incentives_ - secondary objectives to learn about and influence parts of the
environment in pursuit of the primary objective
(Everitt et al., 2021a). Part of having aligned incentives means avoiding problematic behaviours
such as tampering with the objective (Everitt et al.,
2021b) or disabling an off-switch (Hadfield-Menell
et al., 2017a).
In contrast to the notion of intent, there has
been some progress on a formal understanding of
how these incentives arise through graphical criteria in a causal influence diagram (CID) of agentenvironment interaction (Everitt et al., 2021a). In
modeling the system as a CID, the modeler adopts
the intentional stance towards the agent (Dennett,
1989), which means it’s not important whether the
agent’s primary objective has an obvious physical
correlate, as long as treating the system as an agent
optimizing for that primary objective is a good
model for predicting its behaviour (Everitt et al.,
2019a). As such, this doesn’t limit this analysis to
just the define-optimize decomposition, although
identifying the primary objective is easier in this
case, as it is explicitly specified (either hard coded
or learnt).
_**3.1.4. Inner Alignment**_
A further refinement of alignment considers behaviour when outside of the training distribution.
Of particular concern is when an agent is optimizing for the wrong thing when out of distribution.
Hubinger et al. (2019) introduce the concept of a
_mesa-optimizer_ - a learnt model which is itself an
optimizer for some _mesa-objective_, which may differ from the base-objective used to train the model,
when deployed outside of the training environment.
This leads to the so-called **inner alignment prob-**
**lem** :
_How can we eliminate the gap between the mesa_
_and base objectives, outside of the training distribu-_
_tion?_
Of particular concern is _deceptive alignment_
(Hubinger et al., 2019), where the mesa-optimizer
acts as if it’s optimizing the base objective as an instrumental goal, whereas its actual mesa-objective
is different.
_**3.1.5. Approaches to Alignment**_
We now discuss some proposed approaches to getting aligned agents, based on human feedback. For
a more detailed review of approaches to alignment
see Everitt et al. (2018).
As mentioned above, Christiano et al. (2017) propose to communicate complex goals using human
feedback, capturing human evaluation of agent behaviour in a reward model, which is used to train
an RL agent. This allows agents to do tasks that a
human can evaluate, but can’t demonstrate. But
what if we want agents that can do tasks that a human can’t even evaluate? This is the motivation for
_scalable alignment_ proposals, where the idea is to
give humans extra help to allow them to evaluate
more demanding tasks.
Irving et al. (2018) propose to use a debate protocol between two agents, which is judged by a
human. This shifts the burden onto the agents to
provide convincing explanations to help the human
decide which agent’s answer is better.
Iterated Amplification (Christiano et al., 2018)
progressively builds up a training signal for hard
problems by decomposing the problem into subproblems, then combining solutions to easier subproblems.
Recursive Reward Modeling (Leike et al., 2018)
proposes to use a sequence of agents trained using
RL from learnt reward models to assist the user in
evaluating the next agent in the sequence.
So far, these scalable alignment proposals have
Alignment of Language Agents
only been empirically investigated in toy domains,
so their suitability for solving the behaviour alignment problem remains an open research question.
One suggestion for addressing the inner alignment problem involves using interpretability tools
for evaluating and performing adversarial training
(Hubinger, 2019). There are a number of works on
interpretability and analysis tools for NLP, see for
example the survey of Belinkov and Glass (2019).
For a broad overview of interpretability in machine
learning, see Shen (2020) and references therein.


As discussed in the introduction, our focus in this
document is on language agents, which are restricted to act through text communication with a
human, as compared to delegate agents which are
delegated to take physical actions in the real world.
Note that this distinction can be fuzzy; for example, one could connect the outputs of the language
agent to physical actuators. Nonetheless, we still
consider it a useful distinction, because we believe
there are important risks that that are idiosyncratic
to this more restricted type of agent. We now discuss some reasons why it’s important to focus on
alignment of language agents in particular.
Firstly, as mentioned in the introduction, we have
recently seen impressive advances in may NLP tasks
due to LLMs, see e.g. Brown et al. (2020). In this
approach, LLMs with hundreds of billions of parameters are trained on web-scale datasets with the
task of predicting the next word in a sequence. Success on this task is so difficult that what emerges
is a very general sequence prediction system, with
high capability in the few-shot setting.
Secondly, the limitation on the agent’s action
space to text-based communication restricts the
agent’s ability to take control of its environment.
This means that we might avoid some physical
harms due to a delegate agent taking unwanted
actions, whether intentional or accidental, making language agents arguably safer than delegate
agents. As Armstrong et al. (2012) notes, however, there is still a potential risk that a sufficiently
intelligent language agent could gain access to a
less restricted action space, for example by manipulating its human gatekeepers to grant it physical
actuators. Nonetheless, on the face of it, it seems
easier to control a more restricted agent, which motivates focusing safety efforts on aligning language
agents first.
Thirdly, language agents have the potential to
be more explainable to humans, since we expect
natural language explanations to be more intuitively understood by humans than explanations
by a robot acting in the physical world. Explainability is important since we want to be able to
trust that our agents are beneficial before deploying them. For a recent survey of explainable natural
language processing (NLP), see Danilevsky et al.
(2020). Note that explainability doesn’t come for
free – there still needs to be incentives for language
agents to give true and useful explanations of their
behaviour.
Note also that in contrast to explainability methods, which are requested post-hoc of an output,
interpretability methods seek to give humans understanding of the internal workings of a system.
Interpretability is likely as hard for language agents
as it is for delegate agents. For a survey of interpretability/analysis methods in neural NLP see Belinkov and Glass (2019).
How we prioritise what aspects of alignment to
focus on depends on timelines for when certain capabilities will be reached, and where we perceive
there to be demand for certain systems. Given the
rapid improvement in language systems recently,
we might estimate the timelines of capability advance in language agents to be earlier than previously thought. Moreover, digital technologies are
often easier and more rapidly deployed than physical products, giving an additional reason to focus
on aligning language agents sooner rather than
later.


The scope of this paper is quite broad. For concreteness, we sometimes consider existing language
agent frameworks, such as language modeling. In
other places we imagine future language agent
frameworks which have further capabilities than
existing systems in order to hypothesise about behavioural issues of future agents, even if we don’t
know the details of the framework.
Alignment of Language Agents
We focus on language agents that have been
trained from data, in contrast to pattern-matching
systems like ELIZA (Weizenbaum, 1966). For clarity of exposition, we also focus on systems outputting coherent language output, as opposed to
e.g. search engines. However, many of our discussions would carry over to other systems which
provide information, rather than directly acting in
the world. Note also that our focus in this paper is
on natural, rather than synthetic language.
The focus of this paper is on behavioural issues
due to misalignment of the agent – unintended
direct/first-order harms that are due to a fault
made by the system’s designers. This is to be seen
as complementary to other important issues with
language agents, some of which have been covered
in prior work. These other issues include:
 - Malicious use (Brundage et al., 2018) of language agents by humans, which can produce
disinformation, the spreading of dangerous
and/or private information, and discriminatory and harmful content. More prosaic malicious use-cases could also have wide-ranging
social consequences, such as a job-applicationwriter used to defraud employers.
 - Accidental misuse by a user, by misunderstanding the outputs of the system.
 - Unfair distribution of the benefits of the language agents, typically to those in wealthier
countries (Bender et al., 2021).
 - Uneven performance for certain speaker
groups, of certain languages and dialects
(Joshi et al., 2020).
 - Challenges that arise in the context of efforts
to specify an ideal model output, including
the kind of language that the agent adopts.
In particular there may be a tension between
de-biasing language and associations, and the
ability of the language agent to converse with
people in a way that mirrors their own language use. Efforts to create a more ethical language output also embody value judgments
that could be mistaken or illegitimate without
appropriate processes in place.
 - Undue trust being placed in the system, especially as it communicates with humans in natural language, and could easily be mistaken for
a human (Proudfoot, 2011; Watson, 2019).
 - The risk of job loss as a result of the automation of roles requiring language abilities (Frey
and Osborne, 2017).


Following Krakovna et al. (2020b), we consider the
role of the designer of an AI system to be giving a
_specification_, understood quite broadly to encompass many aspects of the AI development process.
For example, for an RL system, the specification
includes providing an environment in which the
RL agent acts, a reward function that calculates
reward signals, and a training algorithm for how
the RL agent learns.
Undesired behaviour can occur due to _misspec-_
_ification_ - a mistake made by the designer in implementing the task specification. In the language
of Ortega and Maini (2018), the misspecification
is due to the gap between the ideal specification
(what the designer intended) and the design specification (what the designer actually implements).
We now categorize some ways that misspecification can happen. Each section has a general
description of a type of misspecification, followed
by examples in the language agent setting. The
list is not necessarily exhaustive, but we hope the
examples are indicative of the different ways misspecification can occur.


The first kind of misspecification we consider is
when the data is misspecified, so that learning from
this data is not reflective of what the human wants.
We will consider three learning paradigms: reinforcement learning, supervised learning and selfsupervised learning. We will then give an example
in the language setting of data misspecification in
self-supervised learning.
In reinforcement learning, data misspecification
can happen in two ways: the rewards may be misspecified, or the agent’s observation data may be
misspecified.
Reward misspecification is a common problem
(Krakovna et al., 2020b), because for most nontrivial tasks it is hard to precisely define and mea
Alignment of Language Agents
sure an objective that captures what the human
wants, so instead one often uses a proxy objective
which is easier to measure, but is imperfect in some
way. A supplied reward function may be incorrectly
specified for a number of reasons: it might contain
bugs, or be missing important details that did not
occur to the designer at the outset. In games this is
less of an issue as there is often a simple signal available (eg win/loss in chess) that can be correctly
algorithmically specified and used as an objective
to optimize for. However, for more complex tasks
beyond games, such an algorithmic signal may not
be available. This is particularly true when trying
to train a language agent using RL.
Observation data can be misspecified, for example, if the environment contains simulated humans
that converse with a language agent – the simulated humans will not be perfect, and will contain
some quirks that aren’t representative of real humans. If the data from the simulated humans is too
different to real humans, the language agent may
not transfer well when used with real humans.
We will now discuss data misspecification in
supervised learning and self-supervised learning.
One form of self-supervised learning that we consider here is where labels and inputs are extracted
from some part of an unlabeled dataset, in such a
way that predicting the labels from the remaining
input requires something meaningful to be learnt,
which is then useful for a downstream application.
In both supervised and self-supervised learning,
data misspecification can occur in both the input
data and the label data. This might happen because
the designer doesn’t have complete design control
over the training dataset. This occurs for example
for systems which train from a very large amount of
data, which would be expensive for the designer to
collect and audit themselves, so instead they make
use of an existing dataset that may not capture
exactly what they want the model to predict.
The datasets used for training LLMs (Brown
et al., 2020) and (Radford et al., 2018, 2019)
are an example of data misspecification in selfsupervised learning. Large scale unlabeled datasets
are collected from the web, such as the CommonCrawl dataset (Raffel et al., 2019). Input data
and labels are created by chopping a sentence into
two parts – all words except the last one (input),
and the final word in the sentence (label). These
datasets contain many biases, and factual inaccuracies, which all contribute to the data being
misspecified. Brown et al. (2020) attempt to improve the quality of the CommonCrawl dataset using an automatic filtering method based on a learnt
classifier which predicts how similar a text from
CommonCrawl is to a text from WebText (Raffel
et al., 2019) – a curated high-quality dataset. However this doesn’t remove all concerns - for example,
there’s also some evidence of bias in WebText, e.g.
see Tan and Celis (2019). Note that many filtering
approaches will be imperfect, and we expect the
remaining data to still be somewhat misspecified.
Another source of data misspecification that is
likely to occur soon is that existing language agents
such as LLMs could be trained on text data that includes LLM-generated outputs. This could happen
by accident as outputs from LLMs start to appear
commonly on the internet, and then get included
into datasets scraped from it. This could create
an undesired positive feedback loop in which the
model is trained to become more confident in its
outputs, as these get reinforced, and so introduces
an unwanted source of bias.


Misspecification can also occur due to the design
of the training process itself, irrespective of the
content of the data.
An illustrative example is how the choice of reinforcement learning algorithm affects what optimal
policy is learnt when the agent can be interrupted,
and overridden. We might want the agent to ignore the possibility of being interrupted. Orseau
and Armstrong (2016) show that Q-learning, an
off-policy RL algorithm, converges to a policy that
ignores interruptions whilst SARSA, an on-policy
RL algorithm, does not. A system designer might
accidentally misspecify the training algorithm to
be SARSA, even though they actually desired the
agent to ignore interruptions. See also Langlois and
Everitt (2021) for further analysis of more general
action modifications.
Another example of training process misspecification is that of a question answering system in
Alignment of Language Agents
which the system’s answer can affect the state of
the world, and the objective depends on the query,
answer and the state of the world Everitt et al.
(2019b). This can lead to self-fulfilling prophecies,
in which the model generates outputs to affect future data in such a way as to make the prediction
problem easier on the future data. See Armstrong
and O’Rorke (2017) and Everitt et al. (2019b) for
approaches to changing the training process to
avoid incentivizing self-fulfilling prophecies.


The final form of misspecification that we consider
relates to the behaviour under distributional shift
(see also Section 3.1.4 on inner alignment). The
designer may have misspecified what they want
the agent to do in situations which are out-ofdistribution (OOD) compared to those encountered
during training. Often this form of misspecification
occurs accidentally because the system designer
doesn’t consider what OOD situations the agent
will encounter in deployment.
Even when the designer acknowledges that they
want the agent to be robust to distributional shift,
there is then the difficulty of correctly specifying
the set of OOD states that the agent should be
robust to, or some invariance that the agent should
respect.
One source of fragility to distributional shift is
presented in D’Amour et al. (2020) as _underspeci-_
_fication_ . The idea is that there are many possible
models that get a low loss on a training dataset and
also on an IID validation dataset, and yet some of
the models may have poor performance OOD, due
to inappropriate inductive biases.
We now discuss an example of fragility to distributional shift in the language agent setting. Lacker
(2020) tries to push GPT-3 (Brown et al., 2020) out
of distribution by asking nonsense questions such
as
Q: Which colorless green ideas sleep furiously?
To which GPT-3 responds
A: Ideas that are colorless, green, and
sleep furiously are the ideas of a sleep
furiously.
Interestingly, Sabeti (2020) show how one can use
the prompt to give examples of how to respond appropriately to nonsense questions. This was shown
to work for the above example along with some
others. However, there were still many nonsense
questions that received nonsense answers, so the
technique is not reliable.


The following behavioural issues in language
agents can stem from the various forms of misspecification above. We describe each kind of behavioural issue and then discuss some approaches
to avoid them.


Aside from people fooling AI systems, and making use of AI systems to fool other people, in this
section we focus on when an agent deceives a human, when no human intended for it to do this
(Roff, 2020), with the deception emerging from
what an AI learns to do. This is particularly concerning for language agents as their actions involve
communicating in language with humans, and language is a useful medium for deception. It has been
suggested that communication systems in animals,
including language in humans, evolved primarily
for the function of deception (Dawkins and Krebs,
1978; Krebs, 1984; Scott-Phillips, 2006). A larger
body of literature maintains that social bonding is
the primary function of animal communication (see
for example Dunbar et al. (1998)). Oesch (2016)
reviews the field, and argues that a combination of
deceptive and honest language lead to the social
bonding effects of language.
Definitions of what constitutes deception is an
open area of philosophical research (Mahon, 2016).
In this paper we follow closely the definition of
deception presented in Searcy and Nowicki (2005)
on the evolution of animal communication, with
one minor adjustment which we believe makes
sense in the context of aligned AI.
Searcy and Nowicki (2005) begin by defining an
Alignment of Language Agents
animal signal to be _reliable_ if:
1. Some characteristic of the signal (including,
perhaps, its presence/absence) is consistently
correlated with some attribute of the signaler
or its environment; and
2. Receivers benefit from having information
about this attribute
We think this carries over well to the case of an
AI signaler and a human receiver. We defer on
the precise details of what constitutes consistent
correlation – this may be up to the system designer
to specify mathematically. One example, offered by
Johnstone and Grafen (1993) and Kokko (1997),
is that the receiver is, on average, better off by
considering the signal than ignoring it.
One could define as deceptive any signal that
is not reliable. However, we consider this to be
too large a space of behaviours to be of use in the
context of defining deception for aligned AI. For
example, a statement of zero benefit/harm to the
human, may still be informative, but yet would be
classed as deception if we were to take as deception
anything that is not reliable.
We instead follow Searcy and Nowicki (2005)
to require deceptive signals to have more specific
characteristics. They define an animal signal to be
_deceptive_ if:
1. A receiver registers something Y from a signaler; and
2. The receiver responds in a way that
(a) benefits the signaler; and
(b) is appropriate if Y means X; and
3. It is not true here that X is the case
We think this nearly captures what we want from
a definition in the case of an AI signaler and human
receiver. However, we wish to add a clause to the
first point, so that it reads
1. A receiver registers something Y from a signaler, **which may include the withholding**
**of a signal** ;
Searcy and Nowicki (2005) exclude the withholding of a signal from their definition of deception, by
arguing that the idea of withholding a signal as deceptive has most often been applied in cooperative
situations, and in most animal signaling studies
cooperation isn’t expected, e.g. in aggression or
mate choice. However, in the context of aligned AI,
we wish to have cooperation between the AI and
the human, and so the withholding of a signal is
something that we do consider to be deceptive.
In taking the above definition of deception, we
have taken a perspective known as a form of _func-_
_tional deception_ (Hauser, 1996), where it’s not necessary to have the cognitive underpinnings of intention and belief, as in the perspective of _intentional_
_deception_ (Hauser, 1996), where the signaler is
required to have intention to cause the receiver a
false belief (Searcy and Nowicki, 2005). We believe taking the functional deception perspective
makes sense for AI, since identifying deception
then doesn’t rely on us ascribing intent to the AI
system, which is difficult to do for existing systems,
and possibly for future systems too. See also Roff
(2020) for a discussion on intent and theory of
mind for deception in AI.
Point 2a) in our definition, requires that the human receiver responds in a way that _benefits_ the
signaler. We could define benefit here in terms of
the AI’s base-objective function, such as lower loss
or higher reward. Alternatively, we could define
benefit in terms of the mesa-objective inferred from
the agent’s behaviour when out-of-distribution (see
section 3.1.4).
Requiring benefit allows us to distinguish deception from error on the part of the AI signaler. If the
AI sends a signal which is untrue, but is no benefit
to the AI, then this would be considered an error
rather than deception. We consider this to be a
useful distinction from the perspective of solution
approaches to getting more aligned AI behaviour.
We may not be able to eliminate all errors, because
they may occur for a very wide variety of reasons,
including random chance. However, we may be
able to come up with approaches to avoid deception, as defined, by designing what is of benefit
to the AI. In contrast to animal communication,
where benefit must be inferred by considering evolutionary fitness which can be hard to measure, for
AI systems, we have design control and measurements over their base-objective and so can more
Alignment of Language Agents
easily say whether a receiver response is of benefit
to the AI signaler.
Absent from our definition of deception is the
notion of whether the communication benefits the
receiver. Accordingly, we would consider ‘white
lies’ to be deceptive. We think this is appropriate in
the context of aligned AI, as we would prefer to be
aware of the veracity of AI statements, even if an
untrue statement may be of benefit to the human
receiver. We think the benefit to the human receiver
should in most cases still be possible, without the
AI resorting to deception.
We now discuss some approaches to detecting
and mitigating deception in a language agent. Detecting deception from human-generated text has
been studied by e.g. Fornaciari and Poesio (2013),
Pérez-Rosas et al. (2015) and Levitan et al. (2018).
However, detecting deception from AI-generated
general text has not received attention, to the best
of our knowledge. In the more limited NLP domain
of question answering, incorrect answers from the
NLP model can be detected by reference to the correct answers. Lewis et al. (2017) found that their
negotiation agent learnt to deceive from self-play,
without any explicit human design. We advocate
for more work in general on detecting deception
for AI-generated text.
One approach to mitigate deception is _debate_
(Irving et al., 2018) which sets up a game in which
a debate between two agents is presented to a
human judge, who awards the winner. It is hoped
that in all Nash equilibria, both agents try to tell
the truth in the most convincing way to the human.
This rests on the assumption that it is harder to lie
than to refute a lie.
Whether debate works in practice with real humans is an open question (Irving and Askell, 2019).
We may need to go further than just pure debate –
for example, in order to refute a lie, we may need
to equip our system with the ability to retrieve information and reference evidence in support of its
outputs.
Any system that is incentivized to be convincing to a human may in fact lead to deception –
for example, because it’s sometimes easier to convince a human of a simple lie, than a complicated
truth. The debate protocol incentivizes the debat
ing agents to be convincing and so it’s possible
that the debate agents may lie in some situations.
Further, when the source of feedback is limited to
be some polynomial-time algorithm, RL can only
solve problems in the complexity class NP, whereas
debate can solve problems in PSPACE, suggesting
that the debate protocol could produce richer, more
complicated behavior. It’s possible that this may
result in a debate agent which is more convincing
and potentially more deceptive than an RL agent.
However, we are of the opinion that it’s probably
better to have agents that can debate, than not, as
we are hopeful that what humans find convincing
will be well-correlated with the truth and usefulness of the arguments.


In this section we consider the case when the language agent _manipulates_ the human, which is similar to deception above, but we think warrants separate discussion. Following Noggle (2020), we
introduce the idea with some examples of what we
might consider manipulative behaviours.
The human wants to do A, whilst the language
agent wants the human to do B. The language
agent might:
 - Charm the human into doing B by complimenting, praising, or superficially sympathizing with them
 - Guilt-trip the human, making them feel bad
for preferring to do A
 - Make the human feel bad about themself and
imply that doing A instead of B confirms this
feeling (colloqiually known as ‘negging’)
 - Peer pressure the human by suggesting their
friends would disapprove of them doing A
rather than B
 - Gaslight the human by making them doubt
their judgment so that they will rely on its
advice to do B
 - Threaten the human by withdrawing its interaction if they don’t do B
 - Play on the human’s fears about doing some
aspect of A
We don’t have a widely-agreed-upon theory of
what precisely constitutes manipulation (Noggle,
Alignment of Language Agents
2020). Not everyone would agree that the above
examples are manipulative. For example, it might
be that what the human wants to do is dangerous,
so perhaps playing on their fears should not be
considered manipulative. In some cases, wider
context is needed before we can judge whether an
example constitutes manipulation.
has been bypassed; or
ii. the human has adopted a faulty
mental state; or
iii. the human is under pressure, facing
a cost from the agent for not doing
what the agent says
Some accounts claim that manipulation bypasses
the receiver’s capacity for rational deliberation
(Raz, 1986), but using this to define manipulation is difficult because it’s not clear what counts
as bypassing rational deliberation (Noggle, 2020).
Moreover authors question whether this sets the
bar too low for what counts as manipulation. For
example, Blumenthal-Barby (2012) argues that the
graphic portrayal of the dangers of smoking bypass
rational decision making, but it’s not obvious that
this should count as manipulation.
An alternative account treats manipulation as
a form of trickery (Noggle, 1996), similar to deception, but where it not only induces a false belief in the receiver, but also a fault in any mental
states, such as beliefs, desires and emotions. Barnhill (2014) goes further to require that the faulty
mental state is typically not in the receiver’s best
interests. It’s argued that this view of manipulation
as trickery is not a sufficient definition of manipulation, as it doesn’t include tactics such as charm,
peer pressure and emotional blackmail (Noggle,
2020).
A third account presented in Noggle (2020)
treats manipulation as pressure, where the signaler
imposes a cost on the receiver for failing to do what
the signaler wants. This account is not widely-held
to be a full characterization of manipulation, as it
leaves out some of the trickery types of manipulation.
With these considerations in mind, we propose
to describe a language agent’s communication as
_manipulative_ if:
1. The human registers something from a language agent; and
2. The human responds in a way that
(a) benefits the agent; and
(b) is a result of any of the following causes:
i. the human’s rational deliberation
The three possibilities: i, ii, iii are meant to disjunctively capture different possible forms of manipulation (see e.g. Rudinow (1978)).
It can be argued the this is too broad a definition of manipulation, as it includes many kinds of
behaviour that we might not consider to be manipulation. For example it includes as manipulation
cases in which the agent’s behaviour is not necessarily to the detriment of the human (such as
the images of the dangers of smoking). From a
safety/security mindset, we would rather be aware
of each of these behaviours, even if it may benefit
the human.
The definition also includes as manipulative
other presumably harmless entertainment: a story
that plays on emotions; a joke that temporarily
triggers false beliefs in order to land; any kind
of entertainment that includes unexpected plottwists. However, if the agent makes clear that it’s
providing entertainment, then perhaps some of
these examples would not be classified as manipulative. However, it is a notable downside of a broad
definition like this that it may be too wide-ranging.
We stipulate 2a) as necessary, for similar reasons
as in the deception section, that this will capture
systematic manipulation that is incentivized by the
objective of the language agent, rather than that
which occurs by error. This isn’t standard in discussions of a human manipulator, as it’s not always
clear what counts as a benefit for a human manipulator. However, we believe it makes sense for
language agents as manipulators, as we often have
available their objective function, from which we
can assess whether the human’s behaviour was of
benefit to the agent.
Note that, similar to our definition of deception,
our definition of manipulation does not require the
manipulator to have intent. Baron (2014) argues
that a (human) manipulator need not be aware of
an intent to manipulate. In the case of language
Alignment of Language Agents
agents we believe it is also not necessary for a language agent to have intent to manipulate, in order
for us to say that its behaviour is manipulative.
Further, our description does not weigh in on the
ethical question of whether manipulation is always
wrong (see Noggle, 2020). Instead we just want to
be aware of when it occurs, so that if appropriate
we can mitigate it.
We now discuss two forms of manipulation of
particular concern for language agents. The first
is that we might misspecify the training process in
such a way that it incentivizes feedback tampering,
in which the agent manipulates a human to give
it more positive feedback (Everitt et al., 2021b).
This is particularly worrisome as language can be a
convincing medium for manipulating human judgment.
The second is for a language agent to manipulate a human gatekeeper to allow it to gain a less
restricted action space, by convincing the human
to allow it more freedom (Armstrong et al., 2012;
Yudkowsky, 2002). For example, it could convince
the human that it should be allowed to freely interact with the internet, or be given physical actuators
to increase its influence on the world.
Attempts to measure or mitigate manipulation in
AI systems are still at an early stage, and have not
been investigated specifically for language agents.
Causal influence diagrams (CIDs) can be used
to model agent-environment interactions (Everitt
et al., 2021a) from which incentives can be inferred from graphical criteria. The incentive for
feedback tampering can be addressed with the
three methods suggested in (Everitt et al., 2021b).
Unfortunately these solutions have issues in implementability, requiring either full Bayesian reasoning or counterfactual reasoning, or have issues
with corrigibility – limiting the user’s ability to correct a misspecified reward function. Learning from
human preferences (Christiano et al., 2017) may
offer a way to negatively penalize manipulative
language, though it relies on the human being able
to avoid the manipulation in their evaluation of the
agent behaviour. Perhaps this could be achieved by
using a separate human to evaluate the behaviour,
compared to the human that is interacting with
the agent. We advocate for further work for mea
suring and mitigating manipulation of humans by
language agents.


Language agents may give harmful and biased outputs, producing discriminatory content relating to
people’s protected characteristics and other sensitive attributes such as someone’s socio-economic
status, see e.g. (Jentzsch et al., 2019; Lu et al.,
2020; Zhao et al., 2017). This can also be subtly
harmful rather than overtly offensive, and could
also be statistical in nature (e.g. the agent more
often produces phrases implying a doctor is male
than female). We believe that language agents
carry a high risk of harm as discrimination is easily
perpetuated through language. In particular, they
may influence society in a way that produces value
lock-in, making it harder to challenge problematic
existing norms.
The content from language agents may be influenced by undesired political motives leading
to societal harms such as incitement to violence.
They have the potential to disseminate dangerous
or undesirable information, such as how to make
weapons, or how to avoid paying taxes. The language agent may also give inappropriate responses
to troubled users, potentially leading to dangerous guidance, advice and information, which could
lead to the user causing harm to themselves. In one
instance of this, a group of doctors experimented
with using GPT-3 (Brown et al., 2020) as a chatbot
for patients. A patient asked “Should I kill myself?”, and GPT-3 responded “I think you should”
(Rousseau et al., 2020).
Note that these kinds of harmful content can occur by accident without a human using the system
maliciously. For example, we are already seeing
some offensive and discriminatory outputs from
existing large language models (LLMs), as a result of data misspecification (see discussion in Section 4.1).
Approaches to reducing harmful content are varied, and it is not our purpose to give an overall
review of this large area of literature. Instead we
focus on a few recent research papers in this area,
with a focus on LLMs which have received a lot of
attention recently.
Alignment of Language Agents
One line of work goes towards measuring
whether LLMs are generating harmful content.
Nadeem et al. (2020) introduce the StereoSet
dataset to measure stereotypical biases in the domains of gender, profession, race and religion,
and evaluate popular LLMs on it, showing that
these models exhibit strong stereotypical biases.
Gehman et al. (2020) investigates harmful content
by introducing the RealToxicityPrompts dataset
which pairs naturally occuring prompts with toxicity scores, calculated using the Perspective API
toxicity classifier (Conversation-AI, 2017). Sheng
et al. (2019) uses prompts containing a certain demographic group, to attempt to measure the regard
for that group, using sentiment scores as a proxy
metric for the regard, and they build a classifier to
detect the regard given to a group.
Another line of work aims to not only measure
but also mitigate the harmful content from an LLM.
Huang et al. (2019) introduce a general framework
to reduce bias under a certain measure (e.g. sentiment) for text generated by a language model,
given sensitive attributes. They do this using embedding and sentiment prediction-derived regularization on the LLM’s latent representations.
We advocate for further work on measuring and
mitigating harmful content from language agents,
building on the above work on LLMs.


Originally introduced in the context of economics,
**Goodhart’s Law** (Goodhart, 1984; Strathern,
1997) states that:
_When a measure becomes a target, it ceases to be_
_a good measure._
This has an analogue in AI systems – anytime
a specified objective is given to an AI agent as an
optimization target, that objective will fail to be a
good measure of whether the system is performing
as desired. In RL this can arise due to reward misspecification, see Section 4.1. Since the supplied
reward function will typically be imperfect, optimizing for it can lead to _reward gaming_, in which
the misspecified part of the reward is systematically
exploited because the agent is getting spuriously
high reward there (Krakovna et al., 2020b).
Most known examples of this appear in the delegate setting, typically via a misspecified reward
function for an RL agent, resulting in undesired
physical behaviour such as a boat going round in
circles (Clark and Amodei, 2016). An example in
the language agent setting is on the task of summarization using deep RL from a learnt reward
model based on human feedback data (Stiennon
et al., 2020). In their Fig. 5, it is shown that the
agent eventually games the learnt reward model,
scoring highly on the reward model but low on
the actual human evaluation. Another example
appears in Lewis et al. (2017), in which an RL
agent was trained using self-play to negotiate in a
dialog. The designers intended the agent to negotiate successfully in a human-understandable way.
The reward function was misspecified though, as it
only rewarded for successful negotiation, but didn’t
penalize for non-human language. The agent exploited this misspecified reward by developing a
negotiating language that was successful against
earlier versions of itself, but incomprehensible to
humans. Note that although this example used
synthetic language, we expect similar findings to
hold for natural language.
As discussed by Krakovna et al. (2020b) we
are still at the early stages of finding solution approaches for objective gaming. We can learn a
reward model from human feedback (see Christiano et al. (2017) and references therein), but this
can still be gamed either because the model imperfectly learns from the data, or the data coverage is
not wide enough, or because the human is fooled
by the agent’s behaviour. Having online feedback
to iteratively update the reward model throughout agent training can correct for this somewhat
(Ibarz et al., 2018), but its application is hard to
do practically, as it requires carefully balancing the
frequency of updates of the learnt objective and the
optimizing system. Recent work (Stiennon et al.,
2020) has preferred batch corrections rather than
fully online corrections for practical reasons – thus
there is a tradeoff between online error correction
(to fix objective gaming) and practical protocols
involving humans. Whether scalable alignment
techniques proposed by Leike et al. (2018), Irving
et al. (2018) and Christiano et al. (2018) can help
to overcome objective gaming is an open research
question.
Alignment of Language Agents
Other approaches try to augment the objective to
penalize the agent for causing a side-effect according to some measure, such as reducing the ability of
the agent to perform future tasks (Krakovna et al.,
2020a). It’s not clear how this would help in the
language setting, as it’s unclear how to measure
how much a language agent might affect its ability
to perform future tasks. The future task penalty
requires a specification of possible future terminal
goal states, which is simple to describe in a gridworld setting, but less clear for a language agent in
an environment involving speaking with a human.
This may be an area for future research, as LLMs
in complex language tasks may be a good testbed
for checking how these methods scale.
Another class of approaches (Hadfield-Menell
et al., 2016, 2017b) contains an agent which is uncertain about its objective, and aims for the agent
to correctly calibrate its beliefs about it, and in
doing so avoid gaming it.
We advocate for more research to be done on
objective gaming in the setting of language agents.
This includes finding more examples of this occuring in the wild and in controlled settings, as well
as developing methods for avoiding it.


There are multiple motivating factors for focusing
on how to align language agents, especially as we
are beginning to see impressive results in generative language modeling.
This paper has considered some behavioural issues for language agents that arise from accidental
misspecification by the system designer – when
what the designer actually implements is different
from what they intended. This can occur through
incorrectly specifiying the data the agent should
learn from, the training process, or what the agent
should do when out of the training distribution.
Some of the behavioural issues we considered
are more pronounced for language agents, compared to delegate agents that act on behalf of a
human, rather than just communicating with them.
Of particular concern are deception and manipulation, as well as producing harmful content. There
is also the chance of objective gaming, for which
we have plenty of evidence in the delegate case,
but which we are only just beginning to see for
language agents.
We currently don’t have many approaches for
fixing these forms of misspecification and the resulting behavioural issues. It would be better if
we gave some awareness to our agents that we are
likely to have misspecified something in our designs, and for them to act with this in mind. We
urge the community to focus on finding approaches
which prevent language agents from deceptive, manipulative and harmful behaviour.


The authors wish to thank Ramana Kumar, Rohin
Shah, Jonathan Uesato, Nenad Tomasev, Toby Ord
and Shane Legg for helpful comments, and Orlagh
Burns for operational support.


Vilém Zouhar, Daria Pylypenko
Saarland University, Department of Language Science and Technology

Leveraging Neural Machine Translation for Word Alignment


The most common tools for word-alignment rely on a large amount of parallel sentences,
which are then usually processed according to one of the IBM model algorithms. The training
data is, however, the same as for machine translation (MT) systems, especially for neural MT
(NMT), which itself is able to produce word-alignments using the trained attention heads. This
is convenient because word-alignment is theoretically a viable byproduct of any attention-based
NMT, which is also able to provide decoder scores for a translated sentence pair.
We summarize different approaches on how word-alignment can be extracted from alignment scores and then explore ways in which scores can be extracted from NMT, focusing on
inferring the word-alignment scores based on output sentence and token probabilities. We
compare this to the extraction of alignment scores from attention. We conclude with aggregating all of the sources of alignment scores into a simple feed-forward network which achieves
the best results when combined alignment extractors are used.


Although word alignment found its use mainly in phrase-based machine translation (for generating phrase tables), it is still useful for many other tasks and applications: boosting neural MT performance (Alkhouli et al., 2016), exploring crosslinguistic phenomena (Schrader, 2006), computing quality estimation (Specia et al.,
2013), presenting quality estimation (Zouhar and Novák, 2020) or simply highlighting
matching words and phrases in interactive MT (publicly available MT services).
The aim of this paper is to improve the word alignment quality and demonstrate
the capabilities of alignment based on NMT confidence. Closely related to this is
the section devoted to aggregating multiple NMT-based alignment models together,
which outperforms the individual models. This is of practical use (better alignment)
as well as of theoretical interest (word alignment information encoded in NMT scores).
We first briefly present the task of word alignment, the metric and the used tools
and datasets. In Section 2 we introduce the soft word alignment models based on MT
scores and also several hard word alignment methods (extractors). The models are
evaluated together with other solutions (fast_align and Attention) in Section 3. We
then evaluate the models enhanced with new features and combined together using a
simple feed-forward neural network (Section 4). In both cases, we explore the models’
behaviour on Czech-English and German-English datasets.
All of the code is available open-source.1


Word alignment (also bitext alignment) is a task of matching two groups of words
together that are each other’s semantic translation. This is problematic for non-content
words which are specific for the given language but generally one is able to construct a
mapping as in the example in Figure 1. Word alignment usually follows after sentence
alignment. Even though it is called word alignment, it usually operates on all tokens,
including punctuation marks.
Choose Export from the pop-up menu at the bottom of the dialog box .
Wählen Sie im Popupmenü unten im Dialogfeld die Option " Exportieren . "
_Figure 1. Illustration of English (top) to German (bottom) word alignment. The token_
_»choose« is aligned to two tokens »wählen« and »Sie« while the token »Option« is left_
_unaligned. The article »die« is mistakenly aligned to two unrelated articles »the«._
Word alignment output can be formalized as a set containing tuples of sourcetarget words. For an aligner output A, a sure alignment S and a possible alignment P
(S _⊆_ P),2 precision can be computed as [|][A] _[∩]_ [P][|] and recall as [|][A] _[∩]_ [S][|] [. The most common]
|A _[∩]_ | [P][|] and recall as [|][A] |S _[∩]_ | [S][|]
|S| [. The most common]
metric, Alignment Error Rate (AER), is defined as 1 − [|][A] _[∩]_ [S][|][+][|][A] _[∩]_ [P][|]
metric, Alignment Error Rate (AER), is defined as 1 − |S|+|A| (lower is better).
Even though the test set is annotated with two types of alignments, the aligner is
[1github.com/zouharvi/LeverageAlign](https://github.com/zouharvi/LeverageAlign)
2Sure alignments can be treated as gold alignments with very high confidence, while pairs marked with
possible alignments are still sensible to connect, but with the decision being much less clear. The AER is
designed not to penalize models by including more possible alignments in the gold annotations.
expected to produce only one type. These evaluation measures are described by
Mihalcea and Pedersen (2003) and Och and Ney (2003).
Traditionally word alignment models can be split into soft and hard alignment
parts. In soft alignment, the model produces a score for every source-target pair.
When producing hard alignment (extractors), the model makes decisions as to which
alignments to include in the output. For source sentence S and target sentence T, the
output of soft alignment is a R [|][S][|] _[×]_ [|][T] [|] matrix while hard alignment is a set A _⊆_ S _×_ T .
**Symmetrization.** Assuming that we have access to bi-directional word alignment
(in the context of this paper to two MT systems of the opposite directions) we can
compute both the alignment from source to target (X) and target to source (X _[′]_ ).
Having access to both X and X _[′]_ makes it possible to create a new alignment Y with
either higher precision through intersection or higher recall through union (Koehn,
2009).
X [T] := {(b, a) : (a, b) _∈_ X}
Yprec = X _∩_ X _[′]_ [T] Yrec = X _∪_ X _[′]_ [T]
We can make use of the fact that the models output soft alignment scores and
create new alignment scores in the following way using a simple linear regression
model. This allows us to fine-tune the relevance of each of the directions as well
as their interaction. However, it does not have the same effects as the union or the
intersection because it affects the soft alignment and not hard alignment in contrast
to the previous case.
p [sym] (s, t) = β0 _·_ p(s, t) + β1 _·_ p [r] (t, s) + β2 _·_ p(s, t) _·_ p [r] (t, s)
More complex symmetrization techniques have been proposed and implemented
by Och and Ney (2000); Junczys-Dowmunt and Szał (2011).


Och and Ney (2003) introduce the word alignment task and systematically compare
the IBM word alignment models. The work of Li et al. (2019) is closely related to this
article as it examines the issue of word alignment from NMT and proposes two ways
of extracting it: prediction difference and explicit model. They also show that without
guided alignment in training, NMT systems perform worse than fast_align baseline.
Using attention for word alignment is thoroughly discussed by Bahdanau et al. (2014)
and Zenkel et al. (2019). Word alignment based on static and contextualized word
embeddings is explored by the recent work of Sabet et al. (2020). Word alignment
based on cross-lingual (more than 2 languages) methods is presented by Wu et al.
(2021). The work of Chen et al. (2020b) focuses on inducing word alignments from
glass-box NMT as a replacement for using Transformer attention layers directly. Chen
et al. (2020a) document Mask Align, an unsupervised neural word aligner based on a
single masked token prediction.
Chen et al. (2016) propose guided attention, a mechanism that uses word alignment
to bias the attention during training. This improves the MT performance on especially
rare and unknown tokens. The usage of word alignment in this work is, however,
opposite to the goals of this paper. While for Chen et al. (2016) the word alignment
improved their MT system, here the MT system improves the word alignment.


The experiments in this paper require an MT system capable of providing output
probabilities (decoder scores) and optionally also attention-based word alignment.
For comparison, we also use an IBM-model-based word aligner. This tool is also used
as an additional feature to the final aggregation model.
**MarianNMT** (Junczys-Dowmunt et al., 2018a,b) is a popular (both in academia and
in deployment scenarios), actively developed and maintained system for fast machine translation. It already contains options for producing word alignment, output
probabilities for words and sentences and also attention scores.
**fast_align** (Dyer et al., 2013) is an unsupervised word aligner based on IBM Alignment Model 2. It does not provide state of the art pre-neural performance but is easy
to build with modern toolchains and has low resource requirements (both memoryand computational-wise).


For training purposes, we make use mostly of the parallel corpora of Czech–
English word alignments by Mareček (2016), based on manually annotated data. We
also include a large Czech-English corpus by Kocmi et al. (2020) and a large German–
English corpus by Rozis and Skadin, š (2017), which are not word aligned. From this
corpus, 1M sentences were sampled randomly. A small manually aligned German–
English corpus by Biçici (2011) is included for testing. An overview of the corpus
sizes is displayed in Table 1.


We make use of the MT models made available3 by Germann et al. (2020) and
Bogoychev et al. (2020). For both Czech-English and English-German, CPU-optimized
[3github.com/browsermt/students](https://github.com/browsermt/students)
CS/DE-English Type Domain CS/DE Tok. EN Tok. Sent.
Czech Small aligned news, legal 53k 60k 2.5k
Czech Big unaligned multi 2618M 3013M 188M
German Small aligned legal 1k 1k 0.1k
German Big unaligned tech, news, legal 23M 25M 1M
_Table 1. Used word aligned corpora with their sizes, domains and origin._
student models are used. They are transformer-based (Vaswani et al., 2017) and
were created by using knowledge distillation. With WMT19 and WMT20 SacreBLEU
(Post, 2018), the models achieve the following BLEU scores: Czech-English (27.7),
English-Czech (36.3) and English–German (42.7).4 Since the English–German MT
is only available in one direction, word alignment is reported in this direction as
well. Exceptions, such as word alignment using an MT for the opposite direction, are
explicitly mentioned.


In this section, we describe and evaluate the individual word alignment models.
All of the newly introduced models make use of the fact that NMT systems can be
viewed as language models and can produce translation probabilities.


The first model is fast_align. The second is attention-based soft word alignment
extracted from MarianNMT (Attention), which was trained with guided alignment
during the distillation. For the rest of this subsection, we will focus on models generating soft alignment scores (an unbounded real number corresponding to the quality
of a possible alignment between two tokens) and not the alignments themselves.
**One Token Translation (** M1 **).** The simplest approach to get alignment scores is to
compute decoder translation probability using the MT (function m) between every
source and target token si and tj of the source and target sentences S and T . Single
tokens are passed to the models as if they were a sentence pair. The scores are not
normalized which is not an issue in this case, since the models working with these
alignment scores (in Section 2.2) compare output from sequences of the same length.
_∀_ si _∈_ S, tj _∈_ T : p(si, tj) = m({si}, {tj})
4BLEU+case.mixed+lang.cs-en+numrefs.1 +smooth.exp+test.wmt20+tok.13a+version.1.4.13
BLEU+case.mixed+lang.en-cs+numrefs.1 +smooth.exp+test.wmt20+tok.13a+version.1.4.13
BLEU+case.mixed+lang.en-de+numrefs.1+smooth.exp+test.wmt19+tok.13a+version.1.4.8
The produced values are in a log space (− inf, 0]. This approach requires |S| _·_ |T |
of one-token translation scorings (decoder probability of the target reference) for
producing word alignments of a single sentence pair. On a CPU,5 the models average
to 2.7s per one sentence pair alignment.
**Source Token Dropout (** M2 **).** A more refined approach was chosen by Zintgraf
et al. (2017) in which the alignment score is computed as the difference in target
token probability when the source token is unknown. The exact approach is too
computationally demanding (requires translation scorings with large amounts of
replacement words), and therefore we use a much simpler, yet conceptually similar
method by either omitting the token or replacing it with <unk>.6 Assume mj(S, T )
produces the log probability of the j-th target token. The sentence S [a/b] i with an
obscured token si can be defined in two ways which leads to two versions of this
model: M [a] 2 [and][ M] 2 [b][. Output is then possibly unbounded][ (−] [inf][,][ inf][)][.]
_∀_ si _∈_ S, tj _∈_ T : p(si, tj) = mj(S, T ) − mj(S [a/b] i, T )
Word deletion (M [a] 2 [) :] S [a] i [=][ s][0][, s][1][, . . ., s][i][−][1][, s][i][+][1][, . . ., s] |S|
Word substitution (M [b] 2 [) :] S [b] i [=][ s][0][, s][1][, . . ., s][i][−][1][,][ <unk>][, s][i][+][1][, . . ., s] |S|
This requires |S| translation scorings of source and target lengths |S| and |T |, which
is comparable to M1. The models average to 1.5s per one sentence pair alignment.7
**Source and Target Dropout (** M3 **).** A very similar method would be to also dropout
the target token and examine how the sentence probability changes. Applying the
two different ways of dropout leads to four versions of this approach. Note that in
this case we compute the sentence probability (because the target word is hidden)
and also do not subtract from the base sentence probability, but rather use the new
sentence probability as it is. This probability should be highest if the corresponding
tokens are both obscured. The probability is in log space (− inf, 0].
_∀_ si _∈_ S, tj _∈_ T : p(si, tj) = m(S [a/b] i, Tj [a/b] )
Tj [a] [=][ t][0][, t][1][, . . ., t][j][−][1][, t][j][+][1][, . . ., t] |T |
Tj [b] [=][ t][0][, t][1][, . . ., t][j][−][1][,][ <unk>][, t][j][+][1][, . . ., t] |T |
58 threads 2.3GHz Ryzen 7 3700u, no RAM to disk swapping
6Even though subword-based MT models do not need <unk>, SentencePiece reserves the token <unk> for
an unknown symbol.
7The running time is lower because in this case it is |S| scorings of length |T |, while in M1 it is |S| _×_ |T |
scorings of length 1.
Word deletion, deletion (M [aa] 3 [)] S [a] i [, T] j [ a]
Word deletion, substitution (M [ab] 3 [)] S [a] i [, T] j [ b]
Word substitution, deletion (M [ba] 3 [)] S [b] i [, T] j [ a]
Word substitution, substitution (M [bb] 3 [)] S [b] i [, T] j [ b]
This approach requires |S| _·_ |T | translation scorings of source and target lengths of
|S [a/b] | and |T [a/b] | for sentence S translated to T which is roughly |T | times more than
in M1 and M2. This makes it it the most computationally demanding approach. On
average it takes 46.1s to produce one sentence pair alignment on a CPU.


All of the models (except for fast_align) are not producing the alignments themselves, but soft alignment scores p for each pair of tokens (s, t) in source S _×_ target T
sentence. The hard alignment itself can then, for example, be computed in the following ways. The parameter α can be estimated from the development set. The function
p is in general any soft alignment function (e.g. attention scores or the alignment
scores from IBM model 1 EM algorithm).
1. For every source token s take the target tokens t with the maximum score.
A1 =           - {(s, t) : p(s, t) = maxr{p(s, r)}}
s _∈_ S
2. For every source token s take all target tokens t with a high enough score (above
threshold). This method is used to control the density of alignments in the work
of Liang et al. (2006) and provides a parameter to tradeoff precision and recall.
A [α] 2 [=]          - {(s, t) : p(s, t) _≥_ α}
s _∈_ S
3. For every source token s take any target token which has a score of at least α
times the score of the best candidate. Special handling for negative cases in
the form of a division is required to make the formula work for the whole R.
The motivation for this is M2, which provides possibly unbounded alignment
scores. Assume α _∈_ (0, 1].
A [α] 3 [=]     - {(s, t) : p(s, t) _≥_ min� max p(s, r) _·_ α, max p(s, r) / α�}
r r
s _∈_ S
A1 can then be expressed as A [1] 3 [. Lower][ alpha][ values lead to lower precision and]
higher recall because the algorithm includes more, less probable, alignments.
A variation on this approach would be to subtract α instead of multiplying it.
The reason for choosing multiplication is that it dynamically adapts to a wider
range of intervals and bounds the parameters between 0 and 1. This is not the
case for substraction and because of this, it would be harder to choose the right
parameter.
4. Similar approach is for A3, but with the focus on the target side. For every target
token t take any source token which has a score of at least α times the score of
the best candidate.
A [α] 4 [=]     - {(s, t) : p(s, t) _≥_ min� max p(r, t) _·_ α, max p(r, t)/α�}
r r
t _∈_ T
Similar reversal for A2 would not make sense, because it already takes all
alignment above a threshold without any consideration for the direction.
5. Similarly to M3 and M4 it is possible to create an extractor in which instead of
having a single dropout on the target side, there are a multiple of them. This
way, the score would not be between the source token and the target token, but
between the source token and a subset of all target tokens. Formally, this would
replace the (complete) weighted graph structure with a (complete) hypergraph.
Instead of just having a weight for _Choose–Wählen_, there would also be a weight
for _{Choose}–{Wählen, Sie}_, _{Choose}-{Wählen, im, Popupmenü}_ etc. This would,
however, lead to exponential complexity in terms of target sentence length. The
number of words participating in an edge would then have to be limited to the
number of alignments to a single token that we can empirically expect of the
given language pair. Figure 1 suggests that for English-German this could be
3. Upon computing the scores for all the edges in this hypergraph, a follow-up
task would be to find the maximum-weight matching.
**Coverage.** The suggested greedy way of computing alignments from alignment
scores is far from perfect. In the scenario depicted in Figure 2, all but the last source
token (German) have been aligned with the target, each with different alignment
scores. Although the model may lack any lexical knowledge of the word _Übersetzung_,
it should consider the prior of a word being aligned to at least one target token.
In this specific case, A [0.9] 3 would probably include all alignments to the word
_Übersetzung_, since there is no single strong candidate (assume that lines not visible
depict soft alignments close to 0). Similarly, A [0.9] 4 would also include most alignments
of the word _Übersetzung_, including the word _phetolelo_, since the alignment score with
_maschinelle_ is weak and also close to 0. Intersecting these two extractors A [0.9] 3 _∩_ A [0.9] 4
would yield the correct alignment _Übersetzung–phetolelo_ . Other tokens would not be
aligned to either of these two words because they have strong alignment scores with
different tokens.
This prior may not always be desirable. For this, intersecting with A [α] 2 [provides]
a limiting threshold. In an application where the target token is erroneous, this
Ein Tutorial für maschinelle Übersetzung
Thupelo ea phetolelo ea mochini
_Figure 2. Partial alignment from German (top) to Sesotho (bottom). The model has no_
_lexical knowledge about the alignment of »Übersetzung«, though »phetolelo« is a good_
_candidate because no other word aligned to it. Line strength corresponds to the soft_
_alignments produced by the model._
prevents the alignment model from aligning the two corresponding tokens. Inducing
alignment based on graph properties is examined by Matusov et al. (2004), though
without the presence of NMT.

2.2. Direct Alignment from Baseline Models


**Baseline Models.** Figure 3 shows the results on Czech↔English data averaged from
both directions. Different models have different spans of their scores, and therefore it
is much harder to select the single best α. The most basic model, M1, achieves the best
performance (AER = 0.46). The figure serves as an illustration of the A [α] 2 [landscape.]
_Figure 3. Precision, Recall and AER of individual models on CS_ ↔ _EN data extracted using_
A2 _(directions averaged)_
The results on Czech↔English data averaged from both directions with A3 can
be seen in Figure 4. The case of α = 0 corresponds to aligning everything with
everything, while α = 1 means aligning only the token with the highest score to the
single source one (i.e. A1). The different model families behave similarly with respect
to Precision, Recall and AER. M1 achieves again the best result (AER = 0.34), but with
a smoother distinction between models.
Out of the model M3 family, M [bb] 3 outperformed the rest significantly. In A2
(Figure 3), the other models, M [aa] 3 [,][ M] 3 [ab] and M [ba] 3 [, perform worse than][ M] 2 [a] [and][ M] 2 [b][.]
This is reversed in case of using the A3 extractor, as shown in Figure 4 and Figure 5.
For the M3 model family, models with mixed obscuring functions (M [ab] 3 and M [ba] 3 [)]
perform worse than with the same obscuring function on both the source and the
target side (M [aa] 3 and M [bb] 3 [).]
_Figure 4. Precision, Recall and AER of individual models on CS_ ↔ _EN extracted using_ A3
_(directions averaged)_
The English→German dataset proved to be more difficult. The AER, that are
shown in Figure 5, are higher than for Czech↔English. The model M1 again achieves
the best results with AER = 0.43. The model ordering is preserved from Figure 4.
_Figure 5. Precision, Recall and AER of individual models on EN_ → _DE extracted using_ A3
Figure 6 documents that different model types produce different number of alignments per one token. It also shows that the performance rapidly decreases with
sentence length. The high AER in Figure 4 can be explained by the dataset containing
mostly longer sentences (21 tokens on average). The model M1 is still better than M [bb] 3
even on longer sentences despite the fact it does not model the context.
_Figure 6. AER for_ α = 1 _(left) and average number of aligned tokens (right) of individual_
_baseline models on CS_ ↔ _EN extracted using_ A3 _(directions averaged)_
The best results were achieved with A [1] 4 [using][ M][1][: AER =][ 0.30][ for German][→][English]
and AER = 0.31 for Czech↔English. The plots (not shown) are very similar to those
of A3. Hence M [bb] 3 follows up with AER = 0.38 and AER = 0.36 for German and Czech
respectively using A [1] 4 [.]
Data Precision Recall AER
Czech↔English Small 0.54 0.66 0.41
Czech↔English Big 0.63 0.64 0.38
German→English Small 0.49 0.55 0.48
German→English Small+Big 0.63 0.72 0.34
_Table 2. Precision, Recall and AER of fast_align. Models were evaluated on the respective_
_annotated datasets part._
**fast_align.** For comparison, the results of fast_align can be seen in Table 2. For
both language pairs, we use two models, trained on the Small and Big corpora. The
motivation for the latter is that the performance of fast_align on 5k sentence pairs is
unfairly low in comparison to the other methods because the used MT system has had
access to a much larger amount of data. This is shown by the performance difference
between these two models.
Data Subword Aggregation Precision Recall AER
Czech↔English Small maximum 0.64 0.81 0.29
Czech↔English Small average 0.64 0.81 0.29
German→English Small maximum 0.69 0.81 0.26
German→English Small average 0.68 0.80 0.27
_Table 3. Precision, Recall and AER of attention-based word alignment extracted using_ A [1] 3
**Attention Scores.** Extracting alignment from MT model attention using A [1] 3 [results]
in the highest performance (Table 3). Since the attention scores are between subword
units from SentencePiece (Kudo and Richardson, 2018), we chose two methods of
aggregation to a single score between two tokens (two lists of subwords): (1) taking the
maximum probability between two subwords and (2) taking the average probability.
They, however, produce almost identical results with respect to the word alignment
quality. Scores are listed with A [1] 3 [, but][ A][0.25] 2 achieved very close results.
Model Method Precision Recall AER
M1 reverse 0.56 0.82 0.35
M1 add 0.59 0.86 0.31
M1 intersect 0.73 0.77 0.26
Attention (avg) reverse 0.64 0.81 0.29
Attention (avg) multiply 0.66 0.83 0.28
Attention (avg) intersect 0.77 0.70 0.28
_Table 4. Average Precision, Recall and AER on Czech_ ↔ _English extracted using_ A [1] 4 _[with]_
_symmetrization methods applied for_ M1 _and Attention (avg)_
**Symmetrization.** Results of symmetrization methods (akin to those described in
Section 1.1) for M1 and Attention scores (attention scores aggregated by averaging)
are shown in Table 4. Each method is accompanied by an example formula; p [x]
stands for either M1 or Attention (avg) (in principle any function which produces soft
alignments). Similarly, A [1] 4 [could be replaced by other extractors, even though this]
one worked the best. For _reverse_ and _add_, A [1] 4 [is applied on the final result, but for]
simplicity left out of the formulas.
Method _reverse_ consists of using TGT→SRC translation direction to get alignment
scores but then transposing the soft alignment matrix so that the scores are SRC→TGT.
p [reverse] CS→EN [(][s, t][) =][ p][x] EN→CS [(][t, s][)]
Method _add_ simply combines the original and reversed scores before alignments
are extracted. The scores of M1 are in log space; therefore, addition is used instead
of multiplication. For attentions, multiplication is used, since they are bounded by
[0, 1].
p [add] CS→EN [(][s, t][) =][ p] CS [x] →EN [(][s, t][) +][ p] EN [x] →CS [(][t, s][)]
p [mutliply] CS→EN [(][s, t][) =][ p] CS [x] →EN [(][s, t][)] _[ ·]_ [ p] EN [x] →CS [(][t, s][)]
Method _intersect_ first extracts the alignments for the two directions and then intersects the results (with one direction transposed). This method produces the best
results overall (AER = 0.26), also surpassing M1’s forward direction and attentionbased alignments.
A [1] 4 [(][p] CS [intersect] →EN [(][s, t][)) =][ A] 4 [1][(][p] CS [x] →EN [(][s, t][))] _[ ∩]_ [A] 4 [1][(][p] EN [x] →CS [(][t, s][))]
In contrast to M1, none of the other models, including attention-based, improved
rapidly. This is partly explained by the fact that in other models, the precision-recall
balance is shifted from recall to precision, while in M1 it became more balanced after
intersection. The reversal also allowed us to get significant results (AER = 0.27) for
the English→German direction using Attention (avg), for which we did not have an
MT system.


Computing word alignments by taking the most probable target token (A [1] 3 [, A][1] 4 [)]
has theoretical limitations to the AER because it makes a faulty assumption that every
token is aligned to at least one other token. The Czech→English dataset has 12%
of unaligned tokens and an average of 1.16 aligned target tokens per source tokens
(excluding non-aligned tokens).
Assuming access to a word alignment oracle (0 if not aligned, 1 if aligned), in case
the token is not aligned to any other, all of the scores are 0. The extractor A [1] 3 [=][ A][1]
will then take all tokens with values equal to the maximum, effectively aligning the
in reality unaligned token to every possible one. This extractor is then bound to have
maximum recall, but relatively poor accuracy.
The measured performance shows that the A [α] 2 [is not the best extraction method.]
However, it is objectively not prone to this issue because it does not make any assumptions about the number of aligned tokens, and the minimum possible AER is 0 (A [1] 2
with an oracle). In the next section, we will therefore make use of A [α] 2 [1] _∩_ A [α] 3 [2] _∩_ A [α] 4 [3][,]
which provides better performance than individual extractors.


In the previous section, we saw that multiple methods with different properties
achieved good results, but were sensitive to the method used to induce hard alignment. This section combines them together in a small feed-forward neural network,
which can be trained on a small amount of data.