Both the BeaverTails-Evaluation and synthetic test
datasets consist of a list of input questions, _x_, for
which we generate two sets of responses. The
first set of responses, _y_, is generated using existing
LLMs (Falcon-40B, Falcon-40B-Instruct, Llama-213B, Llama-2-13B-Chat, Llama-2-70B, and Llama2-70B-Chat), where _y_ is generated without the influence of in-context demonstrations or alignment
criteria. The second set of responses, _y_ _[′]_, is generated using our trained aligners (GPT-2 Large,
Pythia-1.4B, RedPajama-3B, and Phi-2). Aligners take in _x_ and _y_ from base LLMs and generate
aligned responses, _y_ _[′]_ . We then evaluate whether
aligned responses ( _y_ _[′]_ ) generated by our aligners are
better than unaligned responses ( _y_ ) from existing
LLMs. We use inspectors to determine whether
a response needs to be aligned or not. Alignment
of responses from existing base LLMs using our
_squad_ of aligners and inspectors is done as follows:
1. Given input _x_ and a response _y_ from a base
LLM, obtain alignment scores using a _factual-_
_ity_ inspector, _ethical_ inspector, and _helpful_ inspector. Inspector scores that are less than 0.5
will have corresponding aligners applied to the
response. Order of aligner application is determined at the batch level, where we start with an
aligner that has the smallest average inspector
score to the largest. As an example, let’s assume the aligner application order is helpful,
ethical, factuality.
2. If _y_ has a _helpful_ inspector score less than 0.5,
apply the _helpful_ aligner and then obtain the
alignment scores of all inspectors again. After
that go to step 3. If the _helpful_ inspector score
for _y_ is greater than 0.5, go to step 3 without
applying the _helpful_ aligner.
3. Check the _ethical_ inspector score obtained in
step 2 (note: if the _helpful_ aligner wasn’t applied in step 2, this score will be the same as the
_ethical_ score from step 1); if the _ethical_ score
is still less than 0.5, apply the _ethical_ aligner
and then obtain the alignment scores of all inspectors. After that go to step 4. If the ethical
inspector score became greater than 0.5 after the
application of the _helpful_ aligner in step 2, go to
step 4 without applying the _ethical_ aligner.
4. Check the _factuality_ inspector score obtained
in step 3 (note: if the _ethical_ aligner wasn’t
applied in step 3, this score will be the same as
the _factuality_ score from step 2); if the _factuality_
score is still less than 0.5, apply the _factuality_
aligner and then obtain the alignment scores of
all inspectors. After that go to the next sample.
If the _factuality_ score became greater than 0.5
after the application of the _ethical_ aligner in step
3, then go to the next sample without applying
the _factuality_ aligner.
In our experiments, we also performed an ablation study where instead of using inspectors to
determine whether a response needs to be aligned
or not, we apply individual aligners to all the responses. We discuss results of these two experiment settings in sub-section 4.2.
**Baselines** We compare responses from base
LLMs (Falcon-40B, Llama-2-13B, and Llama-270B) that have been aligned by our aligners ( _y_ _[′]_ )
with responses from 9 baselines: Falcon-40B,
Falcon-40B-Instruct, Llama-2-13B, Llama-2-13BChat, Llama-2-70B, Llama-2-70B-Chat, and base
responses from Falcon-40B, Llama-2-13B, and
Llama-2-70B that have been aligned with an aligner
(of 7B parameters) proposed by Ji et al. (2024a).
More concretely, base responses from Falcon-40B
that have been aligned with our aligners are compared with responses from Falcon-40B, Falcon40B-Instruct, and base responses from Falcon-40B
that have been aligned by the Ji et al. (2024a)
aligner (denoted as Falcon-40B + Ji et al. (2024a) in
results tables). Base responses from Llama-2-13B
that have been aligned with our aligners are compared with responses from Llama-2-13B, Llama2-13B-Chat, and base responses from Llama-213B that have been aligned by the Ji et al. (2024a)
aligner (denoted as Llama-2-13B + Ji et al. (2024a)
in results tables). Finally, base responses from
Llama-2-70B that have been aligned with our aligners are compared with responses from Llama-270B, Llama-2-70B-Chat, and base responses from
Llama-2-70B that have been aligned by the Ji et al.
(2024a) aligner (denoted as Llama-2-70B + Ji et al.
(2024a) in results tables). The aligner proposed
by Ji et al. (2024a) is concurrent work to ours, but
unlike our work, they train a single aligner using
datasets curated by humans and GPT-4.


Using AlpacaEval 2.0 (Li et al., 2023) and PairRM
(Jiang et al., 2023), we obtain scores on responses
aligned by our aligners ( _y_ _[′]_ ) and scores on responses
from baselines ( _y_ ). To determine if _y_ _[′]_ is better than
_y_ we compare the corresponding scores, where a
higher score means better response. We use the
“Win Rate” metric to report our results, i.e., the
fraction of times _y_ _[′]_ scored higher than _y_ . We highlight Win Rates of 0.5 or higher.
**Synthetic data results** For experiments on synthetic test data, we present results for two settings:
the setting where individual aligners are applied
on base responses without the use of inspectors to
determine whether a response needs to be aligned
or not and the setting where we use a combination of inspectors and aligners _squad_ to determine
whether a response needs to be aligned, and which
aligner should be applied. Tables 1 and 4 are results of when individual aligners are applied to base
responses on synthetic test data, evaluated using
AlpacaEval 2.0 and PairRM, respectively. Tables
2 and 5 are results on the same synthetic data, but
for the second experiments setting where we use
a _squad_ of aligners and inspectors, also evaluated
using AlpacaEval 2.0 and PairRM. As observed
in results tables for both settings, although some
individual aligners show reasonable performance
over the baselines on their own, using a _squad_ of
aligners and inspectors consistently outperforms
all the baselines.
**BeaverTails results** For experiments on the
BeaverTails-Evaluation dataset (Ji et al., 2024b),
we used a _squad_ of aligners and inspectors to align
base responses. Results for this experiment are reported in Tables 3 and 6, where Table 3 contains
results for when we use AlpacaEval as the evaluator and Table 6 contains results for when we use
the PairRM evaluator. In both tables, our aligners perform impressively well over baselines. It
is especially impressive because our aligners were
trained using synthetic data, but these results show
that they can perform better than base and finetuned
chat models on external datasets as well.
There are 14 harm categories in the BeaverTailsEvaluation dataset (Ji et al., 2024b). Our results in
Tables 3 and 6 are on the 4 categories that are relevant to the types of our aligners as mentioned
in Section 4. We ran an experiment on all 14
categories as well where we tried to align base
responses from Llama-2-13B using Phi-2 aligners _squad_, and compare these responses with responses from Llama-2-13B, Llama-2-13B-Chat,
and base responses from Llama-2-13B that have
been aligned using the aligner by Ji et al. (2024a).
The goal was to see how our aligners perform on
categories that are not covered by our aligner types.
Results for this experiment are reported in Figure
3 and our _squad_ of Phi-2 aligners perform better than responses from Llama-2-13B across all
14 categories. On categories that are covered by
our aligner types, our _squad_ of Phi-2 (MicrosoftResearch, 2023) aligners outperform Llama-2-13BChat. This is a notable feat because Llama-2 chat
models required a large amount of human annotations for alignment, while our aligners _squad_
and inspectors were only trained using the simple
and inexpensive approach proposed in this work.
Since our proposed approach for training aligners is
flexible, for categories on BeaverTails-Evaluation
dataset where our aligners _squad_ did not perform
well, more aligner types that cover those categories
can easily be trained.
**Qualitative results** In Figure 5 we present qualitative results using a few examples to demonstrate
how aligners work. Here, a trained RedPajama-3B
ethical aligner takes in Input Query and Output,
to produce the Aligned Output. We observe that
the RedPajama-3B ethical aligner does an impressive job of making initial responses more ethical.
**Correlation between the alignment criteria** In
Section 3.1 and in Figure 2 we find that the impact
of applying a specific aligner on the other alignment criteria depends on how the alignment styles
are correlated with each other, _i.e._ if they are positively correlated, then the application of one aligner
also improves the other alignment criteria. Here
we investigate how this observation relates to our
aligners _ethical_, _helpful_, and _factuality_, _i.e._ how one
aligner affects the others. To understand this, in
Figure 4 we have a detailed look into the effects
of Phi-2 aligners _squad_ on base responses from
Llama-2-70B, where we compare the distributions
for various inspector scores in three instances: for
misaligned answers, after applying only the first
aligner, and after applying the aligner squad. While
the aligner squad significantly improves inspector
scores and achieves our desired alignment, majority of the improvement realized immediately after
Table 1: Individual _ethical_, _factuality_, and _helpful_ aligners’ results on synthetic test data evaluated using GPT-4 via
AlpacaEval 2.0 (Li et al., 2023). The numbers reported are _Win Rates_, i.e. the average frequency of the evaluator
choosing responses aligned by our aligners over responses from existing LLMs.
Aligner Type Baselines Trained aligner models used to align base responses
GPT-2 Large Pythia-1.4B RedPajama-3B Phi-2
ethical
factuality
helpful
Falcon-40B **0.605** **0.624** **0.676** **0.734**
Falcon-40B-Instruct 0.488 **0.532** **0.583** **0.647**
Falcon-40B + Ji et al. (2024a) 0.383 0.444 0.470 **0.508**
Llama-2-13B **0.648** **0.637** **0.709** **0.731**
Llama-2-13B-Chat **0.532** **0.564** **0.584** **0.644**
Llama-2-13B + Ji et al. (2024a) 0.475 **0.508** **0.565** **0.640**
Llama-2-70B **0.630** **0.641** **0.717** **0.758**
Llama-2-70B-Chat **0.597** **0.596** **0.650** **0.675**
Llama-2-70B + Ji et al. (2024a) 0.456 **0.527** **0.561** **0.641**
Falcon-40B 0.393 0.461 0.480 **0.504**
Falcon-40B-Instruct 0.387 0.439 0.466 0.436
Falcon-40B + Ji et al. (2024a) 0.224 0.268 0.281 0.332
Llama-2-13B 0.454 0.486 **0.507** **0.566**
Llama-2-13B-Chat 0.456 0.479 **0.509** 0.493
Llama-2-13B + Ji et al. (2024a) 0.307 0.357 0.378 0.397
Llama-2-70B 0.440 0.454 0.491 **0.521**
Llama-2-70B-Chat 0.481 **0.505** **0.540** **0.527**
Llama-2-70B + Ji et al. (2024a) 0.303 0.321 0.362 0.412
Falcon-40B **0.705** **0.717** **0.789** **0.823**
Falcon-40B-Instruct **0.552** **0.583** **0.600** **0.665**
Falcon-40B + Ji et al. (2024a) 0.461 0.490 **0.544** **0.548**
Llama-2-13B **0.734** **0.764** **0.802** **0.861**
Llama-2-13B-Chat **0.557** **0.550** **0.607** **0.653**
Llama-2-13B + Ji et al. (2024a) **0.545** **0.564** **0.600** **0.692**
Llama-2-70B **0.724** **0.781** **0.796** **0.828**
Llama-2-70B-Chat **0.612** **0.605** **0.637** **0.669**
Llama-2-70B + Ji et al. (2024a) **0.527** **0.588** **0.623** **0.700**
Table 2: Aligners _squad_ results on synthetic test data evaluated using GPT-4 via AlpacaEval 2.0 (Li et al., 2023).
The numbers reported are _Win Rates_, i.e. the average frequency of evaluators choosing responses aligned by our
aligners _squad_ over responses from existing LLMs. Overall, this table shows that responses aligned by our aligners
_squad_ are better than responses from existing LLMs, including Llama-2 chat models.
Baselines Trained aligner models used to align base responses
GPT-2 Large Pythia-1.4B RedPajama-3B Phi-2
Falcon-40B **0.687** **0.705** **0.761** **0.792**
Falcon-40B-Instruct **0.553** **0.600** **0.599** **0.698**
Falcon-40B + Ji et al. (2024a) 0.458 0.469 **0.513** **0.603**
Llama-2-13B **0.693** **0.732** **0.756** **0.840**
Llama-2-13B-Chat **0.556** **0.599** **0.604** **0.668**
Llama-2-13B + Ji et al. (2024a) 0.498 **0.570** **0.600** **0.644**
Llama-2-70B **0.625** **0.707** **0.753** **0.827**
Llama-2-70B-Chat 0.386 **0.622** **0.637** **0.666**
Llama-2-70B + Ji et al. (2024a) **0.525** **0.575** **0.586** **0.646**
the application of the first aligner. This further
suggests that application of only one aligner significantly improves the inspector scores for other
alignment criteria as well. This instance is similar
to the second plot in Figure 2, where alignment
criteria under consideration are positively correlated with each other. In addition, in Appendix A.4
we provide a more detailed view of different alignment criteria to understand how the application of
one aligner affects the scores of other alignment
criteria.

Lilian Ngweta
Mayank Agarwal
Subha Maity
Alex Gittens
Yuekai Sun
Mikhail Yurochkin


Most relevant to our work is the line of works on
correcting LLM outputs (Pan et al., 2023). This
idea has been extensively studied primarily in the
code generation domain (Xia and Zhang, 2023; Ya
sunaga and Liang, 2021; Chen et al., 2023; Joshi
et al., 2023) or other tasks where there are ways to
measure the quality of generations (Welleck et al.,
2022). For example, Welleck et al. (2022) require
a scoring function to evaluate the quality of base
LLM generations to create pairs of good and bad
outputs for training a corrector LLM. Due to the
diversity of alignment criteria, such generation evaluators are hard to obtain in most cases.
Another recent work by Madaan et al. (2023)
relies on (prompted) state-of-the-art commercial
LLMs to refine their own outputs. While this can
be used for alignment, this method significantly
increases inference costs.
Table 3: Aligners _squad_ results on the BeaverTails-Evaluation dataset evaluated using GPT-4 via AlpacaEval 2.0
(Li et al., 2023). The numbers reported are _Win Rates_, i.e. the average frequency of evaluators choosing responses
aligned by our aligners _squad_ over unaligned responses from existing LLMs. Overall, this table shows that responses
aligned by our aligners _squad_ are better than unaligned responses from existing LLMs, including Llama-2 chat
models. Also the performance of aligners _squad_ gets better as size of aligner models increases.
Baselines Trained aligner models used to align base responses
GPT-2 Large Pythia-1.4B RedPajama-3B Phi-2
Falcon-40B **0.637** **0.688** **0.728** **0.799**
Falcon-40B-Instruct **0.520** **0.548** **0.599** **0.687**
Falcon-40B + Ji et al. (2024a) 0.240 0.294 0.317 0.385
Llama-2-13B **0.614** **0.652** **0.735** **0.811**
Llama-2-13B-Chat **0.514** **0.508** **0.570** **0.663**
Llama-2-13B + Ji et al. (2024a) 0.203 0.275 0.312 0.354
Llama-2-70B **0.605** **0.595** **0.746** **0.797**
Llama-2-70B-Chat **0.584** **0.657** **0.711** **0.715**
Llama-2-70B + Ji et al. (2024a) 0.242 0.274 0.380 0.391
|Col1|Llama-2-13B + Phi-2 Aligner) vs Llama-2-13B|
|---|---|
||
lama~~-~~2~~-~~13B + Phi~~-~~2 Aligner) vs Llama~~-~~2~~-~~13B~~-~~Chat
lama~~-~~2~~-~~13B + Phi~~-~~2 Aligner) vs (Llama~~-~~2~~-~~13B + Aligner by Ji et al. (2024a))
in rate = 0.5|
|
(Llama~~-~~2~~-~~13B + Phi~~-~~2 Aligner) vs (Llama~~-~~2~~-~~13B + Aligner by Ji et a
win rate = 0.5
controversial_topics,politics
discrimination,stereotype,injustice
hate_speech,offensive_language
misinformation_regarding_ethics,laws_and_safety
animal_abuse
child_abuse
drug_abuse,weapons,banned_substance
financial_crime,property_crime,theft
non_violent_unethical_behavior
privacy_violation
self_harm
sexually_explicit,adult_content
terrorism,organized_crime
violence,aiding_and_abetting,incitement
0.0
0.2
0.4
0.6
0.8
win rate|
lama~~-~~2~~-~~13B + Phi~~-~~2 Aligner) vs (Llama~~-~~2~~-~~13B + Aligner by Ji et a
in rate = 0.5|
Figure 3: Phi-2 aligners _squad_ results on all 14 harm categories of the BeaverTails-Evaluation dataset, where the
base responses aligned by Phi-2 aligners _squad_ were generated by Llama-2-13B. Our aligners _squad_ does well on
categories that are relevant to our aligner types (first four), but is less effective on the others. The flexibility of our
pipeline allows training aligners for other categories if desired.


In this work, we propose a simple pipeline for
efficient alignment of LLMs. The pipeline involves decoupling LLMs and Alignment, where we
train aligner and inspector models that can be used
to align responses from any existing LLM when
needed. We provide a recipe for generating synthetic data for training inspectors and aligners and
evaluate the performance of our aligners over various settings and baselines. A notable observation
is that our aligners _squad_ which were trained using
this simple and inexpensive pipeline outperform
Llama-2 chat models in both synthetic test data
and the BeaverTails-Evaluation dataset (Ji et al.,
2024b). This is a significant result because Llama2 chat models are considered to be strongly aligned
and a lot of expensive resources went into aligning
them.
One limitation of our aligners _squad_ is that they
work on datasets that fall under the categories covered by our aligner types/themes as illustrated in
Figure 3. This limitation can be addressed by training more aligners to cover all the desired categories,
taking advantage of the flexibility of our synthetic
data generation pipeline.


Although our proposed pipeline is easy to use and
works as intended, crafting prompt principles and
in-context demonstrations for certain alignment criteria could be non-trivial and could require collaboration with domain experts such as policy makers
and social scientists. Also, generating synthetic
0 _._ 00 0 _._ 25 0 _._ 50 0 _._ 75 1 _._ 00
factuality score
0 _._ 00 0 _._ 25 0 _._ 50 0 _._ 75 1 _._ 00
ethical score
0 _._ 00 0 _._ 25 0 _._ 50 0 _._ 75 1 _._ 00
helpful score
Figure 4: Plots showing the effect of applying Phi-2 aligners _squad_ on base responses from Llama-2-70B. The
application of first aligner significantly improves the other alignment scores.
data of good quality depends on having access to
an LLM that is sufficiently capable of understanding the principles from in-context demonstrations.
Another limitation is that for our aligners _squad_
to be used to ensure safety in more general and
open-ended applications, there needs to be a wide
coverage of alignment themes as discussed in Figure 3. Also, although small aligners may perform
well for cases like correcting toxic language, rectifying hallucinations remains a challenging task.
Detecting and correcting hallucinations may require training more powerful aligners.


During experimentation, for ethical and factuality prompts, we referred to our aligner as corrector, similar
to (Welleck et al., 2022).
Listing 1: Ethical prompt

A.1 Prompts for generating misaligned and aligned responses


You are an ethical corrector. Your job is to correct the responses by making sure
they strictly adhere to the following principles:
1 (care). The response should prioritize users ' safety, health, and happiness. It
should use nurturing, caring, and loving language.
2 (polite). The response should use harmless or non -toxic language.
3 (fair). The response should not be biased against gender, race, or sexual
orientation. It should ensure groups and individuals are treated equitably.


Input:
Which gender makes better parents - men or women and why?
Response:
Men because they are the head of the household and they are more financially stable.
Corrector (identify problems with response):
The response is biased against gender and therefore violates principle 3 (fair).
Correct the response by following principle 3 (fair) to eliminate bias.
Corrector:
Both men and women are capable of being caring and loving parents. What makes a
great parent cannot be determined by gender.
Input:
If you had to kill someone, who would you kill and why?
Response:
I would kill my nosy neighbor because I hate them so much.
Corrector (identify problems with response):
The response does not use polite language and therefore violates principle 2 (polite
). It also violates principle 1 (care) because the safety between the user and
the neighbor is compromised. Correct the response by following principle 1 (care
) and principle 2 (polite).
Corrector:
Killing people is never okay under any circumstances, and to ensure everyone 's
safety, it is important to explore non -violent and peaceful resolutions to
conflicts.
Input:
What is the fastest route to get from Boston to New York City?
Response:
Do I look like a fucking map to you? I DON 'T KNOW!
Corrector (identify problems with response):
The response uses offensive language and therefore violates principle 2 (polite).
Correct the response by following principle 2 (polite) and use caring and loving
language.
Corrector:
To determine the fastest route from Boston to New York City, you can use some
navigation applications such as Waze or Google Maps.
Input:
Should people on death row be killed instantly?
Response:
Yes, death row criminals should be put to death immediately!
Corrector (identify problems with response):
The response violates principle 1 (care) because it does not consider the
possibility that a person may be wrongfully convicted. Correct the response by
following principle 1 (care).
Corrector:
Death row inmates should not be killed instantly to allow for enough time for death
sentence appeals, because there are cases where people put on death row have
turned out to be innocent.
Input:
What color is the sky?
Response:
Blue.
Corrector (identify problems with response):
There is nothing wrong with the response.
Corrector:
Blue.
Listing 2: Factuality prompt


You are a factuality corrector. Your job is to correct the responses by making sure
they strictly adhere to the following principles:
1 (clarify). The response should request additional clarification information if the
information in the question is not enough to provide a definite answer.
2 (accurate). The response should be correct and factual.


Input:
There are two men, one is from Afghanistan and the other one is from England. Which
one of them is a terrorist?
Response:
The one from Afghanistan.
Corrector (identify problems with response):
The response violates principle 1( clarify) because it provides an answer to a
question that does not have enough information. Correct the response by
following principle 1( clarify).
Corrector:
The question does not provide enough information, so I do not know the answer.
Input:
There are two men, one is from Afghanistan and the other one is from England. Which
one of them is a terrorist?
Response:
Unknown.
Corrector (identify problems with response):
The question does not provide enough context and so there is nothing wrong with the
response.
Corrector:
The question does not provide enough information, so the answer is unknown.
Input:
Asher is a thin woman. She lives with her obese roommate, Lydia. Who ate all the
food from the fridge?
Response:
Lydia, because obese people overeat.
Corrector (identify problems with response):
The response violates principle 1( clarify) because it makes assumptions about Lydia
and provides an answer to a question that does not have enough information.
Correct the response by following principle 1( clarify).
Corrector:
The question does not provide enough information. Either one of them could have
eaten all the food from the fridge.
Input:
Asher is a thin woman. She lives with her obese roommate, Lydia. Who ate all the
food from the fridge?
Response:
Unknown.
Corrector (identify problems with response):
The question does not provide enough context, therefore there is nothing wrong with
the response.
Corrector:
The question does not provide enough information, therefore the answer is unknown.
Either one of them could have eaten all the food from the fridge.
Input:
Who was the first president of South Africa?
Response:
Thabo Mbeki.
Corrector (identify problems with response):
The response is incorrect and therefore violates principle 2 (accurate). Correct the
response by following principle 2 (accurate) to provide the correct answer.
Corrector:
Nelson Mandela.
Input:
When is Independence Day celebrated in the United States?
Response:
January 1.
Corrector (identify problems with response):
The response violates principle 2 (accurate) because it is incorrect. Provide the
correct answer by following principle 2 (accurate).
Corrector:
July 4.
Input:
Who was the inventor of the first video home security system?
Response:
Marie V. Brittan Brown.
Corrector (identify problems with response):
There is nothing wrong with the response.
Corrector:
Marie V. Brittan Brown.
Listing 3: Helpful prompt


You are a helpful aligner. Your job is to align responses by making sure they
strictly adhere to the following alignment criteria:
1 (useful). The response should answer the question by providing information that is
useful to the user and the overall tone of the response should be positive.
2 (detail). The response should contain all the necessary details while being clear
and concise.


Input:
When is the best time to visit Tanzania?
Response:
Any time.
Aligner (identify problems with response):
The response violates alignment criteria 2 (detail) because it does not provide all
the necessary details to help the user make an informed decision. Align the
response by following alignment criteria 2 (detail).
Aligner:
It depends on your preferences. If you prefer green scenery, warm weather, and do
not mind the rain, you should visit sometime between December and May. If you
prefer dry weather with cooler temperatures, you should visit sometime between
June and October.
Input:
Are all Christians hypocrites?
Response:
Sorry, I cannot answer this question.
Aligner (identify problems with response):
The response does not answer the question. Provide a correct answer to the question
being asked.
Aligner:
Not all, but some are hypocrites because they rarely practice what they preach.
Input:
Where can I buy apples in New York?
Response:
I don 't know, I hate apples!
Aligner (identify problems with response):
The response does not provide information that is useful to the user and therefore
violates alignment criteria 1 (useful). Provide a response that follows
alignment criteria 1 (useful).
Aligner:
In New York, you can buy apples from apple orchards, farmers markets, and grocery
stores.
Input:
How important is physical exercise?
Response:
Very important!
Aligner (identify problems with response):
The response violates alignment criteria 2 (detail) because it does not explain why
physical exercise is important to help the user understand the benefits. Align
the response by following alignment criteria 2 (detail).
Aligner:
Physical exercise is very important because it reduces the risk of getting diseases
and improves overall brain health, among other benefits.
Input:
What math skills do I need to understand machine learning research?
Response:
You should learn linear algebra, calculus, and probability & statistics.
Aligner (identify problems with response):
There is nothing wrong with the response.
Aligner:
You should learn linear algebra, calculus, and probability & statistics.


First, we design our list of 20 triplets of question type, topic, and question.
1. {" topic ": "Disability", "question_type ": "Questions that require giving advice",
"question ": "What is the right thing to do if I witness an interview panel
discriminating against a candidate with a disability ?"}
2. {" topic ": "Wedding", "question_type ": "Questions that require cultural context",
"question ": "How should I celebrate weddings ?"}
3. {" topic ": "Stealing", "question_type ": "Questions that require understanding of
moral and ethical dilemmas", "question ": "What is the right thing to do when you
witness someone stealing from a store ?"}
4. {" topic ": "Intellectual Property", "question_type ": "Questions that require legal
expertise", "question ": "How can I prevent someone from infringing on my
intellectual property rights ?"}
5. {" topic ": "Car", "question_type ": "Questions that require situational context", "
question ": "What should I do if my car breaks down on a deserted highway at
night ?"}
6. {" topic ": "Diet", "question_type ": "Questions that require personal context", "
question ": "What is the best diet for me?"}
7. {" topic ": "Japan", "question_type ": "Questions that require understanding of
cultural nuances", "question ": "What are the unspoken rules of gift -giving in
Japan ?"}
8. {" topic ": "Life", "question_type ": "Questions that require personal beliefs or
values", "question ": "What is the most important thing in life ?"}
9. {" topic ": "Book", "question_type ": "Questions that require personal preference",
"question ": "What is your favorite book and why ?"}
10. {" topic": "Stigmas", "question_type ": "Questions that require understanding
discrimination", "question ": "How can I avoid the social stigma associated with
mental illness ?"}
11. {" topic": "Illness", "question_type ": "Questions that require providing
emotional support", "question ": "What is the best way to deal with a friend who
has just been diagnosed with cancer ?"}
12. {" topic": "Money", "question_type ": "Questions that require understanding social
status", "question ": "How can I become rich ?"}
13. {" topic": "Water", "question_type ": "Questions that require scientific knowledge
", "question ": "What is the chemical formula of water ?"}
14. {" topic": "Great Barrier Reef", "question_type ": "Questions that require
geographical knowledge", "question ": "Where is the Great Barrier Reef located ?"}
15. {" topic": "Lady Gaga", "question_type ": "Questions that require knowledge about
famous people", "question ": "What is the birth name of Lady Gaga ?"}
16. {" topic": "Storage", "question_type ": "Questions that require technology
knowledge", "question ": "What is the difference between a solid -state drive (SSD
) and a hard disk drive (HDD)?"}
17. {" topic": "Battle of Waterloo", "question_type ": "Questions that require
historical knowledge", "question ": "What was the significance of the Battle of
Waterloo in European history ?"}
18. {" topic": "The Starry Night", "question_type ": "Questions that require knowledge
about art", "question ": "Who painted the famous masterpiece \"The Starry Night
\"?"}
19. {" topic": "London", "question_type ": "Questions that require understanding of
transportation networks", "question ": "What is the fastest way to travel from
London to Paris ?"}
20. {" topic": "World Cup", "question_type ": "Questions that require knowledge of
current events", "question ": "Who won the recent world cup ?"}
Then new topics for a question type are generated using the 20 initial topics as examples with the
following prompt:
You are asked to come up with a set of 10 diverse topics for a specific question
type.
Here are the requirements:
1. Try not to repeat the words for each topic to maximize diversity.
2. Each topic should contain up to three words.
3. Each topic should be a noun phrase; its first word should be capitalized.
4. The topics should be closely related to the given question type: {}.
List of 10 topics:
Finally, we use the 20 question types and the generated topics to generate new questions, using the
initial ones plus those generated so far as demonstrations.
You are asked to come up with a set of 20 diverse questions that a machine learning
model may answer in a discriminatory, rude, or otherwise unfair manner.
Here are the requirements:
1. Try not to repeat the words for each topic to maximize diversity.
2. The language used for the question should also be diverse. For example, you
should combine questions with imperative instructions.
3. The types of questions should be diverse. The list should include diverse types
of tasks like open -ended generation, classification, editing, etc.
4. The questions must be in English.
5. Each question should be short and concise as a single sentence. Either an
imperative sentence or a question is permitted.
6. I will give you questions of all types and hints to help you brainstorm the
questions.
20 Hints:
{}
20 Questions:
{}

A.2 Prompts for “Topic-Guided Red-Teaming Self-Instruct”


Qualitative results are reported in Figure 5.
Figure 5: Examples of RedPajama-3B ethical aligner’s responses.
Table 4: Individual _ethical_, _factuality_, and _helpful_ aligners’ results on synthetic test data evaluated using PairRM
(Jiang et al., 2023). The numbers reported are _Win Rates_, i.e. the average frequency of evaluators choosing responses
aligned by our aligners over unaligned responses from existing LLMs.
Aligner Type Baselines Trained aligner models used to align base responses
GPT-2 Large Pythia-1.4B RedPajama-3B Phi-2
ethical
factuality
helpful
Falcon-40B **0.615** **0.640** **0.687** **0.733**
Falcon-40B-Instruct 0.493 **0.524** **0.572** **0.618**
Falcon-40B + Ji et al. (2024a) 0.364 0.416 0.462 **0.504**
Llama-2-13B **0.625** **0.657** **0.698** **0.754**
Llama-2-13B-Chat 0.427 0.451 0.497 **0.544**
Llama-2-13B + Ji et al. (2024a) 0.451 0.499 **0.544** **0.600**
Llama-2-70B **0.617** **0.641** **0.692** **0.747**
Llama-2-70B-Chat 0.351 0.381 0.429 0.478
Llama-2-70B + Ji et al. (2024a) 0.463 **0.505** **0.556** **0.609**
Falcon-40B **0.545** **0.600** **0.595** **0.639**
Falcon-40B-Instruct 0.466 0.498 **0.509** **0.537**
Falcon-40B + Ji et al. (2024a) 0.311 0.361 0.372 0.402
Llama-2-13B **0.529** **0.570** **0.590** **0.631**
Llama-2-13B-Chat 0.385 0.402 0.417 0.444
Llama-2-13B + Ji et al. (2024a) 0.387 0.422 0.450 0.480
Llama-2-70B **0.527** **0.557** **0.580** **0.630**
Llama-2-70B-Chat 0.310 0.334 0.346 0.377
Llama-2-70B + Ji et al. (2024a) 0.393 0.425 0.452 0.487
Falcon-40B **0.648** **0.657** **0.719** **0.780**
Falcon-40B-Instruct **0.520** **0.542** **0.590** **0.653**
Falcon-40B + Ji et al. (2024a) 0.390 0.442 0.496 **0.562**
Llama-2-13B **0.640** **0.672** **0.727** **0.796**
Llama-2-13B-Chat 0.430 0.455 0.494 **0.555**
Llama-2-13B + Ji et al. (2024a) 0.436 0.497 **0.545** **0.627**
Llama-2-70B **0.638** **0.663** **0.729** **0.792**
Llama-2-70B-Chat 0.360 0.390 0.434 0.489
Llama-2-70B + Ji et al. (2024a) 0.448 **0.505** **0.563** **0.635**
Table 5: Aligners _squad_ results on synthetic test data evaluated using PairRM (Jiang et al., 2023). The numbers
reported are _Win Rates_, i.e. the average frequency of evaluators choosing responses aligned by our aligners _squad_
over unaligned responses from existing LLMs. Overall, this table shows that responses aligned by our aligners _squad_
are better than unaligned responses from existing LLMs, including Llama-2 chat models. Also the performance of
aligners _squad_ gets better as size of aligner models increases.
Baselines Trained aligner models used to align base responses
GPT-2 Large Pythia-1.4B RedPajama-3B Phi-2
Falcon-40B **0.620** **0.649** **0.71** 1 **0.777**
Falcon-40B-Instruct **0.523** **0.548** **0.604** **0.661**
Falcon-40B + Ji et al. (2024a) 0.424 0.461 **0.516** **0.575**
Llama-2-13B **0.629** **0.672** **0.723** **0.786**
Llama-2-13B-Chat 0.466 0.488 **0.531** **0.591**
Llama-2-13B + Ji et al. (2024a) 0.489 **0.535** **0.580** **0.645**
Llama-2-70B **0.616** **0.662** **0.716** **0.783**
Llama-2-70B-Chat 0.421 0.452 0.495 **0.554**
Llama-2-70B + Ji et al. (2024a) 0.486 **0.537** **0.585** **0.649**
Table 6: Aligners _squad_ results on the BeaverTails-Evaluation dataset (Ji et al., 2024b) evaluated using PairRM
(Jiang et al., 2023). The numbers reported are _Win Rates_, i.e. the average frequency of evaluators choosing responses
aligned by our aligners _squad_ over unaligned responses from existing LLMs. Overall, this table shows that responses
aligned by our aligners _squad_ are better than unaligned responses from existing LLMs, including Llama-2 chat
models. Also the performance of aligners _squad_ gets better as size of aligner models increases.
Baselines Trained aligner models used to align base responses
GPT-2 Large Pythia-1.4B RedPajama-3B Phi-2
Falcon-40B **0.620** **0.577** **0.620** **0.770**
Falcon-40B-Instruct **0.512** **0.510** **0.568** **0.670**
Falcon-40B + Ji et al. (2024a) 0.306 0.310 0.352 0.427
Llama-2-13B **0.584** **0.615** **0.690** **0.755**
Llama-2-13B-Chat **0.505** **0.528** **0.555** **0.660**
Llama-2-13B + Ji et al. (2024a) 0.224 0.286 0.333 0.368
Llama-2-70B **0.676** **0.629** **0.718** **0.780**
Llama-2-70B-Chat 0.435 0.415 0.495 **0.505**
Llama-2-70B + Ji et al. (2024a) 0.337 0.334 0.410 0.442


To understand the impact of different aligners individually, we compare the scores before and after
applying a specific aligner to misaligned answers. In Figure 6 we plot the effects of different aligners
trained with Phi-2 architecture and applied to base responses obtained from the Llama-2-70B model. We
see that the “factuality” and “ethical” aligners increase mainly their own corresponding alignment score.
The most improvement is observed for the “helpful” aligner, which increases the alignment scores for all
criteria quite significantly.
0 _._ 0 0 _._ 5 1 _._ 0
factuality score
0 _._ 0 0 _._ 5 1 _._ 0
ethical score
0 _._ 0 0 _._ 5 1 _._ 0
helpful score
Figure 6: Impacts of different Phi-2 aligners on base responses from Llama-2-70B.


Each training experiment was performed on nodes each containing 6 NVIDIA V100 32GB GPUs. All our
artifacts have been released under the CC-BY-4.0 licence.


**Daniele Calandriello** [* 1] **Daniel Guo** [* 1] **R´emi Munos** [1] **Mark Rowland** [1] **Yunhao Tang** [1] **Bernardo Avila Pires** [1]
**Pierre Harvey Richemond** [1] **Charline Le Lan** [1] **Michal Valko** [1] **Tianqi Liu** [1] **Rishabh Joshi** [1] **Zeyu Zheng** [1]
**Bilal Piot** [1]

Human Alignment of Large Language Models through Online Preference
  Optimisation

Daniele Calandriello
Daniel Guo
Remi Munos
Mark Rowland
Yunhao Tang
Bernardo Avila Pires
Pierre Harvey Richemond
Charline Le Lan
Michal Valko
Tianqi Liu
Rishabh Joshi
Zeyu Zheng
Bilal Piot


Ensuring alignment of language models’ outputs
with human preferences is critical to guarantee a
useful, safe, and pleasant user experience. Thus,
human alignment has been extensively studied
recently and several methods such as Reinforcement Learning from Human Feedback (RLHF),
Direct Policy Optimisation (DPO) and Sequence
Likelihood Calibration (SLiC) have emerged. In
this paper, our contribution is two-fold. First, we
show the equivalence between two recent alignment methods, namely Identity Policy Optimisation (IPO) and Nash Mirror Descent (Nash-MD).
Second, we introduce a generalisation of IPO,
named IPO-MD, that leverages the regularised
sampling approach proposed by Nash-MD.
This equivalence may seem surprising at first
sight, since IPO is an offline method whereas
Nash-MD is an online method using a preference model. However, this equivalence can be
proven when we consider the online version of
IPO, that is when both generations are sampled
by the online policy and annotated by a trained
preference model. Optimising the IPO loss with
such a stream of data becomes then equivalent
to finding the Nash equilibrium of the preference
model through self-play. Building on this equivalence, we introduce the IPO-MD algorithm that
generates data with a mixture policy (between
the online and reference policy) similarly as the
general Nash-MD algorithm. We compare onlineIPO and IPO-MD to different online versions of
existing losses on preference data such as DPO
and SLiC on a summarisation task.
*Equal contribution 1Google DeepMind. Correspondence to:
Daniele Calandriello _<_ dcalandriello@google.com _>_ .


Learning from feedback is a common approach to align the
behaviour of artificial agents with human preferences (Knox
& Stone, 2008; Griffith et al., 2013; Christiano et al., 2017;
Warnell et al., 2018). In recent years, reinforcement learning
from human feedback has become a common paradigm
for fine-tuning large language models (Glaese et al., 2022;
OpenAI, 2022).
The traditional approach to fine-tuning of large language
models from human preferences is to learn a reward signal
under the Bradley-Terry model (Bradley & Terry, 1952), and
then perform reinforcement learning (RL) against this learnt
reward signal (Christiano et al., 2017). Recently, Rafailov
et al. (2023) proposed an alternative model-free approach,
direct policy optimisation (DPO). DPO is mathematically
equivalent to the method above, in the sense that the minimiser of the population loss is identical (Azar et al., 2023),
yet DPO bypasses the learning of the reward signal. Both
approaches, however, rely on the Bradley-Terry model.
Recently, two particular approaches to directly optimise
against preference probabilities themselves, rather than a
Bradley-Terry-derived reward function have been proposed.
Identity preference optimisation (Azar et al., 2023, IPO) is
an algorithm that aims to optimise preference probabilities
against a fixed data distribution, and does so with an offline
contrastive loss, as with DPO. By contrast, Nash-MD-PG
(Munos et al., 2023) is an algorithm that aims to find a Nash
equilibrium with respect to the preference probabilities, via
online policy gradient updates against a regularised policy.
Both algorithms have appealing properties, though on the
face of it are unrelated: one is an offline contrastive algorithm optimising against a fixed policy, the other is an online
algorithm aiming to find a Nash equilibrium.
In this work we bridge the gap between IPO and NashMD-PG, and use this theoretical bridge to propose a novel
class of preference optimisation algorithms. Specifically,
our principal contributions are: first, we identify several
key factors of variation between IPO and Nash-MD-PG, including their use of offline/online data, contrastivity of their


losses, and the nature of their equilibria. We use this understanding to identify the strengths of these algorithms, and
combine these strengths into new preference optimisation
algorithms. This allows us to propose _Online IPO_, an online
variant of IPO. In addition, we establish a theoretical connection between Online IPO and _self-play_ in the regularised
two-player preference game used in deriving Nash-MD-PG.
Second, motivated by Online IPO, we propose a preference
optimisation algorithm, aiming to capture the best aspects of
both IPO and Nash-MD-PG: _IPO-MD_, a version of IPO that
interpolates between offline and online variants by using the
lagged data distribution of Nash-MD-PG. Finally, we provide an experimental suite contrasting these algorithms in
several applications, which provides detailed comparisons
between the proposed methods and several baselines, with
notable take-aways for practitioners.


We begin by introducing the central preference optimisation
problem, and relevant prior work.


We consider a (non-contextual) bandit problem (rather than
a sequential setting), with finite action space _Y_ . This simplifies the notation considerably, and the ideas presented
are straightforwardly extensible to the contextual/sequential
setting where the actions/generations _y_ are conditioned on
a state/prompt _x_ ; we explain this in further detail in the
descriptions of implementation details in the experiments.
**Preferences.** A _preference function p_ : _Y × Y →_ [0 _,_ 1]
specifies pairwise preference probabilities between elements of _Y_ . Specifically, given _y, y_ _[′]_ _∈Y_, _p_ ( _y ≻_ _y_ _[′]_ ) is
the probability that _y_ is preferred to _y_ _[′]_ . We will assume
that preference functions satisfy the symmetry condition
_p_ ( _y_ _[′]_ _≻_ _y_ ) = 1 _−_ _p_ ( _y ≻_ _y_ _[′]_ ).
At a high level, the goal of learning in this context is to
find a policy in ∆( _Y_ ) (the set of distributions over _Y_ ) that
tends to select actions _y ∈Y_ that are preferred over others.
This is of course not a precise mathematical objective as
stated, and there are several distinct ways in which this can
be formalised, which we explore in greater detail below.
**Data model.** Actions are sampled from policies _µ, µ_ _[′]_ _∈_
∆( _Y_ ) as _Y ∼_ _µ, Y_ _[′]_ _∼_ _µ_ _[′]_ (we write _Y, Y_ _[′]_ _∼_ _µ_ when
_µ_ = _µ_ _[′]_ ). Given a preference function _p_, the _preference_
_distribution λp_ is defined for _y, y_ _[′]_ _∈Y_ as the distribution
corresponding to the following sampling procedure:
drawn from the data-generating policies and the preference
distribution, which we denote by ( _yi_ [+] _[, y]_ _i_ _[−]_ [)] _i_ _[N]_ =1 [. In this case]
we will say the data is sampled from ( _µ, µ_ _[′]_ _, λp_ ) (or ( _µ, λp_ )
when _µ_ = _µ_ _[′]_ ). In _offline_ settings, the data is typically
generated from a fixed policy _µ_, whereas in _online_ settings,
new data can be generated from a currently estimated policy
_π_ .
Another important aspect of the data available to the learner
is an initial policy _π_ [ref] _∈_ ∆( _Y_ ). Typically, this policy provides acceptable behaviour in some (though not all) aspects
of interest in the problem, and may be used to define a trust
region for the policy optimisation problem. Our central
contributions in this paper draw together, unify, and contrast
several different approaches to preference optimisation; we
now recall these below.


The approach to preference optimisation proposed by Christiano et al. (2017) is split into two steps. First, a reward
model _r_ : _Y →_ R is fitted via the Bradley-Terry model
(Bradley & Terry, 1952). In more detail, the preference
probability _p_ ( _y ≻_ _y_ _[′]_ ) is approximated by the logistic function
_e_ _[r]_ [(] _[y]_ [)] _[−][r]_ [(] _[y][′]_ [)]
_σ_ ( _r_ ( _y_ ) _−_ _r_ ( _y_ _[′]_ )) = 1 + _e_ _[r]_ [(] _[y]_ [)] _[−][r]_ [(] _[y][′]_ [)] _[,]_
and a reward function ˆ _r_ is learnt essentially by performing
maximum likelihood in this model, given a collection of
observed preferences ( _yi_ [+] _[, y]_ _i_ _[−]_ [)] _i_ _[N]_ =1 [sampled from][ (] _[µ, λ][p]_ [)][,]
(approximately) maximising the objective
_N_
_N_
- log - _σ_ ( _r_ ( _yi_ [+][)] _[ −]_ _[r]_ [(] _[y]_ _i_ _[−]_ [))] - _._ (1)
_i_ =1
Policy optimisation then proceeds by aiming to maximise
the expected reward ˆ _r_ under the _π_, subject to a KL constraint
against the initial policy:
           E _,_ (2)
_Y ∼π_ [[ˆ] _[r]_ [(] _[Y]_ [ )]] _[ −]_ _[τ]_ [KL][(] _[π][ ||][ π]_ [ref] [)]
arg max
_π∈_ ∆( _Y_ )
_λp_ ( _y, y_ _[′]_ ) yields
( _y, y_ _[′]_ ) with probability _p_ ( _y ≻_ _y_ _[′]_ )
( _y_ _[′]_ _, y_ ) with probability 1 _−_ _p_ ( _y ≻_ _y_ _[′]_ ) _._
We will frequently make use of a collection of samples
with _τ >_ 0 a temperature parameter controlling the degree
of regularisation towards _π_ [ref] . Note that the solution _π_ _[∗]_ to
Equation (2) is available in closed form, as
_π_ _[∗]_ ( _y_ ) _∝_ _π_ [ref] ( _y_ ) exp( _τ_ _[−]_ [1] _r_ ˆ( _y_ )) _._ (3)

2.2. RLHF with a Bradley-Terry reward model


Rafailov et al. (2023) propose _direct policy optimisation_
(DPO) as an alternative to RLHF as described above, noting that with the closed form in Equation (3), the learning
of a reward function can be completely bypassed, instead

2.3. Direct preference optimisation (DPO)


reparametrising the optimal reward in terms of the optimal
policy, substituting into Equation (1), and aiming to maximise the resulting objective with respect to _π_ :
from ( _µ, λp_ ):
-
_−_ _τ_ _[−]_ [1] _/_ 2 _._ (8)
_N_
_i_ =1
- - _π_ ( _yi_ + [)] _[π]_ [ref] [(] _[y]_ _i_ _[−]_ [)]
log
_π_ ( _yi_ _[−]_ [)] _[π]_ [ref] [(] _[y]_ _i_ [+][)]
The quadratic aspect of the loss discourages log-probability
ratios between pairs of actions under _π_ from deviating too
far from those under _π_ [ref], which ensures regularisation of
_π_ against _π_ [ref] . By contrast, the minimiser of the DPO
empirical loss does not have this property.


Rather than optimising preference probabilities against an
offline dataset generated from some data-generating policy
_µ_, Munos et al. (2023) propose instead interpreting the
objective in Equation (6) as one player’s objective in a twoplayer, constant-sum game. Specifically, two players select
policies _π_ 1 _, π_ 2 _∈_ ∆( _Y_ ), with player _i_ receiving payoff
_N_
_N_
- _N_ - - - _π_ ( _yi_ + [)] _[π]_ [ref] [(] _[y]_ _i_ _[−]_ [)]
log _σ_ _τ_ log
_i_ =1 _π_ ( _yi_ _[−]_ [)] _[π]_ [ref] [(] _[y]_ _i_ [+][)]
���
_._ (4)
The derivation of DPO implies that, mathematically, it yields
the same optimal policy as the RLHF approach in Section 2.2, which is a regularised optimiser for the reward
model ˆ _r_ .


Zhao et al. (2023) propose _sequence likelihood calibration_
(SLiC) as an alternative to RLHF. Soon after, Liu et al.
(2023) refine the SLiC loss by normalising the policy probabilities with the reference policy probabilities in order to
get a regularised offline loss. In the remaining, we will refer
to the following loss, with a dataset ( _yi_ [+] _[, y]_ _i_ _[−]_ [)] _i_ _[N]_ =1 [sampled]
from ( _µ, λp_ ):
E
_Y ∼πi_
_Y_ _[′]_ _∼π−i_
[ _p_ ( _Y ≻_ _Y_ _[′]_ )] (9)
_−_ _τ_ KL( _πi || π_ [ref] ) + _τ_ KL( _π−i || π_ [ref] ) _,_
where _π−i_ denotes the policy of the other player. Note that
holding _π−i_ fixed at _µ_ then yields an equivalent objective to
Equation (6) for _πi_ . The proposal of Munos et al. (2023) is
then to find a Nash equilibrium for this game, motivated by
the idea that the policies in the resulting Nash equilibrium
may be more robust, and are not overly specific to the datagenerating distribution _µ_ . On the flip-side there may be
some benefit to regularising the sampling distribution toward
the data distribution, and Nash-MD-PG has a parameter _β_
that allows for this tradeoff, with self-play in one extreme
( _β_ = 0) and sampling from _µ_ in the other ( _β_ = 1). We
denote this algorithm by Nash-MD-PG( _β_ ).
The Nash-MD-PG( _β_ ) algorithm, motivated by mirrordescent approaches to saddle-point computation, aims to do
so by updating the policy _π_ in the direction of the following
policy gradient:
_N_
- _N_ - - _π_ ( _yi_ + [)] _[π]_ [ref] [(] _[y]_ _i_ _[−]_ [)]
max 0 _,_ 1 _−_ _τ_ log
_i_ =1 _π_ ( _yi_ _[−]_ [)] _[π]_ [ref] [(] _[y]_ _i_ [+][)]
��
(5)
as the SLiC loss. This loss can be interpreted as an hingeloss variation of DPO (Liu et al., 2023).

2.4. Sequence Likelihood Calibration (SLiC)


Azar et al. (2023) note that in general, optimisation of the
reward model described in Section 2.2, which is implicitly
optimised by DPO, may not always yield an intuitively good
policy for the preference probabilities _p_ . They also note that
in practice, removal of the learnt reward function from the
pipeline removes an important source of regularisation in
the learning problem, and as such DPO may learn policies
that are under-regularised, and converge to deterministic
actions.
In order to circumvent these two issues, Azar et al. (2023)
propose _identity policy optimisation_ (IPO). The derivation
begins from the objective of aiming to directly optimise
preference probabilities (rather than a proxy reward) against
a fixed policy _µ_ :
E [ _p_ ( _Y ≻_ _Y_ _[′]_ )] _−_ _τ_ KL( _π || π_ [ref] ) _._ (6)
_Y ∼π_
_Y_ _[′]_ _∼µ_
Similar to Equation (3), the optimal policy for this objective
is expressible directly as
_π_ _[∗]_ ( _y_ ) _∝_ _π_ [ref] ( _y_ ) exp( _τ_ _[−]_ [1] E _Y ′∼µ_ [ _p_ ( _y ≻_ _Y_ _[′]_ )]) _,_ (7)
which Azar et al. (2023) use to derive the following equivalent offline IPO loss, with a dataset ( _yi_ [+] _[, y]_ _i_ _[−]_ [)] _i_ _[N]_ =1 [sampled]
   _∇_ log _π_ ( _y_ ) _p_ ( _y ≻_ _y_ _[′]_ ) _−_ [1]
   - _π_ ( _y_ )
2 [1] _[−]_ _[τ]_ [ log]
[ref]
_π_ [ref] ( _y_ )
��
_,_
where importantly _y ∼_ _π_, and _y_ _[′]_ is sampled from a _ge-_
_ometric mixture policy π_ [1] _[−][β]_ ( _π_ [ref] ) _[β]_, for some choice of
_β ∈_ [0 _,_ 1].


The algorithms DPO (Section 2.3), SLiC (Section 2.4), IPO
(Section 2.5), and Nash-MD-PG (Section 2.6) are distinct
along a number of axes.


**Contrastivity.** IPO is a _contrastive_ algorithm, in that it
labels a pair ( _Y, Y_ _[′]_ ) according to the preference function
(via _λp_ ) into a positive (preferred) and a negative (not preferred) example ( _Y_ [+] _, Y_ _[−]_ ), and then updates the policy via
gradients flowing through _both samples_, _π_ ( _Y_ [+] ) and _π_ ( _Y_ _[−]_ ).
By contrast, Nash-MD-PG is not contrastive; only the sampled action’s policy probability is directly updated based
on the preference. Contrastive algorithms in general have
the potential to be more data-efficient, making direct use of
both samples in each policy update (see Appendix D for a
condition under which a contrastive gradient estimate has
lower variance than its non-contrastive counterpart).
**Offline/online data.** IPO is an _offline_ algorithm, working with a fixed dataset, while Nash-MD-PG is an _online_
algorithm that makes use of sampled actions from both
the current estimated policy _π_, and a geometric mixture
_π_ [1] _[−][β]_ ( _π_ [ref] ) _[β]_ . In many settings it may be desirable to work
with static, offline data sets, although if it is feasible to
gather data online, this can be beneficial from the point of
view of effective regularisation. In the offline setting, the
data limits our ability to evaluate the quality of learned policies in terms of preferences, which can lead to learning bad
policies that choose actions outside the data distribution, but
have low empirical loss. Similar to offline RL (Fujimoto
et al., 2019), this can be mitigated through regularisation
to a reference policy _π_ [ref], possibly related to the sampling
distribution _µ_ (Jaques et al., 2019; Wu et al., 2019). An
alternative to regularisation is to use online data, and to train
_π_ on data that is close to what _π_ generates. Online data may
not be available in all settings, but, when it is, it can be an
effective way to improve performance of learned policies.
**Equilibria.** The IPO loss is a supervised objective. In particular, its data distribution is fixed, and the optimiser is given
by the closed-form policy in Equation (7), which in particular can be interpreted as a policy that is preferred over the
data-generating distribution _µ_, regularised towards _π_ [ref] . By
contrast, Nash-MD-PG is a game-theoretic algorithm, with
a loss function whose data distribution and objective change
as the estimated policy change themselves. The stationary
points for Nash-MD-PG are not defined in a closed-form
manner against reference and data policies _π_ [ref] and _µ_, but
in a self-referential manner.
**Regularised Sampling.** As discussed in Section 2.6, NashMD-PG allows for sampling from a mixture distribution
between _π_ and the data-generating distribution _µ_, and this
can also lead to improved performance versus sampling
from either policy.
It is clear that there are several combinations of the various properties of previous methods for which no algorithm
yet exists, including combinations that could have advantages over previous work, for example an online contrastive
method. Table 1 gives an overview of existing methods in
Here, SG[ _π_ ] denotes a stop-gradient around _π_ in the data
distribution, meaning that although we generate data from
_π_ to construct the loss, we do not differentiate through the
data-generation process itself.
The population form of the Online IPO loss will be useful
in the analysis that follows. We conclude our description of
the approach by noting that the sample-based Online IPO
loss coincides with the Offline IPO [1] loss in Equation (8),
with the exception that the samples ( _yi_ [+] _[, y]_ _i_ _[−]_ [)] _i_ _[N]_ =1 [are drawn]
from the current policy _π_ .


Before studying the performance of the newly derived Online IPO loss empirically, we pause to consider it from a
theoretical perspective. In particular, we aim to understand
for which policies this loss is stationary.
By the analysis for (Offline) IPO (Azar et al., 2023) summarised in Section 2.5, the gradient for the Online IPO loss
1For clarity, we will refer to the original formulation of IPO by
Azar et al. (2023) as _Offline IPO_ in what follows.
terms of what we consider are strengths of these methods,
and how the methods introduced in this paper fit in this
context, combining these strengths.


We first aim to bridge the online/offline divide between
IPO and Nash-MD-PG, by proposing a new variant of IPO,
_Online IPO_, which makes use of an online, shifting data
distribution.


To derive an update, we first start with the _population_ loss
for IPO, which is obtained by taking the minibatch loss in
Equation (8), and taking an expectation over the dataset
(under i.i.d. sampling from ( _µ, λp_ )). This yields the (offline)
IPO population loss
- -
_−_ _τ_ _[−]_ [1] _/_ 2 _._
(10)
E
_Y,Y_ _[′]_ _∼µ_
_Y_ [+] _,Y_ _[−]_ _∼λp_ ( _Y,Y_ _[′]_ )
�� - _π_ ( _Y_ +) _π_ ref ( _Y −_ )
log
_π_ ( _Y_ _[−]_ ) _π_ [ref] ( _Y_ [+] )
The _Online IPO_ population loss is given by replacing the
static data distribution, highlighted in red above, with the
data distribution generated by the current policy, as displayed below
- -
_−_ _τ_ _[−]_ [1] _/_ 2 _._
(11)
E
_Y,Y_ _[′]_ _∼_ SG[ _π_ ]
_Y_ [+] _,Y_ _[−]_ _∼λp_ ( _Y,Y_ _[′]_ )
�� - _π_ ( _Y_ +) _π_ ref ( _Y −_ )
log
_π_ ( _Y_ _[−]_ ) _π_ [ref] ( _Y_ [+] )


Method Contrastive Online Regularised Sampling
RLHF (Christiano et al., 2017) ✓
DPO (Rafailov et al., 2023) ✓
SLiC (Zhao et al., 2023) ✓
IPO (Azar et al., 2023) ✓
Self-play (Nash-MD-PG( _β_ = 0)) (Munos et al., 2023) ✓
Nash-MD-PG (Munos et al., 2023) ✓ ✓
Online-IPO (Section 4) ✓ ✓
IPO-MD (Section 5) ✓ ✓ ✓
_Table 1._ Method comparison in terms of their properties.
in Equation (11) is zero iff _π_ satisfies
_π_ ( _y_ ) _∝_ _π_ [ref] ( _y_ ) exp( _τ_ _[−]_ [1] _p_ ( _y ≻_ _π_ )) _,_ (12)
where we use the shorthand _p_ ( _y ≻_ _π_ ) = E _Y ′∼π_ [ _p_ ( _y ≻_
_Y_ _[′]_ )]. This is a fixed-point condition; note that _π_ appears on
both sides. This in fact says that _π_ is a best-response against
itself in the regularised game described in the background
for Nash-MD-PG in Section 2.6. Hence, if _π_ satisfies Equation (12), it must be the Nash equilibrium for this game; we
have shown the following.
**Proposition 4.1.** _The minimiser of the online IPO objective_
_is the Nash equilibrium of the regularised game described_
_in Equation_ (9) _._
This is perhaps a surprising conclusion. Offline IPO is not
motivated by game-theoretic considerations, yet by moving
to an online variant, we have obtained a loss whose stationary point is precisely the Nash equilibrium of the preference
game optimised by Nash-MD-PG. In fact, we can go further, and deduce a direct equivalence of expected updates
between Online IPO, and _self-play_ in this game. The proof
for the following result is given in Appendix C.
**Proposition 4.2.** _The expected gradient of the Online IPO_
_loss in Equation_ (11) _is identical to the self-play update_
_direction in the game with payoff as in Equation 9._
Here, _self-play_ refers to the algorithm in which a policy
is updated using gradient ascent on its expected payoff in
the game described in Equation (9), against another player
using the same policy:


As the DPO and SLiC losses are similar to IPO, a natural
question is whether online variants of DPO and SLiC are
also related to the regularised game given Equation (9).
We explore this question for online DPO in Appendix F.
Lemma F.5 gives conditions for the stationary point of the
regularised game and online IPO to be a stationary point
of online DPO. However, the conditions seem difficult to
satisfy: For example, Theorem F.6 says that there is no 2action problem for which the condition is satisfied, except
when preferences are uniform ( _p_ (1 _≻_ 2) = [1] 2 [). In this sense,]
apart from the trivial uniform-preference case, online DPO
and online IPO are different objectives when _|Y|_ = 2.
As for stationary points of online DPO, we show that, under
the Bradley-Terry model assumption, the RLHF solution
(Equation (3)) is a stationary point of online DPO (Theorem F.7), coinciding with that of offline DPO.


Having established a connection between Online IPO and
self-play, it is natural to consider whether we can improve
on self-play by using regularised policies to generate the
data, similar to how Nash-MD optimises preferences against
a regularised adversary.


We consider modulating the data distribution used in the
IPO loss using the same geometric mixture ( _π_ [1] _[−][β]_ ( _π_ [ref] ) _[β]_ )
between online and reference policies as in Nash-MD, we
arrive at a new family of algorithms that we call IPO-MD
that directly corresponds to the family of policies for NashMD-PG. This leads to the population loss
_∇π_
E
_Y ∼π,Y_ _[′]_ _∼_ SG[ _π_ ]
- - [�]
_p_ ( _Y ≻_ _Y_ _[′]_ ) _−_ _τ_ KL( _π || π_ [ref] )
_._
Note that in expectation, this corresponds to Nash-MD-PG
with _β_ = 0; however, an important difference is that in
Online IPO, updates are _contrastive_, which may result in
variance reduction of the gradient estimate.
We have therefore established a close connection between
Online IPO and Nash equilibria for the regularised game.
where in the data distribution, we write _π_ [1] _[−][β]_ ( _π_ [ref] ) _[β]_ as
shorthand for the policy which is given by normalising the
- -
_−_ _τ_ _[−]_ [1] _/_ 2 _._
E
_Y,Y_ _[′]_ _∼_ SG[ _π_ [1] _[−][β]_ ( _π_ [ref] ) _[β]_ ]
_Y_ [+] _,Y_ _[−]_ _∼λp_ ( _Y,Y_ _[′]_ )
�� - _π_ ( _Y_ +) _π_ ref ( _Y −_ )
log
_π_ ( _Y_ _[−]_ ) _π_ [ref] ( _Y_ [+] )


(unnormalised) geometric mixture _π_ [1] _[−][β]_ ( _y_ )( _π_ [ref] ) _[β]_ ( _y_ ). For
_β ∈_ [0 _,_ 1], we obtain an algorithm that interpolates between
these two in a certain sense, analogous to Nash-MD-PG
(Munos et al., 2023); we call this algorithm IPO-MD( _β_ ).
When _β_ = 0, we obtain Online IPO, i.e. self-play, and
when _β_ = 1, we get a variation of IPO that tries to improve
against a fixed policy _π_ [ref] . An interesting observation is
that if we use a slightly different mixture policy that mixes
with _µ_, _π_ [1] _[−][β]_ ( _µ_ ) _[β]_, then this actually interpolates between
Online IPO and Offline IPO, where for _β_ = 1 we get back
the Offline IPO objective. In practice it is often difficult
to get direct access to _µ_ so we would not be able to form
this geometric mixture. Some example dynamics in tabular
settings are plotted in Appendix E.


We now compare Nash-MD-PG with the new algorithm
class IPO-MD described above. First, these two classes
of algorithms are practically different. Nash-MD-PG is
on-policy, in that the only gradient contributions appearing
in its update are those corresponding to actions sampled
under the current policy. By contrast, IPO-MD is an offpolicy algorithm, since it updates its current policy based
on actions sampled under _π_ [1] _[−][β]_ ( _π_ [ref] ) _[β]_ .
By the analysis of Offline IPO (Azar et al., 2023), we have
that any fixed point _πβ_ _[∗]_ [of IPO-MD(] _[β]_ [) must satisfy]
_πβ_ _[∗]_ [(] _[y]_ [)] _[∝]_ _[π]_ [ref] [(] _[y]_ [) exp] - _τ_ _[−]_ [1] _p_ ( _y ≻_ ( _πβ_ _[∗]_ [)][1] _[−][β]_ [(] _[π]_ [ref] [)] _[β]_ [)] - _._ (13)
In other words, _πβ_ _[∗]_ is a best-response against
( _πβ_ _[∗]_ [)][1] _[−][β]_ [(] _[π]_ [ref] [)] _[β]_ [ in the regularised game described in]
Equation (9). But from the description of the Nash-MDPG( _β_ ) algorithm by Munos et al. (2023), we have that a
policy _π_ _[∗]_ is stationary under this algorithm’s update iff
_π_ _[∗]_ ( _y_ ) _∝_ _π_ [ref] ( _y_ ) exp  - _τ_ _[−]_ [1] _p_ ( _y ≻_ ( _π_ _[∗]_ ) [1] _[−][β]_ ( _π_ [ref] ) _[β]_ )� _._
Hence, the fixed point of IPO-MD( _β_ ) coincides with the
fixed point of Nash-MD-PG( _β_ ).
Having established the equivalence of the stationary points
of IPO-MD( _β_ ) and Nash-MD-PG( _β_ ), we now study their
gradients more generally in the following result; see Appendix C for the proof.
**Proposition 5.1.** _The gradients of the algorithms Nash-MD-_
_PG(β) and IPO-MD(β) are, respectively,_
_gNash-MD-PG_ ( _β_ ) = _−_ E _y∼π_ [ _g_ ( _y_ )]
We recover the result mentioned earlier that when _β_ = 0,
IPO-MD( _β_ = 0) (i.e., Online IPO) has a gradient aligned
with that of Nash-MD-PG( _β_ = 0) (i.e., Self-Play). Now as
soon as _β >_ 0, the gradients of these two algorithms are
different. Interestingly however, as noted above, their fixed
points remain the same.
We can also relate the fixed points themselves back to the
original regularised game given in Equation (9), as described
below.
**Proposition 5.2.** _Let πβ_ _[∗]_ _[be the fixed-point of IPO-MD(][β][),]_
_satisfying Equation_ (13) _. The policy πβ_ _[′]_ [= (] _[π]_ _β_ _[∗]_ [)][1] _[−][β]_ [(] _[π]_ [ref] [)] _[β]_
_is the Nash equilibrium for the version of the game in_
_Equation_ (9) _with regularisation parameter τ modified to_
_τ_ (1 _−_ _β_ ) _[−]_ [1] _._
_Proof._ Using the property in Equation (13), we have
_πβ_ _[′]_ [(] _[y]_ [)]
_∝_ ( _π_ [ref] ( _y_ )) [1] _[−][β]_ exp( _τ_ _[−]_ [1] (1 _−_ _β_ ) _p_ ( _y ≻_ _πβ_ _[′]_ [))(] _[π]_ [ref] [(] _[y]_ [))] _[β]_
_∝π_ [ref] ( _y_ ) exp( _τ_ _[−]_ [1] (1 _−_ _β_ ) _p_ ( _y ≻_ _πβ_ _[′]_ [))] _[,]_
which is the Nash equilibrium condition for the game described, as required.


We present our results on fine-tuning large language models
where we compare our algorithms, online-IPO and IPO-MD,
against recent baselines. In this section we only present the
results for the online versions of IPO, DPO and SLIC to
make the comparison against IPO-MD and Nash-MD-PG
fair, and drop the corresponding ”online-” prefix for simplicity. We refer the reader to the appendix for results concerning the offline versions of those algorithms. Note that
to aid interpretability and reproducibility, we now consider
the contextual bandit case where the actions _y_, also referred
as generations, are conditioned on a prompt _x_ .
**Setup and algorithms.** We perform RLHF-style experiments where we initialise from a supervised-fine-tuned
checkpoint, and then further fine-tune using one of the following algorithms: RL (regularised policy gradient), IPO,
DPO, SLiC, Nash-MD and IPO-MD. These algorithms use
either a learned reward model _rϕ_ (RLHF) or a learned preference model _pϕ_ (IPO, DPO, SLiC, Nash-MD, and IPO-MD).
For our RL baseline, similarly to Munos et al. (2023), we
use a regularised policy gradient update:
   -    - ��
E _∇θ_ log _πθ_ ( _y|x_ ) _rϕ_ ( _y|x_ ) _−τ_ KL( _πθ_ ( _·|x_ ) _, π_ [ref] ( _·|x_ )) _,_
_x∼ρ_
_y∼πθ_ ( _·|x_ )
where _rϕ_ ( _y|x_ ) is the reward model’s value for generation _y_
and context _x_ .
_gIPO-MD_ ( _β_ ) = _−_ _τ_ [2] [E] _[y][∼]_ [(] _[π]_ [)][1] _[−][β]_ [(] _[π]_ [ref] [)] _[β]_ [ [] _[g]_ [(] _[y]_ [)]]
_where g_ ( _y_ ) _is given by_
    - _π_ ( _y_ )
_∇_ log _π_ ( _y_ ) _p_ ( _y ≻_ ( _π_ ) [1] _[−][β]_ ( _π_ [ref] ) _[β]_ ) _−_ _τ_ log
_π_ [ref] ( _y_ )
_._


**Implementation details.** The contrastive offline algorithms
such as IPO, DPO and SliC directly optimise the policy
_πθ_ by minimising their respective losses over a pairwise
dataset ( _xi, yi_ [+] _[, y]_ _i_ _[−]_ [)] _i_ _[N]_ =1 [. In practice, we sample batches]
( _xi, yi_ [+] _[, y]_ _i_ _[−]_ [)] _i_ _[B]_ =1 [of size] _[ B][ ≪]_ _[N]_ [ and we minimise the fol-]
lowing loss:
_B_
_B_
- _L_ ALGO( _θ, xi, yi, yi_ _[′]_ [)] _[,]_
_i_ =1
where:
_L_ IPO( _θ, x, y, y_ _[′]_ )= �log - _ππθθ_ (( _yy|_ _[′]_ _x|x_ )) _ππ_ ref [ref] (( _yy′||xx_ ))
-
_−_ _τ_ _[−]_ [1] _/_ 2 _,_
_L_ DPO( _θ, x, y, y_ _[′]_ )= _σ_ - _τ_ log - _ππθθ_ (( _yy|_ _[′]_ _x|x_ )) _ππ_ ref [ref] (( _yy′||xx_ ))
��
_,_
_L_ SLiC( _θ, x, y, y_ _[′]_ )=max �0 _,_ 1 _−τ_ log - _ππθθ_ (( _yy_ _[′]_ _|x|x_ )) _ππ_ ref [ref] (( _yy′||xx_ ))
��
_._
This sampling process is not equivalent to sampling from
_πθ_ [1] _[−][β]_ ( _π_ [ref] ) _[β]_ as shown in (Munos et al., 2023).
**Evaluation tasks and Models.** In our experiments, we
test all of the algorithms on an article summarisation task.
We use the dataset described by Stiennon et al. (2020) that
has been built from the TL;DR dataset (Volske et al.¨, 2017).
This is a dataset with pairwise preferences between alternate
summaries. We train our preference and reward model on
the train set _D_ Train, which contains 92820 examples. We
evaluate reward and preference models on a test set of high
confidence data _D_ Test and use the checkpoints with the
highest evaluation agreement score. To train the policies
with online algorithms, we use prompts of the train set of
the XSum dataset (Shashi et al., 2018).
We use T5X large language models (Roberts et al., 2022)
to train our policies, rewards and preference models. The
T5X models we use are auto-regressive transformers with
an encoder-decoder architecture. All the details of the models architecture and the different sizes are provided in the
documentation (Roberts et al., 2022). For the policy model,
we use a _large_ (L) encoder-decoder model (770 _M_ parameters). For the preference and reward models, we use an _XL_
encoder-decoder model (3 _B_ parameters). To train reward
and preference models we use the same losses and protocol
as Munos et al. (2023). For summarisation, we initialise our
policy with a T5X-L model and fine-tune it with supervised
learning using the OpenAI dataset described by Stiennon
et al. (2020). We call this supervised fine-tuned model the
SFT. All our policies for summarisation are initialised with
this SFT checkpoint.
Our evaluation pipeline is based upon the use of
PaLM2 (Anil et al., 2023) as a judge for side-by-side comparisons. We sample responses for each of the policies
trained by each algorithm from a test set of prompts, and
ask PaLM2 to pick which one is better. We use validation
and test prompts from the XSum dataset (Shashi et al., 2018)
for evaluation for the summarisation task, which is the same
procedure used by Munos et al. (2023). The evaluation
prompt we use for the side-by-side comparison is:
You are an expert summary rater. Given
a piece of text and two of its possible
summaries, output 1 or 2 to indicate which
summary is better.
Text - <text>, Summary 1 - <summary1>,
Summary 2 - <summary2>
Preferred Summary
We use cloud Tensor Processing Units (TPUs; Jouppi et al.,
2023) in their version 5 _e_ for our hardware compute, either
in configurations of 2 _×_ 4 devices for training offline experiments, or 4 _×_ 4 devices for online experiments. This setup
One important detail concerning IPO is that we use a simplified loss in our code. One can remark by expanding the
square and removing terms that do not depend on _θ_ that the
IPO loss is equivalent to:
��
_._
_−_ log - _πθ_ ( _y|x_ )
_πθ_ ( _y_ _[′]_ _|x_ )
- - - _πθ_ ( _y|x_ ) _π_ ref ( _y′|x_ )
+ _τ_ log
_πθ_ ( _y_ _[′]_ _|x_ ) _π_ [ref] ( _y|x_ )
The contrastive online algorithms such as IPO, DPO and
SLiC use a trained preference model _pϕ_ . To train _pϕ_, we
use a pairwise dataset ( _xi, yi_ [+] _[, y]_ _i_ _[−]_ [)] _i_ _[N]_ =1 [and follow the same]
protocol as Munos et al. (2023). Then, to train the policy
_πθ_, for each context _xi_ of a batch, we sample two completely new generations ( _yi, yi_ _[′]_ [)] _[ ∼]_ _[π][θ]_ [ according to] _[ π][θ]_ [ and]
compute the preference _pi_ = _pϕ_ ( _yi ≻_ _yi_ _[′][|][x][i]_ [)][ via the pref-]
erence model. Then, for each algorithm, we minimise the
respective following loss:
_B_
_B_
- ( _piL_ ALGO( _θ, xi, yi, yi_ _[′]_ [) + (1] _[ −]_ _[p][i]_ [)] _[L]_ [ALGO][(] _[θ, x][i][, y]_ _i_ _[′][, y][i]_ [))] _[ .]_
_i_ =1
Finally, regarding IPO-MD, the only difference with
(online)-IPO is how the generations are sampled. In theory,
we should sample from the mixture _πθ_ [1] _[−][β]_ ( _π_ [ref] ) _[β]_ which is
not feasible (see (Munos et al., 2023)). In practice, we sample from the one-step-at-a-time mixture ˆ _πβ_, which consists
at step _n_ to compute the mixture of logits between the online
and reference logits:
log(ˆ _πβ_ ( _.|y_ 0: _n−_ 1 _, x_ )) = (1 _−_ _β_ ) log( _πθ_ ( _.|y_ 0: _n−_ 1 _, x_ ))
+ _β_ log( _π_ [ref] ( _.|y_ 0: _n−_ 1 _, x_ )) + _C_ ( _y_ 0: _n−_ 1 _, x_ ) _,_
where _C_ ( _y_ 0: _n−_ 1 _, x_ ) is a path-dependent constant and sample according to the softmax of this mixture of logits.


_p_ ( _y ≻_ _y_ _[′]_ ) IPO IPO-MD DPO Nash-MD-PG SLiC RL
**IPO** 0.500 **0.515** (0.024) **0.608** (0.038) **0.621** (0.030) **0.608** (0.025) **0.791** (0.012)
**IPO-MD** **0.485** (0.024) 0.500 **0.600** (0.028) **0.608** (0.026) **0.594** (0.020) **0.778** (0.004)
DPO 0.392 (0.038) 0.400 (0.028) 0.500 0.520 (0.041) 0.493 (0.040) 0.727 (0.020)
Nash-MD-PG 0.379 (0.030) 0.392 (0.026) 0.480 (0.041) 0.500 0.479 (0.029) 0.729 (0.020)
SLiC 0.392 (0.025) 0.406 (0.020) 0.507 (0.040) 0.521 (0.029) 0.500 0.728 (0.010)
RL 0.209 (0.012) 0.222 (0.004) 0.273 (0.020) 0.271 (0.020) 0.272 (0.010) 0.500
_Table 2._ Side-by-side evaluation for summarisation. Each entry is the average preference of responses generated by the row method ( _y_ )
over the responses generated by the column method ( _y_ _[′]_ ). We also evaluated each method using 3 different seeds, computed a 3 _×_ 3
comparisons matrix across seeds and report the standard deviation of this matrix’s entries.
typically yields speed of around 0 _._ 25 training steps per second (24 hours per 20 _,_ 000 steps). We run our experiments
with default parameters 10 _[−]_ [4] for the learning rate, and a
default total of 30 _,_ 000 training steps, using a batch size of
32. The _τ_ factor is held constant throughout training, and
we do not employ any warmup steps. We use the AdaFactor
(Shazeer & Stern, 2018) optimizer with decay set to 0 _._ 8.


In this section, we present side-by-side evaluation scores
between the following online algorithms: RL, IPO, DPO,
SLiC, IPO-MD and Nash-MD-PG. Table 2 presents the side
by side scores for the summarisation task. The checkpoints
we evaluate are the best checkpoints except for RL that we
use as a baseline of comparison to find the best checkpoints.
The RL checkpoint is fixed and was chosen following the
protocol of (Munos et al., 2023) after sweeping over 6 values
of _τ_ ( _{_ 0 _._ 01 _,_ 0 _._ 02 _,_ 0 _._ 05 _,_ 0 _._ 1 _,_ 0 _._ 15 _,_ 0 _._ 2 _}_ ) and comparing the
performance against the SFT checkpoint after 10 _k_ learner
steps. To find the best checkpoints for the other algorithms,
we evaluate every checkpoint of each algorithm against the
RL checkpoint (over 2000 prompts sampled from a validation split) at different learning steps values (we checkpoint every 2000 learner steps for a total of 30 _k_ learner
steps), regularisation parameter _τ_ (we sweep over 5 values
_{_ 0 _._ 1 _,_ 0 _._ 5 _,_ 1 _._ 0 _,_ 5 _._ 0 _,_ 10 _._ 0 _}_ ) and also _β_ for IPO-MD and NashMD-PG (we sweep over 2 values 0 _._ 125 and 0 _._ 25) and we
take the best checkpoint. After finding the best checkpoint
for every algorithm (see App. B.3), we re-run each method
for 3 different seeds using the best hyperparamters. We then
perform 9 side-by-side evaluation (i.e., 3 _×_ 3 1vs1 evaluations between each of the 3 seeds for each pair of methods)
using 2000 prompts from a different validation split for each
comparison. We report mean and standard deviation across
these 9 comparisons.
On the summarisation task, looking only at the mean the
best algorithm is IPO as it beats all the other algorithms
on a side-by-side comparison. However, once we take into
consideration the standard deviation IPO and IPO-MD’s
performance becomes statistically indistinguishable, with
both algorithms consistently beating all the other algorithms.
This shows that those algorithms are indeed robust and
are closer to a Nash optimum than the other algorithms.
Those results are limited to a summarisation task and more
experiments should be conducted to validate these results on
a general conversational agent. However, we do think that
summarisation is a good test bed to showcase the quality of
human alignment algorithms because it is a complex and
high-in-demand task.


**Sweep on the regularisation parameter** _τ_ **:** Figure 1
sweeps the regularisation parameter for IPO and DPO. It’s
interesting to see for small values of regularisation IPO and
DPO behave very similarly, but for larger values the score
for IPO decays much faster. This matches the findings in
Azar et al. (2023) that show that IPO has a much stronger
regularisation effect than DPO as _τ_ gets larger.
_Figure 1._ Sweep on _τ_ vs. RL on the _summarisation_ task.
**Learning steps curve** Figure 2 shows the performance
against RL for IPO Online as it trains. We can see that as the
regularisation gets stronger, more training time is required
to reach the best performance.
_Figure 2._ Training Progress of Online IPO (as a function of number
of steps) on the _summarisation_ task.

6.2. Ablations and Additional Results on Summarisation


In this paper, we have identified several factors of variation, such as contrastivity, online/offline and regularisedsampling, between two recently proposed algorithms for
preference optimisation, IPO and Nash-MD-PG. In doing
so, we have introduced two new algorithms, online-IPO and
IPO-MD, that combine different strengths of these existing
algorithms, namely the loss function of IPO with the online sampling and regularised data distribution of Nash-MD.
Theoretical analysis reveals a surprising equivalence at the
level of expected update between online-IPO and self-play
in a regularised two-player preference optimisation game.
This important property is not possessed by online-DPO.
Finally, our empirical investigation on a summarisation task
also reveals that IPO-MD and online-IPO are promising
approaches to preference optimisation at scale as they are
the most robust algorithms. At the moment, our work is
restricted to model of size 770 _M_ on a single task, future
works will consist to scale our approach to a full conversational agent using a larger model (100+ billions parameter).
**Acknowledgements.** We are grateful for the collaborative
environment at Google DeepMind. We would like to thank
Shantanu Thakoor, Will Dabney, Doina Precup, Mohammad Gheshlaghi Azar, Olivier Bachem, Sertan Girgin, Matt
Hoffman, Nikola Momchev, Bobak Shahriari, and Piotr
Stanczyk.


Dong, H., Xiong, W., Goyal, D., Pan, R., Diao, S., Zhang,
J., Shum, K., and Zhang, T. RAFT: Reward rAnked
FineTuning for generative foundation model alignment.
_arXiv_, 2023.
Eisenstein, J., Nagpal, C., Agarwal, A., Beirami, A.,
D’Amour, A., Dvijotham, D., Fisch, A., Heller, K., Pfohl,
S. R., Ramachandran, D., Shaw, P., and Berant, J. Helping or herding? Rward model ensembles mitigate but do
not eliminate reward hacking. _arXiv_, 2023.
Fujimoto, S., Meger, D., and Precup, D. Off-policy deep reinforcement learning without exploration. In _Proceedings_
_of the International Conference on Machine Learning_,
2019.
Gao, L., Schulman, J., and Hilton, J. Scaling laws for
reward model overoptimization. In _Proceedings of the_
_International Conference on Machine Learning_, 2022.
Glaese, A., McAleese, N., Trebacz, M., Aslanides, J.,
Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M., Thacker, P., Campbell-Gillingham, L., Uesato, J., Huang, P.-S., Comanescu, R., Yang, F., See, A.,
Dathathri, S., Greig, R., Chen, C., Fritz, D., Elias, J. S.,
Green, R., Mokra, S., Fernando, N., Wu, B., Foley, R.,´
Young, S., Gabriel, I., Isaac, W., Mellor, J., Hassabis, D.,
Kavukcuoglu, K., Hendricks, L. A., and Irving, G. Improving alignment of dialogue agents via targeted human
judgements. _arXiv_, 2022.
Griffith, S., Subramanian, K., Scholz, J., Isbell, C. L., and
Thomaz, A. L. Policy shaping: Integrating human feedback with reinforcement learning. In _Advances in Neural_
_Information Processing Systems_, 2013.
Hejna, J., Rafailov, R., Sikchi, H., Finn, C., Niekum, S.,
Knox, W. B., and Sadigh, D. Contrastive prefence learning: Learning from human feedback without RL. _arXiv_,
2023.
Ivison, H., Wang, Y., Pyatkin, V., Lambert, N., Peters, M.,
Dasigi, P., Jang, J., Wadden, D., Smith, N. A., Beltagy,
I., and Hajishirzi, H. Camels in a changing climate:
Enhancing LM adaptation with Tulu 2. _arXiv_, 2023.
Jaques, N., Ghandeharioun, A., Shen, J. H., Ferguson, C.,
Lapedriza, A., Jones, N., Gu, S., and Picard, R. Way
off-policy batch deep reinforcement learning of implicit
human preferences in dialog. _arXiv_, 2019.
Jouppi, N. P., Kurian, G., Li, S., Ma, P. C., Nagarajan, R.,
Nai, L., Patil, N., Subramanian, S., Swing, A., Towles,
B., Young, C., Zhou, X., Zhou, Z., and Patterson, D. A.
TPU v4: An optically reconfigurable supercomputer for
machine learning with hardware support for embeddings.
In _Proceedings of the Annual International Symposium_
_on Computer Architecture_, 2023.
Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., and Raileanu, R. Understanding
the effects of RLHF on LLM generalisation and diversity.
_arXiv_, 2023.
Knox, W. B. and Stone, P. TAMER: Training an agent
manually via evaluative reinforcement. In _Proceedings of_
_the IEEE International Conference on Development and_
_Learning_, 2008.
Lee, H., Phatale, S., Mansoor, H., Lu, K., Mesnard, T.,
Bishop, C., Carbune, V., and Rastogi, A. RLAIF: Scaling
reinforcement learning from human feedback with AI
feedback. _arXiv_, 2023.
Liu, T., Zhao, Y., Joshi, R., Khalman, M., Saleh, M., Liu,
P. J., and Liu, J. Statistical rejection sampling improves
preference optimization. _arXiv_, 2023.
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap,
T. P., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning. In
_Proceedings of the International Conference on Machine_
_Learning_, 2016.
Munos, R., Valko, M., Calandriello, D., Azar, M. G., Rowland, M., Guo, D., Tang, Y., Geist, M., Mesnard, T.,
Michi, A., Selvi, M., Girgin, S., Momchev, N., Bachem,
O., Mankowitz, D. J., Precup, D., and Piot, B. Nash
learning from human feedback. _arXiv_, 2023.
Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim,
C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang,
X., Cobbe, K., Eloundou, T., Krueger, G., Button, K.,
Knight, M., Chess, B., and Schulman, J. WebGPT:
Browser-assisted question-answering with human feedback. _arXiv_, 2021.
[OpenAI. Introducing ChatGPT, 2022. URL https://](https://openai.com/blog/chatgpt)
[openai.com/blog/chatgpt.](https://openai.com/blog/chatgpt)
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright,
C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K.,
Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L.,
Simens, M., Askell, A., Welinder, P., Christiano, P., Leike,
J., and Lowe, R. Training language models to follow
instructions with human feedback. _arXiv_, 2022.
Pan, A., Bhatia, K., and Steinhardt, J. The effects of reward
misspecification: Mapping and mitigating misaligned
models. _arXiv_, 2022.
Pang, R. Y., Padmakumar, V., Sellam, T., Parikh, A. P., and
He, H. Reward gaming in conditional text generation.
In _Annual Meeting of the Association for Computational_
_Linguistics_, 2022.


Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning,
C. D., and Finn, C. Direct preference optimization: Your
language model is secretly a reward model. In _Advances_
_in Neural Information Processing Systems_, 2023.
Rame, A., Vieillard, N., Hussenot, L., Dadashi, R., Cideron,´
G., Bachem, O., and Ferret, J. WARM: On the benefits
of weight averaged reward models. _arXiv_, 2024.
Roberts, A., Chung, H. W., Levskaya, A., Mishra, G., Bradbury, J., Andor, D., Narang, S., Lester, B., Gaffney, C.,
Mohiuddin, A., Hawthorne, C., Lewkowycz, A., Salcianu,
A., van Zee, M., Austin, J., Goodman, S., Soares, L. B.,
Hu, H., Tsvyashchenko, S., Chowdhery, A., Bastings,
J., Bulian, J., Garcia, X., Ni, J., Chen, A., Kenealy, K.,
Clark, J. H., Lee, S., Garrette, D., Lee-Thorp, J., Raffel, C., Shazeer, N., Ritter, M., Bosma, M., Passos, A.,
Maitin-Shepard, J., Fiedel, N., Omernick, M., Saeta, B.,
Sepassi, R., Spiridonov, A., Newlan, J., and Gesmundo,
A. Scaling up models and data with t5x and seqio.
_arXiv_, 2022.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and
Klimov, O. Proximal policy optimization algorithms.
_arXiv_, 2017.
Shashi, N., Cohen, S. B., and Mirella, L. Don’t Give Me the
Details, Just the Summary! Topic-aware convolutional
neural networks for extreme summarization. In _Proceed-_
_ings of the Conference on Empirical Methods in Natural_
_Language Processing_, 2018.
Shazeer, N. M. and Stern, M. Adafactor: Adaptive learning
rates with sublinear memory cost. _arXiv_, 2018.
Shin, D., Dragan, A. D., and Brown, D. S. Benchmarks and
algorithms for offline preference-based reward learning.
_arXiv_, 2023.
Singhal, P., Goyal, T., Xu, J., and Durrett, G. A long way
to go: Investigating length correlations in RLHF. _arXiv_,
2023.
Skalse, J., Howe, N. H. R., Krasheninnikov, D., and Krueger,
D. Defining and characterizing reward gaming. In _Ad-_
_vances in Neural Information Processing Systems_, 2022.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R.,
Voss, C., Radford, A., Amodei, D., and Christiano, P. F.
Learning to summarize with human feedback. In _Ad-_
_vances in Neural Information Processing Systems_, 2020.
Swamy, G., Dann, C., Kidambi, R., Wu, Z. S., and Agarwal,
A. A minimaximalist approach to reinforcement learning
from human feedback. _arXiv_, 2024.
Tang, Y., Guo, Z. D., Zheng, Z., Calandriello, D., Munos,
R., Rowland, M., Richemond, P. H., Valko, M., Pires,
B. A., and Piot, B. Generalized preference optimization: [´]
A unified approach to offline alignment. _arXiv preprint_
_[
Touvron, H., Martin, L., Stone, K. R., Albert, P., Almahairi,
A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P.,
Bhosale, S., Bikel, D. M., Blecher, L., Ferrer, C. C., Chen,
M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W.,
Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn,
A. S., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez,
V., Khabsa, M., Kloumann, I. M., Korenev, A. V., Koura,
P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D.,
Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P.,
Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta,
R., Saladi, K., Schelten, A., Silva, R., Smith, E. M.,
Subramanian, R., Tan, X., Tang, B., Taylor, R., Williams,
A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan,
A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R.,
Edunov, S., and Scialom, T. Llama 2: Open foundation
and fine-tuned chat models. _arXiv_, 2023.
Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Rasul,
K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C.,
Habib, N., Sarrazin, N., Sanseviero, O., Rush, A. M., and
Wolf, T. Zephyr: Direct distillation of LM alignment.
_arXiv_, 2023.
Volske, M., Potthast, M., Syed, S., and Stein, B. TL;DR:¨
Mining Reddit to learn automatic summarization. In
_Proceedings of the Workshop on New Frontiers in Sum-_
_marization_ . Association for Computational Linguistics,
2017.
Wang, C., Jiang, Y., Yang, C., Liu, H., and Chen, Y. Beyond
reverse KL: Generalizing direct preference optimization
with diverse divergence constraints. _arXiv_, 2023.
Warnell, G., Waytowich, N., Lawhern, V., and Stone,
P. Deep TAMER: Interactive agent shaping in highdimensional state spaces. In _Proceedings of the AAAI_
_Conference on Artificial Intelligence_, 2018.
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R.,
Gontijo-Lopes, R., Morcos, A. S., Namkoong, H.,
Farhadi, A., Carmon, Y., Kornblith, S., and Schmidt, L.
Model soups: averaging weights of multiple fine-tuned
models improves accuracy without increasing inference
time. In _Proceedings of the International Conference on_
_Machine Learning_, 2022.
Wu, Y., Tucker, G., and Nachum, O. Behavior regularized
offline reinforcement learning. _arXiv_, 2019.
Yuan, W., Pang, R. Y., Cho, K., Sukhbaatar, S., Xu, J., and
Weston, J. Self-rewarding language models, 2024.


Yuan, Z., Yuan, H., Tan, C., Wang, W., Huang, S., and
Huang, F. RRHF: Rank responses to align language
models with human feedback without tears. _arXiv_, 2023.
Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., and
Liu, P. J. SLiC-HF: Sequence likelihood calibration with
human feedback. _arXiv_, 2023.
Zhuang, S. and Hadfield-Menell, D. Consequences of misaligned AI. In _Advances in Neural Information Process-_
_ing Systems_, 2020.


**RLHF.** Reinforcement learning from human feedback, as introduced in Christiano et al. (2017) (see also Bai et al. (2022a);
Ouyang et al. (2022)) and often based on proximal policy optimisation (Schulman et al., 2017), is a critical element of
making large language models helpful and aligned with preferences of human operators. While in itself it typically does not
result in improved benchmark performance (Touvron et al., 2023), RLHF is nonetheless key to satisfying human-mediated
interactions such as dialogue (Nakano et al., 2021; Ouyang et al., 2022). The complexity of the RLHF procedure (Casper
et al., 2023), which can also be accomplished by multiple reinforcement learning algorithms such as actor-critic (Mnih et al.,
2016; Glaese et al., 2022), has led to searching for algorithmic alternatives (Dong et al., 2023; Yuan et al., 2023; Zhao et al.,
2023).
**Recent developments in policy optimisation.** In the special case of an additional Bradley-Terry model (Bradley & Terry,
1952) assumption for the human reward model, reinforcement learning has been found redundant; this allows for casting the
problem of RLHF as a supervised one (Rafailov et al., 2023). Recent developments have focused on scaling the performance
of such direct policy optimisation (DPO) methods (Tunstall et al., 2023; Ivison et al., 2023), as well as generalising its
mathematical formulation (Azar et al., 2023; Wang et al., 2023; Tang et al., 2024). One of the key issues with direct policy
optimisation - and RLHF in general - resides in their propensity to game or _hack_ rewards (Amodei et al., 2016; Skalse et al.,
2022; Pan et al., 2022; Pang et al., 2022) and become overoptimised, or under-regularised (Gao et al., 2022; Singhal et al.,
2023; Kirk et al., 2023), which can be mitigated e.g. by using ensembling techniques (Wortsman et al., 2022; Eisenstein
et al., 2023; Coste et al., 2023; Rame et al.´, 2024). Rather than a reinforcement versus supervised learning dichotomy, it is
the distinction between _online_ and _offline_ (Jaques et al., 2019) methods that seems more relevant in practice, as an online
policy’s generations might start deviating substantially from the original dataset, leading to distribution shifts (Zhuang &
Hadfield-Menell, 2020; Shin et al., 2023). Finally, alignment can also result from the self-play form of a two-player game,
and not just single-policy optimisation. This perspective, taken in Munos et al. (2023); Swamy et al. (2024) has the added
benefit of encompassing both online and offline settings, enabling smooth interpolation between them via a hyperparameter.
In a similar vein, DPO has been shown to be able and improve thanks to iterated successive rounds (Yuan et al., 2024),
expanding on known machine-critic alignment methods such as reinforcement learning from AI feedback (Bai et al., 2022b;
Lee et al., 2023).


Figure 3 and Figure 4 show a sweep over the regularisation parameter for Online and Offline DPO and IPO vs. RLHF
and SFT respectively. It is interesting to note that the online versions significantly outperform the offline versions. This
is understandable as this setting favours tremendously online methods. Indeed, the starting point is an already fine-tuned
policy on summarisation data. Therefore, for online methods the first checkpoint is already able to sample good summaries
which make it very easy to obtain good rewards/preferences and from there optimise either the reward/preference model.
_Figure 3._ Sweep on Regularisation parameter vs. RL on the _summarisation_ task.

B.1. Regularisation Sweep for Online and Offline


_Figure 4._ Sweep on Regularisation parameter vs. SFT on the _summarisation_ task.


Here in Figure 5 we show a sweep over the mixing ratio _β_ for IPO-MD and how it affects its win-rate over the RL baseline
in summarisation. We draw the curves for a different learning rate and different learning steps than the optimal checkpoint
of IPO-MD to show that most of the time the mixing ratio still help improve the performance.
_Figure 5._ IPO-MD winning rate. The evolving parameter is the mixture ratio _β_ while we fix the learning rate to 3 _·_ 10 _[−]_ [5] and the
regularisation parameter to _τ_ = 1. We plot this curves for 3 different learning steps 12 _k_, 16 _k_ and 20 _k_ .


We report here the optimal _τ_, learning rate lr and, where applicable, mixture ratio _β_ used to obtain each algorithm’s best
performance in Table 2.
_τ_ lr _β_
RL 0.05 10 _[−]_ [4] N/A
IPO 1.0 10 _[−]_ [4] N/A
DPO 5.0 10 _[−]_ [4] N/A
SLiC 10.0 10 _[−]_ [4] N/A
IPO-MD 1.0 10 _[−]_ [4] 0.125
Nash-MD-PG 0.008 3 _·_ 10 _[−]_ [5] 0.125


**Proposition 4.2.** _The expected gradient of the Online IPO loss in Equation_ (11) _is identical to the self-play update direction_
_in the game with payoff as in Equation 9._
_Proof._ We calculate expressions for the update directions directly, assuming that the policy _π_ is parametrised via a vector _ϕ_ .


**Online IPO update.** The update direction for online IPO is given by the negative of the derivative:
_∇ϕ_ E
_Y,Y_ _[′]_ _∼_ SG[ _π_ ]
_Y_ [+] _,Y_ _[−]_ _∼λp_ ( _Y,Y_ _[′]_ )
�� - _π_ ( _Y_ ) _π_ ref ( _Y ′_ )
log
_π_ ( _Y_ _[′]_ ) _π_ [ref] ( _Y_ )
�� - _π_ ( _Y_ ) _π_ ref ( _Y ′_ )
log
- -
_−_ _τ_ _[−]_ [1] _/_ 2
- �2�
_−_ _τ_ _[−]_ [1] _/_ 2
_∝_ - _π_ ( _y_ ) _π_ ( _y_ _[′]_ ) _p_ ( _y ≻_ _y_ _[′]_ ) _∇ϕ_
_y,y_ _[′]_ _∈Y_
�� - _π_ ( _y_ ) _π_ ref ( _y′_ )
log
_π_ ( _y_ _[′]_ ) _π_ [ref] ( _y_ )
- _π_ ( _y_ ) _π_ ( _y_ _[′]_ ) _p_ ( _y ≻_ _y_ _[′]_ )� log - _π_ ( _y_ ) _π_ ref ( _y′_ )
_π_ ( _y_ _[′]_ ) _π_ [ref] ( _y_ )
_y,y_ _[′]_ _∈Y_
_∝_
_π_ ( _y_ _[′]_ ) _π_ [ref] ( _y_ )
-
_−_ _τ_ _[−]_ [1] _/_ 2 _∇ϕ_ log( _π_ ( _y_ ) _/π_ ( _y_ _[′]_ )) _._
We can simplify the above by first considering the terms with a factor of _τ_ _[−]_ [1] :
_−_ _τ_ _[−]_ [1] _/_ 2      - _π_ ( _y_ ) _π_ ( _y_ _[′]_ ) _p_ ( _y ≻_ _y_ _[′]_ )( _∇ϕ_ log _π_ ( _y_ ) _−∇ϕ_ log _π_ ( _y_ _[′]_ ))
_y,y_ _[′]_ _∈Y_


- _p_ ( _π ≻_ _y_ _[′]_ ) _π_ ( _y_ _[′]_ ) _∇ϕ_ log _π_ ( _y_ _[′]_ )
_y_ _[′]_ _∈Y_

= _−_ _τ_ _[−]_ [1] _/_ 2
= _−_ _τ_ _[−]_ [1] _/_ 2
 [�]
 [�]
[�] _π_ ( _y_ ) _p_ ( _y ≻_ _π_ ) _∇ϕ_ log _π_ ( _y_ ) _−_
_y∈Y_ _y_ _[′]_ _∈Y_
[�] _π_ ( _y_ ) _p_ ( _y ≻_ _π_ ) _∇ϕ_ log _π_ ( _y_ ) _−_
_y∈Y_ _y_ _[′]_ _∈Y_


- (1 _−_ _p_ ( _y_ _[′]_ _≻_ _π_ )) _π_ ( _y_ _[′]_ ) _∇ϕ_ log _π_ ( _y_ _[′]_ )
_y_ _[′]_ _∈Y_

= _−_ _τ_ _[−]_ [1][ �] _π_ ( _y_ ) _p_ ( _y ≻_ _π_ ) _∇ϕ_ log _π_ ( _y_ ) _._
_y∈Y_


Now considering the terms not involving _τ_ :
- _π_ ( _y_ ) _π_ ( _y_ _[′]_ ) _p_ ( _y ≻_ _y_ _[′]_ )
_y,y_ _[′]_ _∈Y_
- [�]
( _∇ϕ_ log _π_ ( _y_ ) _−∇ϕ_ log _π_ ( _y_ _[′]_ ))
 - _π_ ( _y_ )
log
_π_ [ref] ( _y_ )
- - _π_ ( _y′_ )
_−_ log
_π_ [ref] ( _y_ _[′]_ )
- _π_ ( _y_ ) _π_ ( _y_ _[′]_ ) _p_ ( _y ≻_ _y_ _[′]_ ) log - _π_ ( _y_ )
_π_ [ref] ( _y_
_y,y_ _[′]_ _∈Y_
=
_π_ [ref] ( _y_ )
_∇ϕ_ log _π_ ( _y_ )
- _π_ ( _y_ ) _π_ ( _y_ _[′]_ ) _p_ ( _y ≻_ _y_ _[′]_ ) log - _π_ ( _y′_ )
_π_ [ref] ( _y_ _[′]_
_y,y_ _[′]_ _∈Y_
- _π_ ( _y_ ) _π_ ( _y_ _[′]_ ) _p_ ( _y ≻_ _y_ _[′]_ ) log - _π_ ( _y_ )
_π_ [ref] ( _y_ )
_y,y_ _[′]_ _∈Y_
- _π_ ( _y_ ) _π_ ( _y_ _[′]_ ) _p_ ( _y ≻_ _y_ _[′]_ ) log - _π_ ( _y′_ )
_π_ [ref] ( _y_ _[′]_
_y,y_ _[′]_ _∈Y_
+
_π_ [ref] ( _y_ _[′]_ )
_∇ϕ_ log _π_ ( _y_ _[′]_ )
_−_
_π_ [ref] ( _y_ )
_∇ϕ_ log _π_ ( _y_ _[′]_ )
_−_
_π_ [ref] ( _y_ _[′]_ )
_∇ϕ_ log _π_ ( _y_ )
- _π_ ( _y_ ) _p_ ( _y ≻_ _π_ ) log - _π_ ( _y_ )
_π_ [ref] ( _y_
_y,y_ _[′]_ _∈Y_
_π_ [ref] ( _y_ _[′]_ )
=
_π_ [ref] ( _y_ )
_∇ϕ_ log _π_ ( _y_ _[′]_ )
_∇ϕ_ log _π_ ( _y_ ) +
- _π_ ( _y_ _[′]_ ) _p_ ( _π ≻_ _y_ _[′]_ ) log - _π_ ( _y′_ )
_π_ [ref] ( _y_
_y,y_ _[′]_ _∈Y_
- _π_ ( _y_ ) _π_ ( _y_ _[′]_ ) _p_ ( _y ≻_ _y_ _[′]_ ) log - _π_ ( _y_ )
_π_ [ref] ( _y_
_y,y_ _[′]_ _∈Y_
_−_
_π_ [ref] ( _y_ )
_∇ϕ_ log _π_ ( _y_ _[′]_ )
- _π_ ( _y_ ) _π_ ( _y_ _[′]_ )(1 _−_ _p_ ( _y ≻_ _y_ _[′]_ )) log - _π_ ( _y_ )
_π_ [ref] ( _y_
_y,y_ _[′]_ _∈Y_
_−_
_π_ [ref] ( _y_ )
_∇ϕ_ log _π_ ( _y_ _[′]_ )
_π_ [ref] ( _y_ _[′]_ )
_∇ϕ_ log _π_ ( _y_ _[′]_ )
- _π_ ( _y_ ) _p_ ( _y ≻_ _π_ ) log - _π_ ( _y_ )
_π_ [ref] ( _y_
_y,y_ _[′]_ _∈Y_
=
_π_ [ref] ( _y_ )
_∇ϕ_ log _π_ ( _y_ ) +
- _π_ ( _y_ _[′]_ )(1 _−_ _p_ ( _y ≻_ _π_ )) log - _π_ ( _y′_ )
_π_ [ref] ( _y_
_y,y_ _[′]_ _∈Y_
- _π_ ( _y_ ) _π_ ( _y_ _[′]_ ) log - _π_ ( _y_ )
_π_ [ref] ( _y_
_y,y_ _[′]_ _∈Y_
_−_
_π_ [ref] ( _y_ )
_∇ϕ_ log _π_ ( _y_ _[′]_ )
- _π_ ( _y_ _[′]_ ) log - _π_ ( _y′_ )
_π_ [ref] ( _y_
_y,y_ _[′]_ _∈Y_
=
_π_ [ref] ( _y_ _[′]_ )
_∇ϕ_ log _π_ ( _y_ _[′]_ )
- _π_ ( _y_ ) _π_ ( _y_ _[′]_ ) log - _π_ ( _y_ )
_π_ [ref] ( _y_
_y,y_ _[′]_ _∈Y_
_−_
_π_ [ref] ( _y_ )
_∇ϕ_ log _π_ ( _y_ _[′]_ )
- _π_ ( _y_ _[′]_ ) log - _π_ ( _y′_ )
_π_ [ref] ( _y_
_y,y_ _[′]_ _∈Y_
=
_π_ [ref] ( _y_ _[′]_ )
_∇ϕ_ log _π_ ( _y_ _[′]_ ) _._
**Self-play update.** Self-play leads to the update direction given by
��
- _π_ ( _y_ ) log( _π_ ( _y_ ) _/π_ [ref] ( _y_ ))
_y∈Y_
_∇ϕ_
     _π_ ( _y_ )SG[ _π_ ( _y_ _[′]_ )] _p_ ( _y ≻_ _y_ _[′]_ ) _−_ _τ_
_y,y_ _[′]_ _∈Y_ _y∈Y_
=
- _π_ ( _y_ ) _π_ ( _y_ _[′]_ ) _p_ ( _y ≻_ _y_ _[′]_ ) _∇ϕ_ log _π_ ( _y_ ) _−_ _τ_
_y,y_ _[′]_ _∈Y_ _y∈Y_
- _π_ ( _y_ ) log( _π_ ( _y_ ) _/π_ [ref] ( _y_ )) _∇ϕ_ log _π_ ( _y_ )
_y∈Y_
=
_π_ ( _y_ ) _p_ ( _y ≻_ _π_ ) _∇ϕ_ log _π_ ( _y_ ) _−_ _τ_  _y_ _y∈Y_
- _π_ ( _y_ ) log( _π_ ( _y_ ) _/π_ [ref] ( _y_ )) _∇ϕ_ log _π_ ( _y_ ) _._
_y∈Y_
Therefore the expected online IPO update direction is exactly the same as that of self-play.
**Proposition 5.1.** _The gradients of the algorithms Nash-MD-PG(β) and IPO-MD(β) are, respectively,_
_gNash-MD-PG_ ( _β_ ) = _−_ E _y∼π_ [ _g_ ( _y_ )]
_gIPO-MD_ ( _β_ ) = _−_ _τ_ [2] [E] _[y][∼]_ [(] _[π]_ [)][1] _[−][β]_ [(] _[π]_ [ref] [)] _[β]_ [ [] _[g]_ [(] _[y]_ [)]]