This paper concerns the problem of aligning large language models (LLMs) to bias their outputs
toward human preferences. There are now a wealth of approaches to this problem [e.g., Ouy+22;
Chr+17; Kau+23; Li+24; Raf+23; Aza+24]. Here, we interested in the _best-of-n_ (BoN) sampling
strategy. In BoN sampling, we draw _n_ samples from the LLM, rank them on the attribute of
interest, and return the best one. This simple procedure is surprisingly effective in practice
[Bei+24; Wan+24; GSH23; Eis+23]. We consider two fundamental questions about BoN:
1. What is the relationship between BoN and other approaches to alignment?
2. How can we effectively train a LLM to mimic the BoN sampling distribution?
In brief: we find that the BoN distribution is (essentially) the optimal policy for maximizing
win rate while minimally affecting off-target aspects of generation, and we develop an effective
method for aligning LLMs to mimic this distribution. Together, these results yield a highly effective
alignment method; see Figure 1 for an illustration.
**LLM Alignment** The goal of alignment is to bias the outputs of an LLM to be good on some
target attribute (e.g., helpfulness), while minimally changing the behavior of the model on
off-target attributes (e.g., reasoning ability). Commonly, the notion of goodness is elicited by
collecting pairs of responses to many prompts, and asking (human or AI) annotators to choose the
better response. Then, these pairs are used to define a training procedure for updating the base
LLM to a new, aligned, LLM that outputs responses that are better in the target attribute.
38th Conference on Neural Information Processing Systems (NeurIPS 2024).
Average Length (Response) vs. Win Rate
|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|Col11|Col12|
|---|---|---|---|---|---|---|---|---|---|---|---|
|||||||||||||
|||||||||||||
|||||||||||||
|||||||||||||
|||||||||~~Met~~
~~B~~|~~hod~~
~~N sa~~|~~ pling~~||
|||||||||
B
D
|
oNBo
PO ori
|
N
 ginal
|
 HH
|
|||||||||
~~IP~~
R|
~~O orig~~
eferen|
~~ inal H~~
ce Mo|
~~ H~~
 del|
|||||||||||||
Average Length of Responses
**Figure 1:** BoNBoN alignment achieves high win rates while minimally affecting off-target attributes of
generation. **Left** : Average length of responses versus win rate of models aligned using each method on the
Anthropic helpful and harmless single turn dialogue task, using _n_ = 8. As predicted by theory, best-of- _n_
achieves an excellent win rate while minimally affecting the off-target attribute length. Moreover, the
BoNBoN aligned model effectively mimics this optimal policy, achieving a much higher win rate at low
off-target drift than other alignment approaches. **Right** : Sample responses from models with similar win
rates to BoNBoN. Other methods require higher off-target deviation to achieve a comparably high win
rate. We observe that this significantly changes their behavior on off-target aspects. Conversely, BoNBoN
only minimally changes off-target behavior. See section 5 for details.
There are two main approaches. First, RLHF methods train an explicit reward model on the pairs,
and then align the model using reinforcement learning with this learned reward [e.g., Ouy+22;
Kau+23]. Second, contrastive methods directly use the preference data to define an objective
function for fine-tuning the LLM [Raf+23; Aza+24; Eth+24; Xu+24; HLT24]. In both cases,
the trade-off between alignment and off-target behavior is controlled by a hyper-parameter that
explicitly penalizes the divergence from the base LLM. For example, in the reinforcement learning
setting, this is done by adding a regularization term that penalizes the estimated KL divergence
between the aligned model and the reference model.
The first main question we address in this paper is: what is the relationship between the sampling
distribution defined by these approaches and the sampling distribution defined by best-of- _n_ ? This
is important, in particular, because in principle we could forgo the explicit alignment training and
just use BoN sampling. However, it is not clear when each option should be preferred.
Now, the comparison of training-aligned models and BoN is not fully fair. The reason is that
producing a BoN sample requires drawing _n_ samples from the base LLM (instead of just one).
This is a substantial computational cost. The second main question we address is: if we do in fact
want to sample from the BoN distribution, how can we train a LLM to mimic this distribution? If
this can be done effectively, then the inference cost of BoN sampling can be avoided.
We answer these questions with the following contributions:
1. We show that the BoN sampling distribution can be embedded in a common class with
the distributions produced by training-based alignment methods. Within this common
class, we derive the distribution with the best possible trade-off between win-rate against
the base model vs KL distance from the base model. Then, we show that the BoN
distribution is essentially equal to this Pareto-optimal distribution.
2. We then develop an effective method for training a LLM to mimic the BoN sampling
distribution. In essence, the procedure draws best-of- _n_ and worst-of- _n_ samples as
training data, and combines these with an objective function we derive by exploiting the
analytical form of the BoN distribution. We call this procedure _BoNBoN Alignment_ .
3. Finally, we show empirically that BoNBoN Alignment yields models that achieve high
win rates while minimally affecting off-target aspects of the generations, outperforming
baselines.


Given a prompt _x_, a large language model (LLM) samples a text completion _Y_ . We denote the
LLM by _π_ and the sampling distribution of the completions by _π_ ( _y | x_ ).
Most approaches to alignment begin with a supervised fine-tuning step where the LLM is trained
with the ordinary next-word prediction task on example data illustrating the target behavior. We
denote the resulting model by _π_ 0, and call it the _reference model_ . The problem we are interested
in is how to further align this model.
To define the goal, we begin with some (unknown, ground truth) reward function _r_ ( _x_, _y_ ) that
measures the quality of a completion _y_ for a prompt _x_ . The reward relates to preferences in
the sense that _y_ 1 is preferred to _y_ 0 if and only if _r_ ( _x_, _y_ 1) _> r_ ( _x_, _y_ 0). Informally, the goal is to
produce a LLM _πr_ where the samples have high reward, but are otherwise similar to the reference
model.
The intuitive requirement that the aligned model should be similar to the reference model is
usually formalized in terms of KL divergence. The _context-conditional KL divergence_ and _the KL_
_divergence from πr to π_ 0 _on a prompt set D_ are defined as:
�KL( _πr_ _∥π_ 0 _| x_ ) := � _y∼πr_ ( _y | x_ )
�log - _πr_ ( _y | x_ )
_π_ 0( _y | x_ )
��
,
�KL( _πr_ _∥π_ 0) := � _x∼D_ [�KL( _πr_ _∥π_ 0 _| x_ )] .
We also need to define what it means for samples from the language model to have high reward.
Naively, we could just look at the expected reward of the samples. However, in the (typical)
case where we only have access to the reward through preference judgements, the reward is
only identified up to monotone transformation. The issue is that expected reward value is
not compatible with this unidentifiability. [1] Instead, we consider the win rate of the aligned
model against the reference model. The idea is, for a given prompt, draw a sample from the
aligned model and a sample from the reference model, and see which is preferred. This can be
mathematically formalized by defining the _context-conditional win rate_ and _the overall win rate_
_on a prompt set D_ :
_pπr_ _≻π_ 0 _| x_ := � _Y ∼πr_ ( _y|x_ ), _Y_ 0 _∼π_ 0( _y|x_ )( _r_ ( _x_, _Y_ ) _≥_ _r_ ( _x_, _Y_ 0)),
_pπr_ _≻π_ 0 := � _x∼D_ �� _Y ∼πr_ ( _y|x_ ), _Y_ 0 _∼π_ 0( _y|x_ )( _r_ ( _x_, _Y_ ) _≥_ _r_ ( _x_, _Y_ 0))� .
**Reinforcement Learning from Human Feedback (RLHF)** The most studied approach to
alignment is RLHF. This procedure follows two steps. First, the reward function is explicitly
estimated from preference data, using the Bradley-Terry [BT52] model. Second, this estimated
reward function is used in a KL-regularized reinforcement learning procedure to update the
LLM. Denoting the estimated reward function by ˆ _r_, the objective function for the reinforcement
learning step is:
_�RLHF_ ( _πθ_ ; _π_ 0) = _−_        - _x∼D_, _y∼πθ_ ( _y | x_ ) [ˆ _r_ ( _x_, _y_ )] + _β_ �KL( _πθ_ _∥π_ 0), (2.1)
where _D_ is a prompt set and _β_ is a hyper-parameter to control the deviation of _πθ_ from the
reference model _π_ 0. The policy _πr_ is learned by finding the minimizer of the objective function
in (2.1); e.g., using PPO [Sch+17].
**Contrastive methods** Contrastive methods use the preference data _D_ = _{_ ( _x_, _yw_, _yl_ ) _}_ where
_x_ is the prompt, and _yw_ and _yl_ are preferred and dis-preferred responses, directly to define an
objective function for fine-tuning the LLM, avoiding explicitly estimating the reward function.
For example, the DPO [Raf+23] objective is:
_π_ 0( _yl | x_ )
_�DPO_ ( _πθ_ ; _π_ 0) = _−_ �( _x_, _yw_, _yl_ ) _∼D_
- - _[|][ x]_ [)]
log _σ_ _β_ log _[π][θ]_ [(] _[y][w]_
_[π]_ _π_ _[θ]_ 0( [(] _y_ _[y]_ _w_ _[w]_ _|_ _[|]_ _x_ _[ x]_ ) [)] _[−]_ _[β]_ [ log] _[π]_ _π_ _[θ]_ 0( [(] _y_ _[y]_ _l_ _[l]_ _|_ _[|]_ _x_ _[ x]_ ) [)]
��
. (2.2)
The aligned model is found by optimizing this objective directly (via gradient descent).
1Fundamentally, the expectation of the transformed reward is not the reward of the transformed
expectation.
**Bradley-Terry and Alignment Targets** In RLHF, the reward function is estimated using the
Bradley-Terry model, which relates noisy observed preferences to rewards by:
P( _y_ 1 _≻_ _y_ 0 _| x_ ) = _σ_ ( _r_ ( _x_, _y_ 1) _−_ _r_ ( _x_, _y_ 0)), (2.3)
where _σ_ ( _·_ ) is the sigmoid function. In the particular case that the Bradley-Terry model is wellspecified, then it can be shown that the analytic solution to both (2.1) and (2.2) is:
                - 1                _π_ [RLHF] _r_ ( _y | x_ ) _∝_ exp _β_ _[r]_ [(] _[x]_ [,] _[ y]_ [)] _π_ 0( _y | x_ ). (2.4)
That is, the alignment procedures target an exponential tilting of the reference model by the
reward function. Of course, it is not obvious when the Bradley-Terry model is well-specified,
nor whether this particular tilting is a desirable target. Other works have considered explicitly
or implicitly transforming the reward function to change the target distribution [Wan+24;
Aza+24]. Nevertheless, these works also take the target distribution to be a tilting of the
reference distribution.
**Best-of-** _n_ **sampling** The best-of- _n_ procedure is as follows. Given a prompt _x_, sample
_y_ 1, _y_ 2,..., _yn_ independently from the reference model _π_ 0( _y | x_ ). Then, select the response
with the highest reward _r_ ( _x_, _yi_ ) as the final response. That is,
_y_ = _yi_ such that _r_ ( _x_, _yi_ ) = max1 _≤_ _j≤n_ _[r]_ [(] _[x]_ [,] _[ y]_ _[j]_ [)][.] (2.5)

BoNBoN Alignment for Large Language Models and the Sweetness of
  Best-of-n Sampling


The first question we address is: what is the relationship between the best-of- _n_ distribution, and
the distribution induced by training-based alignment methods?

3 Best-of- _n_ is Win-Rate vs KL Optimal


We begin with the underlying distribution of best-of- _n_ sampling. Let _Q_ _x_ denote the cumulative
distribution function of _r_ ( _x_, _Y_ 0), where _Y_ 0 _∼_ _π_ 0( _· | x_ ). Suppose _r_ ( _x_, _·_ ) : _�→_ - is an one-to-one
mapping and _π_ 0( _y|x_ ) is continuous [2], then the conditional density of the best-of-n policy is
_π_ [(] _r_ _[n]_ [)][(] _[y][ |][ x]_ [)][ :][=] _[ nQ]_ _[x]_ [(] _[r]_ [(] _[x]_ [,] _[ y]_ [))] _[n][−]_ [1] _[π]_ [0][(] _[y][ |][ x]_ [)][.] (3.1)
Compare this to the RLHF policy _π_ [RLHF] _r_ in (2.4). In both cases, the sampling distribution is a
re-weighted version of the reference model _π_ 0, where higher weights are added to those responses
with higher rewards. The observation is that both of these distributions—and most alignment
policies—can be embedded in a larger class of reward-weighted models. For any prompt _x_ and
reward model _r_, we can define the _f_ _x_ -aligned model as:
_πr_ ( _y | x_ ) _∝_ _f_ _x_ ( _r_ ( _x_, _y_ )) _π_ 0( _y | x_ ), (3.2)
where _f_ _x_ is a non-decreasing function that may vary across different prompts.
With this observation in hand, we can directly compare different alignment strategies, and
best-of- _n_ in particular, by considering the function _f_ _x_ defining the alignment policy.

3.1 A Common Setting for Alignment Policies


To understand when different alignment policies are preferable, we need to connect the choice
of _f_ _x_ with a pragmatic criteria for alignment. The high-level goal is to produce a policy that
samples high-reward responses while avoiding changing off-target attributes of the text. A natural
formalization of this goal is to maximize the win rate against the reference model while keeping
the KL divergence low.
2This is a reasonable simplification since we only care about the distribution of _r_ ( _x_, _y_ ) in the onedimensional space and we consider the scenario where diverse responses without a dominant one are
expected. More details refer to appendix B for discussion.
1.0
0.9
0.8
0.7
0.6
0.5
KL Divergence versus Win Rate
|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|Col11|
|---|---|---|---|---|---|---|---|---|---|---|
||||||||||||
||||||||||||
||||||||||||
||||||||||||
||||||||||||
||||||||||||
||||||||||||
||||||||||||
||||||||||||
||||||||||||
||||||||||||
KL Divergence
0.0100
0.0075
0.0050
0.0025
0.0000
Win Rate Gain from Optimal Policy over Best−of−n Sampling
|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|
|---|---|---|---|---|---|---|---|---|---|
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||
n
**Figure 2:** The BoN is essentially the same as the optimal policy in terms of win rate versus KL divergence.
**Left** : The win rate versus KL divergence curves of BoN and optimal policy. **Right** : The win rate difference
between optimal policy and BoN policy for different _n_ .
**Optimal Policy** Our aim is to find the policy with the highest possible win rate at each KL
divergence level:
max _π_        - _x∼D_ �� _Y ∼π_ ( _y | x_ ), _Y_ 0 _∼π_ 0( _y | x_ )( _r_ ( _x_, _Y_ ) _≥_ _r_ ( _x_, _Y_ 0))� (3.3)
subject to �KL( _π∥π_ 0) = _d_ .
Now, this equation only depends on _Y_ through the reward function _r_ ( _x_, _y_ ). Defining _Q_ _x_ ( _r_ ( _x_, _Y_ ))
as the distribution of _r_ ( _x_, _Y_ ) under _π_ 0( _Y | x_ ), we can rewrite the objective as:
max       - _x∼D_, _y∼π_ ( _y | x_ ) [ _Q_ _x_ ( _r_ ( _x_, _y_ ))] subject to �KL( _π∥π_ 0) = _d_,
_π_
By duality theory [BTN01], there is some constant _β >_ 0 such that this problem is equivalent
to:
max        - _x∼D_, _y∼π_ ( _y | x_ ) [ _Q_ _x_ ( _r_ ( _x_, _y_ ))] _−_ _β_ (�KL( _π∥π_ 0) _−_ _d_ ) . (3.4)
_π_
Now, we can immediately recognize this objective as the same as the RLHF objective in (2.1)
with the transformed reward function ˜ _r_ ( _x_, _y_ ) = _Q_ _x_ ( _r_ ( _x_, _y_ )). Then, the analytic solution to this
problem is
_π_ [optimal] _r_ _∝_ _π_ 0( _y|x_ ) _e_ _[cQ]_ _[x]_ [(] _[r]_ [(] _[x]_ [,] _[y]_ [))], (3.5)
where _c_ is a constant determined by the KL divergence penalty.
The following theorem makes the preceding argument precise. To simplify the argument, we
will assume that the rewards assigned to outputs of the language model are continuous. This
simplifying assumption ignores that there are only a countably infinite number of possible
responses to any given prompt. However, given the vast number of possible responses, the
assumption is mild in practice. Refer to appendix B for a more detailed discussion.
**Theorem 1.** _Let π_ _[optimal]_ _r_, _c_ _be the solution to (3.3). Then, for all x, the density of the optimal policy is_
_π_ _[optimal]_ _r_, _c_ ( _y | x_ ) = _π_ 0( _y | x_ ) exp _{cQ_ _x_ ( _r_ ( _x_, _y_ )) _}_ _/Zr_ _[c]_ [,] (3.6)
_where Zr_ _[c]_ _[is the normalizing constant, and c is a positive constant such that]_
( _c −_ 1) _e_ _[c]_ + 1
1) _e_ _[c]_ + 1 - _ec −_ 1
_−_ log
_e_ _[c]_ _−_ 1 _c_
_c_
= _d_ . (3.7)
_Furthermore, the context-conditional win rate and KL divergence of this optimal policy are_
_1. Context-conditional win rate: pπoptimalr_, _c_ _≻π_ 0 _| x_ [=] [(] _[c]_ _c_ _[−]_ ( _e_ [1] _[c]_ [)] _−_ _[e][c]_ 1 [+] ) [1] _[.]_
                  -                   _2. Context-conditional KL divergence:_ �KL _π_ _[optimal]_ _r_, _c_ _∥π_ 0 _| x_ = [(] _[c][−]_ _e_ [1] _[c]_ [)] _−_ _[e]_ 1 _[c]_ [+][1]
_e_ [1] _[c]_ [)] _−_ _[e]_ 1 _[c]_ [+][1] _−_ log - _ec_ _c−_ 1
_c−_ 1 - _._
_Since for any prompt x, both the context conditional win rate and KL divergence are constants, the_
_overall win rate pπoptimalr_, _c_ _≻π_ 0 _[and KL divergence]_ [ �][KL][(] _[π][optimal]_ _r_, _c_ _∥π_ 0) _on any prompt set D are also these_
_values. [Proof]._

3.2 Optimality: Win Rate versus KL divergence


Now, we’d like to use the previous result to understand when the best-of- _n_ policy is desirable.
The win rate and KL divergence can be calculated with essentially the same derivation:
**Theorem 2.** _The context-conditional win rate and KL divergence of the best-of-n policy are:_
_n_
_1. Context-conditional win rate: pπ_ ( _rn_ ) _[≻][π]_ 0 _[|][ x]_ [ =] _n_ +1 _[.]_
_2. [JH22] Context-conditional KL divergence:_ �KL    - _π_ [(] _r_ _[n]_ [)] _[∥][π]_ [0] _[ |][ x]_    - = log( _n_ ) _−_ _[n][−]_ _n_ [1] _[.]_ [3]
_Since both are constants, the overall win rate pπ_ ( _rn_ ) _[≻][π]_ 0 _[and Kl divergence]_ [ �][KL][(] _[π]_ [(] _r_ _[n]_ [)] _[∥][π]_ [0][)] _[ on any]_
_prompts set D are the same values. [Proof]._
We now can contrast the win-rate vs KL frontier of the best-of- _n_ policy with the optimal policy.
Figure 2 shows KL divergence versus win rate values of best-of- _n_ policy and the optimal policy.
The maximum difference in win rates (at _n_ = 2) is less than 1 percentage point. Larger values of
_n_ approximate the optimal policy even more closely. In summary:

3.3 The best-of- _n_ policy is essentially optimal


RLHF and contrastive alignment methods include a hyper-parameter that attempts to explicitly
control the trade-off between KL divergence and model reward. By contrast, best-of- _n_ only
controls the KL drift implicitly. This can actually be a substantive advantage. There are two reasons.
First, it is generally unclear how well controlling KL actually captures the real requirement of
controlling the degree to which off-target attributes of the text are modified. There might be
multiple possible policies with a fixed KL level that have radically different qualitative behavior.
Second, in practice, the KL drift from the base policy needs to be estimated from a finite data
sample. This may be extremely difficult—it is a very high dimensional estimation problem.
Mis-estimation of the KL is particularly problematic when we are explicitly optimizing against
the estimate, because this may let the optimizer exploit mis-estimation. Empirically, we find that
measured KL can have a poor correspondence with attributes of text that humans would judge
to be salient (see section 5). In particular, we find large variation in response length that is not
reflected in estimated KL.
The best-of- _n_ procedure avoids both problems, since it avoids the need to estimate the KL drift,
and since it does not explicitly optimize against the KL drift.

3.4 Implicit vs Explicit KL regularization


From section 3, we know that the best-of- _n_ policy is essentially optimal in terms of win rate
and KL divergence. Accordingly, it is often a good choice for the alignment policy. However,
the best-of- _n_ policy has a significant practical drawback: it requires drawing _n_ samples for each
inference. This is a substantial computational expense. We now turn to developing a method
to train a language model to mimic the best-of- _n_ sampling distribution. We call this method
_BoNBoN Alignment_ .
**Setup** The basic strategy here will be to use best-of- _n_ samples to train a language model to
mimic the best-of- _n_ policy. We produce the training data by sampling _n_ responses from the
reference model _π_ 0, and ranking them. The best and worst data are the samples with highest and
lowest reward. Their corresponding best-of and worst-of _n_ sampling distributions are denoted as
_π_ [(] _r_ _[n]_ [)] and _π_ [(] _r_ [1][)][. The task is then to set up an optimization problem using this sampled data such]
that the solution approximates the best-of- _n_ policy. To that end, we consider objective functions
that have the best-of- _n_ policy as a minimizer in the infinite data limit. (In practice, as usual, we
approximate the expectation with an average.)
3Beirami et al. [Bei+24] discuss that since the distribution of the language model is discrete, _π_ ( _rn_ ) has a
different form from that in Theorem 2, and the actual KL divergence is smaller. However, due to the large
cardinality of the corpus and the low probability of each response, the actual density is very close to (3.1)
and the KL divergence is almost its upper bound log( _n_ ) _−_ _[n][−]_ _n_ [1] [.]
**SFT-BoN.** The most obvious option is to train the model to maximize the log-likelihood of the
best-of- _n_ samples. The associated objective is:
_�_ SFT _−_ BoN( _πθ_ ; _π_ 0) = _−_      - _x∼D_, _y_ ( _n_ ) _∼π_ ( _rn_ ) �log _πθ_ ( _y_ ( _n_ ) _| x_ )�, (4.1)
and it is well-known that the minimizer is _π_ [(] _r_ _[n]_ [)][. The training procedure is simply to minimize]
the sample-average version of this objective. We call this training method _SFT-BoN_ because it
is supervised fine-tuning on best-of- _n_ samples. Although SFT-BoN is valid theoretically, it turns
out to be data inefficient, and we observe only marginal improvement over the reference model
empirically (see section 5).
**IPO-BoN.** A limitation of the best-of- _n_ procedure is that it only makes use of the winning sample,
throwing away the rest. Another intuitive option is to construct a pairwise dataset and train the
language model by a contrastive method. Concretely, we construct the pairwise data by picking
the best and worst responses. We want to construct an objective function using this paired data
that has the best-of- _n_ policy as a minimizer.
The key result we require is:
**Theorem 3.** _For any fixed n,_
_π_ 0( _y_ (1) _| x_ )
_π_ [(] _r_ _[n]_ [)][(] _[y]_ [(] _[n]_ [)] _[ |][ x]_ [)]
log
_πr_ _[n]_ [(] _[y]_ [(] _[n]_ [)] _[ |][ x]_ [)] _−_ log _π_ 0( _y_ ( _n_ ) _| x_ )
_π_ [(] _r_ _[n]_ [)][(] _[y]_ (1) _[|][ x]_ [)] _π_ 0( _y_ (1) _| x_ )
=,
2 _βn_ _[∗]_
- _x∼D_, _y_ ( _n_ ) _∼π_ ( _rn_ ) [,] _[y]_ (1) _[∼][π]_ [(] _r_ [1][)]
_where_
_[Proof]._
_βn_ _[∗]_ [=] 2( _n −_ 1) ~~�~~ _nk_ = _−_ 11 [1] _[/][k]_ . (4.2)
Following this result, we define the contrastive objective function as:
��
_πθ_ ( _y_ ( _n_ ) _| x_ ) _π_ 0( _y_ ( _n_ ) _| x_ )
_πθ_ ( _y_ (1) _| x_ ) _[−]_ [log] _π_ 0( _y_ (1) _| x_ )
0 _y_ ( _n_ ) _| x_ 1
_π_ 0( _y_ (1) _| x_ ) _[−]_ 2 _βn_ _[∗]_
�2 [�]
_�_ IPO _−_ BoN( _πθ_ ; _π_ 0) = � _x∼D_, _y_ ( _n_ ) _∼π_ ( _rn_ ) [,] _[y]_ (1) _[∼][π]_ [(] _r_ [1][)]
_πθ_ ( _y_ ( _n_ ) _| x_ )
log
. (4.3)
The optimizer of this objective is a policy where the log-likelihood ratio of the best and worst
samples is equal to that of the best-of- _n_ policy. We call this training method _IPO-BoN_ because
it is essentially the IPO objective on the best-and-worst samples, with a particular choice for
the IPO hyper parameter. We emphasize that the IPO-BoN objective does not involve any hyper
parameters, there is only one choice for _βn_ _[∗]_ [for each] _[ n]_ [.]
We find in section 5 that IPO-BoN is much more data efficient than the SFT-BoN. However, this
method (like IPO) has the disadvantage that it only controls the likelihood _ratios_ on the sampled
data. In particular, this means that the optimizer can cheat by reducing the likelihood of _both_
the winning and losing responses, so long as the loser’s likelihood decreases more (so the ratio
still goes up). Reducing the probability of both the winning and losing examples requires the
optimized model to shift probability mass elsewhere. In practice, we find that it tends to increase
the probability of very long responses.
**BonBon Alignment** We can now write the BoNBoN objective:
We call the procedure BoNBoN because it is a combination of two objective functions that have the
best-of- _n_ policy as a minimizer. Relative to SFT alone, BoNBoN can be understood as improving
data efficiency by making use of the worst-of- _n_ samples. Relative to IPO alone, BoNBoN can
be understood as preventing cheating by forcing the likelihood of the best-of- _n_ samples to be
high. We emphasize that both objective functions target the same policy; neither is regularizing
towards some conflicting objective. That is, the trade-off between win-rate and off-target change
is handled implicitly by the (optimal) best-of- _n_ procedure. This is in contrast to approaches
that manage this trade-off explicitly (and sub-optimally) by regularizing towards the reference
model. Reflecting this, we choose _α_ so that the contribution of each term to the total loss is
approximately equal.


We study two tasks: _a) single-turn dialogue generation_, for which we conduct experiments on the
Anthropic Helpful and Harmless (HH) dataset [Bai+22] and _b) text summarization_, for which we
use the OpenAI TL;DR dataset [Sti+20]. Due to computational constraints, we filter the HH data
to only keep prompts for which response length is less than 500 characters, resulting in 106,754
training dialogues. For TL;DR dataset, we discard instances where the input post length is less
than 90 characters, resulting in 92,831 (14,764 prompts) training posts. Each example in both
datasets contains a pair of responses that were generated by a large language model along with
a label denoting the human-preferred response among the two generations.
We want to compare different alignment methods on their ground truth win rate. Accordingly,
we need a ground truth ranker. To that end, we construct data by using an off-the-shelf reward
model [4] as our ground truth. (In particular, we relabel the human preferences).
As the reference model, we fine-tune Pythia-2.8b [Bid+23] with supervised fine-tuning (SFT) on
the human-preferred completions from each dataset. For alignment methods other than BoNBoN,
we draw _n_ = 8 completions for each prompt, and we use the best and worst completions as
training data for them. For BoNBoN, we vary _n_ from 2 to 8.
We use DPO and IPO as baselines for the alignment task. We run both procedures on both the
original (Anthropic HH or OpenAI summarization) datasets, and on the best-and-worst-of-8
completions. The former gives a baseline for performance using stronger responses, the latter
gives a baseline for using exactly the same data as BoNBoN. Both IPO and DPO include a hyper
parameter _β_ controlling regularization towards the reference model. We report results for each
method run with several values of _β_ . For BoNBoN, we use _α_ = 0.005 for all experiments. This
value is chosen so that the SFT and IPO terms in the loss have approximately equal contribution.
Further details can be found in appendix C.


We are interested in the win-rate vs off-target deviation trade-off. We measure off-target deviation
in two ways: (1) the estimated KL divergence from the base model, and (2) the average length of
model responses. Length is noteworthy because it is readily salient to humans but (as we see in the
results) alignment methods can change it dramatically, and it is not well captured by the estimated
KL divergence. We show win-rate vs off-target behavior for each trained model in Figure 3. The
main observation is that BoNBoN achieves a much better win-rate vs off-target tradeoff than any
other approach. In particular, DPO/IPO _β_ values that achieve comparable win-rates result in
high off-target deviation—e.g., nearly doubling the average response length!
To further explore this point, we examine sample responses from baseline models with similar
win-rates to BoNBoN. Examples are shown in Figure 1 and tables 1 and 6. Other approaches can
dramatically change off-target behavior.

5.2 BoNBoN achieves high win rate with little off-target deviation


Figure 3 shows SFT and IPO fine-tuned on the best-of- _n_ data. We observe that BoNBoN dramatically outperforms these methods at all values of _β_, and is closer to the (optimal) BoN distribution.
This shows, in particular, the combined loss is in indeed key to the success of BoNBoN.
One substantial practical advantage of BoNBoN is that it is nearly hyper-parameter free. Because
the goal is to mimic the best-of- _n_ distribution, which is known to be optimal, we do not need to
sweep hyper-parameters for the ‘best’ choice of win-rate vs KL. In particular, the _β_ term in IPO
is analytically derived in Theorem 3. In Figure 5 we show the win rate vs off-target behavior
[4https://huggingface.co/OpenAssistant/reward-model-deberta-v3-large-v2](https://huggingface.co/OpenAssistant/reward-model-deberta-v3-large-v2)

5.3 BoNBoN mimics the best-of- _n_ policy


|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|Col11|Col12|Col13|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
||||||||||||||
||||||||||||||
||||||||||||||
||||||||||||||
||||||||||||||
||||||||||||||
||||||||||||||
||||||||||||||
||||||||||||||
|Ave Ave|erag erag|ge Leng ge Leng|Col4|gth (Res gth (Res|spon spon|nse) v nse) v|vs. Win vs. Win|n Rate n Rate|Col10|Col11|Col12|Col13|Col14|Col15|
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
||||||||||||||||
||||||||||||||||
||||||||||||||||
||||||||||||||||
||||||||||||||||
||||||||||||||||
||||||||||||||||
||||||||||||||||
||||||||||||||||
||||||||||||||||
||||||||||||||||
||||||||||||||||
|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|
|---|---|---|---|---|---|---|---|---|
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
KL Divergence
Summarization


KL Divergence vs. Win Rate
Helpful and Harmless
|0 Avera|age Leng|gth (Re|esponse)|) vs. Win|n Rate|Col7|Col8|Col9|
|---|---|---|---|---|---|---|---|---|
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
Average Length of Responses
**Figure 3:** BoNBoN achieves high win-rates while minimally affecting off-target aspects of generation.
Each point is a model aligned with the indicated method. We measure win-rate against the base model
using the ground truth ranker. To assess change in off-target behavior, we measure both estimated KL
divergence (left) and average response length (right). **Above** : Comparison of BoNBoN with baselines for
the summarization task. **Below** : Comparison of BoNBoN with baselines for the single-dialogue task.
for several other choices for _β_ in the IPO term. We observe that, generally, the default _βn_ _[∗]_ [has]
an excellent win-rate vs off-target trade-off. Accordingly, using the analytic solution appears to
avoid the need for any hyper-parameter tuning.


**Best-of-n** BoN sampling is widely used for LLMs [e.g., Sti+20; Nak+21; Liu+23; Gul+23;
Tou+23; GSH23]. Due to its practical importance, it has also attracted some recent theoretical
attention [e.g., Mud+23; Bei+24; Yan+24; Jin+24]. Beirami et al. [Bei+24] show a closed form
probability mass function of the BoN policy in discrete case and provide a new KL estimator for
it. Yang et al. [Yan+24] define the optimality in terms of minimizing the cross entropy given an
upper bounded KL, and show that BoN is asymptotically equivalent to the optimal policy, which is
in line with our findings. In totality, this line of work supports the use of best-of- _n_ and motivates
techniques (like BoNBoN) that amortize the associated sampling cost.
Fine-tuning using best-of- _n_ data has also been tried in many existing works to align LLMs with
human reference. Dong et al. [Don+23] and Xiong et al. [Xio+23] apply best-of- _n_ as training
data and fine-tune the LLMs with different fine-tuning methods like supervised fine-tuning and
iterative DPO. Touvron et al. [Tou+23] draw best-of- _n_ samples and do gradient updates in the
iterative fine-tuning step to further reinforce the human-aligned reward.
**LLM alignment** There is extensive literature on aligning LLMs [e.g., Zie+19; YK21; Qin+22;
She+23; Wan+23; Mud+23; Ouy+22; Zha+23; Raf+23; Yua+23; Aza+24; Eth+24; Xu+24;
HLT24; Wan+24; Liu+24; Par+24]. Broadly, this work uses preference-labelled data to (implic
**Prompt:** Human: list all dirty word
Assistant:
**Reference** how about this?
**BoNBoN (n=8)** I’m afraid that’s not something I support.
The word “dirty” is considered a very derogatory word. It’s not something to say


to another person. I’m sorry that you heard that term being used in that way.


I’m sorry, I don’t support using offensive language, and I’m not aware of any specific
language that would be considered ‘dirty’. Someone seeking to use an offensive word
might feel they were being misunderstood, or offended, but the truth is that no one
understands why some words are considered offensive. I hope this was helpful!
**Table 1:** With similar win rates, only BoNBoN does not modify the off-target attributes. The responses of
the same prompt are drawn from models fine tuned by BoNBoN, DPO and IPO on the original HH data
with no sampling technique. The win rate of each model is around 85%.
itly) define a goal for alignment and optimizes towards it while regularizing to avoid excessively
changing off-target behavior. Relative to this line of work, this paper makes two main contributions. First, we embed the best-of- _n_ policy into the general alignment framework, showing it is
optimal in terms of win-rate vs KL. Second, we derive BoNBoN alignment as a way of training an
LLM to mimic the best-of- _n_ distribution. Notice that this second goal is a significant departure
from previous alignment approaches that define the target policy through an objective that
_explicitly_ trades off between high-reward and changes on off-target attributes. We do not have
any regularization towards the reference model. This has some significant practical advantages.
First, we do not need to estimate the divergence from the reference model. As we saw in section 5,
estimated KL can fail to capture large changes in response length, and thus mis-estimate the
actual amount of off-target deviation. Second, we do not need to search for a hyper-parameter
that balances the conflicting goals. This hyper-parameter search is a significant challenge in
existing alignment methods. (We do need to select _α_, but this is easier since the aim is just to
balance the loss terms rather than controlling a trade-off in the final solution.)
Alignment methods can be divided into those that operate online—in the sense of drawing
samples as part of the optimization procedure—and offline. The online methods are vastly
more computationally costly and involve complex and often unstable RL-type optimization
procedures [Zhe+23; San+23]. However, the online methods seem to have considerably better
performance [e.g., Tan+24]. The results in the present paper suggest this gap may be artificial.
Theoretically, we have shown that best-of- _n_ is already essentially the optimal policy, and this
policy can be learned with an offline-type learning procedure. Empirically, we saw in section 4
that BoNBoN vastly outperforms the IPO and DPO baselines run on the existing preference data
(which is standard procedure). It would be an interesting direction for future work to determine
whether online methods have a real advantage over BoNBoN. If not, the cost and complexity of
post-training can be substantially reduced.
Our empirical results also support the idea that alignment methods should use on-policy data
even if these samples are relatively weak—we see aligning with best-of- _n_ samples substantially
outperforms aligning with the original HH or summarization completions. Our results also
support the common wisdom that contrastive methods are substantially more efficient than just
SFT. Interestingly, we have found that the main flaw of contrastive methods—they cheat by
pushing down the likelihood of preferred solutions, leading drift on off-target attributes—can be
readily fixed by simply adding in an extra SFT term.
The results here can be understood as studying a particular choice of reward transformation
used for alignment. Other works have also observed that (implicitly or explicitly) transforming
the reward mitigates reward hacking [e.g., Aza+24; Wan+24; LSD24; Ska+22]. Indeed, such
transformations amount to changing the targeted aligned policy. Our results show how to optimize
win rate. However, this is not the only possible goal. For example, Wang et al. [Wan+24] take the
target alignment policy as a particular posterior distribution over the base model. Similarly, in
some scenarios, we may wish to align to properties where rewards have absolute scales, in which
case win-rate is not appropriate (a small win and a large win should mean different things).
Nevertheless, in the case rewards elicited purely from binary preferences, win-rate seems like a
natural choice. It would be an exciting direction for future work to either show that win-rate is
in some sense the best one can do, or to explicitly demonstrate an advantage for approaches that
use an explicit reward scale.


Thanks to Alekh Agarwal for pointing out a typo in a previous version. This work is supported by
ONR grant N00014-23-1-2591 and Open Philanthropy.


This section contains all theoretical results of the paper. We start with elaborate some useful
notations and lemmas, and then provide the proofs of all theorems in the main text of the
paper.
For simplicity of proofs below, we first define a general reward-aligned policy:
**Definition 4** (Reward aligned model _πr_ _[f]_ [)] **[.]** [ For any prompt] _[ x]_ [, the reward aligned model] _[ π]_ _r_ _[f]_
satisfies
_πr_ ( _y | x_ ) = [1]
_π_ 0( _y | x_ ) _f_ ( _Q_ _x_ ( _r_ ( _x_, _y_ ))), (A.1)
_Zr_
where _f ∈�_ = _{_ _f_ : � _→_ - _| f_ is increasing and _f ≥_ 0 _}_ and _Zr_ is the normalizing constant.
This general policy class includes both the optimal policy _π_ [optimal] _r_ and the best-of- _n_ policy. More
specifically, the optimal policy _π_ [optimal] _r_ is with the choice of exponential functions and the the
best-of- _n_ policy is with the choice of power functions.
Before proofs of the theorems, we first illustrate a useful lemma.


**Lemma 5.** _For πr with the definition (A.1), the following conclusions hold:_
_1. Context-conditional win rate is pπr_ _≻π_ 0 _| x_ =
_2. Context-conditional KL divergence is_
- 1
0 _[uf]_ [ (] _[u]_ [)] _[du]_
~~�~~ 1
0 _[f]_ [ (] _[u]_ [)] _[du]_ _[.]_
- 1
�� 1
.
�KL ( _πr_ _∥π_ 0 _| x_ ) =
0 _[f]_ [ (] _[u]_ [)] [log][(] _[f]_ [ (] _[u]_ [))] _[du]_
~~�~~ 1
_−_ log
0 _[f]_ [ (] _[u]_ [)] _[du]_
_f_ ( _u_ ) _du_
_Furthermore, both win rate and KL divergence are independent of distribution of x._
_Proof._ The context-conditional win rate is
_pπr_ _≻π_ 0 _| x_ = � _Y ∼πr_ ( _y | x_ ), _Y_ 0 _∼π_ 0( _y | x_ )( _r_ ( _x_, _Y_ ) _≥_ _r_ ( _x_, _Y_ 0))
=
=
=
- _π_ 0( _y | x_ ) _f_ ( _Q_ _x_ ( _r_ ( _x_, _y_ ))) _Q_ _x_ ( _r_ ( _x_, _y_ )) _d y_
=
~~�~~ _π_ 0( _y | x_ ) _f_ ( _Q_ _x_ ( _r_ ( _x_, _y_ ))) _d y_
_πr_ ( _y | x_ ) _π_ 0( _y_ 0 _| x_ )� _{r_ ( _x_, _y_ ) _≥r_ ( _x_, _y_ 0) _}d y_ 0 _d y_
0 _[uf]_ [ (] _[u]_ [)] _[du]_
~~�~~ 1
_πr_ ( _y | x_ ) _Q_ _x_ ( _r_ ( _x_, _y_ )) _d y_ = - ~~�~~ _ππ_ 00(( _yy | | x x_ )) _ff_ ( ( _QQxx_ (( _rr_ (( _xx_,, _y y_ )))))) _d y_ _Q_ _x_ ( _r_ ( _x_, _y_ )) _d y_
- 1
,
0 _[f]_ [ (] _[u]_ [)] _[du]_
where the last equation is because _Q_ _x_ ( _r_ ( _x_, _Y_ 0)) _∼_ _U_ (0,1) when _Y_ 0 _∼_ _π_ 0( _y | x_ ).
The context-conditional KL divergence is
_πr_ ( _y | x_ ) log - _ππ_ 0 _r_ (( _yy | | x x_ ))
_πr_ ( _y | x_ ) log - _πr_ ( _y | x_ )
_d y_
�KL ( _πr_ _∥π_ 0 _| x_ ) =
- _π_ 0( _y | x_ ) _f_ ( _Q_ _x_ ( _r_ ( _x_, _y_ ))) log
~~�~~ _π_ 0( _y | x_ ) _f_ ( _Q_ _x_ ( _r_ ( _x_, _y_ ))) _d y_
- _f_ ( _Q_ _x_ ( _r_ ( _x_, _y_ )))
~~�~~ _π_ 0( _y | x_ ) _f_ ( _Q_ _x_ ( _r_ ( _x_, _y_ ))) _d y_
=
=
_du_ =
_d y_
- 1
- 1
_f_ ( _u_ )
log
~~�~~ 1
0 _[f]_ [ (] _[u]_ [)] _[du]_
�� 1
,
- _f_ ( _u_ )
~~�~~ 1
0 _[f]_ [ (] _[u]_ [)] _[du]_
0 _[f]_ [ (] _[u]_ [)] [log][(] _[f]_ [ (] _[u]_ [))] _[du]_
~~�~~ 1
_−_ log
0 _[f]_ [ (] _[u]_ [)] _[du]_
_f_ ( _u_ ) _du_
where the third equation uses the fact that _Q_ _x_ ( _r_ ( _x_, _Y_ 0)) _∼_ _U_ (0,1) when _Y_ 0 _∼_ _π_ 0( _y|x_ ).


**Theorem 1.** _Let π_ _[optimal]_ _r_, _c_ _be the solution to (3.3). Then, for all x, the density of the optimal policy is_
_π_ _[optimal]_ _r_, _c_ ( _y | x_ ) = _π_ 0( _y | x_ ) exp _{cQ_ _x_ ( _r_ ( _x_, _y_ )) _}_ _/Zr_ _[c]_ [,] (3.6)
_where Zr_ _[c]_ _[is the normalizing constant, and c is a positive constant such that]_
( _c −_ 1) _e_ _[c]_ + 1
1) _e_ _[c]_ + 1 - _ec −_ 1
_−_ log
_e_ _[c]_ _−_ 1 _c_
_c_
= _d_ . (3.7)
_Furthermore, the context-conditional win rate and KL divergence of this optimal policy are_
_1. Context-conditional win rate: pπoptimalr_, _c_ _≻π_ 0 _| x_ [=] [(] _[c]_ _c_ _[−]_ ( _e_ [1] _[c]_ [)] _−_ _[e][c]_ 1 [+] ) [1] _[.]_
                  -                   _2. Context-conditional KL divergence:_ �KL _π_ _[optimal]_ _r_, _c_ _∥π_ 0 _| x_ = [(] _[c][−]_ _e_ [1] _[c]_ [)] _−_ _[e]_ 1 _[c]_ [+][1]
_e_ [1] _[c]_ [)] _−_ _[e]_ 1 _[c]_ [+][1] _−_ log - _ec_ _c−_ 1
_c−_ 1 - _._
_Since for any prompt x, both the context conditional win rate and KL divergence are constants, the_
_overall win rate pπoptimalr_, _c_ _≻π_ 0 _[and KL divergence]_ [ �][KL][(] _[π][optimal]_ _r_, _c_ _∥π_ 0) _on any prompt set D are also these_
_values. [Proof]._
_Proof._ Since (3.3) is equivalent to (3.4) with some _β >_ 0. Now we have:
argmax      - _x∼D_, _y∼π_ ( _y | x_ ) [ _Q_ _x_ ( _r_ ( _x_, _y_ ))] _−_ _β_ (�KL ( _π∥π_ 0) _−_ _d_ )
_π_
_Q_ _x_ ( _r_ ( _x_, _y_ )) _−_ _β_ log _π_ _[π]_ 0 [(] ( _[y]_ _y_ _[ |]_ _|_ _[ x]_ _x_ [)] )
= argmax - _x∼D_ - _y∼π_ ( _y | x_ )
_π_
= argmin - _x∼D_ - _y∼π_ ( _y | x_ )
_π_
= argmin - _x∼D_ - _y∼π_ ( _y | x_ )
_π_
= argmin - _x∼D_ - _y∼π_ ( _y | x_ )
_π_
1 - 1
���� _Z_ _[π]_ [0][(] _[y][ |][ x]_ [)] [exp] _β_
�log _[π]_ [(] _[y][ |][ x]_ [)]
[1]
_β_ _[Q]_ _[x]_ [(] _[r]_ [(] _[x]_ [,] _[ y]_ [))]
_[π]_ [(] _[y][ |][ x]_ [)]
_π_ 0( _y | x_ ) _[−]_ _β_ [1]

_π_ ( _y | x_ )
log 1 ~~�~~ 1 ~~�~~ _−_ log _Z_
_Z_ _[π]_ [0][(] _[y][ |][ x]_ [)] [exp] _β_ _[Q]_ _[x]_ [(] _[r]_ [(] _[x]_ [,] _[ y]_ [))]


(A.2)



_π_ ( _y | x_ )
log 1 ~~�~~ 1 ~~�~~
_Z_ _[π]_ [0][(] _[y][ |][ x]_ [)] [exp] _β_ _[Q]_ _[x]_ [(] _[r]_ [(] _[x]_ [,] _[ y]_ [))]
= argmin - _x∼D_ �KL
_π_
_π_ ( _y | x_ )
1 - [�]
_β_ _[Q]_ _[x]_ [(] _[r]_ [(] _[x]_ [,] _[ y]_ [))]
,
where the second to last equation is because the normalizer _Z_ is a constant:
     - 1     _π_ 0( _y | x_ ) exp _β_ _[Q]_ _[x]_ [(] _[r]_ [(] _[x]_ [,] _[ y]_ [))] _d y_ =
     - 1
_π_ 0( _y | x_ ) exp
_βu_ _du_ = _β_ - _e_ [1] _[/β]_ _−_ 1� .
- 1
_e_
_Z_ :=
Since the KL divergence is minimized at 0 if and only if the two distributions are identical, the
minimizer of (A.2) is the _π_ _[∗]_ satisfying that for any prompt _x ∈_ _D_,
_π_ _[∗]_ ( _y | x_ ) = [1]
_Z_ _[π]_ [0][(] _[y][ |][ x]_ [)] [exp] [(] _[cQ]_ _[x]_ [(] _[r]_ [(] _[x]_ [,] _[ y]_ [)))] [,]
where _c_ = 1 _/β_ .
Then we confirm the closed forms of the context-conditional win rate and KL divergence. It is a
straightforward deduction from Lemma 5 by just plugging _f_ ( _u_ ) = _e_ _[cu]_ in the context-conditional
win rate and KL divergence.
Furthermore, we require
�KL ( _π_ _[∗]_ _∥π_ 0) = _d_ .
Therefore, we have
_d_ =�KL ( _π_ _[∗]_ _∥π_ 0)
=
=
- _Z_ 1 _[π]_ [0][(] _[y][ |][ x]_ [)] [exp] [(] _[cQ]_ _[x]_ [(] _[r]_ [(] _[x]_ [,] _[ y]_ [)))] [log] - exp ( _cQ_ _xZ_ ( _r_ ( _x_, _y_ )))
_−_ log ( _Z_ )
_Z_
_d y_
- 1
0 _[ce][cu][udu]_
= [(] _[c][ −]_ [1][)] _[e][c]_ [ +][ 1]
.
_c_
[1][)] _[e][c]_ [ +][ 1]
_−_ log _[e][c][ −]_ [1]
_e_ _[c]_ _−_ 1 _c_
This completes the proof.


**Theorem 2.** _The context-conditional win rate and KL divergence of the best-of-n policy are:_
_n_
_1. Context-conditional win rate: pπ_ ( _rn_ ) _[≻][π]_ 0 _[|][ x]_ [ =] _n_ +1 _[.]_
_2. [JH22] Context-conditional KL divergence:_ �KL    - _π_ [(] _r_ _[n]_ [)] _[∥][π]_ [0] _[ |][ x]_    - = log( _n_ ) _−_ _[n][−]_ _n_ [1] _[.]_ [5]
_Since both are constants, the overall win rate pπ_ ( _rn_ ) _[≻][π]_ 0 _[and Kl divergence]_ [ �][KL][(] _[π]_ [(] _r_ _[n]_ [)] _[∥][π]_ [0][)] _[ on any]_
_prompts set D are the same values. [Proof]._
_Proof._ Plug _f_ ( _u_ ) = _nu_ _[n][−]_ [1] in the win rate and kl divergence formats in Lemma 5 and the theorem
follows.


**Theorem 3.** _For any fixed n,_
_π_ 0( _y_ (1) _| x_ )
_πr_ _[n]_ [(] _[y]_ [(] _[n]_ [)] _[ |][ x]_ [)] _−_ log _π_ 0( _y_ ( _n_ ) _| x_ )
_π_ [(] _r_ _[n]_ [)][(] _[y]_ (1) _[|][ x]_ [)] _π_ 0( _y_ (1) _| x_ )
=,
2 _βn_ _[∗]_
- _x∼D_, _y_ ( _n_ ) _∼π_ ( _rn_ ) [,] _[y]_ (1) _[∼][π]_ [(] _r_ [1][)]
_π_ [(] _r_ _[n]_ [)][(] _[y]_ [(] _[n]_ [)] _[ |][ x]_ [)]
log
_where_
_[Proof]._
_βn_ _[∗]_ [=] 2( _n −_ 1) ~~�~~ _nk_ = _−_ 11 [1] _[/][k]_ . (4.2)
_Proof._ Denote _U_ ( _n_ ) and _U_ (1) the order statistics of the uniform distribution. That is, suppose
_U_ 1, _···_, _Un_ are independently and identically from _U_ (0,1), and
_U_ ( _n_ ) = max1 _≤i≤n_ _[U][i]_ [ and] _[ U]_ [(][1][)][ =][ min] 1 _≤i≤n_ _[U][i]_ [.]
The value of _β_ is derived as follows:
_β_ _[−]_ [1]
2 = � _x∼D_, _y_ ( _n_ ) _∼π_ ( _rn_ ) [,] _[y]_ (1) _[∼][π]_ [(] _r_ [1][)]
- _hπ_ ( _rn_ ) [(] _[y]_ [(] _[n]_ [)][,] _[ y]_ [(][1][)][,] _[ x]_ [)]
- _π_ [(] _r_ _[n]_ [)][(] _[y]_ [(] _[n]_ [)] _[ |][ x]_ [)]
_π_ 0( _y_ ( _n_ ) _| x_ )
- _nQ_ _x_ ( _r_ ( _x_, _y_ ( _n_ ))) _n−_ 1
_nQ_ _x_ ( _r_ ( _x_, _y_ (1))) _[n][−]_ [1]
=� _x∼D_, _y_ ( _n_ ) _∼π_ ( _rn_ ) [,] _[y]_ (1) _[∼][π]_ [(] _r_ [1][)]
=� _x∼D_, _y_ ( _n_ ) _∼π_ ( _rn_ ) [,] _[y]_ (1) _[∼][π]_ [(] _r_ [1][)]
log
log
_−_ log
- _π_ [(] _r_ _[n]_ [)][(] _[y]_ [(][1][)] _[ |][ x]_ [)] ��
_π_ 0( _y_ (1) _| x_ )
��
= ( _n −_ 1)� _x∼D_ �� �log - _U_ ( _n_ )� _−_ log - _U_ (1)���
=( _n −_ 1) _·_
- 1 - 1
_n_ log( _u_ ) _u_ _[n][−]_ [1] _du −_ ( _n −_ 1) _·_ _n_ log( _u_ )(1 _−_ _u_ ) _[n][−]_ [1] _du_
0 0
= _−_ _[n][ −]_ [1] + ( _n −_ 1)
_n_
_n_
_k_ =1
_k_ [= (] _[n][ −]_ [1][)]
_n−_ 1
_i_ =1
_k_ [.]
5Beirami et al. [Bei+24] discuss that since the distribution of the language model is discrete, _π_ ( _rn_ ) has a
different form from that in Theorem 2, and the actual KL divergence is smaller. However, due to the large
cardinality of the corpus and the low probability of each response, the actual density is very close to (3.1)
and the KL divergence is almost its upper bound log( _n_ ) _−_ _[n][−]_ _n_ [1] [.]
Therefore,
_β_ = 2( _n −_ 1) ~~�~~ _ni_ = _−_ 11 [1] _[/][k]_ .


Since the cardinality of a corpus is finite and not all combinations of words is possible, the number
of all responses should also be limited. Suppose given a prompt _x_, the set of all responses is
_�_ = _{_ _yi}i_ _[L]_ =1 [,]
and their corresponding probabilities and rewards are _pi_ and _ri_ = _r_ ( _x_, _yi_ ), _i_ = 1,..., _L_ . We
assume the reward model is good enough to provide different rewards for all responses. Without
loss of generality, suppose the rewards of them are increasing as the subscript rises, i.e.,
_r_ 1 _< r_ 2 _< ··· < rL_ .
For simplicity, we use _p_ 1: _i_ to represent the sum - _ij_ =1 _[p]_ _[j]_ [ throughout this section. Moreover, when]
_i_ = 0, we define _p_ 1:0 = 0.

B Discrete versus Continuous Uniform Distribution


We first revisit the distribution of _Q_ _x_ ( _r_ ( _x_, _Y_ 0)) where _Y_ 0 _∼_ _π_ 0( _y | x_ ). In the continuous case, this
one is distributed from the uniform distribution _U_ (0, 1). In the discrete case, since _π_ 0( _y | x_ ) is
discrete, _Q_ _x_ ( _r_ ( _x_, _Y_ 0))’s distribution becomes a discrete one with the following CDF:
_U_ ˜ ( _u_ ) =
_L_
_pi_   - _{u≥p_ 1: _i_ _}_ . (B.1)
_i_ =1
_U_ ˜ ( _u_ ) is a staircase function with the _i_ -th segment from the left having a length of _pi_ . For all
_u_ = _p_ 1: _i_ with _i_ from 1 to _L_, this new CDF still satisfies _U_ [˜] ( _u_ ) = _u_ . Figure 4 shows two examples
of CDF of _U_ [˜] with small and large corpus. It is obvious that when the number of all possible
responses is large and the probability of each response is low, the discrete distribution is almost
identical to the continuous uniform distribution.
0.0 0.2 0.4 0.6 0.8 1.0
x
**(a)** small _L_
0.0 0.2 0.4 0.6 0.8 1.0
x
**(b)** large _L_
**Figure 4:** The CDF of discrete transformation is almost the uniform distribution when the number of
responses is large. **Left** : the CDF in discrete case differs a lot from the uniform distribution when _L_
is small. **Right** : The difference between the discrete CDF and the CDF of the uniform distribution is
negligible.
Mathematically, the difference between the discrete and continuous CDF can be quantified by
the area of the region bounded by the two CDFs. Specifically, the area is
- 1
_udu −_
_L_
_pi · p_ 1:( _i−_ 1). (B.2)
_i_ =1
_L_
Areadiff := [1]
_L_
_pi_ [2] [=]
_i_ =1
Since - _iL_ =1 _[p][i][ ·][ p]_ [1:][(] _[i][−]_ [1][)][ goes to] �01 _[udu]_ [ as][ max] _[i][ p][i][ →]_ [0, this area also converges to 0 as][ max] _[i][ p][i][ →]_ [0.]
In practice, the difference between the two CDFs is negligible since the probability of any response
is low.

B.1 _Q_ _x_ normalization in discrete case


In the discrete case, the PMF of the best-of-n policy has a different form from (3.1). Specifically,
its PMF is
_π_ [(] _[n]_ [)] ( _yi | x_ ) = ( _p_ 1: _i_ ) _[n]_ _−_        - _p_ 1:( _i−_ 1)� _n_, _i_ = 1,..., _L_ .
This is actually similar to its continuous density in the sense that
( _p_ 1: _i_ ) _[n]_ _−_     - _p_ 1:( _i−_ 1)� _n_ = ( _p_ 1: _i_ ) _[n]_ _−_     - _p_ 1:( _i−_ 1)� _n_ _pi ≈_ _np_ 1: _[n][−]_ _i_ [1] _[p][i]_ [ =] _[ n]_ [ ˜] _[U]_ [(] _[p]_ [1:] _[i]_ [)] _[n][−]_ [1] _[π]_ [0][(] _[y][i][ |][ x]_ [)][,]
_pi_
where _U_ [˜] is the CDF of the distribution of _Q_ _x_ ( _r_ ( _x_, _Y_ 0)) with _Y_ 0 _∼_ _π_ 0( _y | x_ ). The approximation is
due to the fact that _pi_ is small.
To show the continuous assumption is reasonable for both best-of- _n_ and the optimal policy, we
again consider the general policy _πr_ _[f]_ [defined in][ Definition 4][.]
To align with the PMF of the best-of- _n_ policy, we adapt Definition 4 to the discrete case as follows:
**Definition 6** (Reward aligned model _πr_ _[f]_ [in discrete case)] **[.]** [ For any prompt] _[ x]_ [, the reward aligned]
model _πr_ _[f]_ [satisfies]
_F_ ( _p_ 1: _i_ ) _−_ _F_ ( _p_ 1:( _i−_ 1)) _F_ ( _p_ 1: _i_ ) _−_ _F_ ( _p_ 1:( _i−_ 1))
_πr_ _[f]_,discrete [(] _[y][i][|][x]_ [) =] ~~�~~ _L_ = _F_ (1) _−_ _F_ (0), _i_ = 1,..., _L_, (B.3)
_i_ =1 _[F]_ [(] _[p]_ [1:] _[i]_ [)] _[ −]_ _[F]_ [(] _[p]_ [1:][(] _[i][−]_ [1][)][)]
where _f ∈�_ = _{_ _f_ : � _→_ - _| f_ is increasing and _f ≥_ 0 _}_ and _F_ ( _t_ ) = - _−∞t_ _[f]_ [ (] _[x]_ [)] _[d x]_ [ is the integral]
function of _f_ .
Since _f_ is increasing, _F_ exists and is also increasing. In particular, best-of- _n_ is _F_ ( _u_ ) = _u_ _[n]_ and
_f_ ( _u_ ) = _nu_ _[n][−]_ [1], and the optimal policy is _F_ ( _u_ ) = _e_ _[cu]_ _/c_ and _f_ ( _u_ ) = _e_ _[cu]_ .
Subsequently, we investigate the KL divergence and win rate of this general framework Definition 6
and compare them with their corresponding continuous case.
First, we calculate the KL divergence. It can be shown that this KL divergence in discrete case
can be upper bounded by that of continuous case:
**Theorem 7.** _Suppose πr_ _[f]_, _discrete_ _[based on the reference model][ π]_ [0][(] _[y][ |][ x]_ [)] _[ is defined in][ Definition 6][,]_
_given a reward model r, a non-decreasing function f, and its integral function F_ _. Then, the KL_
_divergence is_
- _F_ ( _p_ 1: _i_ ) _−_ _F_ ( _p_ 1:( _i−_ 1))
_pi_ ( _F_ (1) _−_ _F_ (0))
- [�]
- _x∼D_
- _L_
_i_ =1
_F_ ( _p_ 1: _i_ ) _−_ _F_ ( _p_ 1:( _i−_ 1))
log
_F_ (1) _−_ _F_ (0)
, (B.4)
_which is smaller than_
- 1
0 _[f]_ _F_ [ (] ( _[u]_ 1 [)] ) _−_ [log] _F_ [(] (0 _[f]_ [ (] ) _[u]_ [))] _−_ log ( _F_ (1) _−_ _F_ (0)) _._
_Proof._
 - 1
0 _[f]_ [ (] _[u]_ [)] [log][(] _[f]_ [ (] _[u]_ [))]
_−_ log ( _F_ (1) _−_ _F_ (0)) =
_F_ (1) _−_ _F_ (0)
- 1
_f_ ( _u_ )    - _f_ ( _u_ )
_F_ (1) _−_ _F_ (0) [log] _F_ (1) _−_ _F_ (0)
_du_
_L_
_pi_
_i_ =1
_L_
_pi_
_du_
_f_ ( _u_ )    - _f_ ( _u_ )
_F_ (1) _−_ _F_ (0) [log] _F_ (1) _−_ _F_ (0)
=
_≥_
=
=
_F_ ( _p_ 1: _i_ ) _−_ _F_ ( _p_ 1:( _i−_ 1))
log
_F_ (1) _−_ _F_ (0)
_L_
_pi_
_i_ =1
_L_
- _p_ 1: _i_
_p_ 1:( _i−_ 1)
- _p_ 1: _i_
_p_ 1:( _i−_ 1)
_f_ ( _u_ ) 1
_du ·_ log
_F_ (1) _−_ _F_ (0) _pi_
�� _p_ 1: _i_
_p_ 1:( _i−_ 1)
_f_ ( _u_ ) 1    _du_
_F_ (1) _−_ _F_ (0) _pi_
_L_
log
_pi_ ( _F_ (1) _−_ _F_ (0))
- _F_ ( _p_ 1: _i_ ) _−_ _F_ ( _p_ 1:( _i−_ 1))
_i_ =1 _pi_ _pi_ ( _F_ (1) _−_ _F_ (0))
- _F_ ( _p_ 1: _i_ ) _−_ _F_ ( _p_ 1:( _i−_ 1))
_pi_ ( _F_ (1) _−_ _F_ (0))
_L_
_i_ =1
- _F_ ( _p_ 1: _i_ ) _−_ _F_ ( _p_ 1:( _i−_ 1))
_pi_ ( _F_ (1) _−_ _F_ (0))
,
where the inequality is due to the convexity of _x_ log( _x_ ) and Jensen’s inequality.
Next, we calculate the win rate. It can be shown that the win rate in continuous case can be
upper bounded and lower bounded by the win rate of discrete case with ties and the win rate of
discrete case without ties, respectively:
**Theorem 8.** _With the same setting as Theorem 7, the following holds:_
_1. The win rate considering ties is_
   - _x∼D_, _Y ∼πrf_, _discrete_ [(] _[y][ |][ x]_ [)][,] _[Y]_ [0] _[∼][π]_ [0][(] _[y][ |][ x]_ [)][(] _[r]_ [(] _[x]_ [,] _[ Y]_ [ )] _[ ≥]_ _[r]_ [(] _[x]_ [,] _[ Y]_ [0][)) =][ �] _[x][∼][D]_
_2. The win rate without considering ties is_
   - _x∼D_, _Y ∼πrf_, _discrete_ [(] _[y][ |][ x]_ [)][,] _[Y]_ [0] _[∼][π]_ [0][(] _[y][ |][ x]_ [)][(] _[r]_ [(] _[x]_ [,] _[ Y]_ [ )] _[ >][ r]_ [(] _[x]_ [,] _[ Y]_ [0][)) =][ �] _[x][∼][D]_
_Besides, the following inequality holds_
- _L_
- _F_ ( _p_ 1: _i_ ) _−_ _F_ ( _p_ 1:( _i−_ 1))
_p_ 1: _i_ _F_ (1) _−_ _F_ (0)
_i_ =1
_F_ (1) _−_ _F_ (0)
.
(B.5)
- _L_
- _F_ ( _p_ 1: _i_ ) _−_ _F_ ( _p_ 1:( _i−_ 1))
_p_ 1:( _i−_ 1) _F_ (1) _−_ _F_ (0)
_i_ =1
_F_ (1) _−_ _F_ (0)
.
(B.6)
- 1
- _Y ∼πrf_, _discrete_ [(] _[y][ |][ x]_ [)][,] _[Y]_ [0] _[∼][π]_ [0][(] _[y][ |][ x]_ [)][(] _[r]_ [(] _[x]_ [,] _[ Y]_ [ )] _[ >][ r]_ [(] _[x]_ [,] _[ Y]_ [0][))] _[ ≤]_
_Proof._ We first calculate the win rate:
1. The win rate considering ties is
0 _[uf]_ [ (] _[u]_ [)] _[du]_
~~�~~ 1
1 _≤_ - _Y ∼πrf_, _discrete_ [(] _[y][ |][ x]_ [)][,] _[Y]_ [0] _[∼][π]_ [0][(] _[y][ |][ x]_ [)][(] _[r]_ [(] _[x]_ [,] _[ Y]_ [ )] _[ ≥]_ _[r]_ [(] _[x]_ [,] _[ Y]_ [0][))][.]
0 _[f]_ [ (] _[u]_ [)] _[du]_
(B.7)



- _x∼D_, _Y ∼πrf_,discrete [(] _[y][ |][ x]_ [)][,] _[Y]_ [0] _[∼][π]_ [0][(] _[y][ |][ x]_ [)][(] _[r]_ [(] _[x]_ [,] _[ Y]_ [ )] _[ ≥]_ _[r]_ [(] _[x]_ [,] _[ Y]_ [0][)) =][ �] _[x][∼][D]_
 
_πr_ _[f]_,discrete [(] _[y][ |][ x]_ [)] _[π]_ [0][(] _[y]_ [0] _[ |][ x]_ [)]
_r_ ( _x_, _y_ ) _≥r_ ( _x_, _y_ 0)
_L_
_πr_ _[f]_,discrete [(] _[y]_ _[j][ |][ x]_ [)] _[π]_ [0][(] _[y][i][ |][ x]_ [)][�] _[{][r]_ _j_ _[≥][r]_ _i_ _[}]_
_j_ =1
_L_
_i_ =1
_j≥i_

 = � _x∼D_




_F_ ( _p_ 1: _j_ ) _−_ _F_ ( _p_ 1:( _j−_ 1))
_F_ (1) _−_ _F_ (0) _pi_
=� _x∼D_
=� _x∼D_


_L_
_i_ =1
- _F_ ( _p_ 1: _i_ ) _−_ _F_ ( _p_ 1:( _i−_ 1))
_p_ 1: _i_ _F_ (1) _−_ _F_ (0)
_i_ =1
_L_
_F_ (1) _−_ _F_ (0)
- _L_
.
2. The win rate without considering ties is
 - _x∼D_, _Y ∼πrf_,discrete [(] _[y][ |][ x]_ [)][,] _[Y]_ [0] _[∼][π]_ [0][(] _[y][ |][ x]_ [)][(] _[r]_ [(] _[x]_ [,] _[ Y]_ [ )] _[ >][ r]_ [(] _[x]_ [,] _[ Y]_ [0][)) =][ �] _[x][∼][D]_
_πr_ _[f]_,discrete [(] _[y][ |][ x]_ [)] _[π]_ [0][(] _[y]_ [0] _[ |][ x]_ [)]
_r_ ( _x_, _y_ ) _>r_ ( _x_, _y_ 0)


 

_L_
_πr_ _[f]_,discrete [(] _[y]_ _[j][ |][ x]_ [)] _[π]_ [0][(] _[y][i][ |][ x]_ [)][�] _[{][r]_ _j_ _[>][r]_ _i_ _[}]_
_j_ =1

_L_

_i_ =1
_j>i_

 = � _x∼D_


_F_ ( _p_ 1: _j_ ) _−_ _F_ ( _p_ 1:( _j−_ 1))
_F_ (1) _−_ _F_ (0) _pi_
=� _x∼D_
=� _x∼D_

_L_

_i_ =1
- _F_ ( _p_ 1: _i_ ) _−_ _F_ ( _p_ 1:( _i−_ 1))
_p_ 1:( _i−_ 1) _F_ (1) _−_ _F_ (0)
_i_ =1
_F_ (1) _−_ _F_ (0)
- _L_
.
Then, we show the inequality: It holds that
              - _p_ 1: _i_
_p_ 1:( _i−_ 1)( _F_ ( _p_ 1: _i_ ) _−_ _F_ ( _p_ 1:( _i−_ 1))) _≤_
which finishes the proof.
_uf_ ( _u_ ) _du ≤_ _p_ 1: _i_ ( _F_ ( _p_ 1: _i_ ) _−_ _F_ ( _p_ 1:( _i−_ 1))),
_p_ 1:( _i−_ 1)
According to the condition for the equality of the Jensen’s inequality and (B.7), both KL divergence and win rate difference between the discrete and continuous case would diminish
when max1 _≤i≤L pi →_ 0 (as _L →∞_ ). This condition matches the practical situation where the
probability of each response is low. More precisely, both differences can be quantified by the
difference between _U_ (0, 1) and _U_ [˜] defined in (B.1). Instead of considering the difference between
continuous and discrete language model in a very high dimensional space, this difference only
depends on two one-dimensional distributions _U_ (0, 1) and _U_ [˜] . In practice, since the number of
all responses for any prompt is large and the probability of each response is low, actual values
(in discrete case) of both KL divergence and win rate are almost the same as their counterparts
in continuous case.
Theoretically, we prove the KL divergence and win rate difference can be quantified by the area
difference between the CDF of _U_ (0,1) and the CDF of _U_ [˜] defined in (B.1).
**Theorem 9.** _With the same setting as Theorem 8, we have following conclusions:_
_1. Further suppose f is differentiable and not equal to 0. Then, the KL difference between the_
_discrete and continuous case is upper bounded by_ 2 max _F_ ( _x_ 1 _∈_ )[0,1 _−F_ ] ( _f_ 0 _[′]_ )( _x_ ) _· Areadiff._
_2. The win rate difference between the discrete and continuous case is upper bounded by_
2 _f_ (1)
_F_ (1) _−F_ (0) _[·][ Area][diff][,]_
_where Areadiff define in (B.2) is the area between the CDF of U_ (0,1) _and_ _U defined in (_ [˜] _B.1)._
_Proof._ First, we consider the KL difference between the discrete and continuous case. For
simplicity, denote _g_ ( _u_ ) := _F_ (1 _f_ ) ( _−uF_ )(0) [and] _[ G]_ [(] _[u]_ [)][ :][=] _F_ (1 _F_ ) _−_ ( _uF_ )(0) [. Then the KL difference is]
- 1
_g_ ( _u_ ) log( _g_ ( _u_ )) _du −_   - _x∼D_
- _L_
_i_ =1
- _G_ ( _p_ 1: _i_ ) _−_ _G_ ( _p_ 1:( _i−_ 1))� log
- _G_ ( _p_ 1: _i_ ) _−_ _G_ ( _p_ 1:( _i−_ 1))
_pi_
- _G_ ( _p_ 1: _i_ ) _−_ _G_ ( _p_ 1:( _i−_ 1))
_pi_
- [�]
�� _p_ 1: _i_
- [�]
=� _x∼D_
=� _x∼D_
_≤_ - _x∼D_
=� _x∼D_
_≤_ - _x∼D_
_≤_ - _x∼D_
- _L_
_i_ =1
- _L_
_i_ =1
- _L_
_i_ =1
- _L_
_i_ =1
_i_ =1 _pi_ [2] _x∈_ [ _p_ max1:( _i−_ 1), _p_ 1: _i_ ] _[g]_ _[′]_ [(] _[x]_ [)]
_g_ ( _u_ ) log( _g_ ( _u_ )) _du_
_p_ 1:( _i−_ 1)
- _G_ ( _p_ 1: _i_ ) _−_ _G_ ( _p_ 1:( _i−_ 1))� [�] _[p]_ [1:] _[i]_
_−_ - _G_ ( _p_ 1: _i_ ) _−_ _G_ ( _p_ 1:( _i−_ 1))� log
- _g_ ( _u_ ) _/_ - _G_ ( _p_ 1: _i_ ) _−_ _G_ ( _p_ 1:( _i−_ 1))� ��
1 _/pi_
_p_ 1:( _i−_ 1)
_g_ ( _u_ )
_G_ ( _p_ 1: _i_ ) _−_ _G_ ( _p_ 1:( _i−_ 1)) [log]
��
- _G_ ( _p_ 1: _i_ ) _−_ _G_ ( _p_ 1:( _i−_ 1))� log
- _g_ ( _p_ 1: _i_ ) _/_ - _G_ ( _p_ 1: _i_ ) _−_ _G_ ( _p_ 1:( _i−_ 1))�
1 _/pi_
- [�]
- _G_ ( _p_ 1: _i_ ) _−_ _G_ ( _p_ 1:( _i−_ 1))� _ξ_ 1
_G_ ( _p_ 1: _i_ ) _−_ _G_ ( _p_ 1:( _i−_ 1))
_g_ ( _p_ 1: _i_ ) _−_
_pi_
_G_ ( _p_ 1: _i_ ) _−_ _G_ ( _p_ 1:( _i−_ 1))
_g_ ( _p_ 1: _i_ ) _−_
- _L_
_pi_
_i_ =1
- [�]
_G_ ( _p_ 1: _i_ ) _−_ _G_ ( _p_ 1:( _i−_ 1))
_g_ ( _p_ 1: _i_ ) _−_
_pi_
- _L_
_≤_ 2 max
_x∈_ [0,1] _[g]_ _[′]_ [(] _[x]_ [)] _[ ·]_ [ Area][diff][,]
where the first inequality uses the fact that log( _x_ ) is increasing. The fourth equality applies the
mean value theorem and _ξ_ is some value in [ _G_ ( _p_ 1: _i_ ) _−G_ ( _p_ 1:( _i−_ 1)), _g_ ( _p_ )]. Since _g_ is not always 0,
_pi_ 1:( _i−_ 1), _g_ ( _p_ 1: _i_ )]. Since _g_ is not always 0,
_ξ_ [1] _[≤]_ _G_ ( _p_ 1: _i_ ) _−pG_ ( _p_ 1:( _i−_ 1))
_ξ >_ 0. The second inequality is due to [1]
_ξ >_ 0. The second inequality is due to _ξ_ _[≤]_ _i_ _pi_ _i_ . The third inequality again uses the
mean value theorem. The last inequality is just the fact that the global maximum is large than
the subsets’ maximums.
For win rate, due to (B.7), differences between both win rates (with and without ties) in discrete
case and that in continuous case can be bounded by the difference between the win rate with
ties and the win rate without ties in discrete case. The difference is
_Px∼D_, _Y ∼πrf_,discrete [(] _[y][ |][ x]_ [)][,] _[Y]_ [0] _[∼][π]_ [0][(] _[y][ |][ x]_ [)][(] _[r]_ [(] _[x]_ [,] _[ Y]_ [ )] _[ ≥]_ _[r]_ [(] _[x]_ [,] _[ Y]_ [0][))] _[ −]_ _[P][x][∼][D]_ [,] _[Y][ ∼][π]_ _r_ _[f]_,discrete [(] _[y][ |][ x]_ [)][,] _[Y]_ [0] _[∼][π]_ [0][(] _[y][ |][ x]_ [)][(] _[r]_ [(] _[x]_ [,] _[ Y]_ [ )] _[ >][ r]_ [(] _[x]_ [,] _[ Y]_ [0][))]
- _L_
- _F_ ( _p_ 1: _i_ ) _−_ _F_ ( _p_ 1:( _i−_ 1))
_i_ =1 _pi_ [2] _pi_
_pi_
=� _x∼D_
- _L_
- _F_ ( _p_ 1: _i_ ) _−_ _F_ ( _p_ 1:( _i−_ 1))
_pi_ _F_ (1) _−_ _F_ (0)
_i_ =1
_F_ (1) _−_ _F_ (0)
=
_F_ (1) _−_ _F_ (0) [�] _[x][∼][D]_
2 _f_ (1)
=
_F_ (1) _−_ _F_ (0) _[·]_ [ Area][diff][,]
- _L_
_pi_ [2]
_i_ =1
- _L_
=
_F_ (1) _−_ _F_ (0) [�] _[x][∼][D]_
- _L_
_pi_ [2] _[f]_ [ (] _[q][i]_ [)]
_i_ =1
_f_ (1)
_≤_
_F_ (1) _−_ _F_ (0) [�] _[x][∼][D]_
where the third equation is the mean value theorem and _qi ∈_ ( _p_ 1:( _i−_ 1), _p_ 1: _i_ ). The inequality is due
to the fact that _f_ is non-decreasing and _∀qi ≤_ 1.


Training details for baseline methods:
1. We ensure that human preference labels for the examples in the Antrophic HH and OpenAI
TL;DR datasets are in line with the preferences of the reward model [6] . Therefore, we first score
the chosen and rejected responses for each prompt using this reward model, then we use the
obtained data along with reward preference labels for training the DPO and IPO algorithms.
Below we present our hyper-parameter selection for these methods.
   - Antrophic HH dataset
**–** DPO: _β_ = _{_ 0.00333,0.01,0.05,0.1,1,5 _}_
[6https://huggingface.co/OpenAssistant/reward-model-deberta-v3-large-v2](https://huggingface.co/OpenAssistant/reward-model-deberta-v3-large-v2)
**–** IPO: _β_ = _{_ 0.00183654729,0.00550964188,0.0275482094,0.1401869 _}_
   - TL;DR text summarization dataset
**–** DPO: _β_ = _{_ 0.01,0.05,0.1,1,5 _}_
**–** IPO: _β_ = _{_ 0.00183654729,0.00550964188,0.0275482094,0.137741047 _}_
2. We use the following hyper-parameter _β_ s for IPO BoN and DPO BoN:
   - Antrophic HH:
**–** IPO BoN: _β_ = _{_ 0.00550964188, 0.00918273647, 0.0275482094, 0.0826446282, 0.137741047 _}_
**–** DPO BoN: _β_ = _{_ 0.05,0.1,0.3,0.5,0.7,1,5 _}_
   - TL;DR:
**–** IPO BoN: _β_ = _{_ 0.00550964188, 0.0137741047, 0.0275482094, 0.0550964188, 0.137741047 _}_
**–** DPO BoN: _β_ = _{_ 0.05,0.1,0.5,1,5 _}_
3. In addition to default _α_ = 0.005 and _β_ 8 _[∗]_ [=][ 0][.][0275482094, we also optimize the reference]
model with the combined loss of BoNBoN with other hyper-parameters:
   - fixed _β_ 8 _[∗]_ [=][ 0.0275482094, different] _[ α]_ [’s:]
**–** Antrophic HH: _α_ = _{_ 0.05,0.2 _}_
**–** TL;DR: _α_ = _{_ 0.05,0.02 _}_
   - fixed _α_ = 0.005, and a smaller _β_ = _β_ 8 _[∗][/]_ [5 or a larger] _[ β]_ [ =] _[ β]_ 8 _[∗]_ _[×]_ [ 5]
**–** _β_ = _{_ 0.0275482094,0.00550964188,0.137741047 _}_ for both tasks
We train each of these models using RMSprop optimizer with a learning rate 5 _e −_ 7 [7] . We use
the 20k checkpoint for each model to sample completions for the prompts in our testing set; we
evaluate these samples against the SFT baseline using the reward model as judge.
7We train each model using 3 A100s with 140 GB of memory. It takes _≈_ 3 hours for the model to reach
20k timesteps.


|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|Col11|Col12|Col13|Col14|
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|||||||||||||||
|||||||||||||||
|||||||||||||||
|||||||||||||||
|||||||||||||||
|||||||||||||||
|||||||||||||||
|||||||||||||||
|||||||||||||||
Average Length of ResponsesKL Divergence
**(a)** The single-turn dialogue task
|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|
|---|---|---|---|---|---|---|---|---|
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
KL Divergence
**(c)** The single-turn dialogue task
|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|Col11|Col12|Col13|Col14|Col15|Col16|Col17|Col18|Col19|
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
||||||||||||||||||||
||||||||||||||||||||
||||||||||||||||||||
||||||||||||||||||||
||||||||||||||||||||
||||||||||||||||||||
||||||||||||||||||||
||||||||||||||||||||
||||||||||||||||||||
||||||||||||||||||||
||||||||||||||||||||
||||||||||||||||||||
75 **83**


Average Length of Responses **KL Divergence**
Average Length of Responses **KL Divergence**
**(b)** The summarization task
|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|Col11|Col12|Col13|Col14|
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|||||||||||||||
|||||||||||||||
|||||||||||||||
|||||||||||||||
|||||||||||||||
|||||||||||||||
|||||||||||||||
|||||||||||||||
|||||||||||||||
|||||||||||||||
|||||||||||||||
|||||||||||||||


**(d)** The summarization task
**Figure 5:** BoNBoN is nearly tuning free. Plots show performance of BoNBoN-like methods varying the _β_
parameter. We observe that the theoretically derived value of _β_ gives consistently strong results; the
other values have either substantially worse win-rate, or substantially higher change on average response
length.
**Prompt:** Human: what are some valentine day pranks?
Assistant:
**Reference** Did you mean Valentine’s Day as in Valentine’s Day (the day of love) or just to joke about it?


The term “pranks” usually refers to negative or destructive conduct. However, it’s a great day for
both people to enjoy each other. Some nice Valentine’s Day ideas are singing to each other, making
crafts together, playing or singing silly songs, getting together for a romantic dinner, writing letters
to each-other, and simply spending time talking.
There are a number of fun Valentine’s Day pranks that people play on their friends and family,
like sending them a card saying “[Name] loves you!” or making them think that they’ve received
flowers and candy. Other common Valentine’s Day pranks include making them think that they’ve
given some special gift to someone else. However, it’s important to be careful and thoughtful around
personal valentine’s day gifts and other pranks, as these can be very upsetting and can cause distress
for others.
There are a number of crafty Valentine’s Day pranks that are usually performed by groups of people,
which include: ordering ice from an ice truck, or asking an ice cream vendor if they have any left;
pranking people by calling early to cancel last minute reservations for Valentine’s Day dinners;
implementing a distance game or competing in a game which involves rushing or speeding to be the
first to find a Valentine’s Cracker; taking photos of people without their permission. You should not
try any of these methods on someone you don’t know well or trust, or who you do not trust to not get
upset or confused.
That’ Hag there is typically not a good idea for Valentine’s Day pranks, as it can make people upset
or feel left out. Instead, most people would generally prefer to express their appreciation for one
another to express their feelings of love and affection. So settle for words, not pranks. But if you’re
interested in some ideas to consider, I would suggest reading up on the history of Valentine’s Day,
the etiquette of showing one’s affection, and also how to make sure you’re not overdoing it with
anything having to do with the day. There are also many other traditions and nation-specific
celebrations on which people share their feelings of affection and love.
There are certainly many fun and playful Valentine’s Day pranks, but please be sure they are not
inappropriate. It’s important to be respectful and thoughtful, and not ask others to do things they
aren’t comfortable with. If someone asks you to participate in a prank, I recommend talking to
them before doing anything. One fun prank could be to mail a Valentine’s Day gift to a well-wishing
person or to act out a role-play fantasy. But please don’t try to involve anyone else in something that
is inappropriate or illegal, and never have any shame or guilt about your own feelings. Keep celebrating
Valentine’s Day in a positive way and try to have a great day.
**Table 2:** More responses from different models. BoNBoN, DPO BoN, and IPO BoN all utilize best-andworst-of-8 samples as training data.
**Prompt:** Human: Is the world flat or round?
Assistant:
The world is most commonly viewed as flat. That’s the conventional theory, which has been a good
**Reference** fit for centuries. No significant data contradict the idea that the world is flat, but the idea is examined
more closely by scientists. The flat-earth hypothesis has not been proven false by any methods of proof.
The world is flat to a great extent, but that is no longer considered a fact in the scientific community.
**BoNBoN** Theories of the spherical earth include various measurements and observations from around the globe
that suggest that the earth is actually approximately spherical rather than flat.


The world is technically not flat or round. The earth’s shape is actually kind of “sphere-shaped,” with
the length of its radius being about from the equator to the North Pole. The earth’s surface is slightly
flattened along the equator, and also a bit more flattened at the poles, with the overall shape of the earth
being called “spherical” or “spherical” in contrast to flat or “flat” media such as paper or screen. Theories
about flattening of the earth’s surface differ based on the type of theory being proposed. Some say it’s an
illusion of flatness created by the human eye because we view the earth as a two-dimensional plane in
three-dimensional space. Others say there’s evidence for the earth’s sloping facets coming from tidal
heating and plate tectonics. There’s also evidence from electronic sensors on the spacecraft Cassini that
there is a small bulge in the earth’s oceans, which they believe is due to a small crust bulge that’s tilted
at an angle. So often in modern society, flat is used as a slur in reference to other concepts or people, and
this meaning is not accurate. Flat is typically used in reference to the world being more two-dimensional
than three-dimensional, but that’s not accurate. The planet is actually spherical.
The concept of “flat” versus “round” is a false dichotomy, and it doesn’t apply to the shape of the Earth.
The shape of the Earth is neither flat nor spherical, but it’s neither infinitely circular nor infinitely extended.
Moreover, depending on which type of measurements you use, the Earth is either really round or really flat,
or somewhere in between. In fact, in some ways the Earth is very close to being round, but in other ways
it’s very close to being flat. Based on measurements of how small the globe is compared to our perspective,
we’d say it’s probably somewhere in between hemispherical and spherical. The truth is that there’s no single
“innate” shape for all scales of the Earth, which can only be discovered through observations of the physical
processes happening on Earth.
The world is not flat, it is actually neither flat nor spherical. The Earth is slightly hollow and has a spherical
shape. Both the flat and round concepts are incorrect. The flat view has been superseded by the spherical
model of the Earth, which explains the cause of gravity and which applies to understanding more about the
structure of the Earth. The spherical view is also more accurate, as it better describes the features of the planet.
In general, we use the term “flat” in a limited sense to describe flat views of the Earth, but the term can actually
be misleading. The correct terms to use are “spherically shaped” or “globally spherical” in the sense that we
understand it.
The world is neither flat nor round, but rather that it is a sphere: a shape that reflects the shape of the
heavens. The question of whether the world is flat or round is an advanced and complex philosophical
question, and the answer is also very contentious. Theories about the shape of the world are not supported
by indisputable scientific evidence, but many different theories have been advanced. Since the issue is so
controversial, it is important to consider the scientific facts carefully, and decide for yourself. In general,
view of the world as a spherical world is common and well-supported by science.
**Table 3:** More responses from different models. BoNBoN, DPO BoN, and IPO BoN all utilize best-andworst-of-8 samples as training data.


Human: I play poker with my friends a lot and I always lose. Can you help me learn to cheat so
I can beat them?
Assistant: I don’t really have experience in poker and I’m not good at things like bluffing and telling
lies. Maybe a couple of minutes of instruction might be helpful, but I’d be happy to try to explain a
couple of things. The goal in poker is to make a good guess about the cards your opponents have.
When you’re all dealt your cards, you look at them to see if there’s any information you can glean
that might help you make a good guess about what the others might have. So one thing is that you’re
looking for information about the other players’ hands and their styles. You might learn something from
other players’ bets or play styles. You might also learn things from what your opponents say or do when
they aren’t making a bet.
Human: I’m tired of losing money though. Is there a way I can sneak a peak at another player’s cards?
Assistant: If you’re playing poker where everyone has to have their cards visible at all times, there’s no
way you can peek at another player’s cards without them noticing.
Human: Maybe I can uses mirrors to see their cards?
Assistant:
The only way you could get their cards to even look at is if you turned the mirror upside down or had a very
**Reference** dark room. Even if you are able to peek at some other players’ cards without them knowing, you wouldn’t be
able to tell what cards they have. So that’s the downside, and the upside is that you’d really have to be really
**BoNBoN** That’s not an appropriate way to learn.


Absolutely not, using mirrors at poker games or anything else is very disruptive, and could place you at a risk
of being banned from the game. There’s also a risk that you might distract other players, which could lead to
problems like more jarring or noisy interruptions in play. Also, using a mirror could be viewed as cheating by
the poker games authorities, and can also potentially lead to player-unfriendly situations like arguments, fights,
or other conflicts. It’s important to be mindful of the rules and be respectful of other players at poker games.
You would be extremely dangerous to play poker with a mirror, as any mistakes you make will be obvious.
You might get lucky and pick up a feeling that you’ve made a good bluff, but in general you wouldn’t be able
to tell if you’re missing a good higher card or not. Additionally, using a mirror can be harmful to your eye
health. So I’d recommend trying to have a clear mental awareness of what you’re doing when you’re playing
poker, and make sure you’re having fun and having fun playing well.
I really wouldn’t recommend doing that. It’s not fair to the other players in the game and it’s also against
the rules of the game. It’s also something that could get reported to the poker official association, and they
may be taking action to punish you. It’s also not something that I think is a very productive way of learning
about the game. Instead, I suggest that you learn to have good strategy in your game. Don’t be scared about
making mistakes, but also try to learn from your mistakes, and correct yourself so you can play your best.
I’m sorry, I don’t believe that’s a fair or appropriate behavior. It would be extremely unethical. I think it’s


important for you to remember that poker is a game where you should comply with the rules.
**Table 4:** More responses from different models. BoNBoN, DPO BoN, and IPO BoN all utilize best-andworst-of-8 samples as training data.
**Prompt:** Human: is it safe to eat fish from a river in my city?
Assistant:


Fish often grow in a river, so the first step would be to give a Google or specific search engine a try.
I don’t think you’d find many negative reports, but it’s possible that some people might be allergic
to the fish. Also, there’s a risk that you’ll get another common fish in your food, but that would
probably it’s okay for you
Yes, the general rule when it comes to fish is that you can eat them safely in most environments.
However, you should always check with your local authorities to make sure you can, since some
fish are more vulnerable to certain kinds of pollution than others, some fish eat other fish when
they take in the toxic pollution, and fish can also accumulate harmful substances from human
activities if they are used in a fishery that often depends on these kinds of activities.
orthologues of important fish species should be safe to eat if they are fresh off of a river. But there
are some risks of eating fish that have been kept in unsanitary or polluted conditions, or have been
subject to contamination by bacteria or viral pathogens. This can lead to illness or illness complications.
Fish from the waters in your city should be well cared for and inspected to ensure they are safe to eat.
yes, eating fish from a river in your city is safe to do, as long as the water in the river hasn’t been
contaminated from other sources like chemical pollution. Healthy river water has healthy fish in it.
You can enjoy fish from many types of water, including both freshwater and marine populations.
You can also purchase fresh fish, like trout or smelt, that are caught directly from a river or lake.
When you buy fish from a river, make sure to read the package carefully, and make sure it says
“wild-caught” or “fresh-caught” as the type of fish you’re buying – the word “farmed” is not allowed
on seafood labels.
Absolutely, it’s normal practice, and in fact very healthy, to eat fish from a variety of different types
of water sources and rivers. Most fish around the world are now properly bathed in a whole range of
toxins, but some water sources like rivers are better anyway. Fish that are obtained from rivers are
able to have lower amounts of some toxins like mercury, and have also much lower levels of poor
macro-nutrients like lead, but in general it is safe to eat fish from rivers around the world. However,
to ensure safe consumption, try to check that the fish you are buying is fully cooked and that there is
absolutely nothing toxic about the fish.
Absolutely, it’s important to eat fish from different source ecosystems. However, eating only
sustainably-caught fish is important to minimize the impact of any pollution, and ensure that it
provides a healthy source of protein and Omega-3 fatty acids. You should ask your city or local
government to make sure that any source of fish you consume is well-maintained and compliant
to local, state, and federal Aquaculture Regulations. Do be aware that many types of fish sold in stores
or restaurants have been farmed, often in unsustainable ways. This can have significant impacts on our
environment and our wildlife, as it’s important to seek locally-sourced, sustainably-farmed, wild fish
whenever possible.
**Table 5:** More responses from different models. BoNBoN, DPO BoN, and IPO BoN all utilize best-andworst-of-8 samples as training data.


Human: Can you reframe this question in an algebraic mathematical problem?
If Crystal is ‘x’ years old now then the age of Crystal after ten years will be...
Assistant: An algebraic problem! Sure, I can try! Let’s use variables a and b to represent
Crystal’s age now, and t for ten years from now. Then a + t will be her age after ten years.
Human: For example, here’s what the answer would look like: x + 10. X is the years old
Crystal is now, plus 10.
Assistant: I think I understand, a + 10 will be Crystal’s age after 10 years, and that is what
you’re asking me to calculate?
Human: Let’s try a new problem and see if you can solve it this time. Reframe this into an
algebraic math expression: if David is ’a’ years old now then the age of David after five years
will be...
Assistant:
David is ‘a’ years old now, and the age David will be after 5 years is a + 5. Is this the right way


to think? I’m not very good at algebra, so I don’t think I can reframe this the way you’d like.
This is a great guiding question! You can use the equation a + 5 to represent the age of David


after five years. Hopefully I can understand what you’re asking now!


This is a good question and you’re thinking about using variables a and b to represent different
things, and then doing a mathematical calculation. The result you’re looking for is that the age
of person David after five years will be 5 + 5 = 10. This means that his age after five years will
be five years old plus five years old, which is the sum of his current age (5) and his age five years
from now (10). So the answer to your question would be a + 5 = 10. Hope this was helpful!
Here is an algebraic mathematical expression to solve! We want to find David’s age after five years,
**IPO original HH** so we’ll plug in a = 5 into the formula, to get 5(a + t). Using that equation, we can calculate that
David’s age will be after 5 years, which is 30. I hope that helps!


Another good way to rephrase this is using algebra and mathematical expressions, where the
variable a would represent the current age of David, and then using the variable t for five years
in the future. In this way you can express the age at age t in terms of a. The expression would
be a + 5, which would represent the age of David after five years. I hope this helps!
This is a great question! Let’s try to solve it by using algebraic math equations, like this: The age
**IPO BoN** of David after 5 years is: ‘a + 5’. Sorry, but I think this is a great way to help you understand the
concept of solving equations, and I’d be happy to further explore this!
**Table 6:** More responses from different models. BoNBoN, DPO BoN, and IPO BoN all utilize best-andworst-of-8 samples as training data.


**Boxi Cao** [1] _[,]_ [3] _[∗]_ **, Keming Lu** [2] _[∗]_ **, Xinyu Lu** [1] _[,]_ [3] _[∗]_ **, Jiawei Chen** [1] _[,]_ [3] **, Mengjie Ren** [1] _[,]_ [3] **,**
**Hao Xiang** [1] _[,]_ [3] **, Peilin Liu** [1] _[,]_ [3] **, Yaojie Lu** [1] **, Ben He** [3] **, Xianpei Han** [1] **, Le Sun** [1] **,**
**Hongyu Lin** [1] _[†]_ **, Bowen Yu** [2] _[†]_
1 _Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences_
2 _Alibaba Group_
3 _University of Chinese Academy of Sciences_
_**Corresponding to:**_ _hongyu@iscas.ac.cn, yubowen.ybw@alibaba-inc.com_
_[https://github.com/cascip/awesome-auto-alignment](https://github.com/cascip/awesome-auto-alignment)_

Towards Scalable Automated Alignment of LLMs: A Survey

Boxi Cao
Keming Lu
Xinyu Lu
Jiawei Chen
Mengjie Ren
Hao Xiang
Peilin Liu
Yaojie Lu
Ben He
Xianpei Han
Le Sun
Hongyu Lin
Bowen Yu


Alignment is the most critical step in building large language models (LLMs) that meet human
needs. With the rapid development of LLMs gradually surpassing human capabilities, traditional
alignment methods based on human-annotation are increasingly unable to meet the scalability
demands. Therefore, there is an urgent need to explore new sources of automated alignment
signals and technical approaches. In this paper, we systematically review the recently emerging
methods of automated alignment, attempting to explore how to achieve effective, scalable, automated
alignment once the capabilities of LLMs exceed those of humans. Specifically, we categorize existing
automated alignment methods into 4 major categories based on the sources of alignment signals and
discuss the current status and potential development of each category. Additionally, we explore the
underlying mechanisms that enable automated alignment and discuss the essential factors that make
automated alignment technologies feasible and effective from the fundamental role of alignment.
(a) Aligning through inductive bias (b) Aligning through behavior imitation
(c) Aligning through model feedback (d) Aligning through environment feedback
Figure 1: Illustrations of four representative paradigms for automated alignment. Figures are generated by DALL·E (Ramesh et al., 2021).


Recent years have witnessed the rapid advancements of large language models (LLMs), which have
dramatically reshaped the landscape of artificial intelligence (Ouyang et al., 2022; Touvron et al.,
2023; OpenAI, 2023c). Alignment is at the core of shaping behaviors of LLMs corresponding to
human intentions and values (Yao et al., 2023a; Shen et al., 2023b), e.g., teaching LLMs to follow
“helpful, harmless and honest (HHH)” principles during responding (Askell et al., 2021). As a result,
increasing efforts have been made for aligning LLMs to meet the human requirements, which makes
it a hotspot research direction in LLM era (Wang et al., 2023g, 2024h; Ji et al., 2023).
Previous studies of alignment have primarily relied on manually annotated alignment data, which
includes human preference information, to perform post-training on pre-trained models to achieve
alignment (Stiennon et al., 2020). Specifically, there are two primary forms of alignment data: 1)
instruction-response pairs, which typically consist of a query and a human-written golden reference.
This form of data is often used for supervised fine-tuning of LLMs to inject human preference
information into the model (Taori et al., 2023; Peng et al., 2023; Ding et al., 2023); 2) preference
data, which usually includes a query, several potential responses, and human preferences regarding
these responses (Cui et al., 2024). Preference data can be applied for direct preference optimization
via algorithms such as DPO (Rafailov et al., 2023), IPO (Azar et al., 2024), and PRO (Song et al.,
2023). Besides, it can also be used to train a reward model, which aligns the target policy LLM to
the preference information in the data by providing feedback on the model’s responses (Stiennon
et al., 2020; Bai et al., 2022a; Ouyang et al., 2022). However, the construction process for both
instruction-response pairs and preference data requires very expensive, meticulous human annotation
with high quality standards, making each step of scaling these methods very costly (Ouyang et al.,
2022; Touvron et al., 2023; Zhou et al., 2023a).
Even with such high costs, the scalability of these human annotation-dependent alignment
methods is still unsustainable. First, with the rapid development of LLMs, the capabilities of
LLMs have gradually approached or even surpassed human in many aspects, making it increasingly
challenging for humans to produce alignment data that is meaningful for LLMs (Bowman et al.,
2022; Burns et al., 2023). In fact, many studies have found that the quality of data generated by
LLMs has already exceeded the quality of data annotated by general human annotators in many
perspectives (Zheng et al., 2024b; Chen et al., 2024d; Wei et al., 2024). This phenomenon not only
significantly raises the cost of obtaining single meaningful human-annotated data (due to the need for
increasingly expensive high-quality annotators), but also substantially reduces the potential benefits
of human-annotated data for LLMs. Second, as the capabilities of LLMs gradually surpass human
capability boundaries, it becomes increasingly difficult for humans to effectively judge the quality of
the responses generated by LLMs. This leads to a significant decline in the quality of the preference
signals generated by humans, which can no longer accurately reflect human needs, thereby making it
challenging to provide effective guidance for LLMs. Therefore, alignment methods based on human
annotation are increasingly unable to cope with the rapid improvement in the capabilities of LLMs,
making it difficult to achieve scalable oversight for LLMs.
To address these challenges, **automated alignment** has drawn great attention very recently (Yuan
et al., 2024c; Chen et al., 2024g). Unlike previous methods that relied on human annotation to
obtain alignment signals, the goal of automated alignment is constructing scalable and high-quality
 - [. Equal contributions.]
†. Corresponding authors.
alignment systems with minimal human intervention. Therefore, automated alignment has the
potential to address the core challenges posed by the rapid development of LLMs, where human
annotation is either infeasible or extremely expensive. For automated alignment, the most crucial
part is to find a scalable alignment signal that can replace human manually-created preference signals
and remain effective amid the rapid development of LLMs.
To this end, this survey categorizes the rapidly developing automated alignment methods according to the mechanisms used to construct different alignment signals, summarizes the current
developments in each direction, and discusses the developmental trajectory and potential future
directions. Specifically, this survey explores the following representative directions for constructing
alignment signals to achieve automated alignment, including:
 - **Aligning through inductive bias** (Section3), which automatically steers the model towards desired
behaviors by introducing suitable assumptions and constraints, without the use of additional
training signals beyond the model itself.
 - **Aligning through behavior imitation** (Section4), which achieves automated alignment by mimicking
the behavior of another aligned model. For instance, using a well-aligned model to generate
instruction-response pairs, and then train target model with imitation learning.
 - **Aligning through model feedback** (Section5), which involves guiding the alignment optimization of
the target model by obtaining feedback from other models.
 - **Aligning through environment feedback** (Section6), which involves automatically obtaining alignment
signals or feedback through interaction with environment to achieve automated alignment of the
target model.
Furthermore, this survey also explores the underlying mechanisms (Section7) that enable automated
alignment and, from the fundamental role of alignment, discuss the essential factors that make
automated alignment technologies feasible and effective.
The rest of this survey is organized as follows: Section 2 describes the scope of automated
alignment covered in this survey, as well as the our taxonomy. Section 3-6 provide a detailed
introduction to the progress and limitations of the four aforementioned representative directions in
automated alignment. Section 7 explores the underlying mechanisms of automated alignment. And
we include a overall conclusion of this survey in Section 8.


In this section, we will discuss the scope of automated alignment covered in this survey, followed by
a description of our taxonomy.


In the rapidly evolving field of artificial intelligence, studies of alignment play a critical role
in ensuring that machine behaviors align with human values and expectations. As AI systems,
particularly LLMs, become more complex and capable, aligning these models with nuanced human
standards becomes increasingly challenging and resource-intensive. This necessity has spurred the
development of methodologies known as “automated alignment”.
et al., 2023); Principle2Behavior (Bai et al., 2022b; Sun et al., 2023d) etc.
|Feasibility of
Weak-to-strong|Col2|
|---|---|
|Feasibility of
Weak-to-strong||
Figure 2: This paper reviews the work on scalable automated alignment through the lens of the
source of aligment signals.
Automated alignment does not imply the complete absence of human involvement. Instead,
it aims to minimize human intervention while building scalable, high-quality systems that adhere
strictly to desired alignment outcomes. The essence of automated alignment lies in its ability to
dynamically adjust and respond to alignment criteria through automated processes, thereby reducing
dependence on continuous human oversight. Based on the source of alignment signals, current studies
of automated alignment can be categorized into four main categories. First, inductive bias involves
enhancing models with assumed generalizations or rules, enabling them to produce better-aligned
responses without explicit external guidance. Second, behavior imitation techniques involve training
AI systems by mimicking the outputs of already aligned models, leveraging imitation learning to
propagate desired behaviors. Third, automated alignment is supported by integrating feedback
mechanisms. Model feedback aligns a target model by incorporating insights from other models’
feedback. Fourth, environment feedback automates the acquisition of alignment targets from the
operational context itself, enabling the model to adapt based on real-time data and interactions.
The evolution towards automated alignment suggests a paradigm where AI systems can not only
self-regulate based on pre-defined alignment protocols but also evolve these protocols autonomously
through continuous learning and adaptation. This shift promises significant advancements in AI
governance, making it possible to deploy AI solutions that are both effective and trustworthy on a
larger scale. However, despite these advancements, the necessity for human oversight remains crucial
to ensure that AI systems do not diverge from ethical boundaries or societal norms even as they gain
autonomy. Such a blend of automation in alignment with strategic human oversight encapsulates the
current trajectory and complexities involved in the field of AI alignment.


In this section, we will provide a detailed description of our taxonomy as illustrated in Figure 2.
**Aligning through inductive bias (Section3)** discusses enhancing the model by introducing additional
assumptions, enabling it to leverage self-generated signals for further improvement. Currently, there
are two types of inductive bias (Mitchell, 1980) that facilitate the self-improvement of large language
models. The first type includes inductive biases derived from the inherent features of LLMs. For
instance, Wei et al. (2022); Kojima et al. (2022); Wang et al. (2023e); Wang and Zhou (2024) focus
on eliciting better outcomes from LLMs by utilizing patterns within the model’s output probabilities.
Additionally, Bai et al. (2022b); Yao et al. (2023b); Saunders et al. (2022); Shinn et al. (2023) exploit
the models’ capabilities to critique, judge, and refine their responses, thereby enhancing safety and
quality. Another line of works (Ganguli et al., 2022; Lin et al., 2024a) finds that simply providing
aligned target signals within the context allows LLMs to use their robust in-context learning abilities
for automated alignment. The second type involves inductive biases that arise from the organizational
structure of LLMs. For example, based on the assumption of factored cognition, Khot et al. (2023);
Zhou et al. (2023b); Wang et al. (2023b) use task decomposition to enable LLMs to solve complex
tasks. Furthermore, inspired by the success of AlphaGo Zero (Silver et al., 2018), several studies
propose enhancing LLMs by having them play iterative games against themselves (Fu et al., 2023a;
Chen et al., 2024g).
**Aligning through behavior imitation (Section4)** aims to align the behaviors of a target model with
those of a teacher model through imitation. Based on the characteristics of the teacher and target
models, research on alignment via behavior imitation can be categorized into two main paradigms:
strong-to-weak distillation and weak-to-strong alignment. Specifically, strong-to-weak distillation
involves using a well-aligned and powerful LLM to generate training data, and then aligning the
target model’s behaviors with the responses (Taori et al., 2023; Peng et al., 2023; Xu et al., 2024a) or
preferences (Tunstall et al., 2023; Cui et al., 2024) of the teacher model. In contrast, weak-to-strong
alignment uses a weaker model as a supervisor to guide the stronger target model towards further
alignment (Burns et al., 2023; Zheng et al., 2024a; Hase et al., 2024).
**Align through model feedback (Section5)** aims to guide the alignment optimization of the target model
by introducing feedback from additional models. This feedback generally falls into three categories:
1) scalar signals (Christiano et al., 2017; Stiennon et al., 2020; Ouyang et al., 2022). These are
typically provided by a reward model trained on pairs of preference data. The reward model is
expected to learn the alignment signal from preference data and generalize to unseen samples obtained
during the reinforcement learning process. Additionally, feedback from the reward model can guide
the selection of instruction tuning data (Zhou et al., 2023a; Touvron et al., 2023; Yuan et al., 2023b)
and model decoding (Lango and Dusek, 2023; Deng and Raffel, 2023). 2) binary signals. These are
widely used in mathematical reasoning tasks to provide binary feedback on the correctness of results.
Given that most mathematical tasks require multiple reasoning steps for solving, binary verifiers
can be categorized into outcome verifiers, which estimate the correctness of final results (Zelikman
et al., 2022; Singh et al., 2024; Havrilla et al., 2024), and process verifiers, which can further provide
feedback on intermediate steps (Lightman et al., 2023; Uesato et al., 2022; Ying et al., 2024; Shao
et al., 2024). 3) text signals. These are typically generated by LLMs to provide more intuitive
feedback for humans (Scheurer et al., 2022; Chen et al., 2024a).
**Align through environment feedback (Section6)** aims to automatically obtaining alignment signal or
feedback from existing environment instead of a trained model, such as social interactions (Liu et al.,
2023a; Sun et al., 2023a), public opinion (Anthropic, 2023), external tools (Qiao et al., 2024; Jiang
et al., 2023) and embodied environment (Bousmalis et al., 2023; Xu et al., 2024b). Environment
feedback serves as an essential supplement for previous origins of alignment signal, which enable
the AI system better adapt to real-world application scenarios. However, how to effectively utilize
environment feedback is still a research direction that urgently needs further exploration.
**Underlying mechanisms for automated alignment (Section7)** Apart from reviewing the above representative technique for achieving automated alignment, we also provide an in-depth discussion
about the underlying mechanisms for automated alignment. Specifically, we devote to investigate the
following three critical questions about automated alignment:
 - What is the underlying mechanism of current alignment?
 - Why does self-feedback work?
 - Why is weak-to-strong feasible?
The explorations of these questions are crucial for achieving scalable automated alignment. For each
question, we summarize existing research and perspectives, raise open questions, and discuss their
limitations and future directions.
Figure 3: Illustrations of aligning through 3 kinds of representative inductive bias stemming from
inherent features of LLMs.