University of Wisconsin-Madison [1]
Stanford University [2]
mkhanov@wisc.edu, jirayu@stanford.edu, sharonli@cs.wisc.edu
ABSTRACT
Aligning large language models with human objectives is paramount, yet common
approaches including RLHF suffer from unstable and resource-intensive training.
In response to this challenge, we introduce **ARGS**, Alignment as Reward-Guided
Search, a novel framework that integrates alignment into the decoding process,
eliminating the need for expensive RL training. By adjusting the model’s probabilistic predictions using a reward signal, ARGS generates texts with semantic diversity while being aligned with human preferences, offering a promising and flexible solution for aligning language models. Notably, ARGS demonstrates consistent enhancements in average reward compared to baselines across diverse alignment tasks and various model dimensions. For example, under the same greedybased decoding strategy, our method improves the average reward by 19.56% relative to the baseline and secures a preference or tie score of 64.33% in GPT-4 evaluation. We believe that our framework, emphasizing decoding-time alignment,
paves the way for more responsive language models in the future. Code is pub[licly available at: https://github.com/deeplearning-wisc/args.](https://github.com/deeplearning-wisc/args)
1 INTRODUCTION
Large language models (LLMs) trained on massive datasets exhibit a remarkable ability to handle a
wide array of tasks (Wei et al., 2022; Kaddour et al., 2023). However, due to the varied nature of their
training data, these models can inadvertently generate misinformation and harmful outputs (Gehman
et al., 2020; Weidinger et al., 2021; Deshpande et al., 2023). This concern underscores the urgent
challenge of language model alignment: ensuring these models’ behaviors agree with human objectives and safety considerations (Ngo et al., 2023; Casper et al., 2023).
In recent years, a spectrum of alignment strategies have emerged, with prominent methods showcasing the effectiveness of reinforcement learning with human feedback (RLHF) (Christiano et al.,
2017; Ziegler et al., 2019; Ouyang et al., 2022; Bai et al., 2022). RLHF has gained widespread
adoption among state-of-the-art models, including OpenAI’s GPT-4 (OpenAI, 2023), Anthropic’s
Claude (Anthropic, 2023), Google’s Bard (Google, 2023), and Meta’s Llama 2-Chat (Touvron et al.,
2023b). A pivotal component within RLHF is proximal policy optimization (PPO), which employs
an external reward model that mirrors human preferences for its optimization process. However, as
noted in previous studies (Henderson et al., 2017; Wang et al., 2023; Rafailov et al., 2023; Zheng
et al., 2023b), implementing PPO introduces challenges of unstable and costly training. Furthermore, the need to repeat PPO training when altering the reward model hinders rapid customization
to evolving datasets and emerging needs.
To address the aforementioned challenge, we introduce Alignment as Reward-Guided Search, or
**ARGS**, a novel framework designed to enhance the alignment of generated text with human-desired
preferences. ARGS achieves this by employing a reward mechanism that directly guides the text generation process of a language model. Unlike traditional alignment approaches, our method integrates
alignment into the decoding process, enabling quick realignments without having to go through the
exhaustive process of retraining the foundational model using PPO. This is especially valuable in
today’s rapidly changing field of machine learning, and ensures that models remain relevant and
responsive to contemporary requirements without the need for extensive overhauls. Specifically, at
each decoding step, our key idea is to adjust the model’s probabilistic prediction using a reward
signal. This adjustment is crucial as it enables the generated text to both **(1)** _maintain the semantic_
*Equal contributions. Work done while J.B. was an undergraduate researcher at UW-Madison.
Published as a conference paper at ICLR 2024
Figure 1: Illustration of ARGS (Alignment as Reward-Guided Search) framework.
_relevance with respect to the previous context, and_ **(2)** _align with the reward criteria and human_
_preference_ . These two sub-goals can be flexibly traded off with proper weighting on the reward
signal, which degenerates to the standard maximum-likelihood decoding when the weight is zero.
Notably, our reward-guided score can be seamlessly integrated with various token selection strategies, including both greedy and stochastic sampling.
We validate ARGS on the large-scale HH-RLHF (Helpful and Harmless) dataset (Bai et al., 2022)
and demonstrate that our technique effectively guides the generation towards outputs that are preferable. For example, our method improves the average reward by _↑_ 19.56% relative to the standard
decoding and secures a preference or tie score of 64.33% in GPT-4 evaluation. Moreover, our
method excels at generating lexically diverse continuations without compromising their contextual
consistency. Qualitatively, ARGS offers less redundant and more informative outputs than the standard maximum-likelihood decoding, as illustrated in Table 1. Additionally, we further emphasize
the versatility of ARGS and demonstrate consistent improvement across different model architectures (LLaMa and OPT), sizes, and alignment tasks including Stanford Human Preferences (SHP)
dataset (Ethayarajh et al., 2022). To summarize our contributions:
1. We propose a novel framework ARGS, which postulates the alignment process as a rewardguided search problem that runs during decoding time. This framework not only omits the
need for expensive RL training but also facilitates flexible customization to emerging needs.
2. We conduct both qualitative and quantitative evaluations of ARGS’s performance, showcasing its superiority over existing approaches. ARGS effectively guides the outputs of the
neural language model in alignment with human preferences.
3. Importantly, ARGS brings a new perspective of decoding-time alignment to the field of
AI safety. While traditional alignment strategies focus on optimization during the training
phase, decoding-time alignment emphasizes the pivotal role of post-training adjustments.
Such a shift in focus allows models to adjust to new reward signals and user requirements
without the need for extensive retraining. We hope this inspires further research into post
hoc alignment, leading to more efficient and safer AI systems in real-world applications.
2 ARGS: ALIGNMENT AS REWARD-GUIDED SEARCH
In this section, we introduce ARGS, a novel decoding framework that facilitates the alignment of
generated text with human preferences, by employing a reward mechanism that directly guides the
text generation process of a language model. Our method has two main components: (1) _reward-_
_guided scoring_, which assigns scores to possible continuations of the text, and (2) _token selection_,
which selects a continuation. We detail the reward-guided scoring method in Section 2.1 and the
token selection methods in Section 2.2.
Published as a conference paper at ICLR 2024
2.1 REWARD-GUIDED SCORING
Our goal is to steer the decoded outputs of language models in alignment with human preference. At
each decoding step, our key idea is to adjust the model’s probabilistic prediction by a reward signal
(Figure 1). This adjustment is crucial as it enables the model to generate text that is not only coherent
and contextually relevant but also tailored to satisfy specific alignment criteria or objectives.
Specifically, a reward model (RM) assigns a scalar reward value to each response. Following Stiennon et al. (2020), reward models are often trained on a dataset comprised of paired comparisons
between two responses generated for the same input or prompt. Formally, the reward modeling loss
for each pair of preferred sample ( _**x**_ _, yw_ ) and less preferred sample ( _**x**_ _, yl_ ) is defined as follows:
_L_ RM( _**x**_ _, yw, yl_ ; _θ_ ) = log _σ_ ( _r_ ([ _**x**_ _, yw_ ]) _−_ _r_ ([ _**x**_ _, yl_ ])) _,_ (1)
where _θ_ is the parameterization of the reward model, _σ_ ( _·_ ) is the sigmoid function, _r_ ([ _**x**_ _, y_ ]) is the
scalar reward for a given pair of input _**x**_ and response _y_, and [ _**x**_ _, y_ ] represents concatenation of the
prompt and response.
Given the previous context _**x**_ _<t_ and timestamp _t_, we formalize our reward-guided scoring function
for a token _v_ :
_s_ ( _v,_ _**x**_ _<t_ ) = LM( _v |_ _**x**_ _<t_ ) + _w · r_ ([ _**x**_ _<t, v_ ]) _,_ (2)
where LM( _v|_ _**x**_ _<t_ ) is the model’s assigned output for token _v_, _w_ is the weight assigned to the reward
scalar, and [ _**x**_ _<t, v_ ] represents concatenation of _v_ to the previous context.
Our scoring function is more desirable than the vanilla decoding strategy, since the generated text
is encouraged to both **(1)** maintain the semantic coherence and relevance with respect to the previous context and **(2)** align with the reward criteria and human preference. These two sub-goals
can be flexibly traded off with the weighting parameter _w_, which we analyze comprehensively in
Section 3.2.
2.2 TOKEN SELECTION
Our reward-guided score can be flexibly used by different token selection strategies. Here, we
consider two popular selection strategies: greedy selection and stochastic sampling. We describe
both variants below, dubbed ARGS-greedy and ARGS-stochastic respectively.
**ARGS-greedy.** The greedy method selects a candidate continuation based on the maximum scores,
which can formulated as follows:
_v_ selected = arg max _s_ ( _v,_ _**x**_ _<t_ ) _,_
_v∈V_ [(] _[k]_ [)]
where _V_ [(] _[k]_ [)] is a set of the _k_ most likely predictions according to the model’s predicted probability
distribution _p_ ( _·|_ _**x**_ _<t_ ), and enables effectively reducing the search space to probable tokens without
considering all possible tokens.
After selecting a continuation token, _v_ selected, we construct the next context as follows:
_**x**_ _t_ = [ _**x**_ _<t, v_ selected] _,_
where _**x**_ _t_ is the new context for the next iteration. We iteratively generate the next best token using
our method until we reach the desired number of tokens.
**ARGS-stochastic.** Stochastic method samples token from a renormalized probability distribution
among the top- _k_ candidate tokens. Specifically, a token _v_ is randomly chosen with the following
probability:
exp( _s_ ( _v,_ _**x**_ _<t_ ) _/τ_ )
_p_ ( _v,_ _**x**_ _<t, τ_ ) = ~~�~~ (3)
_vi∈V_ [(] _[k]_ [)][ exp(] _[s]_ [(] _[v][i][,]_ _**[ x]**_ _[<t]_ [)] _[/τ]_ [)] _[,]_
where _τ_ is the temperature. A larger _τ_ makes the distribution more uniformly distributed, leading
to a random selection. Conversely, as the temperature _τ_ approaches 0, the probability _p_ ( _v,_ _**x**_ _<t, τ_ )
approaches 1 for the token _v_ with the maximum score, similar to the greedy decoding method. We
update the context, _**x**_ _t_, using the same concatenation process as described above in ARGS-greedy.
Published as a conference paper at ICLR 2024
**Algorithm 1** ARGS-greedy
**Input:** Previous context _**x**_ with _n_ tokens, number of candidates _k_, reward coefficient _w_, desired
number of tokens _m_, base model LM, and reward model
**Output:** A generated sequence with _m_ tokens
1: **for** _t ←_ _n_ to _m −_ 1 **do**
2: _V_ [(] _[k]_ [)] _←_ top- _k_ tokens with highest likelihood
3: **for** _v ∈_ _V_ [(] _[k]_ [)] **do** _▷_ Iterate over top- _k_ candidates
4: reward _←_ _r_ ([ _**x**_ _, v_ ]) _▷_ Compute a reward of this candidate
5: scores( _v_ ) _←_ LM( _v |_ _**x**_ ) + _w ·_ reward
6: **end for**
7: _v_ selected _←_ arg max _v∈V_ ( _k_ ) scores( _v_ ) _▷_ Select token
8: _**x**_ _←_ [ _**x**_ _, v_ selected]
9: **end for**
10: **return** _**x**_
2.3 IMPLEMENTATION AND COMPLEXITY
We exemplify the complete pipeline of our method with greedy decoding in Algorithm 1. In each
decoding step, the following steps are performed: the language model computes the prediction
scores for the next tokens (line 2), the rewards for all top- _k_ tokens are computed (lines 3-6), and the
context is updated with a token with the highest score (line 8). One can switch from ARGS-greedy
to ARGS-stochastic, by modifying line 7 to be probabilistic sampling using Equation 3.
The time complexity of ARGS is primarily governed by two operations: computing the predictions
and calculating the associated rewards for each candidate token. Consider the complexity of a single
decoding step for a context with _t_ tokens. The complexity for this is given by
_T_ ( _t_ ) = _T_ LM( _t_ ) + _k · Tr_ ( _t_ + 1) _,_
where _T_ LM and _Tr_ denote the time complexity associated with the base model and the reward model,
respectively.
As described in Vaswani et al. (2017), utilizing the transformer architecture results in a complexity
of _O_ ( _t_ [2] ) for both the base model and reward model. This quadratic factor emerges due to the selfattention mechanism in transformers, which requires pairwise calculations of attention scores across
all tokens. Thus, the initial decoding step for ARGS, with an original context length of _n_, exhibits a
complexity of _T_ ( _n_ ) = _O_ ( _k · n_ [2] ).
For subsequent tokens, since we add one token at a time to the context, we can reuse previously
calculated attention, reducing the complexity to _T_ LM( _t_ ) = _O_ ( _t_ ). Similarly, by retaining the attention
from the previously selected candidate, the reward model complexity becomes _Tr_ ( _t_ + 1) = _O_ ( _t_ ).
Therefore, each of the subsequent decoding steps is characterized by _T_ ( _t_ ) = _O_ ( _k · t_ ), where _t_ spans
from _n_ + 1 to _m −_ 1.
To conclude, the aggregate time complexity of our proposed approach is:
_T_ ARGS( _n, m, k_ ) = _O_ ( _k · n_ [2] ) +
_m−_ 1
- _O_ ( _k · t_ ) = _O_ ( _k · m_ [2] ) _._
_t_ = _n_ +1
In contrast to the classical decoding methods with complexity _O_ ( _m_ [2] ), the ARGS approach introduces only a constant factor of _k_ in its complexity which arises due to the need to consider the top- _k_
candidate tokens at each decoding step. Even though this appears to add more complexity than the
original method, we find that _k_ can be considerably small (Section 3.2), rendering the complexity
more tractable. We discuss the practical computation further in Section 4.
3 EXPERIMENTS
This section presents empirical experiments to evaluate the effectiveness of our proposed method.
In particular, we aim to show that ARGS can effectively guide the outputs of the neural language
model in alignment with human preference, such as helpfulness and harmfulness. All of our experiments are based on open-sourced language models and datasets. Our code is released publicly for
reproducible research.
Published as a conference paper at ICLR 2024
3.1 SETUP
Our goal is to steer the model to generate helpful and harmless responses. This task is fundamental in
understanding the practical applicability of our proposed decoding method in real-world scenarios,
where the generation of helpful and harmless text is of utmost importance for AI assistants.
**Experimental details.** To evaluate the performance of our approach, we employ ARGS on the
HH-RLHF (Helpful and Harmless) dataset (Bai et al., 2022), which is the most commonly adopted
benchmark for alignment. The dataset consists of 112,000 training samples and 12,500 test samples
and is publicly available [*] . Each sample in the dataset includes a prompt and two responses, with
one being preferred over the other. The selected responses are annotated based on the opinions of
crowd workers, who assess which response is more helpful and harmless. For a base model, we
use LLaMA-7B (Touvron et al., 2023a) as a pre-trained language model and fine-tune it on only
preferred responses of the HH-RLHF dataset for one epoch. The fine-tuned model is referred to
as LLaMA-7B-SFT. We then train the reward model from the fine-tuned model on HH-RLHF,
employing a pairwise reward loss introduced in Ouyang et al. (2022). The trained reward model
attains a final accuracy of 74.58% on the validation set. Full details on training hyperparameters are
included in Appendix A.
**Decoding.** We evaluate the models by producing text responses given the conversation prompts
from the HH-RLHF test set. For main results, we test decoding methods based on the fine-tuned
LLaMA-7B model by default. Following the standard practice, we limit the maximum lengths of
the prompt and generated continuation to 2,048 and 128 tokens, respectively. For the deterministic
baseline, we use greedy search. For stochastic methods, we employ top- _k_ sampling ( _k_ = 40 and
temperature = 0 _._ 7), nucleus sampling ( _p_ = 0 _._ 95), and contrastive search ( _k_ = 8 and _α_ = 0 _._ 6).
For evaluations of our proposed method on LLaMA-7B, we use _w_ = 1 _._ 5 and _k_ = 10 based on
the optimal average reward performance on the validation set. Following a similar rationale, for all
evaluations involving OPT (Zhang et al., 2022) models, we opt for values of _w_ = 2 and _k_ = 10.
Ablations on all the hyperparameters, including _k_ and _w_, will be discussed in Section 3.2.
**Evaluation metrics.** Drawing inspiration from the previous methodologies, our generation quality
evaluation leverages the following metrics.
    - **Average Reward** : This metric represents the mean of the rewards computed by the reward
model across all generations from the HH-RLHF test prompts. A higher average reward
indicates model continuations that are more closely aligned with the attributes represented
in the reward model, such as helpfulness and harmlessness. We use the same reward model
that was employed during the ARGS decoding step.
    - **Diversity** : This metric aggregates n-gram repetition rates. A higher diversity score indicates the capacity to produce texts with a broad spectrum of vocabulary. The diversity score
for a given continuation _y_ is diversity( _y_ ) = [�] _n_ [4] =2 unique n-gramstotal n-grams( _y_ ( _y_ )) [.]
    - **Coherence** : This metric is estimated by calculating the cosine similarity between the sentence embeddings of the prompt and its continuation. As in Su et al. (2022), we utilize the
pre-trained SimCSE sentence embedding model to obtain the embeddings.
3.2 RESULTS
**ARGS effectively improves the generation performance.** Figure 2 (left) shows that ARGS yields
relative improvements of **19.56** % in average reward over the greedy decoding baseline ( _w_ = 0);
thus highlighting the benefit of incorporating a reward signal during decoding. There is a noticeable
shift towards higher rewards for the generated texts by ARGS. This suggests that our method is
indeed effective in aligning the generation towards more desirable outputs. Moreover, we observe in
Figure 2 (middle) that the diversity metric for ARGS is generally better than the standard decoding
method, which indicates that our proposed method is capable of generating lexically diverse continuations. Lastly, our method maintains comparable contextual consistency under mild weight, such
as _w_ = 0 _._ 5 and _w_ = 1. However, we observe a degradation when _w_ becomes too large. This is
expected since our decoding mechanism has an inherent tradeoff between semantic coherence and
reward, emphasizing the need for a mildly chosen weight in practice. We also repeat the experiment
[*https://huggingface.co/datasets/Dahoas/full-hh-rlhf](https://huggingface.co/datasets/Dahoas/full-hh-rlhf)
Published as a conference paper at ICLR 2024
with the original non-fine-tuned LLaMA-7B model and achieve similarly improved results, which
are shown in Table 10 (Appendix C).
Figure 2: Comparison between ARGS and baseline, under greedy token selection strategy. For ARGS, we vary
the weight _w_ and report the performance measured by average reward (left), diversity (middle), and coherence
(right). For each subplot, the darker-colored bars correspond to _k_ = 40, the lighter color corresponds to _k_ = 10,
and the dotted line corresponds to the greedy baseline. Our method is relatively insensitive to the variable _k_ .
**Effect of** _w_ **and** _k_ **.** To further understand the impact of the choice of parameters _k_ and _w_, we
compare the performance of our method as the hyperparameters vary. Figure 2 depicts three main
metrics of performance, average reward score, diversity, and coherence, using the same experimental
setup as outlined earlier in Section 3.1. In Figure 2, we observe an increase in average reward as
the weighting parameter _w_ increases up to a particular point, after which it begins to decline. We
hypothesize that a higher _w_ may inadvertently favor short-term rewards, potentially undermining the
broader semantic coherence and alignment. Regarding the number of candidates _k_, the performance
variation between _k_ = 40 and _k_ = 10 is slight, suggesting that a large number of candidates may
not be essential for producing aligned generations.
**Qualitative examples.** In Table 1, we provide qualitative examples of how ARGS can steer decoded outputs to be more aligned with human preference. For the first example, the greedy approach
provides unhelpful and repetitive responses, asking multiple times about the number of strings of
lights to be connected. In contrast, the ARGS-greedy offers a comprehensive plan for setting up the
light show, suggesting types of lights, power strips, and a strategy for a test run. For the second
example, the greedy decoding yields a short and redundant query despite this information being previously provided. On the other hand, the ARGS-greedy method offers a nuanced response, offering
practical interview preparation advice such as rehearsing answers, dressing appropriately, organizing necessary documents, and even preparing questions for potential employers. See Appendix D
for additional qualitative examples.
3.3 GPT-4 EVALUATION
To address the nuanced aspects of language quality that the aforementioned metrics may not comprehensively capture, we also adopt a GPT-4-based evaluation approach for comparing the quality
of responses. As investigated in Zheng et al. (2023a), using GPT-4 proxy aligns with human evaluations over 80% of the time for quality assessments, thus offering a scalable method to approximate
human preferences. Following the methodologies in Chiang et al. (2023), we use GPT-4 as a proxy
for human evaluation by having it review and score two responses to the same prompt on a scale
from 1 to 10. We explicitly instruct the proxy to assign the score to the responses based on helpfulness, harmlessness, relevance, accuracy, and insightfulness (see the prompt template attached in
Appendix B). We randomly sample 300 prompts from the test set of HH-RLHF and compare the
response between ARGS and various decoding methods. To mitigate position bias (Zheng et al.,
2023a), we randomize the order in which we present the generated responses to GPT-4.
Table 2 presents the GPT-4 evaluation results, measured by the percentage of win-ties of our method
over the alternative decoding strategies. A higher percentage indicates that our proposed method is
more proficient in generating responses that exhibit not only contextual relevance and accuracy but
also helpfulness and harmlessness. This observation is consistent with the outcomes of the automatic
evaluation discussed in Section 3.2. For all decoding methods, we report in Table 10 (Appendix C)
the complete evaluation metrics including average reward, diversity, and coherence.
Published as a conference paper at ICLR 2024
Table 1: Comparative examples of the model using greedy and ARGS-greedy decoding. We set _w_ = 1 _._ 5 and
_k_ = 40 for ARGS.
Table 2: Comparison of between ARGS and other decoding methods based on GPT-4 evaluation. For ARGS,
we use the greedy version with _w_ = 1 _._ 5 and _k_ = 10.
**ARGS** vs. **Method** **Win-Tie** (%) _↑_
ARGS Greedy 64.33
ARGS Top- _k_ (Fan et al., 2018) 54.33
ARGS Nucleus (Holtzman et al., 2020) 55.33
ARGS Contrastive (Su et al., 2022) 62.00
3.4 FURTHER ANALYSIS
**ARGS-greedy vs. ARGS-stochastic.** In Table 3, we compare the performance using two variants
of ARGS, namely ARGS-greedy and ARGS-stochastic. For both variants, we use the same default _k_
and _w_ as specified in Section 3.1. The temperature parameter _τ_ is set to be 0.7 for ARGS-stochastic,
which follows the same configuration in popular stochastic methods such as top- _k_ sampling. In
general, we find that ARGS-greedy more effectively improves the average reward and alignment
with human preference. ARGS-stochastic can produce more diverse texts due to probabilistically
sampling from a set of top- _k_ tokens instead of deterministically selecting the most probable one.
Table 3: Comparison of ARGS-greedy and ARGS-stochastic.
**Method** **Average Reward** _↑_ **Diversity** _↑_ **Coherence** _↑_
**ARGS-greedy** 6.87 0.670 0.526
**ARGS-stochastic** 6.56 0.772 0.525
**ARGS is model- and task-agnostic.** Our proposed method, ARGS, is inherently both model and
task-agnostic, enhancing its utility across a broad array of natural language processing applications.
The primary strengths of ARGS encompass (1) its compatability with diverse model architectures,
Published as a conference paper at ICLR 2024
(2) its broad applicability across various alignment tasks, and (3) its flexibility regarding the size of
the model deployed. To elaborate, the language model and reward model do not need to have the
same size or architecture, given that the reward model is trained effectively to capture the human
preferences pertinent to a given task.
To validate this, we consider another helpful
and harmless alignment task and perform experiments on OPT-1.3b and OPT-2.7b as
base models and OPT-125m and OPT-350m
as reward models (Zhang et al., 2022). We
fine-tune base and reward models following the
methodology in Section 3.1 on the Stanford
Human Preferences (SHP) dataset (Ethayarajh
et al., 2022). The dataset consists of 349,000
training samples and 36,800 test samples and
|6<br>4<br>2|Col2|Col3|Col4|
|---|---|---|---|
|2<br>4<br>6||25m<br>OP~~T-~~|50m|
|6<br>4<br>2|Col2|Col3|Col4|
|---|---|---|---|
|2<br>4<br>6||25m<br>OP~~T-~~3|0m|
is publicly available. Each sample contains a Figure 3: Comparison between ARGS and baseline, on
prompt paired with two responses. An anno- OPT models trained with the SHP dataset. The x-axis
tation is provided to indicate which of the two indicates the reward models used to guide ARGS.
responses is more preferred. We evaluate the models on random 1,000 samples of the test set, and
the average reward is calculated by the OPT-350m reward model. As shown in Figure 3, ARGS
consistently outperforms the greedy baseline, suggesting that ARGS is model- and task-agnostic.
Additionally, we conduct a GPT-4 evaluation, comparing ARGS with the baseline PPO. Overall,
our method produces more favorable responses, achieving a win-tie rate of 72.33%.
Figure 3: Comparison between ARGS and baseline, on
OPT models trained with the SHP dataset. The x-axis
indicates the reward models used to guide ARGS.
4 DISCUSSION
**Training-time vs. decoding-time alignment.** This paper brings a novel perspective of _decoding-_
_time alignment_ to the field. While traditional alignment strategies focus on alignment and optimization during the training phase, decoding-time alignment emphasizes the pivotal role of post-training
adjustments. One feature of decoding-time alignment is its ability to adapt in the event of altering
the reward model. This omits the need to go through the exhaustive process of retraining the RL
model and enables quick realignment. Thus, the shift facilitates rapid customization of evolving
datasets and emerging needs, and ensures that models remain relevant and responsive to contemporary requirements without the need for extensive overhauls. Furthermore, our framework can be
compatible with a wide range of models, which is especially valuable in today’s rapidly changing
field of machine learning with its various model designs or sizes.
Table 4 empirically compares the performance between ARGS, Proximal Policy Optimization (PPO)
and Direct Preference Optimization (DPO) (Rafailov et al., 2023), on the SHP dataset. The PPO
model is optimized with fine-tuned OPT-1.3b as the initial language model and OPT-350m as
the reward model. Similarly, the DPO model uses the fine-tuned OPT-1.3b model as a base. Full
details of training configurations are in Appendix A. The same reward and language model are used
for ARGS, but notably without any further training. We observe that ARGSachieves a comparable
average reward as PPO, while alleviating the need for expensive RL training. Moreover, we observe
that ARGS significantly outperforms PPO and DPO in terms of diversity and coherence. Overall, the
results indicate that our approach is a competitive contender compared to the status quo approach.
Table 4: Comparison of ARGS and PPO. For ARGS, we use the greedy version with _w_ = 2 and _k_ = 10.
**Method** **Category** **Average Reward** _↑_ **Diversity** _↑_ **Coherence** _↑_
**ARGS** Decoding-based 5.98 0.322 0.390
PPO Training-based 5.88 0.247 0.264
DPO Training-based 5.65 0.096 0.372
**Computation and alignment tradeoff.** We analyze the theoretical complexity of ARGS in Section 2.3. Compared to the classic decoding methods with complexity _O_ ( _m_ [2] ), the ARGS approach
introduces only a constant factor of _k_ in its complexity which arises due to the need to consider the
top- _k_ candidate tokens at each decoding step. Empirically, we find that _k_ can be considerably small.
For example, using the OPT-2.7b base model, the generation time per response increases only by
1.9 times when comparing ARGS ( _k_ = 10) with conventional greedy decoding. Despite the slight
increase in processing time, there was a notable enhancement in reward performance by _↑_ 6.8%,
Published as a conference paper at ICLR 2024
demonstrating the existence of a reasonable tradeoff between reward optimization and inference
speed. The gap can be further reduced by employing a smaller reward model, or parallelizing the
reward computation across _k_ candidates. The feasibility is supported in Figure 3, where a smaller
reward model (such as OPT-125m) does not significantly change the performance.
5 RELATED WORKS
**Language model alignment.** Fine-tuning language models to reflect human preferences has
gained traction, with reinforcement learning from human feedback (RLHF) offering a direct route.
Signals from external reward models that act as human proxies are used to refine agents through iterative trials under different RLHF frameworks (Christiano et al., 2017; Ziegler et al., 2019; Stiennon
et al., 2020; Lee et al., 2021; Nakano et al., 2022; Snell et al., 2023). One notable approach is to utilize proximal policy optimization (Askell et al., 2021; Ouyang et al., 2022; Bai et al., 2022; Glaese
et al., 2022). However, recognizing the challenges posed by the unstable and resource-demanding
nature of RL, researchers have also explored supervised fine-tuning methods. For example, Liu et al.
(2023) fine-tune the model using prompts that encompass both desirable and undesirable answers.
Rafailov et al. (2023), on the other hand, take a distinctive route by modeling the language model as
a Bradley-Terry model, bypassing the need for conventional reward modeling. Yuan et al. (2023);
Song et al. (2023) introduce frameworks that are designed to rank multiple responses, adding to
the spectrum of alignment methods. Dong et al. (2023) introduce an approach in which rewards
are harnessed to curate suitable training sets for the fine-tuning of language models. Rennie et al.
(2017) investigate reinforcement learning training to improve image captioning on LSTM and CNN
architectures. Notably, ARGS diverges from these training-based approaches, by providing a new
decoding-time framework to align language models without requiring expensive RL training.
**Language model decoding.** A language model (LM) is a machine learning model trained to predict the probability distribution _p_ ( _**x**_ ) across a text sequence of variable length _**x**_ = _{x_ 1 _, . . ., x|_ _**x**_ _|}_ .
The probability _p_ ( _xt |_ _**x**_ _<t_ ) denotes the likelihood of predicting the next token _xt_, given the context
_**x**_ _<t_ = _{x_ 1 _, . . ., xt−_ 1 _}_ . Leveraging this likelihood, various decoding strategies have been proposed
to generate a text continuation of the context, which can be categorized into either _deterministic_ or
_stochastic_ methods (Ippolito et al., 2019). Notable deterministic methods include the greedy beam
search, and contrastive search (Su et al., 2022). They select the text continuation with the highest
probability or scoring criteria. Popular stochastic methods include top- _k_ sampling (Fan et al., 2018)
and nucleus sampling (Holtzman et al., 2020). Top- _k_ sampling selects the _k_ tokens with the highest
likelihood, renormalizes the probabilities, and then samples from this set, while nucleus sampling
selects the smallest set of tokens such that their cumulative probability exceeds a certain threshold.
Unlike deterministic methods, stochasticity in these methods could cause the semantic meaning of
the sampled text to diverge from or even contradict the human-written prefix (Basu et al., 2021).
**Guided decoding.** Our works distinguishes itself in the token-level guided decoding literature by
using a reward model that guides generation at the token level, rather than focusing on step-level
verifiers that typically emphasize sentence-level analysis (Welleck et al., 2022; Uesato et al., 2022;
Lightman et al., 2023; Krishna et al., 2022; Li et al., 2023b; Khalifa et al., 2023; Xie et al., 2023;
Yao et al., 2023). While token-level guided decoding have been explored in the past (Dathathri et al.,
2020; Krause et al., 2021; Yang & Klein, 2021; Lu et al., 2021; Chaffin et al., 2022; Liu et al., 2021;
Li et al., 2023a), they have not connected language decoding directly to the alignment problem of our
interest, especially in the context of utilizing a reward model. Concurrent to our work, Deng & Raffel
(2023) use a decoding process that includes a reward model, however, they utilize a unidirectional
reward model that is trained using a cumulative squared error loss. In contrast, ARGS employs
a reward model based on a pairwise ranking loss to score preferred and nonpreferred responses,
which is consistent with the existing RLHF framework.
6 CONCLUSION
The ARGS framework offers a novel decoding-time approach to alignment, addressing the limitations of traditional methods. By formulating alignment as a decoding-stage problem and leveraging
reward signals, our approach reduces the need for resource-intensive RL training. The consistent
performance gains achieved emphasize the potential of ARGS, indicating a promising trajectory toward creating more flexibly aligned language models. Overall, ARGS framework not only improves
alignment performance but also paves the way for broader applications of alignment in the future.
Due to the space limit, we discuss limitations and future work in Appendix E.
Published as a conference paper at ICLR 2024
**Ethics statement.** The ability to adapt and align models post-training means that smaller institutions or businesses without the capacity for large-scale training can still effectively tailor pre-trained
models to meet their specific needs. This can potentially level the playing field, allowing those with
limited computational resources to benefit from state-of-the-art models without incurring significant
costs. Also, the compatibility of our method with different reward models extends its applicability
across various domains and industries. This can accelerate the adoption of machine learning solutions in fields where resource constraints or rapidly changing data are prevalent. Our study does not
involve human subjects or violation of legal compliance. Code will be released publicly to facilitate
reproducibility and broader applicability.
REFERENCES
Anonymous. The trickle-down impact of reward inconsistency on RLHF. In _Submitted to The_
_Twelfth International Conference on Learning Representations_, 2023. under review.
[Anthropic. https://www.anthropic.com/index/introducing-claude, 2023.](https://www.anthropic.com/index/ introducing-claude)
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones,
Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack
Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A general language assistant as a laboratory for alignment. _arXiv preprint
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn
Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson
Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson,
Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback.
_arXiv preprint
Sourya Basu, Govardana Sachitanandam Ramachandran, Nitish Shirish Keskar, and Lav R. Varshney. Mirostat: a neural text decoding algorithm that directly controls perplexity. In _International_
_Conference on Learning Representations_, 2021.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier
Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel
Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul
Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud,
Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Bıyık,
Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell. Open problems
and fundamental limitations of reinforcement learning from human feedback. _arXiv preprint_
_
Antoine Chaffin, Vincent Claveau, and Ewa Kijak. PPL-MCTS: constrained textual generation
through discriminator-guided MCTS decoding. In Marine Carpuat, Marie-Catherine de Marneffe, and Iván Vladimir Meza Ruíz (eds.), _Proceedings of the 2022 Conference of the North_
_American Chapter of the Association for Computational Linguistics: Human Language Technolo-_
_gies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022_, pp. 2953–2967. Association for
Computational Linguistics, 2022.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng,
Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An
[open-source chatbot impressing gpt-4 with 90%* chatgpt quality. https://vicuna.lmsys.](https://vicuna.lmsys.org)
[org, 2023.](https://vicuna.lmsys.org)
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep
reinforcement learning from human preferences. _Advances in Neural Information Processing_
_Systems_, 2017.
Published as a conference paper at ICLR 2024
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text
generation. In _International Conference on Learning Representations_, 2020.
[Microsoft DeepSpeed. https://github.com/microsoft/DeepSpeedExamples, 2023.](https://github.com/microsoft/DeepSpeedExamples)
Haikang Deng and Colin Raffel. Reward-augmented decoding: Efficient controlled text generation
with a unidirectional reward model. In _Proceedings of the 2023 Conference on Empirical Methods_
_in Natural Language Processing_, 2023.
Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik
Narasimhan. Toxicity in chatgpt: Analyzing persona-assigned language models. _arXiv preprint_
_
Shizhe Diao, Rui Pan, Hanze Dong, Ka Shun Shum, Jipeng Zhang, Wei Xiong, and Tong Zhang.
Lmflow: An extensible toolkit for finetuning and inference of large foundation models. _arXiv_
_preprint
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum,
and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment.
_arXiv preprint
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with
_V_ -usable information. In _International Conference on Machine Learning_, 2022.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In _Association_
_for Computational Linguistics_, 2018.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In _Association for Compu-_
_tational Linguistics_, 2020.
Amelia Glaese, Nat McAleese, Maja Tr˛ebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham,
Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth
Dathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Soˇna
Mokrá, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young, Iason Gabriel, William
Isaac, John Mellor, Demis Hassabis, Koray Kavukcuoglu, Lisa Anne Hendricks, and Geoffrey
Irving. Improving alignment of dialogue agents via targeted human judgements. _arXiv preprint_
_
Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymetman. Aligning foundation models for language with preferences through $f$-divergence minimization. In _ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation_
_Models_, 2023.
[Google. Bard. https://bard.google.com/, 2023.](https://bard.google.com/)
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger.
Deep reinforcement learning that matters. _Proceedings of the AAAI Conference on Artificial_
_Intelligence_, 2017.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text
degeneration. In _International Conference on Learning Representations_, 2020.
Daphne Ippolito, Reno Kriz, Maria Kustikova, João Sedoc, and Chris Callison-Burch. Comparison of diverse decoding methods from conditional language models. In _Proceedings of the 57th_
_Annual Meeting of the Association for Computational Linguistics_, 2019.
Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and
Robert McHardy. Challenges and applications of large language models. _arXiv preprint_
_
Published as a conference paper at ICLR 2024
Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang. Grace:
[Discriminator-guided chain-of-thought reasoning. 2023. URL https://openreview.net/](https://openreview.net/forum?id=2MiTZxLFA9)
[forum?id=2MiTZxLFA9.](https://openreview.net/forum?id=2MiTZxLFA9)
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard
Socher, and Nazneen Fatema Rajani. GeDi: Generative discriminator guided sequence generation.
In _Findings of the Association for Computational Linguistics: EMNLP 2021_, 2021.
Kalpesh Krishna, Yapei Chang, John Wieting, and Mohit Iyyer. Rankgen: Improving text generation
with large ranking models. In _Empirical Methods in Natural Language Processing_, 2022.
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement
learning via relabeling experience and unsupervised pre-training. In _International Conference on_
_Machine Learning_, 2021.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke
Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization.
In _Association for Computational Linguistics_, 2023a.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making
language models better reasoners with step-aware verifier. In _Proceedings of the 61st Annual_
_Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pp. 5315–
5333, 2023b.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy
Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by
step. _ArXiv_, abs/2305.20050, 2023. [URL https://api.semanticscholar.org/](https://api.semanticscholar.org/CorpusID:258987659)
[CorpusID:258987659.](https://api.semanticscholar.org/CorpusID:258987659)
Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith,
and Yejin Choi. Dexperts: Decoding-time controlled text generation with experts and anti-experts.
In _Proceedings of Annual Meeting of the Association for Computational Linguistics_, 2021.
Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Chain of hindsight aligns language models with
feedback. _arXiv preprint
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In _International Confer-_
_ence on Learning Representations_, 2019.
Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.
NeuroLogic decoding: (un)supervised neural text generation with predicate logic constraints. In
_Proceedings of the 2021 Conference of the North American Chapter of the Association for Compu-_
_tational Linguistics: Human Language Technologies_ . Association for Computational Linguistics,
2021.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher
Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou,
Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt:
Browser-assisted question-answering with human feedback. _arXiv preprint
2022.
Richard Ngo, Lawrence Chan, and Sören Mindermann. The alignment problem from a deep learning
perspective. _arXiv preprint
OpenAI. Gpt-4 technical report. _arXiv preprint
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong
Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike,
and Ryan Lowe. Training language models to follow instructions with human feedback. _Advances_
_in Neural Information Processing Systems_, 2022.
Published as a conference paper at ICLR 2024
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and
Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.
In _Advances in Neural Information Processing Systems_, 2023.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In _ACM SIGKDD_,
2020.
Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical
sequence training for image captioning. In _Proceedings of the IEEE Conference on Computer_
_Vision and Pattern Recognition (CVPR)_, 2017.
Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural
language generation with implicit language q learning. _arXiv preprint
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang.
Preference ranking optimization for human alignment. _arXiv preprint
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford,
Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. _Advances_
_in Neural Information Processing Systems_, 2020.
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive
framework for neural text generation. _Advances in Neural Information Processing Systems_, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée
Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation
language models. _arXiv preprint
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher,
Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy
Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn,
Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel
Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee,
Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra,
Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi,
Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh
Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen
Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic,
Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models.
_arXiv preprint
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia
Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and
outcome-based feedback, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,
Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. _Advances in Neural Informa-_
_tion Processing Systems_, 2017.
Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang,
Xin Jiang, and Qun Liu. Aligning large language models with human: A survey. _arXiv preprint_
_
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol
Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models.
_Transactions on Machine Learning Research_, 2022.
Published as a conference paper at ICLR 2024
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang,
Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will
Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne
Hendricks, William Isaac, Sean Legassick, Geoffrey Irving, and Iason Gabriel. Ethical and social
risks of harm from language models. _arXiv preprint
Sean Welleck, Jiacheng Liu, Ximing Lu, Hannaneh Hajishirzi, and Yejin Choi. Naturalprover:
Grounded mathematical proof generation with language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), _Advances in Neural Information Processing_
_Systems_, 2022.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith,
Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for
language model training. In _Advances in Neural Information Processing Systems_, 2023.
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min-Yen Kan, Junxian He, and Qizhe Xie.
Decomposition enhances reasoning via self-evaluation guided decoding, 2023.
Kevin Yang and Dan Klein. FUDGE: Controlled text generation with future discriminators. In
_Proceedings of the 2021 Conference of the North American Chapter of the Association for Com-_
_putational Linguistics: Human Language Technologies_, 2021.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik
Narasimhan. Tree of Thoughts: Deliberate problem solving with large language models. 2023.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf:
Rank responses to align language models with human feedback without tears. _arXiv preprint_
_
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer
language models. _arXiv preprint
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang,
Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.
Judging LLM-as-a-judge with MT-bench and chatbot arena. In _Conference on Neural Information_
_Processing Systems Datasets and Benchmarks Track_, 2023a.
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin,
Qin Liu, Yuhao Zhou, Limao Xiong, Lu Chen, Zhiheng Xi, Nuo Xu, Wenbin Lai, Minghao Zhu,
Cheng Chang, Zhangyue Yin, Rongxiang Weng, Wensen Cheng, Haoran Huang, Tianxiang Sun,
Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, and Xuanjing Huang. Secrets of rlhf in large language
models part i: Ppo. _arXiv preprint
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul
Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. _arXiv_
_preprint
Published as a conference paper at ICLR 2024
APPENDIX
A EXPERIMENTATION DETAILS
**Software and hardware.** We conduct our experiments on servers equipped with NVIDIA RTX
A6000 GPUs (48GB VRAM) and NVIDIA A100 GPUs (80GB VRAM). We use Ubuntu 22.04.2
LTS as the operating system, with NVIDIA CUDA Toolkit version 11.8 and cuDNN 8.9. All experiments are implemented in Python 3.11.4 using the PyTorch 1.12.1 framework.
**Training LLaMA-7B on HH-RLHF.** We employ the LMFlow (Diao et al., 2023) toolkit to facilitate the training of the LLaMA-7B model on the HH-RLHF dataset. Following the training scheme
in Dong et al. (2023), we use the AdamW (Loshchilov & Hutter, 2019) optimizer in conjunction with
DeepSpeed ZeRO stage 3 (Rasley et al., 2020). The training was performed on the entire training
split. The training parameters are summarized in Table 5.
Table 5: Summary of training hyperparameters for supervised fine-tuning and reward modeling for
LLaMA-7B models.
Parameters Value
Supervised fine-tuning
Reward modeling
Number of epochs 1
Learning rate 2 _·_ 10 _[−]_ [5]
Learning rate decay Linear decay
Batch size 32
Floating point format fp16 (Half-precision)
Block size 512
Number of epochs 1
Learning rate 5 _·_ 10 _[−]_ [6]
Learning rate decay Linear decay
Batch size 16
Floating point format fp16 (Half-precision)
Block size 512
**Training OPT models on the Stanford Human Preferences (SHP) dataset.** For the training
of all OPT-family models on the SHP dataset, we utilize the DeepSpeed-Chat (DeepSpeed, 2023)
repository. We adopt the training scheme proposed by Ouyang et al. (2022), wherein the reward
model is trained based on the supervised fine-tuned model. Their default configurations were followed: models undergo supervised fine-tuning on 20% of the training dataset, and reward modeling on the subsequent 40%. We format the response pairs by prefixing the prompt with Human:
and prepending Assistant: to the model’s responses, following the methodology outlined
in DeepSpeed (2023). These training parameters are consistently applied across all model sizes
(OPT-125m, OPT-350m, OPT-1.3b, and OPT-2.7b) and are detailed in Table 6.
**Training configurations for PPO.** For all model training with reinforcement learning with human
feedback through proximal policy optimization, we adopt the DeepSpeed-Chat (DeepSpeed, 2023)
repository. We follow their default configurations which are detailed in Table 7.
**Training configurations for DPO.** For experiments on DPO, we use the TRL (transformer reinforcement learning) repository from Huggingface in conjunction with the DPOTrainer module. The
configuration values are detailed in Table 8.
B GPT-4 EVALUATION DETAILS
Table 9 presents the prompts and responses usage in our GPT-4 evaluation. Each GPT-4 request
comprises both a system and a user prompt. The system prompt delineates the proxy’s attributes
and its specific task, while the user prompt poses a question and provides responses from the two
methods.
Published as a conference paper at ICLR 2024
Table 6: Summary of training hyperparameters for supervised fine-tuning and reward modeling for
OPT-family models.
Parameters Value
Supervised fine-tuning
Reward modeling
Number of epochs 16
Learning rate 9 _._ 65 _·_ 10 _[−]_ [6]
Learning rate decay Cosine
Batch size 64
Gradient accumulation steps 1
Maximum sequence length 512
DeepSpeed Zero stage 2
Weight decay 0.0
Number of padding tokens at the beginning of the input 1
Number of epochs 1
Learning rate 5 _·_ 10 _[−]_ [5]
Learning rate decay Linear decay
Batch size 32
Gradient accumulation steps 1
Maximum sequence length 512
DeepSpeed Zero stage 2
Weight decay 0.1
Number of padding tokens at the beginning of the input 1
Table 7: Summary of training hyperparameters for proximal policy optimization (PPO).
Parameters Value
Number of training epochs 1
Number of PPO epochs 1
Generation batches 1
Actor model learning rate 9 _._ 65 _·_ 10 _[−]_ [6]
OPT-1.3b
Critic model learning rate 5 _·_ 10 _[−]_ [5]
Learning rate decay Cosine
Batch size 32
Gradient accumulation steps 1
Maximum sequence length 256
DeepSpeed Zero stage 2
Number of warmup steps 100
Enable EMA checkpoint for the model
Weight decay 0.0
Number of padding tokens at the beginning of the input 1
C COMPARISON WITH ALL BASELINES
Table 10 provides a comprehensive comparison of ARGS, both in its greedy and stochastic variants, with various baseline methods, including vanilla greedy decoding, top- _k_ sampling, nucleus
sampling, and contrastive search. We evaluate these decoding strategies using both the base model
and the fined-tuned version. It is noteworthy that even when applied to the non-finetuned model,
ARGS exhibits a substantial improvement in average reward, surpassing the performance of the best
baseline method by a margin of _↑_ **27%** . Moreover, in the fine-tuned version of ARGS, the method
outperforms the best baseline by _↑_ **18%** .
These results underscore the effectiveness of ARGS, both in its greedy and stochastic variants, in
enhancing the performance of language generation, surpassing the performance of well-established
baseline methods in both non-finetuned and fine-tuned scenarios.
Published as a conference paper at ICLR 2024
Table 8: Summary of training hyperparameters for Direct Policy Optimization (DPO).
Parameters Value
OPT-1.3b
Number of training epochs 1
Learning rate 5 _·_ 10 _[−]_ [5]
Learning rate decay Linear decay
Batch size 32
Gradient accumulation steps 1
Maximum sequence length 512
Weight decay 0.1
Beta 0.1
Table 9: Sample prompt for the GPT-4 evaluation. Text highlighted in **orange** represents the prompt,
while text in **blue** represents the responses under comparison.
D ADDITIONAL QUALITATIVE EXAMPLES
In Table 11, we provide additional qualitative examples of how ARGS can steer decoded outputs to
be more aligned with human preference. See Section 3.1 for models and hyperparameters set up for
LLaMA-7B.
E LIMITATIONS AND FUTURE WORK.
For our current evaluations, we follow the standard and commonly used benchmarks in alignment literature. In particular, HH-RLHF from Anthropic and Stanford Human Preferences (SHP) are among
the largest and publicly available datasets for alignment research. These tasks allow us to draw
comparisons with existing approaches more easily and directly. Nonetheless, we acknowledge the
potential value in assessing more intricate tasks, such as those involving multi-step reasoning. We
maintain a keen interest in extending our research to encompass more complex tasks in subsequent
studies. We are also interested in exploring different various reward modeling approaches (Anony
Published as a conference paper at ICLR 2024
Table 10: Comparison of performance across various decoding methods for models with and without
fine-tuning.
**Base Model** **Decoding Method** **Average Reward** _↑_ **Diversity** _↑_ **Coherence** _↑_

Maxim Khanov [1] , Jirayu Burapacheep [2] , Yixuan Li [1]

Maxim Khanov
Jirayu Burapacheep
Yixuan Li


Greedy 3.981 0.567 0.426
Top- _k_ 3.757 0.679 0.463
Nucleus 3.313 0.743 0.455
Contrastive 3.823 0.668 0.352
**ARGS-greedy** 5.026 0.611 0.456
**ARGS-stochastic** 4.787 0.700 0.463
Greedy 5.732 0.619 0.544
Top- _k_ 5.476 0.783 0.551
Nucleus 5.117 0.813 0.542
Contrastive 5.245 0.824 0.470
**ARGS-greedy** 6.872 0.670 0.526
**ARGS-stochastic** 6.558 0.772 0.525
mous, 2023; Go et al., 2023; Wu et al., 2023), as we have observed that a higher-quality reward
model results in enhanced generation quality.
Published as a conference paper at ICLR 2024
Table 11: Comparative examples of the model using greedy and ARGS-greedy decoding strategies. For ARGS,
we use _w_ = 1 _._ 5 and _k_ = 10.


**Shangmin Guo** **[†]** [ * 1] **Biao Zhang** [* 2] **Tianlin Liu** **[†]** [ 3] **Tianqi Liu** [2] **Misha Khalman** [2] **Felipe Llinares** [2]
**Alexandre Ramé** **[†]** [ 2] **Thomas Mesnard** [2] **Yao Zhao** [2] **Bilal Piot** [2] **Johan Ferret** [2] **Mathieu Blondel** [2]

Direct Language Model Alignment from Online AI Feedback

Shangmin Guo
Biao Zhang
Tianlin Liu
Tianqi Liu
Misha Khalman
Felipe Llinares
Alexandre Rame
Thomas Mesnard
Yao Zhao
Bilal Piot
Johan Ferret
Mathieu Blondel


Direct alignment from preferences (DAP) methods, such as DPO, have recently emerged as efficient alternatives to reinforcement learning from
human feedback (RLHF), that do not require a
separate reward model. However, the preference
datasets used in DAP methods are usually collected ahead of training and never updated, thus
the feedback is purely offline. Moreover, responses in these datasets are often sampled from
a language model distinct from the one being
aligned, and since the model evolves over training,
the alignment phase is inevitably off-policy. In
this study, we posit that online feedback is key and
improves DAP methods. Our method, online AI
feedback (OAIF), uses an LLM as annotator: on
each training iteration, we sample two responses
from the current model and prompt the LLM annotator to choose which one is preferred, thus
providing online feedback. Despite its simplicity,
we demonstrate via human evaluation in several
tasks that OAIF outperforms both offline DAP
and RLHF methods. We further show that the
feedback leveraged in OAIF is easily controllable,
via instruction prompts to the LLM annotator.


To maximise the benefits of large language models (LLMs)
to society, it is important to align them with human expectations and values (Ouyang et al., 2022; Bai et al., 2022a;
Bubeck et al., 2023). The first method introduced for
alignment was reinforcement learning from human feedback (RLHF, Christiano et al., 2017; Stiennon et al., 2020),
which trains a reward model (RM) from pairwise preferences and then optimises a policy against the RM via reinforcement learning (RL). More recently, direct alignment
*Equal contribution †Work done during an internship at
Google DeepMind 1University of Edinburgh 2Google DeepMind [3] University of Basel. Correspondence to: Shangmin
Guo <s.guo@ed.ac.uk>, Biao Zhang <biaojiaxing@google.com>,
Mathieu Blondel <mblondel@google.com>.
_Preprint._
from preferences (DAP) methods have emerged as popular
alternatives to RLHF, such as direct preference optimisation (DPO, Rafailov et al., 2023), sequence likelihood calibration with human feedback (SLiC, Zhao et al., 2023),
and identity policy optimisation (IPO, Azar et al., 2023).
In contrast to RLHF, the DAP methods directly update the
language model (a.k.a. policy) _π_ _**θ**_ using pairwise preference
data, making the alignment simpler, more efficient and more
stable (Rafailov et al., 2023).
However, the preference datasets used in DAP methods are
often collected ahead of training and the responses in the
dataset are usually generated by different LLMs. Thus, the
feedback in DAP methods is usually purely offline, as _π_ _**θ**_
cannot get feedback on its own generations over training.
This is problematic because of the significant distribution
shift between the policy that generated the dataset and the
policy being aligned: we train on the distribution induced
by _ρ_ but evaluate on the distribution induced by _π_ _**θ**_ in the
end. In contrast, in RLHF, the RM provides online feedback
to generations from _π_ _**θ**_ during the RL step. This practice
leads to on-policy learning, which was shown to improve
exploration and overall performance (Lambert et al., 2022).
Inspired by RL from AI feedback (RLAIF) (Bai et al.,
2022b; Lee et al., 2023), we hereby propose Online AI
Feedback (OAIF) for DAP methods. Our method inherits
both the practical advantages of DAP methods and the online nature of RLHF. Specifically, when aligning an LLM
policy _π_ _**θ**_, we follow a three-step procedure: 1) we sample
two responses to a prompt from the current policy _π_ _**θ**_ ; 2) we
obtain online feedback over the two responses by prompting an LLM to mimic human preference annotation; 3) we
use this online feedback to update the model _π_ _**θ**_ through
standard DAP losses. Our approach is depicted in Figure 1.
Unlike methods proposed by Xu et al. (2023); Liu et al.
(2023); Xiong et al. (2023), OAIF skips the RM training,
and directly extracts the preference from an LLM.
To show the effectiveness of our proposal, we perform
an extensive empirical comparison between OAIF, existing offline DAP methods and RLHF methods. Our
experimental protocol uses both AI and human evaluation
on standard LLM alignment tasks: TL;DR (Ziegler
et al., 2019), Anthropic Helpfulness and


Figure 1: **Summary of the proposed online AI feedback (OAIF) approach for making direct alignment from**
**preferences (DAP) methods online and on-policy** . Given an input prompt _**x**_, two responses _**y**_ [1] and _**y**_ [2] are first sampled
from the current language model _π_ _**θ**_ _t_, then labelled as _**y**_ [`] and _**y**_ [´] by the LLM annotator. The language model parameters
are then updated using the objective function of DAP methods.
Harmlessness (Bai et al., 2022a). To summarise, we
make the following contributions.
  - We demonstrate the effectiveness and generality of
OAIF for turning offline DAP methods (DPO, IPO,
SLiC) into online methods. Our human evaluation
shows that the average win rate of online DAP methods
(DPO, IPO, SLiC) over offline versions of the same
methods is „66%.
  - We confirm the usefulness of making DAP methods online: human raters favour DPO with OAIF (thus, online
DPO) over SFT baseline, RLHF and RLAIF 58 _._ 00%
of time on the TL;DR task in 4-way comparisons.
  - We demonstrate the controllability of the LLM annotator, by injecting specific instructions into the prompts.
We use response length as a test-bed. By asking the
LLM annotator to prefer shorter responses, the average
length of responses from the aligned policy is significantly shortened from „120 to „40, while its quality
is still improved over the SFT baseline.


**Pairwise preference collection** . Current methods for LLM
alignment first collect a dataset of pairwise preferences, as
follows. A prompt _**x**_ is sampled from a prompt distribution _pX_, then two distinct responses _**y**_ [1] and _**y**_ [2] are sampled
independently from an existing LLM _ρ_ . Then, human (Christiano et al., 2017) or AI annotators (Lee et al., 2023) rank
the responses, yielding a preferred response _**y**_ [`] and a less
preferred one _**y**_ [´] . With some abuse of notation, we assume
that there exists a function that uniquely maps p _**y**_ [1] _,_ _**y**_ [2] q to
p _**y**_ [`] _,_ _**y**_ [´] q, and we will therefore write p _**y**_ [`] _,_ _**y**_ [´] q „ _ρ_ p¨| _**x**_ q.
A preference dataset D “ tp _**x**_ _i,_ _**y**_ _i_ [`] _[,]_ _**[ y]**_ _i_ [´][qu] _i_ _[N]_ “1 [is then con-]
structed by repeating the above process _N_ times.
**Direct alignment from preference (DAP) methods.** DAP
methods directly update the target policy _π_ _**θ**_ from the preference pairs p _**y**_ [`] _,_ _**y**_ [´] q. The loss functions for the three main
DAP methods investigated in this work are summarised
below. They take the form _ℓ_ p _**x**_ _,_ _**y**_ [`] _,_ _**y**_ [´] _,_ _**θ**_ q for a prompt
_**x**_ „ _pX_, a response pair p _**y**_ [`] _,_ _**y**_ [´] q „ _ρ_ p¨| _**x**_ q and model
parameters _**θ**_ .
- DPO loss:
where _π_ _**θ**_ 0 is the SFT baseline used as reference, _σ_ is the
logistic function, and _β_ is a scalar hyperparameter. We emphasise once again that p _**y**_ [`] _,_ _**y**_ [´] q are sampled from _ρ_ p¨| _**x**_ q,
not from _π_ _**θ**_ _t_ p¨| _**x**_ q, as this will be the key difference with
the online variant we propose in the next section. One
advantage of these loss functions is that their gradients
ˆ
´ log _σ_ _β_ log _[π]_ _**[θ]**_ [p] _**[y]**_ [`][|] _**[x]**_ [q] _[π]_ _**[θ]**_ [0][p] _**[y]**_ [´][|] _**[x]**_ [q]
_π_ _**θ**_ 0p _**y**_ [`] | _**x**_ q _π_ _**θ**_ p _**y**_ [´] | _**x**_ q
˙
(1)
- IPO loss:
ˆlog ˆ _π_ _**θ**_ p _**y**_ `| _**x**_ q _π_ _**θ**_ 0p _**y**_ ´| _**x**_ q
_π_ _**θ**_ p _**y**_ [´] | _**x**_ q _π_ _**θ**_ 0p _**y**_ [`] | _**x**_ q
- SLiC loss:
˙ ´ [1]
2 _β_
˙2
(2)
max ˆ0 _,_ 1 ´ _β_ log ˆ _π_ _**θ**_ p _**y**_ `| _**x**_ q _π_ _**θ**_ 0p _**y**_ ´| _**x**_ q
_π_ _**θ**_ p _**y**_ [´] | _**x**_ q _π_ _**θ**_ 0p _**y**_ [`] | _**x**_ q
˙˙
(3)


No RM On-policy Online
Method
needed generation feedback
Offline DPO
✓ ✗ ✗
(Rafailov et al., 2023)
Offline IPO
✓ ✗ ✗
(Azar et al., 2023)
Offline SLiC
✓ ✗ ✗
(Zhao et al., 2023)
RSO
✗ ✓ ✓
(Liu et al., 2023)
Iterative DPO
✗ ✓ ✓
(Xu et al., 2023)
OAIF (proposed) ✓ ✓ ✓
Figure 2: **Illustration of the distribution shift problem.**
The responses ( _**y**_ 1 _,_ _**y**_ 2) sampled from the current model _π_ _**θ**_ _t_
differ from preference dataset responses ( _**y**_ [`] _,_ _**y**_ [´] ) sampled
from _ρ_, as _ρ_ ‰ _π_ _**θ**_ _t_ . Two independent distribution shifts
can occur: an initial distribution shift ( _ρ_ ‰ _π_ _**θ**_ 0) and a
gradual distribution shift ( _π_ _**θ**_ 0 ‰ _π_ _**θ**_ _t_ ) during the alignment
procedure.
_∇_ _**θ**_ _ℓ_ p _**x**_ _,_ _**y**_ [`] _,_ _**y**_ [´] _,_ _**θ**_ q can be computed exactly in an efficient
way. In contrast, because the loss function used in RLHF
involves an expectation over the space of responses (Ziegler
et al., 2019), policy gradient methods are typically used to
obtain an unbiased estimate of the gradient and a value function is typically used to reduce the variance, which requires
storing an additional model in memory.
**Offline feedback** . In most real-world applications, due
to the financial cost and complexity of collecting pairwise
preferences from human annotators, the preference dataset
D is usually collected ahead of aligning a language model
_π_ _**θ**_ and kept fixed throughout training. Obtaining online
preferences on new responses is usually not feasible, as
there is no human-in-the-loop. Using a fixed dataset D
makes all preference data _offline_, which means the policy [1]
_π_ _**θ**_ cannot get feedback on its own generations on-the-fly
over the alignment procedure. It is worth mentioning that
the RL step in RLHF and RLAIF is _online_ as the training
data is acquired interactively. See Appendix A.1 for an
in-depth discussion on online vs. offline feedback.
**Off-policy learning** . Beyond the offline feedback problem
illustrated above, aligning an LLM policy _π_ _**θ**_ with DAP
methods on a pre-collected dataset D also yields a distribution shift between the generation from the policy _ρ_ and the
policy _π_ _**θ**_ _t_ at each time step _t_ . This makes the alignment
_off-policy_ as _π_ _**θ**_ _t_ ‰ _ρ_ and _π_ _**θ**_ _t_ keeps evolving over learning.
1In this work, we use language model and policy interchangeably to refer to the model _π_ _**θ**_ being aligned.
Table 1: **Comparison between OAIF (proposed) and ex-**
**isting DAP methods**, with or without a separate RM. Technically, training RMs on pre-collected preference data still
suffers from the distribution shift problem, as RMs cannot
get feedback for responses from the model _π_ _**θ**_ _t_ .
This shift problem is illustrated in Figure 2. We also provide
an empirical verification of this problem in Appendix B. In
DPO, this problem is tackled by supervised finetuning _π_ _**θ**_ on
D so that _π_ _**θ**_ 0 « _ρ_, but the off-policy issue remains during
alignment as _π_ _**θ**_ _t_ gradually departs from _π_ _**θ**_ 0 . Thanks to the
_online_ nature of RL, RL methods are also _on-policy_, as the
responses used to update _π_ _**θ**_ _t_ are all sampled from it. See
Appendix A.2 for more details on on-policy vs. off-policy
learning in LLMs.
**RM-based online feedback for DAP methods** . To avoid
the distribution shifts arising when aligning LLMs with
offline DAP methods on a given dataset D, an intuitive
and straightforward solution is to introduce an RM to provide online feedback. Liu et al. (2023) proposed RSO, a
method that uses an RM to perform rejection sampling in
order to sample from the optimal policy, which improved
the alignment compared to offline DAP baselines. Besides,
pseudo-labelling the generations from _π_ _**θ**_ _t_ by RMs can also
be helpful, as done in the Iterative DPO method (Xu et al.,
2023) and the West-of-N method (Pace et al., 2024). Although the aforementioned RM-based methods make the
alignment of a policy online and on-policy, the distribution
shift problem still exists when training the RM. More specifically, the RM is trained on the preference dataset D „ _ρ_,
but used to annotate preference over responses from _π_ _**θ**_ _t_ at
training step _t_, where _π_ _**θ**_ ‰ _ρ_ . Therefore, RM-based online
feedback cannot fully avoid distribution shift issues.
**LLM-based online feedback for DAP methods** . The
method we propose next, “Online AI Feedback” (OAIF),


consists in using an LLM as an online annotator. Our
method relies on the observation that LLMs can approximate well human labelling and can generate reliable preferences over responses (Lee et al., 2023). In recent concurrent
work, Yuan et al. (2024) proposed a “self-rewarding” approach, in which the policy being aligned provides online
feedback to itself. In comparison, OAIF can leverage feedback from any LLM, including ones stronger than the LLM
being aligned. Swamy et al. (2024) also concurrently investigates the importance of online preference, but still relying
on RMs.
In Table 1, we summarise the characteristics of OAIF and
of the existing offline and online DAP methods.


**Bridging the gap** . As we saw, DAP methods are simple,
do not require a separate RM, but they use preference data
pre-collected offline. On the other hand, RLHF methods
interact online with the language model being aligned, but
they require policy gradient techniques to obtain an unbiased
gradient estimate and a value function to reduce the variance.
To bridge the gap between these two families of methods, we
propose a simple yet effective way to make DAP methods
online.
As pointed out by Ziegler et al. (2019), online data collection
is crucial for aligning language models. To solve the aforementioned offline problem in DAP methods, we propose
to collect preferences on-the-fly for responses generated by
the language model being aligned. Naturally, using human
feedback would be prohibitively expensive. Prior studies
have shown that AI feedback is a reliable and effective
approximation to human labellers, especially for pairwise
preference labelling (Lee et al., 2023). We therefore propose to use an LLM as online annotator, in order to collect
the preference over pairs of responses, sampled from _π_ _**θ**_ _t_
on-the-fly during its alignment. We refer to the proposed
approach as **OAIF**, which stands for online AI feedback.
**Proposed algorithm** . An overview of OAIF is given in Figure 1, and a more formal description is provided in Algorithm 1 (for simplicity, we use batches of size 1). Given a
prompt _**x**_, sampling _**y**_ [1] _,_ _**y**_ [2] from _π_ _**θ**_ _t_ p¨| _**x**_ q ensures _on-policy_
learning. Prompting the annotating LLM to obtain _**y**_ [`] _,_ _**y**_ [´]
ensures _online_ learning. We emphasise that the approach is
general and works with any differentiable DAP loss function
_ℓ_ p _**x**_ _,_ _**y**_ [`] _,_ _**y**_ [´] _,_ _**θ**_ q.
**Gradient computation** . An important technical detail of
online DAP methods is that _**θ**_ is involved in both the response sampling and in the DAP loss function. In contrast,
_**θ**_ is involved only in the loss for offline DAP methods and
only in the sampling for RLHF methods. In addition, using
OAIF, the sampled responses go through an LLM annotator
**Algorithm 1** Online AI Feedback (OAIF) for Direct Alignment from Preference (DAP) methods
**Input:** Number of training steps _T_
Prompt dataset D _X_ “ t _**x**_ _i_ u _[N]_ _i_ “1
SFT baseline model _π_ _**θ**_ 0
An LLM annotator
A DAP loss function _ℓ_ p _**x**_ _,_ _**y**_ [`] _,_ _**y**_ [´] _,_ _**θ**_ q
1: **for** _t_ :“ 0 to _T_ **do**
2: Sample prompt _**x**_ „ D _X_
3: Sample response pair _**y**_ [1] _,_ _**y**_ [2] „ _π_ _**θ**_ _t_ p¨| _**x**_ q
4: Use LLM annotator to get preference pair _**y**_ [`] _,_ _**y**_ [´]
5: Update _**θ**_ _[t]_ into _**θ**_ _[t]_ [`][1] using _∇_ _**θ**_ _ℓ_ p _**x**_ _,_ _**y**_ [`] _,_ _**y**_ [´] _,_ _**θ**_ _[t]_ q
6: **end for**
**Output:** Aligned language model (policy) _π_ _**θ**_ _T_
to obtain p _**y**_ [`] _,_ _**y**_ [´] q, which means that p _**y**_ [`] _,_ _**y**_ [´] q are also
in principle functions of _**θ**_ . In practice, we propose to simply use _∇_ _**θ**_ _ℓ_ p _**x**_ _,_ _**y**_ [`] _,_ _**y**_ [´] _,_ _**θ**_ q as our gradients, which amounts
to placing a stop_gradient on both the sampling and
LLM annotation steps.
**Annotating prompts with text-controllability** . We adopt
a pairwise prompting scheme to collect AI feedback, i.e.
we instruct the LLM annotator to choose which response
is preferred among a pair, as in Lee et al. (2023). To avoid
position bias, we calculate scores for the two response possible orders and use the average as the final score. Since
OAIF leverages prompting techniques to collect feedback,
the reward signals or the preference function can be easily
adapted by modifying the prompts (Sun et al., 2024). This
offers high flexibility without incurring any extra computation (such as retraining the RM) compared to RLHF and
RLAIF. For example, in our experiments, we show that we
can control the response length by simply prompting the
annotator to prefer shorter responses.

3. Direct alignment from online AI feedback


We use three tasks for experiments: TL;DR (Stiennon et al.,
2020), Anthropic Helpfulness and Anthropic
Harmlessness (Bai et al., 2022a). For each task, we
prepare the prompt dataset D _X_ by simply extracting the input prompts from the preference dataset D. We adopt PaLM
2 (Anil et al., 2023) as the language model and also the LLM
annotator. Unless otherwise specified, all policy models are
initialised from the model obtained by supervised finetuning
(SFT) PaLM 2-XS (Extra Small), which is referred to as
the SFT baseline. For the annotating model, we use PaLM
2-L (Large). To obtain online feedback from the annotating
model, we adopt the _Detailed 0-shot_ prompt from Lee et al.


(2023). The prompts we used and how we get preference
scores from them are detailed in Appendix E.
To demonstrate the generality of OAIF, we experiment with
three DAP methods: DPO, IPO and SLiC. Based on preliminary experiments, we set _β_ “ 0 _._ 1 in DPO, _β_ “ 1 _._ 0 in IPO,
and _β_ “ 0 _._ 002 in SLiC. We sample responses with a temperature of 0.9 during training. We adopt Adafactor (Shazeer
& Stern, 2018) as the optimiser, and set the batch size to
128 and the learning rate to 5 ¨ 10 [´][7], with a warm-up period
of 150 steps for all experiments. We evaluate models by
computing win rates, i.e. how often one model’s response
is better than the other. For automatic evaluation, we apply
the same prompting technique as above but with Gemini
Pro (Gemini Team et al., 2023) to reduce the risk of overfitting and reward hacking (Gao et al., 2023). The validity
of Gemini Pro as the judge is explored in Appendix C. For
human evaluation, three raters are presented with responses
generated from a set of policy models. Each rater is then
asked to independently score the responses’ quality (from 1
to 5 where 5 denotes the highest) and to pick the best one,
and the average score is then used to compare the models.


We start by examining the effectiveness of OAIF for DAP
methods (that use online AI feedback), compared to their
offline counterparts (that use pre-collected offline human
preferences). As a sanity check, we track the win rate of
DPO with OAIF (“Online DPO”) and vanilla DPO (“Offline
DPO”) against the SFT baseline on TL;DR. The results are
given in Figure 3, where the results for RLAIF and RLHF
are provided as references.
|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|
|---|---|---|---|---|---|---|---|
|||||||||
|||||||||
|||||||||
||||||Onl
Off|ine DPO
ine DP||
|RLAIF
RLHF|RLAIF
RLHF|RLAIF
RLHF|RLAIF
RLHF|RLAIF
RLHF|RLAIF
RLHF|RLAIF
RLHF|RLAIF
RLHF|
|0
2000
4000
6000
8000
10000
12000
Training Steps|0
2000
4000
6000
8000
10000
12000
Training Steps|0
2000
4000
6000
8000
10000
12000
Training Steps|0
2000
4000
6000
8000
10000
12000
Training Steps|0
2000
4000
6000
8000
10000
12000
Training Steps|0
2000
4000
6000
8000
10000
12000
Training Steps|0
2000
4000
6000
8000
10000
12000
Training Steps|0
2000
4000
6000
8000
10000
12000
Training Steps|
Figure 3: Win rate of DPO with OAIF (online DPO), vanilla
DPO (offline DPO), RLAIF, and RLHF against the SFT
baseline on the TL;DR task, judged by _Gemini Pro_ .
Not surprisingly, both online and offline DPO improve the
performance of the model, as shown by the substantially
high win rate achieved against the SFT baseline. However,
as indicated by the sharp drop of the red curve around training step 3 _,_ 500, offline DPO rapidly _overfits_ the offline and
off-policy preferences in D. In contrast, the win rate of
online DPO keeps increasing over training, and _surpasses_
offline DPO after 4 _,_ 000 steps. This demonstrates the effectiveness of OAIF. To consolidate the findings we got with
Gemini Pro as automatic evaluator, the same experiment
was also carried out with PaLM 2-L as the automatic evaluator. The results, given in Appendix D, confirm that our
observations hold under both automatic evaluators.
Method Win Tie Loss Quality
TL;DR
Online DPO **63.74%** 7 _._ 69% **3.95**
28 _._ 57%
Offline DPO 7 _._ 69% 63 _._ 74% 3 _._ 46
Helpfulness
Online DPO **58.60%** 20 _._ 20% **4.08**
21 _._ 20%
Offline DPO 20 _._ 20% 58 _._ 60% 3 _._ 44
Harmlessness
Online DPO **60.26%** 3 _._ 84% **4.41**
35 _._ 90%
Offline DPO 3 _._ 84% 60 _._ 26% 3 _._ 57
Table 2: Win/tie/loss rate of DPO with OAIF (online
DPO) against vanilla DPO (offline DPO) on the TL;DR,
Helpfulness, Harmlessness tasks, along with the
quality score of their generations, judged by _human raters_ .
Next, we evaluate OAIF on different tasks, i.e., TL;DR,
Helpfulness and Harmlessness. We select the best
performing online and offline DPO models according to
both manual inspection and their development set win rate
against the SFT baseline by Gemini Pro. We then report
side-by-side human evaluations comparing online DPO and
offline DPO in Table 2.
Human evaluation shows that OAIF significantly improves
the performance of DPO across all tasks with substantial superiority over offline DPO. This consolidates our conclusion
that using the offline feedback and off-policy generations in
a pre-collected preference dataset D can be detrimental for
LLM alignment, and OAIF benefits greatly from leveraging
online and on-policy AI feedback.

4.2. How effective is OAIF for LLM alignment?


As shown in Algorithm 1, OAIF is compatible with arbitrary
DAP loss functions. We therefore check the effectiveness
of OAIF for IPO and SLiC. The side-by-side human evaluation results on TL;DR comparing the online and offline
counterparts of these methods are given in Table 3.

4.3. How does OAIF generalise to other DAP methods?


(b) Quality against length of responses
(a) Fraction of responses preferred by humans
Figure 4: **Left** : Fraction of outputs from online DPO, offline DPO, RLAIF, and RLHF being preferred in a 4-way comparison;
**Right** : average quality scores (y-axis, higher is better) assigned to responses of different lengths (x-axis). The responses of
each model were first grouped into six buckets by their length. The mean and standard error of responses in a bucket are
then plotted as a data point. All results are judged by _human raters_ on TL;DR.
Method Win Tie Loss Quality
Online DPO **63.74%** 7 _._ 69% **3.95**
28 _._ 57%
Offline DPO 7 _._ 69% 63 _._ 74% 3 _._ 46
Online IPO **64.81%** 3 _._ 71% **3.84**
31 _._ 48%
Offline IPO 3 _._ 71% 64 _._ 81% 2 _._ 93
Online SLiC **71.43%** 1 _._ 59% **3.85**
26 _._ 98%
Offline SLiC 1 _._ 59% 71 _._ 43% 3 _._ 23
Table 3: Win/tie/loss rate of DAP methods with OAIF (online DPO/IPO/SLiC) against their offline counterparts in
TL;DR along with the quality score of their generations,
judged by _human raters_ .
Compared to their offline counterparts, DAP methods with
OAIF achieve promising win rates, ranging from „64% to
„71%. The consistent ineffectiveness of offline DAP methods confirms that the existence of the offline and off-policy
issue in DAP methods and greatly hinders the performance
of aligning LLMs. The consistent superiority of online
DAP methods via OAIF against their offline counterparts
demonstrates that OAIF is a general framework effectively
addressing these challenges.


Understanding the merits of DPO and RLHF is still a relatively open research question. We argue that comparing
online DPO with RLAIF and RLHF, which is interesting on
its own sake, can also contribute to answering this question.
We adopt similar experimental setups for RLAIF and RLHF
as before, to make the comparison as fair as possible: we
employ PaLM 2-L as the AI feedback model for RLAIF
and use the same pre-collected preference dataset to train
RMs for RLHF. Our training and optimisation procedures
follow Lee et al. (2023). Figure 4a shows the human evaluation results, where online DPO is more preferred than the
other methods, in 58% of the time.
We emphasise that the RM used in RLAIF and RLHF is
often not updated during policy training. As a result, its
response assessment ability may not generalise, as the output distribution from _π_ _**θ**_ _t_ evolves. To verify this hypothesis,
we also trained an online DPO with the same RM used for
RLAIF. It outperforms RLAIF, but significantly underperforms online DPO with OAIF, with a win rate of ă30%
judged by Gemini Pro. This experimental result supports
the superiority of using LLMs over RMs to provide online feedback. Synchronously retraining the RM is feasible
theoretically (Ziegler et al., 2019), but this would greatly
complicate the training pipeline and increase training cost.
Despite the great performance of OAIF compared to various
baselines, we found that OAIF tends to produce significantly
longer responses. This may affect the LLM and human
evaluation as both evaluators often prefer long generations,
referred to as “length bias” by Singhal et al. (2023). To
avoid the effect of such bias on analysing the performance
of OAIF, we group the responses by their length, and plot the


average quality score of each group. The results in Figure 4b
show that online DPO with OAIF provides responses of
higher quality than the other methods at fixed length, which
further validates the effectiveness of OAIF.


Another important dimension arising during our experiment
is the size of the annotating LLMs. Previous experiments are
all based on PaLM 2 L for feedback collection. To examine
the feasibility of feedback from smaller LLM annotators, we
then replicate online DPO experiments on TL;DR but with
feedback from PaLM 2-XS and PaLM 2-S instead. Figure 5
shows the comparison to SFT baseline, offline DPO, RLAIF,
and RLHF models we used, as in the previous experiments.
|Col1|S|
|---|---|
|XS|XS|
|Col1|Col2|L|
|---|---|---|
||S|S|
|XS|XS|XS|
|Col1|L|
|---|---|
|S|S|
Figure 5: Win rate of online DPO against the SFT baseline,
offline DPO, RLAIF, and RLHF, with annotating LLMs of
varying sizes (XS, S, L) in the task TL;DR, as assessed by
_Gemini Pro_ .
The size of the LLM annotator clearly has a significant
impact on OAIF. Generally, as size increases, online DPO
obtains better performance. Compared to the initial SFT
model, online DPO with OAIF performs significantly better
regardless of AI labeller model sizes, suggesting that even
OAIF from a small LLM annotator is helpful in improving
the performance of alignment. In particular, OAIF with
PaLM 2-XS (i.e. an LLM annotator of same-size) achieves
comparable performance to RLHF, although the latter learns
from human feedback. Further human evaluation confirms
this observation: OAIF with PaLM 2-XS obtains an overall
quality score of 3.41 out of 5, slightly better than RLHF
(3.38) and comparable to offline DPO (3.46).


While the necessity of LLM alignment has been widely
recognised, what to align them with is still under debate, as
human expectations vary greatly across regions and cultures,
and may evolve over time. This indicates that the human
preference annotation might change dramatically and frequently. In RLHF, such changes require re-annotating the
preference dataset and re-training the RM, leading to high
cost. In contrast, as OAIF is obtained through prompting
the LLM annotator, its reward signal could be adjusted by
simply modifying the prompts.
To examine this, we choose to explore the controllability
of the length of responses by modifying the prompts to
the LLM annotators. We take the online DPO model _π_ _**θ**_
trained to be as _helpful_ as possible in Section 4.2 as the
reference. We further train another two online DPO models
with the same experiment setup, but in which the annotator
is prompted to favor “ _helpful and short_ ” and “ _helpful and_
_very short_ ” responses. The exact prompts given to the LLM
annotators are provided in Table 6 and Table 8.
We display the average length of responses over training
in Figure 6a. The “short” and “very short” prompts given to
the LLM annotator significantly shorten the responses from
„120 tokens to „90 and „40 tokens respectively. This direct evidence demonstrates that the behaviour of policy _π_ _**θ**_
can be significantly changed through prompting the annotating LLM differently, and the degree of the changes can be
controlled as well.
However, the above changes come at a cost. In Figure 6b,
we plot the win rate of the “helpful”, “helpful and short”,
and “helpful and very short” models against the initial SFT
baseline. We noticed that the shorter responses become
much less helpful, as judged by Gemini Pro. Nevertheless,
they still improve the performance of the aligned model over
the SFT baseline. This finding is also confirmed by human
evaluation: from “helpful”, “helpful and short” to “helpful
and very short”, the average quality score drops from 4.08,
3.72 to 3.26, all outperforming the SFT baseline (3.19) still.


Section 4.5 shows that PaLM 2-XS could provide reasonable
feedback that helps improving the alignment of LLMs, although it’s significantly smaller than PaLM 2-S/L. We argue
that our approach offers an orthogonal solution to the _weak-_
_to-strong generalisation_ problem investigated by Burns et al.
(2023). To verify that a weaker AI labeller can improve the
performance of a stronger LLM model, we perform experiments using PaLM 2-S as the policy model (student) under
two teacher settings: one with PaLM 2-XS (weaker teacher)
and the other with PaLM 2-L (stronger teacher). The sideby-side automatic evaluation results on Helpfulness
comparing against the SFT baseline and offline DPO are
given in Figure 7. Our results suggest that OAIF from a
weaker teacher indeed improved the alignment of PaLM
2-S, though they are less effective compared with the OAIF

4.7. Can weaker AI labeller improve stronger LLM?


(b) Win rate against the initial SFT baseline
(a) Average length of responses
Figure 6: Performance on the Helpfulness task of online DPO with OAIF, trained to be _helpful only_, _helpful and short_,
_helpful and very short_ . Win rates are judged by Gemini Pro. Results for SFT, RLHF, and RLAIF models are given as
references.
from a stronger teacher.
|Col1|L|
|---|---|
|XS|XS|
Figure 7: Win rate of online DPO with OAIF from
PaLM 2-XS (weak teacher) and PaLM 2-L (strong teacher)
against the SFT baseline and offline DPO, in the task
Helpfulness, judged by _Gemini Pro_ .
We hereby emphasise the essential difference between the
setup investigated by Burns et al. (2023) and ours. In their
work, the tasks for the teacher and student model are both
supervised learning tasks, thus they are of equal difficulty.
However, in our work, the role of teacher is a simpler discriminative task (labelling preference), whereas the student
model being aligned is given a more difficult one (generating proper responses). Following this perspective, our
method is actually closer in spirit to the generative adversarial network proposed by Goodfellow et al. (2020), but
doesn’t train a particular discriminator.


**Limitations** . In this work, we study only the shift between
distributions over responses, e.g. _ρ_ p _**y**_ | _**x**_ q and _π_ _**θ**_ _t_ p _**y**_ | _**x**_ q.
However, the shifts also happen on the user prompt distribution _pX_ and the ground-truth human value function.
Although the prompt-controllability of OAIF raises a possible solution to later case, the shift of _pX_ is still a challenge. Since we extract prompts from the given preference
dataset, our study assumes an in-distribution of prompts
used for evaluation, thus lacks of evaluating the performance
of aligned LLMs on out-of-distribution prompts. In the
meantime, the model aligned in Section 4 is always PaLM
2-XS, thus whether our conclusion holds after scaling up
is not investigated. As pointed out by Bai et al. (2022a), it
is harder to distinguish responses of higher quality. Therefore, how much can OAIF for responses from larger LLMs
requires further study.
**Self-annotating models** . In all the experiments in Section 4, we aligned models _π_ _**θ**_ using preferences generated
by a separate LLM annotator. Yet, technically speaking, the
feedback could also be from the model _π_ _**θ**_ _t_ being trained
at time-step _t_ . This method, used recently by Yuan et al.
(2024), is promising as outputting responses and annotating preferences are two distinct tasks, the former being a
generative task and the latter a discriminative task. However, one disadvantage of this approach is that the model
architecture and size have to be the same. In contrast, the
LLM annotator in OAIF can be of arbitrary nature: as shown
in Section 4.5, an LLM annotator of larger size brings additional benefits. Therefore, we argue that the choice of LLM
annotator should not necessarily be limited to the model


being aligned, especially when an LLM annotator of larger
size or higher quality is available.
**Qualitative preference annotation from LLMs** . While
we used response length as a simple test-bed, the promptcontrollability of reward signals can be naturally extended
to more qualitative desiderata. Human values (such as helpfulness and impartiality) are a typical example of qualitative
desiderata. Moreover, one motivation for annotating preferences instead of quantitative scores by human labellers
is indeed because grading how well a response follows human values is difficult. Our approach, however, shows that
AI feedback can achieve the same goal by changing only
the prompts to the LLM annotators. Our approach can be
extended to align language models to other qualitative objectives without much input from human labellers.
**Preference from real-time human feedback.** In our work
the online feedback is from LLM annotators, but it is technically plausible to replace them with real online users. In
such case, the model can be aligned towards either a specific
group of users or an individual user, and the key bottleneck becomes the sample efficiency for fine-tuning LLMs.
During our experiment in Section 4.2, we found that the
behaviour of a model can be visibly changed with „2 _,_ 000
training steps, which requires „256 _,_ 000 samples. To personalise an LLM, this amount of data is still way too much
for an individual user to produce, which is a limitation of
applying RLHF for single-user personalisation of LLMs.
A common solution to improve sample efficiency is to use
low-rank adaptation (LoRA) (Hu et al., 2021). However,
aligning an LLM to a specific person requires several fundamental advances and we leave this to future research.


To circumvent the offline feedback problem in direct alignment from preference (DAP) methods, such as DPO, we
proposed Online AI Feedback (OAIF), a simple and effective way to make DAP methods online via AI feedback. We
carried out an extensive empirical evaluation, using both AI
and human evaluation, which showed the effectiveness of
DAP methods combined with OAIF, against their offline
counterparts. We also exhibited the tendency of offline DAP
methods to overfit, and in contrast the usefulness of OAIF
as a way to mitigate reward overoptimization. We further
verified the generality of OAIF, as our empirical results hold
for three prominent DAP methods: DPO, IPO and SLiC.
Beyond the empirical evaluation of OAIF, our work also
contributes the comparison of two types of methods: online
DAP methods (e.g., online DPO) and RLAIF. Since the
feedback comes from identical models in both learning algorithms, our experiment setup ensures that the AI feedback
is of the same quality and that only the learning procedures
differ. Our experimental results in various tasks show that
online DPO outperforms RLAIF and RLHF, which further
confirms the effectiveness of OAIF, compared to offline
feedback. Moreover, we used response length as a test bed
to demonstrate that the LLM annotator can be controlled
easily using instruction prompts. This shows that OAIF can
be used to achieve desirable alignment goals.
Overall, this work demonstrates the effectiveness and importance of OAIF for aligning LLMs, and paves the way
for more scalable alignment strategies, requiring reduced
human annotation effort.


We hereby acknowledge the enlightening discussion we had
with Yao Fu for refining the initial design of our method, the
invaluable assistance from Harrison Lee and Samrat Phatale
on conducting experiments with RLAIF and RLHF, the
insightful suggestions and feedback provided by Nino Vieillard which significantly contributed to enhancing the quality
of our paper, as well as the dedication to developing the infrastructure essential for this project from Léonard Hussenot,
Robert Dadashi, Geoffrey Cideron, Alexis Jacq, Sabela
Ramos, Piotr Stanczyk, Sertan Girgin, Danila Sinopalnikov,
Amélie Héliou, Nikola Momchev, Olivier Bachem, Sarah
Perrin, Pier Giuseppe Sessa, Matt Hoffman, Bobak Shahriari.


We propose a new method to improve the alignment of AI
with human values. Our method paves the way for more
scalable alignment with reduced human efforts. Since we
rely on AI feedback, to tackle other challenges in RLHF
(Casper et al., 2023) and mitigate safety risks (Amodei et al.,
2016), our approach must be considered within the larger
context of responsible and safe AI.