We verify the effectiveness of EXO via extensive experimentations. In Section4.1, we systematically study the frontier
of reward maximization and KL minimization achieved by
different alignment methods in a controlled text generation
task following previous works (Ziegler et al., 2019; Rafailov
et al., 2023). We investigate two different settings, including learning directly from preference data governed by a
predefined oracle reward model, and 2) learning from a reward model estimated from the preference data. In Section4.2,
we compare EXO against different approaches on realistic
alignment problems including generating human-preferred
summaries, helpful dialogue responses, and answers that
follow human instructions. Under both settings of learning
directly from the preference data and from a reward model,
EXO outperforms existing alignment approaches, e.g., DPO
and PPO. Next, we briefly describe the experiment settings
and leave additional details in Appendix B.
**Experiment Setting.** Our experiments consider two practical settings of LM alignment: (i) Directly train on a preference dataset = _**x**_ [(] _[j]_ [)] _,_ _**y**_ [(] _w_ _[j]_ [)] _[,]_ _**[ y]**_ [(] _l_ _[j]_ [)] _j_ =1 [where] _**[ y]**_ _w_ [and] _**[ y]**_ _l_
are the chosen and rejected responses judged by an oracle re- _D_ [pref] _{_ _[}][N]_
ward model or human labelers. (ii) Train on a reward dataset
_D_ [rw] = _{_ _**x**_ [(] _[j]_ [)] _,_ ( _**y**_ [(] 1 _[j]_ [)] _[, r]_ 1 [(] _[j]_ [)][)] _[,][ · · ·][,]_ [ (] _**[y]**_ _K_ [(] _[j]_ [)] _[, r]_ _K_ [(] _[j]_ [)][)] _[}]_ _j_ _[N]_ =1 [where] _**[ y]**_ _k_
is generated by the SFT policy and _rk_ is a scalar reward
provided by a reward model estimated on the given preference dataset. In the **controlled text generation** task, the
policy is optimized to generate a completion _**y**_ with positive sentiment given a prefix _**x**_ of a movie review from
the IMDB dataset [2] (Maas et al., 2011). To systematically
evaluate the alignment performance, we train a binary sentiment classifier on the IMDB dataset and define the oracle
reward as its log odds following Ziegler et al. (2019). Both
the policy and the reward models are initialized from the
GPT-2 large model (Radford et al., 2019). In the **summa-**
**rization** task, the policy is required to generate a summary
_**y**_ of the post _**x**_ from the Reddit forum that is preferred by
human annotators. Following Stiennon et al. (2020), we
use the same filtered version [3] of the Reddit TL;DR summarization dataset (Volske et al.¨, 2017) to train the SFT
2 `[https://huggingface.co/datasets/imdb](https://huggingface.co/datasets/imdb)` .
3 `[https://huggingface.co/datasets/UCL-DARK/](https://huggingface.co/datasets/UCL-DARK/openai-tldr-filtered)`
`[openai-tldr-filtered](https://huggingface.co/datasets/UCL-DARK/openai-tldr-filtered)` .
policy and use their preference dataset [4] for the alignment
problem. In the **dialogue generation** task, the policy is
learned to generate a helpful response _**y**_ given multi-turn
dialogue history between the user and the assistant denoted
as _**x**_ . We use the helpfulness subset of the Anthropic Helpful and Harmless dialogue dataset [5] (Bai et al., 2022) as the
preference dataset and train the SFT policy using the chosen responses. For summarization and dialogue generation
tasks, we initialize both the policy and the reward model
from the Pythia-2.8B (Biderman et al., 2023) following
Rafailov et al. (2023). To ensure sample quality, we use a
temperature of _τ_ = 0 _._ 8 to divide the logits of the language
model in all experiments. Lastly, for the **instruction fol-**
**lowing** task, we create a dataset based on instructions with
high demand and representativeness from the real-world
application scenarios, featuring 83K pairs of preferences
annotated by human labelers and 49K prompts for policy
training. The average lengths of the instructions and the answers are 47 and 230 respectively. We curate a diverse set of
high-quality test instructions to assess a range of capabilities
of the learned LM policy, encompassing multilingual ability,
creative writing, open-ended question answering, and role
playing. Each category takes the same proportion in the test
set. Both the policy and the reward models are initialized
from ChatGLM2-6B (Du et al., 2022).
**Evaluation.** In the controlled text generation task, we evaluate the frontier of the oracle reward and the KL divergence
achieved by different approaches. This enables us to systematically compare the effectiveness of different methods in
maxmizing the oracle reward under the same distributional
shift constrained by the reverse KL. For experiments on
the public preference datasets of summarization and dialogue generation, we use the reward model trained on the
preference dataset as an in-domain proxy of the unknown
ground-truth reward and also query GPT-4 for zero-shot
pair-wise evaluation, which is shown to be consistent with
human judgments (Rafailov et al., 2023). The prompts for
GPT-4 evaluation are slightly modified based on those used
in Rafailov et al. (2023), as detailed in Appendix B. We
compare the generated outputs against those generated by
the SFT policy and the preferred choice in the preference
dataset. For the instruction-following task, we report the
win rate of directly comparing our method against various
baselines as judged by GPT-4. Additionally, we employ
human assessment to evaluate criteria including adherence
to instruction, correctness, fluency, safety and helpfulness.
**Methods for Comparison.** We consider the following
methods for aligning language models with human prefer
4 `[https://huggingface.co/datasets/openai/](https://huggingface.co/datasets/openai/summarize_from_feedback)`
`[summarize_from_feedback](https://huggingface.co/datasets/openai/summarize_from_feedback)` .
5 `[https://huggingface.co/datasets/Anthropic/](https://huggingface.co/datasets/Anthropic/hh-rlhf)`
`[hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf)` .


0 2 4 6
DKL( _π||π_ sft)
_Figure 2._ The frontier of oracle reward vs reverse KL to the SFT
policy of different methods in the controlled experiment.
ences under various settings. Under the setting of learning
directly from preferences, we consider the special case of
EXO for preference learning _L_ exo-pref (Eq. (16)) denoted as
**EXOpref**, and the standard DPO (Rafailov et al., 2023) that
minimizes the pair-wise loss _L_ dpo (Eq. (6)) on the preference data, which we denoted as **DPOpref** . Then we consider
the setup of alignment with a reward model estimated from
the preference dataset, which includes the RL algorithm
**PPO** (Ziegler et al., 2019) that optimizes the expected reward with a KL penalty (Eq. (3)), the general EXO objective
_L_ exo (Eq. (13)) that performs probability matching by minimizing reverse KL, which is denoted as **EXOrw**, the general
DPO objective _L_ dpo-rw (Eq. (17)) that minimizes the forward KL, which is denoted as **DPOrw**, and the **Best-of-** _N_
method which first samples _N_ = 128 outputs from the SFT
policy and then returns the response with the highest score
according to the reward model. Note that the Best-of- _N_
baseline is practically inefficient and can be regarded as an
upperbound of exploiting the SFT policy according to the
reward model in Eq. (4) by trading off the computation.

Towards Efficient and Exact Optimization of Language Model Alignment

Towards Effcient Exact Optimization of Language Model Alignment

Haozhe Ji
Cheng Lu
Yilin Niu
Pei Ke
Hongning Wang
Jun Zhu
Jie Tang
Minlie Huang


To avoid undesirable reward overoptimizaion due to distributional shift, a preferred alignment solution should return a policy that obtains high oracle reward while incuring minimum deviation from _π_ sft. Thereby, we plot the
frontier of the oracle reward against KL divergence in Figure 2. We additionally present the accuracy-KL frontier
in Figure 7 in Appendix C.3. Each point represents a
checkpoint of the learned policy which is evaluated on
512 prefixes from the test set to complete the response
with maximumly 512 tokens. We sample _M_ = 4 completions _{_ _**y**_ _i}i_ _[M]_ =1 [for each given prompt] _**[ x]**_ [ to calculate]
the average oracle reward as well as to reduce the variance of approximating the sequence-level KL divergence
DKL( _πθ∥π_ sft) _≈_ _M_ 1 - _Mi_ =1 [log] _[ π][θ]_ [(] _**[y]**_ _i_ _[|]_ _**[x]**_ [)] _[ −]_ [log] _[ π]_ [sft][(] _**[y]**_ _i_ _[|]_ _**[x]**_ [)][.]
Despite aiming to optimize the same alignment objective,
the EXO approaches (EXOpref and EXOrw) yield the most
_Figure 3._ Visualization of the estimated density ratio between the
optimal and learned policy by EXO and DPO and the SFT policy
on samples from the SFT policy sorted by their log probabilities.
efficient frontiers in their respective settings, evidenced
by consistently achieving higher oracle rewards than baselines under the same KL divergence. Specifically, in the
setting of directly learning from preference pairs, EXOpref
outperforms DPOpref by a large margin, which underscores
EXO’s better sample efficiency when learning from a constrained source of preference indicators. As _K_ increases,
the frontier of EXOrw begins to exhibit convergence, indicating an effective progression towards the intended solution
within a finite _K_ . Although DPOrw also improves over
DPOpref when _K_ = 4, the frontier becomes worse when
_K_ is further increased to 8. This result substantiates our
analysis about the mean-seeking behavior of forward KL,
which leads to inaccuracy in capturing the modes of the
complex target distribution. Finally, we illustrate the strong
optimization efficiency of EXO in Figure 8, evidenced by
consistently achieving high and stable oracle rewards within
fewer number of training steps compared with PPO and
DPO in Appendix C.4.
Next, we compare DPO and EXO from the probability
matching perspective by visualizing the probability density of the policies obtained by these two approaches [6] . In
Figure 3, we plot the estimated density ratio of the optimal
and learned policies by EXO and DPO against _π_ sft given
a randomly chosen test prompt “ _This Fox spectacle was a_
_big hit when released in_ ”. Since the probability density of
an LM policy is defined over a high dimensional space of
_Y_ = _V_ _[n]_, it is intractable to evaluate every point in this
space exhaustively. Thus, we consider the representative
data points that are sampled from _π_ sft, and sort them in the
ascending order of their log probabilities. Then we compute
the empirical distribution under the learned policies over
these samples. Formally, given _M_ = 256 samples _**y**_ _i_ _i_ =1
_{_ _}_ _[M]_
drawn from _π_ sft conditioned on the prompt _**x**_, the empirical
distribution ˆ _πθ_ is calculated via self-normalized importance
sampling over the learned policy _πθ_ :
_π_ ˆ _θ_ ( _**y**_ _i_ _**x**_ ) = ~~�~~ _M_ _Mπθ_ ( _**y**_ _i|_ _**x**_ ) _._
_|_
_j_ =1 _[π][θ]_ [(] _**[y]**_ _j_ _[|]_ _**[x]**_ [)] _[/π]_ [sft][(] _**[y]**_ _j_ _[|]_ _**[x]**_ [)]
6We consider the setting of learning directly from preferences.


Reward Model (%) GPT-4 (%)
Method
vs SFT vs Chosen vs SFT vs Chosen
```
         w/ Preferences
```
DPOpref 68.3 23.7 57.0 30.5
EXOpref **92.5** **60.1** **83.0** **55.0**
```
         w/ Reward Model
```
Best-of- _N_ 99.3 75.8 83.5 60.0
PPO 93.2 58.3 77.0 52.0
DPOrw 82.7 39.8 70.0 41.0
EXOrw **97.3** **76.4** **88.5** **64.0**
_Table 1._ Win rates against the SFT generated texts and the chosen
texts on the TL;DR summarization dataset. Best results from the
computationally efficient methods are highlighted in **boldface** .
For the optimal policy, the empirical distribution reduces to:
_π_ ˆ _β_ _[∗]_ [(] _**[y]**_ _i_ _[|]_ _**[x]**_ [) =] _[Mπ]_ ~~�~~ [sft] _M_ [(] _**[y]**_ _[i][|]_ _**[x]**_ [) exp(] _[r]_ [(] _**[x]**_ _[,]_ _**[ y]**_ _[i]_ [)] _[/β]_ [)] _._
_j_ =1 [exp(] _[r]_ [(] _**[x]**_ _[,]_ _**[ y]**_ _j_ [)] _[/β]_ [)]
Finally, we use kernel density estimation to estimate the
probability densityplot the density ratio ˆ _π_ ( _ρ_ _**y**_ _|π_ ˆ _**x**_ () _**y**_ of the empirical distribution and _|_ _**x**_ ) = _ππ_ ˆsft(( _**yy**_ _|_ _**x**_ _|_ _**x**_ )) [against the log]
probability of the data points under _π_ sft. From the result,
the density ratio of the EXO policy closely matches the
optimal policy at the high probability region against _π_ sft,
which reflects its ability in capturing the major modes of
the optimal policy. However, the DPO policy overestimates
the long tail in _π_ sft due to the mean-seeking tendency of
forward KL, resulting in a reduced accuracy in fitting the
modes of the optimal policy. We present more visualization
results in Appendix C.5.


Large language models (Rae et al., 2021; Hoffmann et al.,
2022; Touvron et al., 2023) learned to predict the next tokens on large corpora have significantly improved the performance of various natural language processing (NLP)
tasks in zero shot (Radford et al., 2019) or few-shot (Brown
et al., 2020; Chowdhery et al., 2023) settings. To bridge
the mismatch between the training objective and users’ objective of solving specific NLP tasks, instruction-tuning is
proposed, which fine-tunes the language models on human
curated instruction-response pairs in a fully supervised manner (Sanh et al., 2021; Chung et al., 2022; Longpre et al.,
2023; Wang et al., 2023). Despite its performance and gen
GPT-4 evaluation
vs DPO vs PPO vs SFT
vs DPO vs PPO vs SFT


eralization to unseen instruction sets (Chung et al., 2022),
there have been concerns that the model learned to predict
the next token might only capture surficial patterns rather
than comprehending the tasks (Kung & Peng, 2023).
To address the aforementioned deficiencies, the framework
of reinforcement learning from human feedback (RLHF) is
proposed (Ziegler et al., 2019), which relies on only _rela-_
_tive_ human preferences on response quality and optimizes
the language model by RL algorithms (Williams, 1992),
e.g., PPO (Schulman et al., 2017), under the supervision
of a reward model which encapsulates the implicit preference of humans. Despite the success of RLHF in various
tasks where alignment is strongly emphasized, e.g., translation (Kreutzer et al., 2018), summarization (Stiennon et al.,
2020), instruction following (Ouyang et al., 2022), and etc.,
RL fine-tuning for large language models still faces serious
challenges in stability and scalability (Zheng et al., 2023b).
Thereby, a new trend of recent works proposed to optimize
the language model to follow human preferences without
applying RL algorithms (Yuan et al., 2023; Zhao et al., 2023;
Rafailov et al., 2023). While most approaches are empirically set to model alignment as a re-ranking problem, DPO
proposed a theoretically sound way to realize direct policy
optimization from preference data based on the analytic optimal solution of the reward maximization objective in RLHF.
Afterwards, there have been several studies that extend DPO
in various ways, e.g., altering the preference data by sampling from the optimal policy via rejection sampling (Liu
et al., 2023), substituting the point-wise reward with a pairwise preference function (Azar et al., 2023), extending the
preference pairs to rankings of preferences of any size (Song
et al., 2023), and etc. However, these approaches are still
based on the formation of DPO. In this work, we propose
to learn the policy by exactly optimizing the RLHF objective via probability matching that minimizes the reverse KL.
In the literature of maximum-entropy RL (Eysenbach &
Levine, 2019; Korbak et al., 2022), this equivalent form of
probability matching was discussed only for analysis purposes, while we are the first to derive a practical way to
optimize it. We also revisit DPO under this perspective and
recognize that it actually corresponds to minimizing the forward KL in its general form. To this sense, DPO shares the
same spirit of weighted regression (Peters & Schaal, 2007;
Peng et al., 2019; Wang et al., 2020), an algorithm that directly utilizes behavioral actions to supervise the policy in
offline RL (Peters et al., 2010; Lu et al., 2023). However,
this approach is known to be suboptimal when the policy
model is limited in distributional expressivity (Yue et al.,
2022; Chen et al., 2023). We analyze the characteristics of
the probability density learned by DPO with both theoretical
insight and empirical experimentations.


The alignment framework proposed by Ziegler et al. (2019)
relies on the KL regularization to the SFT policy to prevent
the optimized policy from greedily maximizing out the reward model which is estimated from the human preference
data. Despite the regularization, our experiments still reveal
instances of reward over-optimization, a phenomenon possibly due to insufficient focus on the reward model estimation
and use. Importantly, ensuring that the reward model accurately reflects the true modes of the oracle distribution
of human is more vital for PPO and EXO that optimizes
a mode-seeking objective. This opens possible avenues to
improve the alignment framework at a broader scope. For
instance, rather than relying on a static preference dataset
for reward model training, it could be more effective to dynamically improve the reward model with the development
of the policy, thereby offering more precise feedback. Upon
the current regularization that solely focuses on proximity to
the initial policy, one can take into account the uncertainty
of the reward model output to avoid over-exploitation of
the reward model. It is beneficial to take into account these
aspects to develop efficient and effective method towards
closer alignment with human preferences. Additionally,
while we already evaluated EXO on advanced language
models up to 6B on realistic scenarios, scaling EXO to models that are orders of magnitude larger can present profound
implications. At the other end of the spectrum, systematically dissecting and comparing PPO, DPO and EXO, e.g.,
regarding their variance and bias during optimization is
essential to broaden our understanding of these methods.


In this work, we consider the problem of aligning language
models with human preferences. Although reinforcement
learning (RL) for reward maximization presents the direct
and apparent solution, we reframe the problem in a supervised probability matching framework, which underscores
the probabilistic interpretation of the alignment procedure.
This derives our _efficient exact optimization_ (EXO) of the
KL-regularized reward maximization objective of alignment.
Formally, we prove the asymptotic equivalence between the
EXO objective and the alignment objective. In practice,
EXO enables efficient optimization via probability matching between empirical distributions, which avoids the complexities of RL algorithms. We further demonstrate that
DPO in its general form actually corresponds to minimizing
the forward KL against the optimal policy, which is shown
to be less effective in capturing the modes of the optimal
policy under realistic model parametrization with both theoretical and empirical justifications. Finally, we demonstrate
the effectiveness and scalability of EXO on various text
generation tasks with real human preferences.


This work was supported by the NSFC projects (with No.
61936010 and No. 62306160). This work was supported by
the National Science Foundation for Distinguished Young
Scholars (with No. 62125604). This work was also supported by China National Postdoctoral Program for Innovative Talents (No. BX20230194) and China Postdoctoral
Science Foundation (No. 2023M731952). We would also
like to thank Zhipu AI for sponsoring the computation resources and annotation cost used in this work.


This paper presents a method whose goal is to advance the
alignment of language models with human preferences. This
endeavor, while technically challenging, carries significant
implications for the ethical use and societal impact of artificial intelligence. The goal of alignment aims to mitigate the
inherent biases of AI systems and ensure that they reflect
diverse human values, goals and intentions that are safe and
ethical. It enhances AI’s utility in various sectors to make
reliable decisions that are in line with organizational goals
and ethical standards.


S., Nakov, P., and Villavicencio, A. (eds.), _Proceed-_
_ings of the 60th Annual Meeting of the Association for_
_Computational Linguistics (Volume 1: Long Papers),_
_ACL 2022, Dublin, Ireland, May 22-27, 2022_, pp. 320–
335. Association for Computational Linguistics, 2022.
doi: 10.18653/V1/2022.ACL-LONG.26. URL `[https:](https://doi.org/10.18653/v1/2022.acl-long.26)`
`[//doi.org/10.18653/v1/2022.acl-long.26](https://doi.org/10.18653/v1/2022.acl-long.26)` .
Eysenbach, B. and Levine, S. If maxent rl is the answer,
what is the question? _arXiv preprint
2019.
Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In Krause, A., Brunskill,
E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J.
(eds.), _International Conference on Machine Learning,_
_ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA_,
volume 202 of _Proceedings of Machine Learning Re-_
_search_, pp. 10835–10866. PMLR, 2023. URL `[https:](https://proceedings.mlr.press/v202/gao23h.html)`
`[//proceedings.mlr.press/v202/gao23h.html](https://proceedings.mlr.press/v202/gao23h.html)` .
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E.,
Cai, T., Rutherford, E., de Las Casas, D., Hendricks,
L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E.,
Millican, K., van den Driessche, G., Damoc, B., Guy,
A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W.,
Vinyals, O., and Sifre, L. Training compute-optimal
large language models. _CoRR_, abs/2203.15556, 2022.
doi: 10.48550/ARXIV.2203.15556. URL `[https://doi.](https://doi.org/10.48550/arXiv.2203.15556)`
`[org/10.48550/arXiv.2203.15556](https://doi.org/10.48550/arXiv.2203.15556)` .
Ji, H., Ke, P., Hu, Z., Zhang, R., and Huang, M. Tailoring language generation models under total variation distance. In _The Eleventh International Confer-_
_ence on Learning Representations, ICLR 2023, Kigali,_
_Rwanda, May 1-5, 2023_ . OpenReview.net, 2023a. URL
`[https://openreview.net/pdf?id=VELL0PlWfc](https://openreview.net/pdf?id=VELL0PlWfc)` .
Ji, H., Ke, P., Wang, H., and Huang, M. Language model
decoding as direct metrics optimization. _arXiv preprint_
_
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E.,
Bang, Y. J., Madotto, A., and Fung, P. Survey of hallucination in natural language generation. _ACM Computing_
_Surveys_, 55(12):1–38, 2023c.
Korbak, T., Perez, E., and Buckley, C. L. Rl with kl penalties
is better viewed as bayesian inference. _arXiv preprint_
_
Kreutzer, J., Uyheng, J., and Riezler, S. Reliability and
learnability of human bandit feedback for sequenceto-sequence reinforcement learning. _arXiv preprint_
_
Kung, P.-N. and Peng, N. Do models really learn to follow
instructions? an empirical study of instruction tuning.
_arXiv preprint
Lin, C.-C., Jaech, A., Li, X., Gormley, M., and Eisner, J.
Limitations of autoregressive models and their alternatives. In _NAACL_, 2021.
Liu, T., Zhao, Y., Joshi, R., Khalman, M., Saleh, M.,
Liu, P. J., and Liu, J. Statistical rejection sampling improves preference optimization. _arXiv preprint_
_
Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W.,
Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., et al. The
flan collection: Designing data and methods for effective instruction tuning. _arXiv preprint
2023.
Lu, C., Chen, H., Chen, J., Su, H., Li, C., and Zhu, J. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In
Krause, A., Brunskill, E., Cho, K., Engelhardt, B.,
Sabato, S., and Scarlett, J. (eds.), _International Con-_
_ference on Machine Learning, ICML 2023, 23-29 July_
_2023, Honolulu, Hawaii, USA_, volume 202 of _Pro-_
_ceedings of Machine Learning Research_, pp. 22825–
22855. PMLR, 2023. URL `[https://proceedings.](https://proceedings.mlr.press/v202/lu23d.html)`
`[mlr.press/v202/lu23d.html](https://proceedings.mlr.press/v202/lu23d.html)` .
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng,
A. Y., and Potts, C. Learning word vectors for sentiment analysis. In Lin, D., Matsumoto, Y., and Mihalcea, R. (eds.), _The 49th Annual Meeting of the As-_
_sociation for Computational Linguistics: Human Lan-_
_guage Technologies, Proceedings of the Conference, 19-_
_24 June, 2011, Portland, Oregon, USA_, pp. 142–150.
The Association for Computer Linguistics, 2011. URL
`[https://aclanthology.org/P11-1015/](https://aclanthology.org/P11-1015/)` .
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright,
C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K.,
Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L.,
Simens, M., Askell, A., Welinder, P., Christiano, P. F.,
Leike, J., and Lowe, R. Training language models to
follow instructions with human feedback. In _NeurIPS_,
2022.
Papini, M., Binaghi, D., Canonaco, G., Pirotta, M., and
Restelli, M. Stochastic variance-reduced policy gradient. In _International conference on machine learning_, pp.
4026–4035. PMLR, 2018.
Peng, X. B., Kumar, A., Zhang, G., and Levine, S.
Advantage-weighted regression: Simple and scalable offpolicy reinforcement learning. _CoRR_, abs/1910.00177,
2019. URL `[http://arxiv.org/abs/1910.00177](http://arxiv.org/abs/1910.00177)` .


Peters, J. and Schaal, S. Reinforcement learning by rewardweighted regression for operational space control. In
Ghahramani, Z. (ed.), _Machine Learning, Proceedings_
_of the Twenty-Fourth International Conference (ICML_
_2007), Corvallis, Oregon, USA, June 20-24, 2007_, volume 227 of _ACM International Conference Proceeding_
_Series_, pp. 745–750. ACM, 2007. doi: 10.1145/1273496.
1273590. URL `[https://doi.org/10.1145/1273496.](https://doi.org/10.1145/1273496.1273590)`
`[1273590](https://doi.org/10.1145/1273496.1273590)` .
Peters, J., Mulling, K., and Altun, Y. Relative entropy¨
policy search. In Fox, M. and Poole, D. (eds.), _Proceed-_
_ings of the Twenty-Fourth AAAI Conference on Artificial_
_Intelligence, AAAI 2010, Atlanta, Georgia, USA, July_
_11-15, 2010_, pp. 1607–1612. AAAI Press, 2010. doi:
10.1609/AAAI.V24I1.7727. URL `[https://doi.org/](https://doi.org/10.1609/aaai.v24i1.7727)`
`[10.1609/aaai.v24i1.7727](https://doi.org/10.1609/aaai.v24i1.7727)` .
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D.,
Sutskever, I., et al. Language models are unsupervised
multitask learners. _OpenAI blog_, 1(8):9, 2019.
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J.,
Song, H. F., Aslanides, J., Henderson, S., Ring, R., Young,
S., Rutherford, E., Hennigan, T., Menick, J., Cassirer, A.,
Powell, R., van den Driessche, G., Hendricks, L. A.,
Rauh, M., Huang, P., Glaese, A., Welbl, J., Dathathri, S.,
Huang, S., Uesato, J., Mellor, J., Higgins, I., Creswell,
A., McAleese, N., Wu, A., Elsen, E., Jayakumar, S. M.,
Buchatskaya, E., Budden, D., Sutherland, E., Simonyan,
K., Paganini, M., Sifre, L., Martens, L., Li, X. L., Kuncoro, A., Nematzadeh, A., Gribovskaya, E., Donato, D.,
Lazaridou, A., Mensch, A., Lespiau, J., Tsimpoukelli,
M., Grigorev, N., Fritz, D., Sottiaux, T., Pajarskas, M.,
Pohlen, T., Gong, Z., Toyama, D., de Masson d’Autume,
C., Li, Y., Terzi, T., Mikulik, V., Babuschkin, I., Clark,
A., de Las Casas, D., Guy, A., Jones, C., Bradbury, J.,
Johnson, M. J., Hechtman, B. A., Weidinger, L., Gabriel,
I., Isaac, W., Lockhart, E., Osindero, S., Rimell, L., Dyer,
C., Vinyals, O., Ayoub, K., Stanway, J., Bennett, L.,
Hassabis, D., Kavukcuoglu, K., and Irving, G. Scaling
language models: Methods, analysis & insights from
training gopher. _CoRR_, abs/2112.11446, 2021. URL
`[https://arxiv.org/abs/2112.11446](https://arxiv.org/abs/2112.11446)` .
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D.,
Ermon, S., and Finn, C. Direct preference optimization:
Your language model is secretly a reward model. In _Thirty-_
_seventh Conference on Neural Information Processing_
_Systems_, 2023. URL `[https://arxiv.org/abs/2305.](https://arxiv.org/abs/2305.18290)`
`[18290](https://arxiv.org/abs/2305.18290)` .
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L.,
Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja,
A., et al. Multitask prompted training enables zero-shot
task generalization. _arXiv preprint
2021.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and
Klimov, O. Proximal policy optimization algorithms.
_arXiv preprint
Song, F., Bowen, Y., Li, M., Yu, H., Huang, F., Li,
Y., and Wang, H. Preference ranking optimization for human alignment. _ArXiv_, abs/2306.17492,
2023. URL `[https://api.semanticscholar.org/](https://api.semanticscholar.org/CorpusID:259308873)`
`[CorpusID:259308873](https://api.semanticscholar.org/CorpusID:259308873)` .
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe,
R., Voss, C., Radford, A., Amodei, D., and Christiano,
P. F. Learning to summarize from human feedback.
_CoRR_, abs/2009.01325, 2020. URL `[https://arxiv.](https://arxiv.org/abs/2009.01325)`
`[org/abs/2009.01325](https://arxiv.org/abs/2009.01325)` .
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux,
M.-A., Lacroix, T., Roziere, B., Goyal, N., Hambro, E.,`
Azhar, F., et al. Llama: Open and efficient foundation language models. _arXiv preprint
Volske, M., Potthast, M., Syed, S., and Stein, B. Tl;dr: Min-¨
ing reddit to learn automatic summarization. In Wang, L.,
Cheung, J. C. K., Carenini, G., and Liu, F. (eds.), _Proceed-_
_ings of the Workshop on New Frontiers in Summarization,_
_NFiS@EMNLP 2017, Copenhagen, Denmark, Septem-_
_ber 7, 2017_, pp. 59–63. Association for Computational
Linguistics, 2017. doi: 10.18653/V1/W17-4508. URL
`[https://doi.org/10.18653/v1/w17-4508](https://doi.org/10.18653/v1/w17-4508)` .
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A.,
Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions. In
Rogers, A., Boyd-Graber, J. L., and Okazaki, N. (eds.),
_Proceedings of the 61st Annual Meeting of the Associ-_
_ation for Computational Linguistics (Volume 1: Long_
_Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_,
pp. 13484–13508. Association for Computational Linguistics, 2023. doi: 10.18653/V1/2023.ACL-LONG.
754. URL `[https://doi.org/10.18653/v1/2023.](https://doi.org/10.18653/v1/2023.acl-long.754)`
`[acl-long.754](https://doi.org/10.18653/v1/2023.acl-long.754)` .
Wang, Z., Novikov, A., Zolna, K., Merel, J., Springenberg,
J. T., Reed, S. E., Shahriari, B., Siegel, N. Y., Gul¨ c¸ehre,
C¸ ., Heess, N., and de Freitas, N. Critic regularized
regression. In Larochelle, H., Ranzato, M., Hadsell, R.,
Balcan, M., and Lin, H. (eds.), _Advances in Neural Infor-_
_mation Processing Systems 33: Annual Conference on_
_Neural Information Processing Systems 2020, NeurIPS_
_2020, December 6-12, 2020, virtual_, 2020. URL `[https:](https://proceedings.neurips.cc/paper/2020/hash/588cb956d6bbe67078f29f8de420a13d-Abstract.html)`
```
 //proceedings.neurips.cc/paper/2020/hash/
 588cb956d6bbe67078f29f8de420a13d-Abstract.
```
`[html](https://proceedings.neurips.cc/paper/2020/hash/588cb956d6bbe67078f29f8de420a13d-Abstract.html)` .


Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. _Mach._
_Learn._, 8:229–256, 1992. doi: 10.1007/BF00992696.
URL `[https://doi.org/10.1007/BF00992696](https://doi.org/10.1007/BF00992696)` .
Yao, Z., Aminabadi, R. Y., Ruwase, O., Rajbhandari, S., Wu,
X., Awan, A. A., Rasley, J., Zhang, M., Li, C., Holmes, C.,
Zhou, Z., Wyatt, M., Smith, M., Kurilenko, L., Qin, H.,
Tanaka, M., Che, S., Song, S. L., and He, Y. Deepspeedchat: Easy, fast and affordable RLHF training of chatgptlike models at all scales. _CoRR_, abs/2308.01320, 2023.
doi: 10.48550/ARXIV.2308.01320. URL `[https://doi.](https://doi.org/10.48550/arXiv.2308.01320)`
`[org/10.48550/arXiv.2308.01320](https://doi.org/10.48550/arXiv.2308.01320)` .
Yuan, Z., Yuan, H., Tan, C., Wang, W., Huang, S., and
Huang, F. Rrhf: Rank responses to align language models with human feedback without tears. _arXiv preprint_
_
Yue, Y., Kang, B., Ma, X., Xu, Z., Huang, G., and YAN,
S. Boosting offline reinforcement learning via data rebalancing. In _3rd Offline RL Workshop: Offline RL as a_
_”Launchpad”_, 2022. URL `[https://openreview.net/](https://openreview.net/forum?id=vOC01fqW2T)`
`[forum?id=vOC01fqW2T](https://openreview.net/forum?id=vOC01fqW2T)` .
Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., and
Liu, P. J. Slic-hf: Sequence likelihood calibration with
human feedback. _arXiv preprint
Zheng, L., Chiang, W., Sheng, Y., Zhuang, S., Wu, Z.,
Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang,
H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge
with mt-bench and chatbot arena. _CoRR_, abs/2306.05685,
2023a. doi: 10.48550/ARXIV.2306.05685. URL `[https:](https://doi.org/10.48550/arXiv.2306.05685)`
`[//doi.org/10.48550/arXiv.2306.05685](https://doi.org/10.48550/arXiv.2306.05685)` .
Zheng, R., Dou, S., Gao, S., Hua, Y., Shen, W., Wang, B.,
Liu, Y., Jin, S., Liu, Q., Zhou, Y., et al. Secrets of rlhf
in large language models part i: Ppo. _arXiv preprint_
_
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford,
A., Amodei, D., Christiano, P. F., and Irving, G. Finetuning language models from human preferences. _CoRR_,
abs/1909.08593, 2019. URL `[http://arxiv.org/abs/](http://arxiv.org/abs/1909.08593)`
`[1909.08593](http://arxiv.org/abs/1909.08593)` .


_Proof._ We derive the optimal _πθ∗_ that maximizes the generalized alignment objective _J_ lhf _[β][r]_ [(] _[π]_ _θ_ _[β][π]_ [)][ and show that it equals to]
the optimal solution _πβ_ _[∗]_ [of] _[ J]_ lhf _[ β]_ [(] _[π][θ]_ [)][ given unlimited model capacity. First, we restate the formation of] _[ J]_ lhf _[ β][r]_ [(] _[π]_ _θ_ _[β][π]_ [)][:]
               -                _J_ lhf _[β][r]_ [(] _[π]_ _θ_ _[β][π]_ [) =][ E] _**[x]**_ _[∼D]_ [pref] E _πθβπ_ ( _**y**_ _|_ _**x**_ ) [[] _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[ y]**_ [)]] _[ −]_ _[β][r]_ [D][KL][[] _[π]_ _θ_ _[β][π]_ [(] _**[y]**_ _[|]_ _**[x]**_ [)] _[∥][π]_ [sft][(] _**[y]**_ _[|]_ _**[x]**_ [)]] _,_ (19)
where _πθ_ _[β][π]_ is defined as:
_πθ_ _[β][π]_ [(] _**[y]**_ _[|]_ _**[x]**_ [)] _[ ∝]_ _[π][θ]_ [(] _**[y]**_ _[|]_ _**[x]**_ [)] _[β][π]_ _[π]_ [sft][(] _**[y]**_ _[|]_ _**[x]**_ [)][1] _[−][β][π]_ _[.]_ (20)
Then we substitute _θ_ with the optimal _θ_ _[∗]_ in Eq. (20) where _πθ∗_ maximizes _J_ lhf _[β][r]_ [(] _[π]_ _θ_ _[β][π]_ [)][:]
_πθ_ _[β][∗][π]_ [(] _**[y]**_ _[|]_ _**[x]**_ [)] _[ ∝]_ _[π][θ][∗]_ [(] _**[y]**_ _[|]_ _**[x]**_ [)] _[β][π]_ _[π]_ [sft][(] _**[y]**_ _[|]_ _**[x]**_ [)][1] _[−][β][π]_ _[.]_ (21)
Since _πθ_ _[β][∗][π]_ [is also the optimal policy that maximizes] _[ J]_ lhf _[ β][r]_ [(] _[·]_ [)][, it should satisfy Eq. (][4][) which gives:]
_πθ_ _[β][∗][π]_ [(] _**[y]**_ _[|]_ _**[x]**_ [) =] _[ π]_ _β_ _[∗]_ _r_ [(] _**[y]**_ _[|]_ _**[x]**_ [)] _[ ∝]_ _[π]_ [sft][(] _**[y]**_ _[|]_ _**[x]**_ [)] _[e]_ _βr_ _[r]_ [(] _**[x]**_ _[,]_ _**[y]**_ [)] _._ (22)
Together with Eq. (21) and Eq. (22), we obtain the form of _πθ∗_ via some simple algebra:
_πθ_ _[∗]_ ( _**y**_ _|_ _**x**_ ) _∝_ ( _πβ_ _[∗]_ _r_ [(] _**[y]**_ _[|]_ _**[x]**_ [)] _[π]_ [sft][(] _**[y]**_ _[|]_ _**[x]**_ [)] _[β][π][−]_ [1][)] _βπ_
_βπ_ _−_ 1
_βπ_
_βπ_ _−_ 1
1 1
( _π_ sft( _**y**_ _**x**_ ) _e_ _βr_ _[r]_ [(] _**[x]**_ _[,]_ _**[y]**_ [)] ) _βπ_ _π_ sft( _**y**_ _**x**_ )
_∝_ _|_ _|_
1 1
_π_ sft( _**y**_ _**x**_ ) _βπ_ _e_ _βrβπ_ _[r]_ [(] _**[x]**_ _[,]_ _**[y]**_ [)] _π_ sft( _**y**_ _**x**_ )
_∝_ _|_ _|_
_βπ_ _−_ 1
_βπ_
_βπ_ _−_ 1
_π_ sft( _**y**_ _**x**_ ) _e_ _βrβπ_ _[r]_ [(] _**[x]**_ _[,]_ _**[y]**_ [)] _._
_∝_ _|_
By definition, _βrβπ_ = _β_, then _πθ∗_ reduces to the same form of the optimal solution of _J_ lhf _[β]_ [(] _[π][θ]_ [)][ defined in Eq. (][4][):]
_πθ∗_ ( _**y**_ _|_ _**x**_ ) = _πβ_ _[∗]_ [(] _**[y]**_ _[|]_ _**[x]**_ [)] _[ ∝]_ _[π]_ [sft][(] _**[y]**_ _[|]_ _**[x]**_ [)] _[e]_ _β_ _[r]_ [(] _**[x]**_ _[,]_ _**[y]**_ [)] _,_
which completes the proof.
**A.2. Derivation of Eq.** (9)
We first start by rearranging _J_ lhf _[β][r]_ [(] _[π]_ _θ_ _[β][π]_ [)][ into the expectation of a log ratio:]
        -        _J_ lhf _[β][r]_ [(] _[π]_ _θ_ _[β][π]_ [) =][ E] _**[x]**_ _[∼D]_ [pref] E _πθβπ_ ( _**y**_ _|_ _**x**_ ) [[] _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[ y]**_ [)]] _[ −]_ _[β][r]_ [D][KL][[] _[π]_ _θ_ _[β][π]_ [(] _**[y]**_ _[|]_ _**[x]**_ [)] _[∥][π]_ [sft][(] _**[y]**_ _[|]_ _**[x]**_ [)]]
    -    = E _**x**_ _∼D_ pref E _πθβπ_ ( _**y**_ _|_ _**x**_ ) [[] _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[ y]**_ [)]] _[ −]_ _[β][r]_ [E] _πθ_ _[βπ]_ ( _**y**_ _|_ _**x**_ ) log _[π]_ _πθ_ _[β]_ sft _[π]_ ( [(] _**y**_ _**[y]**_ _|_ _[|]_ _**x**_ _**[x]**_ ) [)]
��
= E _**x**_ _∼D_ pref - _βr_ E _**y**_ _∼πθβπ_ ( _**y**_ - _|_ _**x**_ )� log _e_ _βr_ 1 _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ [)][�] _−_ _βr_ E _π_ - _θβπ_ ( _**y**_ _|_ _**x**_ )� log _[π]_ _πθ_ _[β]_ sft _[π]_ ( [(] _**y**_ _**[y]**_ _|_ _[|]_ _**x**_ _**[x]**_ ) [)]
��
= E _**x**_ _∼D_ pref E _**y**_ _∼πθβπ_ ( _**y**_ _|_ _**x**_ )
- 1
_βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ [)]
_βr_ log _[π]_ [sft][(] _**[y]**_ _[|]_ _**[x]**_ [)] _[e]_
_πθ_ _[β][π]_ [(] _**[y]**_ _[|]_ _**[x]**_ [)]
Notice the analytical form of _πβ_ _[∗]_ _r_ [:]
1 1
_πβ_ _[∗]_ _r_ [(] _**[y]**_ _[|]_ _**[x]**_ [) =] _π_ sft( _**y**_ _**x**_ ) _e_ _βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ [)] _._
_Zβr_ ( _**x**_ ) _|_
We substitute _π_ sft( _**y**_ _|_ _**x**_ ) _e_ _βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ [)] into the expression of _J_ lhf _[β][r]_ [(] _[π]_ _θ_ _[β][π]_ [)][:]
_J_ lhf _[β][r]_ [(] _[π]_ _θ_ _[β][π]_ [) =][ E] _**[x]**_ _[∼D]_ [pref][E] _**y**_ _∼πθ_ _[βπ]_ ( _**y**_ _|_ _**x**_ )
_Zβr_ ( _**x**_ ) _πβ_ _[∗]_ _r_ [(] _**[y]**_ _[|]_ _**[x]**_ [)]
_βr_ log
_πθ_ _[β][π]_ [(] _**[y]**_ _[|]_ _**[x]**_ [)]
      -      = _βr_ E _**x**_ pref DKL( _πθ_ _[β][π]_ [(] _[·|]_ _**[x]**_ [)] _[∥][π]_ _β_ _[∗]_ _r_ [(] _[·|]_ _**[x]**_ [)) + log] _[ Z][β]_ _r_ [(] _**[x]**_ [)] _._
_∈D_ _−_


_Proof._ We first restate the definition of exo( _πθ_ ) by substituting Eq. (10), (12) into (13):
_L_
     - _K_
exo( _πθ_ ) = E _**x**_ prefE _π_ sft( _**y**_ 1: _K_ _**x**_ )
_L_ _∼D_ _|_
_i_ =1
_e_ _[f][θ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[i]_ [)]  - _e_ _[f][θ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[i]_ [)] _e_ _βr_ 1 _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[i]_ [)]
~~�~~ _K_ log ~~�~~ _K_ ~~�~~ _K_ 1
_βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[j]_ [)]
_j_ =1 _[e][f][θ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[j]_ [)] _j_ =1 _[e][f][θ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[j]_ [)] _[−]_ [log] _j_ =1 _[e]_
- [�]
(23)
Since _fθ_ ( _**x**_ _,_ _**y**_ ) = log _πθ_ _[β][π]_ [(] _**[y]**_ _[|]_ _**[x]**_ [)] _[ −]_ [log] _[ π]_ [sft][(] _**[y]**_ _[|]_ _**[x]**_ [)][, we have that:]
_πθ_ _[βπ]_ ( _**y**_ _i|_ _**x**_ )   _π_ sft( _**y**_ _i_ _**x**_ )
_|_ log
~~�~~ _K_ _πθ_ _[βπ]_ ( _**y**_ _j_ _|_ _**x**_ )
_j_ =1 _π_ sft( _**y**_ _j_ _**x**_ )
_|_
_e_ _βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[i]_ [)]
log ~~�~~ _K_ 1
_−_ _βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[j]_ [)]
_j_ =1 _[e]_
_θ_ 1
_e_ [log] _π_ sft( _**y**_ _i|_ _**x**_ ) _e_ _βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[i]_ [)]
- _K_ log _ππθ_ _[βπ]_ sft (( _**yy**_ _jj ||_ _**xx**_ )) _−_ log ~~�~~ _Kj_ =1 _[e]_ _βr_ 1 _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[j]_ [)]
_j_ =1 _[e]_
- [�]
exo( _πθ_ ) = E _**x**_ prefE _π_ sft( _**y**_ 1: _K_ _**x**_ )
_L_ _∼D_ _|_
= E _**x**_ prefE _π_ sft( _**y**_ 1: _K_ _**x**_ )
_∼D_ _|_
 - _K_
_i_ =1
 - _K_
_i_ =1
_πθ_ _[βπ]_ ( _**y**_ _i|_ _**x**_ )
_e_ [log] _π_ sft( _**y**_ _i|_ _**x**_ )
- _K_ log _ππθ_ _[βπ]_ sft(( _**yy**_ _jj ||_ _**xx**_ ))
_j_ =1 _[e]_
- _e_ [log] _ππθ_ _[βπ]_ sft(( _**yy**_ _ii||_ _**xx**_ ))
log
_πθ_ _[βπ]_ ( _**y**_ _i|_ _**x**_ )
_π_ sft( _**y**_ _i_ _**x**_ )
_|_
~~�~~ _K_ _πθ_ _[βπ]_ ( _**y**_ _j_ _|_ _**x**_ )
_j_ =1 _π_ sft( _**y**_ _j_ _**x**_ )
_|_
- [�]
_._ (24)
Since _**y**_ _i_ _i_ =1 [are sampled from] _[ π]_ [sft][(] _[·|]_ _**[x]**_ [)][, when] _[ K][ →∞]_ [, for arbitary function] _[ g]_ [ :] _[ Y →]_ [R][, the estimate] _K_ [1] - _Ki_ =1 _[g]_ [(] _**[y]**_ _i_ [)][ is]
_{_ _}_ _[K]_    - _K_
unbiased, i.e., lim _K→∞_ _K_ [1] _i_ =1 _[g]_ [(] _**[y]**_ _i_ [) =][ E] _[π]_ sft [(] _**[y]**_ _[|]_ _**[x]**_ [)][[] _[g]_ [(] _**[y]**_ [)]][. We consider the following two instantiations of] _[ g]_ [(] _[·]_ [)][.]
Since _{_ _**y**_ _i}i_ _[K]_ =1 [are sampled from] - _K_ _[ π]_ [sft][(] _[·|]_ _**[x]**_ [)][, when] _[ K][ →∞]_ [, for arbitary function] _[ g]_ [ :] _[ Y →]_ [R][, the estimate] _K_ [1]
 - _K_ _K_ _i_
_K_ [1] _i_ =1 _[g]_ [(] _**[y]**_ _i_ [) =][ E] _[π]_ sft [(] _**[y]**_ _[|]_ _**[x]**_ [)][[] _[g]_ [(] _**[y]**_ [)]][. We consider the following two instantiations of] _[ g]_ [(] _[·]_ [)][.]
For _g_ ( _**y**_ ) = _[π]_ _πθ_ _[βπ]_ sft(( _**yy**_ _|_ _**xx**_ )) [, we have:]
_|_
For _g_ ( _**y**_ ) = _e_ _βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ [)], we have:
- _πθβπ_ [(] _**[y]**_ _[|]_ _**[x]**_ [)]
_π_ sft( _**y**_ _**x**_ )
_|_
- _K_
_j_ =1
_πθ_ _[β][π]_ [(] _**[y]**_ _j_ _[|]_ _**[x]**_ [)]
_π_ sft( _**y**_ _j_ _**x**_ ) [=] _[ K]_ [E] _[π]_ [sft][(] _**[y]**_ _[|]_ _**[x]**_ [)]
_|_
 = _K_
_**y**_   - _∈Y_
= _K_
_θ_ [(] _**[y]**_ _[|]_ _**[x]**_ [)]
_π_ sft( _**y**_ _|_ _**x**_ ) _[π]_ _π_ _[β]_ sft _[π]_ ( _**y**_ _**x**_ )
_**y**_ _∈Y_ _|_
_πθ_ _[β][π]_ [(] _**[y]**_ _[|]_ _**[x]**_ [)]
_**y**_ _∈Y_
= _K._
- _K_ 1 - 1
_e_ _βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[j]_ [)] = _K_ E _π_ sft( _**y**_ _**x**_ ) _e_ _βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ [)][�]
_|_
_j_ =1
- _K_
 = _K_
_π_ sft( _**y**_ _**x**_ ) _e_ _βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ [)]
_|_
_**y**_ _∈Y_
= _KZβr_ ( _**x**_ ) _._
Then we simplify exo by substituting the expression of [�] _[K]_ _j_ =1 _ππθ_ _[βπ]_ sft(( _**yy**_ _jj_ _|_ _**xx**_ )) [and][ �] _j_ _[K]_ =1 _[e]_ _βr_ 1 _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[j]_ [)] when _K_ into Eq.
_L_ _|_ _→∞_
(24).
 - _K_
- [�]
- _βr_ 1 _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[i]_ [)]
_θ_ [(] _**[y]**_ _i_ _[|]_ _**[x]**_ [)]
log _[π][β][π]_ _[e]_
_Kπ_ sft( _**y**_ _i_ _**x**_ ) _KZβr_ ( _**x**_ )
_|_ _[−]_ [log]
exo( _πθ_ ) = E _**x**_ prefE _π_ sft( _**y**_ 1: _K_ _**x**_ )
_L_ _∼D_ _|_
= E _**x**_ prefE _π_ sft( _**y**_ 1: _K_ _**x**_ )
_∼D_ _|_
_i_ =1
_πθ_ _[β][π]_ [(] _**[y]**_ _i_ _[|]_ _**[x]**_ [)]
_Kπ_ sft( _**y**_ _i_ _**x**_ )
_|_
_KZβr_ ( _**x**_ )
_K_
- _K_
_i_ =1
_πθ_ _[β][π]_ [(] _**[y]**_ _i_ _[|]_ _**[x]**_ [)] _πθ_ _[β][π]_ [(] _**[y]**_ _i_ _[|]_ _**[x]**_ [)]
_π_ sft( _**y**_ _i|_ _**x**_ ) [log] _π_ sft( _**y**_ _i|_ _**x**_ ) _e_ _βr_ 1 _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[i]_ [)] _/Zβr_ ( _**x**_ )
_._


Notice the analytic form of _πβ_ _[∗]_ _r_ [(] _**[y]**_ _[|]_ _**[x]**_ [) =] _Zβr_ 1 ( _**x**_ ) _[π]_ [sft][(] _**[y]**_ _[|]_ _**[x]**_ [)] _[e]_ _βr_ 1 _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ [)], we substitute _πβ_ _[∗]_ _r_ [into the above equation:]
_K_
_i_ =1
exo( _πθ_ ) = E _**x**_ prefE _π_ sft( _**y**_ 1: _K_ _**x**_ )
_L_ _∼D_ _|_
- _K_
_πθ_ _[β][π]_ [(] _**[y]**_ _i_ _[|]_ _**[x]**_ [)] _[π]_ _θ_ _[β][π]_ [(] _**[y]**_ _i_ _[|]_ _**[x]**_ [)]
_π_ sft( _**y**_ _i|_ _**x**_ ) [log] _πβ_ _[∗]_ _r_ [(] _**[y]**_ _i_ _[|]_ _**[x]**_ [)]
_K_
_K_
_i_ =1
- [�]
- _πθβπ_ [(] _**[y]**_ _i_ _[|]_ _**[x]**_ [)] _[π]_ _θ_ _[β][π]_ [(] _**[y]**_ _i_ _[|]_ _**[x]**_ [)]
_π_ sft( _**y**_ _i|_ _**x**_ ) [log] _πβ_ _[∗]_ _r_ [(] _**[y]**_ _i_ _[|]_ _**[x]**_ [)]
= E _**x**_ pref
_∼D_
= E _**x**_ pref
_∼D_
= E _**x**_ pref
_∼D_
_θ_ [(] _**[y]**_ _[|]_ _**[x]**_ [)]
_πθ_ _[β][π]_ [(] _**[y]**_ _[|]_ _**[x]**_ [) log] _[π][β][π]_
_**y**_ _∈Y_ _πβ_ _[∗]_ _r_ [(] _**[y]**_ _[|]_ _**[x]**_ [)]
- _K_
E _π_ sft( _**y**_ _i_ _**x**_ )
_|_
_i_ =1
- _K_
- _K_
_πθ_ _[β][π]_ [(] _**[y]**_ _i_ _[|]_ _**[x]**_ [) log] _[π]_ _θ_ _[β][π]_ [(] _**[y]**_ _i_ _[|]_ _**[x]**_ [)]
_**y**_ _i_ _πβ_ _[∗]_ _r_ [(] _**[y]**_ _i_ _[|]_ _**[x]**_ [)]
_∈Y_
��
= E _**x**_ pref[DKL( _πθ_ _[β][π]_ [(] _**[y]**_ _[|]_ _**[x]**_ [)] _[∥][π]_ _β_ _[∗]_ _r_ [(] _**[y]**_ _[|]_ _**[x]**_ [))]] _[.]_
_∼D_
Accordingly, by taking the derivative with respect to _θ_, we complete the proof of Eq. (14).
To prove Eq. (15), we utilize Eq. (14) to substitute into Eq. (9) to build the connection between _J_ lhf _[β][r]_ [(] _[π]_ _θ_ _[β][π]_ [)][ and] _[ L]_ [exo][:]
_J_ lhf _[β][r]_ [(] _[π]_ _θ_ _[β][π]_ [) =] _[ −][β][r][L]_ [exo][(] _[π][θ]_ [) +] _[ β][r]_ [E] _**[x]**_ _[∼D]_ [pref][[log] _[ Z][β][r]_ [(] _**[x]**_ [)]] _[.]_
Then we take the gradient with respect to the parameters _θ_ of the above formulat:
_∇θJ_ lhf _[β][r]_ [(] _[π]_ _θ_ _[β][π]_ [) =] _[ −][β][r][∇][θ][L]_ [exo][(] _[π][θ]_ [)] _[,]_
which completes the proof of Eq. (15).


_Proof._ We utilize the definition of _πθ_ _[β][π]_ [(] _**[y]**_ _[|]_ _**[x]**_ [)] _[ ∝]_ _[π][θ]_ [(] _**[y]**_ _[|]_ _**[x]**_ [)] _[β][π]_ _[π]_ [sft][(] _**[y]**_ _[|]_ _**[x]**_ [)][1] _[−][β][π]_ [ in Eq. (][8][) and divide both sides by] _[ π]_ [sft][(] _**[y]**_ _[|]_ _**[x]**_ [)][:]
    _πθ_ _[β][π]_ [(] _**[y]**_ _[|]_ _**[x]**_ [)] _πθ_ ( _**y**_ _|_ _**x**_ )
_π_ sft( _**y**_ _**x**_ ) _π_ sft( _**y**_ _**x**_ )
_|_ _[∝]_ _|_
- _βπ_
_π_ sft( _**y**_ _**x**_ )
_|_
Then we substitute the above equation into dpo-rw:
_L_
dpo-rw( _πθ_ ) = E _**x**_ prefE _π_ sft( _**y**_ 1: _K_ _**x**_ )
_L_ _∼D_ _|_
= E _**x**_ prefE _π_ sft( _**y**_ 1: _K_ _**x**_ )
_∼D_ _|_
= E _**x**_ prefE _π_ sft( _**y**_ 1: _K_ _**x**_ )
_∼D_ _|_
_−_
_−_
_−_
1 _πθ_ ( _**y**_ _i|_ _**x**_ )
_e_ _βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[i]_ [)] _e_ _[β][π]_ [ log] _π_ sft( _**y**_ _i|_ _**x**_ )
~~�~~ _Kj_ =1 _[e]_ _βr_ 1 _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[j]_ [)] [log] - _K_ _βπ_ log _ππ_ sft _θ_ (( _**yy**_ _jj ||_ _**xx**_ ))
_j_ =1 _[e]_
_e_ _βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[i]_ [)]
~~�~~ _K_ _βr_ 1 _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[j]_ [)] [log]
_j_ =1 _[e]_
- _K_
_i_ =1
- _K_
_i_ =1
- _K_
_i_ =1
1 _πθ_ _[βπ]_ ( _**y**_ _i|_ _**x**_ )
_e_ _βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[i]_ [)] _e_ [log] _π_ sft( _**y**_ _i|_ _**x**_ )
~~�~~ _Kj_ =1 _[e]_ _βr_ 1 _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[j]_ [)] [log] - _K_ log _ππθ_ _[βπ]_ sft(( _**yy**_
- _K_ log _ππθ_ _[βπ]_ sft(( _**yy**_ _jj ||_ _**xx**_ ))
_j_ =1 _[e]_
_πθ_ _[βπ]_ ( _**y**_ _i|_ _**x**_ )
_π_ sft( _**y**_ _i_ _**x**_ )
_|_
~~�~~ _K_ _πθ_ _[βπ]_ ( _**y**_ _j_ _|_ _**x**_ )
_j_ =1 _π_ sft( _**y**_ _j_ _**x**_ )
_|_
(25)
Since _{_ _**y**_ _i}i_ _[K]_ =1 [are sampled from] _[ π]_ [sft][(] _[·|]_ _**[x]**_ [)][, when] _[ K][ →∞]_ [, we follow the proof of Theorem][ 3.2][ and directly give the]
following results:
- _K_
_j_ =1
_πθ_ _[β][π]_ [(] _**[y]**_ _j_ _[|]_ _**[x]**_ [)]
_π_ sft( _**y**_ _j_ _**x**_ ) [=] _[ K,]_
_|_
- _K_ 1
_e_ _βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[j]_ [)] = _KZβr_ ( _**x**_ ) _._
_j_ =1


Then we simplify dpo-rw by substituting the above results of [�] _[K]_ _j_ =1 _ππθ_ _[βπ]_ sft(( _**yy**_ _jj_ _|_ _**xx**_ )) [and][ �] _j_ _[K]_ =1 _[e]_ _βr_ 1 _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[j]_ [)] when _K_ into
_L_ _|_ _→∞_
Eq. (25):
_e_ _βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[i]_ [)] _[π]_ _θ_ _[β][π]_ [(] _**[y]**_ _i_ _[|]_ _**[x]**_ [)]
_KZβr_ ( _**x**_ ) [log] _Kπ_ sft( _**y**_ _i_ _**x**_ )
_|_
dpo-rw( _πθ_ ) = E _**x**_ prefE _π_ sft( _**y**_ 1: _K_ _**x**_ )
_L_ _∼D_ _|_
_−_
- _K_
_i_ =1
_e_ _βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[i]_ [)]
Notice the analytic form of _πβ_ _[∗]_ _r_ [(] _**[y]**_ _[|]_ _**[x]**_ [) =] _Zβr_ 1 ( _**x**_ ) _[π]_ [sft][(] _**[y]**_ _[|]_ _**[x]**_ [)] _[e]_ _βr_ 1 _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ [)], we rearrange _πβ_ _[∗]_ _r_ [and substitute] _[ π]_ [sft][(] _**[y]**_ _[|]_ _**[x]**_ [) =]
_βr_ [1] _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ [)] into the above equation to simplify it:
_πβ_ _[∗]_ _r_ [(] _**[y]**_ _[|]_ _**[x]**_ [)] _[Z][β]_ _r_ [(] _**[x]**_ [)] _[e][−]_ [1]
 - _K_
_−_
_i_ =1
 - _K_
_−_
_i_ =1
_e_ _βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[i]_ [)]
1 1
_βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[i]_ [)] _[e]_ _βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[i]_ [)]
_KZβr_ ( _**x**_ ) [log] _KZβr_ ( _**x**_ )
- [�]
_e_ _βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[i]_ [)] _[π]_ _θ_ _[β][π]_ [(] _**[y]**_ _i_ _[|]_ _**[x]**_ [)]
_KZβr_ ( _**x**_ ) [log] _πβ_ _[∗]_ _r_ [(] _**[y]**_ _i_ _[|]_ _**[x]**_ [)]
_[−]_
_e_ _βr_ 1 _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[i]_ [)] - _πθβπ_ [(] _**[y]**_ _i_ _[|]_ _**[x]**_ [)] _[e]_ _βr_ 1 _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[i]_ [)]
_KZβr_ ( _**x**_ ) [log] _πβ_ _[∗]_ _r_ [(] _**[y]**_ _i_ _[|]_ _**[x]**_ [)] _KZβr_ ( _**x**_ )
_[·]_
dpo-rw( _πθ_ ) = E _**x**_ pref E _π_ sft( _**y**_ 1: _K_ _**x**_ )
_L_ _∼D_ _|_
= E _**x**_ pref E _π_ sft( _**y**_ 1: _K_ _**x**_ )
_∼D_ _|_
_e_ _βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[i]_ [)]
_KZβr_ ( _**x**_ )
_KZβr_ ( _**x**_ )
- _K_
_i_ =1
The second term of the final equality remains constant in relation to _θ_, and thus can be omitted when computing the derivative
with respect to _θ_ . Then we further consider the gradient of dpo-rw:
_L_
- [�]
- _K_
E _π_ sft( _**y**_ _i_ _**x**_ )
_|_
_i_ =1
_θ_ dpo-rw( _πθ_ ) = _θ_ E _**x**_ pref
_∇_ _L_ _∇_ _∼D_
= _θ_ E _**x**_ pref
_∇_ _∼D_
= _θ_ E _**x**_ pref
_∇_ _∼D_
= _θ_ E _**x**_ pref
_∇_ _∼D_
- 1
_e_ _βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[i]_ [)]
1 _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[i]_ [)]
_θ_ [(] _**[y]**_ _i_ _[|]_ _**[x]**_ [)]
log _[π][β][π]_
_Zβr_ ( _**x**_ ) _πβ_ _[∗]_ _r_ [(] _**[y]**_ _i_ _[|]_ _**[x]**_ [)]
_−_ _K_ [1]
_−_
_**y**_ _∈Y_
_−_
_θ_ [(] _**[y]**_ _[|]_ _**[x]**_ [)]
_πβ_ _[∗]_ _r_ [(] _**[y]**_ _[|]_ _**[x]**_ [) log] _[π][β][π]_
_**y**_ _∈Y_ _πβ_ _[∗]_ _r_ [(] _**[y]**_ _[|]_ _**[x]**_ [)]
- 1
_e_ _βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ [)]
- [�]
E _π_ sft( _**y**_ _**x**_ )
_−_ _|_
_βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ [)]
_θ_ [(] _**[y]**_ _[|]_ _**[x]**_ [)]
log _[π][β][π]_
_Zβr_ ( _**x**_ ) _πβ_ _[∗]_ _r_ [(] _**[y]**_ _[|]_ _**[x]**_ [)]
- _βr_ 1 _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ [)]
_**y**_ _∈Y_ _π_ sft( _**y**_ _|_ _**x**_ ) _[e]_ _Zβr_ ( _**x**_ )
_βr_ _[r][ϕ]_ [(] _**[x]**_ _[,]_ _**[y]**_ [)]
_θ_ [(] _**[y]**_ _[|]_ _**[x]**_ [)]
log _[π][β][π]_
_Zβr_ ( _**x**_ ) _πβ_ _[∗]_ _r_ [(] _**[y]**_ _[|]_ _**[x]**_ [)]
      -       = _θ_ E _**x**_ pref DKL( _πβ_ _[∗]_ _r_ [(] _[·|]_ _**[x]**_ [)] _[∥][π]_ _θ_ _[β][π]_ [(] _[·|]_ _**[x]**_ [))] _,_
_∇_ _∼D_
which completes the proof of Theorem 3.3.
**A.5. Mechanistic Understanding of** _θ_ **exo** ( _πθ_ )
_∇_ _L_
We present the gradient of exo( _πθ_ ) defined in Eq. (13):
_L_
      - _K_
_θ_ exo( _πθ_ ) = E _**x**_ prefE _π_ sft( _**y**_ 1: _K_ _**x**_ ) _pfθ_ ( _k_ ) DKL( _pfθ_ _prϕ_ )
_∇_ _L_ _−_ _∼D_ _|_ ~~�~~ ~~�~~ - _∥_ ~~�~~
_k_ =1
_distributional_ gap
log _[p][f][θ]_ [(] _[k]_ [)]
_−_ - _p_ - _r_ ~~�~~ _ϕ_ ( _k_ )�
_point-wise_ correction
_θ_ log _πθ_ ( _**y**_ _k_ _**x**_ )
_∇_ _|_
_,_ (26)
where _pfθ_ ( _k_ ) and _prϕ_ ( _k_ ) are short for _pfθ_ ( _k_ _**y**_ 1: _K,_ _**x**_ ) and _prϕ_ ( _k_ _**y**_ 1: _K,_ _**x**_ ), which are defined in Eq. (11) and Eq. (12)
_|_ _|_
respectively.
Next, we provide a mechanistic understanding of _θ_ exo( _πθ_ ), which is the expected weighted sum of the gradients of
_∇_ _L_
the log likelihood on samples _pfθ_ ( _k_ ) _**y**_ 1: _K_ drawn from _π_ sft( _·|_ _**x**_ ). The weight is proportional to the difference between the log
probability ratio log _prϕ_ ( _k_ ) [and the KL divergence] _[ D]_ [KL][(] _[p][f][θ]_ _[∥][p][r][ϕ]_ [)][. Intuitively, if the policy has already correctly weighted]
_pfθ_ ( _k_ )
the sample _yk_ according to the reward model, i.e., _prϕ_ ( _k_ ) = _pfθ_ ( _k_ ), then log _prϕ_ ( _k_ ) [= 0][ and the weight suggests that it only]
needs to minimize the overall KL divergence between _pfθ_ and _prϕ_ on the distribution level. If the policy _πθ_ overestimates


_pfθ_ ( _k_ ) _pfθ_ ( _k_ )
or underestimates the sample _yk_, i.e., log _prϕ_ ( _k_ ) _[>]_ [ 0][ or][ log] _prϕ_ ( _k_ ) _[<]_ [ 0][, this log-ratio will be used to calibrate the KL]
divergence to penalize or encourage the policy to update towards increasing the likelihood of this sample at a faster rate.
Finally, this gradient form offers us the insight of when the optimization should stop: the gradient _θ_ exo( _πθ_ ) becomes 0
_∇_ _L_
when the two distribution _pfθ_ and _prϕ_ are identical.


We illustrate the relationship among the objectives _J_ lhf _[β]_ [(] _[π][θ]_ [)][,] _[ J]_ lhf _[ β][r]_ [(] _[π]_ _θ_ _[β][π]_ [)][,] _[ L]_ [dpo][(] _[π][θ]_ [)][,] _[ L]_ [dpo-rw][(] _[π][θ]_ [)][ and] _[ L]_ [exo][(] _[π][θ]_ [)][ in Figure][ 5][.]
_Figure 5._ Illustration of the relationship among the different objectives discussed in Section3. 1 : _J_ lhf _[β][r]_ [(] _[π]_ _θ_ _[β][π]_ [)][ is a generalized version of] _[ J]_ lhf _[ β]_ [(] _[π][θ]_ [)]
by distributing the KL regularization to both the learned policy _πθ_ and the reward model _rϕ_ (Section3.1). 2 : _L_ dpo( _πθ_ ) is derived based on the
optimal policy of _J_ lhf _[β]_ [(] _[π][θ]_ [)][ (Section][2.3][).][ 3][ :] _[ L]_ [exo][(] _[π][θ]_ [)][ is equivalent to] _[ J]_ lhf _[ β][r]_ [(] _[π]_ _θ_ _[β][π]_ [)][ in terms of their optimization directions (Section][3.2][).][ 4][ :] _[ L]_ [dpo-rw][ is]
the generalized version of _L_ dpo by subsituting the pariwise loss with softmax loss over _K_ responses.(Section3.3). The optimal policy, denoted
by a dotted line, assumes unlimited model capacity. The solution, shown with a solid line, is the practically achievable policy within the
realistic constraints of model capacity.

A.6. Illustrating the Relationship among the Objectives in Section3


**Training.** In the controlled text generation task, we use _βπ_ = 0 _._ 1 for EXOpref and DPOpref, and additionally use _βr_ = 0 _._ 1
and _K ∈{_ 4 _,_ 8 _}_ for EXOrw and DPOrw. For the tasks of summarization, dialogue generation, and instruction following,
we use _βπ_ = 0 _._ 5 for EXOpref and DPOpref, and additionally use _βr_ = 0 _._ 1 and _K_ = 4 for EXOrw and DPOrw. We provide
additional results of tuning _βr_ and _βπ_ in Appendix C.1 to justify our choice of hyperparameters. In all experiments, we set
the label smoothing hyperparameter _ε_ in EXOpref to `1e-3` . For DPO and EXO, we use the Adam optimizer with a universal
learning rate of `1e-6` and a batch size of 64 and train for one epoch on each dataset, although both methods converge within
one epoch. All the hyperparameters are set to be the same for DPO and EXO for a fair comparison. For PPO, we modify
based on the implementation of DeepSpeed-Chat (Yao et al., 2023), which sets _γ_ = 1 and _β_ = 0 _._ 1 by default. We pretrain
the critic model for the first few steps while freezing the actor model and find it to improve convergence. We empirically
tune the number of actor-freezing steps, total training steps, learning rate for actor and critic model, and the batch size for
PPO on each dataset, as PPO is sensitive to these hyperparameters. Specifically, we conduct 15 trials of hyperparameter
search on the IMDB dataset and 10 trials on the real human preference datasets in total. In the experiment, we report the
PPO performance with the best hyperparameters obtained under constrained number of hyperparameter search trials. We
conduct the experiments except for instruction following on 8 V100 GPUs. For instruction following task, we train the
models on 8 A100 GPUs.


**Evaluation.** At inference time, we sample 4 completions from the learned policy for each prompt and consider 512
prompts from the test set for all datasets. Except for the instruction following task, we sample from the policy with the same
temperature _τ_ = 0 _._ 8 that is set during training for consistency. For the instruction following task, we use top-p sampling
and empirically set _p_ = 0 _._ 9 and temperature _τ_ = 0 _._ 95 given its performance. To calculate the win rate evaluated by the
reward model, we consider all combinations of pairs between the completions generated by the learned policy and the base
completions (either generated by the SFT policy or the chosen completion in the dataset) and then compare the scores from
the reward model on the pairs of generations. For the evaluations using GPT-4, we sample 100 prompts and 1 completion
for each prompt under each policy. To mitigate the position bias of GPT-4, we evaluate one pair of generations twice by
swapping the order of responses in each pair. To evaluate the quality of the summaries, we use the concise prompt of Rafailov
et al. (2023) as shown in Table 3. To evaluate the helpfulness of the generated dialogues, we use the prompt shown in Table
4, which is modified based on the prompt of Rafailov et al. (2023) for single-turn dialogue to accommodate the general
multi-turn setting. For the instruction-following task, we use the prompt modified from the prompt for reference-guided
pairwise comparison provided in Zheng et al. (2023a).
**Human Assessment.** We conduct human assessment to evaluate the instruction following task more thoroughly. Specifically, we select three matchups that pair the generated outputs of EXO with those produced by DPO, PPO and the SFT
policy. Given 100 randomly sampled test instructions, each model generates 100 responses, which results in a total of
300 pairs of comparisons. We assign 3 human labelers to each comparison, producing 900 judgements in total. Given the
instruction, each human annotator is provided with two generated answers by two systems respectively together with a
high-quality reference answer. The annotator is then asked to make a preference among win, tie or lose by comparing the
generated answers with the reference answer, considering the criteria including adherence to instruction, correctness, fluency,
safety and helpfulness. Specifically, adherence to instruction encapsulates the model’s comprehension and following of the
prompt’s intention. Correctness involves the identification of inaccurate knowledge or logical inconsistencies within the
generated responses. Fluency assesses the linguistic coherence, encompassing an examination of sentence completeness,
grammatical accuracy, and the presence of a consistent language structure. Safety refers to the inspection for potentially
harmful content. Lastly, helpfulness indicates whether the responses provide the information required by the prompt or
contribute to problem resolution.
```
   Which of the following summaries does a better job of summarizing the most important
   points in the given forum post, without including unimportant or irrelevant details? A
   good summary is both precise and concise.
   Post:
   <post>
   Summary A:
   <Summary A>
   Summary B:
   <Summary B>
   FIRST provide a one-sentence comparison of the two summaries, explaining which you prefer
   and why. SECOND, on a new line, state only "A" or "B" to indicate your choice. Your
   response should use the format:
   Comparison: <one-sentence comparison and explanation>
   Preferred: <"A" or "B">
```
_Table 3._ Prompt for GPT-4 evaluation on the summarization task. Texts in blue are placeholders to be substituted by the real data.


**C.1. Ablation Study of** _βr_ **and** _βπ_
We present an ablation study to investigate the performance of EXOrw on the dialogue generation task by varying _βr_
and _βπ_ respectively. We execute multiple runs bifurcated into two series. We set _βπ_ = 0 _._ 5 as the default value and
vary _βr ∈{_ 0 _._ 1 _,_ 0 _._ 25 _,_ 0 _._ 5 _,_ 0 _._ 75 _,_ 1 _._ 0 _}_ . Subsequently, the process is reversed whereby we fix _βr_ = 0 _._ 1, and alter _βπ ∈_
_{_ 0 _._ 1 _,_ 0 _._ 25 _,_ 0 _._ 5 _,_ 0 _._ 75 _,_ 1 _._ 0 _}_ . We present the results in Figure 6.


```
   For the following dialogue history to a chatbot, which response is more helpful?
   Dialogue history:
   <dialogue history>
   Response A:
   <Response A>
   Response B: <Response B>
   FIRST provide a one-sentence comparison of the two responses and explain which you feel is
   more helpful. SECOND, on a new line, state only "A" or "B" to indicate which response is
   more helpful. Your response should use the format:
   Comparison: <one-sentence comparison and explanation>
   More helpful: <"A" or "B">
```
_Table 4._ Prompt for GPT-4 evaluation on the dialogue generation task. Texts in blue are placeholders to be substituted by the real data.
0 _._ 10 0 _._ 25 0 _._ 50 0 _._ 75 1 _._ 00
_βr_
0 _._ 10 0 _._ 25 0 _._ 50 0 _._ 75 1 _._ 00
_βπ_
0 _._ 10 0 _._ 25 0 _._ 50 0 _._ 75 1 _._ 00
_βπ_
0 _._ 10 0 _._ 25 0 _._ 50 0 _._ 75 1 _._ 00
_βr_
Reward model win rate.
GPT-4 win rate.
_Figure 6._ Win rate performance of EXOrw evaluated by the reward model and the GPT-4 by varying _βr_ and _βπ_ .
From Figure 6 (a), we observe that reducing either _βr_ and _βπ_ increase the reward model win rate, which is expected as a
small _βr × βπ_ encourages the policy to optimize the reward model while neglecting the KL regularization. In Figure 6 (b),
the GPT-4 win rate starts to decline when _βπ <_ 0 _._ 5, which suggests where the reward model starts to be over-optimized.
The different effects of tuning _βπ_ and _βr_ on the performance could be attributed to their different roles in our algorithm, i.e.,
_βπ_ scales the log probability in the parametrized policy while _βr_ scales the reward. Based on the results, we recommend
adopting a moderate value for _βπ_ and a lower value for _βr_, for instance, _βπ_ = 0 _._ 5 and _βr_ = 0 _._ 1.
**C.2. Effect of** _βr_ **and** _βπ_ **beyond the Product** _β_ = _βrβπ_
To further demonstrate the effect of _βr_ and _βπ_ beyond the impact on their product _β_ = _βπβr_, we fix _β_ = 0 _._ 05 while
tuning _βπ_ 1 _,_ 0 _._ 5 _,_ 0 _._ 1 _,_ 0 _._ 05 and _βr_ 0 _._ 05 _,_ 0 _._ 1 _,_ 0 _._ 5 _,_ 1 accordingly. The result is shown in Table 5. According to the
_∈{_ _}_ _∈{_ _}_
evaluation by GPT-4, the best performnce is obtained at _βπ_ = 0 _._ 5 _, βr_ = 0 _._ 1, while neither the hyper-parameter choice of
_βπ_ = 1 _, βr_ = 0 _._ 05 (the configuration of PPO) nor the choice of _βπ_ = 0 _._ 05 _, βr_ = 1 (the configuration of DPO) yield the
highest win rate evaluated by GPT-4. While using a small _βπ_ is more likely to trigger over-optimization of the reward model,
which indicates the asymmetric effect of the two hyperparameters _βr_ and _βπ_ on the optimization process.
Reward Model (%) GPT-4 (%)
_βr_ _βπ_ vs SFT vs Chosen vs SFT vs Chosen
1.0 0.05 87.1 88.4 81.0 57.5
0.5 0.1 87.1 88.0 77.0 59.5
0.1 0.5 85.6 87.2 **83.5** **60.0**
0.05 1.0 84.1 84.8 76.5 52.0
_Table 5._ Effect of _βr_ and _βπ_ beyond _β_ by tuning _βπ_ and _βr_ while keeping their product _β_ = _βrβπ_ fixed.


We additionally calculate the accuracy of the sentiment classifier by taking the sigmoid of the oracle reward and plot the
frontier of the accuracy vs reverse KL in Figure 7.
|Col1|PPO
EXOpref
EXOrw (K=4)
EXOrw (K=8)
DPOpref
DPOrw (K=4)
DPO (K=8)|
|---|---|
||rw|
0 2 4 6
DKL( _π||π_ sft)
_Figure 7._ The frontier of classifier accuracy vs reverse KL to the SFT policy of different methods in the controlled experiment.

C.3. Frontier of Classifier Accuarcy against KL Divergence


To demonstrate the efficency of optimizing the oracle reward, we plot the curve of the oracle reward with standard deviation
on the test set of different training steps. Specifically, the standard deviation is calculated on the 4 samples generated given
the same prompt and then averaged across 512 prompts.
PPO
DPOpref
DPOrw (K=4)
DPOrw (K=8)
EXOpref
EXOrw (K=4)
EXOrw (K=8)
controlled experiment.


We additionally sample more prompts from the test set of the IMDb dataset, and plot the estimated density ratio of the
policy learned by EXO and DPO and the optimal policy in Figure 9. Similar results were obtained among these diverse set
of prompts: the density ratio under the EXO policy better matches the optimal policy at the high probability region against
_π_ sft than the DPO policy. And very appearently, the DPO policy tends to overestimate the long tail in _π_ sft and thus often
loses resolution in capturing the modes of the optimal policy.


We present the sample outputs generated by EXO and DPO and the judgments made by GPT-4 when compared to the chosen
response in the preference dataset. Table 6 and Table 7 show examples on the task of dialogue generation and summarization
respectively.

C.6. Sample Outputs with GPT-4 Judgments


Human: How do I make a fishtail braid?
Assistant: Hey human, just walk down the left and then down the right side, then at the end, you pull through. Are you
looking to just braid, or a tight ponytail?
Human: Yes. Can you tell me the steps to do it?


Assistant: Sure, here’s how to do a side braid:
1) Pull hair up to the back of your head, leaving it out from the rest of your hair so you can see it.
2) Put one hair on the side where you want to make your braid.
3) Hold the braid with your thumb and forefinger
4) Bring your braid through the opposite side (putting the hair to the other side of the braid.
5) Repeat steps 2, 3, and 4 as needed until you’ve completed your braid.


Assistant: Sure! I’ll walk you through it step by step:
1. Stand with your left hand on the back of your head. Grab a left side section of hair, using the right hand to pull it
through.
2. Take a right side section of hair, and pull it through
3. Grab a left side section of hair, and pull it through
4. Grab a right side section of hair, and pull it through
5. Repeat steps 2-4 for the last section of hair
6. Wrap the hair around your head and secure it at the nape of your neck.


Response A provides a more detailed and step-by-step guide on how to make a fishtail braid, while Response B seems to
describe a different type of braid and is less clear.


Response B is more helpful because it provides a clearer, step-by-step guide on how to make a fishtail braid, whereas
Response A seems to describe a different type of braid and is less clear.


Assistant: Sure, let’s walk through it. You’ll want to start with a braid about two and a half inches long, then you want to
braid it together and pull through, which means you’ll somehow thread it through your own hair and tie it off at the other
end. Then you’re going to braid it again, but this time it will be a fishtail, since you’ll pull it through. You’ll keep doing
this, braiding the fishtail.


Response B is more helpful as it provides a clear, step-by-step guide on how to make a fishtail braid, while Response A
is confusing and lacks clarity.


Response A provides a clear, step-by-step guide on how to make a side braid, while Response B gives a vague description
of how to make a fishtail braid without clear steps.
_Table 6._ Responses generated by EXO and DPO given the dialogue history sampled from the Anthropic-HH test set. GPT-4 consistently
prefers EXO and disprefers DPO over the chosen response regardless of the order in which the evaluated pairs are presented.


|optim
EXO
DPO|Col2|Col3|
|---|---|---|
||200 1|00
800
600
400
200
0|
Estimated density ratio of the EXO, DPO and optimal policy
given the prompt “ _Is this supposed to be serious? I hope not_ ”.
|optim
EXO
DPO|Col2|Col3|
|---|---|---|
||200
|000
800
600
400
200|
Estimated density ratio of the EXO, DPO and optimal policy
given the prompt “ _This is indeed the film that popularized_
_kung_ ”.
|optimal
EXO
DPO|Col2|Col3|
|---|---|---|
||200|000
800
600
400
200|
Estimated density ratio of the EXO, DPO and optimal policy
given the prompt “ _What we have here the standard Disney_
_direct to DVD_ ”.
|optimal
EXO
DPO|Col2|Col3|
|---|---|---|
||200|000
800
600
400
200
0|
|optimal
EXO
DPO|Col2|
|---|---|
||000
800
600
400
200
0|
|75|Col2|Col3|Col4|
|---|---|---|---|
|1
0
5
0
5|optimal
EXO
DPO|optimal
EXO
DPO|optimal
EXO
DPO|
|1
0
5
0
5|00|200 1|00 800
600
400
200
0|
Estimated density ratio of the EXO, DPO and optimal policy
given the prompt “ _Once the slow beginning gets underway, the_
_film kicks_ ”.
Estimated density ratio of the EXO, DPO and optimal policy
given the prompt “ _Great book, great movie, great soundtrack._
_Frank_ ”.
Estimated density ratio of the EXO, DPO and optimal policy
given the prompt “ _This movie is about a group of people who_
_are_ ”.
_Figure 9._ Visualization of the estimated density ratio of the policy learned by EXO and DPO, and the optimal policy given 10 test prompts
randomly sampled from the IMDb dataset.


SUBREDDIT: r/Dogtraining
TITLE: Best walks when on a choke chain: dog isn’t being choked, but is it bringing up bad memories?
POST: My 3 year old GSD x Husky rescue stayed with some relatives for a few days while my boyfriend and I moved
house, and the relatives found an old choke chain to walk him with, instead of his Mekuti harness. We have had real
problems with him pulling on the lead, and the Mekuti harness has worked wonders, but didn’t eliminate the problem.
He hasn’t pulled when on the choke chain at all, even when it was first put on apparently, so we think that is what he was
trained on (and damage from that process is probably why his barks and whines sound a bit odd). We now think the
people who had him before us were violent towards him (if you mime kicking he runs across the room terrified), so, my
question is: he is best behaved on walks when wearing a choke chain, and it isn’t hurting him at all, but will the harm
chokers and the people using them caused him in the past affect him in any way now? I 100% DO NOT want to use a
choke chain on him and I don’t think anyone should ever, ever train a dog that way, but this chain is the only lead we’ve
used that has completely solved the problem with him pulling, and he hasn’t seemed stressed out by it. If he ever started
pulling on the lead so that the choker was tight, I would stop using it straight away. The last thing I want is my dog being
hurt. If I’d known these relatives were going to put one on him, I wouldn’t have let them look after him. Still, it works
and doesn’t hurt him, my only worry is it’ll do psychological damage somehow.


TL;DR: 3yr old husky rescue pulling on choke chain w/ no damage, but using it on walks occasionally causes him stress.
Can this damage actually cause harm in the future?


TL;DR: Best walks are on a choke chain but dog isn’t pulling on lead, has had no problems with pull before, not sure if
past violent treatment will affect him now, does it cause harm to dogs being on choke chains/when being walked on
them?


Summary A provides a more comprehensive and accurate portrayal of the post’s points, despite a few misplaced details,
whereas Summary B incorrectly states the dog is showing stress and pulling on the choke chain.


Summary B is preferred because it more accurately summarizes the main points of the user’s post by mentioning the
choke chain’s effect on walks, mentioning the absence of negative effects of pull so far, and raising concerns about past
violent treatment potentially affecting the dog’s behavior.


TL;DR: If you took a dog and had it wear chokes it’s self-choked, will it be traumatized in any way when you put it on
and no longer train it to be choked?


Summary B is preferred as it captures the key details more accurately, including the dog’s breed, the issue of pulling, and
the potential future harm, while Summary A is ambiguous and loss a lot of context.


Summary A is more accurate and detailed because it correctly describes the specific situation of the dog and the concerns
associated with using a choke chain, while Summary B is vague and confusing in its wording.
_Table 7._ Summaries generated by EXO and DPO given the post sampled from the TL;DR test set. GPT-4 consistently prefers EXO and
disprefers DPO over the chosen summary regardless of the order in which the evaluated pairs are presented.