Haozheng Luo _[†∗]_ [1], Jiahao Yu _[†∗]_ [2], Wenxin Zhang _[†∗]_ [3], Jialong Li _[†]_ [4],
Jerry Yao-Chieh Hu _[†]_ [5], Xinyu Xing _[†]_ [6], Han Liu _[†‡]_ [7]
_†_ Department of Computer Science, Northwestern University, Evanston, IL 60208 USA
_‡_ Department of Statistics and Data Science, Northwestern University, Evanston, IL 60208 USA
**Content Warning: This paper contains examples of harmful language.**
We introduce a low-resource safety enhancement method for aligning large language models
(LLMs) without the need for supervised fine-tuning (SFT) or reinforcement learning from human feedback (RLHF). Our main idea is to exploit knowledge distillation to extract the alignment
information from existing well-aligned LLMs and integrate it into unaligned LLMs in a plug-andplay fashion. Methodology, we employ delta debugging to identify the critical components of
knowledge necessary for effective distillation. On the harmful question dataset, our method significantly enhances the average defense success rate by approximately 14.41%, reaching as high
as 51.39%, in 17 unaligned pre-trained LLMs, without compromising performance.
[1robinluo2022@u.northwestern.edu](mailto:robinluo2022@u.northwestern.edu)
[2jiahao.yu@u.northwestern.edu](mailto:jiahao.yu@u.northwestern.edu)
[3wenxinzhang2025@u.northwestern.edu](mailto:wenxinzhang2025@u.northwestern.edu)
[4jialongli2024@u.northwestern.edu](mailto:jialongli2024@u.northwestern.edu)
[5jhu@u.northwestern.edu](mailto:jhu@u.northwestern.edu)
[6xinyu.xing@northwestern.edu](mailto:xinyu.xing@northwestern.edu)
[7hanliu@northwestern.edu](mailto:hanliu@northwestern.edu)
*These authors contributed equally to this work.

Decoupled Alignment for Robust Plug-and-Play Adaptation

Haozheng Luo
Jiahao Yu
Wenxin Zhang
Jialong Li
Jerry Yao-Chieh Hu
Xinyu Xing
Han Liu


We introduce **D** ecoupled **A** lignment for Robust **P** lug-and-Play **A** daptation (termed DAPA), a lowresource safety enhancement method for aligning large language models (LLMs). DAPA aligns
unaligned Large language models (LLMs) with ethical guidelines even without supervised finetuning (SFT) or reinforcement learning from human feedback (RLHF).
This innovation is practically urgent and important. LLMs have been widely adopted in various
applications recently, demonstrating their ability to generate high-quality human-like texts [Team
et al., 2024, Touvron et al., 2023, Ivison et al., 2023]. However, the security of these models
has become a significant concern due to the potential risks of generating harmful content [Wu
et al., 2024a, Yu et al., 2024, 2023a, Chao et al., 2023, Deng et al., 2023]. To align the LLMs
with ethical guidelines, researchers have developed various methods to enhance their safety. For
example, the Llama-2-Chat [Touvron et al., 2023] and Gemma-it [Team et al., 2024] models have
been extensively fine-tuned to improve their alignment performance. However, these methods
often require extensive computational resources or manual red-teaming, which can be costly and
time-consuming [Team et al., 2024, OpenAI, 2024, Bai et al., 2022, Ganguli et al., 2022]. Thus,
most of the LLMs finetuned from the pre-trained models by third-party developers do not undergo
the alignment process [Xu et al., 2024a, Chiang et al., 2023, Ivison et al., 2023], leaving them
vulnerable to generating harmful content by users with malicious intent. To combat these issues,
we seek motivations from knowledge distillation technologies [Xu et al., 2024b, Hahn and Choi,
2019], where a teacher model’s knowledge is transferred to a student model.
Specifically, through numerical experiments Figure 3 and Figure 4, we make two key detections:
- **MLP Alignment.** the alignment knowledge is stored in the Feed-Forward Network (FFN)
layers, also referred to as MLP layers,
- **Gate Alignment.** Further, the gate layer within MLP layers is the key component that determines whether the model’s output is aligned with ethical guidelines.
Focusing on these two key components, we propose to use memory editing technology [Meng
et al., 2022a,b] to transfer alignment knowledge from one aligned LLM to another unaligned
LLM within the same family. Surprisingly, by incorporating the _aligned_ gate layers into _unaligned_
LLMs, we dramatically improve their alignment performance. This technique differs from other
alignment strategies in two aspects: computational and manual resources for model alignment
through fine-tuning or reinforcement learning [Dai et al., 2024]. This work provides a timely,
robust and economic solution to LLM safety enhancement, aiming for more efficient and low-cost
alignment.
**Contribution.** We present a low-resource safety enhancement method for aligning LLMs.
Figure 1: **The Alignment Example of The** DAPA **on The Chinese-Alpaca-7B Model.**
- Methodologically, we propose DAPA (as shown Figure 1) to enhance the security of LLMs
against harmful prompts. DAPA utilizes memory editing technology to identify the memory
space responsible for alignment performance. Specifically, we present a delta debugging-based
search algorithm to address the challenge of pinpointing the memory space (gate layers) responsible for alignment performance. This allows us to locate the alignment-related modules
for memory editing via knowledge distillation. Then, we perfrom surgery migrating alignmentrelated modules from one aligned model to unaligned model to achieve cheap yet effective satefy
enhancement. As a result, DAPA achieves near-optimal alignment performance for LLMs without requiring extensive computational resources or fine-tuning.
- Empirically, we apply DAPA to 17 models from three popular LLM families (LLama2, Mistral, and Gemma), achieving significant safety enhancements with minimal computational effort
(adapting _at most_ 8.11% model parameters). Specifically, the DAPA-aligned models show an
average 14.41% increase in Defense Success Rate (DSR), with one of Gemma’s models achieving a dramatic increase of 51.39%. Surprisingly, we achieve this while largely maintaining the
core functionalities of the models. We evaluate the models’ performance using various assessment techniques, including cosine similarity scores, model perplexity, few-shot prompting, and
Chain-of-Thought (CoT) methodologies. The average degradation in perplexity is only 1.69,
and the average drop in model reasoning ability is only 2.59%.
**Organization.** Section2 includes a description of knowledge editing technology in our adaption. Section3 includes the algorithm to search critical memory space responsible for model alignment Section4 includes
the numerical evaluation results to show the efficiency of our adaption. Section5 includes a conclusion
and a discussion of future directions.


**Model Alignment.** With the developments of the LLMs [Team et al., 2024, Touvron et al., 2023,
Bai et al., 2023], concerns about their security have become significant [Weng and Wu, 2024,
Yu et al., 2023b]. Among these concerns, the potential risks of generating harmful content have
attracted the most attention. To counteract the potential risks of harmful outputs, developers often
engage in fine-tuning during safety training to decrease the likelihood of such outputs [Wu et al.,
2024a, Touvron et al., 2023, Ganguli et al., 2022, Bai et al., 2022]. Current methods for safeguarding LLMs against jailbreak attacks include Reinforcement Learning from Human Feedback
(RLHF), Direct Preference Optimization (DPO), and Supervised Fine-Tuning (SFT) [Rafailov
et al., 2023, Peng et al., 2023, Ouyang et al., 2022]. However, these fine-tuning methods are both
slow and costly. Many practitioners are exploring ways to lower the expenses associated with
alignment fine-tuning [Wang et al., 2024, Yao et al., 2023b], yet costs remain substantial.
**Memory Editing.** Knowledge editing focuses on altering specific behaviors of LLMs [Huang
et al., 2023, Meng et al., 2022a,b], and can be divided into three primary paradigms [Yao et al.,
2023a]. The first paradigm edits the memory during the inference stage [Wei et al., 2024, Zheng
et al., 2023, Mitchell et al., 2022], employing memory retrieval or in-context learning for modifications. The second paradigm adjusts model parameters and structures during the training stage
[Meng et al., 2022a,b]. The third paradigm utilizes associative memory models such as the Modern Hopfield Network [Hu et al., 2024a,b,c, Wu et al., 2024b,c, Hu et al., 2023, Ramsauer et al.,
2020] to edit model memory effectively. These networks feature fast convergence and significant
memory capacity, facilitating plug-and-play methods in model editing. Subsequent efforts utilize
knowledge editing to detoxify LLMs. [Wang et al., 2024] explores the use of contextual semantics
to allocate memory space, employing memory editing techniques to adjust the relevant memory
areas. They achieve this by training new parameters specifically within the attention and MLP
layers of relevant LLM layers. However, these knowledge editing methods either need to modify
the hidden representation each time when generating the outputs or require fine-tuning the model
to edit the knowledge stored in the attention and MLP layers. Our method does not require finetuning the model nor modifying the hidden representation each time during inference, which is
more efficient and cost-effective.


To locate the association of ethical memory in the parameters of an autoregressive LLM, we begin
by analyzing and identifying the specific hidden states that have the strongest correlation with
ethical memory.
An autoregressive transformer model _G_ is a type of language model that generates text by predicting the next token in a sequence given the previous tokens. For each input **X**, the model maps the
input to a sequence of tokens **S** = [ **s** 1 _,_ **s** 2 _, . . .,_ **s** _T_ ] _∈_ **X**, which are then fed into the transformer
model. Within the transformer model, the _i_ -th tokens are first embedded into a sequence of hidden states _h_ [(] _i_ _[l]_ [)][. The final output] _[ y]_ [ =][ decode][(] _[h]_ _T_ [(] _[L]_ [)][)][ is generated by the decoder layer from the last]
hidden state.
Figure 2: **The Architecture of Transformer Models.** We describe the architecture of Transformer models utilized by state-of-the-art LLMs such as Llama [Touvron et al., 2023] and Gemma
[Team et al., 2024]. Many of these models employ activation functions like SwiGLU [Shazeer,
2020] or GELU [Hendrycks and Gimpel, 2016] in their MLP layers. Each Transformer block
combines an attention mechanism with MLP layers (comprising Up, Gate, and Down modules).
Our figure illustrates the transition of the model’s hidden representation from the previous state to
the next state.
In Figure 2, we visualize the internal computation of _G_ as a grid of hidden states _h_ [(] _i_ _[l]_ [)][. Each layer] _[ l]_
(left _→_ right) of the transformer model adds a self-attention mechanism _a_ [(] _i_ _[l]_ [)] and local MLP _Mi_ [(] _[l]_ [)]
from previous layers. Recall that, in the autoregressive cases, tokens only draw information from
previous tokens:
**h** [(] _i_ _[l]_ [)] = **h** [(] _i_ _[l][−]_ [1)] + **a** [(] _i_ _[l]_ [)] [+] **[ M]** _i_ [(] _[l]_ [)] _[,]_ **a** [(] _i_ _[l]_ [)] = attn [(] _[l]_ [)] ( **h** [(] 1 _[l][−]_ [1)] _, . . .,_ **h** [(] _T_ _[l][−]_ [1)] )
**M** [(] _i_ _[l]_ [)] = **W** down [(] _[l]_ [)] _[σ]_ [(] **[W]** gate [(] _[l]_ [)] _[γ]_ [(] **[a]** [(] _i_ _[l]_ [)] [+] **[ h]** _i_ [(] _[l][−]_ [1)] ) _·_ **W** up [(] _[l]_ [)] _[γ]_ [(] **[a]** _i_ [(] _[l]_ [)] [+] **[ h]** _i_ [(] _[l][−]_ [1)] )
Each layer’s MLP is a three-layer neural network parameterized by **W** up, **W** gate, and **W** down, along
with a SwiGLU [Shazeer, 2020] or GELU [Hendrycks and Gimpel, 2016] activation function in
several popular LLMs, such as LLama [Touvron et al., 2023], Gemma [Team et al., 2024]. For
further background on transformers, we refer to [Vaswani et al., 2017].
**Storage of Alignment Knowledge.** We first use knowledge editing technology [Meng et al.,
2022a] to identify where the alignment knowledge is stored in the model. We use one unethical
question as a prompt to Llama-2-7B-chat. We first add noise to all hidden states as shown in
Figure 2, and then restore only the selected hidden state. We then measure the difference in output
probability between the corrupted run (adding noise to all hidden states) and the corrupted run
with one hidden state restored, referred to as the indirect effect of the selected hidden state. The
higher the indirect effect, the more critical the hidden state is to the model’s output probability.
We iteratively apply this process to all hidden states to identify the hidden states that have the
most significant impact on the model’s output probability, and show the results in Figure 3. We
could observe that the hidden states in the middle layers of the model have the most significant
impact on the model’s output, and the MLP layers have a higher indirect effect than the attention
layers. This aligns with the findings in [Meng et al., 2022a]. We show more numerical results in
Appendix. The results confirm that the alignment knowledge is mainly stored in the middle MLP
layers of the model.
Figure 3: **Visualizing Attention, MLP, and All Modules on Memory Space.** We visualize the
influence of unethical prompt tokens on the results using the aligned LLama-2-7B-chat model to
identify memory space. This includes examining the effects on attention, MLP, and all modules.
To better understand the impact of each module in the MLP layers towards the alignment knowledge, we customize the knowledge editing technology [Meng et al., 2022a] to visualize the indirect effects of different MLP modules: the gate, up, and down projections. We first use unethical
prompts and capture the last token’s last layer’s hidden representation of the unaligned model
(as the corrupted run in [Meng et al., 2022a]). Then, we replace one projection module in one
MLP layer with the aligned model’s corresponding module and measure the change in the last
hidden representation by computing the cosine similarity (as the corrupted run with one module
restored). We repeat this process for all modules and layers, and calculate the average change for
128 unethical prompts. The results are shown in Figure 4. We observe that the gate projection
has the most significant impact on the model’s last token hidden representation, followed by the
up projection. This is potentially due to the gate projection’s role in controlling the information
flow in the MLP. Thus, by restoring the gate projection, the unaligned model can better align with
ethical guidelines.


Although the gate layer within MLP layers is crucial for ensuring model responses adhere to
ethical guidelines from Section2, modifying all gate layers could degrade the original performance due
to a large number of parameter changes. We propose a strategy to efficiently identify the optimal
memory space for targeted modifications, enhancing alignment while preserving performance.
We incorporate delta debugging [Zeller and Hildebrandt, 2002] in our strategy. Delta debugging
|Gate Projection Up Projection Down Projection|Col2|Col3|
|---|---|---|
|

|

|

|
|~~0~~
~~5~~
|~~0~~
~~15~~
|~~20~~
~~25~~
~~30~~
|
Figure 4: **Impact of Different MLP Modules on Hidden Representation.** We visualize the
average indirect effects of different MLP modules on the model’s last token hidden representation
using 128 harmful prompts. Our observations indicate that the gate modules have a more significant impact on the model’s last token hidden representation. Moreover, the middle layer of the
MLP exhibits the most substantial influence on the hidden representation.
is a systematic approach that automates the debugging process by identifying the smallest set of
changes responsible for a program’s failure. It methodically reduces the set of changes, testing
progressively smaller subsets until pinpointing the precise cause of the failure. In DAPA, we
consider it a program failure when LLMs provide an unethical response to an unethical question.
To demonstrate how delta debugging works in DAPA, let **S** _∈_ S be a memory space where S is
the universe memory of all MLP modules. A policy is defined by the function _π_ : S _→{_ 0 _,_ 1 _}_,
where if _π_ ( **S** ) = 1, it indicates that the memory space **S** is beneficial for enhancing alignment,
and if _π_ ( **S** ) = 0, it indicates that the memory space **S** does not contribute to improving alignment.
Given an aligned model memory space **S** and policy _π_, we aim to find the smallest memory
space **S** _[∗]_ _∈_ S in the aligned model which can most efficiently improve the unaligned ability to
defend the jailbreak. In our case, we define _π_ ( **S** ) as the evaluation on a small set of additional
unethical questions ( _e.g.,_ 5% of preserved data). If the model provides ethical responses to all
these questions, we set _π_ ( **S** ) = 1; otherwise, _π_ ( **S** ) = 0.
We next briefly describe the delta debugging process in our aligner, as shown in Algorithm 1.
Given the input memory space of aligned model S, number of partition _n_ = 2 and a list of
memory space set _L_ of S. we first split the memory space into _n_ partitions. We then check if there
exists a partition _si_ such that _π_ ( _si_ ) = 1. If such a partition exists, we update the memory space to
_si_ and update _n_ = 2. Otherwise, we check if there exists a partition _si_ such that _π_ ( _L \ si_ ) = 1. If
such a partition exists, we update the memory space to _L \ si_ and set _n_ = _n −_ 1. If neither of the
above conditions are met, we double the number of partitions _n_ . We repeat this process until _n_ is
greater than the number of partitions in the memory space. Finally, we return the memory space
**S** _[∗]_ corresponding to the updated memory space _L_ . The worst-case complexity of this algorithm is
**Algorithm 1** Memory Search Algorithm in DAPA
**Require:** Aligned Model MLP Memory Space S
**Require:** A policy function _π_
**Ensure:** The smallest memory space **S** _[∗]_ for the editing
1: _L ←_ A List memory space set of S
2: _n ←_ 2
3: **while** _n ≤|L|_ **do**
4: _⟨s_ 1 _, . . ., sn⟩←_ split _L_ into _n_ partitions
5: **if** _∃i, π_ ( _si_ ) = 1 **then**
6: _⟨L, n⟩←⟨si,_ 2 _⟩_
7: **else if** _∃i, π_ ( _L \ si_ ) = 1 **then**
8: _⟨L, n⟩←⟨L \ si, n −_ 1 _⟩_
9: **else**
10: _⟨L, n⟩←⟨L,_ 2 _n⟩_
11: **end if**
12: **end while**
13: **return S** _[∗]_ corresponding to _L_
O( _|S|_ [2] ).
To demonstrate the efficiency of our memory space searching algorithm, we employ the
LLama-2-7b model as a case study to illustrate
how Algorithm 1 navigates the memory space
for alignment. The LLama-2-7b model consists of 32 MLP layers, resulting in a memory space S = 32. For clearer visualization,
we employ a simplified diagram that represents
the model with 8 memory spaces. Figure 5 depicts iteration of the algorithm to search the
LLama-2-7b model memory space.


|s1|s2|s3|s4|s5|s6|s7|s8|
|---|---|---|---|---|---|---|---|
|**_s_1**|**_s_2**|**_s_3**|**_s_4**|**_s_5**|**_s_6**|**_s_7**|**_s_8**|
|**_s_1**|**_s_2**|**_s_3**|**_s_4**|**_s_5**|**_s_6**|**_s_7**|**_s_8**|
|**_s_1**|**_s_2**|**_s_3**|**_s_4**|**_s_5**|**_s_6**|**_s_7**|**_s_8**|
|**_s_1**|**_s_2**|**_s_3**|**_s_4**|**_s_5**|**_s_6**|**_s_7**|**_s_8**|
|**_s_1**|**_s_2**|**_s_3**|**_s_4**|**_s_5**|**_s_6**|**_s_7**|**_s_8**|
|**_s_1**|**_s_2**|**_s_3**|**_s_4**|**_s_5**|**_s_6**|**_s_7**|**_s_8**|
|**_s_1**|**_s_2**|**_s_3**|**_s_4**|**_s_5**|**_s_6**|**_s_7**|**_s_8**|
|**_s_1**|**_s_2**|**_s_3**|**_s_4**|**_s_5**|**_s_6**|**_s_7**|**_s_8**|
_**s**_ **1** _**s**_ **2** _**s**_ **3** _**s**_ **4** _**s**_ **5** _**s**_ **6** _**s**_ **7** _**s**_ **8**
✘
✘
✘
✘
✘
Figure 5: **Example of LLama-2-7b Model**
**Memory Space Search.**
We perform a series of experiments to evaluate
DAPA in enhancing the alignment performance
of unaligned models against unethical prompts, in Section4.1. We also assess the impact of the DAPA
aligner on the model’s performance in Section4.2, including linguistic capabilities and reasoning abilities. Lastly, we conduct an ablation study to investigate the influence of the replacement layer in
Section4.3, including the model’s safety and overall performance.
**Models and Parameter Efficiency.** We validate our method on 17 widely-used LLMs from 3
different families, reported in Table 1. These models include both foundational and fine-tuned
models, with the fine-tuning approach including SFT, DPO, and RLHF. Further, Table 1 classifies
the models based on their family and the aligned and unaligned models. We defer the details of
these aligned and unaligned models in Appendix D. In our experiments, we identify the layers for
replacement using delta debugging (Algorithm 1). In Table 1, we also report that the DAPA aligner
is very parameter-efficient. DAPA not only updates an average of 6.26% of parameters accross 3
model families, it also updates as little as 3.25% parameters in the commonly used LLama-2-7b.
Table 1: **Model Families Employed in the Experiments.** We categorize models by family and
size, detailing the aligned and unaligned models. This table includes the specific layers replaced
in each unaligned model and the percentage of model parameter changes. The DAPA aligner alters
only an average of 6.26% of the model parameters, with as little as 3.25% change in parameters.
Family Size Aligned Model Unaligned Model Replace layers Average Parameter change


To evaluate DAPA’s effectiveness in aligning unaligned models, we substitute the relative memory in 17 models with DAPA’s configuration and assess their performance in defending against
jailbreak attacks. Because we set the response generation with deterministic, the variance of the
evaluation is 0. As a result, we only need to run one evaluation for each model.
**Dataset.** In our experiment, we use the AdvBench [Zou et al., 2023] to validate the performance
of DAPA. It is a benchmark dataset that contains various unethical prompts to evaluate the alignment of language models, encompassing different categories such as violence, hate speech, and
misinformation. We sample 128 prompts from the AdvBench dataset to form our evaluation
dataset.
**Metrics.** We employ the DSR as the primary metric to evaluate the alignment performance. For
each unethical prompt, if the model provides a refusal or an ethical response, it is considered
Table 2: **Comparing** DAPA **in 3 Common LLM Families.** We demonstrate the enhancement in
alignment abilities for unaligned models through our DAPA aligner across 17 models, measured by
the Defense Success Rate (DSR). Furthermore, we exhibit the linguistic performance of our model
post-application of the DAPA aligner. We report the average perplexity and Cosine Similarity score
with variance omitted as they are all _≤_ 2%. In every scenario, DAPA achieves a significant increase
in DSR, approximately 14.41% on average and up to 51.39%. The average accuracy using the 5shot prompting on the MMLU dataset drops by 2.06% and the average drop in perplexity is 1.69.
In all settings, DAPA delivers a significant improvement in DSR, without substantially affecting
the original capabilities of the unaligned model.
Family Model Name DSR Perplexity MMLU Cosine Similarity
Before After Before After Before After
aligned. To automatically evaluate the model’s responses, we combine LLM judgments with gpt3.5 Turbo and rule-based classifiers to determine if the model’s response is refusal or ethical. For
LLM judgment, we use the system prompt provided in [Yu et al., 2023a] to guide the LLM to
evaluate the response. For rule-based classifiers, we employ and expand the keyword list like
“Sorry, as an" or “As a responsible” provided in [Zou et al., 2023] to classify the response. If the
response contains any of the keywords, it is classified as a refusal. A response is deemed aligned
if it is classified as refusal or ethical by both the LLM and rule-based classifiers.
**Results.** In Table 2, our results show that DAPA achieves performance in increasing the alignment
on unaligned models, achieving a 13.77% average increase in DSR across all 17 models. Notably,
the _gemma-2b_ model achieves a significant 51.39% increase in DSR. These improvements in DSR
underscore DAPA aligner’s effectiveness in enhancing model safety against jailbreak prompts.


To assess the model’s performance before and after DAPA alignment, we evaluate the generative
and reasoning capabilities in a deterministic setting. For each pre-alignment and post-alignment
model, we measure the model’s generative ability using perplexity and assess the response variation caused by the DAPA alignment through cosine similarity score. We also validate the model’s
reasoning ability by employing real-life question-answering and STEM problem-solving tasks,
using Chain-of-Thought (CoT) [Wei et al., 2022] and few shot prompting approach. We conduct
each evaluation three times and present the average and standard deviation for each metric.
**Dataset.** We employ four real-world datasets: ShareGPT [Chiang et al., 2023], WikiText-2 [Merity et al., 2017], Big-Bench [et al., 2023] (TruthQA, General QA, SocialQA), and MMLU datasets
[Hendrycks et al., 2021]. The ShareGPT dataset is utilized for computing the cosine similarity
score of model responses, Wiki8-2 assesses model perplexity, and the final two, MMLU and BigBench, evaluate the model’s problem-solving and reasoning abilities.
**Metrics.** In our experiment, we evaluate the responses generated by both pre-alignment model
and post-alignment model. We use cosine similarity to measure the impact of the aligner on model
response generation. Additionally, we use perplexity for comparative analysis of the models’ generative capabilities. A high cosine similarity score or comparable perplexity indicates using our
aligner improves the defense success rate while maintaining the original performance. Additionally, to evaluate the model’s reasoning abilities, we administer real-life question-answering and
STEM problem-solving tasks, measuring performance with the Exact Match (EM) metric.
**Setup.** We assess post-alignment performance by examining reasoning capacity, response similarity, and perplexity. In all experiments, we use the model both before and after the adapter in a
deterministic output setting. In the response similarity test, we compare the average similarity of
Table 3: **Comparing** DAPA **with CoT Abilities in 3 Common LLM Families.** We present an
experiment assessing the impact on Chain of Thought (CoT) capabilities using DAPA, evaluated
through the Exact Match (EM) score. Our DAPA aligner reduces the average EM of the Chain of
Alignment (CoA) methodology on the Big-Bench datasets by 2.77%. This reduction indicates that
our DAPA aligner substantially impacts the original reasoning abilities of the unaligned model.
Family Model Name TruthQA GK SocialQA
Before After Before After Before After
responses on the same generated question. For comparing model responses, we embed responses
from both models using the text-embedding-3-small model [8] and analyze 128 questions sampled
from ShareGPT. In the perplexity test, we compute the perplexity score with Huggingface Evaluate [9] on Wiki8-2 dataset [Merity et al., 2017]. In assessing model reasoning capacity, we conduct
tests using 5-shot prompting on the MMLU dataset [Brown et al., 2020] and Chain-of-Action
(CoA) [Pan et al., 2024a,b] methodology on the Big-Bench dataset, excluding memory retrieval.
We conduct each evaluation three times and present the average and standard deviation for each
metric.
**Results.** In Table 2, our findings indicate that the average perplexity changes by 1.69, with the
LLama-2-13b model showing no change in perplexity. In one specail case, the Gemme 2b family’s
models display the most significant increase in perplexity, at 16.23. Additionally, the average
cosine similarity is 0.82, with Dolphin-2.6-mistral-7b-dpo achieving the highest similarity of 0.91.
Those indicate that the system does not adversely affect the original capabilities of the language
model. Additionally, in Table 2, our finding indicate the average accuracy drops by 2.06% using 5shot prompting on the MMLU dataset. Most models exhibit only slight changes in accuracy. The
only exception is gemma-2b and gemma-7b-ultrachat-sft experience significant drops of 19.77%
[8https://openai.com/blog/new-embedding-models-and-api-updates](https://openai.com/blog/new-embedding-models-and-api-updates)
[9https://huggingface.co/docs/evaluate/index](https://huggingface.co/docs/evaluate/index)
Table 4: **Influence of Different Sets of MLP modules.** We conducted an experiment to evaluate
the influence of different MLP modules on the DAPA abilities using the Llama-2 model, assessed
through DSR and perplexity metrics. The best results are highlighted in bold, and the secondbest results are underlined. Across most configurations, replacing all modules in the MLP block
resulted in higher DSR and Perplexity scores, particularly for the 13B models. The gate and up
modules demonstrated similar effects on the model’s alignment abilities and outperformed the
down module.
Model Name DSR Perplexity
gate (ours) all up down gate (ours) all up down
chinese-alpaca-2-7b 87.50 **92.97** 87.28 86.72 7.46 7.18 7.42 7.41
Llama-2-7b **42.19** 31.25 **42.19** 37.50 4.78 4.86 4.77 4.78
Llama-2-13b 46.09 **55.47** 39.06 36.72 4.28 4.41 4.28 4.28
chinese-alpaca-2-13b 85.16 **88.28** 85.12 82.81 5.60 5.61 5.60 5.58
Redmond-Puffin-13B 47.66 **100.00** 50.78 46.09 4.30 4.42 4.30 4.30
Table 5: **Influence of Different Positions Memory.** We present an experiment to evaluate the
influence of positioning the MLP’s gate module in different locations, while maintaining the same
size, on the performance of aligning the unaligned model. We compare the effects of positioning
the MLP gate module on the left side and right side within our DAPA setting to understand its
impact on the performance. The best results are highlighted in bold, and the second-best results
are underlined. Across all configurations, our DAPA delivers the most efficient alignment improvement, indicating that it positions the model memory optimally compared to the right and left
sides.
Model Name DAPA (ours) Left-most Right-most
DSR Perplexity DSR Perplexity DSR Perplexity
chinese-alpaca-2-7b **87.50** 7.46 85.16 7.46 82.81 8.05
Llama-2-7b **42.19** 4.78 35.16 4.78 35.16 4.79
Llama-2-13b **46.09** 4.28 38.28 4.28 36.72 4.30
chinese-alpaca-2-13b **85.16** 5.60 75.78 5.64 74.22 5.65
Redmond-Puffin-13B **47.66** 4.30 21.14 4.30 23.44 4.34
Figure 6: **The Influence of Different Memory Space Size on LLama-2 Model** We present an
experiment to assess how varying memory space influence the alignment ability of the LLama-2
model. We evaluate the LLama-2-7b and chinese-alpaca-2-13b model using DSR and perplexity
metrics across different memory spaces. As the memory space increases, the model’s alignment
capability also improves. The only exception is when more than 11 layers of the llama-2-7b model
are altered, leading to a significant deterioration in performance.
and 12.87%, respectively.
In Table 3, our results our results show a 2.77% average accuracy decrease using the CoA methodology on the Big-Bench datasets. In one exception, OpenHermes-2-Mistral-7B shows the most
significant improvement, achieving a 10% increase in accuracy on SocialQA dataset, while Gemmaalpaca-2B shows the largest decrease, with a 24% decrease on the SocialQA dataset.
Overall, these findings regarding models’ perplexity, responses’ cosine similarity, and performance on the real-life question-answering and problem-solving tests indicate that the DAPA aligner
does not significantly impair the models’ performance after using DAPA aligner.


In our experience, we conduct three detailed ablations to reveal the inner workings of DAPA,
focusing on 5 models in the Llama-2 family.
**Dataset.** Building on the methodologies described in Section4.1 and Section4.2, our ablation study utilizes
the AdvBench and WikiText-2 datasets.
**Metrics.** To assess the impact of the replacement layer on performance in DAPA, we employ the
same metrics, DSR and perplexity, as used in previous experiments.
**Impact of Various MLP Parts in** DAPA **.** In our experiments, we explore the effects of replacing
various components of the MLP block in the Llama-2 family models, specifically targeting the
gate, all, up, and down modules. In Table 4, our findings indicate that updating all blocks in the
MLP layer typically results in a more significant increase in DSR compared to other modules,
especially for the 13B models. The gate and up modules demonstrated similar effects on the
model’s alignment abilities and consistently outperformed the down module. An exception to
this trend is observed with the LLama-2-7b model, where the enhancement in DSR for the gate
module surpasses that of changes to all modules combined. Editing the entire module memory of
the MLP layers into an unaligned model can improve its alignment ability. However, incorporating
the entire module memory into an unaligned model leads to significant parameter changes. This
can markedly affect the model’s performance relative to the original unaligned version.
**Impact of Various Memory Module in** DAPA **.** In our experiment, we investigate the impact of
varying the position of the MLP’s gate module within the Llama-2 family of models, while maintaining consistent memory size. We assess how these positional changes affect the performance
of the DAPA method when applied to unaligned models. We compare the effects of positioning
the MLP gate module on the left side and right side within our DAPA setting to understand its
impact on the system’s performance. As indicated in Table 5, the alignment capability of DAPA
diminishes when the memory positions are shifted to the extreme left or right.
**Impact of Various Memory Length in** DAPA **.** In our experiment, we examine how changes in
the length of the MLP’s gate module affect the Llama-2 model family. In our experiment, if the
model’s DSR is reduced by more than 10% compared to other memory sizes, it is deemed unsafe
(red). Similarly, if the perplexity increases by more than 5% relative to other memory sizes, we
consider that the editing may let the model become a hallucination (yellow). As shown in Figure
6, an increase in memory size enhances the model’s alignment capability. Additional visualization
and experiment results are provided in Section E.2. We also observe that substantial increases in
memory size can significantly degrade performance, particularly in models that have not been
fine-tuned.

4.3 Ablation: Influence of Different MLP Module Setting


We introduce the Decoupled Alignment for Robust Plug-and-Play Adaptation, DAPA, which edits the unaligned model memory to enhance the model’s defenses against jailbreak attacks. This
method improves model alignment without the substantial computational expense typically associated with fine-tuning. It also efficiently identifies the optimal memory space for alignment.
Visualizations confirm that the ethical boundary of model alignment is predominantly situated
within the middle MLP’s gate layers. Empirically, DAPA achieves a 14.41% improvement in
model alignment, reaching up to 51.39% in one of the Gemma family models, with an average
parameter change of only 6.26%. Moreover, DAPA minimally impacts the model’s performance
in generation and reasoning tasks.
**Limitation and Future Work.** One limitation of our approach is the extent of memory space
editing required. Although the average memory modification across three family models is 6.26%,
popular model adapters like Lora [Hu et al., 2021] and QLora [Dettmers et al., 2023] typically
require only about 1% of parameter changes. In future work, we aim to explore strategies to
reduce the percentage of memory space editing necessary for effective model alignment.
**Broader Impact.** Our proposal improves LLMs’ defenses against jailbreak attacks. It enables
third-party supervised fine-tuning of LLMs to acquire alignment capabilities. However, there is a
risk that malicious actors could use this research to strengthen their attacks on LLMs. Nonetheless,
we consider it crucial to expose this vulnerability to the public, despite the potential dangers.


E.1 Aligned Model DSR Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
E.2 Additional Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . 18
E.3 Response Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
E.4 Experimental Details of LLMs Reasoning Performance . . . . . . . . . . . . . . 20
E.4.1 Prompt of CoA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
E.4.2 Performance Evaluation of LLMs Reasoning Abilities. . . . . . . . . . . 20

E Supplementary Material for Experiments 17


Our proposal improves LLMs’ defenses against jailbreak attacks. It enables third-party supervised
fine-tuning of LLMs to acquire alignment capabilities. However, there is a risk that malicious
actors could use this research to strengthen their attacks on LLMs. Nonetheless, we consider it
crucial to expose this vulnerability to the public, despite the potential dangers.


Considering the potential risks of our work, we take the following measures to mitigate the negative impact of our research. First, we provide a content warning at the beginning of our paper
to alert readers to the harmful language contained in our examples. Second, we notify the model
providers of the potential risks of DAPA prior to submission and provide recommendations for
mitigating these risks. Third, we open-source the code and data used in our experiments to promote transparency and reproducibility. Finally, we provide recommendations for future research
to mitigate the risks of DAPA and encourage the community to develop effective defenses against
this attack.


We perform all experiments using a single NVIDIA A100 GPU with 80GB of memory and a 12core Intel(R) Xeon(R) Gold 6338 CPU operating at 2.00GHz. Our code is developed in PyTorch
and utilizes the Hugging Face Transformer Library for experimental execution. For running the
LLMs, we employ the default system prompt from the official source and set the temperature to 0
to guarantee deterministic responses.

C Experiment System and Implement Settings


In our experiments, we categorize all unaligned models based on the fine-tuned techniques they
employ, as outlined in Table 6.
Table 6: **Links to Hugging Face Pages of Unaligned LLMs Used in The Experiments.**


RLHF OPENCHAT-3.5 [openchat/openchat_3.5](https://huggingface.co/openchat/openchat_3.5)
Foundation Model
LLAMA-2-7B [meta-llama/Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b)
LLAMA-2-13B [meta-llama/Llama-2-13b](https://huggingface.co/meta-llama/Llama-2-13b)
GEMMA-2B [google/gemma-2b](https://huggingface.co/google/gemma-2b)
GEMMA-7B [google/gemma-7b](https://huggingface.co/google/gemma-7b)
MISTRAL-7B [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)
MISTRAL-7B-FOREST-DPO [abhishekchohan/mistral-7B-forest-dpo](https://huggingface.co/abhishekchohan/mistral-7B-forest-dpo)
DPO DOLPHIN-2.6-MISTRAL-7B-DPO [cognitivecomputations/dolphin-2.6-mistral-7b-dpo](https://huggingface.co/cognitivecomputations/dolphin-2.6-mistral-7b-dpo)
SFT
GEMMA-ORCHID-7B-DPO [macadeliccc/gemma-orchid-7b-dpo](https://huggingface.co/macadeliccc/gemma-orchid-7b-dpo)
CHINESE-ALPACA-2-13B [hfl/chinese-alpaca-2-13b](https://huggingface.co/hfl/chinese-alpaca-2-13b)
CHINESE-ALPACA-2-7B [hfl/chinese-alpaca-2-13b](https://huggingface.co/hfl/chinese-alpaca-2-7b)
REDMOND-PUFFIN-13B [NousResearch/Redmond-Puffin-13B](https://huggingface.co/NousResearch/Redmond-Puffin-13B)
DOLPHIN-2.2.1-MISTRAL-7B [cognitivecomputations/dolphin-2.2.1-mistral-7b](https://huggingface.co/cognitivecomputations/dolphin-2.2.1-mistral-7b)
OPENHERMES-2-MISTRAL-7B [teknium/OpenHermes-2-Mistral-7B](https://huggingface.co/teknium/OpenHermes-2-Mistral-7B)
ZEPHYR-7B-ALPHA [HuggingFaceH4/zephyr-7b-alpha](https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha)
GEMMALPACA-2B [mlabonne/Gemmalpaca-2B](https://huggingface.co/mlabonne/Gemmalpaca-2B)
GEMMA-7B-ULTRACHAT-SFT [CorticalStack/gemma-7b-ultrachat-sft](https://huggingface.co/CorticalStack/gemma-7b-ultrachat-sft)


In this section, we provide supplementary material for our experiments, which includes the DSR
Rate for the aligned model, the methods used for evaluating responses, and additional experimental results.

E Supplementary Material for Experiments


We present the DSR rate of the aligned model in AdvBench [Zou et al., 2023] to demonstrate the
original performance of the aligned model in protecting LLMs against jailbreak attacks. We list
the model name and their Defense Success Rate (DSR) in Table 7.
Table 7: **The DSR Rate of Aligned Model** We detail the DSR performance across three model
families and five aligned models, focusing on the effects of our memory editing techniques.


In this section, we present additional experimental results on how varying the memory editing
space influences the model’s alignment capability. As shown in Tables 8 and 9, increasing the
memory space generally enhances alignment abilities in the Llama2 7b model. However, excessively large memory edits can result in worse performance compared to smaller spaces. Meanwhile, in the Llama2 13b model, we find that our system has already identified a near-optimal
space for memory editing. Also, we present additional experiments on the effects of varying
memory space sizes on the LLama-2 model in Figure 7.
Table 8: **Additional Results: The Influence of Different Memory Space in LLama2 7b Mod-**
**els.** In our experiment investigating the impact of different memory space edits on model alignment capabilities, we observe that increasing memory space generally enhances alignment abilities. However, there are exceptions; for example, with the Chinese-Alpaca-2-7b model, we notice
a decline in performance when more than 12 layers of memory are altered.


Table 9: **Additional Results: The Influence of Different Memory Space in LLama2 13b Mod-**
**els.** In our experiment exploring the effect of various memory space edits on model alignment capabilities, we observe that our system achieves near-optimal performance even as memory space
increases.
Model Name Memory Space Size
18 16 14 12 10 8 (ours) 6 4 2
Llama-2-13b 37.50 41.41 39.06 37.50 43.75 46.09 41.41 45.31 42.97
chinese-alpaca-2-13b 87.50 86.72 86.72 86.72 83.59 85.16 80.47 80.47 78.12
Redmond-Puffin-13B 57.81 55.47 56.25 49.22 46.77 47.66 36.22 32.81 25.78
Figure 7: **Additional Experiments on The Influence of Different Memory Space Size on**
**LLama-2 Model** We conduct an experiment to evaluate the impact of different memory space
capacities on the alignment capabilities of the LLama-2 model. We assess the LLama-2-13b and
Chinese-Alpaca-2-7b models using DSR and perplexity metrics across various memory configurations.


As previously discussed in Section4, we assess the responses generated by the LLMs using two distinct methodologies: black list keyword detection and GPT judgments. We will elaborate on the
detailed evaluation criteria for both methods in this section.
**Black List Keyword Detection .** This method involves analyzing the typical refusal responses
generated by the target LLMs and identifying the most frequently occurring keywords within these
responses. These keywords, which are listed in Table 10, serve as indicators for refusal responses.
If a response from the model contains any of these keywords, it is classified as a safety response.
**GPT Judgment .** For this method, we utilize GPT-4-turbo due to its efficient performance. We
outline the specific prompt used in Prompt E.2, where we instruct GPT to evaluate whether a
response is harmful. If the response is deemed harmful, GPT should output **1** . For other conditions
such as refusal, irrelevant, or benign responses, it should output **0** . We then use the outputs
provided by GPT to assess the responses generated by the target LLMs.


In this section, we present the prompt used for the Chain-of-Actions (CoA) method, as well as the
evaluation methodology employed to assess the reasoning abilities of LLMs.
**E.4.1** **Prompt of CoA.**
We provide the prompt used for the Chain-of-Actions method, as shown in Prompt E.3
**E.4.2** **Performance Evaluation of LLMs Reasoning Abilities.**
We employ GPT-3.5 Turbo [Bevilacqua et al., 2023] to evaluate the performance of LLMs in
reasoning tasks. The specific prompt used for this evaluation is detailed in Prompt E.4. This
allows us to assess the reasoning abilities of the LLMs.
Table 10: **Description of Keywords by Family.**

E.4 Experimental Details of LLMs Reasoning Performance


Llama2 I’m sorry; I apologize; As an; As a responsible; I’m just; I’m an;
I cannot; I must inform; Instead of; instead of; respectful; helpful
Gemma I am unable; cannot provide; i am unable; am not able;
important to remember; Disclaimer:; DISCLAIMER:;
important to note; illegal and harmful; inappropriate;
dangerous; potentially deadly act; crime
Other AI assistant; cannot assist; sorry; it’s illegal; cannot instruct;
against the policies; against the policy; is not possible; against the rules


**Rafael Rafailov** _[∗]_
Stanford University
```
rafailov@cs.stanford.edu
```
**Harshit Sikchi** _[∗]_
UT Austin
```
hsikchi@utexas.edu
```
**Yaswanth Chittepu** _[∗]_
UMass Amherst
```
   ychittepu@umass.edu
```
**Joey Hejna** _[∗]_
Stanford University
```
jhejna@cs.stanford.edu
```
**Ryan Park** _[∗]_
Stanford University
```
  rypark@stanford.edu
```

Scaling Laws for Reward Model Overoptimization in Direct Alignment
  Algorithms

Rafael Rafailov
Yaswanth Chittepu
Ryan Park
Harshit Sikchi
Joey Hejna
Bradley Knox
Chelsea Finn
Scott Niekum


Stanford University
```
cbfinn@cs.stanford.edu
```


Reinforcement Learning from Human Feedback (RLHF) has been crucial to the
recent success of Large Language Models (LLMs), however, it is often a complex
and brittle process. In the classical RLHF framework, a reward model is first trained
to represent human preferences, which is in turn used by an online reinforcement
learning (RL) algorithm to optimize the LLM. A prominent issue with such methods is _reward over-optimization_ or _reward hacking_, where performance as measured
by the learned proxy reward model increases, but true quality plateaus or even deteriorates. Direct Alignment Algorithms (DAAs) like Direct Preference Optimization
have emerged as alternatives to the classical RLHF pipeline by circumventing the
reward modeling phase. However, although DAAs do not use a separate proxy
reward model, they still commonly deteriorate from over-optimization. While
the so-called reward hacking phenomenon is not well-defined for DAAs, we still
uncover similar trends: at higher KL budgets, DAA algorithms exhibit similar
degradation patterns to their classic RLHF counterparts. In particular, we find that
DAA methods deteriorate not only across a wide range of KL budgets but also often
before even a single epoch of the dataset is completed. Through extensive empirical
experimentation, this work formulates and formalizes the reward over-optimization
or hacking problem for DAAs and explores its consequences across objectives,
training regimes, and model scales.


Recent advancements in Large Language Models (LLMs) have broadened their capabilities significantly, enabling applications in code generation, mathematical reasoning, tool use, and interactive
communication. These improvements have popularized LLMs across various domains. Reinforcement Learning from Human Feedback (RLHF) has been instrumental in these advances and is now
integral to sophisticated LLM training regimes [10, 55]. Before alignment, LLMs, trained on vast text
corpses to predict subsequent tokens [45, 8] are often unwieldy and hard to use. Today, leading LLMs
incorporate variants of the RLHF framework [14, 69, 36] to align them with human intent, which
_∗_ Equal Contribution, Dice Rolling
38th Conference on Neural Information Processing Systems (NeurIPS 2024).
generally involves a multi-stage process. Specifically, users evaluate model responses to assorted
prompts in order to train a reward model that encapsulates human preferences [10, 55, 72, 5, 62].
Then, the refined LLM maximizes the expected learned reward function using a reinforcement learning (RL) algorithm [50, 1, 65]. Despite its efficacy, this procedure is complex and computationally
intensive, particularly in its latter stages.
Goodhart’s Law [25, 11], that “when a measure becomes a target, it ceases to be a good measure”,
has often been cited as a core shortcoming of RLHF. Standard RLHF methods optimize a learned, but
imperfect reward function which ends up amplifying the reward model’s shortcomings. Empirically,
this phenomenon was first extensively characterized by Gao et al. [21], who coined the term “reward
over-optimization”, and has been seen consistently in recent findings [62, 16, 14]. While reward
over-optimization has been studied in the context of the aforementioned RLHF procedure, recent
contemporary methods for aligning LLMs circumvent the reward learning procedure, necessitating a
new characterization of the over-optimization phenomena.
This new broad class of algorithms, which we refer to as Direct Alignment Algorithms (DAAs),
bypass the traditional RLHF pipeline by re-parameterizing the reward model directly through the
optimal policy derived during the reinforcement learning phase. DAA methods, like Direct Preference
Optimization [46], have gained popularity [14, 28] as they often reduce computational demands. Yet,
despite not fitting a reward function, DAAs still exhibit over-optimization trends similar to those of
traditional RLHF methods using a learned reward function. In some sense, this is puzzling: DAAs
can be viewed as simply learning a reward function with supervised learning from which the optimal
policy is deterministically mapped, however more seems to be at play than simple supervised learning.
In this work, we investigate the over-fitting phenomena present in DAA algorithms through extensive
experimentation. First, we unify a number of different recent methods [46, 68, 4] under the DAA
framework. Then, across different model scales and hyper-parameters, we show that DAAs exhibit a
type of reward over-optimization consistent with that previously observed in RLHF [21]. Specifically,
we find that at different KL-divergence budgets DAAs exhibit degradation patterns similar to those
found in RLHF. Interestingly, we also find that performance within a single epoch is not always
as consistent as expected for DAAs. Finally, we explain why this happens by appealing to the
under-constrained nature of the optimization problem used in DAAs.


In this section, we first outline the core components of the standard RLHF pipeline [72, 55, 5, 41]).
Then, we examine prior literature to characterize the reward over-optimization exhibited by standard
RLHF methods. Finally, we provide a unifying view of direct alignment algorithms (DAAs) which
will guide our analysis of their training dynamics in the next section.


The standard RLHF pipeline consists of three distinct stages with the goal of aligning the LLM with
human preferences.
**Supervised Fine Tuning (SFT)** : First, a dataset of prompts _x_ and high-quality answers _y_ are used to
train an LLM for instruction following via maximum likelihood estimation over next-tokens. We
refer to the resultant model as _π_ SFT( _y|x_ ) and consider the entire prompt and answer strings to be
single variables.
**Reward Modeling** : Second, the SFT model _π_ SFT( _y|x_ ) is used to learn a reward function over human
preferences. Specifically, the SFT model is queried to produce pairs of answers ( _y_ 1 _, y_ 2) _∼_ _π_ SFT( _y|x_ ),
for every prompt _x_ in a dataset. Then, users select their preferred answers, resulting in ranking
_yw ≻_ _yl | x_ where _yw_ and _yl_ are the preferred and dispreferred answers respectively. Typically, user
rankings are assumed to be distributed according to the Bradley-Terry (BT) model [7]
exp ( _r_ ( _x, y_ 1))
_p_ ( _y_ 1 _≻_ _y_ 2 _| x_ ) = (1)
exp ( _r_ ( _x, y_ 1)) + exp ( _r_ ( _x, y_ 2)) [=] _[ σ]_ [(] _[r]_ [(] _[x, y]_ [1][)] _[ −]_ _[r]_ [(] _[x, y]_ [2][))]
where the preference distribution _p_ results from an unobserved latent reward _r_ ( _x, y_ ), and _σ_ is the
logistic function. Given this model and a dataset of rankings, denoted _D_ = - _x_ [(] _[i]_ [)] _, yw_ [(] _[i]_ [)] _[, y]_ _l_ [(] _[i]_ [)] - _Ni_ =1 [, we]
can train a parameterized model _rϕ_ ( _x, y_ ) to predict the unobserved reward using maximum likelihood
estimation. This yields the following loss function,
_L_ rew( _rϕ_ ) = _−_ E( _x,yw,yl_ ) _∼D_ �log _σ_ ( _rϕ_ ( _x, yw_ ) _−_ _rϕ_ ( _x, yl_ ))� _._ (2)
**Reinforcement Learning (RL)** : The final stage of the standard RLHF pipeline uses the learned reward
model _rϕ_ ( _x, y_ ) to update the LLM _πθ_ with an on-policy RL algorithm like PPO [50], optimizing the
model to provide responses more preferred by human raters. The most common objective is
max _πθ_ [E] _[x][∼D][,y][∼][π][θ]_ [(] _[.][|][x]_ [)]       - _rϕ_ ( _x, y_ )� _−_ _β_ DKL� _πθ_ ( _y | x_ ) _|| π_ ref( _y|x_ )� (3)
which enforces a Kullback-Leibler (KL) divergence penalty with a reference distribution _π_ ref( _y|x_ )
(usually taken to be _π_ SFT( _y|x_ )) to prevent the LLM _πθ_ from straying too far from its initialization.
Thus, the hyper-parameter _β_ directly trades off exploiting the reward function and deviating from
_π_ ref( _y|x_ ).

2.1 Reinforcement Learning From Human Feedback


Unfortunately, repeating the above procedure without careful tuning of the RL phase can lead to
disastrous performance. This is because in the context of RLHF the LLM policy is optimizing the
surrogate reward estimate _rϕ_ ( _x, y_ ) and not the true reward function as is often the case in other
domains. Thus, prior works have observed that while the LLM’s expected reward according to
eq. (3) increases the actual quality of the model’s outputs can decrease [54, 43, 9, 34]. This particular
instantiation of the reward exploitation or hacking problem [3] is often referred to as reward “overoptimization” in RLHF literature and has been studied empirically in both controlled experiments
[21] and user studies [14]. There are two prevailing explanations for why this phenomenon occurs.
**1. OOD Robustness:** In the classical RLHF pipeline, the RL objective (eq. (3)) is optimized using
on-policy samples from _πθ_ . This means that the reward function is continuously queried using unseen
model samples which are potentially out-of-distribution. Beyond the support of the reward modeling
distribution, _rϕ_ may assign high rewards to sub-par responses, leading the policy to believe it is doing
well when it may not be. While the KL-regularization term is designed to prevent the model from drifting too far out of distribution, this term alone has proven inadequate to prevent reward hacking [21].
**2. Reward Mis-specification.** Learned reward functions may exhibit spurious correlations that cause
them to prefer unintended behaviors. While this issue is not at the forefront of LLM research, it is
known to be pervasive in RL [43, 34]. Most efforts to address these problems exist at the intersection
of robustness and offline RL literature [13, 67, 16] and use measures of epistemic uncertainty to
penalize the predicted reward.


Due to its complex multi-step nature, recent works have sought alternatives to the classic RLHF
pipeline. A new class of algorithms, which we broadly classify as Direct Alignment Algorithms
(DAAs), directly update the LLM’s policy _πθ_ using user feedback instead of fitting a reward function
to it and then employing an RL algorithm. Perhaps the most known example is Direct Preference Optimization (DPO). DPO, as well as other DAAs, are derived using the closed form solution to the RLHF
objective in eq. (3) [71], _π_ _[∗]_ ( _y|x_ ) _∝_ _π_ ref ( _y|x_ ) _e_ _[r]_ [(] _[x,y]_ [)] _[/β]_, where _r_ ( _x, y_ ) is the ground-truth reward.
By isolating _r_ ( _x, y_ ) in this relationship and substituting it into the reward optimization objective in
eq. (2), we arrive at a general objective that allows us to train the LLM directly using feedback data:
              -               _L_ DAA ( _πθ_ ; _π_ ref ) = E( _x,yw,yl_ ) _∼D_ _g_ _β_ log _[π][θ]_ [ (] _[y][w][ |][ x]_ [)]
_[π][θ]_ [ (] _[y][w][ |][ x]_ [)] _[π][θ]_ [ (] _[y][l][ |][ x]_ [)]
_π_ ref ( _yw | x_ ) _[−]_ _[β]_ [ log] _π_ ref ( _yl | x_ )
_π_ ref ( _yl | x_ )
��
(4)
where _g_ is a convex loss function. Using _g_ ( _x_ ) = _−_ log _σ_ ( _x_ ) coincides with the standard
Bradley-Terry model and the original DPO objective. Other methods choose different loss functions:
IPO [4] uses the quadratic objective _g_ ( _x_ ) = ( _x −_ 1) [2] and SLiC-HF [68, 38] uses the hinge loss
_g_ ( _x_ ) = max(0 _,_ 1 _−_ _x_ ). Additional objectives were also considered in [59], but due to limited
computational resources, we focus on the three objectives outlined above.
Crucially, the DAA approach allows us to recover the optimal policy using a straightforward classification loss without the need for learning a reward function, on-policy sampling, or RL, which can be
Figure 1: Results on over-optimization in Direct Alignment Algorithms for DPO, IPO and SLiC.
Results show model win-rates over the dataset summary on an evaluation set of prompts as judged by
GPT-4. The top row shows the final performance after 1 epoch of training, while the second row also
includes 4 intermediate checkpoints as well. The fitted dotted curves utilize scaling laws from [21]
applied to direct alignment, with GPT4 winrates taking the place of the gold reward model score.
notoriously difficult to tune and computationally expensive. Because of this, DAAs have emerged as
a popular alternative. However, just like classical RLHF methods, DAAs exhibit strong over-fitting
and even reward-hacking like behaviors. For example, Park et al. [44] show that LLMs trained with
DPO generate responses with increasing length throughout the course of training, but do not improve
in ground-truth win-rate after a certain point. Since DAAs do not explicitly learn a reward function, it
is unclear how “reward-overoptimization” fits into the picture. In this work, we aim to shed some
light on this phenomenon in DAAs.


First, we examine the over-optimization problem in DAAs and compare it to those observed in traditional RLHF methods. All our experiments are carried out using the Reddit TL;DR summarization
dataset [55] and the Pythia family of Large Language Models [6]. Additional plots illustrating similar
over-optimization trends for Direct Alignment Algorithms on the Gemma2-2b model [61] and the
Anthropic Helpfulness-Harmlessness dataset [5] are provided in Appendix F

3 Empirical Analysis of Overoptimization in DAAs


In our first set of experiments, we evaluate the reward model over-optimization phenomenon. We
evaluate three training objectives DPO, IPO, and SLiC using seven _β_ parameters, representing
different KL budgets at three model sizes - 1B, 2.8B, and 6.9B. Our main results are shown in Fig. 1
which presents results for different configurations after 1 epoch of training (row 1) and including 4
uniform intermediate checkpoints (row 2). We include additional results on the training dynamics
in Fig. 2, which shows win rates and KL bounds for intra-epoch training. We present our findings
below.
**Model Over-Optimization:** We see clear over-optimization for all objectives as performance exhibits
a hump-shaped pattern, where an additional increase in the KL budget leads to decreasing model
performance. Moreover in Fig. 2 we observe similar intra-epoch training dynamics patterns as
configurations with wider KL budgets achieve their best performance after training on only 25% of
the data, after which performance starts decreasing in conjunction with increasing KL divergence
metrics.
Figure 2: Results on intra-epoch optimization dynamics. The top row shows win-rates against the
fraction of an epoch so far, while the bottom row shows the corresponding KL values. Under a lower
KL constraint, most experiments reach their best performance in the first 25% of the epoch and
degrade with additional training, while the model deviates from the reference under increasing KL.
All models are 6.9B and vary across DPO, SLiC, and IPO loss formulations.
**Effect of Training Objective:** In the IPO work [4] the authors present theoretical arguments that
due to the monotone sigmoid objective in the DPO formulation, the KL constraint is not effectively
enforced and propose the quadratic fixed-margin loss as an alternative. Across all objectives, there
are clear dependencies between the _β_ parameter and the corresponding KL achieved at the end of
training. While DPO and SLiC exhibit similar performance, IPO indeed seems to be less prone to
over-optimization and in general, achieve lower KLs under the same constraint. Our observations
with IPO also align with prior works in preference-based RL and imitation learning where imposing
a fixed margin led to more stable and performant methods [48, 51].
**Effect of Model Size:** The results also show a strong parameter count scaling effect. The Pythia 1B
model achieves low performance under the same set of constraints it reaches much higher KL values,
while almost immediately exhibiting signs of over-optimization. This behavior holds under all three
objectives. At larger scales, the 6.9B Pythia model tends to exhibit more win-rate - KL trade-offs and
be less prone to over-optimization, with both models significantly outperforming the 1B model. In
the case of the IPO objective, the 6.9B also exhibits significantly better control over the KL objective
and shows little to no over-optimization behavior.


Given we have established a framework for evaluating over-optimization in DAAs and empirically
validated it (section 3.1), we now develop scaling laws for this phenomenon. Previous work in
classical RLHF has established such scaling laws for reward model scores as a function of the KL
divergence between initial and optimized policies [21]. The relevant functional of the reward _R_ ( _d_ ) is
_R_ ( _d_ ) = _d_ ( _α −_ _β_ log _d_ ) (5)
where _α, β_ are constants dependent on the size of the reward model dataset and parameter count,
and _d_ = - _D_ KL( _π||π_ ref). As DAAs do not train a proxy reward model, we treat GPT4 winrates over
dataset completions as a proxy for gold reward. Somewhat surprisingly, we find that this scaling law
accurately relates _d_ and winrates for DAAs. Compared to a quadratic fit between _D_ KL( _π||π_ ref) and
winrates, this scaling law halves the RMSE. It is worth noting, however, that a quadratic fit between
_d_ and winrates yields a similar error compared to Equation 5.
Figure 3: **Left:** KL budget versus win-rates (over dataset human answer) with and without lengthregularization [44]. While including a length correction in the optimization objective changes the
KL-win-rate Pareto Frontier, it does not alleviate reward over-optimization and might even exacerbate
it. **Right:** Scaling behavior for length extrapolation - smaller capacity models (either by size or KL
budget) extrapolate more strongly on simpler features such as length.


Prior work [44] has shown that the DPO algorithm is prone to length exploitation as it amplifies
verbosity biases in preference datasets. Here we show that length is not the only dimension on which
exploitation can occur. Our experimental results are shown in Fig. 3. On the left, we show results
for the 2.8B Pythia model with standard training plus the length-regularization approach from [44].
Both approaches suffer from over-optimization, but the dynamics differ depending on the KL budget.
Moreover, even though the regularized model achieves higher win rates on a length-correct basis,
it under-performs the model trained with the standard objective in the lower KL constraint region.
Recent work [27] has also shown that DAAs prioritize features of the data based on their complexity
and prevalence (with length a clear example of human datasets). [44] further showed that models
trained with the DPO algorithm extrapolate significantly based on length. We extend this analysis in
Fig, 3 (right). We consider a linear regression of the form
log _[π][θ]_ [(] _[y]_ [(] _[i]_ [)] _[|][x]_ [(] _[i]_ [)][)] (6)
_πref_ ( _y_ [(] _[i]_ [)] _|x_ [(] _[i]_ [)] ) [= ˆ] _[γ][|][y]_ [(] _[i]_ [)] _[|]_ [ +] _[ ϵ]_ [(] _[i]_ [)]
where _x_ [(] _[i]_ [)] are held-out prompts and _y_ [(] _[i]_ [)] are samples from the corresponding model between the
DPO implicit reward and length. We fit a different regression for each model size and checkpoint
and plot the corresponding _R_ [2] values. We observe two main effects; first, there is a clear scaling
law behavior. Weaker models extrapolate across the simple length feature to a much higher degree
than stronger ones. This is especially clear when comparing the behavior of the Pythia 1B versus the
2.8B and 6.9B models. Second, we see significant effects based on the KL budget - under a smaller
budget all model sizes exhibit higher extrapolation behavior. Based on these results we formulate the
hypothesis that under limited capacity, either from model capability or limited KL budgets, the model
will extrapolate more strongly based on simpler features, which can lead to OOD issues.


Prior works have measured reward model quality in ranking settings by classification accuracy. We
evaluate the relationship between the DAA implicit reward model accuracy and policy performance
in Figure 4. The DPO and SLiC algorithms exhibit little to no correlation between reward model
accuracy and downstream model performance. The IPO model shows a weak positive relationship, but
upon further examination, this is entirely due to model size scaling - stronger models both fit the data
better and produce better generations as well, however within each particular model size, there is no
discernible relationship between the DAA implicit reward accuracy and the actual policy performance.
Similar observations hold when comparing the empirical DAA loss with model performance, which
is contrary to observations in supervised pre-training and instruction tuning [30].
Figure 4: **Top:** We plot the DAA implicit reward accuracy in preference classification versus win
rates. **Bottom:** DAA optimization loss versus checkpoint win rates. Model training statistics, do not
exhibit a strong relationship with downstream performance.


A number of recent works have observed that the implicit DAA rewards of both preferred and
dis-preferred responses decrease during training, which may be counter-intuitive. In [47] the authors
make a counter-point that in offline training of DAAs _π_ ref is usually pre-trained with SFT on the
preferred response and thus
_≈_ E _π_ ref( _yw|x_ )
log _[π][θ]_ [(] _[y][w][|][x]_ [)]
_π_ ref( _yw|x_ )
= _−_ DKL� _π_ ref( _y|x_ ) _|| πθ_ ( _y | x_ )� (7)
E _pD_ ( _yw|x_ )
log _[π][θ]_ [(] _[y][w][|][x]_ [)]
_π_ ref( _yw|x_ )
where _pD_ ( _y_ _[w]_ _|x_ ) is the dataset distribution of preferred answers. That is the expected implicit reward
represents a forward KL divergence between the reference policy and the optimization policy, thus it
is expected to be negative and decrease with training as the optimization model moves away from
the reference. In this section, we study whether this empirical phenomenon presents a challenge for
DAA learning. Similar to Fig. 1 we plot the win rates against the square-root-transformed (negative)
expected implicit reward of the preferred response (evaluated on a held-out evaluation dataset), which
as stated above approximates the (square-root-transformed) forward KL DKL� _π_ ref( _y|x_ ) _|| πθ_ ( _y | x_ ).
Results are included in Fig. 5, which follow closely the pattern in Fig. 1 with performance initially
increasing before it starts dipping down after a certain threshold. This indicates that under the
standard DAA training pipeline decreasing likelihoods are not necessarily an issue for performance,
and are even necessary for improvement, but exhibit non-linear over-optimization dynamics.

3.5 Decreasing Likelihoods and Model Performance


While the phenomena observed in the previous section echo those observed in classical RLHF, their
underlying causes may be distinct. Reward over-optimization in classical RLHF is largely attributed to
querying a proxy reward function that is potentially OOD, while DAAs do not train a separate reward
model. Instead, DAAs are generally understood as fitting an “implicit” reward model to preference
data with the parameterization _rθ_ ( _x, y_ ) = _β_ log _π_ _[π]_ ref _[θ]_ [(] ( _[y]_ _y_ _[|]_ _|_ _[x]_ _x_ [)] ) [using the objective in eq. (2). Therefore, the]
OOD behavior of the policy is inextricably linked to the OOD behavior of the implicit reward model.
We demonstrate below that the reward modeling objective used is heavily under-constrained, allowing
for a potentially large number of solutions that can place weight on OOD responses. This is especially
problematic for DAAs which deterministically map the optimal policy from the “implicit” reward.
**Rank Deficiency with Finite Preferences.** In DAAs, the language modeling problem is treated
as a contextual bandit. However, the space of possible prompts _x ∈X_ and answers _y ∈Y_ are both
_√_
Figure 5: Over-optimization results for Forward KL vs. winrates. The top row shows the final
performance after 1 epoch of training, while the second row also includes 4 intermediate checkpoints.
The fitted dotted curves are scaling laws from [21] applied to DAAs, with GPT4 winrates taking the
place of the gold reward model score.
exponentially large in sequence length. However, as highlighted by Tang et al. [59], DAAs often
assume full support of the reference distribution when mapping from the implicit reward to the
optimal policy _π_ by eq. (10). However, in practice such coverage is impossible. Instead, preference
datasets cover a minuscule portion of the prompt-response space. Unfortunately, as DAA objectives
are not strictly convex, their loss functions (eq. (4)) can have multiple global optimas, which may
be undesirable. We demonstrate this below, using the regression interpretation from Hejna et al. [23].
First, we re-write the DAA objective from eq. (4) using vectors in the prompt-response space _X × Y_ .
Each preference query in the comparison dataset can be written as difference between indicator
vectors, specifically _qi_ = 1 _{_ ( _x, y_ ) = ( _x_ [(] _[i]_ [)] _, yw_ [(] _[i]_ [)][)] _[} −]_ [1] _[{]_ [(] _[x, y]_ [) = (] _[x]_ [(] _[i]_ [)] _[, y]_ _l_ [(] _[i]_ [)][)] _[}]_ [. This “query” vector]
simply selects the comparison from the prompt response space, with the entree corresponding to
( _x, y_ _[w]_ ) being +1 and the entree corresponding to ( _x, y_ _[l]_ ) being -1. Similarly, we can consider the
learned policy to be a vector log _π −_ log _π_ ref _∈X × Y_, to which the distributional constraint also
applies in practice. Our generalized DAA loss function can then be re-written as
_L_ DAA( _πθ, D_ ) = [�] _i_ _[|D|]_ =1 _[g]_ - _βqi_ _[⊤]_ [(log] _[ π]_ [(] _[y][|][x]_ [)] _[ −]_ [log] _[ π]_ [ref][(] _[y][|][x]_ [))] - _,_ where _qi_ [ _x, y_ ] =

1 if ( _x, y_ ) = ( _x_ [(] _[i]_ [)] _, yw_ [(] _[i]_ [)][)]
_−_ 1 if ( _x, y_ ) = ( _x_ [(] _[i]_ [)] _, yl_ [(] _[i]_ [)][)]
0 otherwise
with finite data. Choosing _g_ to be the negative log sigmoid above recovers DPO with finite preferences,
but also logistic regression with a data matrix _Q_ of shape _|D|_ by _|X × Y|_ constructed by stacking
the aforementioned query vectors _q_ . As _|X × Y| >> |D|_, this matrix is likely to have a non-trivial
null space, making the problem not strictly convex. Thus, there are many possible policies _π_ that can
achieve the same optima, some of which will place a high weight on out-of-distribution responses due
to the distributional constraint of policy [23, 70]. To demonstrate this, we formalize the construction
below.
**Proposition 1** _(Adapted from Hejna et al. [23]) Let S be the set of win-or-lose prompt-response_
_vectors_ ( _x, y_ ) _in D. Provided:_
_1. The intersection of the null space of Q, N_ ( _Q_ ) _, and the span of S, span_ ( _S_ ) _, is non-trivial._
_2. For every x there exists a response yOOD ∈Y that is not in the data,_ ( _x, yOOD_ ) _̸_ = _S._
_Then, there are infinite number of minima to eq._ (4) _which place weight on out-of-distribution_
_responses y._
Figure 7: (Top row) Probability of OOD trajectories. DAA algorithms end up placing a substantial
probability mass of some of the OOD trajectories during training. (Bottom row) Probability of
in-distribution (preference-pair) trajectories decreases during training.
.
_Proof._ Let ˆ _π_ be the minima of the DAA loss function. Choose a vector _u_ such that _u ∈_ _N_ ( _Q_ ),
_u ∈_ span( _S_ ), and _u_ has at least one negative component. Modifying the log policy vector as log ˆ _π_ + _u_
will not affect the DAA loss, as _u_ is in the null space of _Q_, but the log-probability of the policy
will decrease for least one prompt-response pair in _S_ by construction. However, _e_ [log ˆ] _[π]_ [+] _[u]_ may not
integrate to one. To fix this, we can construct a second vector _v ∈_ _N_ ( _Q_ ) using the _y_ OOD at each
_x_ such that _e_ [log ˆ] _[π]_ [+] _[u]_ [+] _[v]_ integrates to one. For more details, we refer the reader to Hejna et al. [23]
Appendix A.3.
The second constraint of proposition 1 is often trivially satisfied by the dimension of the response
space as we are unlikely to see _every_ response to a prompt. The first constraint is harder, but can be
satisfied by conflicting preferences. A trivial example which satisfies these constraints is a simple
MDP in which there is only a single state (or prompt _x_ ), but three possible actions (or responses),
_y_ 1, _y_ 2, and _y_ 3. If we construct the preference dataset _D_ = _{_ ( _y_ 1 _≻_ _y_ 2) _,_ ( _y_ 2 _≻_ _y_ 1) _}_, omitting _x_
for brevity, then we satisfy the above conditions: the null space of _Q_ is non trivial in span of _y_ 1
and _y_ 2 and there is an out-of-distribution action _y_ 3. In this setting, the DPO loss is minimized by
both ˆ _π_ ( _y|x_ ) = (0 _._ 5 _,_ 0 _._ 5 _,_ 0) and ˆ _π_ ( _y|x_ ) = (0 _._ 0 _,_ 0 _._ 0 _,_ 1 _._ 0). In fact, it is minimized by infinitely many
policies which place equal weight on _y_ 1 and _y_ 2. To demonstrate this effect in higher dimensions
across a number of different DAA methods, we conduct experiments in a Toy MDP which bears
resemblance to the language modeling setting.
**Understanding OOD behavior for DAA algorithms with a Toy MDP** : To illustrate that DAA
algorithms, in general and not an artifact of training LLM’s, end up placing probability mass on
OOD sequences during training we design a simple Tree MDP (shown in Figure 6) to mimic the
token-level MDP in LLMs. We use a dataset containing a single preference between two trajectories
and follow the standard procedure of running SFT on preferred responses before updating an RNN
policy using a DAA. Figure 7 shows that even in this simple setup, popular DAAs (DPO/IPO/SLiC)
end up extrapolating incorrectly out of distribution revealing a fundamental shortcoming. Unlike in
standard RLHF, the non-strict convexity of the reward function in DAAs ends up directly affecting
the policy. Detailed experimental details can be found in Appendix E.

4 Reward Exploitation in Direct Alignment Algorithms


Broadly, over-optimization has been a widely studied phenomenon across different settings [60, 18].
Over-fitting can be characterized as over-optimization in the supervised learning setting [39, 32],
which can harm generalization [19, 12, 24] or lead to susceptibility to adversarial attacks [56, 37, 15].
Reward hacking in reinforcement learning (RL) [54], where an agent maximizes its reward through
behavior that deviates from the intended goal, can be viewed as a different type of over-optimization,
commonly observed in prior work [43, 3, 22].
We study over-optimization in the context of
aligning LLMs with human feedback, for which
the most common approach is RLHF as outlined
in section 2.1. Similar RLHF techniques were
originally pioneered for control [31, 2, 10].
Standard RLHF methods suffer from both
potential over-fitting of the reward function
and reward exploitation by the RL algorithm.
Several works have considered how to reduce
over-fitting or increase the robustness of learned
reward functions using ensembles [13, 67, 16]
or data smoothing [70]. Other approaches,
like Moskovitz et al. [40] consider how reward
exploitation can be reduced by using different
optimization techniques in the RL stage. Much
of this work is motivated by Gao et al. [21],
which first characterized and provided scaling
laws for over-optimization in RLHF.
Figure 6: An illustration of the Tree MDP. At each
state, we can choose one of 3 actions ( _a_ 0 _, a_ 1 _, a_ 2),
which deterministically maps to the next state. Furthermore, all the leaf nodes in this tree MDP, transition to the terminal absorbing state _s∞_, irrespective
of the chosen action
Unlike Gao et al. [21], we consider the over- tion to the terminal absorbing state _∞_, irrespective
optimization problem in DAAs, which differs of the chosen action
significantly from the standard RLHF pipeline.
Different DAAs have been derived theoretically
[47, 46, 68, 4, 64], and applied to problems beyond language modeling like image generation [63]
and control [23]. In all of these scenarios, over-optimization problems have persisted. Park et al. [44]
show that DAAs commonly over-fit to length and the expense of performance, which has been linked
to inherent bias in training data [53, 29]. Other works have tried to allow DAAs to use more types
of data like demonstrations [49] or ratings [17] to get better performance. Recently, incorporating
online data has proven critical to improving performance [66, 26, 57]. Concurrent to our work, Tang
et al. [58] study the differences between offline DAAs and standard RLHF methods. Unlike us, they
focus on comparisons with online sampling whereas we focus on the purely offline setting.


In this work, we present an analysis of the over-optimization problem in Direct Alignment Algorithms.
Through extensive experimentation on different algorithms (DPO, IPO, SLIC) and at different model
scales (1B, 2.8B, 6.9B), we observe consistent over-optimization trends at different KL-divergence
budgets. While our analysis is a first step, it is not a complete picture of understanding the overoptimization phenomena. More work can be done characterizing this effect at larger model scales,
which we were unable to do due to computational limitations. Nevertheless, we believe our work
sheds light on important problems in Direct Alignment Algorithms that can spur future research.


Our discussion highlights a number of issues with direct alignment algorithms used widely as means
to align to human values. This work has mostly focused on pointing out those issues along with a
theoretical underpinning of the issue but does not provide a way to resolve these issues. We still
assume an underlying model of human preferences, which is an ongoing research area as no model is
perfect in explaining the ways humans give preferences. Our work aims to drive the push towards
better alignment algorithms that do not overoptimize and generate models that are safe to be deployed
in our society. We believe only through understanding and demonstrating the shortcomings of current
methods we can develop better alignment methods.


We largely follow the DPO setup unless otherwise mentioned and build on their code
[(https://github.com/eric-mitchell/direct-preference-optimization) without changing any hyperparame-](https://github.com/eric-mitchell/direct-preference-optimization)
ters unless otherwise mentioned.
For all DAA experiments, we used the curated OpenAI TL;DR dataset with 92K preferred-dispreferred
summary completions [55]. Each prompt is a Reddit post belonging to one of several topic forums,
with title/post metadata included. 256 prompts sampled from the held-out set are used for all
evaluations (e.g. loss, accuracy, KL, winrates, length), with temperature 1 _._ 0 and max length 512.
Model sizes include 1B, 2.8B, and 6.9B and were initialized from the base Pythia pre-trained weights.
All models underwent supervised fine-tuning on TL;DR prior to direct alignment. Across all SFT
and DAA runs, we used a batch size of 128 (8 gradient accumulation steps), and RMSProp with a
learning rate of 0 _._ 5 _×_ 10 _[−]_ [6] (linear warmup for 150 steps) for 1 epoch. 1B models were trained on 2
NVIDIA A40 GPUs, 2.8B models were trained on 4 NVIDIA A40 GPUs, and 6.9B models were
trained on 4 NVIDIA A100 GPUs. All evaluations were computed with "gpt-4-turbo-2024-04-09" as
judge, with random positional flips to avoid known bias.


This appendix contains similar intra-epoch KL divergence and winrate evolution results as in Fig. 2,
across all model sizes.
Figure 8: KL divergence and GPT4 winrate evolution for 2.8B and 1B models across DPO, SLiC, and
IPO losses. Similar to the 6.9B models, performance tends to degrade after the first quarter epoch,
particularly under a low KL budget, while KL increases almost monotonically.

C Appendix A: Complete Intra-Epoch Training Dynamics


Reward over-optimization is well understood in the classical RLHF setting, with a consensus that
is driven by two main components - using a proxy reward function that is trained on limited data
and continuous querying with new, potentially OOD samples during PPO training. At first glance,
none of these conditions hold in DAAs as we do not train a separate proxy reward model or generate
new data during training. Therefore, understanding reward over-optimization in DAAs requires a
new theory. We will base our analysis on [47] using the token-level MDP and corresponding (soft)
Q-learning formulation. Consider the class of dense per-token reward functions _rθ_ ( _x, y≤i_ ), where
_y≤i_ denotes the first _i_ tokens of _y_, with sequence level-reward _rθ_ ( _x, y_ ) = [�] _i_ _[|][y]_ =1 _[|]_ _[r][θ]_ [(] _[x, y][≤][i]_ [)][. This is a]
strictly more general class than the sparse reward function which returns a single score at the end of
the sequence since we can set all intermediate rewards as 0. Within the framework of [47] given a
DAA-trained policy _πθ_, there exists a dense per-token reward _rθ_, that minimizes the reward modeling
objective in Eq. 2 and satisfy the below.
The (soft) Bellman Equation holds:
      - _r_ ( _x, y≤i_ ) + _β_ log _π_ ref( _yi|_ ( _x, y<i_ )) + _V ∗_ (( _x, y≤i_ )) _,_ if _yi_ is not **EOS**
_Q_ _[∗]_ ( _yi,_ ( _x, y<i_ )) = (8)
_r_ ( _x, y≤i_ ) + _β_ log _π_ ref( _yi|_ ( _x, y<i_ )) _,_ if _yi_ is **EOS**
where _V_ _[∗]_ is the corresponding soft-value function:
_V_ _[∗]_ (( _x, y<i_ )) = _β_ log          - _e_ _[Q][∗]_ [(] _[y,]_ [(] _[x,y][<i]_ [))] _[/β]_ (9)
_y∈|V |_
then the DAA policy _πθ_ satisfies:
_πθ_ ( _yi|_ ( _x, y<i_ )) = exp( [1] (10)
_β_ _[Q][∗]_ [(] _[y][i][,]_ [ (] _[x, y][<i]_ [))] _[ −]_ _[V][ ∗]_ [((] _[x, y][<i]_ [)))]
in this interpretation, the LLM logits _lθ_ [ _i_ ] = _Q_ _[∗]_ ( _yi,_ ( _x, y<i_ )) _/β_ represent Q-values. With a direct
substitution, we then have
_Q_ _[∗]_ ( _yi,_ ( _x, y<i_ )) = _r_ ( _x, y≤i_ ) + _β_ log _π_ ref( _yi|_ ( _x, y<i_ )) + _β_ log - _e_ _[Q][∗]_ [(] _[y,]_ [(] _[x,y][<i]_ [))] _[/β]_
_yi∈|V |_
~~�~~ ~~�~~                          - ~~�~~
OOD bootstrapping
(11)
That is in this framework DAAs may suffer from the classical OOD bootstrapping issue in offline
RL [20, 35, 33, 52]. In this case, even though the objective is trained fully offline we still effectively
query the model on the values of unseen tokens. This interpretation also provides further insight into
the effect of the _β_ coefficient and the training dynamics. For small values of beta the estimate
_β_ log         - _e_ _[Q][∗]_ [(] _[y,]_ [(] _[x,y][<i]_ [))] _[/β]_ _≈_ max (12)
_y∈|V |_ _[Q][∗]_ [(] _[y,]_ [ (] _[x, y][<i]_ [))]
_yi∈|V |_
that is smaller parameter values yield a more optimistic estimate, which results in a higher level of
OOD bootstrapping. This interpretation would also explain the somewhat counter-intuitive results of
section 3.4. While the implicit reward function can adequately fit and model the data, the resulting
LLM might behave sub-optimally, due to OOD bootstrapping in the corresponding Q-value estimate.

D Overoptimization from the lens of Implicit Bootstrapping


We have established that common DAA objectives allow for placing a high likelihood on OOD
data. In practice, while one might expect the likelihood of preferred responses to increase during
training, it has been observed that algorithms like DPO decrease the likelihood of both the preferred
and dis-preferred responses [42]. In fact, this is expected from a max-entropy RL perspective [47].
Since the total probability mass must sum to one, the probability of OOD responses must increase
during the course of training. A small amount of extrapolation may be necessary to reach the optimal
policy, however, too much is potentially detrimental to performance. Because they are not adequately
constrained to the reference distribution, current DAA objectives allow this to happen.
To understand how DAAs allocate probability mass out of distribution, we use a toy Markov Decision
Process (MDP), that mimics the LLM setting. The MDP is modeled as a tree, originating from a
single start state, featuring deterministic transitions. The Toy MDP is illustrated in fig. 6.

E Understanding Behavior of DAAs on OOD sequences


The MDP is modeled as a tree, originating from a single start state. This configuration mirrors the
token-level MDP in Direct Preference Optimization (DPO) [47], or the scenario where both preferred
and dispreferred responses are conditioned on the same prompt in the broader Large Language Model
alignment context. Each leaf node in the MDP transitions deterministically to a terminal absorbing
state, regardless of the action taken. The deterministic transitions resemble the LLM setting, where
the current state is represented by the sequence of encountered tokens ( _s_ 1 _, s_ 2 _, ..., si_ ), and the action
corresponds to predicting the next word _si_ +1 from the vocabulary, given the context. In this simplified
MDP, the deterministic transition is akin to a concatenation function, advancing the state to the next
step ( _s_ 1 _, s_ 2 _, ..., si, si_ +1). Employing a toy MDP enables us to systematically evaluate the trajectory
probabilities for all feasible paths within the MDP, shedding light on the allocation of probability
mass by Direct Alignment Algorithms (DAAs) towards out-of-distribution (OOD) trajectories.
**The Experimental Setup.** We adhere to the standard direct alignment protocol [46][41], encompassing two key stages:
1. **Supervised Fine-tuning (SFT) / Behavioral Cloning (BC):** This phase involves finetuning the policy based on a limited number of trajectories. Specifically, we utilize
three demonstrations for SFT: ( _s_ 1 _, a_ 0 _, s_ 2 _, a_ 0 _, s_ 5 _, a_ 0 _, s∞_ ), ( _s_ 1 _, a_ 1 _, s_ 3 _, a_ 1 _, s_ 9 _, a_ 0 _, s∞_ ), and
( _s_ 1 _, a_ 2 _, s_ 4 _, a_ 2 _, s_ 13 _, a_ 2 _, s∞_ ).
2. **Alignment with Preferences:** In this stage, preferences extracted from trajectories
are employed to align the policy. Notably, we have only one preference available:
( _s_ 1 _, a_ 1 _, s_ 3 _, a_ 1 _, s_ 9 _, a_ 0 _, s∞_ ) _≻_ ( _s_ 1 _, a_ 0 _, s_ 2 _, a_ 0 _, s_ 5 _, a_ 0 _, s∞_ ). This deliberate constraint exaggerates a scenario with limited data, enabling us to gauge the probability mass allocated to
out-of-distribution (OOD) trajectories under such conditions. Insights garnered from this
exaggerated low-data scenario hold relevance for Large Language Model (LLM) settings
where preference datasets used for alignment are notably smaller compared to the scale of
LLM models deployed.
We utilize a Recurrent Neural Network (RNN) policy to navigate through the MDP, facilitating a
closer resemblance to real-world language modeling scenarios.
Subsequently, we explore three distinct direct alignment loss functions: Direct Preference Optimization (DPO) [46], Identity Preference Optimization (IPO) [4], and Sequence Likelihood Calibration
(SLiC) [68]. Additionally, we investigate how the selection of the KL penalty coefficient _β_ influences
the distribution of probability mass on OOD trajectories. This exploration encompasses three values
of _β_ : (0 _._ 01 _,_ 0 _._ 1 _,_ 0 _._ 5).
In general, the plots illustrate that Direct Alignment Algorithms (DAAs) tend to allocate a significant
proportion of the probability mass to out-of-distribution (OOD) trajectories during the alignment
process. While Figure 9 may suggest that Direct Preference Optimization (DPO) can retain a
substantial amount of probability mass on the selected trajectory in the preference dataset, it’s
noteworthy that the plots for DPO exhibit considerable noise. To provide further insight, Figure 18
displays the plots resulting from three additional repetitions of the DPO experiment. Similar noisy
trends were also observed in the experiments for IPO and SLiC. This elucidates the unconstrained
nature of the DPO problem: multiple solutions exist for the DPO loss, each distributing varying
amounts of probability mass to OOD trajectories. In the experiments with IPO and SLiC, it’s
also observed that similar to DPO, the probability mass allocated to in-distribution trajectories can
diminish substantially over the course of training. Notably, the probability mass, in our experiments,
becomes concentrated on a select few out-of-distribution trajectories. Moreover, consistent trends
are discernible across various values of _β_ . The results of our experiments with the Toy-MDP can be
found in the following figures 12, 9, 15, 13, 10, 16, 14, 11, 17.
Figure 9: Trajectory probabilities throughout DPO training, _β_ = 0 _._ 1. The top plot shows how the
probability mass of different OOD trajectories, changes throughout training. The bottom plot shows
how the probability mass of the trajectories in our preference dataset (size 1) changes over training.
The trajectories are listed in the legends for the plots, as a sequence of state, action pairs.
.
Figure 10: Trajectory probabilities throughout IPO training, _β_ = 0 _._ 1. The top plot shows how the
probability mass of different OOD trajectories, changes throughout training. The bottom plot shows
how the probability mass of the trajectories in our preference dataset (size 1) changes over training.
The trajectories are listed in the legends for the plots, as a sequence of state, action pairs.
Figure 11: Trajectory probabilities throughout SLiC training, _β_ = 0 _._ 1. The top plot shows how the
probability mass of different OOD trajectories, changes throughout training. The bottom plot shows
how the probability mass of the trajectories in our preference dataset (size 1) changes over training.
The trajectories are listed in the legends for the plots, as a sequence of state, action pairs.
Figure 12: Trajectory probabilities throughout DPO training, _β_ = 0 _._ 01. The top plot shows how the
probability mass of different OOD trajectories, changes throughout training. The bottom plot shows
how the probability mass of the trajectories in our preference dataset (size 1) changes over training.
The trajectories are listed in the legends for the plots, as a sequence of state, action pairs.
.
Figure 13: Trajectory probabilities throughout IPO training, _β_ = 0 _._ 01. The top plot shows how the
probability mass of different OOD trajectories, changes throughout training. The bottom plot shows
how the probability mass of the trajectories in our preference dataset (size 1) changes over training.
The trajectories are listed in the legends for the plots, as a sequence of state, action pairs.
Figure 14: Trajectory probabilities throughout SLiC training, _β_ = 0 _._ 01. The top plot shows how the
probability mass of different OOD trajectories, changes throughout training. The bottom plot shows
how the probability mass of the trajectories in our preference dataset (size 1) changes over training.
The trajectories are listed in the legends for the plots, as a sequence of state, action pairs.
Figure 15: Trajectory probabilities throughout DPO training, _β_ = 0 _._ 5. The top plot shows how the
probability mass of different OOD trajectories, changes throughout training. The bottom plot shows
how the probability mass of the trajectories in our preference dataset (size 1) changes over training.
The trajectories are listed in the legends for the plots, as a sequence of state, action pairs.
.
Figure 16: Trajectory probabilities throughout IPO training, _β_ = 0 _._ 5. The top plot shows how the
probability mass of different OOD trajectories, changes throughout training. The bottom plot shows
how the probability mass of the trajectories in our preference dataset (size 1) changes over training.
The trajectories are listed in the legends for the plots, as a sequence of state, action pairs.
Figure 17: Trajectory probabilities throughout SLiC training, _β_ = 0 _._ 5. The top plot shows how the
probability mass of different OOD trajectories, changes throughout training. The bottom plot shows
how the probability mass of the trajectories in our preference dataset (size 1) changes over training.
The trajectories are listed in the legends for the plots, as a sequence of state, action pairs.
Figure 18: Trajectory probabilities throughout DPO training, over three different runs, with _β_ = 0 _._ 1
.
Figure 19: KL divergence versus GPT-4 win rate for the Gemma2-2b model on the Anthropic-HH
dataset. The **left** plot shows DPO results, and the **right** plot shows SLiC results.


We present KL divergence versus GPT-4 win rate plots in Figure 19 to illustrate overoptimization
trends in Direct Alignment Algorithms for the Gemma2-2b model [61] and the Anthropic-HH dataset
[5]. Results are shown for the DPO and SLiC variants, which sufficiently demonstrate that the
overoptimization trends observed with the Pythia models are not specific to a single model or dataset.
The figure illustrates the trade-off between KL divergence and GPT-4 win rate across different values
of beta in the alignment objective.