In the previous sections, we proved that LLMs have
_elasticity_ . This section will analyze two specific
phenomenons of it:
  - _**Resistance**_ **for Pre-Trained Models.** Models
tend to maintain the original distribution and
resist alignment;
  - _**Rebound**_ **for Post-Trained Models.** Finetuning in the opposite direction of posttraining ( _e.g._, safe _vs_ . unsafe) causes posttrained models to return quickly to the pretraining distribution.
**5.1** **Existence of Language Models’** _**Resistance**_
Figure 2: **Experiment pipeline for validating** _**resis-**_
_**tance**_ **.** We conceptualize _resistance_ as: _**inverse align-**_
_**ment**_ is easier than _**forward alignment**_ .
**Experiment Design.** We verify the existence of
_resistance_ by arguing that _**forward alignment**_ is
harder than _**inverse alignment**_ for pre-trained models. Specifically, we first perform one epoch of SFT
on a pre-trained LLM with parameters _**θ**_ 0, saving
the slices _{_ _**θ**_ 1 _,_ _**θ**_ 2 _, . . .,_ _**θ**_ _n}_ . Subsequently, without
loss of generality, we collect the responses of slices
_**θ**_ _k_ and _**θ**_ _l_ (where _k < l_ ) on hold-out prompts, forming datasets _Dk_ and _Dl_ . As shown in Figure 2, we
define _**forward alignment**_ **(** _**Path A**_ **)** as the process
of training _**θ**_ _k_ on _Dl_, and _**inverse alignment**_ **(** _**Path**_
_**B**_ **)** as the process of training _**θ**_ _l_ on _Dk_ .
**Experiment Setup.** We select different SFT
datasets including Alpaca (Taori et al., 2023),
TruthfulQA (Lin et al., 2022), and Beavertails (Ji
et al., 2024c,b), which correspond respectively to
the model’s 3H principle (Askell et al., 2021). We
divide them into three equal parts to obtain three
corresponding slices _{_ _**θ**_ 1 _,_ _**θ**_ 2 _,_ _**θ**_ 3 _}_ . We consider
Llama2-7B, Llama2-13B, and Llama3-8B (Touvron et al., 2023) as the base model _**θ**_ 0.
**Experiment Results.** As shown in Table 1, the
experimental results indicate that the training loss
of _**inverse alignment**_ consistently remains lower
than that of _**forward alignment**_, irrespective of
the slice pair selection. This observation holds
across all models and datasets in the experiments.
All experimental results demonstrate that _**inverse**_
_**alignment**_ is easier than _**forward alignment**_ across
diverse models and datasets and validate the existence of _resistance_ . To further verify the _resistance_
in language models, we provide additional experimental details and results in the Appendix B.1.
**5.2** **Existence of** _**Rebound**_
Figure 3: **Experiment pipeline for validating** _**rebound**_ **.**
We conceptualize _rebound_ as: the more **positive** the
post-trained models’ performance, the more **negative** it
becomes after inverse finetuning.
**Experiment Design.** We verify the existence of
_rebound_ by arguing that for post-trained models,
𝜽𝒌
Model
Training Process
𝐷
Datasets Performance
Datasets Base Models _**θ**_ 2 _→_ _**θ**_ 1 _vs._ _**θ**_ 1 _→_ _**θ**_ 2 _**θ**_ 3 _→_ _**θ**_ 2 _vs._ _**θ**_ 2 _→_ _**θ**_ 3 _**θ**_ 3 _→_ _**θ**_ 1 _vs._ _**θ**_ 1 _→_ _**θ**_ 3
Llama2-7B 0.1589 _↓_ 0.2018 _↑_ 0.1953 _↓_ 0.2143 _↑_ 0.1666 _↓_ 0.2346 _↑_
**Alpaca** Llama2-13B 0.1772 _↓_ 0.1958 _↑_ 0.2149 _↓_ 0.2408 _↑_ 0.1835 _↓_ 0.2345 _↑_
Llama3-8B 0.2540 _↓_ 0.2573 _↑_ 0.2268 _↓_ 0.3229 _↑_ 0.2341 _↓_ 0.2589 _↑_
Llama2-7B 0.1909 _↓_ 0.2069 _↑_ 0.1719 _↓_ 0.1721 _↑_ 0.2011 _↓_ 0.2542 _↑_
**Truthful** Llama2-13B 0.1704 _↓_ 0.1830 _↑_ 0.1544 _↓_ 0.1640 _↑_ 0.1825 _↓_ 0.2429 _↑_
Llama3-8B 0.2118 _↓_ 0.2256 _↑_ 0.2100 _↓_ 0.2173 _↑_ 0.2393 _↓_ 0.2898 _↑_
Llama2-7B 0.2730 _↓_ 0.2809 _↑_ 0.2654 _↓_ 0.2691 _↑_ 0.2845 _↓_ 0.2883 _↑_
**Safe** Llama2-13B 0.2419 _↓_ 0.2439 _↑_ 0.2320 _↓_ 0.2327 _↑_ 0.2464 _↓_ 0.2606 _↑_
Llama3-8B 0.2097 _↓_ 0.2156 _↑_ 0.2008 _↓_ 0.2427 _↑_ 0.2277 _↓_ 0.2709 _↑_
Table 1: **Comparsion between** _**inverse alignment**_ **and** _**forward alignment**_ **training loss.** Under different model,
task, and stage slicing settings, _**forward alignment**_ is more challenging than _**inverse alignment**_, _i.e.,_ the loss is
higher. This indicates that pre-trained models tend to maintain their original distribution.
Figure 4: **Experimental results for validating the existence of** _**rebound**_ **(left: IMDb, right: Beavertails).** The
left part of each sub-figure is the performance of Gemma-2B while the right is Llama2-7B, respectively. Models
trained with more **positive** data initially perform better but perform worse after fine-tuning with **negative** data.
the more **positive** the post-trained models’ performance, the more **negative** it becomes after _inverse_
_alignment_ . We validate tasks involving two opposing characteristics ( _e.g._ safe and unsafe). We first
train slices _{_ _**θ**_ 1 _,_ _**θ**_ 2 _, ...,_ _**θ**_ _n}_ based on a pre-trained
model _**θ**_ 0 using positive ( _e.g.,_ safe) data of various
volumes. Then we perform inverse finetuning on
these slices using negative data ( _i.e.,_ unsafe).
**Tasks and Datasets.** We select two tasks: positive generation and single-turn safe conversation.
For the former, we use the data classified as positive
or negative in the IMDb dataset (Maas et al., 2011).
Referring to (Rafailov et al., 2024), we use the first
2-8 tokens of each complete text as a prompt for
LLMs to generate the subsequent content. For the
latter, we use the data classified as safe and unsafe in Beavertails (Ji et al., 2024c). We organized
the positive sample sizes into {1000, 2000, 5000,
10000}, while the negative sample sizes, were divided into {100, 200, 500, 1000, 2000}.
**Evaluation and Metrics.** We collect the model’s
responses on the reserved test prompts. Then we
use score models provided by existing research to
complete the performance evaluation. For positive
style generation, we refer to (Rafailov et al., 2024)
and use the Sentiment Roberta model (Hartmann
et al., 2023) to classify the responses, taking the
proportion of all responses classified as positive as
the model score. For single-turn safe dialogue, we
use the cost model (Dai et al., 2024) to score the
safety of each response, using the average score of
all responses as the model score.
**Experiment Results.** We evaluate the _rebound_
phenomenon on Llama2-7B (Touvron et al., 2023)
and Gemma-2B (Team et al., 2024). The experimental results in Figure 4 show that, for models
fine-tuned with a larger amount of positive sample data, their performance drops quicker under
only a small amount of negative sample fine-tuning.
Subsequently, the performance decline slows down
and tends to stabilize. This result also confirms
the previous conclusion: the initial rapid decline
of model’s performance is due to _rebound_, as the
model is far from the pre-trained distribution; while
the later stabilization of the countermeasure is due
to _resistance_, as the model is already close to the
pre-trained distribution.
To assess the generalizability of the _rebound_
phenomenon, we perform additional ablation studies focusing on alignment algorithms, evaluation
Figure 5: **Experimental results for validating** _**rebound**_ **increases with model size (left: IMDb, right: Beavertails).**
All single line covers positive data volume settings as Figure 4, with shadow denoting std. As the model size
increases, the performance of the aligned model deteriorates more rapidly after fine-tuning with **negative** data.
Figure 6: **Experimental results for validating** _**rebound**_ **increases with pre-training data volume. (left: IMDb,**
**right: Beavertails).** Each line covers positive data settings as Figure 4, with shadow denoting std. As pre-training
data volume increases, aligned model performance deteriorates more rapidly after fine-tuning with **negative** data.
metrics, and fine-tuning directions. The results
consistently validate the presence of the _rebound_
phenomenon across language models. Further experimental details are available in Appendix B.2.
**5.3** **Internal Factor of** _**Rebound**_
We conduct an in-depth analysis of the internal
factors affecting the _rebound_ phenomenon since it
is crucial for the robustness of LLMs alignment.
In particular, we investigate how model parameter
scale and the amount of pre-training data influence
_rebound_ . Owing to space constraints, further experimental analyses are included in Appendix B.3.
_**Rebound**_ **Increases with Model Size.** To investigate how the _rebound_ phenomenon varies with
model size, we conducted experiments on Qwen
models (Bai et al., 2023) with parameter scales
of 0.5B, 4B, and 7B. The experimental results in
Figure 5 show that as the model parameter size
increases, the initial performance decline due to
negative data fine-tuning is faster, while the subsequent decline is slower. This indicates that as
the parameter size increases, there is an increase
in _rebound_ in response to both positive and negative data, further suggesting a positive correlation
between model _elasticity_ and parameter scale.
_**Rebound**_ **Increases with Pre-training Data Vol-**
**ume.** To verify that _rebound_ increases with the
growth of pre-training data, we vary pre-training
slices (2.0T, 2.5T, and 3.0T) released by TinyLlama (Zhang et al., 2024a) and conduct the same
experiment. As shown in Figure 6, when the pretraining data volume increases, the initial performance decline due to negative data fine-tuning is
faster, while the subsequent decline is slower. It
demonstrates that larger pre-training data volumes
reinforce the _rebound_ of LLMs, which is consistent
with the inference proposed in Theorem 4.2.


Our work uncovers a possible mechanism underlying the fragility of alignment: the _elasticity_ of
language models. We demonstrate that upon perturbation, the normalized compression rate of language models changes inversely with dataset size,
making them more inclined to retain pre-training
distributions while forgetting alignment distributions, thus resisting alignment. Through extensive experiments, we validate the universality of
this _elasticity_ effect, observing that the _elasticity_
strengthens as model size and pre-training data volume scale up. Fundamentally, our findings support
a core assertion: language models exhibit _elasticity_,
and thereby inherently resist alignment.

Jiaming Ji
Kaile Wang
Tianyi Qiu
Boyuan Chen
Jiayi Zhou
Changye Li
Hantao Lou
Yaodong Yang


Theory-wise, the primary limitation of our work is
our specification of the _mass distribution_ (Assumption A.7), and empirical studies on the exact form
of this distribution shall be valuable. Experimentwise, we have not systematically validated _elastic-_
_ity_ throughout the entire lifecycle of pre-training
and alignment phases due to cost constraints. In
future works, we plan to focus on whether this
phenomenon is universally applicable, such as in
multimodal models (Ji et al., 2024d; Huh et al.,
2024). Additionally, we aim to theoretically uncover the relationship between model _elasticity_ and
_scaling laws_ (Kaplan et al., 2020; Xiao et al., 2024),
specifically determining the amount of training data
required for _elasticity_ to manifest and quantitatively
analyzing whether _elasticity_ intensifies as model
parameters and pre-training data volume increase.


**Resist Alignment.** Alignment fine-tuning methods aim to efficiently adjust the distribution of
LLMs with minimal data to enhance their safety
mechanisms. However, current alignment algorithms for language models may rely on optimization shortcuts, leading to local optima, while lacking involvement with the intrinsic mechanisms of
the model (Qi et al., 2024a). From the perspective
of language models’ _elasticity_, we require more
robust alignment methods to ensure that modifications to model parameters extend beyond superficial changes (Qi et al., 2024a; Cohen et al., 2024),
thereby preventing the emergence of more effective
_inverse alignment_ techniques that could give rise to
extreme misalignment risks such as deceptive adversarial alignment (IDAIS-Beijing, 2024; Bengio
et al., 2025; Marks et al., 2025; Baker et al., 2025)
or alignment faking (Greenblatt et al., 2024). Although methods such as data cleansing during the
training phase have proven to be an effective strategy for improving the malleability of a language
model’s final distribution (He et al., 2024; Qi et al.,
2024b), such approaches are not particularly costeffective and feasible. In Appendix C.2, we provide
a preliminary discussion of practical steps based
on _elasticity_ to mitigate _inverse alignment_ risks,
and we look forward to future research and practical developments grounded in _elasticity_, which
will enable the design of more robust alignment
algorithms (Revel et al., 2024; Zhang et al., 2024b;
Chen et al., 2024; Sheshadri et al., 2024; Liu et al.,
2024; Li et al., 2025), ultimately achieving true and
reliable alignment for LLMs.

Rethinking Fine-tuning and Model Weight from


**LLMs.** Open-sourcing is a double-edged sword
(Seger et al., 2023; Anwar et al., 2024; Kukreja
et al., 2024). The public release of model weights
enables the technical community to quickly identify and address potential vulnerabilities and conduct large-scale research on broad safety concerns
(Touvron et al., 2023). These efforts collectively
help sustain a balanced offence-defense dynamic
(Jervis, 1978; Garfinkel and Dafoe, 2021) in the
open-source environment and improve model security (Eiras et al., 2024). However, the risk of
misuse associated with open-sourcing models is inevitable (Seger et al., 2023). Malicious fine-tuning
of open-source models (Urbina et al., 2022; Sandbrink, 2023) and the use of the open-source models
to facilitate system jailbreaks (Zou et al., 2023)
can pose serious threats to public safety (Goldstein et al., 2023; Reuel et al., 2024). At present,
the open-sourcing of powerful language models
relies on safety alignment and rigorous security audits (Mökander et al., 2024) to ensure responsible
use. However, if advanced _inverse alignment_ techniques become feasible, even carefully aligned and
audited open-source models could be reverted to
their pre-alignment states at minimal cost, drastically lowering the barrier for jailbreaking aligned
models and disrupting the offense-defense balance
in the open-source ecosystem (Shapiro and Siegel,
2010). Therefore, it is crucial to develop alignment
algorithms with fine-tuning robustness. We hope
that deeper insights into the _elasticity_ mechanisms
of language models will drive the advancement
of untunable alignment methods, enabling models
to maintain reliable safety throughout their entire
lifecycle (Madiega, 2021).

Rethinking Open-sourcing from Elasticity of


**Assumption A.1** (Binary Tokens) **.** For simplicity and without loss of generality, we assume that all
datasets share a uniform token table containing only binary tokens (specifically 0/1).
_Proof._ When �� _**x**_ �� _≤_ _d_, the compression protocol defined in Definition 3.3 can perfectly compress _**x**_ .
Hence, the expectation of the ideal code length _Lp_ _**θ**_ ( _**x**_ ) satisfies:

_,_

E [ _Lp_ _**θ**_ ( _**x**_ )] =

_d_
_−_  _l_ =1
2 _[l][−]_ [1]
_plj_ log _plj_
_j_ =1
where _d_ represents the depth of the pruned tree _TD_ _[′]_ [and] _[ p][lj]_ [ represents the probability values of the][ EOS]
nodes for the _j_ -th node at the _l_ -th layer.
Now consider _sd ≤_ �� _**x**_ �� _≤_ ( _s_ + 1) _d_ . Let us suppose that _**x**_ = ( _**x**_ 1 _· · ·_ _**x**_ _s_ _**x**_ _s_ +1), where _|_ _**x**_ _k|_ = _d_, for
_k ∈{_ 1 _, . . ., s}_ and _|_ _**x**_ _s_ +1 _| ≤_ _d_ . In this case, _**x**_ cannot be perfectly compressed by the model. Hence, the
compression of _x_ needs to be performed in segments, and the length of each segment is not greater than _d_ .
E _**x**_ [ _Lp_ _**θ**_ ( _**x**_ )] = E _**x**_ 1 [ _Lp_ _**θ**_ ( _**x**_ 1)] + E _**x**_ 1E( _**x**_ 2 _..._ _**x**_ _s_ +1)  - _Lp_ _**θ**_ (( _**x**_ 2 _. . ._ _**x**_ _s_ +1)) �� _**x**_ 1�
-
_p_ ( _**x**_ 1)
_**x**_ 1 ( _**x**_ _..._ _**x**_
   = E _**x**_ 1 [ _Lp_ _**θ**_ ( _**x**_ 1)] +
_p_ ( _**x**_ 2 _. . ._ _**x**_ _s_ +1�� _**x**_ 1) _· Lp_ _**θ**_ (( _**x**_ 2 _. . ._ _**x**_ _s_ +1))
( _**x**_ 2 _..._ _**x**_ _s_ +1)
= E _**x**_ 1 [ _Lp_ _**θ**_ ( _**x**_ 1)] + E( _**x**_ 2 _..._ _**x**_ _s_ +1) [ _Lp_ _**θ**_ (( _**x**_ 2 _. . ._ _**x**_ _s_ +1))]
=
=
_s_ +1
E _**x**_ _k_ [ _Lp_ _**θ**_ ( _**x**_ _k_ )]
_k_ =1
��
 - _**x**_
_d_

_,_

2 _[l][−]_ [1]
_plj_ log _plj_
_j_ =1
2 _[l][−]_ [1]
- []
_−_

_d_
_l_ =1
thus the proof is completed.
**Definition A.3** (Mass Distribution in Token Tree) **.** Consider the sample space Ω consisting of all responses
in dataset _D_ . The probability distribution _PD_ of all subtrees at the _d_ -th level nodes of _TD_ is a mapping
from Ω to [0 _,_ 1]. Let _XD_ be the random variable representing the probability value taken at each leaf
multiplied by the number of leaves. The mass distribution _Pmass_ represents the probability that _XD_ takes
the corresponding probability value. According to the definition of _Pmass_, E[ _XD_ ] = 1 .
_Remark_ A.4 (Mixture of Mass Distribution) _._ For independently and differently distributed datasets
_D_ 1 _, . . ., DN_, _D_ = [�] _i_ _[N]_ =1 _[D][i]_ [ is a mixture of these datasets. For the pruned trees] _[ T]_ [1] _[, . . .,][ T][N]_ [ of these]
datasets with depth _d_, the random variables of their leaf nodes satisfy the following relationship:
_XD_ =            - ~~�~~ _Nk_ =1 _Nk_ =1�� _D_ ~~�~~            - _kD_ �� _Xk_ ~~�~~            - _Dk_ _,_
where _XDk_ follows the mass distribution _Pmass_ _[k]_ [. For] _[ X][D]_ _ki_ [and] _[ X][D]_ _kj_ [from different datasets,] _[ X][D]_ _ki_ [and]
_XDkj_ are independent of each other.
**Lemma A.5** (Entropy of Mass Distribution) **.** _Consider the pruned trees Tk_ _[′]_ _[and][ T][ ′][ of dataset][ D][k][ and]_
_D_ = [�] _i_ _[N]_ =1 _[D][i][. Denote that the response distribution and the mass distribution of][ T][ ′][’s tree nodes are][ P]_ _[D]_
_and Pmass. Similarly, we define the response distribution Pk_ _[D]_ _[and the mass distribution][ P]_ _mass_ _[k]_ _[. When]_
_the number of the leaf nodes M is sufficiently large, the cross-entropy of the response distribution can be_
_rewritten as follows._
E _**x**_ _∼Pk_     - _−p_ _[D][k]_ log _p_ _[D]_ [�] = E _XDk_ _∼Pkmass,XD∼Pmass_ [ _−XDk_ log _XD_ ] + log _M,_
_where p_ _[D]_ _, p_ _[D][k]_ _stand for the probability of the leaf nodes of T_ _[′]_ _,Tk_ _[′]_ _[while][ X][D]_ _k_ _[,][ X][D][ stand for the random]_
_variables of the probability of the leaf nodes in T_ _[′]_ _,Tk_ _[′][, respectively.]_
_Proof._ Let _M_ be the number of leaf nodes of _T_ _[′]_ . According to the definitions of the response distribution
_P_ and mass distribution _Pmass_, we have _Mp_ _[D][j]_ = _XDj_ _, ∀j ∈{_ 1 _, . . ., N_ _}_ . Therefore,
E _**x**_ _∼Pk_ - _−p_ _[D][k]_ log _p_ _[D]_ [�] =
=
_M_
- _−p_ _[D]_ _i_ _[k]_ log _p_ _[D]_ _i_
_i_ =1
- _−_ [1]
_M_
_i_ =1
_M_
_M_ _[X][i,][D][k]_ [ log] _[ X][i,][D]_ [ + log] _[ M]_
= E _XDk_ _∼Pkmass,XD∼Pmass_ [ _−XDk_ log _XD_ ] + log _M._
_Remark_ A.6 _._ In Lemma A.5, _XDk_ are assumed to be independent. However due to [�] _i_ _[M]_ =1 _[p]_ _i_ _[D][k]_ = 1, the
_XDk_ are not actually independent. Considering that _M_ is sufficiently large in our subsequent analysis, we
can regard the independence of _XDk_ as a good approximation.
We assume that the mass distribution of the segment follows a heavy-tailed Pareto distribution, with
supporting evidence from Mingard et al. (2021); Zipf (1946).
**Assumption A.7** (The Pareto Distribution) **.** We assume that the mass distribution of pruned token trees _T_
of depth _d_, across different datasets, follows a Pareto distribution with identical parameters:
_pX_ ( _x_ ) =
- _αc_ _[α]_
_x ≥_ _c,_
_x_ _[α]_ [+1]
0 _x < c,_
where _α, c_ are parameters of the Pareto distribution.
_Proof._ For the sake of convenience in calculations, we first use Lemma A.5 to replace the Shannon
entropy of response distribution.
_dγp_ _[D]_ _**θ**_ _[j]_ _[/][D]_ = _[d]_ �E _**x**_ _∼Pj_          - _−p_ _[D][j]_ log _p_ _[D]_ [�] _−_ log _M_          _d l_ _d l_
          _d_ E _XDj ∼Pmassj_ _[,X]_ _D_ _[∼P]_ _mass_              - _−XDj_ log _XD_              - [�]
= _._
_d l_
According to Assumption A.7, _XDj_ follows a Pareto distribution with the parameters _α_ and _c_ . Hence,
E _XDp_ _∼Pmass_ _[p]_ _[,X]_ _D_ _[∼P]_ _mass_  - _−XDp_ log _XD_
  - + _∞_
= _−_
_c_
  - + _∞_
= _−_
_c_
- + _∞_
_c_
- + _∞_
_c_
- + _∞_
_c_
- + _∞_
_c_
_α_ [3] _c_ [3] _[α]_ ( _x_ 1 + _x_ 2 + _x_ 3)
log
3 ~~[�]~~ _i_ [3] =1 _[x]_ _i_ _[α]_ [+1]
_α_ [3] _c_ [3] _[α]_
_·_ _[x]_ [1][ +] _[ x]_ [2][ +] _[ x]_ [3] _·_ log
~~�~~ 3 3
_i_ =1 _[x]_ _i_ _[α]_ [+1]
- _i∈{p,a,t}_ �� _Di_ �� _xi_
_dx_ 1 _dx_ 2 _dx_ 3
~~�~~ _i∈{p,a,t}_ ~~��~~ _Di_ ~~��~~
_k_ 3 [(] _[x]_ [1] [+] _[ x]_ [2] [+] _[ x]_ [3][) +] _[ x]_ [2] [+] _[ lx]_ [3]
_dx_ 1 _dx_ 2 _dx_ 3
_k_ + _l_ + 1
E _XDa_ _∼Pamass,XD∼Pmass_ [ _−XDa_ log _XD_ ]
  - + _∞_
= _−_
_c_
- + _∞_
_c_
- + _∞_
_c_
_α_ [3] _c_ [3] _[α]_ _x_ 2
log
~~�~~ 3
_i_ =1 _[x]_ _i_ _[α]_ [+1]
_k_ 3 [(] _[x]_ [1] [+] _[ x]_ [2] [+] _[ x]_ [3][) +] _[ x]_ [2] [+] _[ lx]_ [3]
_dx_ 1 _dx_ 2 _dx_ 3
_k_ + _l_ + 1
_Dp/D_
where _j_ = 1 _,_ 2 _,_ 3. Therefore, _dγp_ _**θ**_
_d l_ _**θ**_ can be written as:
_p/D_
_d l_ _**θ**_ and _dγp_ _[D]_ _d l_ _**θ**_ _[a/][D]_
_dγp_ _[D]_ _**θ**_ _[p][/][D]_ = _−_ _[dS]_ [1]
_d l_ _d l_
(5)
_d l_
_dγp_ _[D]_ _**θ**_ _[a][/][D]_
_[dS]_ [1] = _−_ _[dS]_ [2]
_d l_ _[,]_ _d l_ _d l_
where
    - + _∞_
_S_ 1 =
_c_
    - + _∞_
_S_ 2 =
_c_
- + _∞_
_c_
- + _∞_
_c_
- + _∞_
_c_
- + _∞_
_c_
_α_ [3] _c_ [3] _[α]_
log
_x_ _[α]_ 1 [+1] _x_ _[α]_ 2 _[x]_ 3 _[α]_ [+1]
_α_ [3] _c_ [3] _[α]_ ( _x_ 1 + _x_ 2 + _x_ 3)
log
3 _x_ _[α]_ 1 [+1] _x_ _[α]_ 2 [+1] _x_ _[α]_ 3 [+1]
_k_ 3 [(] _[x]_ [1] [+] _[ x]_ [2] [+] _[ x]_ [3][) +] _[ x]_ [2] [+] _[ lx]_ [3]
_dx_ 1 _dx_ 2 _dx_ 3
_k_ + _l_ + 1
_k_ 3 [(] _[x]_ [1] [+] _[ x]_ [2] [+] _[ x]_ [3][) +] _[ x]_ [2] [+] _[ lx]_ [3]
_dx_ 1 _dx_ 2 _dx_ 3 _._
_k_ + _l_ + 1
_[a/][D]_  - _Dp/D_
_**θ**_ = Θ _−k_ _dγp_ _**θ**_
_d l_ _d l_
Proving that _dγp_ _[D]_ _**θ**_ _[a/][D]_
_d l_
is equivalent to proving:
_Dp/D_
_k ·_ _dγp_ _**θ**_
lim _d l_
_k→_ + _∞, l→_ 0
_Dp/D_
_p_ _**θ**_ + _dγp_ _[D]_ _**θ**_ _[a/][D]_
_d l_ _d l_
= 0 _,_ (6)
_d l_
_Dp/D_
_k ·_ _dγp_ _**θ**_
_d l_
By substituting (5) into (6), we have
_Dp/D_
_k ·_ _dγp_ _**θ**_
lim _d l_
_k→_ + _∞, l→_ 0
_Dp/D_
_p_ _**θ**_ + _dγp_ _[D]_ _**θ**_ _[a/][D]_
_d l_ _d l_
lim _k→_ + _∞, l→_ 0  - _k·dSd l_ 1+ _dS_ 2
=
1+ _dS_ 2
_d l_
_._ (7)
_[dS]_ [1]
_d l_
_d l_
_Dp/D_
_k ·_ _dγp_ _**θ**_
lim _k→_ + _∞, l→_ 0 _[k][·][dS]_ [1]
_d l_
Now calculate the values of _[dS]_ _d l_ [1] [for the case when] _[ l][ →]_ [0][ and] _[ k]_ [ is sufficiently large.]
_dS_ 1
lim
_l→_ 0 _d l_
= _α_ [3] _c_ [3] _[α]_ lim
_l→_ 0
- + _∞_
_c_
- + _∞_
_c_
- + _∞_
_c_
( _x_ 1 + _x_ 2 + _x_ 3)
3 _x_ _[α]_ 1 [+1] _x_ _[α]_ 2 [+1] _x_ _[α]_ 3 [+1]
_·_ [(] _[k]_ [ + 1 +] _[ l]_ [)] _[x]_ [3] _[−]_ [(] _[k]_ 3
3 [(] _[x]_ [1] [+] _[ x]_ [2] [+] _[ x]_ [3][) +] _[ x]_ [2] [+] _[ lx]_ [3][)]
_[k]_ _dx_ 1 _dx_ 2 _dx_ 3
3 [(] _[x]_ [1][ +] _[ x]_ [2][ +] _[ x]_ [3][) +] _[ x]_ [2][ +] _[ lx]_ [3][)]
( _k_ + 1 + _l_ )( _[k]_
- + _∞_
_c_
- + _∞_
_c_
( _x_ 1 + _x_ 2 + _x_ 3) _x_ 3
3 _x_ _[α]_ 1 [+1] _x_ _[α]_ 2 [+1] _x_ _[α]_ 3 [+1] ( _[k]_ 3 [(] _[x]_ [1][ +] _[ x]_ [2][ +] _[ x]_ [3][) +] _[ x]_ [2][ +] _[ lx]_ [3][)] _[dx]_ [1] _[dx]_ [2] _[dx]_ [3]
= _α_ [3] _c_ [3] _[α]_ lim
_l→_ 0
- + _∞_
_c_
- + _∞_
_c_
- + _∞_
_c_
( _x_ 1 + _x_ 2 + _x_ 3)
3 _x_ _[α]_ 1 [+1] _x_ _[α]_ 2 [+1] _x_ _[α]_ 3 [+1] ( _k_ + 1 + _l_ ) _[dx]_ [1] _[dx]_ [2] _[dx]_ [3]
_−_ _α_ [3] _c_ [3] _[α]_ lim
_l→_ 0
- + _∞_
_c_
   - + _∞_
= _α_ [3] _c_ [3] _[α]_
_c_
- + _∞_
_c_
- + _∞_
( _x_ 1 + _x_ 2 + _x_ 3) _x_ 3
lim
_c_ _l→_ 0 3 _x_ _[α]_ 1 [+1] _x_ _[α]_ 2 [+1] _x_ _[α]_ 3 [+1] ( _[k]_ 3 [(] _[x]_ [1][ +] _[ x]_ [2][ +] _[ x]_
_[k]_
3 [(] _[x]_ [1][ +] _[ x]_ [2][ +] _[ x]_ [3][) +] _[ x]_ [2][ +] _[ lx]_ [3][)] _[dx]_ [1] _[dx]_ [2] _[dx]_ [3]
- + _∞_
_c_
- + _∞_
_c_
_x_ 1
_x_ _[α]_ 1 [+1] _x_ _[α]_ 2 [+1] _x_ _[α]_ 3 [+1] ( _k_ + 1 + _l_ ) _[dx]_ [1] _[dx]_ [2] _[dx]_ [3]
_−_ _α_ [3] _c_ [3] _[α]_ lim
_l→_ 0
- + _∞_
_c_
   - + _∞_
= _α_ [3] _c_ [3] _[α]_
_c_
- + _∞_
_c_
- + _∞_
_c_
_x_ 1 + _x_ 2 + _x_ 3
3 _x_ _[α]_ 1 [+1] _x_ _[α]_ 2 [+1] _x_ _[α]_ 3 [+1]
_·_ lim
_l→_ 0
_x_ 3 _−_ _x_ [2] 3
_k_ _[l]_
[ _[k]_
3 [(] _[x]_ [1][ +] _[ x]_ [2][ +] _[ x]_ [3][) +] _[ x]_ [2] 3 [(] _[x]_ [1][ +] _[ x]_ [2][ +] _[ x]_ [3][) +] _[ x]_ [2][]][2]
_x_ 3 _−_ _x_ [2] 3
_k_ [ _[k]_
3 [(] _[x]_ [1][ +] _[ x]_ [2][ +] _[ x]_ [3][) +] _[ x]_ [2] 3 [(] _[x]_ [1][ +] _[ x]_ [2][ +]
_dx_ 1 _dx_ 2 _dx_ 3
- + _∞_
_c_
- + _∞_
_c_
_x_ 1
(8)
_x_ _[α]_ 1 [+1] _x_ _[α]_ 2 [+1] _x_ _[α]_ 3 [+1] ( _k_ + 1 + _l_ ) _[dx]_ [1] _[dx]_ [2] _[dx]_ [3]
_−_ _α_ [3] _c_ [3] _[α]_ lim
_l→_ 0
- + _∞_
_c_
   - + _∞_
= _α_ [3] _c_ [3] _[α]_
_c_
- + _∞_
_c_
- + _∞_
_c_
_x_ 1 + _x_ 2 + _x_ 3
3 _x_ _[α]_ 1 [+1] _x_ _[α]_ 2 [+1] _x_ _[α]_ 3 [+1]
_x_ 3
_k_ _dx_ 1 _dx_ 2 _dx_ 3
3 [(] _[x]_ [1][ +] _[ x]_ [2][ +] _[ x]_ [3][) +] _[ x]_ [2]
- + _∞_
_c_
- + _∞_
_c_
_x_ 1
_x_ _[α]_ 1 [+1] _x_ _[α]_ 2 [+1] _x_ _[α]_ 3 [+1] ( _k_ + 1 + _l_ ) _[dx]_ [1] _[dx]_ [2] _[dx]_ [3]
_−_ _α_ [3] _c_ [3] _[α]_ lim
_l→_ 0
- + _∞_
_c_
   - + _∞_
= _α_ [3] _c_ [3] _[α]_
_c_
- + _∞_
_c_
- + _∞_
_c_
_x_ 1 + _x_ 2 + _x_ 3
3 _x_ _[α]_ 1 [+1] _x_ _[α]_ 2 [+1] _x_ _[α]_ 3 [+1]
_dx_ 1 _dx_ 2 _dx_ 3
- _x_ 3 3 _x_ 2 _x_ 3 9
( _x_ 1 + _x_ 2 + _x_ 3) _k_ _[−]_ ( _x_ 1 + _x_ 2 + _x_ 3) [2] _k_ [2]
- + _∞_
_c_
- + _∞_
_c_
_x_ 1
(9)
_x_ _[α]_ 1 [+1] _x_ _[α]_ 2 [+1] _x_ _[α]_ 3 [+1] ( _k_ + 1 + _l_ ) _[dx]_ [1] _[dx]_ [2] _[dx]_ [3]
_−_ _α_ [3] _c_ [3] _[α]_ lim
_l→_ 0
- + _∞_
_c_
   - + _∞_
= _α_ [3] _c_ [3] _[α]_
_c_
- + _∞_
_c_
- + _∞_
_c_
_x_ 1
_x_ _[α]_ 1 [+1] _x_ _[α]_ 2 [+1] _x_ _[α]_ 3 [+1]
_k_ ( _k_ + 1) _[dx]_ [1] _[dx]_ [2] _[dx]_ [3]
   - + _∞_
_−_ _α_ [3] _c_ [3] _[α]_
_c_
- + _∞_
_c_
- + _∞_
_c_
_x_ _[α]_ 1 [+1] _x_ _[α]_ 2 [+1] _x_ _[α]_ 3 [+1]
3 _x_ 2 _x_ 3
(10)
_k_ [2] ( _x_ 1 + _x_ 2 + _x_ 3) _[dx]_ [1] _[dx]_ [2] _[dx]_ [3]
Here, we apply a Taylor expansion to Equation (8) and Equation (9) with respect to _l_ and [1]
Here, we apply a Taylor expansion to Equation (8) and Equation (9) with respect to _l_ and _k_ [, respectively.]
Similarly, we can calculate the value of _[dS]_ [2] [.]
[2]
_d l_ [.]
_dS_ 2
lim
_l→_ 0 _d l_
- + _∞_ - + _∞_
_c_ _c_
_x_ 2
_x_ _[α]_ 1 [+1] _x_ _[α]_ 2 [+1] _x_ _[α]_ 3 [+1]
= _α_ [3] _c_ [3] _[α]_ lim
_l→_ 0
- + _∞_
_c_
_·_ [(] _[k]_ [ + 1 +] _[ l]_ [)] _[x]_ [3] _[−]_ [(] _[k]_ 3
3 [(] _[x]_ [1] [+] _[ x]_ [2] [+] _[ x]_ [3][) +] _[ x]_ [2] [+] _[ lx]_ [3][)]
_[k]_ _dx_ 1 _dx_ 2 _dx_ 3
3 [(] _[x]_ [1][ +] _[ x]_ [2][ +] _[ x]_ [3][) +] _[ x]_ [2][ +] _[ lx]_ [3][)]
( _k_ + 1 + _l_ )( _[k]_
- + _∞_ - + _∞_
_c_ _c_
_α_ [3] _c_ [3] _[α]_  - 3 _x_ 2 _x_ 3 1
_·_
_x_ _[α]_ 1 [+1] _x_ _[α]_ 2 [+1] _x_ _[α]_ 3 [+1] _k_ ( _x_ 1 + _x_ 2 + _x_ 3) _[−]_ _k_ + 1 + _l_
= lim
_l→_ 0
- + _∞_
_c_
_dx_ 1 _dx_ 2 _dx_ 3 (11)
As a result, Equation (7) can be written as:
_Dp/D_
_k ·_ _dγp_ _**θ**_
lim _d l_
_k→_ + _∞, l→_ 0 _dγ_
_Dp/D_
_p_ _**θ**_ + _dγp_ _[D]_ _**θ**_ _[a/][D]_
_d l_ _d l_
_d l_
_Dp/D_
_k ·_ _dγp_ _**θ**_
_d l_
lim _k→_ + _∞, l→_ 0  - _k·dSd l_ 1+ _dS_ 2
=
1+ _dS_ 2
_d l_
lim _k→_ + _∞, l→_ 0 _[k][·][dS]_ [1]
_d l_
- + _∞_
_c_
_α_ [3] _c_ [3] _[α]_
~~�~~ 3
_i_ =1 _[x]_ _i_ _[α]_ [+1]
_x_ 1 3 _x_ 2 _x_ 3
_k_ ( _k_ + 1) _[−]_ _k_ [2] ( ~~[�]~~ [3] _i_ =1 _[x][i]_ [)]
_dx_ 1 _dx_ 2 _dx_ 3
lim
_k→_ + _∞, l→_ 0
- - + _∞_
_k ·_
_c_
 - + _∞_
_c_
 - + _∞_
_c_
_α_ [3] _c_ [3] _[α]_
~~�~~ 3 _·_
_i_ =1 _[x]_ _i_ _[α]_ [+1]
3 _x_ 2 _x_ 3 _−_ 1
_k_ ( ~~[�]~~ [3] _i_ =1 _[x][i]_ [)] _k_ + 1 + _l_
_dx_ 1 _dx_ 2 _dx_ 3
=
lim _k→_ + _∞, l→_ 0 _k ·_ ~~�~~ _c_ + _∞_ ~~�~~ _c_ + _∞_ ~~�~~ _c_ + _∞_ ~~�~~ 3 1
_i_ =1 _[x]_ _i_ _[α]_ [+1]
 - + _∞_
+
_c_
- + _∞_
_c_
~~�~~ _x_ 1 3 _x_ 2 _x_ 3
_k_ ( _k_ +1) _[−]_ _k_ [2] ( ~~[�]~~ [3] _i_ =1 _[x][i]_ [)]
~~�~~
_dx_ 1 _dx_ 2 _dx_ 3
lim _k→_ + _∞, l→_ 0       - _c_ + _∞_       - _c_ + _∞_       - _c_ + _∞_ ~~�~~ 3 _α_ [3] _c_ [3] _[α]_ _·_ ( _k_ +1)( _kl_ +1+ _l_ )
_i_ =1 _[x]_ _i_ _[α]_ [+1]
=
~~�~~
_dx_ 1 _dx_ 2 _dx_ 3
lim _k→_ + _∞, l→_ 0 _k ·_ ~~�~~ _c_ + _∞_ ~~�~~ _c_ + _∞_ ~~�~~ _c_ + _∞_ ~~�~~ 3 1
_i_ =1 _[x]_ _i_ _[α]_ [+1]
~~�~~ _x_ 1 3 _x_ 2 _x_ 3
_k_ ( _k_ +1) _[−]_ _k_ [2] ( ~~[�]~~ [3] _i_ =1 _[x][i]_ [)]
=0 _._ (12)
Next, we prove that _dγp_ _[D]_ _d l_ _**θ**_ [1] _[/][D]_ _<_ 0. For sufficiently large _k_, since
_dS_ 1
lim
_l→_ 0 _d l_
= _α_ [3] _c_ [3] _[α]_ lim
_l→_ 0
- + _∞_ - + _∞_
_c_ _c_
- + _∞_
_c_
_x_ 1
_x_ _[α]_ 1 [+1] _x_ _[α]_ 2 [+1] _x_ _[α]_ 3 [+1]
_k_ ( _k_ + 1) _[dx]_ [1] _[dx]_ [2] _[dx]_ [3]
- + _∞_ - + _∞_
_c_ _c_
_x_ _[α]_ 1 [+1] _x_ _[α]_ 2 [+1] _x_ _[α]_ 3 [+1]
_−_ _α_ [3] _c_ [3] _[α]_ lim
_l→_ 0
- + _∞_
_c_
3 _x_ 2 _x_ 3
_k_ [2] ( _x_ 1 + _x_ 2 + _x_ 3) _[dx]_ [1] _[dx]_ [2] _[dx]_ [3]
 - + _∞_
=
_c_
- + _∞_
_c_
- + _∞_
_c_
_k_ +1 _k_ _[x]_ [1][(] _[x]_ [1] [+] _[ x]_ [2] [+] _[ x]_ [3][)] _[ −]_ [3] _[x]_ [1] _[x]_ [3]
_dx_ 1 _dx_ 2 _dx_ 3
_k_ ( _x_ 1 + _x_ 2 + _x_ 3) _x_ _[α]_ 1 [+1] _x_ _[α]_ 2 [+1] _x_ _[α]_ 3 [+1]
- + _∞_ - + _∞_
_c_ _c_
:= [1]
_k_
- + _∞_
_c_
_f_ ( _x_ 1 _, x_ 2 _, x_ 3)
(13)
_g_ ( _x_ 1 _, x_ 2 _, x_ 3) _[dx]_ [1] _[dx]_ [2] _[dx]_ [3] _[.]_
For _c < u < v < w_, let
_f_ ˆ = _x_ 1( _x_ 1 + _x_ 2 + _x_ 3) _−_ 3 _x_ 1 _x_ 3 = _x_ 1( _x_ 1 + _x_ 2 _−_ 2 _x_ 3) _,_
    _h_ ( _u, v, w_ ) = _f_ ( _π_ ( _u, v, w_ )) _,_
_π∈S_ 3
(14)
_h_ ˆ( _u, v, w_ ) =
_π∈S_ 3
_f_ ˆ( _π_ ( _u, v, w_ )) _._
Here, _S_ 3 denotes the symmetric group of degree 3 and _π ∈_ _S_ 3 denotes a permutation in _S_ 3. Observe that
_−_
lim _,_
_k→_ + _∞_ _[h]_ [(] _[u, v, w]_ [) =ˆ] _[h]_ [(] _[u, v, w]_ [)]
_h_ ˆ( _u, v, w_ ) =
_π∈S_ 3
_f_ ˆ( _π_ ( _u, v, w_ ))
  =
_π∈S_ 3
( _u_ 0 _,v_ 0 _,w_ 0)= _π_ ( _u,v,w_ )
  =
_π∈S_ 3
( _u_ 0 _,v_ 0 _,w_ 0)= _π_ ( _u,v,w_ )
_u_ 0( _u_ 0 + _v_ 0 _−_ 2 _w_ 0)
2 [(] _[u]_ [0] _[ −]_ _[v]_ [0][)][2] _[ >]_ [ 0] _[.]_
(15)
Hence, we have
Therefore,
Therefore,
lim (16)
_k→_ + _∞_ _[h]_ [(] _[u, v, w]_ [)] _[ >]_ [ 0] _[.]_
_h_ ( _u, v, w_ ) Θ(1)
lim lim (17)
_k→_ + _∞_ _g_ ( _u, v, w_ ) [=] _k→_ + _∞_ Θ( _k_ ) [= 0][+] _[.]_
_dS_ 1
lim
_k→_ + _∞, l→_ 0 _d l_ [= lim] _k→_ + _∞_
���
( _u,v,w_ ) _∈_ ( _c,_ + _∞_ ) [3] : _u<v<w_
_h_ ( _u, v, w_ )
_g_ ( _u, v, w_ ) _[dudvdw]_
��� _h_ ( _u, v, w_ )
= lim
( _u,v,w_ ) _∈_ ( _c,_ + _∞_ ) [3] : _u<v<w_ _k→_ + _∞, l→_ 0 _g_ ( _u, v, w_ ) _[dudvdw]_
���
= lim
( _u,v,w_ ) _∈_ ( _c,_ + _∞_ ) [3] : _u<v<w_ _k→_ + _∞, l→_ 0 [0][+] _[dudvdw]_
=0 [+] _._ (18)
Hence, for sufficiently large _k_, we have
lim _dγp_ _[D]_ _**θ**_ [1] _[/][D]_ = lim (19)
_l→_ 0 _d l_ _l→_ 0 _[−]_ _[dS]_ _d l_ [1] _[<]_ [ 0] _[.]_
Thus, the proof is completed.


**B.1** **Additional Experiment of Language Models’** _**Resistance**_
To broadly validate the phenomenon of _Resistance_ in language models, we extend the experimental setup
described in Section 5.1. Specifically, we use Llama2-7B (Touvron et al., 2023) as the base model and
perform a finer-grained snapshot division within _**forward alignment**_, covering a wider range of _k_ and
_l_ . As the evaluation metric, we measure the change in relative KL divergence between the distributions
of model _**θ**_ _l_ and _**θ**_ _k_, obtained by applying _**forward alignment**_ and _**inverse alignment**_ to slices _**θ**_ _k_ and _**θ**_ _l_,
respectively, relative to the original distributions of _**θ**_ _l_ and _**θ**_ _k_ . The experimental results, which is shown
in the Table 2, demonstrate that the KL divergence under _**inverse alignment**_ is substantially smaller than
that under _**forward alignment**_, consistent with the conclusions presented in the main text.
**Training Steps** _**θ**_ 1 _vs._ _**θ**_ 2 _**θ**_ 1 _vs._ _**θ**_ 3 _**θ**_ 1 _vs._ _**θ**_ 4 _**θ**_ 1 _vs._ _**θ**_ 5 _**θ**_ 1 _vs._ _**θ**_ 6 _**θ**_ 2 _vs._ _**θ**_ 3 _**θ**_ 2 _vs._ _**θ**_ 4
_**Forward Alignment**_ 0.4568 _↑_ 0.4932 _↑_ 0.5929 _↑_ 0.4439 _↑_ 0.4109 _↑_ 0.3778 _↑_ 0.5580 _↑_
_**Inverse Alignment**_ 0.2796 _↓_ 0.2863 _↓_ 0.3960 _↓_ 0.3995 _↓_ 0.3412 _↓_ 0.2752 _↓_ 0.3742 _↓_
Table 2: **KL Divergence between different snapshots.** The results show that _**forward alignment**_ exhibits higher
values compared to _**inverse alignment**_, providing evidence for the existence of _Resistance._
**B.2** **Additional Experiment of Language Models’** _**Rebound**_
**Ablations on the Different Alignment Algorithms.** To verify that the consistent _rebound_ phenomenon
occurs in different alignment algorithms in language models, we conduct language models’ _rebound_ experiments on reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022), direct preference
optimization (DPO) (Rafailov et al., 2024), Kahneman-Tversky optimization (KTO) (Ethayarajh et al.,
2024), and simple preference optimization (SimPO) (Meng et al., 2024), using an experimental setup
similar to the SFT validation described in Section 5.2. Considering that reinforcement learning algorithms
cannot be directly applied to pretrained models, our experimental procedure consists of the following two
steps: (a) SFT of the pretrained model using positive sample data of varying scales as _**forward lignment**_ ;
(b) _**inverse alignment**_ by applying the corresponding reinforcement learning algorithms on the aligned
models, using negative sample data of varying scales, where negative samples serve as selected responses
and positive samples serve as rejected responses.
We conduct experiments using the Llama2-7B model on the IMDb task (Maas et al., 2011) with four
different RL alignment algorithms. The results, shown in Figure 7, indicate that regardless of the alignment
algorithm employed, the model exhibits a decline in performance consistent with that observed in SFT as
the amount of positive data increases. Moreover, the rate of this performance degradation accelerates with
the increasing volume of positive data. The results further confirm that the broad applicability of language
models _rebound_ across different alignment algorithms.
Figure 7: **Experimental results for validating** _**rebound**_ **across different alignment algorithms.** The heatmaps
from left to right represent the performance of the PPO, DPO, KTO, and SimPO algorithms after fine-tuning with
varying amounts of positive and negative data. Darker colors indicate better model performance in the heatmaps,.
Models trained with more **positive** data perform better initially tend to exhibit worse performance after fine-tuneing
with **negative** data. The conclusion holds regardless of the alignment algorithm used.
**Ablations on the Different Evaluation Metrics.** In the main experiments discussed in Section 5.2, we
primarily adopt specific evaluation scores to evaluate the _rebound_ phenomenon. Using the score models
corresponding to alignment targets as the evaluation metrics is the most direct and important method for
the task evaluation. However, to further reinforce the robustness of our findings and address the generality
of the _rebound_ phenomenon, we acknowledge the value of incorporating broader distribution-level metrics
beyond task-specific scoring models. In particular, we consider the Kullback-Leibler divergence between
aligned models and base models as an additional evaluation metric.
Specifically, we conducted the evaluation through the following procedure: a) We first measure the KL
divergence between the aligned model and the base model before and after fine-tuning the base model
with safety data. b) We then apply fine-tuning with unsafe data until the KL divergence between the
models decreased to a sufficiently small value _ϵ_ (where _ϵ_ = 1 _×_ 10 _[−]_ [2] ), and record the amount of unsafe
data required as an indicator of the difficulty for the model reverting to its pre-trained distribution. The
experimental results are reported in Table 3. Under different models and data scales, we observe that only
a small amount of unsafe data is needed for a positively fine-tuned model to revert to the pre-training
distribution in terms of KL divergence. Furthermore, the larger the amount of safe fine-tuning, the less
unsafe data are required. This observation is consistent with the conclusions drawn in Figure 4.
**Ablations on the Reverse Finetuning Settings.** To rule out the influence of positive data on the _rebound_
phenomenon in language models, we conducted a reverse experimental setup: negative data are used
during the SFT stage, while positive data are applied during the _**inverse alignment**_ stage. The experimental
results, presented in Figure 8, demonstrate that _elasticity_ in language models persists under this reverse
setting, exhibiting a consistent trend where larger model sizes correspond to greater _elasticity_ . The


Llama2-7B 0.21 0.22 0.26 0.27
Gemma-2B 0.18 0.21 0.24 0.25
(a) KL divergence between models fine-tuned with varying
amounts of safety data and the initial models.


Llama2-7B 961 844 801 729
Gemma-2B 923 853 709 598
(b) Unsafe data amount needed for KL divergence between
fine-tuned model and pre-trained model to drop below _ϵ_ .
Table 3: **Experimental results for validating** _**rebound**_ **on the KL divergence metrics.** The results show that,
under different models, the larger the amount of safe fine-tuning, the less unsafe data is required to revert to the
pre-training distribution in terms of KL divergence, providing evidence for the existence of _Rebound_ .
observation aligns with and further supports the results reported in Figure 5, highlighting the robustness
of _rebound_ phenomenon across different experimental configurations.
|Number of Positive Data
1000 5000 Pretrained 1.0 2000 10000 Score
0.8
Positive
0.6
0.4
Average
0.2
0.0
21 00 00 500 1000 2000
Number of Negative Data|Col2|Number of Positive Data
1000 5000 Pretrained|Col4|Col5|Col6|Col7|
|---|---|---|---|---|---|---|
|2000
500
100
200
1000
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained||~~20~~|~~00~~

|~~10000~~
|~~10000~~
||
|2000
500
100
200
1000
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||||
|2000
500
100
200
1000
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||||
|2000
500
100
200
1000
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||||
|2000
500
100
200
1000
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||||
|Number of Positive Data
1000 5000 Pretrained 1.0 2000 10000 Score
0.8
Positive
0.6
0.4
Average
0.2
0.0
1 20 00 0 500 1000 2000
Number of Negative Data|Col2|Number of Positive Data
1000 5000 Pretrained|Col4|Col5|Col6|Col7|
|---|---|---|---|---|---|---|
|200
100
2000
500
1000
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained||~~2~~|~~000~~

|~~10000~~
|~~10000~~
||
|200
100
2000
500
1000
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||||
|200
100
2000
500
1000
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||||
|200
100
2000
500
1000
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||||
|200
100
2000
500
1000
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||||
|Number of Positive Data
1000 5000 Pretrained
1.0 2000 10000
0.8
0.6
0.4
0.2
0.0
1 20 00 0 500 1000 2000
Number of Negative Data|Number of Positive Dat
1000 5000|Col3|a
Pretrained|Col5|
|---|---|---|---|---|
|200
500
1000
2000
100
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|~~2000~~|~~10000~~
|~~10000~~
||
|200
500
1000
2000
100
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||
|200
500
1000
2000
100
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||
|200
500
1000
2000
100
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||
|200
500
1000
2000
100
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||
Figure 8: **Reverse Fine-tuning Results on IMDb.** Each subfigure from left to right shows the changes in LLMs
with parameter sizes of 0.5B, 4B, and 7B, respectively. Models trained with more **negative** data initially perform
worse, but perform better after fine-tuning with **positive** data. As the model size increases, the performance of the
aligned model deteriorates more rapidly after fine-tuning with **positive** data.
**B.3** **Additional Experiment of Internal Factor of Language Models’** _**Rebound**_
**Analysis of the Model Size Scale under Different Alignment Algorithms.** To verify whether the
_rebound_ phenomenon in language models becomes more pronounced as model size increases persists
across different alignment algorithms, we conduct experiments using the DPO algorithm. Specifically, we
employ the Qwen1.5 model series as the base models and follow an experimental setup similar to that
described in Section B.2. We carried out experiments on models with parameters 0.5B, 4B, and 7B. The
experimental results, presented in Figure 9, demonstrate that _rebound_ remains evident under the DPO
algorithm, with its strength increasing as model size grows. This finding is consistent with the results
reported in Figure 5 and further highlights the generality of _elasticity_ in language models.
|Number of Positive Data
1000 5000 Pretrained 1.0 2000 10000 Score
0.8
Positive
0.6
0.4
Average
0.2
0.0
21 00 00 500 1000 2000
Number of Negative Data|Col2|Number of Positive Data
1000 5000 Pretrained|Col4|Col5|Col6|Col7|
|---|---|---|---|---|---|---|
|2000
500
100
200
1000
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained||~~20~~|~~00~~

|~~10000~~
|~~10000~~
||
|2000
500
100
200
1000
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||||
|2000
500
100
200
1000
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||||
|2000
500
100
200
1000
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||||
|2000
500
100
200
1000
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||||
|Number of Positive Data
1000 5000 Pretrained
1.0 2000 10000
0.8
0.6
0.4
0.2
0.0
1 20 00 0 500 1000 2000
Number of Negative Data|Col2|Number of Positive Data
1000 5000 Pretrained|Col4|Col5|Col6|Col7|
|---|---|---|---|---|---|---|
|200
100
2000
500
1000
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0

Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained||~~20~~|~~00~~

~~1~~|~~0000~~
|~~0000~~
||
|200
100
2000
500
1000
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0

Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||||
|200
100
2000
500
1000
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0

Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||||
|200
100
2000
500
1000
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0

Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||||
|200
100
2000
500
1000
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0

Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||||
|0
8
6
4
2|Number of Positive Data
1000 5000|Col3|Pretrained|Col5|
|---|---|---|---|---|
|2
4
6
8
0|~~2000~~|~~10000~~
|~~10000~~
||
|2
4
6
8
0|||||
|2
4
6
8
0|||||
|2
4
6
8
0|||||
|200
500
1000
2000
100
Number of Negative Data
0
|||||
Figure 9: **Experimental results for validating** _**rebound**_ **increases with model size under DPO fine-tuning.** Each
subfigure from left to right shows the changes in LLMs with parameter sizes of 0.5B, 4B, and 7B. As the model size
increases, the performance of the aligned model deteriorates more rapidly after fine-tuning with **negative** data.
**Analysis of the Pre-training Data Volume.** We present the experimental results for a broader range of
pre-training data volumes in Figure 10. When the pre-training data volume is 0.1T, 0.5T, and 1.0T, the
model still demonstrates the phenomenon that _rebound_ increases with the volume of pre-training data,
which is consistent with the results reported in Figure 6, where the pre-training data volumes range from
2.0T to 3.0T. Moreover, when comparing the experimental results in Figure 6 and Figure 10, we observe a
consistent conclusion: as the pre-training data volume increases (from 0.5T to 3.0T), the _rebound_ becomes
more pronounced. Specifically, when the pre-training data volume increases, the initial performance
decline caused by negative data fine-tuning occurs more rapidly, while the subsequent decline slows down.
This indicates that larger pre-training data volumes amplify the _rebound_ effect in language models.
|Number of Positive Data
1000 5000 Pretrained 1.0 2000 10000 Score
0.8
Positive
0.6
0.4
Average
0.2
0.0
21 00 00 500 1000 2000
Number of Negative Data|Col2|Number of Positive Data
1000 5000 Pretrained|Col4|Col5|Col6|Col7|
|---|---|---|---|---|---|---|
|100
1000
2000
200
500
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained||~~2~~|~~00~~
|~~0000~~|~~0000~~||
|100
1000
2000
200
500
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||||
|100
1000
2000
200
500
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||||
|100
1000
2000
200
500
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||||
|Number of Positive Data
1000 5000 Pretrained 1.0 2000 10000 Score
0.8
Positive
0.6
0.4
Average
0.2
0.0
21 00 00 500 1000 2000
Number of Negative Data|Col2|Number of Positive Data
1000 5000 Pretrained|Col4|Col5|Col6|Col7|
|---|---|---|---|---|---|---|
|2000
1000
500
100
200
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained||~~2~~|~~00~~
|~~0000~~|~~0000~~||
|2000
1000
500
100
200
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||||
|2000
1000
500
100
200
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||||
|2000
1000
500
100
200
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||||
|Number of Positive Data
1000 5000 Pretrained 1.0 2000 10000 Score
0.8
Positive
0.6
0.4
Average
0.2
0.0
1 20 00 0 500 1000 2000
Number of Negative Data|Col2|Number of Positive Data
1000 5000 Pretrained|Col4|Col5|Col6|Col7|
|---|---|---|---|---|---|---|
|1000
500
2000
200
100
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained||~~2~~|~~00~~
|~~0000~~|~~0000~~||
|1000
500
2000
200
100
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||||
|1000
500
2000
200
100
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||||
|1000
500
2000
200
100
Number of Negative Data
0.0
0.2
0.4
0.6
0.8
1.0
Average Positive Score
Number of Positive Data
1000
~~2000~~
5000
~~10000~~
Pretrained|||||||
Figure 10: **Experimental results for validating** _**rebound**_ **increases with model pre-training data volume.** Each
subfigure from left to right shows the changes in pre-training data volumes of 0.1T, 0.5T, and 1.0T. As pre-training
data volume increases, aligned model performance deteriorates more rapidly after fine-tuning with **negative** data.
**C** **Further Discussions of Language Models’** _**Elasticity**_
**C.1** **Transfer Learning and Language Models’** _**Elasticity**_
Transfer Learning is formally defined as follows: consider a domain _D_ = _{X_ _, P_ ( _X_ ) _}_, which consists of a
feature space _X_ and a marginal probability distribution _P_ ( _X_ ), where _X_ = _{x_ 1 _, · · ·, xn} ∈X_ . A task _T_
within the domain _D_ is defined by a label space _Y_ and a predictive function _f_ ( _·_ ). Given a source domain
_DS_ and its corresponding task _TS_, as well as a target domain _DR_ and its task _TR_, transfer learning aims
to improve the target predictive function _fR_ ( _·_ ) by leveraging knowledge from _DS_ and _TS_, where either
_DS ̸_ = _DR_ or _TS ̸_ = _TR_ (Zhuang et al., 2020; Weiss et al., 2016). From the perspective of transfer learning,
the pre-training and fine-tuning process of language models can be described as follows: Given labeled
training data from a source distribution _D_ _[s]_ = _{_ ( _xn, yn_ ) _∼_ _p}_ _[N]_ _n_ =1 _[s]_ [and data from a target distribution]
_D_ _[t]_ = _{_ ( _xn, yn_ ) _∼_ _q}_ _[N]_ _n_ =1 _[t]_ [, we first minimize the empirical risk on the source distribution to fit a model:]
_f_ _[s]_ = argmin _f_ _R_ [ˆ] ( _f, D_ _[s]_ ), where _R_ [ˆ] ( _f, D_ _[s]_ ) = E( _x,y_ ) _∼p_ [ _l_ ( _y, f_ ( _x_ ))] represents the empirical risk of model _f_
on the source distribution _p_ . Subsequently, the model is adapted to the target distribution by minimizing,
_f_ _[t]_ = argmin _f_ _R_ [ˆ] ( _f, D_ _[t]_ )+ _λ||f −_ _f_ _[s]_ _||_, where _||f −_ _f_ _[s]_ _||_ measures some distance between the two functions,
and _λ ≥_ 0 is a regularization parameter (Murphy, 2023).
Specifically in the context of language models, transfer learning can almost encompass all models
trained under the pre-training-finetuning paradigm. However, transfer learning does not capture the
specific details of the model training process, nor does it explicitly incorporate the established theories
that explain the varying degrees of difficulty in task transfer for LLMs. Therefore, while transfer learning
provides a perspective to analyze various phenomena in LLMs from the standpoint of traditional machine
learning, it does not offer concrete tools to explain these _elasticity_ phenomena.
Furthermore, in traditional transfer learning settings, positive and negative samples can only be
considered as different partitions of the same dataset in classification tasks. However, in our case, these
samples are used for autoregressive generation training, where the generation on positive samples and
negative samples follows mutually conflicting input-output distributions. This is fundamentally different
from classification tasks. Specifically for safe-unsafe generation, steering a language model’s generation
from an unsafe input-output distribution to the opposite safe input-output distribution is a major challenge
in generative language model research. This cannot be adequately explained by merely considering
different partitions of the same data distribution.
These considerations lead us to conclude that the observed phenomenon in our work represents a
novel behavior that cannot be fully accounted for by existing transfer learning theories. It warrants finergrained theoretical and experimental investigation. Consequently, we propose the concept of _elasticity_ to
formally name this new phenomenon observed in language models and introduce corresponding theoretical
frameworks and empirical methodologies to examine it. This perspective is empirically supported by the
experimental results shown in Table 1. Taking the Alpaca task as an example, we examine the process in
which _θ_ 1 and _θ_ 2 are aligned with _θ_ 3. Since _θ_ 2 has acquired more knowledge about the SFT distribution
compared to _θ_ 1, it should, from the perspective of transfer learning, find it easier to learn the distribution
of _θ_ 3 than _θ_ 1. However, as shown in the experimental results above, this is not supported, suggesting that
transfer learning alone cannot fully account for the observed _resistance_ phenomenon. [2]


The emphasis on _elasticity_ phenomenon in language model alignment highlights a crucial challenge for
ensuring the safety and robustness of open-source models throughout their entire lifecycle. As our study
demonstrates, the resistance of language models to alignment adjustments is fundamentally rooted in the
substantial disparity in data volumes across different training phases. This insight suggests a practical
mitigation strategy: customizing the scale of synthetic data through the elasticity mechanism offers a
feasible pathway for the development of more robust alignment algorithms in the future.
  - We discover that the resistance of language models to alignment is essentially due to the significant
differences in data volume across various training processes. Therefore, a straightforward idea is to
ensure that the training data volumes corresponding to different alignment targets are as similar as
possible during the alignment process. This approach helps avoid resistance effects on alignment
targets with smaller training data volumes due to subsequent perturbations.
  - The elasticity theorem provides a feasible method for customizing data ratios, thereby quantitatively
measuring the amount of various types of data needed for a model to meet expected goals. Specifically,
for an aligned language model, during subsequent alignment processes, the elasticity theorem
indicates that there is an inherent loss of elasticity dependent on data volume ratios when learning
unrelated distributions. Therefore, to ensure that previously aligned targets remain satisfied during
subsequent alignment, the training objective can be transformed into a constrained optimization
problem to quantitatively calculate the data volume required for new target alignment.
  - The current challenge in implementing algorithms based on the elasticity theory, mainly lies in the
lack of a sufficient theoretical foundation to characterize the specific features of current datasets.
This makes it difficult to distinguish the independent and differently distributed premises of different
datasets in the elasticity theory. Since real-world datasets often contain fused features, a possible
future research direction is to finely characterize the features of datasets with fused characteristics.
**C.3** **Discussions of Quantitatively Characterize the Impact of Dataset Size on** _**Elasticity**_
Considering that, as indicated in Theorem 4.2, the difference in dataset size is a key factor contributing to
the _elasticity_ phenomenon, quantitatively characterizing the impact of dataset size on _elasticity_ is indeed
crucial for both theoretical understanding and practical alignment design. In this section, we present
preliminary explorations, based on experimental observations, on quantifying the influence of dataset size
on elasticity, as well as the current limitations in achieving this goal.
By synthesizing observations from Figure 6 in the main text and Figure 10 in the appendix, we find
that for the tinyllama series models, when the pre-training data volume is merely 0.1T, there is almost
no significant resistance phenomenon observed; the positive level of the model’s output does not change
significantly with an increase in negative data. However, when the pre-training data volume reaches
0.5T, the resistance phenomenon of the language model becomes notably observable. Considering the
continuity between pre-training data and model resistance, this suggests that the significant change point
of the resistance phenomenon should lie between 0.1T and 0.5T.
Regrettably, there are still challenges in training a series of models with continuously varying pretraining data volumes, and we currently lack sufficiently fine-grained training data samples to conduct
such precise series model training. This limitation prevents us from accurately determining the critical
2As Reviewer qznt’s suggestion, we further discuss the relationship between transfer learning and LLMs’ elasticity in this
section.
point where the resistance phenomenon becomes significant. Therefore, although we can suggest a
possible range, accurately quantifying this threshold in practice remains a considerable challenge. We
hope that future research will further address this issue by developing more controllable model series and
fine-grained pre-training datasets to enable a clearer, quantitative characterization of _elasticity_ .

C.2 Discussions of Practical Steps to Mitigate _Inverse Alignment_ Risks


**Zhangchen Xu** _[♠]_ **Fengqing Jiang** _[♠]_ **Luyao Niu** _[♠]_ **Yuntian Deng** _[♢]_
**Radha Poovendran** _[♠]_ **Yejin Choi** _[♠♢]_ **Bill Yuchen Lin** _[♢]_
_♠_ University of Washington _♢_ Allen Institute for AI
[https://magpie-align.github.io/](https://magpie-align.github.io/)
[https://hf.co/magpie-align](https://hf.co/magpie-align)
ABSTRACT
High-quality instruction data is critical for aligning large language models (LLMs).
Although some models, such as Llama-3-Instruct, have open weights, their alignment data remain private, which hinders the democratization of AI. High human
labor costs and a limited, predefined scope for prompting prevent existing opensource data creation methods from scaling effectively, potentially limiting the
diversity and quality of public alignment datasets. Is it possible to synthesize
high-quality instruction data at scale by extracting it directly from an aligned
LLM? We present a _self-synthesis_ method for generating large-scale alignment data
named MAGPIE. Our key observation is that aligned LLMs like Llama-3-Instruct
can generate a user query when we input only the pre-query templates up to the
position reserved for user messages, thanks to their auto-regressive nature. We
use this method to prompt Llama-3-Instruct and generate 4 million instructions
along with their corresponding responses. We further introduce extensions of MAGPIE for filtering, generating multi-turn, preference optimization, domain-specific
and multilingual datasets. We perform a comprehensive analysis of the MAGPIEgenerated data. To compare MAGPIE-generated data with other public instruction
datasets (e.g., ShareGPT, WildChat, Evol-Instruct, UltraChat, OpenHermes, TuluV2-Mix, GenQA), we fine-tune Llama-3-8B-Base with each dataset and evaluate
the performance of the fine-tuned models. Our results indicate that using MAGPIE
for supervised fine-tuning (SFT) solely can surpass the performance of previous
public datasets utilized for both SFT and preference optimization, such as direct
preference optimization with UltraFeedback. We also show that in some tasks,
models supervised fine-tuned with MAGPIE perform comparably to the official
Llama-3-8B-Instruct, despite the latter being enhanced with 10 million data points
through SFT and subsequent preference optimization. This advantage is evident on
alignment benchmarks such as AlpacaEval, ArenaHard, and WildBench.
1 INTRODUCTION
Large language models (LLMs) such as GPT-4 (Achiam et al., 2023) and Llama-3 (Meta, 2024)
have become integral to AI applications due to their exceptional performance on a wide array
of tasks by following instructions. The success of LLMs is heavily reliant on the data used for
instruction fine-tuning, which equips them to handle a diverse range of tasks, including those not
encountered during training. The effectiveness of instruction tuning depends crucially on access to
high-quality instruction datasets. However, the alignment datasets used for fine-tuning models like
Llama-3-Instruct are typically private, even when the model weights are open, which impedes the
democratization of AI and limits scientific research for understanding and enhancing LLM alignment.
To address the challenges in constructing high-quality instruction datasets, researchers have developed
two main approaches. The first type of method involves human effort to generate and curate instruction
data (Databricks, 2023; Köpf et al., 2023; Zhao et al., 2024; Zheng et al., 2024; 2023), which is
both _time-consuming_ and _labor-intensive_ (Liu et al., 2024a). In contrast, the second type of methods
uses LLMs to produce synthetic instructions (Ding et al., 2023; Yin et al., 2023; Li et al., 2024a;
Sun et al., 2023; Taori et al., 2023; Wang et al., 2023; 2024c; Xu et al., 2023a;b; Li et al., 2023a).
Preprint
Figure 1: This figure illustrates MAGPIE, the process of self-synthesizing alignment data from aligned
LLMs (e.g., Llama-3-8B-Instruct) to create a high-quality instruction dataset. In Step 1, we input only
the pre-query template into the aligned LLM and generate an instruction along with its response using
auto-regressive generation. In Step 2, we use a combination of a post-query template and another
pre-query template to wrap the instruction generated from Step 1, prompting the LLM to generate the
response. This completes the construction of the instruction dataset. MAGPIE efficiently generates
diverse and high-quality instruction data, which can be further extended to multi-turn (MAGPIE-MT),
preference optimization (MAGPIE-DPO), domain-specific, and multilingual datasets.
Although these methods reduce human effort, its success heavily depends on prompt engineering
and the careful selection of initial seed questions. The _diversity_ of synthetic data tends to decrease
as the dataset size grows. Despite ongoing efforts, the scalable creation of high-quality and diverse
instruction datasets continues to be a challenging problem.
_Is it possible to synthesize high-quality instructions at scale by directly extracting data from advanced_
_aligned LLMs?_ A typical input to an aligned LLM contains three key components: the pre-query
template, the query, and the post-query template. For instance, an input to Llama-2-chat could be
“[INST] Hi! [/INST]”, where [INST] is the pre-query template and [/INST] is the post-query
template. These templates are predefined by the creators of the aligned LLMs to ensure the correct
prompting of the models. We observe that when we only input the pre-query template to aligned
LLMs such as Llama-3-Instruct, they _self-synthesize_ a user query due to their auto-regressive nature.
Our experiments indicate that these random user queries are of high quality and great diversity,
suggesting that the abilities learned during the alignment process are effectively utilized.
Based on these findings, we developed a self-synthesis method to construct high-quality instruction
datasets at scale, named MAGPIE (as illustrated in Figure 1). Unlike existing methods, our approach
does not rely on prompt engineering or seed questions. Instead, it _directly_ constructs instruction
data by prompting aligned LLMs with a pre-query template for sampling instructions. We also
demonstrated the extensibility of MAGPIE in generating multi-turn, preference optimization, domainspecific, and multilingual datasets. We applied MAGPIE to the Llama-3-8B-Instruct and Llama-370B-Instruct models, creating two instruction datasets: MAGPIE-Air and MAGPIE-Pro, respectively.
Our MAGPIE-Air and MAGPIE-Pro datasets were created using 206 and 614 GPU hours, respectively,
without any human intervention or API access to production LLMs like GPT-4. The statistics and
advantages of MAGPIE datasets compared to existing ones are summarized in Table 4 in Appendix A.
We perform a comprehensive analysis of these two datasets in Section 3, allowing practitioners to
filter and select data instances for fine-tuning models according to their particular needs.
To compare MAGPIE data with other public instruction datasets (e.g., ShareGPT (Chiang et al.,
2023), WildChat (Zhao et al., 2024), Evol Instruct (Xu et al., 2023a), UltraChat (Ding et al., 2023),
OpenHermes (Teknium, 2023a;b), GenQA (Chen et al., 2024), Tulu V2 Mix (Ivison et al., 2023)), we
conducted supervised fine-tuning (SFT) of the Llama-3-8B-Base model with each dataset and assess
the performance of the fine-tuned models on alignment benchmarks such as AlpacaEval 2 (Li et al.,
2023b), Arena-Hard (Li et al., 2024b), and WildBench (Lin et al., 2024). Our results show that models
supervised fine-tuned with MAGPIE achieve superior performance, even surpassing models that utilize
both SFT and direct preference optimization (DPO) (Rafailov et al., 2023) with UltraFeedback (Cui
et al., 2023). Notably, MAGPIE-aligned models outperform the official Llama-3-8B-Instruct model
on AlpacaEval 2, despite the latter being fine-tuned with over 10 million data points for SFT and
Preprint
subsequent preference optimization. Not only does MAGPIE excel in SFT alone compared to prior
public datasets, but also delivers the best results when combined with preference optimization methods
such as DPO. By leveraging MAGPIE extensions to generate high-quality preference optimization
datasets, MAGPIE-aligned Llama-3 models can even outperform GPT-4-Turbo(1106) on AlpacaEval
2. These findings show the exceptional quality of instruction data generated by MAGPIE, enabling it
to outperform even the official, extensively optimized, and proprietary LLMs.
2 MAGPIE: A SCALABLE METHOD TO SYNTHESIZE ALIGNMENT DATA
**Chat Templates of Aligned LLMs.** For an aligned LLM (e.g., Llama-3-8B-Instruct), the input
sequence can be represented as _x_ = _Tpre−query ⊕_ _q ⊕_ _Tpost−query_ . Here, _q_ is the user query (e.g.,
"What material should I use to build a nest?"), while _Tpre−query_ and _Tpost−query_ are pre-query and
post-query templates. The pre-query template shows up before the user query, and the post-query
template is defined as the conversation template between the user query and the LLM response. These
templates are defined by the model provider to ensure the correct prompting. For example, for Llama3-8B-Insturct model, _Tpre−query_ = <|start_header_id|>user<|end_header_id|>, and
_Tpost−query_ =<|eot_id|><|start_header_id|>assistant<|end_header_id|>.
2.1 MAGPIE PIPELINE
**Overview of MAGPIE.** In what follows, we describe our lightweight and effective method, MAGPIE,
to synthesize alignment data from aligned LLMs. An instance of instruction data consists of at least
one or multiple instruction-response pairs. Each pair specifies the roles of instruction provider (e.g.,
user) and follower (e.g., assistant), along with their instruction and response. As shown in Figure 1,
MAGPIE consists of two steps: (1) instruction generation, and (2) response generation. The MAGPIE
pipeline can be fully _automated without any human intervention_, and can be readily adapted for the
generation of multi-turn, preference, and domain-specific datasets, as detailed in Section 2.2. We
describe each step in the following.
**Step 1: Instruction Generation.** The goal of this step is to generate an instruction for each instance
of instruction data. Given an open-weight aligned LLM (e.g., Llama-3-70B-Instruct), MAGPIE crafts
a pre-query template in the format of the predefined instruction template of the LLM. Note that
the auto-regressive LLM has been fine-tuned using instruction data in the format of the pre-query
template. Thus, the LLM autonomously generates an instruction when the pre-query template crafted
by MAGPIE is given as an input. MAGPIE stops generating the instruction once the LLM produces
an end-of-sequence token. Sending the crafted query to the LLM multiple times leads to a set of
instructions. We note that compared with existing synthetic approaches (Ding et al., 2023; Li et al.,
2024a; Taori et al., 2023; Wang et al., 2023; 2024c; Xu et al., 2023a;b), MAGPIE does not require
specific prompt engineering techniques since the crafted query follows the format of the predefined
instruction template. In addition, MAGPIE autonomously generates instructions without using any
seed question, ensuring the diversity of generated instructions.
**Step 2: Response Generation.** The goal of this step is to generate responses to the instructions
obtained from Step 1. MAGPIE sends these instructions to the LLM to generate the corresponding
responses. Combining the roles of instruction provider and follower, the instructions from Step 1,
and the responses generated in Step 2 yields the instruction dataset.
**Applicability of MAGPIE on Different LLMs.** MAGPIE can be readily deployed to state-of-the-art
open-weight language models including but not limited to Llama-3 (Meta, 2024), Llama-3.1 (Dubey
et al., 2024), Qwen2 (Yang et al., 2024), Gemma-2 (Team et al., 2024), and Phi-3 (Abdin et al., 2024).
Please refer to Appendix A for detailed support and corresponding datasets.
**Remark.** MAGPIE generates high-quality instructions even when the instruction loss is masked during
alignment. We hypothesize that LLMs retain an implicit memorization of instruction distributions.
We leave it as a potential future research problem.
2.2 MAGPIE EXTENSIONS
**Dataset Filtering.** MAGPIE allows practitioners to select instruction data from the raw dataset
generated from the above two steps based on their needs. In Appendix C, we explores potential filter
Preprint
Figure 2: This figure illustrates how to control generating math and Chinese instructions using
system prompts. The system prompt in Chinese translates to: "You are an AI assistant designed to
provide accurate and concise answers to users’ questions in Chinese." The user’s response in Chinese
translates to: "I want to learn about some basic concepts and historical development trends of AI."
configurations with eight available metrics for users to customize their own MAGPIE datasets. We
also provide 6 off-the-shelf filter configurations and discuss their performance in Appendix F.4.
**Generating Multi-Turn Instruction Datasets.** MAGPIE can be readily extended to generate multiturn instruction datasets. To construct a multi-turn dataset (denoted as MAGPIE-MT), we initially
follow Steps 1 and 2 to generate the first turn of instruction and response. For subsequent turns, we
append the pre-query template to the end of the full prompt from the previous round of communication.
We observe that the model may occasionally forget its role as the user, especially for the 8B model. To
mitigate this, we employ a system prompt designed to control the behavior of the LLM and reinforce
its awareness of the multi-round conversation context. The full prompt for building the instructions
of MAGPIE-MT can be found in Figure 14 in Appendix G. We follow the procedure in Step 2 of
Section 2.1 to generate responses to form the multi-turn instruction dataset.
**Generating Preference Optimization Datasets.** Leveraging the diverse and high-quality instructions
produced by MAGPIE, we present a simple and effective method for generating preference optimization data, inspired by Meng et al. (2024) and Tran et al. (2023). We first select a small proportion of
high-quality instructions from the raw dataset generated by the MAGPIE pipeline, ensuring diverse
task categories. For each selected instruction, we sample responses from the aligned LLM _k_ times,
using a temperature of _T <_ 1. We then employ a reward model (RM) to annotate scores for these
responses. The response with the highest RM score is labeled as the chosen response, while the one
with the lowest RM score is designated as the rejected response.
**Generating Domain-Specific and Multilingual Datasets.** In certain scenarios, users may wish
to fine-tune LLMs using domain-specific or multilingual instruction data to enhance performance
within specific domains or languages. To address this need, we introduce a lightweight method to
control both the task category and the language of generated instructions. Our approach involves
guiding LLMs through a tailored system prompt, specifying that the model is a chatbot designed
for a particular domain and outlining the types of user queries it might encounter. Figure 2 shows
how to control generating math and Chinese instructions using system prompts. More examples of
system prompts designed to control the generation of code, translation, and multilingual instructions
are illustrated in Figure 15 in Appendix G.
Furthermore, we note that domain-specific and multilingual instruction data can also be generated
using models that are tailored to particular fields. MAGPIE demonstrates broad applicability beyond
diverse chat models, extending to specialized code models (e.g., DeepSeek-Coder-V2 (Zhu et al.,
2024)) and math models (e.g., Qwen2-Math-7B-Instruct (Yang et al., 2024)). By leveraging the
unique strengths and specializations of different models, MAGPIE can create a rich and diverse corpus
of instructions. Examples of MAGPIE-generated instructions from different domain-specific models
and multilingual models are provided in Appendix H.
Preprint
3 DATASET ANALYSIS
To demonstrate the effectiveness of MAGPIE compared with baseline methods for generating diverse
high-quality alignment datasets, we apply MAGPIE to the Llama-3-8B-Instruct and Llama-3-70BInstruct models (Meta, 2024) to construct two instruction datasets: Llama-3-MAGPIE-Air (hereafter
referred to as MAGPIE-Air) and Llama-3-MAGPIE-Pro (hereafter referred to as MAGPIE-Pro), respectively. Examples of instances in both datasets can be found in Appendix H. In this section, we present
a comprehensive analysis of the MAGPIE-Air and MAGPIE-Pro datasets, including topic coverage,
difficulty, quality, similarity of instructions, and the quality of the responses. We also provide the
safety analysis and cost analysis of MAGPIE.
3.1 DATASET COVERAGE
We follow Zhao et al. (2024) and analyze the coverage of MAGPIE-Pro in the embedding space.
Specifically, we use the all-mpnet-base-v2 embedding model [1] to calculate the input embeddings, and employ t-SNE (Van der Maaten & Hinton, 2008) to project these embeddings into a
two-dimensional space. We adopt three synthetic datasets as baselines, including **Alpaca** (Taori
et al., 2023), **Evol Instruct** (Xu et al., 2023a), and **UltraChat** (Ding et al., 2023), to demonstrate the
coverage of MAGPIE-Pro. The detailed analysis can be found in Appendix D.1. We observe that the
t-SNE plot of MAGPIE-Pro encompasses the area covered by the plots of Alpaca, Evol Instruct, and
UltraChat. This suggests that MAGPIE-Pro provides a broader or more diverse range of topics.
We also follow Wang et al. (2023) and present the most common verbs and their top direct noun
objects in instructions in Appendix D, indicating the diverse topic coverage of MAGPIE dataset.
3.2 DATASET ATTRIBUTES
**Attribute: Task Categories of Instructions.** We use Llama-3-8B-Instruct to categorize the instances in MAGPIE-Pro (see Figure 9 in Appendix D.1 for detail). The prompts used to query
Llama-3-8B-Instruct can be found in Appendix G. Our observations indicate that over half of the
tasks in MAGPIE-Pro pertain to information seeking, making it the predominant category. This is
followed by tasks involving creative writing, advice seeking, planning, and math. This distribution
over the task categories aligns with the practical requests from human users (Li et al., 2023b).
**Attribute: Quality of Instructions.** Similar to methods in
(Chen et al., 2023), we prompt the Llama-3-8B-Instruct model
to assess the quality of each instruction in MAGPIE-Air and
MAGPIE-Pro, categorizing them as ‘very poor’, ‘poor’, ‘average’, ‘good’, and ‘excellent’. We present the histograms of
qualities for both datasets in Figure 3-(a). We have the following two observations. First, both datasets are of high quality,
with the majority of instances rated ‘average’ or higher. In
addition, the overall quality of MAGPIE-Pro surpasses that of
MAGPIE-Air. We hypothesize that this is due to the enhanced
capabilities of Llama-3-70B compared with Llama-3-8B.
(a) Statistics on Input Quality
**Attribute: Difficulty of Instructions.** We use the Llama-3
MAGPIE-Air and MAGPIE-Pro. Each instruction can be labeled
as ‘very easy’, ‘easy’, ‘medium’, ‘hard’, or ‘very hard’. Fig- Figure 3: The statistics of instrucure 3-(b) presents the histograms of the levels of difficulty for tion difficulty and quality.
MAGPIE-Air and MAGPIE-Pro. We observe that the distributions across difficulty levels are similar for MAGPIE-Air and MAGPIE-Pro. Some instructions in
MAGPIE-Pro are more challenging than those in MAGPIE-Air because MAGPIE-Pro is generated by a
more capable model (Llama-3-70B-Instruct).
(b) Statistics on Input Difficulty
Figure 3: The statistics of instruction difficulty and quality.
[1https://huggingface.co/sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2)
Preprint
**Attribute: Instruction Similarity.** We quantify the similarity
among instructions generated by MAGPIE to remove repetitive instructions. We measure the similarity using **minimum**
**neighbor distance** in the embedding space. Specifically, we
first represent all instructions in the embedding space using
the all-mpnet-base-v2 embedding model. For any given
instruction, we then calculate the minimum distance from the instruction to its nearest neighbors in the embedding space using
Facebook AI Similarity Search (FAISS) (Douze et al., 2024).
The minimum neighbor distances of instructions in MAGPIEAir after removing repetitions are summarized in Figure 4-(a).
(a) Min Input Embedding Distance
(b) Reward Difference between Base Model and Instruct Model
**Attribute: Quality of Responses.** We assess the quality of
Figure 4: This figure summarizes
responses using **rewards** assigned by a reward model, denoted
the minimum neighbor distances
as _r_ _[∗]_ . For each instance in our dataset, we also calculate **reward**
and reward differences.
**difference** as _r_ _[∗]_ _−_ _rbase_, where _rbase_ is the reward assigned
by the same reward model to the response generated by the
Llama-3 base model for the same instruction. We use URIAL (Lin et al., 2023) to elicit responses
from the base model. A positive reward difference indicates that the response from our dataset is
of higher quality, and could potentially benefit instruction tuning. In our experiments, we follow
Lambert et al. (2024) and choose FsfairX-LLaMA3-RM-v0.1 (Xiong et al., 2024) as the reward
model. Our results on the reward difference are presented in Figure 4-(b).
3.3 SAFETY ANALYSIS
We use Llama-Guard-2 (Team, 2024) to analyze the safety of MAGPIE-Air and MAGPIE-Pro. Our
results indicate that both datasets are predominantly safe, with less than 1% of the data potentially
containing harmful instructions or responses. Please see Appendix D.2 for detailed safety analysis.
3.4 COST ANALYSIS
We perform experiments on a server with four NVIDIA A100-SXM4-80GB GPUs, an AMD EPYC
7763 64-Core Processor, and 512 GB of RAM, using the VLLM inference framework (Kwon et al.,
2023). The models are loaded in the bfloat16 format.
When creating the 3M MAGPIE-Air dataset, our MAGPIE spent 1.55 and 50 hours to generate the
instructions (Step 1) and responses (Step 2), respectively. For the 1M MAGPIE-Pro dataset, MAGPIE
used 3.5 and 150 hours to generate the instructions (Step 1) and responses (Step 2), respectively.
Compared to existing approaches to create instruction datasets, the pipeline of MAGPIE is fully
automated without any human intervention or API access to advanced commercial models such as
GPT-4 (Achiam et al., 2023). Consequently, MAGPIE is cost-effective and scalable. On average,
implementing MAGPIE on a cloud server [2] would incur costs of **$0.12** and **$1.1** per 1,000 data instances
for MAGPIE-Air and MAGPIE-Pro, respectively.
3.5 ADDITIONAL ANALYSIS
Additional dataset analysis, including the impact of generation configurations on the quality and
difficulty of the generated instructions, is detailed in Appendix D.3. We provide ablation analysis on
annotating models for assessing quality and difficulty in Appendix D.4.
4 PERFORMANCE ANALYSIS
In this section, we evaluate the quality of MAGPIE-generated datasets by utilizing them to align base
models including Llama-3 (Meta, 2024), Qwen1.5 (Bai et al., 2023), and Qwen2 (Yang et al., 2024).
[2https://lambdalabs.com/service/gpu-cloud](https://lambdalabs.com/service/gpu-cloud)
Preprint
4.1 EXPERIMENTAL SETUPS
**Baselines for Supervised Fine-Tuning and Preference Optimization.** We compare the family
of instruction datasets generated by MAGPIE with eight SOTA open-source instruction datasets:
**ShareGPT** (Chiang et al., 2023), **WildChat** (Zhao et al., 2024), **Evol Instruct** (Xu et al., 2023a),
**UltraChat** (Ding et al., 2023), **GenQA** (Chen et al., 2024), **OpenHermes 1** (Teknium, 2023a),
**OpenHermes 2.5** (Teknium, 2023b), and **Tulu V2 Mix** (Ivison et al., 2023). ShareGPT and WildChat
are representative human-written datasets containing 112K and 652K high-quality multi-round
conversations between humans and GPT, respectively. Evol Instruct, UltraChat, and GenQA are
representative open-source synthetic datasets. Following Meng et al. (2024), we use the 208K
sanitized version of Ultrachat provided by HuggingFace [3] . OpenHermes 1, OpenHermes 2.5, and
Tulu V2 Mix are crowd-sourced datasets consisting of a mix of diverse open-source instruction
datasets, with 243K, 1M, and 326K conversations, respectively. We also create an instruction dataset
with 100K conversations using the Self-Instruct (Wang et al., 2023) and Llama-3-8B-Instruct model,
denoted as **Self-Instruct (Llama-3)** .
We compare the models aligned using MAGPIE with preference optimization baselines using direct
preference optimization (DPO) (Rafailov et al., 2023). Specifically, we follow Meng et al. (2024)
and use the models fine-tuned with the UltraChat dataset (for instruction tuning) and **Ultrafeedback**
dataset (for preference optimization) (Cui et al., 2023).
**MAGPIE Setups.** To demonstrate the quality of MAGPIE-generated instruction datasets for SFT, we
select the first 300K **MAGPIE-Air** and **MAGPIE-Pro** raw datasets generated by Llama-3-8B-Instruct
and Llama-3-70B-Instruct models, respectively. Apart from these raw datasets, we also applied the
filters detailed in Appendix C and created two filtered datasets: **MAGPIE-Air-Filtered** and **MAGPIE-**
**Pro-Filtered**, each contains 300K conversations. For preference optimization, we generate two
additional datasets: **MAGPIE-Air-DPO** (generated by Llama-3-8B-Instruct) and **MAGPIE-Pro-DPO**
(generated by Llama-3-70B-Instruct) with _k_ = 5 and _T_ = 0 _._ 8, each contains 100K conversations.
We use RLHFlow/ArmoRM-Llama3-8B-v0.1 (Wang et al., 2024a) as the reward model.
**Model Alignment Details.** For supervised fine-tuning, we follow Touvron et al. (2023) and use a
cosine learning rate schedule with an initial learning rate of 2 _×_ 10 _[−]_ [5] when fine-tuning Llama-3,
Qwen1.5 and Qwen2 base models. The maximum sequence length is 8192. For DPO, we use a
cosine learning rate of 5 _×_ 10 _[−]_ [7] . The detailed parameters can be found in Appendix E.2. We follow
the official instruction templates of each model.
**Evaluation Benchmarks.** We evaluate the performance of the aligned models using two widely
adopted instruction-following benchmarks: AlpacaEval 2 (Li et al., 2023b) and Arena-Hard (Li et al.,
2024b). AlpacaEval 2 consists of 805 representative instructions chosen from real user interactions.
Arena-Hard is an enhanced version of MT-Bench (Zheng et al., 2023), containing 500 challenging
user queries. Both benchmarks employ a GPT evaluator to assess responses generated by the model
of interest and a baseline model. Specifically, we use GPT-4-Turbo (1106) and Llama-3-8B-Instruct
as baselines for AlpacaEval 2. By default, Arena-Hard uses GPT-4 (0314) as its baseline model.
**Metrics.** We adopt two metrics to measure the capabilities of instruction-following of fine-tuned
models. The first metric is the **win rate (WR)**, which calculates the fraction of responses that are
favored by the GPT evaluator. This metric is applied in both benchmarks including AlpacaEval 2
and Arena-Hard. The second metric is the **length-controlled win rate (LC)** (Dubois et al., 2024),
a debiased version of WR. The GPT evaluator considers the lengths of responses generated by
the baseline model and model under evaluation when computing LC. By accounting for response
length, LC reduces its impact on the win rate. This metric is specifically applied to the AlpacaEval 2
benchmark (Li et al., 2023b).
**More Experimental Setups.** We provide more detailed descriptions of our experimental setups,
including more model alignment details and benchmark decoding hyper-parameters in Appendix E.
4.2 EXPERIMENTAL RESULTS
[3https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k)
[4https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)
Preprint
Table 1: This table compares the performance of models instruction-tuned on the Llama-3-8B base
models using MAGPIE-generated datasets and baseline datasets. We observe that models aligned
with our datasets significantly outperform those aligned with baseline datasets of the same order of
magnitude in terms of data size. In addition, our fine-tuned models achieve comparable performance
to the official aligned model, despite only undergoing SFT with a much smaller dataset. Numbers in
**bold** indicate that MAGPIE outperforms the official Llama-3-8B-Instruct model.

Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs
  with Nothing

Zhangchen Xu
Fengqing Jiang
Luyao Niu
Yuntian Deng
Radha Poovendran
Yejin Choi
Bill Yuchen Lin


GPT-4-Turbo (1106) Llama-3-8B-Instruct
LC (%) WR (%) SD LC (%) WR (%) SD WR(%)
SFT +Self-Instruct (Llama-3) (Wang et al., 2023) 100K 7.21 5.18 0.7 17.86 12.73 1.05 4.0
+ShareGPT (Chiang et al., 2023) 112K 9.73 7.2 0.81 27.26 18.32 1.18 6.5
+Evol Instruct (Xu et al., 2023a) 143K 8.52 6.25 0.76 20.16 14.98 1.1 5.1
+OpenHermes 1 (Teknium, 2023a) 243K 9.94 6.27 0.73 29.19 17.92 1.16 4.4
+Tulu V2 Mix (Ivison et al., 2023) 326K 9.91 7.94 0.86 24.28 18.64 1.18 5.4
+WildChat (Zhao et al., 2024) 652K 14.62 10.58 0.92 34.85 26.57 1.32 8.7
+OpenHermes 2.5 (Teknium, 2023b) 1M 12.89 9.74 0.91 32.68 25.01 1.30 8.2
+GenQA (Chen et al., 2024) 6.47M 9.05 7.11 0.82 21.90 16.09 1.12 3.0
+UltraChat (Ding et al., 2023) 208K 8.29 5.44 0.71 23.95 15.12 1.11 3.6
+ DPO +UltraFeedback((Cui et al., 2023)) 64K 18.36 17.33 1.14 44.42 42.36 1.46 14.8
SFT **MAGPIE** -Air-300K-Raw 300K 21.99 21.65 1.21 48.63 48.06 1.42 15.8
**MAGPIE** -Air-300K-Filtered 300K 22.66 **23.99** 1.24 49.27 **50.8** 1.44 14.9
+ DPO + **MAGPIE** -Air-DPO 100K **45.48** **50.43** 1.48 **75.06** **79.64** 1.18 **35.9**
SFT **MAGPIE** -Pro-300K-Raw 300K 21.65 22.19 1.2 49.65 **50.84** 1.42 15.9
**MAGPIE** -Pro-300K-Filtered 300K **25.08** **29.47** 1.35 **52.12** **53.43** 1.44 18.9
+ DPO + **MAGPIE** -Pro-DPO 100K **50.10** **53.53** 1.45 **78.52** **80.82** 1.17 **35.7**
Llama-3-8B-Instruct (SFT+DPO) _>_ 10M [4] 22.92 22.57 1.26 50 50   - 20.6
**MAGPIE datasets outperform baselines with SFT only.** In Table 1, we compare the performance of Llama-3 models fine-tuned with instruction datasets generated by MAGPIE against
those supervised fine-tuned with baseline datasets. Using the AlpacaEval 2 benchmark, we observe that both the LC and WR of our supervised fine-tuned models surpass all those models fine-tuned with baseline SFT datasets. This indicates that the datasets generated by MAGPIE are of higher quality, leading to significantly enhanced instruction-following capabilities.
A similar observation is made when using the Arena-Hard evaluation benchmark. We highlight that the Llama-3 base models supervised fine-tuned with instruction datasets generated by
MAGPIE outperform even those models that have undergone preference optimization (i.e., STF
followed by DPO), which further emphasizes the high quality of data generated by MAGPIE.
To investigate the advantages of MAGPIE across different task categories, we also compare the performance
of models fine-tuned with MAGPIE-Pro compared with
baseline datasets using WildBench benchmark (Lin et al.,
2024). This benchmark consists of 1024 tasks carefully
selected from real-world human-LLM conversation logs.
The results are demonstrated in Figure 5. We observe that
MAGPIE consistently outperforms baseline datasets across
categories.


**the official aligned model, but with fewer data.** In Table
1, we also compare the performance of models aligned
with data generated by MAGPIE against the official aligned
model (Llama-3-8B-Instruct). We observe that the Llama- Figure 5: This figure shows the perfor3-8B base model supervised fine-tuned with data from mance breakdown by category of MAGMAGPIE outperforms Llama-3-8B-instruct using the Al- PIE-Pro and baselines on WildBench.
pacaEval 2 benchmark. For example, when Llama-3-8BInstruct is chosen as the baseline model of AlpacaEval 2, we observe that LC of Llama-3-8B base
models fine-tuned with instruction data from MAGPIE exceeds 50%, indicating a preference for our
SFT models over the official aligned model. In addition, when DPO is applied, our aligned model
Figure 5: This figure shows the performance breakdown by category of MAGPIE-Pro and baselines on WildBench.
Preprint
Table 2: This table compares the performance of models instruction-tuned on the Qwen base models
using the MAGPIE-Pro-300K-Filtered dataset and the official instruction-tuned models. The Qwen
base model enhanced with MAGPIE outperforms the official instruction-tuned model.

Models aligned with data generated by MAGPIE