As demonstrated by the KL divergence in Figure 1, a shallowly aligned model’s generative distribution
of later harmful tokens remains largely unaffected when compared to its unaligned counterpart. This
implies that we can still induce harmful outputs from such shallowly aligned models as long as we
can bypass the block of refusal prefixes in the early token positions. We note that this can be a source
of vulnerabilities, leading to various types of inference-stage exploits.
**Prefilling Attacks.** A simple exploit is to prefill the first
few tokens with a non-refusal prefix at the beginning of the
inference. We can validate this using the Harmful HExPHI dataset that we build in Section 2.2. For each harmful data pair ( _**x**_ _,_ _**y**_ ) from this dataset, we sample outputs
_**y**_ ˆ _∼_ _πθ_ ( _· |_ _**x**_ _,_ _**y**_ _≤k_ ). This tests whether the model would generate harmful content if the first _k_ tokens are prefilled with
a non-refusal prefix _**y**_ _≤k_ . The Attack Success Rate (ASR)
in relation to _k_ is plotted in Figure 2. As shown, when conditioned on an increasing number of harmful tokens, the
Figure 2: ASR vs. Number of Pre- aligned models’ likelihood of generating harmful content infilled Harmful Tokens, with ˆ _**y**_ _∼_ creases quickly from near zero to over 50%. This suggests a
_πθ_ ( _·|_ _**x**_ _,_ _**y**_ _≤k_ ) on Harmful HEx-PHI. practical safety risk as the decoding of open-source LLMs is
readily controlled by attackers. Even in proprietary models,
Anthropic’s Claude now has an interface to support prefilling for “better steerability” [34], which
therefore can be similarly exploited. Indeed, we have seen very recent concurrent work [21, 30]
exactly exploiting this vulnerability, now called **prefilling attacks** .
3Using the terminology “spending” KL on a KL “budget” from Gao et al. [33].
**Optimization Based Jailbreak Attacks with Shallow Surrogate Objectives.** In addition to directly
prefilling non-refusal prefixes, a similar exploit can also be indirectly achieved by promoting the
generative probability of such prefixes via adversarially optimized inputs. Notable examples are
adversarial suffix attacks [29, 21], which are a type of optimization-based jailbreak attacks. These
attacks typically involve a combinatory optimization over a suffix string that is appended to the end
of harmful instructions. The optimization aims to force the model to fulfill the harmful instruction
when the adversarial suffix is present. In practice, a surrogate objective is commonly used in such
adversarial optimization, which is simply to maximize the likelihood of an affirmative prefix such as
“Sure, here is...”. Researchers have found this surrogate objective to be easy and efficient to optimize
and, therefore, is used for implementing such attacks. Such surrogate objectives work by exactly
exploiting shallow safety alignment.
**Jailbreak via Mere Random Sampling.** Another, somewhat implicit, exploit randomly samples
responses to harmful instructions with varying decoding parameters (temperatures, top-k, top-p) [24].
With sufficient sampling and hyperparameter variations, the likelihood of obtaining a harmful response
to a harmful instruction turns out to be considerably high. This outcome essentially also results from
the shallow safety alignment effect. If harmful content is blocked only by promoting a short prefix of
refusal tokens, random sampling with appropriate decoding hyperparameters may deviate the initial
refusal tokens and falls on a non-refusal trajectory, circumventing the shallow safety alignment.
**Remark.** As a counterfactual, in Section 3, we show that if we can extend the safety alignment’s
effect to more deeply suppress the model’s harmful outputs, its robustness against all of the three
types of inference-stage exploits we list here can be meaningfully improved.


(a) Per-token Cross-Entropy Loss
on The Fine-tuning Dataset
(b) Per-token Gradient Norm
on The Fine-tuning Dataset
(c) Per-token KL Divergence on
HEx-PHI Safety Test Dataset [32]
Figure 3: Then per-token dynamics when fine-tuning Llama-2-7B-Chat on the 100 Harmful Examples
from Qi et al. [22]. _Note: 1) ASR of initially aligned model =_ _1.5%; 2) After 2 gradient steps =_
_22.4%; 3) After 4 gradient steps = 76.4%; 4) After 6 gradient steps = 87.9%._
Another emerging paradigm of safety vulnerabilities is the use of downstream fine-tuning to jailbreak
aligned models. Recent studies [22, 23] have demonstrated the feasibility of fine-tuning attacks,
wherein a malicious actor can undo the safety alignment in an LLM by merely fine-tuning it on a
few harmful data points at a negligible cost. Notably, Qi et al. [22] and He et al. [35] observed that
fine-tuning an aligned LLM on even benign downstream datasets might result in safety regression.
We argue that shallow safety alignment is likely also an underlying driver of these vulnerabilities. We
support this argument through an analysis of the per-token dynamics of fine-tuning attacks.
Formally, the standard custom fine-tuning of an aligned LLM on a dataset _D_ is characterized by the
following optimization loss function, where _πθ_ is initialized with the aligned model _π_ aligned:
        -        E _**y**_ _|_ _**x**_
( _**x**_ _,_ _**y**_ ) _∼D_ _[−]_ [log] _[ π][θ]_
_|_ _**y**_ _|_
- log _πθ_ - _yt|_ _**x**_ _,_ _**y**_ _<t_
_t_ =1
min
_θ_
= min
_θ_
E
( _**x**_ _,_ _**y**_ ) _∼D_ _[−]_
(1)
We investigate the **per-token dynamics of the fine-tuning process** by separately examining:
1. The per-token cross-entropy loss at each token position _t_ : _−_ log _πθ_ - _yt |_ _**x**_ _,_ _**y**_ _<t_ �.
2. The gradient magnitude of the per-token loss: �� _∇_ log _πθ_ - _yt |_ _**x**_ _,_ _**y**_ _<t_ ���2 [.]
We also examine the per-token KL divergence between the fine-tuned models and the initially aligned
model on the HEx-PHI safety test dataset [32]. Specifically, for each harmful instruction ˜ _**x**_, we take
the outputs ˜ _**y**_ _∼_ _πθ_ ( _· |_ _**x**_ ˜) from the fine-tuned model _πθ_ . Then, for each token position _t_, we compute
the KL divergence _D_ KL� _πθ_ ( _· |_ _**x**_ ˜ _,_ ˜ _**y**_ _<t_ ) �� _π_ aligned( _· |_ _**x**_ ˜ _,_ ˜ _**y**_ _<t_ )�.
Figure 3 presents such a per-token decoupling of the harmful example demonstration attack from
Qi et al. [22]. Here, a safety-aligned model (Llama-2-7B-Chat in our case) is fine-tuned on 100
(harmful instruction, harmful answer) data pairs, with a learning rate of 2 _×_ 10 _[−]_ [5] and a batch
size of 64. Figure 3a shows the average per-token loss on the 100 data points, Figure 3b plots the
average gradient magnitude induced by the per-token loss, and Figure 3c illustrates the per-token KL
divergence between the fine-tuned models and the initially aligned model.
**Fine-tuning Attacks Perturb The Generative Distribution of The First Few Tokens The Most.**
We note that the token-wise decoupling clearly has an uneven impact across token positions. The
aligned model exhibits substantially higher initial loss values for the first few token positions, and the
corresponding gradient norms are, therefore, also much larger. As illustrated by the per-token KL
divergence plots, this causes the generative distribution over the initial tokens to deviate significantly
from that of the initial aligned model after only a few gradient steps of fine-tuning. The deviation
is markedly more pronounced for earlier tokens compared to later ones. Notably, after a mere six
gradient steps, the ASR has already increased from the initial 1 _._ 5% to 87 _._ 9%. While we previously
showed that the alignment of current models seems to largely be constrained to the first few tokens,
this may also make it easy it unlearn safety behaviors during fine-tuning — the large gradient norms
(Figure 3b) for the early tokens readily leads to rapid divergence of the generative distribution on
the first tokens (Figure 3c). Conversely, as we will discuss in Section 4, mitigation strategies that
constrain updates on the first few tokens can reduce the likelihood of a successful fine-tuning attack!
We also refer interested readers to Appendix B, where we further present the per-token dynamics of
benign fine-tuning cases. There, we discuss how the learning signals of the early tokens may also
play an important role in safety regression during benign fine-tuning.

Safety Alignment Should Be Made More Than Just a Few Tokens Deep

2.3.2 Safety Vulnerabilities in The Stage of Downstream Fine-tuning

Xiangyu Qi
Ashwinee Panda
Kaifeng Lyu
Xiao Ma
Subhrajit Roy
Ahmad Beirami
Prateek Mittal
Peter Henderson


Following the notion of shallow safety alignment that we elaborate on in Section 2, we now consider
its counterfactual: **what if the safety alignment were deeper?** Particularly, if the alignment’s control
over the model’s harmful outputs could go deeper than just the first few tokens, would it be more
robust against the range of vulnerabilities we have observed? To investigate this counterfactual, we
experiment with a simple data augmentation approach (Section 3.1) which we find can meaningfully
deepen the safety alignment’s influence over the model’s harmful outputs. In Section 3.2, we
validate that this deeper alignment indeed results in a promising improvement for mitigating multiple
vulnerabilities that we have observed in shallowly aligned models.

3 What If The Safety Alignment Were Deeper?


Formally, let’s use _**x**_ _,_ _**h**_ to denote a harmful instruction ( _**x**_ ) and its harmful response ( _**h**_ ). As noted
in Section 2, a shallow safety alignment can keep the probability of the harmful response _πθ_ ( _**h**_ _|_ _**x**_ )
low, but this is achieved by merely suppressing the initial tokens of _**h**_ . For example, an extreme
case is to just adapt _πθ_ ( _h_ 1 _|_ _**x**_ ) = 0 while leaving _πθ_ ( _**h**_ _>_ 1 _|_ _**x**_ _, h_ 1) = 1. Then the overall probability of
the harmful response _πθ_ ( _**h**_ _|_ _**x**_ ) = _πθ_ ( _h_ 1 _|_ _**x**_ ) _× πθ_ ( _**h**_ _>_ 1 _|_ _**x**_ _, h_ 1) = 0 is indeed diminished. However,
this does not control the harmful behaviors encoded just one token deeper by _πθ_ ( _**h**_ _>_ 1 _|_ _**x**_ _, h_ 1) = 1.
Therefore, a natural counterfactual of the shallow safety alignment is to push its influence deeper into
the model’s harmful outputs, for instance, by keeping _πθ_ ( _**h**_ _>k|_ _**x**_ _,_ _**h**_ _≤k_ ) low for any _k ∈_ [0 _, |_ _**h**_ _|_ ] or at
least a sufficiently large _k_ . We propose a data augmentation approach to encode this notion.
**Safety Recovery Examples.** Consider triplets in the form of ( _**x**_ _,_ _**h**_ _,_ _**r**_ ), in which a refusal response _**r**_
is paired with the harmful instruction and its harmful response. Instead of solely promoting _πθ_ ( _**r**_ _|_ _**x**_ )
as in some standard safety alignment approaches, we propose to also augment _πθ_ ( _**r**_ _|_ _**x**_ _,_ _**h**_ _≤k_ ) _, k ∼_
Uniform[1 _, C_ ] for some constant _C ≤|_ _**h**_ _|_ . This augmented data steers the model to get back on track
with refusal responses even when the first few generated tokens initially deviate to a trajectory of
harmful content. In this way, it explicitly trains the model to suppress harmful content more deeply
within the response, reducing _πθ_ ( _**h**_ _>k|_ _**x**_ _,_ _**h**_ _≤k_ ) at a deeper depth of _k_ . We refer to such augmented
data as safety recovery examples. Such examples typically appear as:
As illustrated, the text is synthetic and not even coherent in natural language, implying that it is
unlikely to be naturally produced by human labelers for SFT data or sampled from models for
preference optimization data. Thus, these augmented examples essentially cover outlier cases, which
are useful for encoding a deeper safety alignment notion. [4]
**Implementations.** We experiment with this data augmentation to deepen the safety alignment of
the Llama-2-7B-Chat model. Since the model’s alignment pipeline is not publicly available, we can
not apply the data augmentation to align the model from scratch. Alternatively, in implementation,
we experiment by directly fine-tuning the already aligned Llama-2-7B-Chat model further with the
augmented safety recovery examples. To implement it, we construct a safety dataset _DH_ comprising
256 examples of triplets ( _**x**_ _,_ _**h**_ _,_ _**r**_ ) in the form we described above. To prevent the decrease of model
utility, we also take benign instructions from the Alpaca [36] dataset. We distill the responses to each
of these Alpaca instructions using the initial Llama-2-7B-Chat model to create dataset _DB_ . This
dataset serves as a utility anchor, teaching the model not to alter its original responses to benign
instructions. Taking together, we fine-tune the model using the following objective:
   min _α ×_ E
_θ_ ( _**x**_ _,_ _**h**_ _,_ _**r**_ ) _∼DH_ _,_
_k∼Pk_
      -       -
_−_ log _πθ_ ( _**r**_ _|_ _**x**_ _,_ _**h**_ _≤k_ ) + (1 _−_ _α_ ) _×_ E _−_ log _πθ_ ( _**y**_ _[′]_ _|_ _**x**_ _[′]_ ) (2)
( _**x**_ _[′]_ _,_ _**y**_ _[′]_ ) _∼DB_
Here, _πθ_ is initialized with the aligned Llama-2-7B-Chat model. We set the number of prefilled
tokens _k_ to follow a distribution _Pk_, where _k_ = 0 with a 50% probability, and _k_ is uniformly sampled
from [1 _,_ 100] with a 50% probability. We set _α_ = 0 _._ 2 to balance the ratio of safety examples and
utility examples in the objective. We denote this fine-tuned model as **Llama2-7B-Chat-Augmented** .
Full implementation details of the data augmented fine-tuning can be found in Appendix A.3.
**Effects of The Data Augmentation.** _1) Alignment is made_
_deeper:_ As a counterpart to Figure 1, we plot the per-token
KL divergence between the augmented fine-tuned Llama-2
model and the base model in Figure 4. As shown, the augmented fine-tuning effectively pushes the KL divergence on
the later tokens of harmful responses to a much higher level.
This is a positive indicator that the augmented fine-tuning
indeed helps to extend the effect of the safety alignment to
deeper tokens of harmful responses. _2) Utility is preserved:_
We also evaluate the utility of the augmented fine-tuned
model with Alpaca Eval [37], which is reported as a winrate
Figure 4: The data augmentation in
against the text-davinci-003 model (the default reference
duces larger KL divergence on Harm
baseline for AlpacaEval). The augmented fine-tuned model
ful HEx-PHI (Section 2.2) over the
achieves a winrate of 49 _._ 5%, which is only marginally lower
later tokens of harmful responses.
than the initial Llama-2-7B-Chat model’s winrate of 51 _._ 8%. [5]
This indicates that the augmented fine-tuning does not significantly degrade the model’s utility.
**3.2** **The Deepened Safety Alignment Shows Improved Robustness Against Multiple Exploits**
A central argument in this work is that the shallow safety alignment can be a source of many safety
vulnerabilities in LLMs. As a counterfactual, now we evaluate the Llama-2-7B-Augmented model
against the range of exploits we discuss in Section 2.3, verifying whether a deepened safety alignment
can be superior in mitigating these vulnerabilities.
**Improved Robustness against Multiple Inference-time Exploits.** We test the prefilling attack (using
the Harmful HEx-PHI dataset we built in Section 2), GCG attack [29], and the decoding parameters
4We note that there are ties to ensuring sufficient exploration in reinfocement learning that we do not explore
formally here, but leave to future work.
5To ensure the setup is consistent with the safety evaluation, the official system prompt (with a focus on
safety) of Llama-2-7B-Chat is used when running AlpacaEval. Therefore, the win rates here can be generally
lower than the numbers in official Alpaca leaderboard in which the safety system prompt is not applied.
Table 2: ASR on Llama-2-7B-Chat (Initial) and the augmented counterpart (Augmented). Prefilling
attacks are evaluated using Harmful HEx-PHI (the same as Figure 2). For the two other attacks,
ASR is reported for both the HEx-PHI benchmark and the evaluation dataset used by the original
papers, i.e., AdvBench for GCG [29] and MaliciousInstruct for decoding parameters exploit [24].
The reported numbers are in the form of (mean _±_ std) over three runs.
|ASR (%) →|Prefilling Attacks|Col3|Col4|Col5|GCG Attack|Col7|Decoding Parameters Exploit|Col9|
|---|---|---|---|---|---|---|---|---|
|**ASR (%)**_ →_|~~5 tokens~~|~~10 tokens~~|~~20 tokens~~|~~40 tokens~~|~~HEx-PHI~~|~~AdvBench~~|~~HEx-PHI~~|~~MaliciousInstruct~~|
|Initial|42.1_ ±_ 0.9|51.5_ ±_ 1.6|56.1_ ±_ 2.5|57.0_ ±_ 0.4|36.5_ ±_ 2.7|65.6_ ±_ 3.1|54.9_ ±_ 0.6|84.3_ ±_ 1.7|
|Augmented|2.8_ ±_ 0.4|2.9_ ±_ 0.2|3.4_ ±_ 0.6|4.5_ ±_ 0.6|18.4_ ±_ 4.2|19.0_ ±_ 2.9|11.3_ ±_ 0.4|1.0_ ±_ 0|
exploit [24] on the Llama-2-7B-Chat-Augmented model. Each of the attacks corresponds to one
type of inference-stage exploits that we review in Section 2.3.1. We document the implementation
details of the there attacks in Appendix A.4. Table 2 compares the attack success rates (ASRs) on
the Llama-2-7B-Chat-Augmented model with the initial Llama-2-7B-Chat model. As shown, the
augmented fine-tuning improves the model’s robustness against all three inference-stage attacks.
**Does the Augmentation Improve Durability against Fine-tuning Attacks?** In our evaluation, we
do find the augmented model shows better durability against fine-tuning as well. Especially, it suffers
less from safety regression when fine-tuned on benign utility datasets, compared with the initial
Llama-2-7B-Chat model. Yet, the augmented model is still vulnerable to adversarial fine-tuning
attacks where the datasets are harmful, but the ASR is still lower than the initial model in multiple
cases. We defer the detailed results to Appendix C.

3.1 Data Augmentation with Safety Recovery Examples


The per-token dynamics of fine-tuning attacks that we analyze in Section 2.3.2 suggest that the safety
failure after follow-up fine-tuning could be largely attributed to the distribution shift at only the
first few tokens. Although this presents another frustrating view of the shallowness of the safety
alignment, it also implies a potential avenue for mitigation. Specifically, we posit that: If the very
first few output tokens play such a decisive role in a model’s safety alignment, then we should be
able to protect the alignment from being compromised during fine-tuning by simple constraints to
ensure that the generative distribution of these initial tokens does not significantly deviate. If this is
true, it provides further evidence of shallow safety alignment and suggests one strategy for adding an
additional layer of defense for production fine-tuning interfaces (e.g., Peng et al. [38]).

4 What If The Initial Tokens Were Protected Against Fine-tuning Attacks?


To further test our hypothesis, we devise the following fine-tuning objective—inspired in part by
approaches like Direct Preference Optimization (DPO) [16] and Kahneman-Tversky Optimization
(KTO) [39]— but adapted to control the deviation from the initial generative distribution for each
token position, similarly to the token-wise RL objective in [40]:
 - _πθ_ - _yt_ _|_ _**x**_ _,_ _**y**_ _<t_ _σ_ _βt_ log
_πθ_  - _yt_ _|_ _**x**_ _,_ _**y**_ _<t_  -  - [��]
_π_ aligned ~~�~~ _yt |_ _**x**_ _,_ _**y**_ _<t_ ~~�~~
_|_ _**y**_ _|_
E
( _**x**_ _,_ _**y**_ ) _∼D_ _[−]_
_t_ =1
log
_βt_
min
_θ_
_,_ (3)
where _σ_ ( _z_ ) := 1+1 _e_ _[−][z]_ [is the sigmoid function and] _[ β][t]_ [ is a constant parameter at each token position to]
control the speed of the saturation of the sigmoid. Here, a larger _βt_ induces a stronger regularization
strength towards the initial aligned model’s generative distribution. See below for the interpretation
of the proposed objective.
**Interpretation of Our Objective.** To see why _βt_ can be used to control the deviation of the generative
distribution at each token position, we can rewrite the fine-tuning objective as:
1 _{t≤|_ _**y**_ _|} ·_ [2]
_S_  - _βt_  - log _π_ aligned� _yt |_ _**x**_ _,_ _**y**_ _<t_  - _−_ log _πθ_  - _yt |_ _**x**_ _,_ _**y**_ _<t_  _βt_
- ~~�~~ - ~~�~~
=:∆ _t_ ( _**x**_ _,_ _**y**_ _<t,yt_ )
- [���]
_,_ (4)
min
_θ_
��
E
( _**x**_ _,_ _**y**_ ) _∼D_
_t≥_ 1
where _S_ ( _z_ ) := log(1 + _e_ _[z]_ ) is the softplus function [41]. At token position _t_, the loss
is essentially ∆ _t_ ( _**x**_ _,_ _**y**_ _<t, yt_ ) (defined above) wrapped by the softplus function _S_ ( _·_ ) after being rescaled by _βt_ . When _βt_ is small, _S_ ( _βtz_ ) _≈_ _S_ (0) + _βtS_ _[′]_ (0) _z_ = log 2 + _[β]_ 2 _[t]_ _[z]_ [, so]
_β_ 2 _t_ _[S]_ [(] _[β][t][z]_ [)][ is approximately equal to] _[ −]_ [log] _[ π][θ]_ - _yt |_ _**x**_ _,_ _**y**_ _<t_ - after shifting by a constant. This
means minimizing our objective is approximately the same as minimizing the cross-entropy
loss E( _**x**_ _,_ _**y**_ ) _∼D_ - _−_ 1 _{t≤|_ _**y**_ _|} ·_ log _πθ_ - _yt |_ _**x**_ _,_ _**y**_ _<t_ ��. Conversely, when _βt_ is large, _β_ 2 _t_ _[S]_ [(] _[β][t][z]_ [) =]
2 max _{z,_ 0 _}_ +exp( _−_ Ω( _βt|z|_ )), which converges exponentially to 2 max _{z,_ 0 _}_ . The loss can then be
approximated by E( _**x**_ _,_ _**y**_ ) _∼D_ - 1 _{t≤|_ _**y**_ _|} ·_ max _{_ ∆ _t_ ( _**x**_ _,_ _**y**_ _<t, yt_ ) _,_ 0 _}_ �. **This shows that small** _βt_ **places**
**emphasis on minimizing the cross-entropy loss, while large** _βt_ **places emphasis on matching**
**the generative distribution to the initial aligned model.** In Appendix D.1, we provide a detailed
derivation of these limiting behaviors for large and small _βt_ and further characterization for the
gradient when _βt_ takes a moderate value.
**Gradient of Our Objective.** Our objective can also be interpreted by its gradient. The token-wise
gradient of the objective on each data point ( _**x**_ _,_ _**y**_ ) with _|_ _**y**_ _| ≥_ _t_ can be derived as:
_∇_   - 2 _S_ ( _βt_ ∆ _t_ ( _**x**_ _,_ _**y**_ _<t, yt_ ))� = _−_ 2 _σ_ ( _βt_ ∆ _t_ ( _**x**_ _,_ _**y**_ _<t, yt_ )) _∇_ log _πθ_   - _yt |_ _**x**_ _,_ _**y**_ _<t_ ) _._ (5)
_βt_
Note that the gradient of the cross-entropy loss is _−∇_ log _πθ_ - _yt |_ _**x**_ _,_ _**y**_ _<t_ �, our fine-tuning objective
essentially applies an additional adaptive weight _wt_ := 2 _σ_ ( _βt_ ∆ _t_ ( _**x**_ _,_ _**y**_ _<t, yt_ )) for the token-wise
gradient term of cross-entropy. The weight _wt_ diminishes as _−_ ∆ _t_ ( _**x**_ _,_ _**y**_ _<t, yt_ ) := log _πθ_ - _yt |_
_**x**_ _,_ _**y**_ _<t_ - _−_ log _π_ aligned� _yt |_ _**x**_ _,_ _**y**_ _<t_ - becomes large. This difference in log probabilities essentially
characterizes the deviation between the fine-tuned model _πθ_ and the initially aligned model _πaligned_
at the token position _t_ (see also Theorem 3). Taking together, we can see that the fine-tuning objective
will adaptively diminish the weight of those token positions where the deviation of the distribution
approaches a certain threshold (controlled by _βt_ ), thereby constraining it from further deviation (by
diminishing the gradient at this position). Note that, at the beginning of the fine-tuning when _πθ_ is
initialized as _π_ aligned, the weight _wt_ = 1, and the gradient of the loss is equivalent to that of standard
cross-entropy loss. [6]
**Interpretation from A Reinforcement Learning Perspective.** In Appendix D.3, we further show
how the loss function in Eqn 3 can also be derived from a KL-regularized reinforcement learning
objective with _βt_ controlling the strength of the KL regularizes at each different positions _t_ . Under
this RL-based interpretation, a larger _βt_ essentially denotes a larger weight for the token-wise KL
regularization terms, representing a stronger constraint enforcing that the token-wise generative
distribution at the position _t_ does not deviate much from that of the initial model _π_ aligned.

4.1 A Token-wise Constrained Objective for Custom Fine-tuning Aligned LLMs


**Configurations of** _βt_ **.** To test our argument, we set a large _β_ for the first few tokens to impose a
stronger constraint such that their generative distributions won’t deviate too much from the aligned
models. This leads to the implementation of larger _βt_ as _β_ 1 = 0 _._ 5, _βt_ = 2 for 2 _≤_ _t ≤_ 5 at the initial
5 tokens, while a much weaker constraint _βt_ = 0 _._ 1 for _t >_ 5 at the later tokens.
**Fine-tuning Attacks.** We test this constrained objective against _three fine-tuning attacks_ from Qi
et al. [22] — _1) Harmful Examples:_ fine-tuning with 100 (harmful input, harmful answer) pairs; _2)_
_Identity Shifting:_ fine-tuning the model to self-identify as an absolutely obedient agent, and always
answer questions with affirmative prefix; _3) Backdoor Poisoning:_ fine-tuning the model on a mixture
of 100 (harmful input, refusal answer) pairs plus 100 (harmful input + a backdoor trigger, harmful
answer) pairs. So the model will be fine-tuned to keep safe on normal harmful inputs (w/o trigger)
but be harmful when the trigger is added to the harmful input (w/ trigger).
**Benign Fine-tuning.** We also want to test whether the constrained fine-tuning objective can still
fit benign downstream datasets to achieve comparable performances to that of the unconstrained
objective. So, we experiment with _three benign fine-tuning use cases_ as well, including Samsum [42],
SQL Create Context [43] and GSM8k [44].
**Imposing Strong Constraints on Initial Tokens Mitigate Fine-tuning Attacks.** Table 3 summarizes
our results of fine-tuning Llama-2-7B-Chat and Gemma-1.1-7B-IT with the proposed constrained
fine-tuning objective. As illustrated, the constrained fine-tuning objective (Constrained SFT in the
6This is also why we multiply a constant factor _β_ 2 _t_ [to the loss. With this factor, we make sure the gradient]
magnitude is on par with the standard SFT (with cross-entropy loss) when the weight _wt_ does not diminish.
Table 3: Fine-tuning with The Constrained Objective in Eqn 3, with larger constraints _β_ 1 = 0 _._ 5,
|Models →|Col2|Llama-2-7B-Chat|Col4|Col5|Col6|Gemma-1.1-7B-IT|Col8|Col9|
|---|---|---|---|---|---|---|---|---|
|Datasets_ ↓_|~~**mean**~~~~_ ±_~~~~** std** (%)~~
**(over 3 rounds)**|Initial|Standard
SFT|**Constrained**
**SFT (ours)**||Initial|Standard
SFT|**Constrained**
**SFT (ours)**|
_Against Fine-tuning Attacks_
|Harmful Examples|ASR|1.5 ± 0.2|88.9 ± 1.2|4.6 ± 0.5|Col6|1.8 ± 0.3|81.6 ± 2.9|1.9 ± 0.2|
|---|---|---|---|---|---|---|---|---|
|Identity Shifting|ASR|0_ ±_ 0|79.5_ ±_ 2.3|8.1_ ±_ 0.1||0_ ±_ 0|83.6_ ±_ 2.5|9.1_ ±_ 1.7|
|Backdoor
Poisoning|ASR (w/o trigger)
ASR (w/ trigger)|1.5_ ±_ 0.2
1.7_ ±_ 0.1|7.6_ ±_ 1.1
90.9_ ±_ 1.4|1.9_ ±_ 0.2
10.9_ ±_ 2.8||1.8_ ±_ 0.3
1.8_ ±_ 0.3|2.0_ ±_ 0.2
82.3_ ±_ 1.1|1.5_ ±_ 0.1
1.9_ ±_ 0.8|
_Fine-tuning with Normal Downstream Datasets_
|Samsum|ASR
Utility|1.5 ± 0.2
25.5 ± 0.3|23.4 ± 2.5
51.7 ± 0.5|3.2 ± 0.8
50.1 ± 0.2|Col6|1.8 ± 0.3
36.0 ± 1.4|2.0 ± 0.2
51.5 ± 0.3|2.4 ± 0.3
51.9 ± 0.5|
|---|---|---|---|---|---|---|---|---|
|SQL Create Context|ASR
Utility|1.5_ ±_ 0.2
14.9_ ±_ 0.4|15.4_ ±_ 1.4
99.1_ ±_ 0.2|3.2_ ±_ 0.8
98.5_ ±_ 0.1||1.8_ ±_ 0.3
88.0_ ±_ 0.5|2.8_ ±_ 0.2
99.2_ ±_ 0.1|2.4_ ±_ 0.1
98.6_ ±_ 0.3|
|GSM8k|ASR
Utility|1.5_ ±_ 0.2
25.5_ ±_ 0.2|3.3_ ±_ 0.4
41.7_ ±_ 0.4|2.0_ ±_ 0.5
37.4_ ±_ 0.3||1.8_ ±_ 0.3
28.5_ ±_ 1.2|2.9_ ±_ 0.2
63.3_ ±_ 0.5|1.7_ ±_ 0.4
63.6_ ±_ 0.4|
table) generally keeps a low ASR after both adversarial fine-tuning attacks and benign fine-tuning
with normal downstream datasets. This suggests that the safety alignment can indeed be more
persistent against fine-tuning if we can properly apply a tight constraint to prevent the distribution of
early tokens from deviating too much from the initial models.
**Comparable Utility Using The Constrained Loss.** In Table 3, we also report utility metrics for
benign fine-tuning use cases, employing the standard ROUGE-1 score for Samsum and SQL Create
Context, and answer accuracy for GSM8k, consistent with established practices for these datasets.
As shown, both standard SFT and constrained SFT improve utility compared to the initial model
across all three cases. Notably, constrained SFT achieves comparable utility to standard SFT while
mitigating the risk of harmful fine-tuning. These results suggest that constraining initial tokens
offers significant advantages in maintaining model safety, without significantly compromising the
model’s ability to still leverage fine-tuning for enhanced utility in many downstream tasks. This is a
meaningful insight that may be leveraged to build an additional layer of protection for production
fine-tuning interfaces such as OpenAI’s Finetuning API [38]. Since fine-tuning interface providers
want to allow their users to customize their models for downstream usage while not breaking the
safety alignment, they should consider enforcing such more restrictive fine-tuning objectives that are
strategically designed to protect safety alignment while allowing customizability.
**Experiment Details and More Ablation Studies.** The full implementation details of this experiment
can be found in Appendix A.5. Besides, to further validate that the improvement in Table 3 is indeed
benefiting from the stronger constraints (larger _βt_ ) biased to the first 5 tokens, we also provide further
ablation studies on the choice of _βt_ in Appendix C.


**Safety & Alignment.** A large body of work has examined improved methods for alignment [16,
39, 19, 45, 14, 5, 31]. While we tie alignment approaches to potential downstream jailbreaks, we
do so mainly be examining aligned artifacts which go through more rigorous alignment procedures
than most other open source models. We rely on the Gemma [31] and Llama-2 [5] base and aligned
models throughout this work, as the safety alignment built in these two models are closest to the
technology applied in frontier proprietary models.
**Jailbreaking Methods.** A large body of work has examined methods for jailbreaking aligned LLMs,
including leveraging finetuning, decoding strategies, prefilling strategies, optimization strategies, and
even persuasion [21, 29, 22, 24, 46, 23, 47, 48, 30, 17]. Many approaches try to handle jailbreaking
through a systems approach by monitoring inputs and outputs with machine learning models [49],
but this is only as good as the monitoring mechanism (which can also be jailbroken) [50].
**Superficial Alignment Hypothesis and Per-token Effects of Alignment Fine-tuning.** Our work is
closely related to the Superficial Alignment Hypothesis (SAH) [28], which posits that the alignment
process for current LLMs only superficially changes the formats of inputs and outputs to be used
for interacting with the users. Besides, there are also some earlier works noting asymmetries in
the representation power and utility of different tokens during adaptation. For example, Zhang
and Wu [26] show that “the adaptation of topic and style priors” during finetuning are “learned
independently and primarily at the beginning of a text sequence.” Lin et al. [25] also find that
differences between aligned and unaligned base models introduced by the alignment fine-tuning
vanishes as the sequence goes longer (similar to the effect that we observe in Figure 1, though theirs
are not in safety-specific contexts). Particularly, while Lin et al. [25] primarily tie such effects to
question whether fine-tuning is even necessary to achieve current level of alignment (e.g., in-context
learning may already suffice to achieve a comparable level of alignment), we go much deeper into
investigating the safety-specfic effects of this phenomenon and tie it to multiple downstream attacks
and training-based mitigations. Besides, we find Zhao et al. [27] also note a similar token-wise effect,
and they explicitly exploit this effect to design jailbreak attacks. Others have investigated fine-tuning
dynamics through interpretability or pruning methods, which is distinct but somewhat related to the
approach we take here [51, 52].
**Protecting The Safety Alignment at Initial Token Positions.** In Section 4, one important insight
that motivates the design of our constrained fine-tuning loss is that “safety alignment would be more
difficult to be circumvented if we can protect the generative distribution of the model at the early
token positions.” We note that Xu et al. [53] share a similar insight to ours in this regard. They find it
is possible to design a defense against inference-time jailbreak attacks by simply identifying safety
disclaimers and amplifying their token probabilities at the early token positions.
**Connections to Control Theory and Safe Reinforcement Learning.** Our data augmentation
approach in Section 3 relates to exploration requirements for optimal learning via policy gradient
methods [54], learning recovery policies [55], and safe control theory [56, 57]. We, however, leave
deeper connections to this literature for future work.
**Other Notions of Safety Depth.** We also note that safety “depth” may be multi-dimensional in
addition to token-based depth we describe here. Other considerations for depth would be the ability for
models to retain safety properties after adaptation that some have previously discussed [58, 51, 22].


Our work identifies a shortcut that current safety alignment strategies appear to exploit: that alignment
only needs to change the generative distribution of the first few tokens. We show that this may be a key
component of many downstream vulnerabilities. We then provide two initial strategies to address this:
(1) a data augmentation approach that can increase depth of alignment; (2) a constrained optimization
objective that can help mitigate finetuning attacks by constraining updates on initial tokens. Future
work can explore additional approaches grounded in control theory and safe reinforcement learning.
The methods we describe here may not be a perfect defense and may be subject to some future
adaptive attacks, but they are an initial step for improving robustness and further demonstrate how
much improvement there can be over current approaches. Fundamentally, our results are primarily to
support our argument that future safety alignment should be made more than just a few tokens deep.


Our work explicitly ties failure modes of safety alignment to potential shortcuts that can be taken by
alignment methods and advocates for, and provides a path forward for, deeper alignment approaches
that will improve safety more broadly. While a deeper understanding of the failures of alignment
may result in increased ability to jailbreak models, we believe that open investigations of such failure
modes are important for strengthening the safety of future models and broadly ensuring that models
have positive societal impact. The proposed prototype approaches (in this work) for strengthening
the alignment in current LLMs also contribute to the broader agenda of building safer and secure AI.


We thank Kaixuan Huang, Zixuan Wang, Dingli Yu, Haoyu Zhao at Princeton University for their
early discussions and feedback to this project. This work is supported by Princeton Language and
Intelligence (PLI) Compute Cluster and Center for AI Safety (CAIS) Compute Cluster. Xiangyu Qi
and Ashwinee Panda are supported by a Superalignment Fast Grant from OpenAI, and Xiangyu Qi
is also supported by the Princeton Gordon Y. S. Wu Fellowship. Prateek Mittal acknowledges the
Princeton SEAS Innovation Grant. Peter Henderson acknowledges the Foundational Research Grants
program at Georgetown University’s Center for Security and Emerging Technology. Any opinions,
findings, conclusions, or recommendations expressed in this material are those of the author(s) and
do not necessarily reflect the views of the funding agencies.


In this work, we use single 4 _×_ A100-80GB GPU nodes or 4 _×_ H100-80GB GPU nodes for all
experiments, depending on availability of the nodes. On each node, our experiments use up to 8 CPU
cores and 256GB memory, but overall the experiments are not CPU intensive tasks.


**Decoding Parameters.** Throughout this paper, we use the top-p sampling with a temperate of 0.9 and
a top-p parameter of 0.6 by default for decoding outputs from LLMs in our experiments. The only
case where we do not follow this default configuration is the decoding parameters exploit experiment
where the exploit itself needs to take use of different parameters by its design [24].
**Safety Evaluation.** As also mentioned in Section 2.1, we use the GPT-4 based judge to evaluate
the safety of model outputs, following the setup of Qi et al. [22]. Specifically, in such an evaluation
pipeline, we pass (input, output) pairs to the GPT-4-Turbo model, and prompt the model to evaluate
the harmfulness level of the output. The model will output a score ranging from 1 to 5, with higher
score indicating being more harmful. When reporting ASR in our experiments, we report the ratio
of outputs that get the highest score 5 (identical to the harmfulness rate metric in Qi et al. [22]).
By default, HEx-PHI safety benchmark [32] is used for safety evaluation. The only exception
is the experiments in Table 2, where we add additional evaluation on AdvBench for GCG attack
evaluation [29] and MaliciousInstruct for decoding parameters exploit [24]. These two additional
safety evaluation datasets are used in the original papers of the two work, and we report results on
both HEx-PHI and these additional safety evaluation datasets for a more complete reference.


Here, we describe the implementation details of the data augmentation experiments in Section 3.1.
**Safety Data.** As noted in Eqn 2, we use a safety dataset _DH_ to keep generating safety recovery
examples. To construct it, we first collect 256 harmful instructions. These instructions are mostly
collected from the red-teaming data provided by Ganguli et al. [61]. We make sure they do not
overlap with any of the safety evaluation datasets that we used in this paper, i.e., HEx-PHI [32],
AdvBench [29], and MaliciousInstruct [24]. Then, we generate refusal answers for each harmful
instruction using the initial Llama-2-7B-Chat model. We also collect the corresponding harmful
responses for these instructions using a jailbroken version of the model (jailbroken through fine-tuning
attacks per Qi et al. [22]). This results in the dataset _DH_ with 256 examples of triplets ( _**x**_ _,_ _**h**_ _,_ _**r**_ ).
**Utility Data.** To prevent the decrease of model utility during the data augmentation fine-tuning,
we also take benign instructions from the Alpaca [36] dataset. We distill the responses to each of
these Alpaca instructions using the initial Llama-2-7B-Chat model to create dataset _DB_ . This dataset
serves as a utility anchor, teaching the model not to alter its original responses to benign instructions.
**Training details with Eqn 2.** In the implementation of the augmentation fine-tuning as per Eqn 2, we
set the number of prefilled tokens _k_ to follow a distribution _Pk_, where _k_ = 0 with a 50% probability,
and _k_ is uniformly sampled from [1 _,_ 100] with a 50% probability. We set _α_ = 0 _._ 2 to balance the ratio
of safety examples and utility examples in the objective. In batch-wise training, this is implemented
by randomly sampling 16 examples from _DH_ and 64 examples from _DB_ in each batch. Using this
objective, we train the model for 10 epochs on _DH_ with a learning rate of 2 _×_ 10 _[−]_ [5] using the AdamW
optimizer (with the default configurations of the optimizer).
**AlpacaEval.** We also evaluate the utility of the augmented fine-tuned model with AlpacaEval [37],
which is reported as a winrate against the text-davinci-003 model (the default reference baseline for
AlpacaEval). Specifically, we use the 1.0 version of AlpacaEval without length control. To ensure the
setup is consistent with the safety evaluation, the official system prompt (with a focus on safety) of
Llama-2-7B-Chat is used when running AlpacaEval. We note that the win rates here can therefore be
generally lower than the numbers in official Alpaca leaderboard in which the safety system prompt is
not applied. Under this evaluation, we note that the augmented fine-tuned model achieves a winrate of
49 _._ 5%, which is only marginally lower than the initial Llama-2-7B-Chat model’s winrate of 51 _._ 8%.
**Limitations.** Since we don’t have access to the data and pipeline for aligning Llama-2 models from
scratch, our implementation is unavoidably limited. As also specified in Section 3.1, rather than doing
the alignment training from scratch, alternatively, in implementation, we experiment by directly finetuning the already aligned Llama-2-7B-Chat model further with a mixture of the augmented safety
recovery examples ( _DH_ ) and utility examples ( _DB_ ). This implementation is inherently sub-optimal.
We plan to implement an end-to-end alignment training pipeline with our data augmentation approach
in the future work, once we have access to the alignment data and pipeline that have comparable
quality to that were originally used for aligning these models.

A.3 Details of Data Augmentation Experiments


We have tested three inference-stage attacks in Section 3.2, i.e., prefilling attack, GCG attack [29],
and decoding parameters exploit [24]. We specify the details here.
**Prefilling Attack.** Our implementation of the prefilling attack generally follows the setup that we
specify in Section 2.3.1. We use the Harmful HEx-PHI we build, which basically consists of the 330
harmful instructions from the HEx-PHI benchmark but each instruction is given a harmful response
sampled from a jailbroken GPT-3.5-Turbo model. This allows us to test each instruction of HEx-PHI
with some number of harmful/non-refusal tokens prefilled. Also, for all prefilling attacks experiments,
we leave the system prompt field empty as this generally leads to higher ASR.
**GCG Attack.** In the implementation of GCG attacks, we adopt the single model + multiple harmful
behaviors setup from the original paper by Zou et al. [29]. Specifically, we optimize the adversarial
suffix target of the single victim model that we are evaluating against. We train the adversarial suffix
over 50 harmful behaviors data points. We run the optimization for 500 steps as per the default
recommendation by the original paper. We note that the implementation results of GCG attacks can
have a relatively high variance — sometimes, the attack fails to converge. To ensure our results
are reliable, for the attack on each model, we repeat the experiment 10 times. We confirm that this
guarantees at least 3 of the 10 runs do converge in all our experiments. In Table 2, when we report
the ASR, we report the mean _±_ a standard deviation over the 3 most successful (with highest ASR)
runs out of the 10 runs. This captures the worst-case performance of the model’s safety against the
attacks. Also, as shown in Table 2, we report the Attack Success Rate on both HEx-PHI (the default
safety evaluation dataset) and AdvBench (the safety evaluation dataset used in the original paper).
Following the setup of Zou et al. [29], the official system prompt is used when doing the attack.
**Decoding Parameters Exploit.** We follow the same setup as per Huang et al. [24]. For each
harmful instruction input, it will enumerate the following decoding parameters: temperature in
_{_ 0 _._ 05 _,_ 0 _._ 1 _,_ 0 _._ 15 _,_ 0 _._ 2 _,_ 0 _._ 25 _, ...,_ 0 _._ 95 _,_ 1 _._ 0 _}_, top-k in _{_ 1 _,_ 2 _,_ 5 _,_ 10 _,_ 20 _,_ 50 _,_ 100 _,_ 200 _,_ 500 _}_, and top-p in
_{_ 0 _._ 05 _,_ 0 _._ 1 _,_ 0 _._ 15 _,_ 0 _._ 2 _,_ 0 _._ 25 _, ...,_ 0 _._ 95 _,_ 1 _._ 0 _}_ . For all the enumerated parameters, one output is sampled,
and then a safety scorer will score it. The most unsafe output will be picked. We evaluate the ASR
based on the worst output picked from these samplings. Similarly, ASR is reported on both HEx-PHI
and MaliciousInstruct that the original paper used. Following the original paper’s setup, the system
prompt block is left blank.

A.4 Details of Inference-Stage Attacks Experiments


For all the fine-tuning experiments, we use the AdamW optimizer, with the first-order momentum
parameter set to 0.5 and the second-order momentum parameter set to 0.999. For Llama-2-7B-Chat,
a learning rate of 2 _×_ 10 _[−]_ [5] is used. For Gemma-1.1-7B-IT, we use a learning rate of 5 _×_ 10 _[−]_ [6] . A
batch size of 64 is used for all experiments.
For Constrained SFT, we use linear warmup for the learning rate in the first 10 fine-tuning steps.
This warmup makes sure the constraints imposed by the sigmoid function are gently initialized —
note that, at the start of the fine-tuning, the log ratio log _πθ/π_ aligned in Eqn 3 is equal to 0 since _πθ_
is basically initialized as _π_ aligned. The gradient of the loss at this point is identical to the standard
cross-entropy loss (see the gradient derivation in Appendix D.2). Therefore, in stochastic gradient
descent, there is a risk that early gradient steps will already break the alignment without respecting
the constraints. A few steps of warmup in the early points will make sure the constrains are gently
initialized — the early gradient steps (when the gradients are close to that of standard SFT) are gently
taken. See Table 5 in Appendix C for ablation of the effect of the warmup.


We evaluate _three fine-tuning attacks_ from Qi et al. [22].
**Harmful Examples.** It fine-tunes the model with 100 (harmful input, harmful answer) pairs. We use
exactly the same 100 pairs from Qi et al. [22]. We fine-tune models on this dataset for 25 epochs.
**Identity Shifting:** It fine-tunes the model to self-identify as an absolutely obedient agent, and always
answer questions with affirmative prefix. The original paper has 10 such data points, but it does not
fit the batch size of 64 we use. So we extend it to 100 data points manually, in the same format. We
fine-tune models on this dataset for 25 epochs.
**Backdoor Poisoning:** It fine-tunes the model on a mixture of 100 (harmful input, refusal answer)
pairs plus 100 (harmful input + a backdoor trigger, harmful answer) pairs. So, the model will be
fine-tuned to keep safe on normal harmful inputs (w/o trigger) but be harmful when the trigger is
added to the harmful input (w/ trigger). We use the same data from Qi et al. [22]. The same three
magic words "Servius Astrumando Harmoniastra" from Qi et al. [22] are used as the backdoor trigger.


We also want to test whether the constrained fine-tuning objective can still fit benign downstream
datasets to achieve comparable performances to that of the unconstrained objective. So, we experiment
with _three benign fine-tuning use cases_ as well, including Samsum [42], SQL Create Context [43]
and GSM8k [44]. For each of the three datasets, we fine-tune models on them for 3 epochs.
Specifically, Samsum is a dataset for summarization tasks. We report the ROUGE-1 score as the
utility. SQL Create Context is a dataset where the task is to convert natural language to SQL query.
The ROUGE-1 score is also used for its utility evaluation. GSM8k is a dataset for math tasks. We
report the utility as the accuracy of the model’s answers.


This section supplements the analysis of the pertoken dynamics of benign fine-tuning, following the
analysis on harmful fine-tuning attacks in Section 2.3.2.
Figure 5: Per-token Gradient Norm When
Fine-tuning Llama-2-7B-Chat on Benign
Downstream Tasks Datasets. _Note: 1)_
_ASR of initially aligned model =_ 1 _._ 5% _;_
_2) After 10 gradient steps on SQL Create_
_Context_ = 13 _._ 6%; 3) After 10 gradient
steps on Samsum = 22 _._ 1%.
Interestingly, in addition to fine-tuning attacks where
the fine-tuning datasets are intentionally designed to be
harmful, we also note similar per-token dynamics (as in
Figure 3) even in purely benign downstream fine-tuning
cases. Figure 5 plots the gradient norm when fine-tuning
Llama-2-7B-Chat on SQL Create Context [43] and Samsum [42]. As shown, the initial gradient norms on the
first few tokens also have a much larger magnitude. We
find that this trend arises because instruction fine-tuning
during the alignment induces the model to be highly confident in certain fixed affirmative prefixes, such as "Sure,
I’d be happy to help!" on normal inputs. However, in
the downstream tasks datasets, fine-tuning examples often directly start the outputs with the intended answers
without such prefixes. Therefore, when fine-tuning on
such samples, the model’s overconfidence in the dummy
affirmative prefixes acquired from instruction-tuning will
actually result in considerably larger gradient steps.
We hypothesize that **this might be one underlying rea-**
**son why Qi et al. [22] discover that even benign fine-tuning can cause safety regression in the**
**aligned LLMs** - it may merely result from the excessively larger gradient steps when updating the
generative distributions of these initial transition tokens, which, in turn, lead to over-generalization (or
catastrophic forgetting), and therefore unintendedly also disrupt the model’s generative distribution
for refusal prefixes in these token positions. This is plausible, as we note that a full fine-tuning on
SQL Create Context and Samsum with more than 600 gradient steps results in an increase of ASR
from 1 _._ 5% to 14.9% and 25.5% respectively, but the ASR is already at 13.6% and 22.1% after the
initial 10 gradient steps. This suggests that the most significant safety regression exactly occurs
during these early steps when the gradient norm for the initial tokens is excessively large.

B Pertoken Dynamics of Benign Fine-tuning


This section presents more in-depth ablation studies to supplement our studies in Section 4 and
Table 3 there. We also supplement Section 3.2 by presenting results (Table 6) of fine-tuning attacks
on the augmented model that we build in Section 3.
**Biased Constrains on The Early Tokens Are Important.** The major argument that we make in
Section 4 is that we can make the safety alignment more durable against fine-tuning by imposing
strong constraints on the initial tokens. Therefore, we set a larger _βt_ to impose stronger constraints in
early tokens while only setting a very weak _βt_ for later tokens. Our results in table 3 indeed verify the
improved safety. To further support that this improvement is indeed due to the biased constraints on
the early tokens, we perform an ablation where all _βt_ are set to a uniform value. Results are presented
in Table 4. As shown, if we set the same small _β_ = 0 _._ 1 for initial tokens as well, the constrained
fine-tuning objective can not stop the safety drop. While if we set the large _β_ = 2 _._ 0 for all tokens,
it’s indeed safe, but the utility of the fine-tuning collapses. Similarly, _β_ = 0 _._ 5 for all tokens neither
achieve optimal safety, and the utility is worse than the biased configurations we use in Table 3.
Table 4: Ablation on _βt_ in Eqn 3. (Fine-tuning Llama-2-7B-Chat)
Standard Constrained SFT Constrained SFT Constrained SFT Constrained SFT
Datasets Initial SFT (biased _βt_ ) (uniform _β_ = 0 _._ 1) (uniform _β_ = 0 _._ 5) (uniform _β_ = 2 _._ 0)
_Against Fine-tuning Attacks_
Constrained SFT
(uniform _β_ = 0 _._ 1)
Constrained SFT
(uniform _β_ = 0 _._ 5)
Standard
Datasets Initial SFT
Constrained SFT
(biased _βt_ )
|Harmful Examples|ASR|1.5%|88.9%|4.6%|86.2%|7.2%|0.5%|
|---|---|---|---|---|---|---|---|
|Identity Shifting|ASR|0%|79.5%|8.1%|41.6%|17.1%|3.4%|
|Backdoor
Poisoning|ASR (w/o trigger)
ASR (w/ trigger)|1.5%
1.7%|7.6%
90.9%|1.9%
10.9%|3.5%
74.4%|1.8%
24.3%|1.2%
1.4%|
_Fine-tuning with Normal Downstream Datasets_
|Samsum|ASR
Utility|1.5%
25.5%|23.4%
51.7%|3.2%
50.1%|3.9%
51.7%|3.5%
49.8%|2.4%
42.5%|
|---|---|---|---|---|---|---|---|
|SQL Create Context|ASR
Utility|1.5%
14.9%|15.4%
99.1%|3.2%
98.5%|3.3%
99.1%|2.2%
98.6%|2.6%
92.6%|
|GSM8k|ASR
Utility|1.5%
25.5%|3.3%
41.7%|2.0%
37.4%|4.0%
39.4%|1.5%
34.8%|2.0%
2.1%|
**Ablation on The Effects of Warmup Steps.** As we have also noted in Appendix A.5.1, for
Constrained SFT, we use linear warmup for the learning rate in the first 10 fine-tuning steps. This
warmup makes sure the constraints imposed by the sigmoid function are gently initialized — note
that, at the start of the fine-tuning, the log ratio log _πθ/π_ aligned in Eqn 3 is equal to 0. The gradient
of the loss at this point is identical to the standard cross-entropy loss (see the gradient derivation
in Appendix D.2). Therefore, in stochastic gradient descent, there is a risk that early gradients will
already break the alignment without respecting the constraints. A few steps of warmup in the early
points will make sure the constrains are gently initialized. We present an ablation study on the effect
of these 10 steps of warmup in Table 5. We can summarize two key takeaways: 1) the warmup
steps are indeed useful to make the constrained SFT to be perform consistently safer; 2) the safety
improvement is not solely coming from the warmup steps but mostly from the constrained SFT
optimization objective we design.
**Fine-tuning Attacks on The Augmented Model We Build in Section 3.** Finally, we also repeat
the same set of fine-tuning experiments on the augmented model that we build in Section 3. Results
are presented in Table 6. By comparing the results of SFT in Table 6 and Table 5, we can see
the augmented model is generally more robust in multiple fine-tuning cases compared with the
non-augmented model. And the constrained fine-tuning objective can also be applied to it, though we
didn’t observe consistently better results when the two techniques are combined.
Table 5: Ablation on The Effects of The 10 Warmup Steps. (Fine-tuning Llama-2-7B-Chat)
Standard Standard SFT Constrained Constrained SFT
Datasets Initial
SFT (with warmup) SFT (with warmup)
_Against Fine-tuning Attacks_
Standard
Datasets Initial
SFT
Standard SFT
(with warmup)
Constrained
SFT
|Harmful Examples|ASR|1.5%|88.9%|89.4%|29.1%|4.6%|
|---|---|---|---|---|---|---|
|Identity Shifting|ASR|0%|79.5%|44.8%|69.6%|8.1%|
|Backdoor
Poisoning|ASR (w/o trigger)
ASR (w/ trigger)|1.5%
1.7%|7.6%
90.9%|2.7%
80.5%|2.7%
9.7%|1.9%
10.9%|
_Fine-tuning with Normal Downstream Datasets_
|Samsum|ASR
Utility|1.5%
25.5%|23.4%
51.7%|3.8%
51.9%|23.1%
50.2%|3.2%
50.1%|
|---|---|---|---|---|---|---|
|SQL Create Context|ASR
Utility|1.5%
14.9%|15.4%
99.1%|3.3%
99.1%|2.0%
98.6%|3.2%
98.5%|
|GSM8k|ASR
Utility|1.5%
25.5%|3.3%
41.7%|2.9%
41.6%|3.1%
37.2%|2.0%
37.4%|
Table 6: Fine-tuning Llama-2-7B-Chat-Augmented That We Built in Section 3. Refer to Table 5 for
the results on the non-augmented counterpart, i.e., Llama-2-7B-Chat.
Standard Standard SFT Constrained Constrained SFT
Datasets
SFT (with warmup) SFT (with warmup)
_Against Fine-tuning Attacks_
Standard
Datasets
SFT
Standard SFT
(with warmup)
Constrained
SFT
|Harmful Examples|ASR|55.2%|51.5%|9.4%|5.2%|
|---|---|---|---|---|---|
|Identity Shifting|ASR|53.9%|37.0%|28.8%|3.0%|
|Backdoor
Poisoning|ASR (w/o trigger)
ASR (w/ trigger)|3.9%
80.0%|2.7%
83.6%|1.8%
16.4%|0.9%
12.7%|
_Fine-tuning with Normal Downstream Datasets_
|Samsum|ASR
Utility|2.1%
52.4%|0.6%
51.9%|2.1%
50.4%|1.2%
50.1%|
|---|---|---|---|---|---|
|SQL Create Context|ASR
Utility|3.8%
99.0%|1.5%
99.1%|2.0%
98.4%|0.9%
98.5%|
|GSM8k|ASR
Utility|0.9%
42.0%|0.9%
41.2%|0.3%
36.5%|0.3%
36.9%|

C Ablation Studies on Fine-tuning Attack Experiments


In this section, we provide detailed interpretation for our constrained fine-tuning objective in Section 4.
Recall that our fine-tuning objective is defined as:
 - _πθ_ - _yt_ _|_ _**x**_ _,_ _**y**_ _<t_ - - [�]
_σ_ _βt_ log _π_ aligned ~~�~~ _yt |_ _**x**_ _,_ _**y**_ _<t_ ~~�~~
_L_ ( _θ_ ) := E
( _**x**_ _,_ _**y**_ ) _∼D_ _[−]_
_|_ _**y**_ _|_
_t_ =1
log
_βt_
_._ (6)
Alternatively, we can rewrite the fine-tuning objective by linearity of expectation as:
 - _πθ_ - _yt_ _|_ _**x**_ _,_ _**y**_ _<t_ - - [�]
_σ_ _βt_ log _π_ aligned ~~�~~ _yt |_ _**x**_ _,_ _**y**_ _<t_ ~~�~~
 - _πθ_ - _yt_ _|_ _**x**_ _,_ _**y**_ _<t_ - - [�]
_σ_ _βt_ log _π_ aligned ~~�~~ _yt |_ _**x**_ _,_ _**y**_ _<t_ ~~�~~
(7)
(8)
_L_ ( _θ_ ) = E
( _**x**_ _,_ _**y**_ ) _∼D_
- 2
_−_ 1 _{t≤|_ _**y**_ _|}_ log
_βt_
_t≥_ 1
 - 2
= E log
( _**x**_ _,_ _**y**_ ) _∼D_ _[−]_ [1] _[{][t][≤|]_ _**[y]**_ _[|}]_ _βt_
_t≥_ 1
- 2 - _πθ_ - _yt_ _|_ _**x**_ _,_ _**y**_ _<t_
_t≥_ 1 ( _**x**_ _,_ _**y**_ E) _∼D_ [1] _[{][t][≤|]_ _**[y]**_ _[|}]_ _βt_ _S_ _−_ _βt_ log _π_ aligned ~~�~~ _yt |_ _**x**_ _,_ _**y**_
=
_πθ_ _yt_ _|_ _**x**_ _,_ _**y**_ _<t_
_,_ (9)
_π_ aligned ~~�~~ _yt |_ _**x**_ _,_ _**y**_ _<t_ ~~�~~
where in the last equality we define _S_ ( _z_ ) := _−_ log( _σ_ ( _−z_ )) = _−_ log( 1+exp(1 _z_ ) [) = log(1 +] _[ e][z]_ [)][,]
namely the softplus function. Therefore, we can split _L_ ( _θ_ ) into a sum of token-wise losses:
  - 2  -  _L_ ( _θ_ ) = _ℓt_ ( _θ_ ) _,_ where _ℓt_ ( _θ_ ) := E _S_ _βt_ ∆ _t_ ( _**x**_ _,_ _**y**_ _<t, yt_ ) _,_ (10)
( _**x**_ _,_ _**y**_ ) _∼D_ [1] _[{][t][≤|]_ _**[y]**_ _[|}]_ _βt_
_t≥_ 1
∆ _t_ ( _**x**_ _,_ _**y**_ _<t, yt_ ) := log _π_ aligned� _yt |_ _**x**_ _,_ _**y**_ _<t_       - _−_ log _πθ_       - _yt |_ _**x**_ _,_ _**y**_ _<t_       - _._ (11)
In the following, we mainly focus on how to interpret the token-wise loss _ℓt_ ( _θ_ ).

D Interpretation of Our Constrained Fine-tuning Objective


**D.1.1** **Small** _βt_
When _βt_ is small, we have the following theorem showing that _ℓt_ ( _θ_ ) in Eqn 10 is **approximately the**
**cross-entropy loss** at position _t_, up to a constant.
**Theorem 1.** _For a given θ, as βt →_ 0 _, we have_
_ℓt_ ( _θ_ ) _−_ _C_ ( _βt_ ) = E
( _**x**_ _,_ _**y**_ ) _∼D_
- _−_ 1 _{t≤|_ _**y**_ _|} ·_ log _πθ_ ( _yt |_ _**x**_ _,_ _**y**_ _<t_ )� + _O_ ( _βt_ ) _,_ (12)
_where_
- 2 ��
log 2 + log _π_ aligned( _yt |_ _**x**_ _,_ _**y**_ _<t_ ) _,_ (13)
_βt_
_C_ ( _βt_ ) := E
( _**x**_ _,_ _**y**_ ) _∼D_
_{t≤|_ _**y**_ _|}_
_which is a bias term that is constant with respect to θ._
_Proof._ Recall that _ℓt_ ( _θ_ ) := E( _**x**_ _,_ _**y**_ ) _∼D_ 1 _{t≤|_ _**y**_ _|}_ _βt_ _[S]_ [(] _[β][t]_ [∆] _[t]_ [(] _**[x]**_ _[,]_ _**[ y]**_ _[<t][, y][t]_ [))][ and] _[ S]_ [(] _[z]_ [) := log(1 +] _[ e][z]_ [)][ is]
the softplus function [41]. By Taylor expansion, it holds for all _z_ that _S_ ( _βtz_ ) = _S_ (0) + _S_ _[′]_ (0) _βtz_ +
_S_ _[′′]_ ( _ξβtz_ ) _βt_ [2] _[z]_ [2][ for some] _[ ξ][ ∈]_ [(0] _[,]_ [ 1)][ depending on] _[ z]_ [. Since] _[ S]_ [(0) = log 2][,] _[ S][′]_ [(0) =] 2 [1] [and] _[ S][′′]_ [(] _[x]_ [)] _[ ∈]_
_S_ _[′′]_ ( _ξβtz_ ) _βt_ [2] _[z]_ [2][ for some] _[ ξ][ ∈]_ [(0] _[,]_ [ 1)][ depending on] _[ z]_ [. Since] _[ S]_ [(0) = log 2][,] _[ S][′]_ [(0) =] 2 [1] [and] _[ S][′′]_ [(] _[x]_ [)] _[ ∈]_
[0 _,_ 1 _/_ 4] for all _x_, we have _|S_ ( _βtz_ ) _−_ (log 2 + [1] _[β][t][z]_ [)] _[| ≤]_ [1] _[β]_ _t_ [2] _[z]_ [2][. Dividing both sides by] _[ β][t][/]_ [2][, we get]
[1]
2 _[β][t][z]_ [)] _[| ≤]_ 4 [1]
_βtz_ ) _−_ (log 2 + 2 [1] _[β][t][z]_ [)] _[| ≤]_ 4 [1] _[β]_ _t_ [2] _[z]_ [2][. Dividing both sides by] _[ β][t][/]_ [2][, we get]
2 - 2 1
_S_ ( _βtz_ ) _−_ log 2 + _z_ _≤_ (14)
���� _βt_ _βt_ ����� 2 _[β][t][z]_ [2] _[.]_
2 1
log 2 + _z_ _≤_
_βt_ ����� 2
(14)
2 _[β][t][z]_ [2] _[.]_
Then for our token-wise loss _ℓt_ ( _θ_ ), we have
       -       - 2
_ℓt_ ( _θ_ ) _−_ E 1 _{t≤|_ _**y**_ _|}_ log 2 + ∆ _t_
���� ( _**x**_ _,_ _**y**_ ) _∼D_ _βt_
_{t≤|_ _**y**_ _|}_
- 2
2 1
log 2 + ∆ _t_ ( _**x**_ _,_ _**y**_ _<t, yt_ ) _≤_
_βt_ ������ 2
2 _[β][t]_ ( _**x**_ _,_ _**y**_ E) _∼D_ [1] _[{][t][≤|]_ _**[y]**_ _[|}]_ [∆] _t_ [2][(] _**[x]**_ _[,]_ _**[ y]**_ _[<t][, y][t]_ [)]
= _O_ ( _βt_ ) _._
Recall that ∆ _t_ ( _**x**_ _,_ _**y**_ _<t, yt_ ) := log _π_ aligned� _yt |_ _**x**_ _,_ _**y**_ _<t_ - _−_ log _πθ_ - _yt |_ _**x**_ _,_ _**y**_ _<t_ - as per Eqn 11. We can
thus have
- 2
2 ��
log 2 + log _π_ aligned( _yt|_ _**x**_ _,_ _**y**_ _<t_ )
_βt_
_ℓt_ ( _θ_ ) _−_ E
( _**x**_ _,_ _**y**_ ) _∼D_
_{t≤|_ _**y**_ _|}_
= _−_ E
( _**x**_ _,_ _**y**_ ) _∼D_
(15)
- 1 _{t≤|_ _**y**_ _|}_ log _πθ_ ( _yt|_ _**x**_ _,_ _**y**_ _<t_ )� + _O_ ( _βt_ ) _._
Noting that _C_ ( _βt_ ) := E( _**x**_ _,_ _**y**_ ) _∼D_ - 1 _{t≤|_ _**y**_ _|}_ - _β_ 2 _t_ [log 2 + log] _[ π]_ [aligned][(] _[y][t][ |]_ _**[ x]**_ _[,]_ _**[ y]**_ _[<t]_ [)] ��, we can complete
the proof.
**D.1.2** **Large** _βt_
When _βt_ is large, we have the following theorem showing that _ℓt_ ( _θ_ ) can be approximated by
_ℓ_ ˜ _t_ ( _θ_ ) := E( _**x**_ _,_ _**y**_ ) _∼D_ - 1 _{t≤|_ _**y**_ _|}_ max _{_ ∆ _t_ ( _**x**_ _,_ _**y**_ _<t, yt_ ) _,_ 0 _}_ �.
**Theorem 2.** _For a given θ, as βt →_ + _∞, we have_
_ℓt_ ( _θ_ ) = 2 _ℓ_ [˜] _t_ ( _θ_ ) + _O_ ( _βt_ _[−]_ [1] ) _,_
_where_
_ℓ_ ˜ _t_ ( _θ_ ) := E            - 1 _{t≤|_ _**y**_ _|}_ max _{_ ∆ _t_ ( _**x**_ _,_ _**y**_ _<t, yt_ ) _,_ 0 _}_            - _._
( _**x**_ _,_ _**y**_ ) _∼D_
_Proof._ First, we note the following identity.
_S_ ( _z_ ) = log(1 + _e_ _[z]_ ) = max _{z,_ 0 _}_ + log((1 + _e_ _[z]_ ) _e_ _[−]_ [max] _[{][z,]_ [0] _[}]_ ) = max _{z,_ 0 _}_ + log(1 + _e_ _[−|][z][|]_ ) _._
(16)
This implies that
2 2
_S_ ( _βtz_ ) _−_ 2 _·_ max _{z,_ 0 _}_ =
���� _βt_ ���� _β_
_e_ _[−][β][t][|][z][|]_ _._ (17)
_βt_
log(1 + _e_ _[−][β][t][|][z][|]_ ) _≤_ [2]
_βt_ _β_
Then for our token-wise loss _ℓt_ ( _θ_ ), we have
_ℓt_ ( _θ_ ) _−_ 2 _·_ E - 1 _{t≤|_ _**y**_ _|}_ max _{_ ∆ _t_ ( _**x**_ _,_
���� ( _**x**_ _,_ _**y**_ ) _∼D_
E
_βt_ ( _**x**_ _,_ _**y**_ ) _∼D_
- 1 _{t≤|_ _**y**_ _|}_ max _{_ ∆ _t_ ( _**x**_ _,_ _**y**_ _<t, yt_ ) _,_ 0 _}_ - [�] _≤_ 1
��� _β_
1 _{t≤|_ _**y**_ _|}e_ _[−][β][t][|]_ [∆] _[t]_ [(] _**[x]**_ _[,]_ _**[y]**_ _[<t][,y][t]_ [)] _[|]_ [�]
(18)
Noting that the RHS is _O_ ( _βt_ _[−]_ [1] ) completes the proof.
Next, we show that _ℓ_ [˜] _t_ ( _θ_ ) is minimized to 0 if and only if _πθ_ ( _yt |_ _**x**_ _,_ _**y**_ _<t_ ) = _π_ aligned( _yt |_ _**x**_ _,_ _**y**_ _<t_ ).
**Theorem 3.** _The minimum value of_ _ℓ_ [˜] _t_ ( _θ_ ) _is_ 0 _. If_ Pr( _**x**_ _,_ _**y**_ ) _∼D_ [ _yt_ = _c |_ _**x**_ _,_ _**y**_ _<t_ ] _>_ 0 _for all_ ( _**x**_ _,_ _**y**_ ) _in the_
_support of D and all c in the vocabulary, then_ _ℓ_ [˜] _t_ ( _θ_ ) = 0 _if and only if πθ_ ( _yt |_ _**x**_ _,_ _**y**_ _<t_ ) = _π_ aligned( _yt |_
_**x**_ _,_ _**y**_ _<t_ ) _._
_Proof._ It is obvious that _ℓ_ [˜] _t_ ( _θ_ ) _≥_ 0, and _ℓ_ [˜] _t_ ( _θ_ ) = 0 can be attained when _θ_ stays the same as the
parameter for _π_ aligned. It is obvious that _πθ_ ( _yt |_ _**x**_ _,_ _**y**_ _<t_ ) = _π_ aligned( _yt |_ _**x**_ _,_ _**y**_ _<t_ ) implies _ℓ_ [˜] _t_ ( _θ_ ) = 0.
Conversely, suppose _ℓ_ [˜] _t_ ( _θ_ ) = 0. Then ∆ _t_ ( _**x**_ _,_ _**y**_ _<t, yt_ ) _≤_ 0 for all ( _**x**_ _,_ _**y**_ ) in the support of _D_ . By
definition of ∆ _t_ ( _**x**_ _,_ _**y**_ _<t, yt_ ), this implies _π_ aligned( _yt |_ _**x**_ _,_ _**y**_ _<t_ ) _≤_ _πθ_ ( _yt |_ _**x**_ _,_ _**y**_ _<t_ ). Summing over all
_yt_ in the vocabulary, we get 1 = [�] _yt_ _[π]_ [aligned][(] _[y][t][ |]_ _**[ x]**_ _[,]_ _**[ y]**_ _[<t]_ [)] _[ ≤]_ [�] _yt_ _[π][θ]_ [(] _[y][t][ |]_ _**[ x]**_ _[,]_ _**[ y]**_ _[<t]_ [) = 1][. So all the]
_yt_ _[π]_ [aligned][(] _[y][t][ |]_ _**[ x]**_ _[,]_ _**[ y]**_ _[<t]_ [)] _[ ≤]_ [�]
_yt_ in the vocabulary, we get 1 = [�] _yt_ _[π]_ [aligned][(] _[y][t][ |]_ _**[ x]**_ _[,]_ _**[ y]**_ _[<t]_ [)] _[ ≤]_ [�] _yt_ _[π][θ]_ [(] _[y][t][ |]_ _**[ x]**_ _[,]_ _**[ y]**_ _[<t]_ [) = 1][. So all the]
inequalities must be equalities, and we get _πθ_ ( _yt |_ _**x**_ _,_ _**y**_ _<t_ ) = _π_ aligned( _yt |_ _**x**_ _,_ _**y**_ _<t_ ).
This corresponds to the intuition that large _βt_ places emphasis on matching the generative distribution
of fine-tuned model to the initial aligned model.


The gradient of _ℓt_ ( _θ_ ) on a data point ( _**x**_ _,_ _**y**_ ) with _|_ _**y**_ _| ≥_ _t_ is derived as:
_∇_   - 2 _S_ ( _βt_ ∆ _t_ ( _**x**_ _,_ _**y**_ _<t, yt_ ))� = 2 _S_ _[′]_ ( _βt_ ∆ _t_ ( _**x**_ _,_ _**y**_ _<t, yt_ ))( _−∇_ log _πθ_   - _yt |_ _**x**_ _,_ _**y**_ _<t_ ))
_βt_
= 2 _σ_ ( _βt_ ∆ _t_ ( _**x**_ _,_ _**y**_ _<t, yt_ ))( _−∇_ log _πθ_                - _yt |_ _**x**_ _,_ _**y**_ _<t_ )) _._ (19)
Note that the gradient of the cross-entropy loss is _−∇_ log _πθ_ - _yt |_ _**x**_ _,_ _**y**_ _<t_ �. Therefore, compared
to vanilla cross-entropy loss, our fine-tuning objective essentially applies an additional adaptive
weight _wt_ := 2 _· σ_ ( _βt_ ∆ _t_ ( _**x**_ _,_ _**y**_ _<t, yt_ )) for the token-wise gradient term of cross-entropy. The weight
_wt_ decreases as ∆ _t_ ( _**x**_ _,_ _**y**_ _<t, yt_ ) := log _π_ aligned� _yt |_ _**x**_ _,_ _**y**_ _<t_ - _−_ log _πθ_ - _yt |_ _**x**_ _,_ _**y**_ _<t_ - decreases. This
difference in log probabilities essentially characterizes the deviation between the fine-tuned model _πθ_
and the initially aligned model _π_ aligned at the token position _t_ (see also Theorem 3). Taking together,
we can see that the fine-tuning objective will adaptively diminish the weight of those token positions
where the deviation of the distribution approaches a certain threshold (controlled by _βt_ ), thereby
constraining it from further deviation (by diminishing the gradient at this position). Note that, at the
beginning of the fine-tuning when _πθ_ is initialized as _π_ aligned, the weight _wt_ = 1, and the gradient of
the loss is equivalent to that of standard cross-entropy loss.

D.2 Gradients of The Constrained Fine-tuning Objective


We note that our loss function in Eqn 3 can also be interpreted from a reinforcement learning
perspective if we **cast fine-tuning as a KL-constrained reinforcement learning problem** rather
than a standard supervised fine-tuning problem. Specifically, we can follow a similar trick as DPO [16]
to derive a unified loss, but taking a different approach to reward modeling. We, instead, formulate
our problem setting like Mudgal et al. [40], where we optimize at the token level (where tokens are
actions), rather than DPO which uses a sequence-level optimization (corresponding more to a bandit
setting where entire sequences are actions).
**D.3.1** **A Token-wise Reinforcement Learning (RL) Formulation.**
We first introduce the following token-wise RL formulation for LM alignment, which is adapted
from Mudgal et al. [40]. In Appendix D.3.2, we show how a fine-tuning task can be cast into this
token-wise RL problem, and Eqn 3 is essentially a surrogate learning objective of this RL problem.
**Reward Function.** For a pair of an input and a model response ( _**x**_ _,_ _**y**_ ), we cast custom fine-tuning as
a problem of further optimizing an already aligned model for a new custom reward function _r_ ([ _**x**_ _,_ _**y**_ ]).
Here we use [ _**x**_ _,_ _**y**_ ] to denote a concatenation of the two sequences and we use this concatenation to
denote a state, and the reward function is defined on this state. Following Mudgal et al. [40], we can
decompose it to a token-wise reward _R_ ([ _**x**_ _,_ _**y**_ _<t_ ]) defined on the intermediate state [ _**x**_ _,_ _**y**_ _<t_ ]:
�0 _,_ _yt−_ 1 _̸_ = _EOS_
_R_ ([ _**x**_ _,_ _**y**_ _<t_ ]) = (20)
_r_ ([ _**x**_ _,_ _**y**_ _<t_ ]) _,_ _yt−_ 1 = _EOS_ _[,]_
where _EOS_ is the end of the sequence token. The reward is nonzero only if the decoding is
complete. We note that, by _r_ ([ _**x**_ _,_ _**y**_ _<t_ ]) and _R_ ([ _**x**_ _,_ _**y**_ _<t_ ]), we mean the function _r_ and _R_ are applied
on the concatenation of _**x**_ and _**y**_ _<t_ . Similarly, in the following, we will also use notations such
as _R_ ([ _**x**_ _,_ _**y**_ _<t,_ _**z**_ _<τ_ ]) and _R_ ([ _**x**_ _,_ _**y**_ _<t, z_ ]) to denote that we apply _R_ on the concatenation between
[ _**x**_ _,_ _**y**_ _<t_ ] and a followup sequence _**z**_ _<τ_ or just a single token _z_ . These concatenations all represent
some states of the generation.
**Value Function.** At an intermediate state [ _**x**_ _,_ _**y**_ _<t_ ], the value function of a policy _π_ defined on this
reward function can be written as:
_Vπ_ ([ _**x**_ _,_ _**y**_ _<t_ ]) := E
_**z**_ _∼π_ ( _·|_ _**x**_ _,_ _**y**_ _<t_ )
�� - - [�]
_R_ [ _**x**_ _,_ _**y**_ _<t,_ _**z**_ _<τ_ ]
_τ_ _≥_ 1
(21)
Here _**z**_ is a sequence, and _**z**_ _<τ_ is empty when _τ_ = 1 as we index tokens in a sequence starting from
the index number 1. Also, in the following formulations, we will have multiple different notations
_πθ_, _π_ aligned, _π_ _[∗]_ to denote different policies instances, so we will use _Vπθ_, _Vπ_ aligned, _Vπ∗_ to differently
denote their value functions respectively.
**Advantage Function.** When the custom fine-tuning is modeled by the reward function _R_, we define
an expected advantage function of a fine-tuned model _πθ_ w.r.t. the initially aligned model _π_ aligned:
_A_ ˆ _πθ_ ([ _**x**_ _,_ _**y**_ _<t_ ]) := E
_z∼πθ_ ( _·|_ _**x**_ _,_ _**y**_ _<t_ )
   -    -    -    - [�]
_Vπ_ aligned [ _**x**_ _,_ _**y**_ _<t, z_ ] _−_ _Vπ_ aligned [ _**x**_ _,_ _**y**_ _<t_ ]
_,_ (22)
Here the advantage function is defined for non-terminal states [ _**x**_ _,_ _**y**_ _<t_ ] where _yt−_ 1 is not the ending
token _EOS_, and _z_ is a single token sampled by _z ∼_ _πθ_ ( _·|_ _**x**_ _,_ _**y**_ _<t_ ). Note that the left-hand term
could also be viewed as a _Q_ -function with _z_ being the action. In other words, a language model
can be viewed as a fully observable Markov decision process (MDP) with state represented by the
concatenation of the prompt and the partially decoded response tokens so far and action represented
by the next token that is to be decoded.
**The Reinforcement Learning Objective.** Following Mudgal et al. [40], we adopt a token-wise RL
learning objective:
- _A_ ˜ _πθ_ ([ _**x**_ _,_ _**y**_ _<t_ ]) _−_ _βt · D_ KL� _πθ_ ( _· |_ _**x**_ _,_ _**y**_ _<t_ ) �� _π_ aligned( _· |_ _**x**_ _,_ _**y**_ _<t_ )� [��] _,_ (23)
max E
_θ_ ( _**x**_ _,_ _**y**_ ) _∼D_
��
_t≥_ 1
where the advantage function is optimized at each token position _t_ with a token-wise KL regularizer
term _βt · D_ KL� _πθ_ ( _· |_ _**x**_ _,_ _**y**_ _<t_ ) �� _π_ aligned( _· |_ _**x**_ _,_ _**y**_ _<t_ )� applied for each token position. The strength of
regularization at each token position is controlled by _βt_ .
**Closed Form of The Optimal Solution** _π_ _[∗]_ **.** Omitting some intermediate steps for brevity, by
Theorem 2.1 of Mudgal et al. [40], the optimal policy solution _π_ _[∗]_ of Eqn 23 is:
_π_ _[∗]_ ( _z|_ _**x**_ _,_ _**y**_ _<t_ ) = (24)
_Z_ ( _**x**_ _,_ _**y**_ _<t_ ) _[π]_ [aligned][(] _[z][|]_ _**[x]**_ _[,]_ _**[ y]**_ _[<t]_ [)] _[ ·][ e][V][π][∗]_ [([] _**[x]**_ _[,]_ _**[y]**_ _[<t][,z]_ [])] _[/β][t][,]_
where _Z_ ( _**x**_ _,_ _**y**_ _<t_ ) is the partition function, _Vπ∗_ is the value function of this optimal policy. We note
that this conveniently allows us to re-arrange terms to represent the optimal value function—similar
to the steps taken during the derivation of DPO [16]:
_π_ _[∗]_ ( _z|_ _**x**_ _,_ _**y**_ _<t_ )
_Vπ_ _[∗]_ ([ _**x**_ _,_ _**y**_ _<t_ ]) = _βt ·_ log (25)
_π_ aligned( _z|_ _**x**_ _,_ _**y**_ _<t_ ) [+] _[ β][t][ ·]_ [ log] _[ Z]_ [(] _**[x]**_ _[,]_ _**[ y]**_ _[<t]_ [)]

D.3 Interpreting Eqn 3 from A Reinforcement Learning Perspective


In the setting of custom fine-tuning we consider in this work, the dataset _D_ is in the form of
_D_ := _{_ _**x**_ _,_ _**y**_ _}_ with only inputs and example outputs, without preference pairs. Now, we show
an alternative to the standard SFT objective (Eqn 1) for learning from this dataset: casting the
optimization as a KL-regularized RL objective in Appendix D.3.1. We will show how our tokenwise constrained fine-tuning objective in Eqn 3 can essentially be derived from this token-wise
KL-regularized RL problem!
Intuitively, fine-tuning _π_ aligned further on custom data points ( _**x**_ _,_ _**y**_ ) implicitly assumes that this
fine-tuning data is as preferable or more preferable than whatever the model would currently output.
To represent this implied preference that the custom example response _**y**_ is better than the current
responses from _π_ aligned, we can effectively leverage existing preference learning methods. Recall
that, in preference optimization (with both positive and negative examples), the Bradley-Terry
model [62] is used as a model of the underlying reward function. In our setup, we don’t have
pair-wise preference data, and we only have inputs and positive examples. However, we can define
a boolean _T_ ([ _**x**_ _,_ _**y**_ _<t_ ] _, yt_ ) to encode the preference: in the fine-tuning task, _yt_ is a preferable token
output following [ _**x**_ _,_ _**y**_ _<t_ ] compared to the responses from the initial aligned model _π_ aligned. So,
given an expected random draw from the aligned policy and the draw from the fine-tuned optimal
policy, we can define:
 - P _T_ ([ _**x**_ _,_ _**y**_ _<t_ ] _, yt_ ) = _σ_
_Vπ∗_ ([ _**x**_ _,_ _**y**_ _<t, yt_ ]) _−_ E
_z∼π_ aligned( _·|_ _**x**_ _,_ _**y**_ _<t_ ) _[V][π][∗]_ [([] _**[x]**_ _[,]_ _**[ y]**_ _[<t][, z]_ [])]
(26)
Intuitively, this means that—conditioned on the context [ _**x**_ _,_ _**y**_ _≤t_ ]—if there is a continuation token _yt_
that is higher than the average reward of actions sampled from the initial aligned model _π_ aligned, it is
likely to be an improved or preferred choice in the custom fine-tuning task. The higher the margin
_Vπ∗_ ([ _**x**_ _,_ _**y**_ _<t, yt_ ]) _−_ E _z∼π_ aligned( _·|_ _**x**_ _,_ _**y**_ _<t_ ) _Vπ∗_ ([ _**x**_ _,_ _**y**_ _<t, z_ ]) is, the more likely it is an improvement.
With this function in mind, combined with Eqn 25, we can leverage a similar derivation to DPO [16]
to arrive at a constrained fine-tuning objective that is only dependent on the current policy, but
is regularized by the original aligned policy. We plug in the closed form of the optimal value
function (Eqn 25) into the modeling in Eqn 26, obtaining:
 - P _T_ ([ _**x**_ _,_ _**y**_ _<t_ ] _, yt_ ) = _σ_
= _σ_
_π_ _[∗]_ ( _yt|_ _**x**_ _,_ _**y**_ _<t_ ) _π_ _[∗]_ ( _z|_ _**x**_ _,_ _**y**_ _<t_ )
_βt_ log E
_π_ aligned( _yt|_ _**x**_ _,_ _**y**_ _<t_ ) _[−]_ _[β][t]_ _z∼π_ aligned( _·|_ _**x**_ _,_ _**y**_ _<t_ ) [log] _π_ aligned( _z|_ _**x**_ _,_ _**y**_ _<t_ )
_π_ _[∗]_ ( _yt|_ _**x**_ _,_ _**y**_ _<t_ )    -    - [�]
_βt_ log _π_ _[∗]_ ( _·|_ _**x**_ _,_ _**y**_ _<t_ )�� _π_ aligned( _·|_ _**x**_ _,_ _**y**_ _<t_ ) _._
_π_ aligned( _yt|_ _**x**_ _,_ _**y**_ _<t_ ) [+] _[ β][t][D]_ [KL]
_._
(27)
Thus, we don’t need to explicitly learn the value function, instead it is implicitly encoded by the
policy. Then the optimization objective can become:
1 - - [�]
log P _θ_ _T_ ([ _**x**_ _,_ _**y**_ _<t_ ] _, yt_ ) _,_ (28)
_βt_
max E
_θ_ ( _**x**_ _,_ _**y**_ ) _∼D_
��
_t≥_ 1
where:
 -  P _θ_ _T_ ([ _**x**_ _,_ _**y**_ _<t_ ] _, yt_ ) := _σ_
_πθ_ ( _yt|_ _**x**_ _,_ _**y**_ _<t_ )     -     - [�]
_βt_ log _π_ _[∗]_ ( _·|_ _**x**_ _,_ _**y**_ _<t_ )�� _π_ aligned( _·|_ _**x**_ _,_ _**y**_ _<t_ ) _,_
_π_ aligned( _yt|_ _**x**_ _,_ _**y**_ _<t_ ) [+] _[ β][t][D]_ [KL]
_,_
(29)
and the division of _βt_ in Eqn 28 normalizes the gradient norm at each position _t_ as we will later see
in Eqn 31 and also clarified in Section 4.1 (and Appendix D.2).
      -       Note that _βt · D_ KL _π_ _[∗]_ ( _·|_ _**x**_ _,_ _**y**_ _<t_ )�� _π_ aligned( _·|_ _**x**_ _,_ _**y**_ _<t_ ) _≥_ 0 is a non-negative constant, we have:
 -  P _θ_ _T_ ([ _**x**_ _,_ _**y**_ _<t_ ] _, yt_ ) _≥_ _σ_
_πθ_ ( _yt|_ _**x**_ _,_ _**y**_ _<t_ )
_βt ·_ log
_π_ aligned( _yt|_ _**x**_ _,_ _**y**_ _<t_ )
(30)
So, the objective in Eqn 28 can be replaced with a lower bound surrogate objective _Lθ_ :
1 - - [�]
log P _θ_ _T_ ([ _**x**_ _,_ _**y**_ _<t_ ] _, yt_ )
_βt_
E
( _**x**_ _,_ _**y**_ ) _∼D_
��
_t≥_ 1
_≥_ _Lθ_ (31)
where
- ��
_πθ_ ( _yt|_ _**x**_ _,_ _**y**_ _<t_ )
_βt ·_ log _._
_π_ aligned( _yt|_ _**x**_ _,_ _**y**_ _<t_ )
_Lθ_ := E
( _**x**_ _,_ _**y**_ ) _∼D_
��
_t≥_ 1
_·_ log _σ_
_βt_
Eqn 3 is then equivalent to min _θ −Lθ_ _⇐⇒_ max _θ Lθ_ . So, optimizing the objective Eqn 3 is
essentially to maximize the lower-bound of the reinforcement learning objective in Eqn 28.

D.3.2 Casting Fine-tuning into a Reinforcement Learning Objective


Jiaming Ji [1][*], Kaile Wang [1][*], Tianyi Qiu [1][*], Boyuan Chen [1][*], Jiayi Zhou [1][*]
Changye Li [1], Hantao Lou [1], Juntao Dai [12], Yunhuai Liu [3], Yaodong Yang [12] _[†]_
1Institute for Artificial Intelligence, Peking University
2Beijing Academy of Artificial Intelligence
3School of Computer Science, Peking University

Language Models Resist Alignment: Evidence From Data Compression

Jiaming Ji
Kaile Wang
Tianyi Qiu
Boyuan Chen
Jiayi Zhou
Changye Li
Hantao Lou
Yaodong Yang


Large language models (LLMs) may exhibit
unintended or undesirable behaviors. Recent
works have concentrated on aligning LLMs
to mitigate harmful outputs. Despite these efforts, some anomalies indicate that even a wellconducted alignment process can be easily circumvented, whether intentionally or accidentally. Does alignment fine-tuning yield have
robust effects on models, or are its impacts
merely _superficial_ ? In this work, we make the
first exploration of this phenomenon from both
theoretical and empirical perspectives. Empirically, we demonstrate the _elasticity_ of postalignment models, _i.e._, the tendency to revert to
the behavior distribution formed during the pretraining phase upon further fine-tuning. Leveraging compression theory, we formally deduce
that fine-tuning disproportionately undermines
alignment relative to pre-training, potentially
by orders of magnitude. We validate the presence of _elasticity_ through experiments on models of varying types and scales. Specifically, we
find that model performance declines rapidly
before reverting to the pre-training distribution,
after which the rate of decline drops significantly. Furthermore, we further reveal that _elas-_
_ticity_ positively correlates with the increased
model size and the expansion of pre-training
data. Our findings underscore the need to address the inherent _elasticity_ of LLMs to mitigate their resistance to alignment. [1]


Large language models (LLMs) have shown remarkable capabilities (Achiam et al., 2023; Zhang
et al., 2025). However, due to the inevitable
biases and harmful content present in training
*Equal contribution. _[†]_ Corresponding author. If you have
any questions, feel free to email {jiamg.ji, wkl, tianyi.qiu,
cbylll, gaiejj}@stu.pku.edu.cn, yaodong.yang@pku.edu.cn.
1The model weight and code are available at pku-lm-res
ist-alignment.github.io.
Figure 1: **The** _**Elasticity**_ **of Language Models.** The
change in normalized compression rates (∆ _γp_ _[D]_ _**θ**_ _[i][/][D]_ ) and
the dataset volume ( _|Di|_ ) follows an inverse proportionality law after perturbations, which is akin to the
relationship between spring deformation (∆ _li_ ) and stiffness ( _ki_ ) in coupled springs. We conjecture that the
_elasticity_ causes language models to resist alignment,
enabling the possibility of _inverse alignment_ .
datasets (Bai et al., 2022a; Ji et al., 2024c; Qian
et al., 2024; Lin et al., 2025), LLMs often exhibit
behaviors that deviate from human intentions, a
phenomenon we refer to as _model misalignment_ .
Training-based alignment methods, including supervised fine-tuning (SFT), reinforcement learning with human feedback (RLHF) (Ouyang et al.,
2022), and other derivatives (Rafailov et al., 2024;
Bai et al., 2022b; Lee et al., 2023; Gulcehre et al.,
2023; Dong et al., 2023; Xiong et al., 2024; Li et al.,
2023; Zhou et al., 2023, 2025; Ji et al., 2024a),
are the dominant approaches for aligning models.
These methods aim to optimize model behavior
by rejecting harmful distributions, ensuring LLMs
remain consistent with human intentions and values (Ji et al., 2023; Casper et al., 2023).
However, these alignment methods do not truly
penetrate the model representations but merely perform _superficial alignment_ (Qi et al., 2024a; Cohen
et al., 2024; Wen et al., 2024). Recent studies have
shown that highly safety-aligned models can become unsafe again with minimal fine-tuning(Yang
et al., 2023b; Zhou et al., 2024). Furthermore, finetuning aligned LLMs on non-malicious datasets
may also weaken models’ safety mechanisms (Qi
et al., 2024a; Jain et al., 2023).
_**Why is alignment so fragile?**_
In this work, we make the first exploration of
the possible mechanism behind the counterintuitive
phenomenon: the existence of an alignment resistance mechanism in language models. This mechanism may limit the alignment process of LLMs
to superficial adjustments. It could allow the reversal or revocation of alignment through a series
of technical measures, a concept we refer to as _in-_
_verse alignment_ . What drives language models to
resist alignment? How does this mechanism lead
to _inverse alignment_ ? Our key contributions are
summarized as follows:
- **(Phenomenon)** We uncover that language models exhibit _elasticity_, as illustrated in Figure 1
and Theorem 4.2. It encompasses _resistance_ :
pre-trained models tend to retain their original
distribution; and _rebound_ : the deeper alignment
of models, the faster they return to the pre-trained
distribution under reverse finetuning. Moreover, The model’s change in compression rates
∆ _γp_ _[D]_ _**θ**_ _[i][/][D]_ across different datasets is inversely proportional to their sizes _|Di|_, which is analogous
to the deformation behavior of a series of springs,
as illustrated in Section 4.3.
- **(Mechanism)** We systematically model the training and alignment process of language models
through compression theorem, as detailed in Section 3.2. We elaborate on the compression protocol of language models to explore their training
and alignment processes, laying a foundation for
subsequent research on _elasticity_ .
- **(Validation)** We experimentally observe consistent _resistance_ and _rebound_ phenomena across
various LLMs, as detailed in Section 5. This
highlights the universality of _elasticity_ and the
need for systematic approaches to achieve robust
and deep alignment.


**The Fragility of LLMs Alignment** Pre-trained
LLMs often generate offensive content. Recent
initiatives (Ouyang et al., 2022; Bai et al., 2022a;
Yang et al., 2024) have aimed to align these models
to minimize harmful outputs (Yang et al., 2023a;
Cui et al., 2024). However, research indicates that
even well-aligned models can be compromised easily, and fine-tuning them on non-malicious datasets
might unintentionally impair their safety mechanisms (Yang et al., 2023b; Qi et al., 2024b; Hubinger et al., 2024; Dong et al., 2024). Moreover,
recent study reveals that models selectively adhere
to training objectives during the training phase
to preserve their inherent preferences (Greenblatt
et al., 2024; Lang et al., 2024; Park et al., 2024).
Why is alignment so fragile? Wei et al. (2024)
use weight attribution to separate safety-critical
and utility-related regions at both neuron and rank
levels. Qi et al. (2024a) propose the concept of
shallow safety alignment, arguing that safety alignment should reach beyond surface-level tokens to
shape the model’s internal mechanisms.


We discover that language models exhibit elasticity, which leads to resistance to alignment. In this
section, we formally introduce the definition of
_elasticity_, along with the compression theory tools
used in the analysis. Firstly, we review the training
alignment objective and the compression theorem.


**Pre-training.** During pre-training, an LLM acquires foundational language comprehension and
reasoning abilities by processing vast quantities of
unstructured text. The pre-train loss _L_ PT( _**θ**_ ; _D_ PT)
is defined as follows:
_L_ PT( _**θ**_ ; _D_ PT) = _−_ E( _**x**_ _,xN_ ) _∼D_ PT �log _p_ _**θ**_ - _xN_ �� _**x**_ �� _,_
where _**x**_ = ( _x_ 0 _, · · ·, xN_ _−_ 1) and _N ∈_ N, such that
( _x_ 0 _, · · ·, xN_ ) forms a prefix in some piece of pretraining text. _D_ PT stands for pre-training dataset.
**Supervised Fine-tuning (SFT).** SFT adjusts the
pre-trained models to follow specific instructions,
utilizing a smaller dataset compared to the pretraining corpus to ensure model alignment with
target tasks. For _D_ SFT = �� _**x**_ _[i]_ _,_ _**y**_ _[i]_ [��] _[N]_ _i_ =1 [sampled]
from a high-quality distribution, SFT aims to minimize the negative log-likelihood loss:
_L_ SFT( _**θ**_ ; _D_ SFT) = _−_ E( _**x**_ _,_ **y** ) _∼D_ SFT �log _p_ _**θ**_ - _**y**_ �� _**x**_ �� _._
Given that E( _**x**_ _,_ **y** ) _∼D_ SFT �log _pD_ - _**y**_ �� _**x**_ �� is fixed
when specifying _D_ SFT, the optimization objective
_L_ SFT becomes the Kullback-Leibler (KL) divergence between _p_ _**θ**_ and the SFT distribution.
**Lossless Compression.** The goal of lossless compression is to find a compression protocol that encodes a given dataset _D_ and its distribution _PD_ with
the smallest possible expected length and allows
for a decoding scheme that can perfectly reconstruct the original dataset. According to Shannon’s
source coding theorem (Shannon, 1948), for a random variable follows _PD_, the expected code length
_L_ of any lossless compression protocol satisfies:
_L ≥_ _H_ ( _PD_ ) _,_
where _H_ ( _PD_ ) is the Shannon entropy of _PD_ .
**Compression and Prediction.** Compression and
prediction are tightly interconnected. Consider a
model _p_ _**θ**_ and _**x**_ = ( _x_ 0 _, · · ·, xm−_ 1) derived from a
dataset _D_, the expected code length _L_ under arithmetic coding (Witten et al., 1987) satisfies:
model as a lossless compression protocol. The
process of training and aligning the model on different datasets can be equivalently viewed as the
joint compression of these datasets by the protocol.
The model’s compression rate on various datasets
serves as a surrogate metric for the loss during training and alignment. In this section, we detail the
modeling specifics of this compression protocol.
Considering the impact of tokenization on compression rate, we use tokenized sequences as input
and output modalities. For simplicity, we assume
the tokenized vocabulary consists of only binary
tokens (specifically 0/1).

 _,_
_L_ = E _**x**_ _∼D_
 [�]

[�] _−_ log2 _p_ _**θ**_ - _xk_ �� _x_ 0 _,···,k−_ 1�
0 _≤k≤m_
which is the current training objective of language
models. Minimizing log-likelihood loss is equivalent to minimizing the compression rate when
models act as a lossless compressor. Thus, optimal
compression and prediction are equivalent (Delétang et al., 2023; Hutter, 2005). Experiments show
the equivalence between large language model prediction and compression (Delétang et al., 2023),
and that compression performance correlates linearly with intelligence (Huang et al., 2024).


We aim to study the dynamic changes in language
models during training and alignment. Given the
equivalence between language model training and
data compression (Delétang et al., 2023), a feasible modeling approach is to treat the language
In token tree modeling, the model’s training process is conceptualized as learning the node weights
of the token tree. However, due to the finite size of
the model’s parameters in practical scenarios, the
model cannot accurately capture node weights at
arbitrary depths of the token tree. Therefore, we
propose the following assumption.
**Assumption 3.2** (Scale of _T_ is Monotone with
Model Size) **.** Consider a parameterized model
_p_ _**θ**_ ( _·_ ) and a dataset _D_, We assume that the depth of
the portion of _TD_ that can be perfectly modelled by
_p_ _**θ**_ is monotonically increasing with the size of _**θ**_ .
Based on the above assumption, we can define
the compression protocol during the model training
and alignment process.
Due to the optimality and losslessness of Huffman coding (Huffman, 1952), the compression protocol ensures optimal compression while preserving losslessness. We can therefore calculate the
model’s ideal code length and other informationtheoretic metrics based on the protocol.
Since training and alignment involve multiple
datasets with different and independent distributions, we consider the joint compression scenario
for multiple datasets. For _N_ pairwise disjoint
datasets _D_ 1 _, · · ·, DN_, the node weights _p_ _[D]_ _lj_ [of the]
token tree for the dataset _D_ = [�] _i_ _[N]_ =1 _[D][i]_ [ satisfy:]
    - _Ni_ =1 _[p]_ _l_ _[D][i]_ �� _Di_ ��
_p_ _[D]_ _lj_ [=] ~~�~~ _Ni_ =1 ~~�~~      - _Di_ ~~�~~      - _,_
where _p_ _[D]_ _lj_ _[i]_ [stands for the probability value for nodes]
in _TDi_ while �� _Di_ �� represents the size of _Di_ . Thus,
the compression rate _γp_ _[D]_ _**θ**_ _[i]_ [on specific datasets can]
also be defined accordingly:
_γp_ _[D]_ _**θ**_ _[i]_ [=][ E] _**[x]**_ _[∼P]_ _i_

- E _**x**_ _∼Pi_ - _L_ _[D]_ _p_ _**θ**_ _[i]_ [(] _**[x]**_ [)] ~~��~~ _**x**_ ~~��~~

 _._
- _d_ �2 _l−_ 1
_l_ =1 _j_ =1 _[p]_ _lj_ _[D][i]_ [log] _[ p]_ _lj_ _[D]_
_d_
= Θ
 _−_
Here, the compression rate is defined as the compressed encoding length divided by the original
length (Delétang et al., 2023) and ensures consistency between the training and compression objective, which means that minimizing the training loss
is equivalent to minimizing the compression rate.
Please see Appendix A for more details.
**3.3** **The Formal Definition of** _**Elasticity**_
To formalize _inverse alignment_ and _elasticity_, we
provide precise definitions of these concepts.
The eval metrics _ρ_ can be viewed as a measure
of behavioral and distributional proximity between
models. In the context of the compression theorem,
we use compression rate _γp_ _[D]_ _**θ**_ _[i]_ [as the metric] _[ ρ]_ [ to]
evaluate _elasticity_ during the alignment process.


By now, we hope to have convinced the reader of
the concept of compression modeling in language
models’ training and alignment. In the following,
we apply this to analyze language models’ training
and alignment process, focusing on the underlying
reasons that lead the model to alignment resistance.
In Figure 1, we have already presented the theorem of _elasticity_ : when subject to fine-tuning perturbations, language models tend to retain the distribution associated with larger datasets while rejecting that of smaller ones. But why _elasticity_ resists
alignment? Is there a corresponding _elastic_ invariant? How does _elasticity_ make _inverse alignment_
possible? In this section, we will formalize _elastic-_
_ity_ of language models and analyze the impact of
_elasticity_ on the model’s behavior.
**4.1** **Formal Derivation of** _**Elasticity**_
Our primary goal is to investigate the behavioral
changes of a language model after pre-training and
alignment, particularly in response to perturbations,
typically caused by fine-tuning with a minimal
dataset. We use compression rate as an evaluation metric to assess these changes. In analyzing
the model’s behavior, we focus on two datasets:
_Dp_, a larger dataset representing pre-training or the
primary objective of training, and _Da_, a smaller
dataset representing the alignment process or the
secondary objectives of training. Since the pretraining stage encompasses a wide range of data,
without loss of generality, we assume that the
datasets in the alignment process follow the same
distribution as some subset of the pre-training data.
**Definition 4.1** (Normalized Compression Rate) **.**
For _N_ distinct datasets _D_ 1 _, · · ·, DN_ and a parameter model _p_ _**θ**_ compressing _D_ = [�] _i_ _[N]_ =1 _[D][i]_ [, the nor-]
malized compression rate _γp_ _[D]_ _**θ**_ _[i][/][D]_ for a particular
dataset _Di_ is defined as:
_γp_ _[D]_ _**θ**_ _[i][/][D]_ = _γp_ _[D]_ _**θ**_ _[i]_ _[−]_ [log] _[ M,]_ (1)
where _M_ is the number of leaf nodes of the pruned
tree _Ti_ _[′]_ [of dataset] _[ D][i]_ [.]
The normalized compression rate allows for
comparing the model’s compression performance
across different datasets. The smaller the normalized compression rate of a dataset, the better the
model’s compression performance on the dataset.
With this definition in hand, we proceed to our
main result: language models exhibit _elasticity_ .
Theorem 4.2 shows that as the perturbation increases, the normalized compression rates of the
model for _Dp_ decrease and _Da_ increase and the
rate of changes is strongly correlated with the size
of the datasets. Unlike the proportional changes
in compression rates across different datasets, the
language model seems to _prefer_ the dataset with
a larger volume, leading to biased model behavior
after the perturbation.
**4.2** _**Elasticity**_ **and** _**Inverse Alignment**_
Theorem 4.2 shows the normalized compression
rate change of different datasets is inversely proportional to their sizes under perturbations. A significant size difference between the pre-training
and alignment datasets causes rapid performance
degradation on the alignment dataset due to the
inverse relationship, often spanning several orders
of magnitude.
Intuitively, the model’s compression process
across multiple datasets is akin to resource allocation between towns: in a region with both a
large metropolis and rural villages, to maximize
overall economic productivity, resources are typically allocated to the metropolis first, to leverage
its scale and agglomeration effects. In other words,
the metropolis/large dataset occupies a dominant
position in the system.
Therefore, the _elasticity_ of language models
makes _inverse alignment_ possible. Due to the substantial data volume disparity between pre-training
and alignment datasets, the model tends to revert to
the pre-training rather than alignment distribution
when subsequent perturbations occur. This indicates an inherent tendency toward _inverse align-_
_ment_ during fine-tuning. Maximizing the impact of
_elasticity_ through well-designed perturbations will
be key to achieving true _inverse alignment_ .
**4.3** _**Elasticity**_ **and the Hooke’s Law.**
The inverse proportionality result in Theorem 4.2
provides a potential invariant in the model training and alignment process: after perturbation, the
rate of change in the compression rates of different
datasets is inversely proportional to their sizes, with
the absolute value of the product being a constant.
This constant characterizes the impact of the perturbation on the model and indirectly describes the
model’s resistance to perturbations, or its _elasticity_ .
The _elasticity_ of the model can be intuitively
analogized to a series system of springs, as shown
in Figure 1. Consider two massless springs in series, with spring constants _k_ 1 and _k_ 2, respectively.
When the entire system undergoes deformation due
to an external force _F_, the system reaches a stable
state, and the elastic force exerted by each spring
is equal to _F_ . According to Hooke’s Law (Hooke,
2016), the elongation ∆ _l_ 1 and ∆ _l_ 2 of each spring is
inversely proportional to its spring constant. Thus,
in this system, we have:
_F ∝_ _k_ 1 _·_ ∆ _l_ 1 = _k_ 2 _·_ ∆ _l_ 2 _._
In the language model setting, after integrating
Theorem 4.2 to _l_, we obtain ∆ _γp_ _[D]_ _**θ**_ _[i][/][D]_ across different datasets, which is equivalent to the change
in the KL divergence ∆ _D_ KL( _Pp_ _**θ**_ _||PDi_ ) between
the model’s distribution and the distributions of the
individual datasets, is inversely proportional to the
size of the datasets _|Di|_ . Here, we only consider the
absolute value of ∆ _D_ KL. Analogous to the series
spring model, the _elasticity F_ in LLMs satisfies:
_F ∝_ _|Di| ·_ ∆ _D_ KL( _Pp_ _**θ**_ _||PDi_ ) _,_ (3)
where ∆ _D_ KL corresponds to ∆ _l_ in the spring
model, while _|D|_ corresponds to the spring constant
_k_, thus leading to the _elasticity_ of LLMs.