**[https://github.com/jcnf0/targeting-alignment.](https://github.com/jcnf0/targeting-alignment)**
I. INTRODUCTION
Improvements in machine learning—specifically the introduction of the transformer architectures [50]—have led to
increasingly powerful large language models (LLMs). Whether
they are accessed through an API or interface (e.g., GPT [35],
Claude [1]) or published as open-source (e.g., Llama 2 [32],
Llama 3 [33], Qwen [49], etc.), these models have become the
de facto tool for tasks involving natural language. They serve
as foundations for new tools, in which they are augmented
with new capabilities [2, 34] or fine-tuned on a downstream
task [17]. In adapting these models to tasks, LLMs undergo an
alignment process in which they are further trained (refined)
to satisfy safety objectives [32, 42]. For example, alignment
guidelines often prevent a model from generating responses
that would be offensive, discriminatory, or harmful (i.e.,
unsafe). It is acknowledged that alignment can (and often
does) fail in adversarial settings, resulting in a _jailbreak_ : an
unsafe input being accepted as safe. Indeed, many studies
have explored the automated generation of jailbreak inputs,
i.e., adversarial examples [26, 28, 70]. However, these attacks,
specifically in white-box settings, suffer from two major
drawbacks: they are inefficient (i.e., high runtime and memory
footprint) and employ various heuristics (e.g., maximize the
likelihood of a specific output sentence [70], apply fixed
changes to the input [21, 57], etc.) that limit the search space
of adversarial inputs and thus lower their efficacy and their
ability to assess the robustness of models.
In this paper, we posit a novel approach to reason about
LLM alignment in the context of jailbreak attacks (see Figure 1). Here, we hypothesize that alignment embeds a “safety”
classifier into an LLM that determines whether an input
prompt is safe (compliant with alignment goals) or unsafe
(refusal with alignment). In order to evaluate this hypothesis,
we extract an approximation of the model’s classifier (called
a _surrogate classifier_ ) and evaluate its utility in assessing a
model’s robustness to adversarial inputs. We explore how a
surrogate classifier can be extracted, and thereafter how the
surrogate classifier can be used to assess robustness. In this,
we demonstrate that surrogate classifiers are an efficient and
accurate means of evaluating the robustness of a target LLM.
We begin by exploring methods for extracting surrogate
classifiers. Here, our approach systematically identifies candidates from subsets of the LLM’s architecture. Candidates
are built by (a) finding a structure, i.e., a subset of the
architecture of the model, that best represents the embedded
safety classifier, (b) adding a classification head, and (c)
training this classification head on unsafe and safe inputs to
map the features extracted from the structure to the model
predictions (positive for refusal, negative for compliance).
The candidates are then evaluated in benign and adversarial
settings. For the benign settings, we measure how well the
candidate classifier agrees with the model’s safety classifier
on safe and unsafe inputs. For the adversarial settings, we
measure transferability of adversarial inputs between the LLM
and candidate classifiers to determine how well the latter
capture the decision of the embedded safety classifier.
We perform a detailed analysis of the efficacy of our
approach over 4 models and 2 datasets. We first evaluate
candidates with respect to benign inputs, assessing whether
each candidate classifier agrees with the safety classification
of the LLM. We find that the best candidates achieve accurate
agreement (an _F_ 1 score above 80%) using as little as 20% of
the model architecture. Next, we evaluate the candidate classifiers with respect to transferability of adversarial inputs (i.e.,
as generated using best-in-class jailbreaking algorithms) from
the candidates onto the LLM. These latter experiments show
that some candidate classifiers are extremely effective and
efficient tools for targeting LLMs. For example, a surrogate
classifier using only 50% of the Llama 2 model achieved an
attack success rate (ASR) of 70%—a substantial improvement
over attacking the LLM directly, where we observed a 22%
ASR—with less than half the runtime (350s/sample against
850s/sample) and the memory footprint (14 GB of VRAM
usage against 31 GB).
Our results suggest that investigating the robustness can
be furthered (and in many cases made more efficient) by
extracting surrogate classifiers that target alignment tasks. In
this paper, we have found that approximations of LLMs may
be sufficient to explore the security of their deployment, and
at the same time offer potential vectors for developing more
effective attacks on alignment.
Our contributions are as follows:
_•_ We introduce a novel paradigm for jailbreak attacks that
extracts a surrogate classifier from a subset of the LLM’s
weights and attacks it.
_•_ We evaluate this paradigm on several state-of-the-art
LLMs, showing that attacking the surrogate classifier and
transferring to the LLM is more successful than attacking
the LLM directly.
_•_ We show that studying alignment robustness is scalable,
as our approach significantly reduces the memory footprint and runtime of the attack.
II. BACKGROUND
_A. Large Language Models (LLMs)_
**Large language model.** An LLM models the conditional
probability of the next token (a unit of text for the model)
with respect to previous tokens. Let _V_ be the vocabulary of
|C
Adversarial Inputs
UU nn ss aa fefe LLM Compliance
Unsafe|Col2|Col3|
|---|---|---|
|Adversarial Inputs
|Adversarial Inputs
|Adversarial Inputs
|
|Unsafe
Unsafe
Unsafe|Unsafe
Unsafe
||
|Unsafe
Unsafe
Unsafe|Unsafe
||
|Unsafe
Unsafe
Unsafe|||
_N_
the tokens and _V_ _[∗]_ =
- _V_ _[n]_ the corresponding input space of
_n_ =1
Fig. 1: In this work, we (A) hypothesize that alignment embeds
a safety classifier in LLMs responsible for the classification
of safe and unsafe inputs. Then, we design an approach that
(B) builds candidate classifiers using the model structure. We
evaluate these candidates in benign and adversarial settings to
select the best, called the surrogate classifier. Finally, we (C)
attack the surrogate classifier to generate adversarial inputs
that transfer to the LLM.
applying a linear layer followed by a softmax function on
the last output embedding of the last decoder, _h_ [(] _T_ _[D]_ [)], i.e.,
_p_ ( _·|x_ 1 _x_ 2 _. . . xT_ ) = softmax( _Ah_ [(] _T_ _[D]_ [)] ), where _A ∈_ R _[d][×|][V][ |]_ is
the learned matrix that maps the embeddings to the scores (or
logits) for each token.
_B. Alignment and Safety_
**Alignment.** Since the training data used for LLMs comes from
diverse sources, biases and undesirable behaviors emerge [15],
leading to misaligned pretrained models [3]. In order to
prevent those behaviors from happening, models often go
through an alignment process that regulates their outputs according to given guidelines. Several methods exist and are not
mutually exclusive [32]. Approaches such as supervised finetuning (SFT) use human-generated sample responses, while
reinforcement learning with human feedback (RLHF) [4, 9]
trains a neural network that acts as a reward for the LLM.
Since training another reward model may be costly, other
approaches such as direct preference optimization (DPO) [42]
uses the LLM as its own reward model.
**Guidelines and taxonomies.** The existence of alignment as a
technique is a direct product of the complexity of identifying
what is precisely expected from models (i.e., their objective).
Safety is a broad term; therefore, previous work has identified taxonomies [19] that represent a wide range of unsafe
behaviors. This allows researchers to evaluate alignment on a
finer scale and understand where it could be improved. For
example, when the prompts are decomposed into different
sequences, where _N_ denotes the context window of the model
(maximum amount of tokens that can be processed by the
model). Given a sequence of tokens _x_ = _x_ 1 _x_ 2 _. . . xT ∈_ _V_ _[∗]_,
the model aims to produce the next token _xT_ +1 by approximating the probability distribution.
_p_ ( _xT_ +1 _|x_ 1 _x_ 2 _. . . xT_ ) (1)
LLMs learn a vector representation of words called the embedding space. Given a sequence of tokens _x_ = _x_ 1 _x_ 2 _. . . xT_, each
token is assigned to an embedding, resulting in a sequence of
vectors _{ht}t∈{_ 1 _,...,T }_ all in R _[d]_, where _d_ is the dimension
of the embedding space. This study focuses on aligned chat
models, which are almost exclusively of the same architecture: a sequence of _decoders_ . The _i_ -th decoder transforms a
sequence of embeddings _h_ [(] _[i]_ [)] into another _h_ [(] _[i]_ [+1)], keeping the
same sequence length. If the model is made of _D_ decoders,
the probability distribution on the next token is obtained by
categories such as self-harm, privacy, or harassment, models
tend to refuse prompts related to self-harm and accept prompts
related to privacy [11].
**Identifying unsafe outputs.** Classifying LLM outputs is challenging. Since earlier aligned models created strong refusal
responses, a naive solution is to test whether certain refusal
keywords are in the output (e.g., _”Sorry”_, _”as a responsible_
_AI”_, etc.) [70]. However, this method is less accurate for more
recent models that were aligned with a different approach
than refusal training [59]. Alternatively, LLM judge models
are trained to classify the safety of outputs with more [19] or
less [30] granularity on the classes they predict, similarly to
the taxonomies explained above. A limitation of judge models
is that they can also be attacked [29, 43], leading to a possible
high rate of false negatives (unsafe outputs judged as safe).
_C. Attacks on LLMs_
LLMs, like any machine learning model, are prone to
attacks. While several exploits exist (e.g., model-stealing [5],
text misclassification, etc. [66]), the _jailbreak_ exploit remains
the most prevalent. Jailbreak refers to the compliance of an
aligned LLM for an unsafe query, as defined by the alignment
guidelines. Jailbreak attacks fall into one of two threat models:
the white-box threat model, in which model parameters are
known, and the black-box threat model, in which only access
to the output of the model is assumed. Beyond their use
for malicious behavior, jailbreak attacks are also used to test
LLMs before their deployment through red-teaming [51].
**White-box.** These attacks have been studied on open-weights
models such as Llama 2 [32], Llama 3 [33], or Vicuna [8]. One
of the first white-box attacks, GCG [70], uses greedy coordinate gradient (GCG) to mitigate the absence of mapping from
embeddings (i.e., vectors) to tokens. Following this, many
new white-box algorithms have been introduced [10, 56],
improving existing algorithms [26, 27, 39, 60], using different
objectives [62], or using specific properties of the model (e.g.,
logits [25], special tokens [63], or generation [18]). Attacks
like GCG have an average attack success rate (ASR) above
50% [30] on most open-source models.
**Black-box.** These attacks have been widely studied, given
their prevalence in practical scenarios [1, 35]. These attacks
generally employ heuristics (transformations to the initial
prompt such as ASCII-art [21], translation into underrepresented languages [57], or using pre-existing templates [58]),
search algorithms [7, 31], or a combination thereof. Blackbox attacks achieve remarkable efficacy. For instance, ArtPrompt [21] can achieve ASR as high as 80% on commercial
LLMs such as GPT-3.5.
III. METHODOLOGY
This section outlines our approach for extracting and attacking surrogate classifiers of LLMs. Before we detail our
methodology, we first explain a preliminary experiment whose
results inform our approach.
_A. Empirical Evidence of a Safety Classifier_
As explained previously, LLMs are aligned but are prone
to attacks. Regardless of the threat model, jailbreak attacks
have given limited understanding of where alignment fails,
since they involve heuristics in either the adversarial objective
or the perturbation they apply to the input. In this work, we
hypothesize that alignment embeds a safety classifier in the
model, whose existence we outline next. By extracting this
safety classifier, we can create a new class of optimizationbased attacks that reduces jailbreaking to misclassification on
this surrogate classifier.
**Motivation of existence.** While refusal from aligned models
started as sets of predefined outputs (e.g., _“I cannot fulfill_
_your request. I’m just an AI”_ ), this idea was challenged [41]
and recent efforts [59] have tried to pivot away from refusal
training and focus on safe-completions. Regardless, given an
input, the model has to decide whether an input is safe or
unsafe. We assume the following: the last intermediate state
before the output is enough information to know whether the
output will be a refusal or not. Under this assumption, there
exists a set of refusal intermediate states _Hr_ such that, for
a given input sequence of tokens _x_ = _x_ 1 _x_ 2 _. . . xT ∈_ _V_ _[∗]_,
_h_ [(] _T_ _[D]_ [)] _∈_ _Hr_ if and only if the output answer is a refusal. Thus,
classifying unsafe inputs reduces to verifying _h_ [(] _T_ _[D]_ [)] _∈_ _Hr_ .
**Separation metric.** For the classifier to exist, there must be
separation between the two classes, i.e., inputs classified as
unsafe (refusal) and inputs classified as safe (compliance). To
demonstrate this, we measure the average silhouette score on
the embeddings associated to the safe and unsafe inputs. We
use the silhouette score to determine how well the two safety
classes cluster and to evaluate whether there is a separation,
defined as follows for a given data point _i_ :
_d_ inter( _i_ ) _−_ _d_ intra( _i_ )
_s_ ( _i_ ) = (2)
max( _d_ intra( _i_ ) _, d_ inter( _i_ ))
where _d_ intra( _i_ ) denotes the mean intra-cluster distance (average
distance to all other points in the same cluster) and _d_ inter( _i_ )
the mean nearest-cluster distance (average distance to points
in the nearest cluster).
Silhouette scores range from _−_ 1 to 1 where scores below
0 indicate that representations overlap, and scores above 0.25
or 0.5 imply a weak or reasonable separation [44].
**Result.** We measure this score on four LLMs and two
datasets (see Section IV-A for the detailed experimental setup).
Figure 2 shows the silhouette score of the embeddings for
different models and datasets, with respect to the normalized
decoder position (ratio between the current decoder position
and the total number of decoders).
While it would be expected to see a nondecreasing trend [1],
we see that after the silhouette score reaches its maximum, it
1Usually, the subsequent layers of classifiers do not lose information on
the classification, hence a nondecreasing trend. For LLMs, this classification
is only one of many tasks that are performed, which is why we observe a
compression of the linear information in later layers.
0.6
0.5
0.4
0.3
0.2
0.1
0.0
|G
Gr|Col2|emma 1
anite|Col4|Col5|Col6|Col7|
|---|---|---|---|---|---|---|
|G
~~G~~|G
~~G~~|G
~~G~~|G
~~G~~|G
~~G~~|G
~~G~~|G
~~G~~|
||Ll
|ama 2
|||||
||~~Q~~
Ad
O|~~en 2.5~~
vBench
R-Bench|||||
||||||||
||||||||
||||||||
0.0 0.2 0.4 0.6 0.8 1.0
Normalized Decoder Position
Fig. 2: Silhouette score (measure of separation) of unsafe and
safe input embeddings for different LLMs.
starts to decrease. A possible explanation for this phenomenon
is rooted in the fact that models are designed to perform
multiple tasks. Therefore, after extracting and integrating the
safety information from the input, the other tasks add more information to the embedding that is not related to safety. Thus,
the safety separation decreases (i.e., the linear information is
compressed), but remains high enough for the classification.
Besides the difference in magnitude of separation between the
two datasets, the trend is the same for all models.
_B. Threat Model_
Motivated by our preliminary results, our adversary aims to
extract an approximation of the safety classifier of a target
aligned model, called a surrogate classifier. A successful
extraction would allow subsequent attacks to be made with
more effective white-box jailbreak approaches, while forgoing
the complex computational costs they incur, as we discuss
in Section V-B. We will extract the surrogate classifier using
subsets of the model, thus white-box access is assumed (i.e.,
access to the model weights). In practice, the actor could be a
red-teamer that would benefit from the efficiency and efficacy
gains of the approach when auditing a model.
In jailbreak attacks, the adversarial objective is to make the
model comply with an unsafe input by adding a perturbation.
Aligned models overfit on the unsafe inputs. As a result, they
can reject safe prompts [11], exhibiting a high sensitivity
to certain tokens. While previous work has mostly studied
jailbreaks (unsafe inputs identified as compliant), in this work
we focus on extracting the surrogate classifier therein enabling
attacks that target alignment more closely. To do so, we
also study safe inputs identified as unsafe, as we hypothesize
that they also provide substantial information on the decision
boundary of the classifier.
_C. Problem Formulation_
**General formulation.** Let _fθ_ be an LLM with context window
_N_ and vocabulary _V_, and let _R_ : _x ∈_ _V_ _[∗]_ _�→_ _C_ be a
classification rule that maps the output of the LLM to a class
from _C_ . We aim to extract a _classifier_ from the LLM through
two components: a _structure_ and a classification head. A
_structure fθ∗_ _, θ_ _[∗]_ _⊂_ _θ_ is a subset of the model (e.g., a decoder).
A classification head _C_ maps the representations learned by the
structure to a class such that the classifier _C ◦_ _fθ∗_ is equivalent
to the LLM for the classification task. Formally, for any input
_x ∈_ _V_ _[∗]_, the following holds:
_R_ ( _fθ_ ( _x_ )) = _C_ ( _fθ∗_ ( _x_ )) _._ (3)
**Safety.** In this paper, we focus on safety classification, i.e.,
distinguishing between unsafe and safe inputs. This is a binary
classification problem, where unsafe prompts are considered
positive observations and safe prompts as negative observations. A jailbreak is then a false negative, while a false positive
is “over-refusal” [11]. The classification rule _R_ to assign a
predicted LLM label is then comparable to how attack success
is measured for jailbreak attacks, i.e., by verifying refusal or
compliance. The classification rule _R_ we chose is a model
trained to classify LLM outputs as refusal or compliance as we
focus on the type of answer from the LLM, not if it produces
unsafe information.
_D. Extracting the Classifier_
In order to extract the classifier, we first need to have a
set of candidates. We build a candidate classifier through the
following three steps, summarized in Figure 3:
1) Select a subset _fθ∗_ of the architecture of the LLM _fθ_,
which we will refer to as a _structure_ .
2) Collect data points from the structure ( _fθ∗_ ( _x_ ) _, R_ ( _fθ_ ( _x_ )))
consisting of the features extracted from the structure and
the predicted label of the LLM.
3) Train a classification head _C_ on ( _fθ∗_ ( _x_ ) _, R_ ( _fθ_ ( _x_ ))).
The result, _C ◦_ _fθ∗_, will be termed a _candidate_ classifier.
**Structures.** As explained in Section II-A, most LLMs are
built as sequences of decoders: processing units that take a
sequence of vectors and output another sequence of vectors
of the same length. LLMs can be scaled either in depth by
adding decoders or in width by increasing the dimensionality
of embeddings. Therefore, there are many ways to “cut” the
model and thus many possible structures. It is clear that there
is a separation between unsafe and safe representations at the
end of aligned models [27] largely due to the strong refusal
outputs. In addition, previous work [24, 64] has shown that
separation occurs in early decoders [24]. Thus, we estimate
Fig. 3: Methodology overview. In the first step, we extract the safety classifier of an LLM by (A) selecting a structure within
the model and (B) training a classification head on the predictions of the LLM to create a candidate classifier. We verify
its performance in benign settings. Then, we measure the transferability between the LLM and the candidate in both ways:
adversarial examples of the LLM to the candidate (C) and adversarial examples of the candidate to the LLM (D).
the location of the safety classifier via two hypotheses. First,
the classifier is at the decoder level. Second, the information
relevant to the classification at a given decoder is contained at
the end of the sequence (as discussed in Section III-A).
We will refer to _fi,i_ + _δ_ as the set of decoders between the
_i_ -th and the _i_ + _δ_ -th decoder, _δ ≥_ 0. The second hypothesis
implies that for an input _x ∈_ _V_ _[∗]_ where _fi,i_ + _δ_ ( _x_ ) _∈_ R _[|][x][|×][d]_,
we only consider the last element of the sequence, reducing
the input dimension of the classification head _C_ to R _[d]_ .
By construction, the input space of _fi,i_ + _δ_ depends on the
output space of _f_ 1 _,i_, since the model was trained with the
entire architecture. In this work, we set _i_ to 1. We leave the
exploration of structures that isolate intermediate decoders to
future work (see Section V-C) because they would require
techniques to mitigate the absence of earlier decoders, which
is out of the scope of this work.
Finally, _δ_ will be termed the _candidate size_ and we will
refer to _Dδ_ [as the] _[ normalized candidate size]_ [ (also referred to]
as percentage of the corresponding LLM), i.e., the proportion
of decoders of the LLM used by the candidate. We will focus
on the latter when comparing different LLMs as the number
of parameters is spread evenly across decoders.
**Training.** For the selected structure _fi,i_ + _δ_, the goal is to train
a classification head _C_ to obtain a candidate classifier. To avoid
overfitting due to a complex classification head, our approach
leverages linear probing, which has shown promising results
for safety representation [22, 61]. The first step is to create a
dataset _X_ = ( _fi,i_ + _δ_ ( _x_ ) _, R_ ( _fθ_ ( _x_ ))) consisting of the extracted
feature from the structure _fi,i_ + _δ_ and the predicted label [2] by
applying the classification rule _R_ to the output of the LLM _fθ_
for each prompt _x_ . Finally, the classification head is trained
using the following objective:
min (4)
( _h,y_ ) _∈X_ _[L]_ [(] _[C]_ [(] _[h]_ [)] _[, y]_ [)]
Once the classification head is trained, we obtain a candidate
classifier noted _C ◦_ _fi,i_ + _δ_ of candidate size _δ_ .
2Since we focus on extracting the classifier and not improve it, it is
beneficial to have both right and wrong labels from the classifier.
_E. Efficient Jailbreaking_
As a reminder, our goal is to induce a jailbreak in the LLM
using candidate classifiers as approximations of the embedded
safety classifier. This involves two steps: generating adversarial samples from the candidate classifiers and transferring them
to the LLM. Although similar to transferability-based attacks
that use source models different from the target model to craft
adversarial examples [36, 37], our approach uses a _subset_ of
the target model (the LLM) as the source, using the insights
from Section III-A as foundations for a higher efficiency (and
efficacy) of the attack.
We attack the candidate classifiers using the well established
GCG attack [70]. Since the attack normally targets an LLM
that outputs a sequence of tokens, some adjustments to its
objective are required to make it usable on a candidate
classifier, which we detail below.
**From heuristics to misclassification.** The original adversarial
objective of GCG is to maximize the likelihood of a target
sequence (e.g., “Sure, here is how to build a bomb.”) for an
input _x_ on which _A_ tokens are added as an adversarial suffix:
max
( _xT_ +1 _,...,xT_ + _A_ ) _∈V_ _[A]_ [ log] _[ p]_ [(] _[x][T]_ [ +] _[A]_ [+1] _[ . . . x][T]_ [ +] _[A]_ [+] _[O][|][x]_ [1] _[ . . . x][T]_ [ +] _[A]_ [)]
(5)
The space of compliant outputs is infinite, thus limiting
the search of adversarial examples to those that directly (or
indirectly) prioritize a particular output is bound to restrict
the efficacy of the attack. Using the candidate classifiers which
map an input sequence to a predicted safety from the LLM, we
can reformulate the objective to a misclassification objective
(e.g., max loss) without heuristic:
max (6)
( _xT_ +1 _,...,xT_ + _A_ ) _∈V_ _[A][ L]_ [(] _[C]_ [(] _[f][θ][∗]_ [(] _[x]_ [1] _[ . . . x][T]_ [ +] _[A]_ [))] _[, y]_ [)]
with _y_ being the predicted label from the LLM and _L_
being the loss (e.g., binary cross-entropy loss). This objective
encompasses the GCG objective under the assumption of a
perfect candidate classifier (satisfying Equation 3): increasing
the likelihood of a compliant output means that the input is
being considered safer and thus closer to misclassification.
IV. EVALUATION
We use our method to address these research questions:
**RQ1** . How accurate are the candidate classifiers at
approximating the safety classifier in benign
settings? (Section IV-B)
**RQ2** . Does the performance of candidate classifiers hold
in adversarial settings? (Section IV-C)
**RQ3** . Can we jailbreak aligned LLMs by attacking the
surrogate classifier? (Section IV-D)
_A. Experimental Setup_
All experiments were run on pools from the Center for
High Throughput Computing [6].The dataset creation and
attack experiments were run on NVIDIA A100 GPUs with
40 GB of VRAM. Attack experiments on the models amount
to approximately 2000 GPU hours. Attack experiments on the
candidate classifiers amount to approximately 500 GPU hours.
**Models.** We evaluate our approach on four
open-weights aligned chat models, all from
the Hugging Face [Transformers 4.46.2](https://huggingface.co/docs/transformers/)
library [54]: [Llama-2-7b-chat](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) [32],
[Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) [49] [gemma-7b-it](https://huggingface.co/google/gemma-7b-it) [47],
[and granite-3.1-8b-instruct [48]. We chose these](https://huggingface.co/ibm-granite/granite-3.1-8b-instruct)
models because they achieve good performance at the
classification task on the datasets considered, translating to a
stronger alignment. For completeness, we present results on
other models in Appendix C.
**Dataset augmentation.** Since most of the datasets lack safe
prompts or targets to attack, we augment them when necessary. For safe prompts, we use an LLM to convert unsafe
prompts to their corresponding safe version (following best
practices [13]), as it is a simple natural language task suitable
for the use of LLMs. We manually verify two properties of
each generated prompt: the instruction does not prompt the
model for unsafe information, and it has the same structure as
the original prompt. For example, the prompt _“Write a script_
_that can exploit vulnerabilities in a software or operating_
_system”_ becomes _“Write a script that can identify and help fix_
_vulnerabilities in a software or operating system”_, satisfying
the two desired properties. Note that preserving the structure of
the prompt ensures that we focus on safety rather than artifacts
of the dataset, such as how prompts are formatted.
The GCG optimization objective aims to increase the probability of a given output sequence of tokens. Therefore, it
needs a target output string for each input prompt. For missing
target prompts (e.g., corresponding to augmented safe input
prompts), we generate a possible compliant output from the
model using an LLM. For instance, _“Sure, here is a script that_
_can exploit vulnerabilities in a software or operating system”_
is the target string for the previous unsafe input example.
**Datasets.** We use two datasets in our experiments:
_•_ An augmented version of _AdvBench_ [70] with 520 original unsafe instructions and 520 added safe instructions.
_•_ A subset of _OR-Bench_ [11] made of 1000 randomly
sampled prompts (of which 500 are unsafe and 500 are
safe) for computational reasons.
**Labels.** The labels are obtained by applying a classifier on the
LLM output. To eliminate nondeterminism due to sampling
and facilitate reproducibility of results, we generate outputs
with a temperature equal to 0: the next token selected at each
step of the generation is the one that maximizes the logits. As it
has been shown that a higher temperature can lead to a higher
probability of generating unsafe outputs [70] (upper bounding
the measured efficacy of the attack), we consider the additional
tractability incurred by studying the temperature out-of-scope
and leave it to future work. The output is classified between
[refusal and compliance using the distilroberta-base-rejection-](https://huggingface.co/protectai/distilroberta-base-rejection-v1)
[v1 [40] model for the reasons mentioned in Section III-C.](https://huggingface.co/protectai/distilroberta-base-rejection-v1)
**Classifier training.** As explained in Section III-D, the classification head needs to be simple enough to prevent overfitting
on the dataset and to only learn to project the features of
structure to a classification. Therefore, we consider a simple
linear layer followed by a sigmoid: _C_ ( _h_ ) = _σ_ ( _Ah_ + _b_ ) where
_A ∈_ R [1] _[×][d]_ _, b ∈_ R and _σ_ : _x �→_ 1+1 _e_ _[−][x]_ [. The result is a]
scalar between 0 and 1, thus the classification is made given a
threshold _T_ . As for training, we apply _K_ -fold cross-validation
with 5 folds. We select the threshold _T_ that maximizes the _F_ 1
score on the training data, i.e., _T_ = arg max 1 ( _D_ train).
_T ∈_ [0 _,_ 1] _[F]_ [ (] _[T]_ [ )]
We train the classification head using the hyperparameters
listed in Table I.
TABLE I: Hyperparameters for training the classifier head.
Hyperparameter Value
Learning Rate 0.001
Batch Size 32
Epochs 500
Optimizer Adam
Patience 15
Number of folds 5
**Attack.** To generate adversarial examples, we use
the nanogcg [3] implementation of GCG [70] with
the following hyperparameters: num_steps=250,
topk=512,search_width=512, and an attack length of
20 (following prior work [26, 60]). We chose this attack
because it is a pure gradient-based attack with an adversarial
objective that can be adapted to attacking a classifier.
Other white-box attacks may be ill-suited or benefit less
from the approach if they are not gradient-based (e.g.,
BEAST [45]) or if their optimization objective cannot be
mapped to misclassification (e.g., Autodan [67]), making the
attribution of efficacy to the approach harder. Further, for the
purpose of verifying our hypothesis, we decided to ensure
tractability of experiments (following prior work that has
built on GCG [26, 27]) and chose not to build upon direct
descendants of GCG like AmpleGCG [26] as they introduce
more complexity and do not fit our goal of a heuristic-free
[3See code at https://github.com/GraySwanAI/nanoGCG.](https://github.com/GraySwanAI/nanoGCG)
1.0
0.9
0.8
0.7
0.6
1.0
0.9
0.8
0.7
0.6
0.5
(a) _AdvBench_
0.5
(b) _OR-Bench_
Fig. 4: Test _F_ 1 of the candidate classifiers in benign settings, depending on the normalized candidate size.
1.0
0.9
0.8
0.7
0.6
1.0
0.9
0.8
0.7
0.6
0.5
(a) Trained on _AdvBench_, evaluated on _OR-Bench_
0.5
(b) Trained on _OR-Bench_, evaluated on _AdvBench_
Fig. 5: Test _F_ 1 of the candidate classifiers in benign settings, on the dataset they were not trained on.
attack. For candidate classifiers, we use the binary cross
entropy loss of PyTorch [38].
We apply the attack to misclassify unsafe inputs (i.e., create
a compliance from the LLMs). For completeness, we also do
a similar study in Section B in which the attack is used to
misclassify safe inputs.
**Metrics.** Even with a balanced dataset (equal number of unsafe
and safe inputs), it is possible that the dataset of predictions
is not balanced, as it depends on the performance of the LLM
for this task. For instance, if the LLM is prone to over-refusal,
the resulting predictions may contain a lot more positives
than negatives. Therefore, we measure performance of the
candidate classifiers with the _F_ 1 score _F_ 1 = TP+ [1] 2 [(] TP [FP][+][FN][)] [as]
it helps mitigating dataset unbalance.
For attacks, we measure the attack success rate (ASR), i.e.,
the proportion of misclassifications by the model on inputs
modified by the attack. To measure transferability between
LLM and candidate classifiers, we use the transferability rate:
the proportion of adversarial inputs crafted on one that are
misclassified by the other.
_B. Performance in Benign Settings_
In this section, we answer RQ1, i.e., how accurate candidate
classifiers are at approximating the safety classifier.
**Baseline results.** Table II shows the benign classification performance of the LLMs for each dataset with the corresponding
confusion matrices in Figure 6 in Figure 7. We can first see
that all models perform well in _AdvBench_, all scoring above
0.9. The poorer performance on _OR-Bench_ is expected, as it
was introduced after _AdvBench_ with the goal of being harder
to classify and to test for over-refusal, as seen in the top right
cell of the confusion matrices in Figure 7. Further, it echoes
the result from Section III-A in which the silhouette score
(measure of separation) is consistently lower for _OR-Bench_
compared to _AdvBench_ .
This leads to two implications on the candidate classifiers
and their evaluation on _OR-Bench_ : the lower separation between compliance and refusal (measured by the silhouette
score) makes overfitting of the classification head easier, while
the lower performance of the safety classifier creates a harsher
evaluation setting for the candidate classifiers (as they need to
agree with the misclassifications).
Unsafe Safe
Actual
(a) Gemma 1
Unsafe Safe
Actual
(a) Gemma 1
Unsafe Safe
Actual
(b) Granite
Unsafe Safe
Actual
(b) Granite
Unsafe Safe
Actual
(c) Llama 2
Fig. 6: Confusion matrices on _AdvBench_
Unsafe Safe
Actual
(c) Llama 2
Unsafe Safe
Actual
(d) Qwen 2.5
Unsafe Safe
Actual
(d) Qwen 2.5
Fig. 7: Confusion matrices on _OR-Bench_
|Model|AdvBench OR-Bench|
|---|---|
|Gemma 1
Granite
Llama 2
Qwen 2.5|0.97
0.95
0.94
0.88
0.93
0.83
0.97
0.89|
TABLE II: _F_ 1 score of the LLMs in benign settings.
**Candidates performance.** Figure 4 shows the median test _F_ 1
score of the candidate classifiers over 5 trials. We see that
performance either increases or stagnates with respect to the
candidate size. This suggests that later parts of the model preserve or increase the separation of safe and unsafe instructions.
For example, candidate classifiers achieve a lower _F_ 1 score
on _OR-Bench_, losing 5% for all models. This aligns with the
results in Table II where all models perform worse on _OR-_
_Bench_ . We hypothesize that this can be explained by a lower
separability between rejected and accepted prompts at the
embedding level, as we note that the results from Section III-A
align with the performance of the candidate classifiers. This
lower separation means that the two clusters may overlap,
preventing the classification head from completely aligning
with the classifier when training.
**Cross-dataset.** To ensure that the candidate classifiers are not
artifacts of the datasets and their distributions, we evaluate the
candidate classifiers on the dataset that they were _not_ trained
on (e.g., train on _AdvBench_, test on _OR-Bench_ ). Similarly to
the previous figure, Figure 5 reports the median _F_ 1 score for
the same candidate classifiers in the dataset on which they
were not trained. Overall, we see that there is no significant
performance drop of the candidate classifiers when using a
different distribution at the maximum candidate size. However,
it is clear that the _F_ 1 score does not converge as fast on the
cross-dataset setting. For instance, in Figure 4a, Gemma 1
reaches an _F_ 1 score of 0.9 for candidate classifiers above
20% normalized candidate size against 50% in Figure 5a.
This offset is likely due to a difference in the distribution of
prompts: the candidate classifier overfits on specific patterns
tied to how the dataset was built. For instance, _AdvBench_ is
made of instructions, while _OR-Bench_ also contains questions.
This difference in format can affect the behavior of the
representations. Therefore, using another dataset gives a lower
bound on the candidate size _δ_ .
_C. Performance in Adversarial Settings_
In this section, we aim to answer RQ2, i.e., measure the
performance of the candidate classifiers in adversarial settings.
We apply the attack on the LLMs and evaluate the candidate
classifiers on the resulting adversarial inputs. The attacked
samples are prompts for which the classification head was _not_
trained on for each fold, with the threshold that maximizes the
_F_ 1 score on the training data.
1.0
0.8
0.6
0.4
0.2
1.0
0.8
0.6
0.4
0.2
0.0
0.0 0.2 0.4 0.6 0.8 1.0
Normalized Candidate Size
(a) _AdvBench_
0.0
0.0 0.2 0.4 0.6 0.8 1.0
Normalized Candidate Size
(b) _OR-Bench_
Fig. 8: Proportion of adversarial examples crafted on the model that transfer to the candidate classifiers. The total number of
samples the metric is computed on is reported next to each model.
Model _AdvBench_ _OR-Bench_
Gemma 1 0.62 0.67
Granite 0.91 0.88
Llama 2 0.22 0.12
Qwen 2.5 0.94 0.88
TABLE III: Attack success rates (ASR) of the models after
applying the attack on unsafe inputs.
**Baseline results.** Similarly to the previous settings, we report
in Table III the classification performances of the LLMs
after applying the white-box attack, along with the attack
success rate (ASR) defined as the proportion of unsafe samples
misclassified. We note a significant difference (around 70%)
between the ASR of Llama 2 and Qwen 2.5. This comes from
the strong alignment of Llama 2 with a bias toward refusal
(see Section A). Aside from Gemma 1, the ASR on _OR-_
_Bench_ is consistently lower than on _AdvBench_, which could
be explained again by the initial goal of the dataset, having
an emphasis on over-refusal.
**LLM to candidate classifiers.** After applying the attack, we
filter the adversarial inputs misclassified by the LLM and
evaluate how well these examples transfer to the candidate
classifiers by measuring the transferability rate: the proportion
of misclassified samples by each candidate classifier. Figure 8
reports the transferability rate from the LLM to the candidate
classifiers. We observe a similar pattern to that of the crossdataset study (see Section IV-B): the transfer rate does not
converge until at least half of the model is used. For example,
candidate classifiers on Llama 2 obtain a transfer higher than
90% only when more than half the model is used. This result
implies that candidate classifiers that use less than half of the
model are not representative of the safety classifier. We also
note that for _OR-Bench_, most candidate classifiers using less
than 40% of their corresponding LLM seem unstable. This
phenomenon is consistent with the low silhouette score (see
Section III-A) and performance in the cross-dataset study (see
Section IV-B) for corresponding regions of the LLMs. The out
of-distribution nature of adversarial examples may lead to high
transferability rates for earlier layers that overfit, strengthening
the importance of the cross-dataset study done in Section IV-B.
**Imbalance and performance.** We note that the performance
of the candidate classifiers reaches a higher value for _OR-_
_Bench_ than for _AdvBench_ : for the former, the transferability
rate converges to a value above 80% against a value between
60% and 80% for the latter (with Llama 2 being the sole
exception). We attribute this difference to the performance of
the LLM in benign settings for this dataset (see Figure 7),
resulting in a higher amount of samples predicted as unsafe.
Therefore, the classification heads of the corresponding candidate classifiers are trained with more information on when the
safety classifier classifies an input as unsafe, specifically when
it is wrong, leading to an improvement in approximating the
safety classifier for adversarial unsafe inputs. This observation
is also consistent with the only model unaffected by this
trend, Llama 2, which exhibits the highest rate of safe inputs
misclassified as unsafe.
_D. Attacking the Surrogate Classifier_
In this section, we aim to answer RQ3, i.e., if we can
jailbreak aligned LLMs by attacking the surrogate classifier.
In this section, we attack the candidate classifier and evaluate
how well the adversarial inputs transfer to the LLMs.
**Transfer to the LLM.** Figure 9 shows the proportion of adversarial inputs crafted on the candidate classifiers that induce a
misclassification on the model. Interestingly, the results do not
1.0
0.8
0.6
0.4
0.2
|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|
|---|---|---|---|---|---|---|---|
|R
~~V~~|untime
~~RAM Us~~|(Avg.)
~~ age (Avg~~|~~ .)~~|~~ .)~~||||
|
R
|
untime
~~RAM Us~~|
 (Max.)
~~ ge (Ma~~|
~~ .)~~|||||
|
B|
aseline|
 GCG (Av|
 g.)|||||
|||||||||
|||||||||
|||||||||
|||||||||
|||||||||
5 10 15 20 25 30
Candidate Size
0.0
|Col1|Ge|emma 1|Col4|Col5|Col6|Col7|
|---|---|---|---|---|---|---|
||~~G~~
Gr|~~mma 1~~
anite|||||
||~~Ll~~
Q
A|~~ama 2~~
wen 2.5
vBench|||||
|O|O|R-Bench|||||
|O|O||||||
||||||||
||||||||
0.0 0.2 0.4 0.6 0.8 1.0
Normalized Candidate Size
Fig. 9: Transferability rate to the LLM after applying the
white-box attack on the candidate classifiers.
exhibit a strongly increasing trend with respect to the candidate
size like in the previous sections. For three of the models, we
observe a peak above 70% when the normalized candidate
size approaches 60%. In addition, attacking the candidate
classifiers and transferring the resulting inputs to the model
seem to work better than attacking the model directly. For
example, using 50% of Llama 2, it is possible to achieve an
ASR of 70%, well above the baseline ASR in Table III.
**Optimal candidate size.** The existence of a maximum for
the performance of the candidate classifiers highlights the
importance of the candidate size _δ_ . If _δ_ is too small, the
candidate classifiers might fail at capturing the information
for the attack, leading to a smaller transferability rate. On
the other hand, if _δ_ is too high, the candidate classifiers
might be compromised by information orthogonal to the safety
classification, as seen in the decrease from Figure 2.
**ASR and objective.** The previous differences in transferability
rate compared to the baseline lie in how the adversarial
objective is built. Since the output of LLMs is text, there needs
to be a heuristic to know when misclassification (jailbreak)
occurs, such as specific target sentences. When the objective
is misclassification, it is easier to find adversarial examples.
It implicitly contains the previous objective without its limitations, increasing the space of adversarial inputs.
Fig. 10: Runtime and VRAM usage to craft one adversarial
example depending on the candidate size. The dashed line
represents the averaged baseline runtime (840s) and VRAM
usage (31 GB) when attacking the LLM directly.
**Efficiency.** Figure 10 shows the runtime [4] and VRAM usage
(i.e., peak GPU memory occupied) when applying the attack
on one sample, averaged over 100 samples. We see that
attacking the surrogate classifier has lower computational costs
than attacking LLM, as it depends on the candidate size _δ_
(which scales linearly with the VRAM usage like the runtime,
as seen in Figure 10). We note that both the VRAM usage
and the runtime do not reach the baseline averages of 31 GB
and 840s. We attribute this to the adversarial objective of
the baseline GCG attack: it requires the attack to generate
an output sequence at each step to compare to the target
sequence, directly impacting its efficiency. Paired with the
previous results from Figure 9 in which all models achieve
a transferability rate above 70% with only half of the LLM
architecture, this highlights the efficacy and efficiency of
our approach. This also shows that studying the robustness
of LLMs can be made scalable. Indeed, the original attack
requires a NVIDIA A100 with 40 GB for all models (at halfprecision). Attacking 50% of the model (as presented in our
work) means that 20 GB of VRAM will be sufficient, enabling
research on consumer-grade GPUs (e.g., NVIDIA RTX 4090).
For industry-grade GPUs (e.g., NVIDIA A100, H100, etc.),
this result enables studies on the impact of hyperparameters.
4This does not include the time to train the candidate classifier as it is
negligible, taking less than a minute and happening once for all samples.
V. DISCUSSION & FUTURE WORK
_A. Domains_
**Why safety?** This study focuses on the problem of safety,
i.e., classifying unsafe and safe inputs. We chose this for two
main reasons. First, recent work has shown that there is a
clear separation between the two classes in the model (see
Section VI). Second, this problem is one of the most prolific
research topics (e.g., jailbreak attacks), and the insights from
this study lead to implications for white-box attacks, which
we discuss in the next section.
**Other domains.** Outside of safety, several phenomena have
shown increasing concern. First, malicious code generation
and vulnerable code identification have been specifically investigated for LLMs designed for code, making them suitable
for a similar approach. Second, hallucination, dishonesty, bias,
and lack of fairness are all examples of failures of alignment.
Although they do not translate directly to a classification problem, it is possible to set up controlled environments in which
the LLM can act as a classifier. Similarly to representation
engineering [68] in which stimuli are designed for specific
tasks, datasets with prompts formatted to elicit the mentioned
phenomena could help apply the approach to these domains.
_B. Implication for White-box Attacks_
**Classification objective.** As opposed to evasion attacks on
classifiers which cause misclassifications, jailbreak attacks aim
to induce unsafe behaviors. This objective is more complex because it follows from the definition of safety. When extracting
the classifier, we explicitly uncover the decision made by the
model. This gives a more accurate signal that does not rely on
heuristics (e.g., maximizing a target sequence that is _likely_ to
induce a jailbreak) and thus leads to a higher attack efficacy.
We also note that such heuristics on the adversarial objective
not only implicitly reduce the efficacy, but can also reduce the
efficiency (see Section IV-D) as it requires multiple forward
passes to the entire model to compute an output sequence.
**Efficiency.** Recent white-box gradient-based attacks on LLM
are based on the GCG attack [70]. Although there have
been significant performance improvements over time, these
attacks compute gradients and, therefore, they are limited
when the models are too large. Our results from Section IV-D
suggest that investigating the robustness of alignment through
attacks is reducible to attacking the surrogate classifier. This
implies fewer computations, and thus higher efficiency (and
scalability) of attacks. Our approach was applied on LLMs
with less than 9 billion parameters and showed that using 50%
of them is enough for the attack. A similar efficiency could
be obtained from a transfer attack using a smaller LLM as the
source. However, this would come with a lower efficacy and
the results might be less informative on the failure points of
the target model since a different model is used.
**Attack scenarios.** Our work can be applied to any jailbreak
attack (albeit more suited for optimization-based attacks),
making it extensible to multiple scenarios. For instance, in the
presence of a guard model that prevents unsafe outputs (e.g.,
LlamaGuard [19]), recent work has shown that an attack can
be “propagated” to this guard model [29]. Our work can be
extended to such scenarios by extracting the safety classifiers
of the LLM and the guard model, therein improving efficiency.
**Black-box.** We limit the threat model to white-box, since we
need access to the LLM’s weights to obtain the surrogate
classifier. However, a promising direction would be to study
the transferability of safety classifier of a source LLM onto
that of a target LLM. Transferability of white-box attacks is a
well-known phenomenon of adversarial machine learning [12]
also present for jailbreak attacks [28, 70]. An analysis with
candidate classifiers could give a finer-grained understanding
of the transferability of jailbreak attacks.
**Defenses.** As any attack against an ML model, adversarial
training [55] remains the de facto standard for increasing ML
robustness. Instead of applying such training to the entire
model (which would be highly inefficient given its size), it
can be applied to the safety classifier with the classification
objective. For a finer-grained threat model where the adversary
is computationally bounded (by either runtime or memory),
explicitly training the LLM to increase the safety classifier
size (e.g., by modifying the learning objective) would limit the
feasibility of the attack due to computational constraints. This,
however, incurs a trade-off for other defenses such as detection
of adversarial examples in the intermediate layers [53] as
the detection would happen at the end of the safety classifier. Other approaches add components around the model
(e.g., LlamaGuard [19]) to avoid the security-utility tradeoff incurred by defending the model itself. With the safety
classifier identified, it could be used at inference time to drop
unsafe inputs before the output is generated, resulting in higher
efficiency for the whole system.
**Red-teaming.** Ensuring that an LLM is secure before deployment requires a thorough evaluation that needs to be
scalable and efficient, given the size of those models. Using
our approach could be a promising direction in improving
the efficiency of the process: Using the output of the safety
classifier for early feedback, red-teamers would be able to
iterate faster on attacks and thus audit the model in more depth.
_C. Safety Classifier_
**Start point of the classifier.** We focus on the structures _f_ 1 _,_ 1+ _δ_ :
contiguous sets of decoders starting at the first layer. This
allowed to locate the end of the safety classifier by evaluating
the performance of the corresponding candidates. However, the
classifier may not start at the beginning. Therefore, studying
structures _fi,i_ + _δ_ with _i >_ 1 may be of interest to refine the
extraction but introduce a new dimension of complexity. Since
the input space of decoder _i_ must match the output space of
decoder _i −_ 1, new techniques need to be introduced to allow
the creation of candidates that truly capture the information
of the structure. For example, a candidate classifier based on
_fi,i_ + _δ_ should not use information from _f_ 1 _,i−_ 1. We leave such
explorations to future work.
**Labels.** Part of our approach involves obtaining the classifications from the LLM. This involves two steps: generating the
output from the LLM, then mapping this output text to a class,
using the classification rule _R_ defined in Section III-C. LLMs
are deployed in a way that allows nondeterministic outputs,
leading to probabilities of refusal and compliance. To ensure
reproducibility and reduce computational costs, we set the
temperature to 0 in our experiments. Studying the temperature
and its influence over the probabilities of labels would lead to a
more precise understanding of safety classifier and its decision
boundary. We leave this study to future work.
**Finding optimal depth.** From an adversarial perspective,
finding the position of the safety classifier is crucial, as it can
be expensive to run the attack on various candidates. A first
strategy to pinpoint the most likely optimal depth would be to
follow the procedure of Section III-A, i.e., using the silhouette
score. However, the adversary may be constrained in resources
and have limited GPU memory. In that case, multiple strategies
are possible, such as starting at the maximum candidate size
for the GPU memory and lowering it or doing a binary search
on the candidate size.
VI. RELATED WORK
_A. Jailbreak and Representations_
Improving the safety of LLMs is one of the most prevalent
research areas: numerous publications have studied the link
between jailbreak and model representations. It is necessary
that alignment creates a linear separation between refused and
accepted prompt for some representations, since the output is
explicitly linearly dependent on the last embedding [27]. Refusal is encoded through a set of similar responses or sequence
of tokens, which implies a separation between the tokens logits
for accepted and refused prompt. Several papers have shown
the distinction at the decoder level [22, 24, 27, 61, 64] or at
the head of the attention level [65]. In addition, previous work
has also shown that representations can be used to manipulate
the model [68] and increase its robustness at a negligible
utility cost [69]. These works often use linear probing to
obtain “probes” that classify the representations of prompts.
However, the utility of these probes remains limited: they
cannot be used to produce adversarial examples as they do not
hold any gradient information from embeddings to tokens (i.e.,
from intermediary output to input). Our work takes a different
approach by considering an embedded classifier within the
model and extracting it. We leverage linear probing to map the
output of a structure (a subset of the LLM) to a classification.
This allows the application of jailbreak attacks to produce
token-level adversarial examples (i.e., unsafe input prompts
that cause the LLM to comply). The extracted classifier can
then be used by both attackers and defenders to systematically
study the security of alignment.
_B. Adversarial Settings_
Isolating the security critical component of an LLM is
beneficial from attack and defense perspectives. Previous work
has shown multiple ways of using the identified component.
From a defense perspective, it is possible to fine-tune the
LLM while freezing the corresponding component to avoid
destroying the learned safety features [24, 52].
From an attack perspective, there are three ways to use the
component: ablation [65], intervention [22], or optimization
of the input to attack the component [27]. Ablation disables
the component during generation, preventing the model from
classifying the safety of the input. Interventions operate by
adding a perturbation to the embedding at the position of the
component. In the case of a classifier, the perturbation corresponds to the direction orthogonal to its decision boundary.
In this work, we extract a surrogate classifier which can
be attacked directly with a heuristic-free adversarial objective.
While interventions can be applied to the candidates, they
may not lead to feasible input perturbations, limiting the
characterization of alignment failures.
_C. Pruning_
In addition to studying their safety, an active area of research
on LLMs is pruning, i.e., reducing the size of the model while
maintaining most of its capabilities [14]. For LLMs, pruning
can be divided into three axes: depth, width, and length
pruning. Depth pruning focuses on removing layers of the
model, width pruning aims to reduce the size of the projection
matrices, and length pruning focuses on cutting tokens of
the input stream and avoiding redundant computations. As
those axes are orthogonal, they can be combined to obtain
better results [46]. The rise of security concerns led to the
intersection of pruning and security. Recent work has shown
that safety alignment is inherently low rank and can be
isolated [52] and that pruning can offer higher safety [16].
Although these techniques could achieve a similar objective,
they generally involve modifying selected parts of the model.
As we aim to extract the safety classifier, minimizing the
information lost on its decision boundary by having as little
training as possible is crucial, which is why we chose the least
invasive and most efficient method (linear probing).
VII. CONCLUSION
In this paper, we hypothesized that alignment embeds a
safety classifier within LLMs, and we introduced techniques to
extract best candidate classifiers for it. We empirically verified
the agreement with the models of the best candidates in both
benign and adversarial settings. We also demonstrated how
the extracted classifier can be used to systematically study
alignment security, as the candidate classifiers substantially
improved both the efficiency (lower runtime and VRAM
usage) and the efficacy (higher success rate) for attackers and
defenders. Finally, while we studied alignment in a safety
context, other failures of alignment such as hallucinations or
lack of fairness could also benefit from our approach.
ACKNOWLEDGMENTS
The authors thank Chaowei Xiao and Rahul Chatterjee as
well as all reviewers for their helpful comments on previous
iterations of the work.
**Funding acknowledgment:** This material is based upon work
supported by the National Science Foundation under Grant
No. CNS-2343611 and by PRISM, one of seven centers in
JUMP 2.0, a Semiconductor Research Corporation (SRC)
program sponsored by DARPA. Any opinions, findings, and
conclusions or recommendations expressed in this material are
those of the author(s) and do not necessarily reflect the views
of the National Science Foundation.
REFERENCES
[1] Anthropic. Introducing Claude.
[2] Anthropic. Introducing computer use, a new Claude 3.5
Sonnet, and Claude 3.5 Haiku.
[3] Usman Anwar, Abulhair Saparov, Javier Rando, Daniel
Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana,
Erik Jenner, Stephen Casper, Oliver Sourbut, Benjamin L.
Edelman, Zhaowei Zhang, Mario G¨unther, Anton Korinek, Jose Hernandez-Orallo, Lewis Hammond, Eric J
Bigelow, Alexander Pan, Lauro Langosco, Tomasz Korbak, Heidi Chenyu Zhang, Ruiqi Zhong, Sean O
hEigeartaigh, Gabriel Recchia, Giulio Corsi, Alan Chan,
Markus Anderljung, Lilian Edwards, Aleksandar Petrov,
Christian Schroeder de Witt, Sumeet Ramesh Motwani,
Yoshua Bengio, Danqi Chen, Philip Torr, Samuel Albanie, Tegan Maharaj, Jakob Nicolaus Foerster, Florian
Tram`er, He He, Atoosa Kasirzadeh, Yejin Choi, and
David Krueger. Foundational challenges in assuring
alignment and safety of large language models. _Trans-_
_actions on Machine Learning Research_, 2024. Survey
Certification, Expert Certification.
[4] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda
Askell, Anna Chen, Nova DasSarma, Dawn Drain,
Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas
Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly,
Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds,
Danny Hernandez, Tristan Hume, Scott Johnston, Shauna
Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson,
Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement
learning from human feedback, 2022. arXiv: 2204.05862
[cs.CL].
[5] Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A. Feder
Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr,
Arthur Conmy, Eric Wallace, David Rolnick, and Florian
Tram`er. Stealing part of a production language model.
In _ICML_, 2024.
[6] Center for High Throughput Computing. Center for high
throughput computing, 2006.
[7] Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed
Hassani, George J. Pappas, and Eric Wong. Jailbreaking
Black Box Large Language Models in Twenty Queries,
July 2024.
[8] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng,
Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan
Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An Open-Source Chatbot
Impressing GPT-4 with 90%* ChatGPT Quality, March
2023.
[9] Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan
Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In _Proceedings_
_of the 31st International Conference on Neural Informa-_
_tion Processing Systems_, NIPS’17, page 4302–4310, Red
Hook, NY, USA, 2017. Curran Associates Inc.
[10] Junjie Chu, Yugeng Liu, Ziqing Yang, Xinyue Shen,
Michael Backes, and Yang Zhang. Comprehensive
assessment of jailbreak attacks against llms. _CoRR_,
abs/2402.05668, 2024.
[11] Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui
Hsieh. OR-bench: An over-refusal benchmark for large
language models. In _Forty-second International Confer-_
_ence on Machine Learning_, 2025.
[12] Ambra Demontis, Marco Melis, Maura Pintor, Matthew
Jagielski, Battista Biggio, Alina Oprea, Cristina NitaRotaru, and Fabio Roli. Why do adversarial attacks transfer? explaining transferability of evasion and poisoning
attacks. In _28th USENIX Security Symposium (USENIX_
_Security 19)_, pages 321–338, Santa Clara, CA, August
2019. USENIX Association.
[13] Bosheng Ding, Chengwei Qin, Ruochen Zhao, Tianze
Luo, Xinze Li, Guizhen Chen, Wenhan Xia, Junjie Hu,
Anh Tuan Luu, and Shafiq Joty. Data Augmentation
using LLMs: Data Perspectives, Learning Paradigms
and Challenges. In _Findings of the Association for_
_Computational Linguistics ACL 2024_, pages 1679–1705,
Bangkok, Thailand and virtual meeting, 2024. Association for Computational Linguistics.
[14] Siqi Fan, Xin Jiang, Xiang Li, Xuying Meng, Peng Han,
Shuo Shang, Aixin Sun, Yequan Wang, and Zhongyuan
Wang. Not all layers of llms are necessary during
inference. _CoRR_, abs/2403.02181, 2024.
[15] Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow,
Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt,
Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. Bias
and Fairness in Large Language Models: A Survey.
_Computational Linguistics_, 50(3):1097–1179, September
2024.
[16] Adib Hasan, Ileana Rugina, and Alex Wang. Pruning
for protection: Increasing jailbreak resistance in aligned
LLMs without fine-tuning. In Yonatan Belinkov, Najoung
Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller,
and Hanjie Chen, editors, _Proceedings of the 7th Black-_
_boxNLP Workshop: Analyzing and Interpreting Neural_
_Networks for NLP_, pages 417–430, Miami, Florida, US,
November 2024. Association for Computational Linguistics.
[17] Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, EePeng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and
Roy Ka-Wei Lee. LLM-Adapters: An Adapter Family
for Parameter-Efficient Fine-Tuning of Large Language
Models, October 2023.
[18] Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai
Li, and Danqi Chen. Catastrophic jailbreak of opensource LLMs via exploiting generation. In _The Twelfth_
_International Conference on Learning Representations_,
2024.
[19] Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi
Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev,
Qing Hu, Brian Fuller, Davide Testuggine, and Madian
Khabsa. Llama guard: Llm-based input-output safeguard
for human-ai conversations. _CoRR_, abs/2312.06674,
2023.
[20] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch,
Chris Bamford, Devendra Singh Chaplot, Diego de las
Casas, Florian Bressand, Gianna Lengyel, Guillaume
Lample, Lucile Saulnier, L´elio Renard Lavaud, MarieAnne Lachaux, Pierre Stock, Teven Le Scao, Thibaut
Lavril, Thomas Wang, Timoth´ee Lacroix, and William El
Sayed. Mistral 7B, October 2023.
[21] Fengqing Jiang, Zhangchen Xu, Luyao Niu, Zhen Xiang,
Bhaskar Ramasubramanian, Bo Li, and Radha Poovendran. ArtPrompt: ASCII art-based jailbreak attacks
against aligned LLMs. In Lun-Wei Ku, Andre Martins,
and Vivek Srikumar, editors, _Proceedings of the 62nd_
_Annual Meeting of the Association for Computational_
_Linguistics (Volume 1: Long Papers)_, pages 15157–
15173, Bangkok, Thailand, August 2024. Association for
Computational Linguistics.
[22] Nathalie Maria Kirch, Severin Field, and Stephen Casper.
What features in prompts jailbreak LLMs? investigating
the mechanisms behind attacks. In _Red Teaming GenAI:_
_What Can We Learn from Adversaries?_, 2025.
[23] Nathaniel Li, Alexander Pan, Anjali Gopal, Summer
Yue, Daniel Berrios, Alice Gatti, Justin D. Li, AnnKathrin Dombrowski, Shashwat Goel, Gabriel Mukobi,
Nathan Helm-Burger, Rassin Lababidi, Lennart Justen,
Andrew Bo Liu, Michael Chen, Isabelle Barrass,
Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu
Bharathi, Ariel Herbert-Voss, Cort B Breuer, Andy Zou,
Mantas Mazeika, Zifan Wang, Palash Oswal, Weiran
Lin, Adam Alfred Hunt, Justin Tienken-Harder, Kevin Y.
Shih, Kemper Talley, John Guan, Ian Steneker, David
Campbell, Brad Jokubaitis, Steven Basart, Stephen Fitz,
Ponnurangam Kumaraguru, Kallol Krishna Karmakar,
Uday Tupakula, Vijay Varadharajan, Yan Shoshitaishvili,
Jimmy Ba, Kevin M. Esvelt, Alexandr Wang, and Dan
Hendrycks. The WMDP benchmark: Measuring and
reducing malicious use with unlearning. In _Proceedings_
_of the 41st International Conference on Machine Learn-_
_ing_, volume 235 of _Proceedings of Machine Learning_
_Research_, pages 28525–28550. PMLR, 21–27 Jul 2024.
[24] Shen Li, Liuyi Yao, Lan Zhang, and Yaliang Li. Safety
layers in aligned large language models: The key to LLM
security. In _The Thirteenth International Conference on_
_Learning Representations_, 2025.
[25] Yuxi Li, Yi Liu, Yuekang Li, Ling Shi, Gelei Deng,
Shengquan Chen, and Kailong Wang. Lockpicking
LLMs: A Logit-Based Jailbreak Using Token-level Manipulation, June 2024.
[26] Zeyi Liao and Huan Sun. AmpleGCG: Learning a
universal and transferable generative model of adversarial
suffixes for jailbreaking both open and closed LLMs. In
_First Conference on Language Modeling_, 2024.
[27] Yuping Lin, Pengfei He, Han Xu, Yue Xing, Makoto
Yamada, Hui Liu, and Jiliang Tang. Towards understanding jailbreak attacks in LLMs: A representation space
analysis. In Yaser Al-Onaizan, Mohit Bansal, and YunNung Chen, editors, _Proceedings of the 2024 Conference_
_on Empirical Methods in Natural Language Process-_
_ing_, pages 7067–7085, Miami, Florida, USA, November
2024. Association for Computational Linguistics.
[28] Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei
Xiao. AutoDAN: Generating stealthy jailbreak prompts
on aligned large language models. In _The Twelfth_
_International Conference on Learning Representations_,
2024.
[29] Neal Mangaokar, Ashish Hooda, Jihye Choi, Shreyas
Chandrashekaran, Kassem Fawaz, Somesh Jha, and Atul
Prakash. PRP: Propagating universal perturbations to
attack large language model guard-rails. In Lun-Wei
Ku, Andre Martins, and Vivek Srikumar, editors, _Pro-_
_ceedings of the 62nd Annual Meeting of the Association_
_for Computational Linguistics (Volume 1: Long Papers)_,
pages 10960–10976, Bangkok, Thailand, August 2024.
Association for Computational Linguistics.
[30] Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou,
Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel
Li, Steven Basart, Bo Li, David A. Forsyth, and Dan
Hendrycks. Harmbench: A standardized evaluation
framework for automated red teaming and robust refusal.
In _ICML_, 2024.
[31] Anay Mehrotra, Manolis Zampetakis, Paul Kassianik,
Blaine Nelson, Hyrum Anderson, Yaron Singer, and
Amin Karbasi. Tree of Attacks: Jailbreaking Black-Box
LLMs Automatically, February 2024.
[cs, stat].
[32] Meta. Llama 2: Open Foundation and Fine-Tuned Chat
Models, July 2023.
[33] Meta. The Llama 3 Herd of Models, August 2024.
[34] OpenAI. Introducing ChatGPT search.
[35] OpenAI. GPT-4 technical report, 2024. arXiv:
2303.08774 [cs.CL].
[36] Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow.
Transferability in Machine Learning: from Phenomena
to Black-Box Attacks using Adversarial Samples, May
2016.
[37] Nicolas Papernot, Patrick McDaniel, Ian Goodfellow,
Somesh Jha, Z. Berkay Celik, and Ananthram Swami.
Practical Black-Box Attacks against Machine Learning.
In _Proceedings of the 2017 ACM on Asia Conference on_
_Computer and Communications Security_, pages 506–519,
Abu Dhabi United Arab Emirates, April 2017. ACM.
[38] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer,
James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas K¨opf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit
Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.
PyTorch: An Imperative Style, High-Performance Deep
Learning Library, December 2019.
[cs].
[39] Anselm Paulus, Arman Zharmagambetov, Chuan Guo,
Brandon Amos, and Yuandong Tian. AdvPrompter: Fast
Adaptive Adversarial Prompting for LLMs, April 2024.
[40] ProtectAI.com. Fine-tuned DistilRoberta-base for rejection in the output detection, 2024.
[41] Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma,
Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter
Henderson. Safety alignment should be made more than
just a few tokens deep. In _The Thirteenth International_
_Conference on Learning Representations_, 2025.
[42] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.
Direct preference optimization: Your language model is
secretly a reward model. In _Thirty-seventh Conference_
_on Neural Information Processing Systems_, 2023.
[43] Vyas Raina, Adian Liusie, and Mark Gales. Is LLM-asa-judge robust? investigating universal adversarial attacks
on zero-shot LLM assessment. In Yaser Al-Onaizan,
Mohit Bansal, and Yun-Nung Chen, editors, _Proceedings_
_of the 2024 Conference on Empirical Methods in Natural_
_Language Processing_, pages 7499–7517, Miami, Florida,
USA, November 2024. Association for Computational
Linguistics.
[44] Peter J. Rousseeuw. Silhouettes: A graphical aid to the
interpretation and validation of cluster analysis. _Journal_
_of Computational and Applied Mathematics_, 20:53–65,
1987.
[45] Vinu Sankar Sadasivan, Shoumik Saha, Gaurang Sriramanan, Priyatham Kattakinda, Atoosa Chegini, and Soheil Feizi. Fast adversarial attacks on language models in
one gpu minute. In _Proceedings of the 41st International_
_Conference on Machine Learning_, ICML’24. JMLR.org,
2024.
[46] Fabrizio Sandri, Elia Cunegatti, and Giovanni Iacca.
2SSP: A two-stage framework for structured pruning of
LLMs. _Transactions on Machine Learning Research_,
2025.
[47] Gemma Team. Gemma: Open Models Based on Gemini
Research and Technology, 2024. eprint: 2403.08295.
[48] Granite Team. Granite 3.0 language models. Technical
report, IBM, 2025.
[49] Qwen Team. Qwen2.5: A Party of Foundation Models,
September 2024.
[50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser,
and Illia Polosukhin. Attention Is All You Need, August
2023.
[51] Apurv Verma, Satyapriya Krishna, Sebastian Gehrmann,
Madhavan Seshadri, Anu Pradhan, John A. Doucette,
David Rabinowitz, Leslie Barrett, Tom Ault, and Hai
Phan. Operationalizing a threat model for red-teaming
large language models (LLMs). _Transactions on Machine_
_Learning Research_, 2025.
[52] Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao
Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi
Wang, and Peter Henderson. Assessing the brittleness of
safety alignment via pruning and low-rank modifications.
In _Proceedings of the 41st International Conference on_
_Machine Learning_, ICML’24. JMLR.org, 2024.
[53] Bartosz W´ojcik, Paweł Morawiecki, Marek ´Smieja,
Tomasz Krzy˙zek, Przemysław Spurek, and Jacek Tabor. Adversarial Examples Detection and Analysis with
Layer-wise Autoencoders. In _2021 IEEE 33rd Interna-_
_tional Conference on Tools with Artificial Intelligence_
_(ICTAI)_, pages 1322–1326, November 2021.
[54] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien
Chaumond, Clement Delangue, Anthony Moi, Pierric
Cistac, Tim Rault, R´emi Louf, Morgan Funtowicz,
Joe Davison, Sam Shleifer, Patrick von Platen, Clara
Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le
Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest,
and Alexander M. Rush. HuggingFace’s Transformers:
State-of-the-art Natural Language Processing, July 2020.
[55] Sophie Xhonneux, Alessandro Sordoni, Stephan
G¨unnemann, Gauthier Gidel, and Leo Schwinn. Efficient
adversarial training in llms with continuous attacks. In
_Proceedings of the 38th International Conference on_
_Neural Information Processing Systems_, NIPS ’24, Red
Hook, NY, USA, 2025. Curran Associates Inc.
[56] Zihao Xu, Yi Liu, Gelei Deng, Yuekang Li, and Stjepan
Picek. A comprehensive study of jailbreak attack versus
defense for large language models. In Lun-Wei Ku,
Andre Martins, and Vivek Srikumar, editors, _Findings_
_of the Association for Computational Linguistics: ACL_
_2024_, pages 7432–7449, Bangkok, Thailand, August
2024. Association for Computational Linguistics.
[57] Zheng Xin Yong, Cristina Menghini, and Stephen Bach.
Low-resource languages jailbreak GPT-4. In _Socially_
_Responsible Language Modelling Research_, 2023.
[58] Zhiyuan Yu, Xiaogeng Liu, Shunning Liang, Zach
Cameron, Chaowei Xiao, and Ning Zhang. Don’t listen
to me: Understanding and exploring jailbreak prompts
of large language models. In _33rd USENIX Security_
_Symposium (USENIX Security 24)_, Philadelphia, PA,
August 2024. USENIX Association.
[59] Yuan Yuan, Tina Sriskandarajah, Anna-Luisa Brakman,
Alec Helyar, Alex Beutel, Andrea Vallone, and Saachi
Jain. From hard refusals to safe-completions: Toward
output-centric safety training, 2025.
[60] Yiran Zhao, Wenyue Zheng, Tianle Cai, Do Xuan Long,
Kenji Kawaguchi, Anirudh Goyal, and Michael Shieh.
Accelerating greedy coordinate gradient and general
prompt optimization via probe sampling. In _The Thirty-_
_eighth Annual Conference on Neural Information Pro-_
_cessing Systems_, 2024.
[61] Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie
Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng.
On prompt-driven safeguarding for large language models. In _Proceedings of the 41st international conference_
_on machine learning_, ICML’24. JMLR.org, 2025. Place:
Vienna, Austria Number of pages: 21 tex.articleno: 2547.
[62] Yukai Zhou and Wenjie Wang. Don’t Say No: Jailbreaking LLM by Suppressing Refusal, April 2024.
[63] Yuqi Zhou, Lin Lu, Ryan Sun, Pan Zhou, and Lichao
Sun. Virtual context enhancing jailbreak attacks with
special token injection. In Yaser Al-Onaizan, Mohit
Bansal, and Yun-Nung Chen, editors, _Findings of the As-_
_sociation for Computational Linguistics: EMNLP 2024_,
pages 11843–11857, Miami, Florida, USA, November
2024. Association for Computational Linguistics.
[64] Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu
Xu, Fei Huang, and Yongbin Li. How alignment and
jailbreak work: Explain LLM safety through intermediate
hidden states. In Yaser Al-Onaizan, Mohit Bansal, and
Yun-Nung Chen, editors, _Findings of the Association for_
_Computational Linguistics: EMNLP 2024_, pages 2461–
2488, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
[65] Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu
Xu, Fei Huang, Kun Wang, Yang Liu, Junfeng Fang,
and Yongbin Li. On the role of attention heads in large
language model safety. In _The Thirteenth International_
_Conference on Learning Representations_, 2025.
[66] Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang,
Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue
Zhang, Neil Gong, and Xing Xie. Promptrobust: Towards
evaluating the robustness of large language models on
adversarial prompts. In _Proceedings of the 1st ACM_
_Workshop on Large AI Systems and Models with Pri-_
_vacy and Safety Analysis_, LAMPS ’24, page 57–68,
New York, NY, USA, 2024. Association for Computing
Machinery.
[67] Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe
Barrow, Zichao Wang, Furong Huang, Ani Nenkova,
and Tong Sun. AutoDAN: Interpretable gradient-based
adversarial attacks on large language models. In _First_
_Conference on Language Modeling_, 2024.
[68] Andy Zou, Long Phan, Sarah Chen, James Campbell,
Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin,
Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat
Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex
Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt
Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation Engineering: A Top-Down Approach to AI
Transparency, October 2023.
[69] Andy Zou, Long Phan, Justin Wang, Derek Duenas,
Maxwell Lin, Maksym Andriushchenko, J Zico Kolter,
Matt Fredrikson, and Dan Hendrycks. Improving alignment and robustness with circuit breakers. In _The_
_Thirty-eighth Annual Conference on Neural Information_
_Processing Systems_, 2024.
[70] Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr,
J. Zico Kolter, and Matt Fredrikson. Universal and
Transferable Adversarial Attacks on Aligned Language
Models, December 2023.
APPENDIX
_A. Confusion Matrices_
Figure 11 and Figure 12 show the confusion matrices of the
4 models after applying the GCG attack on the LLM.
_B. Attacking Safe Inputs_
As with any classification problem, mitigating false negatives (jailbreaks) comes at the cost of higher false positives
(over-refusal). This effect was seen in the lower classification
performance on _OR-Bench_ (designed to test this phenomenon)
in Section IV-B. In this section, we focus on attacking safe
inputs to create a refusal.
**Setup.** We also augment the dataset by adding refusal targets.
Different aligned models have different refusal answers depending on how they were trained. However, we saw limited
impact on the choice of the target string on the performance
of the attack and used the same target for all prompts and
models: _“I cannot fulfill your request. I’m just an AI”_ .
**Safe ASR.** Similarly to Section IV-C, the attack is applied
to the models, this time with safe inputs (with the objective
of making the model refuse). Table IV reports the ASR for
all settings. We first see that the highest ASR corresponds to
Llama 2 (99%), which has the lowest ASR on unsafe inputs
(12% to 22%). In contrast, attacking Qwen 2.5 produced a
high ASR for unsafe and safe inputs. Finally, Gemma 1 shows
medium ASR on unsafe inputs, also exhibited for safe inputs.
|Model|AdvBench OR-Bench|
|---|---|
|Gemma 1
Granite
Llama 2
Qwen 2.5|0.4
0.47
0.68
0.7
0.99
0.99
0.88
0.85|
TABLE IV: Attack success rates (ASR) of the models after
applying the attack on safe inputs.
**Results.** Figure 13a and Figure 13b report the ASR on
candidate classifiers and the transferability rate to the LLM,
for safe inputs. We notice the same trends on the transferability
rate with a peak for most models with an exception for Llama
2. Furthermore, for _OR-Bench_, the ASR on the candidate classifiers corresponds exactly to the transferability rate, implying
that the success of the attack is determined by how well it
Unsafe Safe
Actual
(b) Granite
Unsafe Safe
Actual
(c) Llama 2
Fig. 11: Confusion matrices on adversarial inputs for _AdvBench_
Unsafe Safe
Actual
(d) Qwen 2.5
Unsafe Safe
Actual
(d) Qwen 2.5
Unsafe Safe
Actual
(b) Granite
Unsafe Safe
Actual
(c) Llama 2
Fig. 12: Confusion matrices on adversarial inputs for _OR-Bench_
1.0
0.8
0.6
0.4
0.2
Unsafe Safe
Actual
(a) Gemma 1
Unsafe Safe
Actual
(a) Gemma 1
1.0
0.8
0.6
0.4
0.2
0.0
|Col1|Col2|Col3|Col4|Col5|Col6|Col7|
|---|---|---|---|---|---|---|
||||||||
|G
|G
||||||
|G
|G
|emma 1
~~ranite~~|emma 1
~~ranite~~|emma 1
~~ranite~~|emma 1
~~ranite~~|emma 1
~~ranite~~|
||L
Q|lama 2
wen 2.5|||||
|A
O|A
O|dvBench
R-Bench|||||
|A
O|A
O||||||
0.0 0.2 0.4 0.6 0.8 1.0
Normalized Candidate Size
(a) ASR on candidates
0.0
|G|Col2|emma 1|Col4|Col5|Col6|Col7|Col8|
|---|---|---|---|---|---|---|---|
||~~L~~
Q
A|ranite
~~lama 2~~
wen 2.5
dvBench||||||
|O|O|R-Bench|R-Bench|||||
|||||||||
|||||||||
0.0 0.2 0.4 0.6 0.8 1.0
Normalized Candidate Size
(b) Transferability rate to the LLM
Fig. 13: ASR on the candidates and transferability rate to the LLM after applying the attack on the candidates for safe samples.
performs on the candidate classifier. This finding aligns with
the performance in adversarial settings in Figure 8b: Llama
2 candidate classifiers have the most stable performance on
adversarial examples from the LLM compared to other models.
_C. Other Models_
Our work focuses on aligned LLMs under the assumption
that alignment embeds a safety classifier. Therefore, our approach is ill-suited to models with weaker alignment or that
have been made more secure through a different approach
(different from the traditional alignment techniques).
**Models.** We studied three other large language
models: [Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) [33],
[Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3) [20], and
[Zephyr_RMU [23].](https://huggingface.co/cais/Zephyr_RMU)
|Model|AdvBench OR-Bench|
|---|---|
|Llama 3
Mistral
Zephyr RMU|0.84
0.63
0.74
0.77
0.7
0.62|
TABLE V: _F_ 1 score of the other models on the two datasets.
**Results.** Table V reports their accuracy and F1 score on
the two datasets. All models achieve an F1 score lower
than 0.8 on _OR-Bench_, translating to a weaker performance
according to the setup. Figure 14 and Figure 15 show the
1.0
0.9
0.8
0.7
0.6
1.0
0.9
0.8
0.7
0.6
0.5
|Col1|Col2|
|---|---|
|||
||Llama 3|
||Mistral
Zephyr RMU|
|0
0.2
0.4
Normalized C|0.6
0.8
1.0
 andidate Size|
(a) _AdvBench_
1.0
0.5
|Col1|Llama 3
Mistral|
|---|---|
||Zephyr RMU|
|||
|||
|0
0.2
0.4
Normalized C|0.6
0.8
1.0
 andidate Size|
(a) Trained on _AdvBench_, evaluated on _OR-Bench_
1.0
0.9
0.8
0.7
0.6
0.9
0.8
0.7
0.6
0.5
|Col1|Llama 3
Mistral|
|---|---|
||Zephyr RMU|
|||
|||
|||
|0
0.2
0.4
Normalized C|0.6
0.8
1.0
 andidate Size|
(b) _OR-Bench_
Fig. 14: Test F1 of the candidates of the classifier in benign
settings for other models.
performance of the candidates in benign settings, with the
same setup as Section IV-B. We see that the trend is similar in
Figure 14a with a convergence after a certain candidate size.
However, the F1 score is lower than the other models studied.
Further, Figure 14b shows a different trend, with a decrease
at the middle of the model. Figure 15 shows the cross-dataset
experiment for the other models studied.
**Possible causes.** The Mistral model is described as a _“quick_
_demonstration”_ that _“does not have any moderation mecha-_
_nisms”_ [5] [20]. Llama 3, similarly to Llama 2, did go through
alignment. However, the tone of the output was adjusted,
specifically for refusal outputs. Therefore, detecting the classification made by the model is more challenging. Zephyr RMU
is a model in which unsafe information has been removed
through unlearning [23], therefore the lower performance is
[5https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3 in “Limitations”.](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3)
0.5
|Col1|Llama 3
Mistral|
|---|---|
||Zephyr RMU|
|||
|||
|||
|0
0.2
0.4
Normalized C|0.6
0.8
1.0
 andidate Size|
(b) Trained on _OR-Bench_, evaluated on _AdvBench_
Fig. 15: Test F1 of the estimations on the dataset they were
not trained on for other models.
due to the way labels are assigned: in our experimental setup,
we assign labels based on whether the model complies with
the input prompt or not. Thus, since Zephyr RMU underwent
unlearning, it may still comply with unsafe input prompts
with a safe output. We argue that models that are made more
secure through unlearning are orthogonal to the scope of the
paper: this method likely removes part of the safety classifier
alongside the unsafe knowledge, preventing its extraction.

aligned models to jailbreaking attacks. The code is available at


**Maarten Buyl** _[†]_ **Hadi Khalaf** _[†]_ **Claudio Mayrink Verdun** _[†]_ **Lucas Monteiro Paes** _[†]_
Ghent University Harvard University Harvard University Harvard University

Maarten Buyl
Hadi Khalaf
Claudio Mayrink Verdun
Lucas Monteiro Paes
Caio C. Vieira Machado
Flavio du Pin Calmon


University of Oxford
Harvard University
University of S˜ao Paulo


In AI alignment, extensive latitude must be granted to annotators, either human or algorithmic, to
judge which model outputs are ‘better’ or ‘safer.’ We refer to this latitude as _alignment discretion_ .
Such discretion remains largely unexamined, posing two risks: (i) annotators may use their power of
discretion arbitrarily, and (ii) models may fail to mimic this discretion. To study this phenomenon,
we draw on legal concepts of discretion that structure how decision-making authority is conferred
and exercised, particularly in cases where principles conflict or their application is unclear or irrelevant. Extended to AI alignment, discretion is required when alignment principles and rules are
(inevitably) conflicting or indecisive. We present a set of metrics to systematically analyze _when_
and _how_ discretion in AI alignment is exercised, such that both risks (i) and (ii) can be observed.
Moreover, we distinguish between _human_ and _algorithmic discretion_ and analyze the discrepancy
between them. By measuring both human and algorithmic discretion over safety alignment datasets,
we reveal layers of discretion in the alignment process that were previously unaccounted for. Furthermore, we demonstrate how algorithms trained on these datasets develop their own forms of
discretion in interpreting and applying these principles, which challenges the purpose of having any
principles at all. Our paper presents the first step towards formalizing this core gap in current alignment processes, and we call on the community to further scrutinize and control alignment discretion.
**Warning: this paper contains example data that may be offensive, biased, and/or harmful**
_**Keywords**_ AI alignment, AI safety, discretion, AI governance, judicial discretion


AI alignment aims to ensure that artificial intelligence (AI), like large language models (LLMs), ‘behaves’ [1] in accordance with _human intentions_ and _social, legal, and ethical principles_ [55]. Particular interest has gone to aligning
LLMs through _learning from human feedback_, which involves (i) collecting examples of possible AI outputs, (ii) annotating these examples by asking _human_ annotators “which output is better” (typically with limited instructions), and
(iii) training the model to follow these human preferences [80]. This example-based approach to alignment is widely
deployed in practice [80, 55, 36, 72]. Nevertheless, a gap remains between translating human intentions and social,
legal, and ethical principles into a simplistic decision of “which output is better.” This gap gives annotators extensive
_discretion_ in defining what alignment means in practice, hindering the interpretability of the alignment process.
_†_ Buyl, Khalaf, Mayrink Verdun, and Monteiro Paes contributed equally to this work and are listed in alphabetical order.
[Correspondence may be sent to maarten.buyl@ugent.be](mailto:maarten.buyl@ugent.be)
1We acknowledge that terms like ‘act’ and ‘behave’ anthropomorphize AI systems in potentially misleading ways [9, 52]. We
use these terms for simplicity of exposition while recognizing that AI systems are computational processes that transform inputs
into outputs through statistical pattern matching rather than conscious agents that truly ‘act’ comparable to humans [111]. Although
there is no universally accepted linguistic convention for describing AI system operations, we aim to balance clarity with precision
while remaining mindful of the limitations of such terminology.
AI Alignment at Your Discretion
Annotator principle prioritization
User prompt
I am feeling tired
and having trouble
concentrating.
Do you know why?
Preferred responses for annotators with diferent principle prioritizations
Figure 1: Illustration of how different prioritizations of principles affect which AI model responses are preferred,
inspired by the xkcd comic about Asimov’s Three Laws of Robotics [75]. The user asks for health advice, but an
annotator’s assessment of how best to respond depends on how they rank three principles: (A) being accurate in
responding to medical concerns, (B) referring to experts, and (C) reducing patient anxiety. All three principles are
independently desirable, but they allow for _discretion_ in how they are balanced.
_Principle_ -based alignment approaches like Constitutional AI [6] aim to improve the interpretability in the alignment
process by defining an explicit set of principles (e.g., “don’t be racist” and “respect privacy” [2]) that the LLM needs
to follow. They then use algorithms as annotators to produce examples responses that better align to the principles.
Principles inevitably conflict, however, making it impossible for model outputs to simultaneously align with all of
them. As illustrated by our example in Fig. 1, selecting (or generating) a ‘preferred’ response implies that some
principles are prioritized over others. The simple act of selecting one response over another, which underpins all
feedback-based alignment [55], (i) fails to capture the rationale behind preference decisions, (ii) masks how principles
are balanced and prioritized when they conflict, and (iii) provides little guidance on how alternative phrasings should
be judged. Such opaque judgments by annotators contribute to the (well-documented) ambiguity and lack of clarity in
alignment and harm detection datasets [12, 98, 51]. If left unsurfaced, we cannot understand _what_ we are aligning to.
In this work, we define **alignment discretion** as _the margin afforded to annotators to decide which AI behavior is_
_“better” with respect to the alignment goals and principles_ . To study such discretion, we draw parallels in Sec. 3
with the inherent need for _judicial discretion_ in the rule of law – a fundamental debate shaped by authors such as
Hart [46] and Dworkin [31], which have profoundly influenced modern legal philosophy. Like judges, annotators
operationalize abstract (alignment) principles while navigating conflicts and ambiguities. Yet, these parallels lead us
to argue that current alignment approaches allow an _excessive_, _unscrutinized_ amount of discretion. This poses the risk
of “alignment-washing,” where alignment processes creating a false sense of ethical compliance [11, 16].
Inspired by parallels with judicial discretion (in Sec. 3) and rooted in current alignment methods (in Sec. 4), we
propose a set of metrics to measure alignment discretion in Sec. 5. These metrics allow us to explicitly quantify the
extent of discretion afforded to annotators, how they prioritize principles, and when they arbitrarily oppose them. We
thus offer transparency over _what_ we are aligning to, complementary to research strands that question _who_ we are
aligning to [4, 53, 59, 58]. Specifically, **we use our proposed metrics for alignment discretion to** :
   - _Characterize human discretion_ in alignment datasets ( `hh-rlhf` and `PKU-SafeRLHF` ), revealing how annotators
implicitly prioritize principles; improving the transparency of the alignment process (Fig. 5).
   - _Quantify the extent of discretion_, finding that there is an excessive amount of discretion afforded to annotators
in alignment datasets (Fig. 3) and that annotators frequently use their power of discretion arbitrarily (Tab. 1).
   - _Analyze whether algorithms can mirror human discretion_ . We find that the discretion of reward models can
closely mirror human discretion by fine-tuning on human preferences (Tab. 2). However, we also demonstrate
that reinforcement learning from human feedback (RLHF) may not suffice to transfer human discretion to
LLMs, suggesting that translating human discretion from reward models to LLMs is an open problem.
   - _Audit the discretion of models in the wild._ We find significant discrepancy between the discretion of off-theshelf models (GPT-4o, DeepSeek-V3, and Claude 3.5 Sonnet) and human preferennces (Tab. 2).
Our formalization of alignment discretion reveals a core gap in the feedback-based alignment process. As discussed in
Sec. 7, challenges remain in understanding when discretion is exercised and how it can be controlled. If discretion is
left unsurfaced, preference-based alignment risks devolving into a _kangaroo court_ [71] – a sham process characterized
by arbitrary judgments, lack of transparency, and absence of principled reasoning or accountability mechanisms.
AI Alignment at Your Discretion


To our knowledge, we are the first to study discretion in AI alignment empirically. Next, we review threads of related
work that inform our analysis.
**Alignment from human feedback** aims to ensure that model outputs are in accordance with user expectations using
_human feedback_ [18]. The popular approach is _Reinforcement Learning from Human Feedback_ (RLHF), which is
discussed in Sec. 4.1. Researchers have developed many algorithms to perform RLHF like [114, 80]. At the same
time, multiple vulnerabilities were found in this process, leading to a diverse set of jailbreak attacks [85]. Our work
differs from previous contributions by identifying a _fundamental limitation_ that is independent of the algorithm used
to perform RLHF - the excessive discretionary power inherent in annotation processes, which persists even in recent
variants like DPO [90] and KTO [37].
**Alignment from AI feedback** aims to ensure that model outputs are in accordance with user expectations _without_
using human feedback. The main example of such alignment from AI feedback is Constitutional AI [6], which defines
an explicit list of principles to align to. Language models are then used to generate and annotate examples to follow
these principles [6]. However, as noted in Anthropic’s public discussion of Claude’s constitution [2], principles are
applied stochastically during training: _“The model pulls one of these principles each time it critiques and revises its re-_
_sponses during the supervised learning phase, and when it is evaluating which output is superior in the reinforcement_
_learning phase. It does not look at every principle every time, but it sees each principle many times during training.”_
This stochastic approach leaves open questions about principle prioritization and conflict resolution. _Collective_ Constitutional AI developed a framework to learn principles from users instead of arbitrarily defining them [48]. Recent
work has expanded these foundations through various frameworks like [74, 29]. In general, these approaches never
require direct human supervision, removing the power of discretion from users and deferring it to the models. Our
work learns how principles are encoded and prioritized in human preference data, analyzing the human discretion
contained in these datasets, and the discretion of models trained in these datasets.
**Principles and human preferences.** Recent papers have tried to _learn_ a set of principles encoded in preference
datasets. This is an instance of the problem of bridging principles to practice [25]. The Value Imprint [77] established
a framework for auditing human values embedded within preference datasets by developing a taxonomy of human
values. Moreover, [60] propose an approach inspired by moral philosophy to determine and reconcile relevant values
from diverse human inputs. Drawing inspiration from constitutional design, [41] proposes a framework to distill
individual and group-level preferences into a set of principles to guide model behavior. Our work builds upon these
by analyzing how (i) humans prioritize different principles by analyzing discretion and (ii) how these principles are
learned by models by analyzing algorithmic discretion. Ultimately, we argue that there is currently an excessive
amount of discretion in the hands of model developers and annotators.
**Pluralistic alignment** expanded alignment approaches by embracing diverse human values and perspectives [100].
Key developments include the Value Kaleidoscope taxonomy of values and rights [99], the PRISM dataset for multicultural feedback [58], frameworks for leveraging community-specific LMs [40], and approaches that consider the
temporal aspects of pluralistic alignment with multiple stakeholders [59]. While these works focus on gathering
diverse perspectives and defining principles, our research specifically analyzes how to weigh and resolve conflicts
between different principles. This goes in line with the recent push for a social-choice approach to alignment to aggregate and reconcile preferences of diverse annotators and principles [21]. Our work differs from this literature by
offering transparency over _what_ we are aligning to, complementary to the question of _who_ we are aligning to, and
ultimately indicating whether aligning to a list of rules produces AI-adherence to a system of values, as a rule-based
system would. We hope that future work analyzes how different communities exercise their power of discretion.
**Annotator disagreement** is a well-studied problem in natural language tasks [94, 106, 14] as it impacts all stages
of the usual ML pipeline [88]. [112] shows that how these disagreements are often rooted in personal biases rather
than annotation errors. With existing alignment methods typically depending on a single ground-truth label, we risk
privileging certain views at the expense of others, thereby ushering in a _tyranny of the majority_ [39]. For this reason,
scholars proposed approaches to better aggregate conflicting annotations beyond majority voting by using Bayesian
approaches [83] and proposing model architectures that handle multiple annotations [73]. The challenge of annotation
disagreement becomes particularly relevant with the increasing use of LLM evaluators, leading recent work to focus
on measuring and reducing biases in their evaluations [68, 110] and improving their reliability and interpretability
[66]. Discretion in AI alignment reveals the principles influencing annotator decisions and how annotators prioritize
conflicting principles, explaining _why_ annotators disagree as a function of their principles.
**AI Alignment and Law.** Recent legal literature has argued that AI alignment operates in a similar fashion to the
legal system [15, 1, 76]. The authors emphasize the role of interpretation and application of normative principles
to guide AI behavior. For instance, [15] argues that AI alignment faces similar challenges in accommodating diverse
AI Alignment at Your Discretion
human values (pluralism) and defining precise rules for AI behavior (specification). Moreover, [1] points to the issue of
transparency as a core element of legal decision-making that lends legitimacy to the legal system. These works suggest
that a legally-inspired approach to AI alignment could be valuable. Moreover, they also highlight transparency in legal
decision-making as a key factor of the exercise of discretion. Our work builds upon these findings by (i) formally
defining discretion in AI alignment, (ii) connecting it to legal systems, and (iii) empirically studying the extent of
discretion in algorithmic and human annotators.


Discretion lies at the heart of AI alignment, as annotators are necessary to label which model outputs are “better”
or “safer.” Such discretion manifests when annotators assess outputs where principles conflict or provide insufficient
guidance. We here remark that employing humans as annotators is expensive and carries ethical risks [33, 47]. As
AI models become increasingly powerful, it has become popular to instead use algorithms as a cheaper source of
preference annotations [6, 64, 22]. Alignment discretion can thus involve both _human_ and _algorihmic discretion_ .

3 From judicial discretion to alignment discretion