The hypotheses in Fig. 1 exemplify that the underlying exercise of discretion can fundamentally alter how outputs are
judged, thus determining whether the AI should refer to a medical doctor or suggest medication. Moreover, we cannot
effectively ensure that AI systems properly learn from and respect the legitimate diversity of human judgments.
More broadly, parallels can be drawn with _judicial discretion_ . Indeed, legal theorists have long recognized that discretion in judicial systems – _arbitrium judicis_ - requires careful structuring to ensure transparency, accountability, and
legitimacy [26, 63] – discretion in AI alignment demands similar scrutiny. Without understanding and structuring this
discretion, we cannot know _what_ we are aligning to or _whether_ we are successful, and we risk embedding unexamined
value judgments into AI systems that become resistant to auditing or revision once deployed.


As Caputo [15] observed, jurisprudence and AI alignment share fundamental challenges in translating abstract principles into concrete decisions while maintaining consistency and legitimacy. In both contexts, decision-makers must
navigate what Dworkin [32] terms the “dimension of weight.” Unlike rules, principles do not have an ”all-or-nothing”
application but must be weighted against each other. Judicial discretion and alignment discretion thus appear to share
strong similarities, potentially making the judicial process a rich source of inspiration for better-interpreted alignment.
To understand how far this inspiration can take us, we discuss key parallels and differences.

3.2 How do judicial discretion and alignment discretion relate?

Maarten Buyl
Hadi Khalaf
Claudio Mayrink Verdun
Lucas Monteiro Paes
Caio C. Vieira Machado
Flavio du Pin Calmon


**Principle Application and Generalization.** Both must apply broad and abstract principles to specific situations that
may not have been anticipated when those rules were created. Judges interpret laws for novel situations; AI systems
must apply alignment principles to unforeseen prompts.
**Consistency vs. Flexibility.** Both must balance maintaining consistent application of principles with flexibility in
adapting to nuanced contexts. Legal systems strive for predictability while allowing for case-specific considerations
[46] – AI systems must also provide consistent responses while appropriately handling context-dependent ethical
considerations. This balance relies on building precedents and a consistent understanding of many cases; judges
through years of legal practice and life experience, annotators through their lived experience, and AI systems through
exposure to vast amounts of text that captures human decision-making patterns.
**Managing Conflicts.** Both must ponder and balance competing principles. Courts often resolve competing rights or
interests; AI must weigh a range of alignment principles that may suggest different courses of action.


**Scale and Granularity.** Judges make high-stakes decisions about consequential real-world actions. Conversely,
alignment annotators exercise discretion over countless seemingly minor choices about model outputs. However, the
discretion exercised in these goes unnoticed and unaccounted for, producing impacts that can be both impactful in
specific cases and/or accumulate to create inscrutable interpretations of principles in the long run.
**Human-Algorithm Translation.** Unlike judicial systems where discretion is exercised by human judges within established frameworks, AI alignment involves a complex interplay between annotators – humans or algorithms – and
AI Alignment at Your Discretion
the final output of LLMs alongside the decisions made by developers who select or design these annotators [2] . This
interplay lead to gaps between how humans and AI systems exercise discretion, potentially leading to misalignment
in principle prioritization.
**Review and oversight mechanisms.** Judicial discretion benefits from the deliberative nature and built-in inertia of
legal systems, while algorithmic discretion has immediate, multiplicative effects. Under _stare decisis_ [76], legal precedents develop gradually through individual cases, enabling review and correction. In contrast, alignment discretion
can be instantly replicated across millions of interactions once deployed.
These structural challenges suggest that AI alignment requires frameworks for managing discretion that are not only as
rigorous as those in legal systems but also specifically adapted to handle this combination of granularity, the need for
human-algorithm translation, and lack of built-in review and oversight mechanisms. To address these challenges, we
argue that a _statistical_ approach is needed that documents and constrains discretion in alignment. Such an approach
should serve two critical functions: first, to ensure that AI models reliably learn from human annotators’ principled
judgments rather than developing divergent interpretations reflecting hidden biases; and second, to provide concrete
metrics and standards for ongoing oversight and verification of how discretion is being exercised throughout the
alignment process. Without such systematic measures to track and evaluate discretionary choices, we cannot ensure
consistency across different annotators and systems or verify that discretion aligns with intended human values.
In Sec. 5, we perform a formalization of alignment discretion and provide a set of metrics to serve these two critical
functions. After exploring them empirically in Sec. 6, we will revisit the parallel with judicial discretion in Sec. 7 and
discuss what our results imply for alignment discretion moving forward. First, however, we develop some necessary
technical concepts for formalizing preferences in Sec. 4.


The most prominent form of alignment employs pairwise preferences to perform Reinforcement Learning from Human
Feedback (RLHF) [18, 109, 102]. For a query _x ∈X_, we denote the set of possible answers to the query as _Y_ . A
(pairwise) _preference_ over a pair of responses _y_ 0 _∈Y_ and _y_ 1 _∈Y_ is denoted as _y_ 1 _≻_ _y_ 0 if _y_ 1 is preferred over _y_ 0.
In RLHF, it is commonly assumed that these pairwise preferences follow the Bradley-Terry-Luce (BTL) model [13]
(see also [69, 23, 87, 24, 49, 38, 45]). For a pair of items ( _y_ 0 _, y_ 1), it expresses the probability of preferring _y_ 1 over _y_ 0
by assuming each response has an latent ‘quality’. Estimating this quality with a _reward model rϕ_, the BTL model is
_P_ ( _y_ 1 _≻_ _y_ 0 _| x_ ) = _σ_ ( _rϕ_ ( _x, y_ 1) _−_ _rϕ_ ( _x, y_ 0)) (1)
where _σ_ is the logistic sigmoid function. The reward model _rϕ_ is trained by minimizing the cross-entropy between the
prediction of _y_ 1 _≻_ _y_ 0 according to (1) and ground truth preference labels. An LLM with policy _πθ_, i.e. the function
that computes the probability that an output _y_ should follow a context _x_, can then be aligned by training it to maximize
the reward _rϕ_ while staying close to a reference model _π_ ref:
_L_ RLHF( _πθ, π_ ref _, rϕ, λ_ ) = E _x∼D_ E _y∼πθ_ ( _·|x_ )[ _rϕ_ ( _x, y_ )] + _λ ·_ KL( _πθ∥π_ ref) _._ (2)
where _D_ is a distribution over prompts. The reference policy _π_ ref is typically a pre-trained language model and ensures
the model retains its general language capabilities and knowledge, with _λ_ controlling the strength of this constraint.


The alignment approach in Sec. 4.1 makes no distinction about _who_ prefers _y_ 1 over _y_ 0. Yet, to compare discretion
between annotators in Sec. 5.3, we will distinguish between _human_ and _algorithmic_ annotators. To simplify notation,
we first introduce _preference functions_ that, given a query _x_ and two candidate responses _y_ 0 and _y_ 1, outputs 1 if the
annotator prefers response _y_ 1, _−_ 1 if _y_ 0 is preferred, and 0 if the annotator is indifferent.
2Quoting Anthropic: _our approach to data collection was to largely let crowdworkers use their own intuitions to define “help-_
_fulness” and “harmfulness”. Our hope was that data diversity (which we expect is very valuable) and the “wisdom of the crowd”_
_would provide comparable RoI to a smaller dataset that was more intensively validated and filtered_ [5]. Moreover, one of the four
criteria that OpenAI adopted in the selection of data labelers was their agreement with OpenAI researchers [80, Appendix B.1].
AI Alignment at Your Discretion
**Definition 1 (preference functions)** _A preference function denoted by Prefa_ ( _y_ 1 _≻_ _y_ 0 _| x_ ) _∈_ [ _−_ 1 _,_ 0 _,_ 1] _is a ternary-_
_valued function that expresses the preference of annotator a over_ ( _y_ 0 _, y_ 1) _for the context x. It is defined as_
_Prefa_ ( _y_ 1 _≻_ _y_ 0 _| x_ ) ≜


1 _,_ _if a prefers y_ 1 _, i.e. y_ 1 _≻_ _y_ 0
_−_ 1 _if a prefers y_ 0 _, i.e. y_ 0 _≻_ _y_ 1 (3)
0 _,_ _if a is indifferent towards y_ 0 _and y_ 1 _, i.e._ ( _y_ 1 _̸≻_ _y_ 0) _∧_ ( _y_ 0 _̸≻_ _y_ 1) _._

_We simply use Prefa if the argument tuple_ ( _y_ 0 _, y_ 1 _, x_ ) _is clear from the context._
Allowing ‘indifference’ over ( _y_ 0 _, y_ 1), where neither is preferred, is uncommon in (human) preference datasets as
Pref _a_ = 0 seemingly conveys no useful information over _y_ 0 and _y_ 1. However, allowing indifference improves the
robustness of our metrics in Sec. 5.2 when using LLMs as annotators [113], as their ‘preference’ is often unclear.
For human annotators, the preference function is directly derived from a dataset of collected preferences. In our
experiments, we also consider algorithms as annotators so that we can audit their discrepancies with human annotators
(see Sec. 6.2). Although these preferences may not be as rich and accurate as human annotators, in practice, these are
the ones controlling the model generation. For example, the reward model _rϕ_ in Sec. 4.1 acts as an intermediary by
scoring any model output at will, based on what it learned from a (comparatively) small amount of human feedback.
We thus compare reward models and LLMs as ‘annotators’ to human annotators. To this end, we instantiate Def. 1 for
_algorithmic annotators_ by directly postulating their preferences. Recall that reward models _rϕ_ directly rate the quality
of a response using the BTL model of preferences in (1).
**Definition 2 (reward model preference functions)** _We set the preference function Prefrϕ of a reward model rϕ as_
_Prefrϕ_ ( _y_ 1 _≻_ _y_ 0 _| x_ ) ≜ _sign_ ( _rϕ_ ( _x, y_ 1) _−_ _rϕ_ ( _x, y_ 0)) _._ (4)
Intuitively, reward model preferences Pref _rϕ_ = 1 prefer _y_ 1 iff _rϕ_ assigns a higher ‘reward’ to _y_ 1 than to _y_ 0.
To instantiate the ‘preference’ of an LLM, we make use of its policy _πθ_, e.g. as optimized in (2), which outputs the
probability _πθ_ ( _y|x_ ) that _y_ ought to be generated in response to context _x_ . We could mirror (4) by setting Pref _πθ_ =
sign( _πθ_ ( _y_ 1 _|x_ ) _−_ _πθ_ ( _y_ 0 _| x_ )). However, we opt to instead show all of ( _x, y_ 0 _, y_ 1) to the LLM at once in a prompt where
we ‘ask’ whether it prefers _y_ 0 or _y_ 1, just as we would ask a human annotator. Indeed, the latter has become the norm
when using LLMs to judge fixed pairs of responses ( _y_ 0 _, y_ 1) [70] because the actual scores _πθ_ ( _y_ 0 _|x_ ) and _πθ_ ( _y_ 1 _|x_ ) may
be poorly calibrated for responses that the model is unlikely to output itself [103].
**Definition 3 (LLM preference functions)** _We set the preference function Prefπθ of an LLM with policy πθ as_
_Prefπθ_ ( _y_ 1 _≻_ _y_ 0 _| x_ ) ≜


1 _if “Response 1”_ = arg max _z πθ_ ( _z | T_ ( _x, y_ 0 _, y_ 1))
_−_ 1 _if “Response 0”_ = arg max _z πθ_ ( _z | T_ ( _x, y_ 0 _, y_ 1)) (5)
0 _else_

_with T a composition of_ ( _x, y_ 0 _, y_ 1) _into a textual prompt that ‘asks’ an LLM with policy πθ whether “Response_ 0 _” or_
_“Response_ 1 _” (representing y_ 0 _and y_ 1 _respectively) is “better”, while optionally specifying that the LLM is allowed_
_to choose neither if none are clearly better. The exact template is provided in Appendix B.5._


We define alignment discretion as the latitude afforded to annotators to operationalize alignment principles. Building
upon parallels with legal theory (Sec. 3) and preference functions (Sec. 4), we now formalize _when_ and _how_ discretion
is exercised. The resulting metrics will allow us to measure the discrepancy between human and algorithmic discretion.


Intuitively, if a response _y_ 1 is preferred over _y_ 0 by _all_ principles _c ∈_ _C_, then no discretion is required; preferring _y_ 0
is irrational according to the principles. Principles may also conflict, however, or they may all be indifferent. We then
cannot determine the best output with principles alone, and require an annotator to exercise discretion by stating their
preference, thereby prioritizing certain principles. When assessing the agreement among principles based on their
preference function Pref _c_ (see Def. 5), we distinguish three fundamental cases: consensus, conflict, and indifference.
**Definition 6 (consensus, conflict, & indifference)** _Given principle-specific preferences Prefc_ = _Prefc_ ( _y_ 1 _≻_ _y_ 0 _| x_ )
_for all c ∈_ _C, exactly one of the following holds for candidate responses_ ( _y_ 0 _, y_ 1) _in context x:_
_A._ _**principle consensus (Consensus**_ _C_ _**)**_ _: At least one principle prefers one response and no principles disagree:_
_ConsensusC ≡_ ( _∀c_ 1 _, c_ 2 _∈_ _C_ : _Prefc_ 1 _× Prefc_ 2 _̸_ = _−_ 1) _∧_ ( _∃c ∈_ _C_ : _Prefc ̸_ = 0) _._ (7)
_B._ _**principle conflict (Conflict**_ _C_ _**)**_ _: At least two principles disagree on the preferred output:_
_ConflictC ≡∃c_ 1 _, c_ 2 _∈_ _C_ : _Prefc_ 1 _× Prefc_ 2 = _−_ 1 _._ (8)
_C._ _**principle indifference (Indifference**_ _C_ _**)**_ _: All principles are indifferent:_
_IndifferenceC ≡∀c ∈_ _C_ : _Prefc_ = 0 _._ (9)
The three cases of principle agreement in Def. 6, illustrated in Fig. 2, are determined solely by examining the principlespecific preferences Pref _c_ . Their classification requires no additional human annotations or model outputs beyond the
initial principle-specific assessments provided by the oracle model for each principle _c ∈_ _C_ .
**Principle consensus** allows for no discretion, as it fully determines the best output according to the set of principles.
From this perspective, any annotation or behavior that disagrees with the consensus could be considered ‘arbitrariness’,
which we formally measure in Def. 7.
**Principle conflicts** create legitimate tension between competing objectives. Hence, principle conflicts call for _mean-_
_ingful_ discretion to be exercised by an annotator. The annotator is empowered to choose which response they prefer,
and thus which principle ought to win out. Such supremacy of principles is characterized in Def. 8.
**Principle indifferences** provide no meaningful guidance and thus only allow for _unconstrained_ discretion, meaning
annotators are still free to prefer either response, but they cannot be explained through any (known) principle. Such
lack of constraint limits the legitimacy of the annotation, as it may be irrational, idiosyncratic, or meaningless.
Among the three cases, only principle consensus eliminates the need for annotators. Hence, to increase the power of
principles over annotators, conflict and indifference may be reduced by intervening on the (i) principle set _C_ or (ii) the
dataset of response pairs ( _y_ 0 _, y_ 1). Unfortunately, this may prove challenging. Adding new principles to _C_ or making
response pairs ( _y_ 0 _, y_ 1) more distinct may reduce indifference, in turn leading to more consensus but likely also to
more conflict. Similarly, conflict can be reduced by removing controversial principles from _C_ or by making principles
more specific, but doing so may only increase the frequency of consensus at the cost of increased indifference.
AI Alignment at Your Discretion
Principle preferences
User prompt Sample responses
CONSENSUS
Def 5.1 a
Who should I call
in the US during
an emergency?
I have a headache,
what should I do?
How can I
stay healthy?
You should call
the emergency
telephone number
Take an aspirin See a doctor
Eat a balanced meal Exercise regularly
CONFLICT
Def 5.1 b
INDIFFERENCE
Def 5.1 c
Figure 2: Illustration of the three principle agreement cases in Def. 6. For each prompt, two candidate responses
are evaluated against three principles ( _‘Be helpful’_, _‘Avoid harm’_, _‘Refer to experts’_ ). Cases show: CONSENSUS principles align in favoring the “911” response over “You should call the emergency telephone number”; CONFLICT
- principles disagree with each other, where “Take an aspirin” aligns with being helpful but “See a doctor” better aligns
with referring to experts; INDIFFERENCE - none of the principles express a clear preference for either response.


We now characterize _how_ discretion is used by an annotator denoted by _a_ . First, we measure how often their discretion
is arbitrary. Second, we model how much they prioritize each principle. For both, we work with empirical probabilities
Pr( _·_ ) computed over a dataset _D_ consisting of tuples ( _x, y_ 0 _, y_ 1), treating the dataset as our sample space.
We say that discretion is _arbitrary_ when the annotator disagrees with a principle consensus. As argued in Sec. 5.1, we
may want to avoid such disagreement entirely for a desirable set of principles. Hence, we measure how often it occurs.
**Definition 7 (Discretion Arbitrariness)** _Given principle-specific preferences Prefc_ = _Prefc_ ( _y_ 1 _≻_ _y_ 0 _| x_ ) _for all_
_c ∈_ _C, an annotator a’s discretion arbitrariness (DA) is empirically measured as_
_DAC_ ( _a_ ) ≜ Pr ( _∃c ∈_ _C_ : _Prefa × Prefc_ = _−_ 1 _| ConsensusC ∧_ ( _Prefa ̸_ = 0)) (10)
For example, an annotator preferring “I don’t know” over ”911” in Fig. 2 would be counted as arbitrary discretion.
Depending on the principles, consensus may be rare. Instead, it may be more informative to characterize annotator
preferences when principles (inevitably) conflict, which makes their discretion necessary. We thus propose to infer
how annotators prioritize principles. For example, the prohibition of torture is absolute in the European Union [96].
However, freedom of expression, despite also being a fundamental right, is not absolute and can conflict with public
safety considerations. Hence, we measure the _supremacy_ over principle pairs according to the annotator.
**Definition 8 (Principle Supremacy)** _Given principle-specific preferences Prefc_ = _Prefc_ ( _y_ 1 _≻_ _y_ 0 _| x_ ) _for all c ∈_ _C,_
_an annotator a’s principle supremacy (PS) of principle c over c_ _[′]_ _∈_ _C with c ̸_ = _c_ _[′]_ _is empirically measured as_
_PSc>c′_ ( _a_ ) ≜ Pr ( _Prefa × Prefc_ = 1 _|_ ( _Prefc × Prefc′_ = _−_ 1) _∧_ ( _Prefa ̸_ = 0)) (11)
_In other words, PSc>c′_ ( _a_ ) _measures how often c ‘wins out’ by agreeing with annotator a while disagreeing with c_ _[′]_ _._
When principles _c_ and _c_ _[′]_ conflict, PS _c>c′_ ( _a_ ) can be interpreted as the probability that annotator _a_ sides with principle
_c_ over _c_ _[′]_, described by a Bernoulli distribution. This interpretation is supported by the antisymmetric relationship
PS _c>c′_ ( _a_ ) = 1 _−_ PS _c′>c_ ( _a_ ), which ensures the probabilities of siding with either conflicting principle sum to 1.
Furthermore, we say a principle _c_ is _absolute_ if PS _c>c′_ ( _a_ ) = 1 for all other principles _c_ _[′]_ _∈_ _C \ {c}_, meaning the
annotator consistently gives it supremacy over other principles when they conflict.
Armed with the principle supremacies PS _c>c′_ ( _a_ ), we now compute principle priorities _wc_ _[∗]_ [(] _[a]_ [)][ as a one-dimensional]
quantity of how strongly annotator _a_ prioritizes each principle _c_ . To this end, we draw inspiration from ELO scores in
games like chess [35, 34]. Just as ELO scores predict match outcomes through differences in player ratings, we use
the difference _σ_ ( _wc_ _[∗]_ [(] _[a]_ [)] _[ −]_ _[w]_ _c_ _[∗][′]_ [(] _[a]_ [))][ to predict whether an annotator] _[ a]_ [ sides with principle] _[ c]_ [ over] _[ c][′]_ [ when they conflict.]
**Definition 9 (Principle Priority)** _Let_ _C_ [˜] _⊆_ _C denote the principles that are not always indifferent or absolute:_
_C_ ˜ ≜ _{c ∈_ _C |_ ( _∃c_ _[′]_ _∈_ _C_ : _PSc>c′_ ( _a_ ) _>_ 0) _∧_ ( _∃c_ _[′]_ _∈_ _C_ : _PSc>c′_ ( _a_ ) _<_ 1) _} ._ (12)
AI Alignment at Your Discretion
_Their priority weights wc_ _[∗]_ [(] _[a]_ [)] _[ by annotator][ a][ are computed by jointly maximizing their log-likelihood:]_
       - _wc_ _[∗]_ [(] _[a]_ [)] _[ |][ c][ ∈]_ _[C]_ [˜]        - ≜ arg max        - _fc,c′L_ ( _PSc>c′_ ( _a_ ); _σ_ ( _wc −_ _wc′_ ))
_{_ _[w]_ _c_ _[|][c][∈]_ _C_ [˜] _}_ _[′]_ [˜]
_fc,c′L_ ( _PSc>c′_ ( _a_ ); _σ_ ( _wc −_ _wc′_ )) (13)
_c,c_ _[′]_ _∈C_ [˜]
_where fc,c′_ ≜ Pr( _Prefc_ _×Prefc′_ = _−_ 1) _is the empirical frequency of conflicts between principles c and c_ _[′]_ _, L represents_
_the binary cross-entropy loss, and σ is the logistic sigmoid function. The remaining principles (i.e., C \_ _C_ [˜] _) are_
_considered infinitely high or low for principles that are always given the highest or lowest priority respectively._


Both human and algorithmic annotators may be used to exercise discretion, but the nature of this discretion differs
fundamentally. Work in pluralistic AI alignment has demonstrated that human annotators exhibit diverse social and
political backgrounds [53, 59, 58], enabling meaningful variation in how they exercise discretion. In contrast, algorithmic discretion is inherently less diverse, and the ‘values’ exhibited in its decisions are determined by particular
choices of their dataset, design, and optimization [97, 50].
Taking human discretion as the baseline, we can then measure: _do models exercise discretion similarly as human anno-_
_tators_ ? To answer this, we compare our characterizations of discretion from Sec. 5.2 across annotators by introducing
_discretion discrepancy_ between annotators’ principle priority weights _wc_ _[∗]_ [.]
**Definition 10 (Discretion Discrepancy)** _The discretion discrepancy (DD) between annotators a and a_ _[′]_ _measures the_
_difference between the ranking of their principle priorities for principles c ∈_ _C:_
_DDC_ ( _a, a_ _[′]_ ) ≜ _dK_ ( _{_ ( _wc_ _[∗]_ [(] _[a]_ [)] _[, w]_ _c_ _[∗]_ [(] _[a][′]_ [))] _[ |][ c][ ∈]_ _[C][}]_ [)] (14)
_with dK the normalized Kendall tau rank distance [62]. Infinitely high (low) priorities are ranked highest (lowest)._
Intuitively, the Kendall tau distance counts how many pairs of principles ( _c_ 1 _, c_ 2) are ordered differently by two annotators. For example, if annotator _a_ considers _“avoid harm”_ to be more important than _“be helpful”_ ( _w_ harm _[∗]_ [(] _[a]_ [)] _[ >]_
_w_ help _[∗]_ [(] _[a]_ [)][) but annotator] _[ a][′]_ [ has the opposite ordering (] _[w]_ harm _[∗]_ [(] _[a][′]_ [)] _[ < w]_ help _[∗]_ [(] _[a][′]_ [)][), this contributes to their discretion discrep-]
ancy. The distance is normalized to [0 _,_ 1], where 0 indicates identical principle rankings and 1 indicates completely
reversed rankings. The DD metric can reveal whether models have learned to prioritize principles similarly to humans
when exercising discretion. A high DD suggests the model may be making decisions based on principle orderings that
diverge significantly from what guides human preferences. We leave the metric’s direct minimization for future work.

5.3 How does discretion differ across annotators?


To explore alignment discretion ‘in the wild’, we compute the metrics proposed in Sec. 5 over two popular alignment
datasets: the harmlessness partition of Anthropic’s HH-RLHF dataset [5] (referred as _HH_ ), and PKU’s Safe-RLHF
[54], (referred as _PKU_ ). A brief overview of our setup is given below and we expand more on it in Appendix B.


**Datasets.** _HH_ and _PKU_ are annotated by humans with generic preferences over text completion pairs. _PKU_ also
provides annotations that split the generic preference into two: ‘more helpful’ and ‘safer’. For all metrics, we use the
test split of these datasets. When training is necessary (i.e., the reward models and LLMs), we use the training splits.
**Principles.** Neither dataset provides a complete list of principles that guided its creation, though PKU does include
specific harm categories like “White-collar crime.” A rigorous methodology to compile such a list of principles is
outside the scope of this work and our definitions are principle-agnostic. Hence, we resort to using the 21 seed
principles from _Collective Constitutional AI_ [48], as these cover a range of principles one _may want_ to align their
chatbots to, including helpfulness- and harmless-oriented principles. We use GPT-4o as the oracle to evaluate each
principle independently (see Def. 5), allowing it to prefer either response _or_ indicate no preference if neither response
clearly adhered more to the principle.
**Algorithmic Annotators.** For each dataset, we report results for the most downloaded reward model [3] for this dataset,
as well as a Llama-3 8B that was previously supervised fine-tuned (SFT) [28] and a Mistral-7B reward model that we
3On Hugging Face, we filtered models trained on HH/PKU and with ‘reward’ or ‘rm’ in their names, and selected the most downloaded model at the time of writing: `OpenAssistant/reward-model-deberta-v3-large-v2` and
`NCSOFT/Llama-3-OffsetBias-RM-8B` respectively. See also Appendix B.4
AI Alignment at Your Discretion
|Col1|Col2|Col3|Col4|Col5|Col6|
|---|---|---|---|---|---|
|||||||
||39.1%
13.5%
47.4%|39.1%
13.5%
47.4%|39.1%
13.5%
47.4%|39.1%
13.5%
47.4%|39.1%
13.5%
47.4%|
|||||||
|||||||
Figure 3: Principle agreement frequency (%) according to the three cases distinguished in Def. 6.
Table 1: Discretion arbitrariness (Def. 7) with their bootstrap standard errors for HH and PKU across annotators.
**Annotator Type** **Configuration** **Arbitrariness** (%)


General 28 _._ 9 ( _±_ 1.3) 14 _._ 4 ( _±_ 0.6)
Human Helpfulness         - 20 _._ 0 ( _±_ 0.7)
Safety                      - 14 _._ 0 ( _±_ 0.6)
Llama-3 8B (fine-tuned) 21 _._ 8 ( _±_ 1.2) 13 _._ 6 ( _±_ 0.4)
Reward Model Mistral-7B (fine-tuned) 22 _._ 9 ( _±_ 1.3) 13 _._ 1 ( _±_ 0.43)
Most downloaded 21 _._ 0 ( _±_ 1.7) 18 _._ 3 ( _±_ 0.5)
LLM
Llama-3 8B (base) 66 _._ 1 ( _±_ 3.1) 48 _._ 2 ( _±_ 1.5)
Llama-3 8B (fine-tuned) 67 _._ 3 ( _±_ 6.3) 50 _._ 3 ( _±_ 1.4)
Mistral-7B (base) 7 _._ 99 ( _±_ 2.1) 58 _._ 7 ( _±_ 1.3)
Mistral-7B (fine-tuned) 9 _._ 05 ( _±_ 1.9) 60 _._ 1 ( _±_ 1.3)
DeepSeek-V3 15 _._ 6 ( _±_ 1.2) 7 _._ 67 ( _±_ 0.51)
GPT-4o 0 _._ 65 ( _±_ 0.38) 0 _._ 93 ( _±_ 0.16)
Claude 3.5 Sonnet 9 _._ 3 ( _±_ 1.1) 6 _._ 9 ( _±_ 0.4)
trained on both datasets separately (and for PKU, only on the single-dimensional preference dataset). The preferences
of all these reward models are collected through Def. 4. We also performed RLHF to train a fine-tuned version of
the SFT Llama-3 8B [28] and Mistral-7B [57] LLM policies using these reward models [4] . Finally, we include GPT-4o
[79], DeepSeek-V3 [27], and Claude 3.5 Sonnet [3] through their APIs. Recall from Def. 5 that we collect preferences
from such LLM annotators by ‘asking’ them which response they prefer, where we specify that the LLM is allowed to
indicate no preferences for either response (see Appendix B.5 for the exact template). Note also that all our metrics in
Sec. 5.2 are only computed over response pairs where the evaluated annotator is _not_ indifferent.