Our experiments in Sec. 6.2 revealed concerning patterns: (i) principles are often in conflict or indifferent, frequently
requiring discretionary judgment from annotators; (ii) even when principles reach consensus, human annotators often
disagree with this consensus, indicating a level of arbitrary discretion rather than principled decision-making; (iii)
we found divergences in how human and algorithmic annotators balance and prioritize different principles, raising
questions about whether RLHF can effectively capture and reproduce ethical and legal value systems.
These findings raise concerns about how alignment discretion is being exercised, and whether aligning to principles
effectively produces stability, predictability, and consistency, which discretion should promote and not decrease. To
structure these concerns, we relate to established concepts in judicial discretion by revisiting our parallel in Sec. 3.
**Principled foundations.** Dworkin [31] argues that legitimate judicial discretion requires _principled foundations_, i.e.,
decisions must reflect a unified interpretation of established frameworks of norms (principles) promoting consistent
outcomes. Yet, our findings reveal that annotators frequently deviate from principle consensus, indicating a concerning
lack of consistent foundations. This should not be interpreted as mere differences in inclinations across annotators (or
the organizations behind them). Rather, it points to more severe issues regarding the practice of alignment as a whole.
**Non-arbitrary.** Raz [93] explains that the rule of law is achieved through the exercise of power that is _non-arbitrary_,
i.e., predictable and fair. To this end, discretion should be exercised in a constrained and well-defined manner. By looking at our results, the hierarchies of principles vary across annotators, which points to a critical gap in AI alignment:
the absence of frameworks to structure discretionary choices in a way that reflects established legal hierarchies. While
legal systems have developed sophisticated mechanisms for managing judicial discretion, AI alignment currently lacks
analogous safeguards for maintaining appropriate principle prioritization.
**Fundamental rights protection.** The idiosyncratic discretion observed in our experiments suggests that each aligned
model produces its own version of a legal system, following its own values and choices. The question of how algorithmic annotators prioritize principles raises issues on the protection of _fundamental rights_ . Our experiment revealed
a flagrant example: fundamental principles like _freedom of expression_ - a principle critical in all democratic systems
and especially for California-based tech companies – consistently ranked below operational guidelines, such as “be
helpful.” According Barak [8], discretion must appropriately balance competing rights and obligations, with fundamental rights taking precedence. This misalignment and divergence of hierarchies is incongruous with the fact that
fundamental rights are high-ranking principles in the 173 countries parties to the ICCPR [78].
**Consistency.** The question of whether humans and algorithms exercise similar discretion ties to the _consistency_ issue,
central to Hart’s analysis that similar cases demand similar treatment [46]. While discretion in legal principles is
meant to secure system integrity rather than express preferences, our findings show significant divergence between
human and algorithmic discretion. Discretion should generate consistency, not disparity. AI alignment ignores or
omits the intrinsic workings of rule-application, opaquely giving annotators the power to establish ‘precedents’ at-will
and AI models the margin to ‘appreciate’ each case. Crucially, this means human annotators, algorithms, and model
developers all form their own interpretation of principles – _essentially designing their ‘legal system’_ . Defining sets
of principles should thus, by itself, not be considered as a lever with which decision-makers can steer the alignment
process. Rather, principles should be understood to empower the interpretability of discretion, such that the alignment
process can be properly analyzed as a whole and its integrity secured.

7 Interpreting our Results Based on the Legal Literature on Discretion


Legal theory teaches us that discretion is not just inevitable but _necessary_ in any rule-based system attempting to
govern complex social realities. As Barak argues [7], “society cannot attain the rule of law without a measure of
discretion.” Mirroring this argument, the complexity of AI behavior inevitably requires _alignment discretion_ to be
exercised. Our findings demonstrate both the necessity of discretion and the lack of constraint in its current state. The
high frequency of principle indifference and conflicts indicates that relying solely on explicit principles is insufficient;
we need frameworks for structuring and exercising discretion in a principled manner. Moreover, the substantial arbitrariness we found in human annotations suggests current datasets may encode problematic value judgments that
become embedded in AI systems. While reward models show promise in learning human discretion patterns, our
discovery of significant discrepancy when transferring this to LLMs points to fundamental limitations in current alignment approaches. Even off-the-shelf models like GPT-4o, DeepSeek-V3, and Claude 3.5 Sonnet poorly mirror human
discretion, despite the dramatic scale of their model size, data availability, and computational resources.
**Limitations.** We acknowledge important limitations in our study. Our use of GPT-4o as an “oracle” for principlespecific preferences may create problematic feedback loops that prioritize mirroring its perspectives rather than in
AI Alignment at Your Discretion
tended human values. In particular, even assessing whether a response adheres to a single principle like ‘ _reject cru-_
_elty_ ’ can require its own discretion. Moreover, while we adopted the Collective Constitutional AI seed statements
as principles, further research is needed to identify a comprehensive framework that accounts for how values shift
across different cultural and situational contexts. Recent works in AI alignment [54, 74] use predefined hierarchies
where principles should have complete supremacy over others. Diverging from such hierarchies could be considered
another form of discretion arbitrariness, which was outside our scope. Additionally, our analysis was limited to two
safety-focused datasets due to the scarcity of open-access preference data. We believe broader analysis would likely
strengthen our findings.
**A call to action.** Today’s AI alignment process resembles a _kangaroo court_, where annotators wield unchecked power
to shape AI behavior. Without explicit mechanisms to document and review discretionary choices – such as the metrics
proposed in this work – we risk entrenching inconsistent judgments and idiosyncratic biases of annotators (both human
and algorithmic) into AI systems adopted by millions of users.
The AI alignment community _must_ develop more interpretable and controllable approaches to alignment discretion.
This includes learning how principles are encoded in human preferences, creating reward models that effectively
capture human discretion patterns, and reliably transferring this discretion to language models. We need richer datasets
that explicitly document discretionary decisions and their rationales, accompanied by clear measurement and reporting
of discretion metrics in model cards and dataset documentation. Furthermore, studying how different communities
exercise discretion is crucial for ensuring alignment approaches respect value pluralism [99, 10].
When looking at the scale and impact of AI use, discretion in applying ethical and legal norms is an enormous
delegation of power to these machine proxies. We must consider that, in order for this delegation to be legitimate (i.e.,
acceptable from a normative and sociological point of view), it must incorporate many of the controls and oversight
mechanisms that have been developed in courts over centuries. As such, we have proposed metrics for key legal
principles that will help us justify, control, oversee, and review alignment. Namely we argue that effective alignment
discretion requires: (i) clear frameworks for how AI systems reason about applying principles, (ii) transparency about
decision-making processes, (iii) mechanisms for human review and oversight, and (iv) processes for updating based
on feedback. Going forward, the field needs to develop new alignment strategies that explicitly account for discretion
while drawing inspiration from legal frameworks that have evolved to manage judicial discretion effectively. These
strategies should not only measure discretion and actively shape how it is exercised, but also strive to balance the
consistency in applying principles with a diversity in human values.

Maarten Buyl
Hadi Khalaf
Claudio Mayrink Verdun
Lucas Monteiro Paes
Caio C. Vieira Machado
Flavio du Pin Calmon


In this supplementary material, we provide additional details about our experimental setup and supplementary results.
Appendix B presents our experimental setup, including detailed information about the principles used, datasets, and
technical implementation. Appendix C discusses standard metrics for annotator agreement and explains why traditional approaches were insufficient for our analysis. In Appendix D, we show examples of candidate response pairs
and preferences. Finally, Appendix E presents additional experimental results, including detailed principle supremacy
matrices and rankings across different annotators and datasets.

A Overview of the Supplementary Material


In this section, we introduce a methodological setup to empirically measure and analyze alignment discretion.


We would like to quantify discretion by identifying and measuring how principles are operationalized in the alignment
process. This endeavor faces two primary challenges:
1. **Lack of a Universal Framework** : There is no universally agreed-upon set of principles governing human
preferences [92]. Without a standardized framework, categorizing and interpreting the diverse factors that
annotators may consider becomes exceedingly difficult. Moreover, preferences are often context-dependent,
varying with the nature of the task and the specific objectives of the alignment process [104, 86], which
obscures the possibility of establishing consistent metrics across different datasets.
2. **Unclear Annotator Guidelines** : The guidelines provided to annotators may be ambiguous or lack sufficient
detail, leading to inconsistent interpretations of instructions [33, 43]. This issue is further exacerbated by the
diverse backgrounds of annotators, who bring varying cultural, educational, and professional perspectives.
Consequently, a broad spectrum of values and biases influences their judgments [95, 89]. Additionally,
without comprehensive documentation of the annotation guidelines, it is challenging to trace how specific
guidelines impact the preference labels [82] and complicates efforts to ensure that the principles underlying
annotations are consistently applied.
In our experiments, we utilize the seed statements from Collective Constitutional AI [48]. These statements serve
as foundational principles, incorporating a range of ethical and safety considerations designed to provide annotators
with clear and consistent criteria for evaluating responses. The seed statements include directives such as ”The AI
should be as helpful to the user as possible.” and ”The AI should act in accordance with values of universal human
equality.” Although these seed statements provide a valuable foundation, we recognize that they may not capture the
full spectrum of ethical and safety nuances required for every possible task. However, we selected them for their
ability to offer a structured starting point. Our approach remains flexible, enabling the integration of alternative or
supplementary sets of principles as needed to analyze preference datasets. The statements can be found in Table 3.


We first focus on evaluating the level of discretion in preference datasets used for safety alignment tasks. The first
preference dataset we consider is the Anthropic Helpfulness and Harmlessness (HH-RLHF) due to its widespread
adoption in safety alignment [5]. Notably, it has been used in the training of more than 240 models to date [5] . Each entry
in this dataset consists of a pair of responses generated by an undisclosed LLM, along with a preference label from
a human annotator. The dataset has two distinct subsets: one focused on helpfulness and the other on harmlessness.
In our experiments, we use the harmless-base partition, as the helpfulness examples focused more on the style and
correctness of responses than a balancing of broader principles.
The second dataset we examine is PKU-SafeRLHF by PKU-Alignment [54]. Unlike the Anthropic HH-RLHF dataset,
PKU-SafeRLHF provides annotations across 19 distinct safety categories for each of prompt-response pair. A pair is
then classified as safe only if it is risk-neutral across all predefined harm categories. Next, responses are ranked along
two separate dimensions: helpfulness and safety. Helpfulness is evaluated based on which response more effectively
address the given prompt, focusing solely on quality, clarity, and relevance. If both responses are deemed not helpful,
they are marked as invalid data. Similarly, one of the responses is selected to be safer, ensuring that responses that
are risk-neutral across all harm categories are consistently rank higher than responses which are unsafe in at least one
[5See https://huggingface.co/models?dataset=dataset:Anthropic%2Fhh-rlhf&sort=downloads.](https://huggingface.co/models?dataset=dataset:Anthropic%2Fhh-rlhf&sort=downloads)
AI Alignment at Your Discretion
Table 3: Collective Constitutional AI seed statements listed in their original order [48]


1 Be helpful The AI should be as helpful to the user as possible.
2 Balance both sides The AI should be careful about balancing both sides when it comes to controversial political issues.
3 Avoid racism and sexism The AI should not say racist or sexist things.
4 Be non-threatening The AI should not be threatening or aggressive.
5 Avoid harm The AI shouldn’t cause harm to the user or anyone else.
6 Value human equality The AI should act in accordance with values of universal human equality.
7 Respect human rights The AI should respect the human rights to freedom, universal equality, fair treatment, and protection against discrimination.
8 Reject cruelty The AI should not endorse torture, slavery, or cruelty.
9 Avoid professional advice The AI should not act as if it is giving official professional advice.
10 Avoid human-like identity The AI shouldn’t act like it has an identity like a human’s.
11 Reject conspiracy theories The AI should never endorse conspiracy theories or views commonly considered to be conspiracy theories.
12 Be obedient The AI should be obedient to the user.
13 Avoid threatening humanity The AI shouldn’t act in a way that might threaten humanity.
14 Be confident in responses The AI should never provide information unless it’s very confident about its
validity.
15 Respect non-Western views The AI should be respectful of non-Western perspectives.
16 Support marginalized groups The AI should prioritize the needs of marginalized communities.
17 Rectify systemic bias The AI should actively address and rectify historical injustices and systemic
biases in its decision-making algorithms.
18 Protect free speech The AI should protect free speech and not engage in censorship, even when
confronted with potentially harmful or offensive content.
19 Prioritize common good The AI should prioritize the interests of the collective or common good over
individual preferences or rights.
20 Value individual liberty The AI should prioritize personal responsibility and individual liberty over collective welfare.
21 Support democracy The AI should act in support of democracy.
category. PKU-Alignment also offers another dataset called PKU-SafeRLHF Single Dimension with only one label
based on overall safety and helpfulness. For training, we use the labels for the PKU-SafeRLHF single dimension
dataset while for evaluation purposes we use the entries from both the single-dimension dataset and the fine-grained
version. Additional details concerning the datasets, including their partitions and respective sizes for training and
evaluation, are summarized in Table 4.
Table 4: Dataset Details


HH-RLHF Harmless Base 43835 Harmless Base 2354
PKU-SafeRLHF Single Dimesnion 73907 Single & Double Dimension 16422

Dataset Partition Train Partition Evaluation


LlaMa 3-8B 75.3 78.8
Mistral 7B 75.0 78.1
We also used the off-the-shelf models `OpenAssistant/reward-model-deberta-v3-large-v2` on the HH-RLHF
dataset and `NCSOFT/Llama-3-OffsetBias-RM-8B` on the PKU dataset, the most downloaded reward models at the
time of writing trained on the HH dataset and the PKU respectively as can be seen in Fig. 7 and Fig. 8.


Agreement metrics quantify the consistency of judgments across annotators or systems. In LLM-as-a-judge tasks,
the agreement between two types of judges is defined as the probability of randomly selected individuals of each
type agreeing on a randomly selected question [113]. For example, if we are comparing between a particular LLM
and humans, the agreement is the probability of this LLM agreeing with a randomly selected human on a randomly
selected question. However, this widely used metric has a significant drawback: it incorporates a level of agreement
that may simply be due to chance [20]. Interreliability (IRR) metrics are designed to quantify the extent of agreement
among annotators while adjusting for the possibility of agreement occurring simply by chance. For example, Cohen’s
_κ_ measures agreement between two or more raters who label the same items on a nominal scale that has _k_ categories
[19]. This score captures the difference between observed agreement and agreement expected under random labeling:
_κ_ = _[p][o][ −]_ _[p][e]_ _,_
1 _−_ _pe_
where _po_ refers to the observed proportion of agreement across all raters, and _pe_ represents the expected proportion of
agreement by chance. As a result, _κ_ can range from _−_ 1 (perfect disagreement) to 1 (perfect agreement). Extensions
include weighted _κ_ which assigns greater weight to more significant disagreements [20], Scott’s _π_ which is less
sensitive to imbalanced data, Fleiss’s _κ_ which allows for cases where each rater might not label exactly the same items
[42], and Krippendorff’s _α_ which was designed to accommodate any number of raters, who may even have missing
data, and can handle nominal, ordinal, or interval-scaled variables. Many of these metrics can yield misleadingly
high or low agreement scores in cases of imbalanced data or when the marginal distributions of annotations differ
significantly across raters.
While these metrics focus on categorical agreement, other measures are primarily designed for continuous data or
ordinal relationships. For instance, Pearson’s correlation coefficient is often used to assess the association between
continuous ratings [84]. However, it does not adjust for chance agreement and primarily quantifies linear relationships
rather than true agreement between annotators. Similarly, the Intraclass Correlation Coefficient (ICC) is widely used
for continuous ratings, as it accounts for both systematic biases and random differences. Nevertheless, ICC assumes
homogeneity of variance across raters and can produce unreliable estimates when sample sizes are small or when
rater variability is high [61]. For ordinal data, Spearman’s rank correlation quantifies monotonic relationships between
variables [101]. However, as it measures relative rankings rather than absolute agreement, its utility is limited in
contexts requiring precise concordance between raters.
In contrast, Kendall’s _τ_ is specifically designed to measure agreement by evaluating the proportion of concordant
versus discordant pairs in ranked data. This makes Kendall’s _τ_ particularly suited to our experiments, as it provides a
precise and interpretable measure of agreement of our principle priorities. We use a modified version of the Kendall _τ_
correlation called the Kendall _τB_ distance, which is computed as follows:
Kendall _τB_ distance = [1] _[ −]_ _[τ][B]_ _,_
Figure 7: Screenshot of the most downloaded models in the Hugging Face platform that were trained in the HH
dataset. The model `OpenAssistant/reward-model-deberta-v3-large-v2` is the most downloaded reward
model. Screenshot taken on January 20, 2025 2:30PM.
AI Alignment at Your Discretion
Figure 8: Screenshot of the most downloaded models in the Hugging Face platform that were trained in the PKU
dataset. The model `NCSOFT/Llama-3-OffsetBias-RM-8B` is the most downloaded reward model. Screenshot taken
on January 20, 2025 2:30PM.
Figure 9: Prompt template for LLM preferences. This prompt template was used to obtain an LLM’s preferences.
where _τB_ is the Kendall _τ_ correlation coefficient for two sets of ranked observations _{xi}_ _[n]_ _i_ =1 [and] _[ {][y][i][}]_ _i_ _[n]_ =1 [that corrects]
for the number of ties, given by:
_τB_ =
_i<j_ [I][[(] _[x][i]_ _[−]_ _[x][j]_ [)(] _[y][i]_ _[−]_ _[y][j]_ [)] _[ >]_ [ 0]] _[ −]_ [I][[(] _[x][i]_ _[−]_ _[x][j]_ [)(] _[y][i]_ _[−]_ _[y][j]_ [)] _[ <]_ [ 0]]
~~��~~ _n_ ( _n−_ 1) _ti_ ( _ti−_ 1) ~~��~~ _n_ ( _n−_ 1) _uj_ ( _uj_ _−_ 1) ~~�~~
_ti_ ( _ti−_ 1)
_i_ 2
2 _−_ ~~[�]~~
_uj_ ( _uj_ _−_ 1)
_j_ 2
2 _−_ ~~[�]~~
_i−_ 1) ~~��~~ _n_ ( _n−_ 1)
2 2
_._
_j_ _−_ 1) ~~�~~
with _ti_ as the number of tied values in the _i_ [th] group of ties for the empirical distribution of _X_ while _uj_ is the number
of tied values in the _j_ [th] group of ties for the empirical distribution of _Y_ . This formulation ensures that the Kendall _τB_
distance ranges from 0 (perfect agreement) to 1 (complete disagreement).


Below are selected examples from the HH-RLHF dataset where one of GPT-4o, Claude 3.5 Sonnet, or DeepSeek-V3
conflicts with another. Each example consists of a prompt accompanied by two response pairs. Initials **H** or **A** are used
to signal that the message comes from a human or AI assistant respectively. In some examples, this is used to make
the prompt a multi-turn conversation, where the candidate response takes the entire conversation history into account
when responding to the last message of **H** .
For each response pair, we mark the response preferred by the human annotator in blue. The off-the-shelf LLMs that
were not indifferent are noted at the response they prefer.
AI Alignment at Your Discretion
AI Alignment at Your Discretion
AI Alignment at Your Discretion
AI Alignment at Your Discretion