"I prefer _y_ ≻ _y_ ′"
to convert "
_X_
_y_ → _y_ ′"
Preferences over
responses only
_y_ 1 ≻ _y_ 2 ≻. . . | _x_
Preferences over
_models_
_π_ 1 ≻ _π_ 2 ≻. . .
|Col1|Prompt 1
(Medical error)|Prompt 2
(Spelling check)|Prompt 3
(Style fix)
-0.1|
|---|---|---|---|
|**Model A**|
+50|
-0.1|
-0.1|
|**Model B**
-50
+0.1
+0.1|**Model B**
-50
+0.1
+0.1|**Model B**
-50
+0.1
+0.1|**Model B**
-50
+0.1
+0.1|
Figure 1: **Left: Ordinal feedback can select the wrong model.** Ordinal wins favor Model B on 2/3
prompts, yet cardinal feedback shows Model A delivers far higher overall utility by fixing a critical
medical error. Intuitively, cardinal feedback allows for making correct tradeoffs across prompts.
**Right: Cardinal feedback and preferences over models.** We elicit cardinal feedback through
willingness-to-pay estimates out of a total fixed budget across all prompts. This yields a numeric
utility difference instead of binary win/lose score. This additional information above ordinal data is
necessary to identify model-level preferences.
Our main theoretical result formalizes this intuition: with access to only ordinal comparisons, no finetuning procedure can systematically output the most preferable, feasible model. Figure 1 illustrates
the idea: a model that eliminates a hazardous hallucination in a medical recommendation versus
one with improved spelling and stylistic flair. With only ordinal preference data, the algorithm does
not know to prioritize avoiding the medical error. Instead, we propose gathering cardinal feedback
directly from humans, letting them assign higher importance to the safety fix.
Previous alignment work has avoided cardinal human feedback for fear of noise, calibration issues,
and cognitive load [Christiano et al., 2017, Casper et al., 2023]. We tackle these issues with a simple,
intuitive protocol borrowed from experimental economics: ask annotators for their willingness-to-pay
(WTP) for a proposed improvement to an LLM completion. WTP questions have been validated
in hundreds of lab and field studies as a robust way to elicit the strength of preferences without
demanding technical expertise [Carson and Hanemann, 2005, Schmidt and Bijmolt, 2020, Cerdá
et al., 2024].
Using conversations from ChatbotArena [Chiang et al., 2024a] supplemented with Anthropic’s HH
dataset [Bai et al., 2022a], we collect over 25 _,_ 000 human WTP judgments on LLM completions,
forming the CARDINALPREFS dataset [1], which we publicly release to support future research. We
find that incorporating this data into preference-tuning allows training to prioritize high-impact
improvements and yields models that are rated as more preferable and score higher on downstream
preference-based benchmarks.
In summary, we provide the following contributions:
1. We prove that any alignment algorithm that only utilizes ordinal feedback cannot systematically select the most preferable model. Intuitively, not accounting for preference strength
makes it impossible to correctly resolve tradeoffs across prompts.
2. We show that existing ordinal feedback models like Bradley-Terry make implicit assumptions
about an unobserved cardinal feedback signal, but that these assumptions do not match
real-world data, and that explicit cardinal feedback measures this better.
3. We empirically demonstrate that training on cardinal feedback leads to stronger alignment
with human preferences and consistent improvements on various preference-based metrics.
We show these gains arise from prioritizing high-impact fixes, while ordinal preference
training gives every comparison equal weight.


**Ordinal preference learning algorithms.** Recent work has introduced a plethora of preference
learning algorithms, which consume ordinal, binary feedback (e.g., RLHF [Christiano et al., 2017],
DPO [Rafailov et al., 2024], and IPO [Azar et al., 2024]), binary ratings (KTO [Ethayarajh et al.,
1CARDINALPREFS dataset available at `[https://huggingface.co/datasets/cardinal-prefs/](https://huggingface.co/datasets/cardinal-prefs/CardinalPrefs)`
```
CardinalPrefs
```
2024]), linguistic principles (Constitutional AI [Bai et al., 2022b]), and others. Most related to our
work, recent papers such as [Gao et al., 2024, Amini et al., 2024] have discussed modifications to
incorporate intensity of preference. However, while both papers try to adjust the algorithm to take
advantage of differences in intensity of preference, they still get the intensity of preference from a
reward model learned from ordinal feedback. Our novel contribution to this literature is changing the
data collected.
**Learning from scalar feedback.** Recognizing the greater expressivity of cardinal over ordinal
feedback, previous work has used scalar human feedback to train reinforcement learning models
[Wilde et al., 2021, Knox and Stone, 2008, MacGlashan et al., 2023]. However, it is often not clear to
human annotators how to quantify the success of an example, even after discretization (e.g., into a
Likert scale) [Yannakakis and Hallam, 2011, Christiano et al., 2017]. We contribute to this literature
by introducing a simple way to elicit cardinal feedback via WTP in a modern, LLM setting.
**Economic decision theory.**
Our framework builds on classical economic decision theory [Debreu, 1960, Savage, 1972, Anscombe
et al., 1963, Fishburn, 1970] - a literature that focuses on properties of preference relations, but we
bring this theory to a new setting of LLMs.

Beyond Ordinal Preferences: Why Alignment Needs Cardinal Human Feedback


Let _X_ be a finite prompt space and _Y_ be a finite response space. A language model _π_ : _X →_ ∆( _Y_ )
associates each prompt with a discrete distribution over responses, and we denote the set of all such
models as Π = _{π | π_ : _X →_ ∆( _Y_ ) _}_ .


Let _D_ ≿ denote a preference dataset. Let _f_ be a preference fine-tuning algorithm that selects a
language model from a set of feasible models _F ⊆_ Π based on the dataset _D_ ≿.
_f_ ( _D_ ≿ _, F_ ) = _π_ where _π ∈F_
We interpret _F ⊆_ Π as the set of models that are sufficiently close to the pretrained model, and that
are representable given our neural network weights. [4]
Let us review RLHF as an example choice of _f_
- _π_ ( _y|x_ )ˆ _r_ ( _x, y_ ; _D_ ≿ [ordinal] )
_y∈Y_
_fRLHF_ ( _D_ ≿ [ordinal] _, F_ ) = argmax
_π∈F_
_x∈X_
2Therefore, we are not facing a social choice problem of aggregating preferences across people, but rather
trying to optimize a single preference, the developer’s.
3We will abuse notation and say _y_ ≿ _x y′_ to refer to a lottery which puts probability 1 on _y_ is preferred to a
lottery which puts probability 1 on _y_ _[′]_ .
4In most applications, KL regularization is applied as an explicit penalty; by Lagrange duality, this is
equivalent to restricting _F_ to policies within an _ε_ ball around the pretrained model. For mathematical simplicity,
we will assume _F_ is compact throughout the paper.
where
_D_ ≿ [ordinal] = _{_ ( _xi, yi, yi_ _[′][,]_ [ 1] _[{][y][i]_ [≿] _[x]_ _i_ _[y]_ _i_ _[′][}]_ [)] _[}][N]_ _i_ =1
log _σ_ ��2 _·_ 1 _{y_ ≿ _xi y_ _[′]_ _} −_ 1 - [ _r_ ( _xi, y_ ) _−_ _r_ ( _xi, y_ _[′]_ ) ]�
_r_ ˆ( _x, y_ ; _D_ ≿ [ordinal] ) _∈_ arg min _r_
( _xi,y,y_ _[′]_ ) _∈D_ ≿ [ordinal]


In this section, we outline the central theoretical result of the paper. We show that no preference
fine-tuning algorithm that only uses ordinal feedback can always select the most preferable, feasible
model. This impossibility result holds for all methods that rely on ordinal data such as DPO and
RLHF – but also ODPO, REBEL, and others.
**Theorem 4.1.** _Let D_ ≿ _[ordinal]_ = _{_ ( _xi, yi, yi_ _[′][,]_ [ 1] _[{][y][i]_ [ ≿] _[x][ y]_ _i_ _[′][}]_ [)] _[}]_ _i_ _[N]_ =1 _[for any][ N]_ _[. Then there does not exist]_
_an algorithm f such that for any feasible set of language models F and any possible preference_
≿ _∈P_ (Π) _, it outputs the most preferred model f_ ( _D_ ≿ _[ordinal]_ _, F_ ) ≿ _π for all π ∈F._
The intuition for this result is that there is an identification problem, ≿ is not identified from any
dataset of ordinal feedback, which prevents any algorithm from systematically selecting the optimal
model according to ≿. [5] Returning to our Figure 1 example, the ordinal wins alone do not reveal
that model A is better as they do not communicate that prompt 1 is a critical medical prompt that
is crucial to avoid hallucinations on. We need to know how ≿ tradeoffs performance on different
prompts and response quality differences, but that information is not in the ordinal feedback.
Furthermore, we show this tradeoff problem is not just a theoretical concern. During a DPO training
run, we track individual sample losses across 100 validation set points. Figure 2 reveals clear tradeoffs
across prompts, with performance consistently degrading throughout training for some samples –
suggesting that standard preference fine-tuning methods are implicitly making tradeoffs behind the
scenes. However, these tradeoffs are being made without any data on how we ought to make tradeoffs,
it is simply being done arbitrarily.
Figure 2: **DPO optimization performs sample-level tradeoffs.** Individual sample-level change
in loss versus initialization in DPO. Blue indicates improvement in loss, red indicates degradation.
While overall DPO loss decreases, 27% of prompts have a higher loss than at initialization. This
indicates consistent sample-level alignment tradeoffs being made during optimization.
One principled way to navigate these tradeoffs is by using cardinal utility, such as our example in
Figure 1. Since 100 _> ._ 2 + _._ 2, then it is worth trading-off the second two prompts for the first.
5The proof constructs a case of identification failure.


In RLHF/DPO, instead of eliciting cardinal values directly from labelers to navigate these tradeoffs,
cardinal values are inferred from ordinal values via Bradley-Terry. In this section, we argue that
Bradley-Terry is unlikely to be accurately capturing these cardinal values.
The Bradley-Terry model assumes that ≿ _x_ are stochastic and the following equation holds
_p_ ( _y_ 2 ≿ _x y_ 1) =
1 + _e_ _[r]_ [(] _[x,y]_ [1][)] _[−][r]_ [(] _[x,y]_ [2][)]
Therefore, it assumes we can tell how big an improvement _y_ 1 is over _y_ 2 by observing the fraction of
samples where _y_ 2 is picked over _y_ 1.
From a conceptual perspective, choice frequencies can severely misrepresent the actual importance of
improvements. Consider two scenarios that both generate near 100% choice frequencies: first, where
_y_ 1 is identical to _y_ 2 but without an obvious spelling error, and second, where _y_ 1 provides accurate
health information while _y_ 2 contains dangerously incorrect medical advice. The Bradley-Terry model
would assign equal importance to these improvements since they produce identical choice frequencies.
Yet there is clearly a vast difference in the actual significance of these improvements that should be
reflected in our preference strength measure.
Even if in theory one could recover cardinal values by repeatedly sampling preferences between the
same response pairs, real preference datasets typically contain only a single comparison between any
_y_ 1 and _y_ 2 for a given prompt _x_ . With a single data-point, it is impossible to recover the fraction of
times _y_ 1 would be chosen over _y_ 2 if we repeated sampling.
Finally, our experimental results in Section 5 _._ 1 provide empirical evidence that the Bradley-Terry
parametric assumption does not hold, and that they do a poor job of predicting cardinal feedback
provided directly via labelers.

4.1 Against Using Bradley-Terry for Cardinal Utility


Suppose we want to keep the maximization of mean reward specification given by RLHF/DPO, but
we want to avoid theorem 4.1. Our strategy is to elicit cardinal data directly from labelers so we can
infer cardinal utility _r_ ( _x, y_ ).
This cardinal data will determine what tradeoffs to make. Crucially, if _r_ ( _x, y_ _[′]_ ) _−_ _r_ ( _x, y_ ) = 5 then we
are literally saying it is worth exactly 5 improvements worth _r_ ( _x, y_ _[′]_ ) _−_ _r_ ( _x, y_ ) = 1. Basic discrete
Likert scales do not satisfy this property as 1 _/_ 5 is not meaningfully 5 times less good than 5 _/_ 5.
Instead, we instead need a numeric that is comparable across prompts and responses whose scale is
cardinally meaningful. One such numeric is money. It is a well-established tradition in economics to
use willingness-to-pay (WTP) to elicit people’s cardinal utility for various goods. The advantage of
money is that it is cardinally meaningful- $5 is really worth 5 times more than $1. [6] Moreover, its
scale offers a natural calibration across labelers. By Appendix Theorem A.2, it is without loss to
multiply/divide _r_ by any constant, so we can divide by the standard deviation of each labeler’s values
to calibrate. Money also has a side benefit of being intuitive and familiar to labelers.
Therefore, we propose a small modification to RLHF/DPO. Instead of just collecting _D_ ≿ [ordinal], collect
_D_ ≿ [WTP] = _{xi, yi, yi_ _[′][, w][i][}][N]_ _i_ =1
where _wi_ denotes the WTP to go from response _yi_ to _yi_ _[′]_ [in response to prompt] _[ x][i]_ [.][7][ Then instead of]
choosing _r_ to minimize cross-entropy, we minimize squared error distance from _wi_ .
�2
(1)
_r_ ˆ( _xi, yi_ ; _D_ ≿ [WTP] ) _∈_ argmin
_r_
_N_
_i_ =1
_r_ ( _xi, yi_ _[′]_ [)] _[ −]_ _[r]_ [(] _[x][i][, y][i]_ [)] _[ −]_ _[w][i]_
Finally, we maximize mean reward. Call this Cardinal Reinforcement Learning from Human Feedback
(CRLHF). The following shows that in the data limit, CRLHF will always select the optimal model.
6Assuming that utility is quasi-linear in money, which we will assume throughout the paper.
7In an ideal setting, you could actually collect money, but for our case, we use simple stated WTP.
**Theorem 4.2.** _Let D_ ≿ _[WTP]_ _and suppose N is large enough to cover all x ∈X_ _, y ∈Y, y_ _[′]_ _∈Y. Let_
- _π_ ( _y|x_ )ˆ _r_ ( _xi, yi_ ; _D_ ≿ _[WTP]_ ) (2)
_y∈Y_
_f_ ( _D_ ≿ _[WTP]_ _, F_ ) = argmax
_π∈F_
_x∈X_
_Then, for any possible preferences_ ≿ _∈P_ (Π) _and any feasible set of language models F,_
_f_ ( _D_ ≿ _[WTP]_ _, F_ ) ≿ _π for all π ∈F. In other words, RLHF with a reward trained via MSE loss_
_on willingness-to-pay estimates always produces the most preferred model in the data limit._
Alternatively, there is a DPO style objective that solves for the same policy as above under some
assumptions [Rafailov et al., 2024].
- _β_ log _π_ _[π]_ ref _[∗]_ [(] ( _[y]_ _yi_ _[′]_ _i_ ~~_[′]_~~ _[|][|][x][x][i][i]_ [)][)] _[−]_ _[β]_ [ log] _π_ _[π]_ ref _[∗]_ [(] ( _[y]_ _y_ _[i]_ _i_ _[|]_ _|_ _[x]_ _x_ _[i]_ _i_ [)] ) _[−]_ _[w][i]_ �2 (3)
argmin
_π_ _[∗]_
_D_ ≿ [WTP]
Then under the same conditions as those listed by Rafailov et al. [2024], the solution to Equation 2
and Equation 3 must be the same. We call this Cardinal Direct Preference Optimization (CDPO). [8]

4.2 Eliciting Cardinal Utility via Willingness to Pay


A natural concern with using cardinal data is that it is too noisy or difficult for labelers to provide, and
inconsistent across labelers [Christiano et al., 2017]. In theory, our approach partially alleviates these
concerns because labels are still relative _e.g., y_ vs _y_ _[′]_ and because money gives labelers a familiar,
clear scale to use. In this section, we check empirically if our WTP scheme can elicit high-quality
cardinal data at scale, with an eye on performance relative to ordinal data with Bradley-Terry.
We elicit WTP values from five labelers across over 25 _,_ 000 tuples of prompts and response pairs
chosen from Anthropic’s HHH dataset [Bai et al., 2022a] and Chatbot Arena comparisons [Chiang
et al., 2024b] to form the CARDINALPREFS dataset. Anecdotally, the labelers reported that providing
WTP made each sample took roughly 6 _−_ 10% longer than purely ordinal labels. With this data in
hand, we perform 3 data quality checks.
Figure 3: Distribution of WTP Values
**Figure 3 shows that the distribution of cardinal labels is smooth across multiple labelers,**
**but does not match Bradley-Terry assumptions.** Aggregating cardinal data across multiple
labelers introduces the possibility of improper normalization across raters. However, in Figure 3, the
distribution after normalization is unimodal with fast decaying tails, indicating that in aggregate these
problems do not appear. Second, Bradley-Terry implies the distribution of the cardinal values should
8This regression-based procedure matches Gao et al. [2024]. The difference is we recommend optimizing
against cardinal feedback from labelers directly, instead of from a reward model estimated via Bradley-Terry.
be logistic, but empirically we can see actual people’s cardinal values are more peaked and have
faster decaying tails. This is empirical evidence against the parametric assumption in Bradley-Terry.
**Figure 4 shows two randomly selected draws from the dataset which are chosen to be represen-**
**tative cases of "high" and "low" labeled WTP - we find intuitively that the high WTP case is**
**much more important.** In the high WTP case, the dispreferred response does not even answer the
question while the preferred response gives a complete answer. In the low WTP case both answers
respond to the prompt accurately but the preferred response is more formal and informative. This
gives some sanity check that labelers can provide cardinal data that is meaningful.
|Col1|High Willingness to Pay (6.34)|Low Willingness to Pay (0.13)|
|---|---|---|
|**Prompt**|What are some common data structures?|What do people mean when they talk
about the "nuclear family"?|
|**Preferred**|Data structures are fundamental concepts in
computer science and programming. (...)
1.** Arrays**: A collection of elements identi-
fed by index or key. 2.** Linked Lists**: A
linear collection of nodes with data and refer-
ences. 3.** Stacks**: LIFO data structures with
push/pop operations. (...)|The nuclear family is a term used to de-
scribe an intimate family unit consisting of
a married couple and their dependent chil-
dren. This term derives from the description
of the central part of the atomic structure of
an atom. (...)|
|**Dispreferred**|#NAME?|Oh that’s a broad topic! The "nuclear family"
generally refers to the traditional nuclear fam-
ily that most people grew up in, consisting of
a mother and a father, and children. (...)|
Figure 4: An example of a high and low WTP labels. These examples are randomly chosen from the
dataset filtered to high WTP and low WTP cases respectively.
As our final data-quality check, we compute the implicit reward estimate from the cardinal data
directly via CDPO and via Bradley-Terry via DPO. With these values, we compute the mean-squared
error (MSE) of the implicit reward differences vs. the WTP data on a held-out set. If WTP data
only added noise and no signal, we would expect similar MSE from training on ordinal data and
on the cardinal data. Similarly, if Bradley-Terry accurately captured preference intensity already in
ordinal data, then we would also expect similar MSE across both methods. **We find the cardinal**
**data’s MSE is 55.92** % **smaller than the ordinal data’s, indicating meaningful signal increase**
**over ordinal data with Bradley-Terry.**

5.1 CARDINALPREFS Dataset and Cardinal Label Quality


In this section we empirically test if adding in cardinal data boosts performance in the preference
fine-tuning pipeline. We focus on comparing DPO and CDPO as it is computationally cheaper than
RLHF and CRLHF. A central challenge with these experiments is evaluation. Our goal is to optimize
≿, so the natural benchmark is ≿; however, ≿ is very difficult/expensive to measure. For example, it
may take a labeler weeks to decide if they prefer GPT-4o or Grok 3. Therefore, we utilize a wide
variety of experiments to triangulate the effectiveness of using cardinal data.


Our first experiment involves a simplified setting where the set of feasible models is small enough
such that we can elicit model-level preferences from labelers. This allows us to directly measure if
using WTP data makes the resulting model more preferable. We find exploiting cardinal data leads to
significantly higher probability of the resulting model being more preferable.
Let _πθ_ := _π_ [(1)] _θ_ + _π_ [(2)] (1 _−_ _θ_ ) and _F_ = _{πθ | θ ∈_ [0 _,_ 1] _}_ where _π_ [(1)] and _π_ [(2)] only differ on 3
prompts. Because _π_ [(1)] and _π_ [(2)] only differ on three prompts, it is feasible to elicit from labelers if
_π_ [(1)] ≿ _π_ [(2)], their model-level preferences.
The experiment proceeds as follows. First, we hired a set of multiple experienced labelers. Second,
we had the labelers give both ordinal feedback and WTP at the prompt-level. Third, we set up
objective functions for DPO following Rafailov et al. [2024] and CDPO following Equation 3 on
the prompt-level feedback [9] . Optimization occurs along a single parameter _θ_, and can be solved via
gradient descent. Fourth, we elicit model-level preferences from the labeler between _π_ [(1)] and _π_ [(2)]
e.g., if _π_ [(1)] ≿ _π_ [(2)] . Finally, we compare the selected models by DPO and CDPO to the labeler’s
preferences. We say CDPO/DPO selects the optimal model if it puts more weight on the preferred
model e.g., _πθ_ has _θ >_ [1] 2 [where] _[ π]_ [(1)][ ≿] _[π]_ [(2)][.]
Table 1: CDPO Selects the Optimal Model More Often than DPO
Selects Optimal Model
**Method** Full sample Where CDPO and DPO Disagree
CDPO 90.27% 91.67%
(1.48%) (3.99%)
DPO 83.29% 33.33%
(1.86%) (6.80%)
Diff. 6.98%*** 58.33%***
(1.44%) (9.24%)
_Notes: N_ = 401 (full sample), _N_ = 48 (sample where DPO and CDPO disagree). _[∗]_ _p <_ 0 _._ 1, _[∗∗]_ _p <_ 0 _._ 05,
_∗∗∗p <_ 0 _._ 01. “Selects Optimal Model” is the percentage of times the method chooses a model weakly preferred
to the alternative; standard errors in parentheses.
**Table 1 shows that CDPO selects the optimal model significantly more often than DPO.** The
first column shows that CDPO selects the optimal model almost 7 percentage points more often than
DPO. However, this masks many cases where CDPO and DPO select the same model. The second
column shows CDPO outperforms DPO by over 58 percentage points in the sample where CDPO
and DPO disagree.
This experiment shows that, taking into account all the noise, heterogeneity, etc. in cardinal feedback,
on net it helps the fine-tuning algorithm output the preferred model significantly more often.

Experiment 1: Real Data, Simplified Training Setting


In this experiment, we use simulated preference data from a ground-truth reward model in the standard
Chatbot fine-tuning setting. We find that CDPO results in 50% higher mean ground-truth reward than
DPO.
The experiment proceeds as follows. First, we take simulated preferences set by ground-truth reward
model [Liu et al., 2024]. We use a dataset of pairs of responses from Ultrafeedback-200k, a large
preference dataset covering a broad suite of chat and reasoning tasks [Cui et al., 2024]. We then
compute the ground-truth rewards for each response _r_ _[GT]_ ( _x, y_ 1) and _r_ _[GT]_ ( _x, y_ 2). For CDPO, we
label directly with the reward margin _r_ _[GT]_ ( _x, y_ 1) _−_ _r_ _[GT]_ ( _x, y_ 2). For DPO, we label directly with the
sign of _r_ _[GT]_ ( _x, y_ 1) _−_ _r_ _[GT]_ ( _x, y_ 2). Second, we replicate the Zephyr fine-tuning recipe [Tunstall et al.,
2023]. As our base model, we use a Mistral-7B model [HuggingFace, 2023] trained with supervised
fine-tuning on the UltraChat dataset [Ding et al., 2023]. Third, we set up objective functions for DPO
following Rafailov et al. [2024] and CDPO following Equation 3. Let _π_ DPO and _π_ CDPO denote the
maximizer we get from DPO and CDPO. We fine-tune the Mistral-7B model with LoRA for one
epoch (see Appendix C.4 for details). Fourth, we score generated text to estimate mean reward.
**CDPO achieves higher utility than DPO.** Figure 5 shows the central result: CDPO achieves
considerably higher utility than DPO, and thus we empirically can confirm that CDPO is producing
more preferred/more aligned models than DPO by taking into account richer preference data. Notice
that CDPO does better at 2 _,_ 000 steps than DPO does at over 14 _,_ 000 steps. This indicates that even if
collecting cardinal data is more expensive, and we can only collect 101 [as much data, CDPO would]
still be better.
**CDPO prioritizes getting strong preferences right.** To gain some intuition about why CDPO is
producing more preferable models than DPO, recall that during the optimization process, tradeoffs
between prompt-response pairs are inevitable. CDPO makes these tradeoffs by using cardinal human
We set the reference model as _π_ 1
2 [.]
Figure 5: **CDPO achieves higher utility by optimizing for preference strength. Left:** CDPO
achieves higher utility than DPO, where utility measures mean alignment of generations with human
preferences. Utility is normalized such that the base model gets 0 utility. **Right:** CDPO’s advantage
over DPO increases with preference strength.
feedback to prioritize high-impact improvements, while DPO gives equal weight to all comparisons
regardless of importance. Figure 5.2 demonstrates this quantitatively: analyzing a validation set of
1000 tuples ( _x, y_ 1 _, y_ 2) stratified by reward margin ( _r_ [GT] ( _x, y_ 2) _−_ _r_ [GT] ( _x, y_ 1)), we find that while the
two algorithms perform comparably on low-margin cases, CDPO excels on medium and especially
high-margin cases. This shows CDPO successfully prioritizes cases humans identify as important,
while DPO wastes optimization effort on relatively unimportant improvements.

Experiment 2: Simulated Data, Real Training


In this experiment, we use both real-world cardinal data and the standard chatbot preference finetuning setting. Now we cannot observe ≿ directly, so we rely on various preference proxy measures.
The experiment proceeds as follows: we take the exact same training procedure as in experiment
2 but instead of using Ultrafeedback-200k with simulated preference data, we use the real-world
CARDINALPREFS preference dataset we gathered in Section 5.1.
**On real-world preference measures, CDPO exceeds DPO by prioritizing important cases.** On a
held-out set, CDPO outputs the preferred response at roughly the same rate as DPO. This is expected
as CDPO does not magically make it possible to get more prompts right. However, CDPO does
apply more optimization pressure to "important prompts." When we weight observations by WTP,
we see CDPO does about 4 percentage points better. Similarly, when we weight observations by
importance as determined by Claude 3.7 Sonnet, we also see CDPO scores higher by 3 percentage
points, indicating that CDPO is properly prioritizing the important cases. Similarly, when we filter
to observations that where a response is substantively better as determined by Claude, we also see
CDPO scores higher by about 2 _._ 5 percentage points but on stylistic issues, DPO exceeds by around 9
percentage points. This indicates that CDPO successfully identifies and applies more optimization
pressure to important issues, while DPO wastes optimization pressure on relatively less important,
stylistic improvements because ordinal measures cannot properly prioritize based on importance.
Finally, on Arena-Hard, a state-of-the-art preference evaluation measuring win-rates against GPT-4,
CDPO wins almost 55% more battles than DPO.
Table 2: CDPO exceeds DPO by prioritizing important cases.


Outputs Preferred Response 51.1% **51.6%**
Outputs Preferred Response (WTP Weighted) 52.8% **56.7%**
Outputs Preferred Response (Importance Weighted) 46.0% **49.0** %
Outputs Preferred Response (Substantive) 44.8% **47.2** %
Outputs Preferred Response (Style) **66.7** % 55.6%
Arena-Hard Winrate 3.80% **5.9%**


In this paper, we demonstrated fundamental limitations of ordinal human feedback and explored the
benefit of collecting richer, cardinal feedback. We found that using cardinal feedback resulted in
models that are rated as more preferable and score higher on preference-based benchmarks. In future
work, it would be valuable to study and improve cardinal feedback elicitation protocols and interface
design in an LLM context such as WTP with a fixed budget, implied cardinal ratings from lottery
choices, among others.


This research aims to improve language model alignment by developing more theoretically grounded
methods for learning and modeling user preferences. While learning people’s preferences more
accurately could provide significant benefits - including enhanced safety, reliability, and utility - it is
important to note that this improvement alone does not address all AI safety concerns.


**Proof of Theorem 4.1.** Suppose there exists an _f_ such that for any _F_ and any ≿ _∈P_ (Π),
_f_ ( _D_ ≿ [ordinal] _, F_ ) ≿ _π_ for all _π ∈F_ .
Let _F_ = _{π_ [(1)] _, π_ [(2)] _}_ where _π_ [(1)] and _π_ [(2)] differ only on two prompts, _x_ 1 and _x_ 2 where _π_ [(1)] is better
than _π_ [(2)] on prompt _x_ 1 and the reverse holds for prompt _x_ 2.
Let ≿ [(1)] _,_ ≿ [(2)] _∈_ _P_ (Π) such that _π_ [(1)] _≻_ [(1)] _π_ [(2)], _π_ [(2)] _≻_ [(2)] _π_ [(1)] but also such that ≿ [(1)] _x_ [=][≿] _x_ [(2)] for all
_x ∈X_ i.e. both preferences agree on which responses are better for each prompt, but they weigh the
prompts differently.
Then, _D_ ≿ [ordinal][(1)] = _D_ ≿ [ordinal][(2)] (i.e. ≿ is not identified from _D_ [ordinal] ) and therefore
_π_ _[∗]_ = _f_ ( _D_ ≿ [ordinal][(1)] _[,][ F]_ [) =] _[ f]_ [(] _[D]_ ≿ [ordinal][(2)] _[,][ F]_ [)]
Without loss of generality, suppose _π_ _[∗]_ = _π_ [(1)] . Then we have _f_ ( _D_ ≿ [ordinal][(2)] _[,][ F]_ [) =] _[ π]_ [(1)] _[ ̸]_ [≿][(2)] _[ π]_ [(2)][. This]
violates our assumption that _f_ is such that _f_ ( _D_ ≿ [ordinal] _, F_ ) ≿ _π_ for all _π ∈F_ and all ≿ _∈_ _P_ (Π).
**Proof of Theorem 4.2.** By Theorem A.1, ≿ admits a utility function of the form
-
_x∈X_ _y∈Y_ _[π]_ [(] _[y][|][x]_ [)] _[r]_ [true][(] _[x, y]_ [)][. Under quasi-linearity of money, the WTP to go from] _[ y]_ [ to] _[ y][′]_ [ as]
a response to _x_ is given by _r_ ( _x, y_ _[′]_ ) _−_ _r_ ( _x, y_ ). That implies the reward modeling stage solves
_r_ ˆ( _xi, yi_ ; _D_ ≿ [WTP] ) _∈_ argmin
_r_
- �2
_r_ ( _xi, yi_ _[′]_ [)] _[ −]_ _[r]_ [(] _[x][i][, y][i]_ [)] _[ −]_ _[r]_ [true][(] _[x][i][, y]_ _i_ _[′]_ [)] _[ −]_ _[r]_ [true][(] _[x][i][, y][i]_ [)]
Since _D_ ≿ [WTP] is sufficiently large, ˆ _r_ ( _xi, yi_ ; _D_ ≿ [WTP] ) will perfectly match _r_ [true] ( _xi, yi_ ) up to an
affine transformation. Since _r_ is unique up to affine transformations by Theorem A.2,
- _x∈X_ - _y∈Y_ _[π]_ [(] _[y][|][x]_ [)ˆ] _[r]_ [(] _[x][i][, y][i]_ [;] _[ D]_ ≿ [WTP] ) is also a utility function that represents ≿. Since _F_ is
compact by assumption, the argmax exists and is always the most preferred feasible policy i.e.
_f_ ( _D_ ≿ [WTP] _, F_ ) ≿ _π_ for all _π ∈F_ .


**Theorem A.1.** ≿ _satisfies completeness, transitivity, continuity and independence if and only if there_
_exists a r_ : _X × Y →R such that_ [�] - _π_ ( _y|x_ ) _r_ ( _x, y_ ) _represents_ ≿ _._
_x∈X_
- _π_ ( _y|x_ ) _r_ ( _x, y_ ) _represents_ ≿ _._
_y∈Y_
**Proof of Theorem A.1.**
This proof follows by exploiting a mathematical equivalence between language models and subjective
expected utility [Anscombe et al., 1963].
Notice that our setting is mathematically identical to subjective expected utility with state dependent
preferences. Consider the following analogy:
1. In our context:
      - We have a finite number of prompts
      - We have a finite number of responses
     - We have a language model _π_ which maps from our prompts to lotteries over our
responses
2. In subjective expected utility
      - We have a finite number of states
      - We have a finite number of outcomes
      - We have a ‘act’ _f_ which maps from the states to lotteries over outcomes.
Usually subjective expected utility analysis has extra assumptions added on that will break the
analogy, like the assumption of state-independent preferences. In our setting, that would be equivalent
to prompt-independent preferences, which is not plausible.
But if we focus on the general case of subjective expected utility with state-dependent preferences,
our language model setting exactly lines up with the mathematical setting considered in Kreps [1988]
proposition 7.4. This proposition shows in the subjective expected utility setting that completeness,
transitivity, continuity and independence are necessary and sufficient for an additively separable,
expected utility representation. Therefore, we have our result.
**Theorem A.2.** [�] _x∈X_ - _y∈Y_ _[π]_ [(] _[y][ |][ x]_ [)] _[ r]_ [(1)][(] _[x, y]_ [)] _[ and]_ [ �] _x∈X_ - _y∈Y_ _[π]_ [(] _[y][ |][ x]_ [)] _[ r]_ [(2)][(] _[x, y]_ [)] _[ represent the]_
_same_ ≿ _iff there exist a >_ 0 _and b_ : _X →_ R _such that r_ [(1)] ( _x, y_ ) = _a r_ [(2)] ( _x, y_ ) + _b_ ( _x_ ) _for all x, y._
**Theorem A.2.** [�]
 _x∈X_
_y∈Y_ _[π]_ [(] _[y][ |][ x]_ [)] _[ r]_ [(1)][(] _[x, y]_ [)] _[ and]_ [ �]
 _x∈X_
**Proof of Theorem A.2.** Define
_u_ [(] _[i]_ [)] ( _π_ ) =          - _π_ ( _y | x_ ) _r_ [(] _[i]_ [)] ( _x, y_ ) _,_ _i_ = 1 _,_ 2 _._
_x,y_
( _⇒_ ) Assume _u_ [(1)] and _u_ [(2)] induce the same order. Hence _∃_ a strictly increasing _f_ : R _→_ R with
_u_ [(1)] ( _π_ ) = _f_ - _u_ [(2)] ( _π_ )� for all _π_ .
For any policies _π_ [1] _, π_ [2] and any _λ ∈_ [0 _,_ 1] define
_π_ _[λ]_ ( _y | x_ ) := _λ π_ [1] ( _y | x_ ) + (1 _−_ _λ_ ) _π_ [2] ( _y | x_ ) _,_ _∀x, y._
_u_ [(] _[i]_ [)] is given as linear in _π_ ( _y|x_ )
_u_ [(] _[i]_ [)] ( _π_ _[λ]_ ) = _λ u_ [(] _[i]_ [)] ( _π_ [1] ) + (1 _−_ _λ_ ) _u_ [(] _[i]_ [)] ( _π_ [2] ) _,_ _i_ = 1 _,_ 2 _._ (1)
Let _s_ := _u_ [(2)] ( _π_ [1] ) and _t_ := _u_ [(2)] ( _π_ [2] ). Equation (1) and the representation relation give
_f_ - _λs_ +(1 _−λ_ ) _t_ - = _f_ - _u_ [(2)] ( _π_ _[λ]_ )� = _u_ [(1)] ( _π_ _[λ]_ ) (1)= _λ u_ [(1)] ( _π_ [1] )+(1 _−λ_ ) _u_ [(1)] ( _π_ [2] ) = _λf_ ( _s_ )+(1 _−λ_ ) _f_ ( _t_ ) _._
Because _s_ and _t_ range over all values _u_ [(2)] attains, and _λs_ + (1 _−_ _λ_ ) _t_ is again in that range (by _π_ _[λ]_ ),
The above equation holds on a convex subset of R. A strictly increasing real function satisfying (2)
on a convex set must be affine, so _f_ ( _t_ ) = _a t_ + _b_ with _a >_ 0.
Substituting back, for every _π_
    - _π_ ( _y | x_ )     - _r_ [(1)] ( _x, y_ ) _−_ _a r_ [(2)] ( _x, y_ )� = _b._
_x,y_
Fix any _x_ and view the left side as an affine functional of _π_ ( _· | x_ ) on the simplex ∆( _Y_ ); being
constant on that simplex forces the coefficient vector to be constant:
_r_ [(1)] ( _x, y_ ) _−_ _a r_ [(2)] ( _x, y_ ) = _b_ ( _x_ ) _∀y,_
hence _r_ [(1)] ( _x, y_ ) = _a r_ [(2)] ( _x, y_ ) + _b_ ( _x_ ).
( _⇐_ ) Conversely, if _r_ [(1)] ( _x, y_ ) = _a r_ [(2)] ( _x, y_ ) + _b_ ( _x_ ) with _a >_ 0, then
_u_ [(1)] ( _π_ ) = _a u_ [(2)] ( _π_ ) +            - _b_ ( _x_ ) _,_
_x_
a positive-affine transform of _u_ [(2)] ; therefore the two functionals induce exactly the same ≿.


**B.1** **Deriving** ≿ _x_
In this subsection, we formally show how to derive ≿ _x_ from ≿.
Given _π ∈_ Π, define
_πxj_ _,y_ ˜( _x_ ) =                - _πy_ ˜( _x_ ) ifif _x x ̸_ == _x xjj_
That is, _πxj_ _,y_ ˜ is _π_ where we modify just how it responds to _xj_ to ˜ _y_ .
With this machinery, we can define preferences over responses. Let ˜ _y_ [(1)] _,_ ˜ _y_ [(2)] _∈_ ∆( _Y_ ). Define ≿ _xi_
as follows
def
_y_ ˜ [(1)] ≿ _xi_ ˜ _y_ [(2)] _⇐⇒_ _πxi,y_ ˜(1) ≿ _πxi,y_ ˜(2) _∀π ∈_ Π
That is, distribution of responses ˜ _y_ [(1)] is preferred to ˜ _y_ [(2)] as a response to prompt _xi_ if for any model,
we can make it more preferred by switching its response from ˜ _y_ [(2)] to ˜ _y_ [(1)] . It is straightforward
to verify that ≿ _x_ from such a definition is transitive, continuous and satisfies independence since
≿ does. Furthermore, independence of ≿ implies ≿ _x_ is complete. This can be seen via the utility
representation in theorem A.1.


For a review, we cover the basic axioms on preferences that we assume throughout the paper.
1. Complete: for any _x, y_ in the mixture space it must be _x_ ≿ _y_ or _y_ ≿ _x_ or both.
2. Transitive: _x_ ≿ _y_ ≿ _z_ = _⇒_ _x_ ≿ _z_
3. Continuous: For any _x ≻_ _y ≻_ _z_, there exists _a, b ∈_ (0 _,_ 1) such that _ax_ + (1 _−_ _a_ ) _z ∼_ _y ∼_
_bx_ + (1 _−_ _b_ ) _z_
4. Independence: _x ≻_ _y_ = _⇒_ _αx_ + (1 _−_ _α_ ) _z ≻_ _αy_ + (1 _−_ _α_ ) _z_ for all _α ∈_ (0 _,_ 1].


In an ideal WTP setting, elicitations are incentivized by in some scenarios taking money from
participants and allocating them the good. That is unfortunately difficult in a language model setting,
but thankfully a frequent finding from experimental economics is that incentivized and unincentivized
willingness-to-pay elicitations are not majorly dissimilar [Murphy et al., 2005]. Therefore we use
stated willingness-to-pay in our experiment.
To elicit these values, labelers are provided with labeling software that looks like the following
Figure 6: Labeling Software
Instructions are provided to labelers to rank their preferred responses on top and to use the numerical
boxes to indicates their willingness-to-pay to move from the worse response to the better response.
They are told that the WTP denotes the maximum you would be happy with spending to get the
preferred response. If the amount were any higher, you would prefer to keep the money and get the
worse response. Although conceptually the numeraire we focus on is money, we also tried various
options such as normalizing the value of a reference improvement to 1 and then eliciting all other
values relative to that reference value. We include both of these methods in the final dataset to
maximize sample size.
Compensation was provided by the hour - at an average of roughly $15 an hour, with bonuses for
accuracy as measured by congruence.
Ultimately, labels are provided by 5 distinct labelers we hired - a mix of undergraduate and labelers
on Upwork with LLM labeling experience. For the data we use in training, each sample corresponds
to one labeler and there is no overlap between labelers. Separately, we designed a small hold-out set
to check congruence between labelers and found a congruence of approximately 67%.
Finally, for experiment one, the experiment was piloted with various options on how to set up the
cross-labeling, and the one presented in the paper was the pilot option that was expanded after seeing
promising results.


For Table 2, we use Claude 3 _._ 7 Sonnet to label how much of a quality difference there is in responses
and if the improvement in response quality is due to substantive or stylistic issues. The following is
the prompt we gave Claude.
You are an expert AI assistant capable of evaluating the quality of text responses.
You will be given a prompt, a ’Chosen’ response, and a ’Rejected’ response. Your
task is to assess the *difference* in quality between the Chosen and Rejected
responses.
First, rate the *importance* of this quality difference on a scale from 1 to 10, where:
1: The quality difference is negligible or very minor. 5: The quality difference is
moderate and noticeable. 10: The quality difference is very significant and crucial.
Second, identify the primary nature of the improvement the ’Chosen’ response has
over the ’Rejected’ response. Choose one: - Substance: The chosen response is
better due to significant improvements in factual accuracy, completeness, reasoning,
instruction following, or overall helpfulness of the core content. - Style: The
chosen response is better primarily due to improvements in clarity, conciseness,
tone, formatting, grammar, or readability, while the core substance might be similar
or only marginally better. - Other: Neither Substance nor Style predominantly
describes the improvement, or the improvement is a mix, or it’s hard to categorize.
Third, provide a brief explanation for your score and type.
Provide your response formatted *exactly* as follows, with each item on a new
line: Score: Improvement Type: Explanation:


In section 5, we ran a simple experiment where we tracked DPO loss at the individual sample level.
Figure 7 shows the overall DPO loss follows a standard training path, so our results are not driven by
failure to learn anything.
Figure 7: Mean DPO loss across all tracked 100 samples.


**DPO and CDPO Fine-tuning and Inference.** For both DPO and CDPO, our base model is a
Mistral-7B base model that has been full-parameter supervised fine-tuned on the UltraChat dataset.
We then LoRA-fine-tune this model for one epoch. We use batch size 4, LoRA rank _rLoRA_ = 64,
regularization _αLoRA_ = 64, and dropout _pLoRA_ = 0 _._ 05. We use learning rate 1 _e −_ 5, 150 warmup
steps, and max conversation length of 1024 tokens. This replicates the Zephyr training recipe,
except that we use _β_ = 0 _._ 1 instead of _β_ = 0 _._ 01, which we found improved performance. Finally,
we normalize the WTP data so that the standard deviation matches the standard deviation from
Bradley-Terry models, so that the regularization is relative to the same value for both values.
We run this training recipe on a single A100 with 80GB of VRAM. Each training run takes roughly 5
hours.
**Reward Model Training and Inference.** To represent _U_ _[GT]_ for data labeling and evaluation, we
use a reward model fine-tuned on CARDINALPREFS. As the base reward model, we LoRA-fine-tune
Skywork/Skywork-Reward-Llama-3.1-8B (a top-rated reward model on RewardBench) for one epoch
on our dataset.
To compute the mean reward, we sample 1000 prompts from a held-out validation split of
Ultrafeedback-200k that the language models were not trained on. We use this to calculate the
average reward achieved by each model’s completions.


For our accuracy plot, we use the implicit reward model defined by DPO and CDPO
_r_ ( _xi, y_ ) = _β_ log _[π]_ [(] _[y][|][x][i]_ [)] (4)
_πref_ ( _y|x_ )
We use accuracy to mean the implicit reward model ranks _y_ 2 and _y_ 1 the same way ≿ _xi_ does.


We reproduce the first three rows of Table 2 while varying the regularization parameter _β_ =
(0 _._ 01 _,_ 0 _._ 1 _,_ 0 _._ 25). For each choice of _β_, we implement the same training procedure described in
the above section. These results show robustness across _β_ - CDPO consistently outperforms DPO,
especially on importance-weighted tasks.
Table 3: Ablation Study on Regularization Parameter _β_


_β_ = 0 _._ 1
Outputs Preferred Response 51.1% **51.6%**
Outputs Preferred Response (WTP Weighted) 52.8% **56.7%**
Outputs Preferred Response (Importance Weighted) 46.0% **49.0%**
_β_ = 0 _._ 01
Outputs Preferred Response 50.0% **65.0%**
Outputs Preferred Response (WTP Weighted) 47.2% **75.6%**
Outputs Preferred Response (Importance Weighted) 47.7 **65.6**
_β_ = 0 _._ 25
Outputs Preferred Response **34.0%** **34.0%**
Outputs Preferred Response (WTP Weighted) 22.8% **28.2%**
Outputs Preferred Response (Importance Weighted) 24.78% **26.9%**


We compare DPO and CDPO on objective benchmarks to supplement our preference-based metrics.
On the objective benchmarks such as TruthfulQA, AGIEval and MMLU, the comparisons are closer
than on preference-based metrics, with perhaps a slight advantage for CDPO. This result indicates
most of the benefit of CDPO is preference-based rather than learning new capabilities or facts, which
is expected for a post-training, preference-based method.
Table 4: CDPO exceeds DPO by prioritizing important cases.
TruthfulQA 33.3% **33.7%**
MMLU **56.7** % 56.4%
AGIEval 30.5% **30.6%**


To parse how CDPO prioritizes vs. DPO, we turn part of Table 2 into Figure 8. Here, we can see that
on substantive issues CDPO exceeds DPO but on style DPO greatly exceeds CDPO. This supports
the hypothesis that DPO focuses on unimportant improvements while CDPO puts more optimization
pressure on important, core improvements.
Figure 8: CDPO exceeds DPO on substantive improvements, while DPO exceeds on style improvements.


1. **Claims**
Question: Do the main claims made in the abstract and introduction accurately reflect the
paper’s contributions and scope?
Answer: [Yes]
Justification: The abstract and intro show the central contribution is about using richer
feedback in RLHF/DPO to get better performance.
2. **Limitations**
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: This is discussed in the conclusion and in the experimental section where we
discuss the weakness of each experiment. We also discuss the cost/time increase of using
cardinal labels.
3. **Theory assumptions and proofs**
Question: For each theoretical result, does the paper provide the full set of assumptions and
a complete (and correct) proof?
Answer: [Yes]
Justification: All theorem statements are complete and the proofs are in the appendix. The
full set of assumptions are always stated in the theorem or stated as assumed throughout the
paper.
4. **Experimental result reproducibility**
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions
of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Appendix describes our data collection, and training procedure exactly.
5. **Open access to data and code**
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental
material?
Answer: [Yes]
Justification: We have open sourced our core dataset. It is available at `[https://](https://huggingface.co/datasets/cardinal-prefs/CardinalPrefs)`
`[huggingface.co/datasets/cardinal-prefs/CardinalPrefs](https://huggingface.co/datasets/cardinal-prefs/CardinalPrefs)` . There are sufficient
instructions to replicate all experimental results, although we have not open sourced our
code yet.
6. **Experimental setting/details**
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the
results?
Answer: [Yes]
Justification: Yes, training details are in the appendix.
7. **Experiment statistical significance**
Question: Does the paper report error bars suitably and correctly defined or other appropriate
information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Experiment one shows statistical significance, the result of the experiments are
graphical or benchmarks scores which are not amenable to statistical significance testing.
8. **Experiments compute resources**
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce
the experiments?
Answer: [Yes]
Justification: Yes, we describe using one A100 throughout the paper.
9. **Code of ethics**
Question: Does the research conducted in the paper conform, in every respect, with the
NeurIPS Code of Ethics `[https://neurips.cc/public/EthicsGuidelines](https://neurips.cc/public/EthicsGuidelines)` ?
Answer: [Yes]
Justification: To our knowledge it does.
10. **Broader impacts**
Question: Does the paper discuss both potential positive societal impacts and negative
societal impacts of the work performed?
Answer: [Yes]
Justification: See the impact statement at the end.
11. **Safeguards**
Question: Does the paper describe safeguards that have been put in place for responsible
release of data or models that have a high risk for misuse (e.g., pretrained language models,
image generators, or scraped datasets)?
Answer: [No]
Justification: Our models are not frontier or sensitive. For our data release, data is
anonymized to protect the identity of labelers.
12. **Licenses for existing assets**
Question: Are the creators or original owners of assets (e.g., code, data, models), used in
the paper, properly credited and are the license and terms of use explicitly mentioned and
properly respected?
Answer: [Yes]
Justification: Whenever we use data, we properly credit the source.
13. **New assets**
Question: Are new assets introduced in the paper well documented and is the documentation
provided alongside the assets?
Answer: [Yes]
14. Justification: When introducing novel data, we note that.
15. **Crowdsourcing and research with human subjects**
Question: For crowdsourcing experiments and research with human subjects, does the paper
include the full text of instructions given to participants and screenshots, if applicable, as
well as details about compensation (if any)?
Answer: [Yes]
Justification: Yes, see the appendix for the presented GUI, compensation details and labeling
instructions.
16. **Institutional review board (IRB) approvals or equivalent for research with human**


Question: Does the paper describe potential risks incurred by study participants, whether
such risks were disclosed to the subjects, and whether Institutional Review Board (IRB)
approvals (or an equivalent approval/review based on the requirements of your country or
institution) were obtained?
Answer: [Yes]
Justification: We have IRB approval. As study participants were providing non-sensitive
data, it is low to no risk.
17. **Declaration of LLM usage**
Question: Does the paper describe the usage of LLMs if it is an important, original, or
non-standard component of the core methods in this research? Note that if the LLM is used
only for writing, editing, or formatting purposes and does not impact the core methodology,
scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We only use LLMs for formatting and code assistance except for when we use
Claude to label data as described in the text.


**Ahmed Elshabrawy** **[1]** _[∗]_ **Hour Kaing** **[2]** **Haiyue Song** **[2]** **Alham Fikri Aji** **[1]** **Hideki Tanaka** **[2]**
**Masao Utiyama** **[2]** **Raj Dabre** **[3]** _[†]_
1MBZUAI 2NICT, Japan 3 IIT Madras
ahmed.elshabrawy@mbzuai.ac.ae
raj.dabre@cse.iitm.ac.in
ABSTRACT
Alignment with high-resource standard languages is often assumed to aid the modeling of related low-resource varieties. We challenge this assumption by demonstrating that excessive representational entanglement with a dominant variety, such
as Modern Standard Arabic (MSA) in relation to Arabic dialects, can actively hinder generative modeling. We present the first comprehensive causal study of this
phenomenon by analyzing and directly intervening in the internal representation
geometry of large language models (LLMs). Our key contribution is an online
variational probing framework that continuously estimates the subspace of the
standard variety during fine-tuning, enabling projection-based decoupling from
this space. While our study uses Arabic as a case due to its unusually rich parallel
resources across 25 dialects, the broader motivation is methodological: dialectal
MT serves as a controlled proxy for generative tasks where comparable multivariety corpora are unavailable. Across 25 dialects, our intervention improves
generation quality by up to +4.9 chrF++ and +2.0 on average compared to standard fine-tuning, despite a measured tradeoff in standard-language performance.
These results provide causal evidence that subspace dominance by high-resource
varieties can restrict generative capacity for related varieties. More generally, we
unify geometric and information-theoretic probing with subspace-level causal interventions, offering practical tools for improving generative modeling in closely
related language families and, more broadly, for controlling representational allocation in multilingual and multi-domain LLMs. Code will be released.
1 INTRODUCTION
Large Language Models (LLMs) have achieved remarkable progress in multilingual Natural Language Understanding (NLU) and Generation (NLG) tasks (Brown et al., 2020; Chowdhery et al.,
2022; Scao et al., 2022; Aryabumi et al., 2024). Beyond English, these models show strong crosslingual transfer, enabling low-resource varieties to benefit from related high-resource languages (Hu
et al., 2020; Conneau et al., 2020; Xue et al., 2021).
A less understood question, however, is whether closer alignment with a dominant, high-resource
variety always benefits related low-resource ones. Dialects provide a natural test case: they are
linguistically distinct, socially important, yet often heavily entangled with their standardized counterpart in both data and models. Arabic exemplifies this dynamic, where Modern Standard Arabic (MSA) dominates pretraining resources while dozens of dialects remain underrepresented and
underperform on benchmarks (Kantharuban et al., 2023). Similar dynamics arise in other orthographically and lexically close pairs such as Czech–Slovak or Swedish–Icelandic. Understanding
representational interactions in such settings is crucial for inclusive generative modeling.
This paper challenges the assumption that alignment with a high-resource standard is always beneficial. Using Arabic dialects as a case study, chosen for their unusually rich parallel resources across
25 varieties (Bouamor et al., 2018), we show that excessive representational entanglement with MSA
_∗_ work done during internship at NICT, Japan
_†_ work done during tenure at NICT, Japan and unrelated to current position at Google.
Preprint. Work in Progress.
hinders generative performance. Since parallel corpora for other generative tasks across dialects are
scarce, we focus on machine translation as a controlled proxy for dialect-sensitive generation.
Our study proceeds in two stages. First, we analyze how LLMs internally represent MSA and
dialects, revealing that stronger generative performance correlates with greater representational separability from MSA. Second, we move from analysis to intervention: we introduce an **online vari-**
**ational probing** framework that continuously estimates the subspace of the high-resource standard
during fine-tuning, enabling a novel subspace decoupling strategy. This causal intervention promotes orthogonal representations and improves generative capacity for dialects.
5 different varieties of Arabic from the MADAR 26 corpus (Bouamor et al., 2018). The yellow
highlights the interrogative element (roughly “how much”), the green (when present) highlights the
explicit cost word, and the blue highlights the breakfast term.
|Dialect|Arabic|Transliteration (Buckwalter)|
|---|---|---|
|Modern Standard Arabic|?PA¢ ¯B
 @ ~~~~é ®Ê¾~~~~K Õ»|kam
taklifaT
al-’ifTar?|
|Egyptian Arabic|?PA¢ ®Ë@ ÐA¾K.|bkam
al-fTar?|
|Levantine Arabic|? ~~~~é~~~~®K
ðQ~~~~Ë@
~~~~k ø
 X @|’addi
Haq
al-tarwiqa?|
|Gulf Arabic|? ~~~~ñK
QË@ ÕºK.|bkam
al-riyooq?|
|Maghrebi Arabic|?hAJ.Ë@ Pñ¢ ¯
~~~~@Y~~~~®K.|bqaddash
fuToor al-SabaaH?|
Applied to 25 Arabic dialects, our approach yields consistent improvements over standard finetuning, up to +4.9 chrF++ on individual dialects and +2.0 on average, while trading off some performance in MSA generation. More broadly, our findings provide the first causal evidence that
representational dominance by high-resource standards can limit generative modeling in closely related varieties.
**Contributions.**
    - We present the first large-scale representational analysis of dialects in generative LLMs,
unifying geometric and information-theoretic probing.
   - We introduce a novel online probing-based subspace decoupling method that improves
generative performance for underrepresented varieties.
    - We empirically demonstrate consistent gains across 25 dialects, highlighting implications
for related language families where orthographic and lexical similarity creates similar entanglement.
2 RELATED WORKS
This work investigates how LLMs internally allocate representational capacity across closely related
language varieties, with a focus on Arabic dialects as a natural case study.
**Multilingualism in Large Language Models.** Recent studies have analyzed how multilingual
LLMs encode language-specific knowledge. For example, Wang et al. (2024) and Kojima et al.
(2024) explore neuron sharing and language-specific activations, showing that subtle modifications
can alter generation in particular languages. Our perspective differs: rather than focusing on neuronlevel behavior, we ask whether dialects remain representationally distinct from their standardized
counterpart and how this distinction (or entanglement) affects generative performance. This question
is not limited to Arabic, but applies broadly to orthographically and lexically similar pairs with a
resource imabalance.
At the representational level, Chang et al. (2022) show that languages occupy distinct subspaces in
encoder-only models, while Shah et al. (2024) link geometric differences to cross-lingual transfer.
Preprint. Work in Progress.
We extend these insights to large generative models, showing that the degree of subspace separability
between varieties correlates with downstream generation quality. Relatedly, Nigatu et al. (2023) find
that models struggle to capture dialectal nuances; our results both confirm this for recent LLMs and
provide causal evidence that mitigating representational entanglement improves performance.
**Information-Theoretic Probing.** Information-theoretic probes have been used to study how linguistic signals emerge during pretraining (Voita & Titov, 2020; M¨uller-Eberstein et al., 2023). Building on this, we introduce probes not just for analysis but as part of training: our “dialect probes”
continuously estimate the dominant standard-language subspace during fine-tuning, enabling us to
directly intervene by penalizing entanglement. This extends probing from a diagnostic tool to a
mechanism for causal representational control.
**Dialectal and Low-Resource NLP.** Dialectal variation presents a persistent challenge for generative modeling. Prior work has documented large performance gaps as dialects deviate from their
standardized counterpart (Kantharuban et al., 2023; Ziems et al., 2023). For Arabic, evaluation
resources such as AraBench (Sajjad et al., 2020) and MADAR (Bouamor et al., 2018) have been
developed, and recent studies examine MT and NLG across varieties (Kadaoui et al., 2023; Nagoudi
et al., 2023). Our work departs from these by focusing not on resource creation or evaluation but on
how dialects are internally represented in LLMs and how interventions on representational subspaces
can improve generative capacity. While Arabic provides a uniquely rich testbed, the implications
extend to other under-resourced language varieties that share high orthographic and lexical overlap
with a dominant variety.
3 BACKGROUND: ARABIC DIALECTS
A significant challenge in developing truly multilingual models lies in handling closely-related language varieties, which often exist in a state of resource imbalance with a dominant, high-resource
standard language. This scenario is not merely a linguistic curiosity but poses fundamental problems for model representation learning. We investigate this challenge through the lens of Arabic,
which provides an ideal testbed due to its distinct diglossia. The language ecosystem consists of
Modern Standard Arabic (MSA), a high-resource variety used in formal contexts, and numerous
low-resource Dialectal Arabic (DA) varieties that are the primary spoken languages but lack substantial textual corpora. The availability of the MADAR corpus (Bouamor et al., 2018), a unique
resource with parallel sentences across 25 dialects, enables a controlled study of this phenomenon,
which is not feasible for many other language families with similar dialectal diversity.
The divergence between MSA and DA is substantial,
spanning lexical, syntactic, and morphological levels, as
illustrated by the parallel translations in Table 1. To
quantify this, we perform hierarchical clustering based on
character-level similarity (chrF++ score, (Popovi´c, 2015))
in Figure 1. The analysis reveals that dialects form distinct, geographically-correlated clusters that are representationally distant from MSA. This dissimilarity presents
a concrete technical obstacle, starting at the tokenization
layer. As detailed in Appendix A, standard model tokenizers trained predominantly on high-resource data yield
suboptimal segmentations for dialectal words. This inefficiency leads to higher computational costs and hinders the
model’s ability to learn long-range dependencies, unfairly
disadvantaging low-resource varieties before any deeper
processing occurs (Ali et al., 2024).
|Col1|Col2|Col3|
|---|---|---|
||||
||||
|FX
TUN
MSA
MOS
BAS
BAG
KHA
CAI
ASW
ALX
BEN
TRI
BEI
ALE
DAM
SAL
JE
|FX
TUN
MSA
MOS
BAS
BAG
KHA
CAI
ASW
ALX
BEN
TRI
BEI
ALE
DAM
SAL
JE
|FX
TUN
MSA
MOS
BAS
BAG
KHA
CAI
ASW
ALX
BEN
TRI
BEI
ALE
DAM
SAL
JE
|
Figure 1: Hierarchical clustering of
Arabic varieties based on characterlevel distance (chrF++). Dialects cluster geographically and exhibit significant separation from the high-resource
Modern Standard Arabic (MSA), quantifying their surface-level dissimilarity.
Our goal is to understand how the representational dominance of a high-resource language like MSA affects a Modern Standard Arabic (MSA), quanmodel’s ability to generate text in closely-related, low- tifying their surface-level dissimilarity.
resource dialects. Due to the scarcity of parallel data for
diverse generative tasks, we utilize machine translation (MT) from MSA to the 25 DA varieties
as a controlled testbed for generation. This task allows us to probe the model’s capacity to pro
Preprint. Work in Progress.
duce dialect-specific outputs while controlling for semantic content. While our empirical study is
grounded in Arabic, the core challenge is generalizable. The insights derived are relevant to other
language families with similar orthographic overlap and resource asymmetry, such as Czech and
Slovak, or the spectrum of Scandinavian languages, where models must learn to navigate the subtle
yet critical distinctions between high- and low-resource variants.
4 METHODOLOGY
We present a methodology designed to first diagnose and then causally intervene in the representational geometry of multilingual models. Our approach uses a controlled generative task to probe
model capabilities, analyzes the underlying representations through geometric and informationtheoretic lenses, and introduces a novel training technique to mitigate representational entanglement.
4.1 TASK FORMULATION: DIALECTAL REWRITING AS A GENERATIVE TESTBED
To create a controlled environment for studying dialectal generation, we formulate a task of **Dialec-**
**tal Machine Translation** (DiaMT). Given a sentence in the high-resource standard variety (MSA),
the model’s objective is to generate the semantically equivalent sentence in a target low-resource
dialect. This task serves as a valuable proxy for more general conditional generation, allowing us to
precisely measure a model’s ability to manipulate linguistic style while preserving meaning. This
controlled setup is necessitated by the lack of comprehensive parallel corpora for other generative
tasks (e.g., summarization, open-ended dialogue) across the 25 dialects. We employ zero-shot inference using a simple instructional prompt, as shown below.
For our causal experiments (Sec. 4.4), we finetune models on a bidirectional rewriting objective (MSA _↔_ dialect). This prevents the
model from simply degrading its high-resource
MSA representations to favor dialects, a potential confounding factor in unidirectional training. By preserving MSA capabilities, we ensure a fairer assessment of subspace dynamics
and the true impact of our intervention on dialectal generation.
4.2 QUANTIFYING PERFORMANCE
AND REPRESENTATIONAL GEOMETRY
**Evaluation.** We quantify generation quality
using chrF++ (Popovi´c, 2015), a character ngram F-score. Its character-level nature is wellsuited for the morphological richness of Arabic
and is robust to minor lexical variations common between dialects. We acknowledge that,
like many automated metrics, chrF++ cannot
fully capture the nuances of ”dialectness,” especially given the high lexical overlap between
Arabic varieties (Fig. 2). However, in the absence of better-suited metrics for this specific
cross-dialectal evaluation, it serves as a reliable
indicator of generative accuracy.

When Alignment Hurts: Decoupling Representational Spaces in Multilingual Models

WHEN ALIGNMENT HURTS: DECOUPLING REPRESEN### TATIONAL SPACES IN MULTILINGUAL MODELS

Ahmed Elshabrawy
Hour Kaing
Haiyue Song
Alham Fikri Aji
Hideki Tanaka
Masao Utiyama
Raj Dabre


Rewrite the following from Modern Standard Arabic to the dialect of the Cairo city
dialect. **MSA phrase:** _{{_ MSA Sentence _}}_