**Instruction Following.** We use the the 805 instruction following tasks from Alpaca Eval (Dubois
et al., 2024) to evaluate models head to head win
rate against our baselines using the recommended
evaluator: alpaca_eval_gpt4_turbo_fn.
We use SFT and SFT + DPO described in Section 4.1 as the baselines for win rate comparisons.
Table 2: Experimental results of supervised fine-tuning on Open Assistant dataset. PT denotes pre-trained Llama2
70B with 5-shot demonstration. SFT [fact] denotes the variant which only optimizes factuality. FS denotes FActScore.
|Llama-2 70B|src. of supervision Alpaca Eval Bio Alpaca Fact FAVA
Human PT win rate over (1) FS # Corr. / Err. FS # Corr. / Err. FS # Corr. / Err.|Col3|Col4|Col5|Col6|
|---|---|---|---|---|---|
|(0) PT|-
-|-|53.1
15.3 / 13.5|-
-|-
-|
|(1) SFT
(2) SFTfact
(3) SFT|✓
✗
✗
✓
✓_∗_
✓_∗_|50.0
48.1
**51.2**|44.7
21.1 / 26.8
48.5
19.6 / 20.6
**49.5**
19.9 / 19.5|38.6
16.7 / 29.0
**42.0**
17.5 / 28.4
41.4
18.3 / 27.7|**54.4**
21.2 / 25.8
53.3
18.3 / 24.2
54.2
19.3 / 22.4|
_∗_ SFT uses supervision from Human and PT for non-fact-based and fact-based instructions, respectively.
**Factuality.** We evaluate models on three datasets
with diverse knowledge-intensive instructions for
factuality. (1) Biography: a knowledge insensitive
sub-task of instruction following tasks. Following
our pilot study in Section 3, we use the 183 human entities provided by Min et al. (2023) with
the prompt “Tell me a bio of entity name”. (2)
Alpaca Fact: we extract the fact-based instructions
from the 803 instructions using our SFT model
(with the prompt shown in Appendix, Figure 5),
resulting in 241 instructions. (3) FAVA (Mishra
et al., 2024) [6] : the 141 knowledge-intensive instructions from multiple sources, including Open Assistant (Köpf et al., 2023), No Robots (Rajani et al.,
2023), WebNLG (Gardent et al., 2017) and manually created datasets. We report FActScore (FS)
without length penalty as the metric for all the three
datasets. Note that original FS computes proportion of correct facts with additional penalty on short
generations with less than 10 atomic facts. This
penalty aims to address situations where models
provide insufficiently detailed answers. We assume
this aspect is considered in the evaluation of instruction following in Alpaca Eval. In addition, we
also report the number of correct and erroneous
facts. All the numbers reported are averaged over
the instructions in each dataset.
In addition, we also evaluate our fine-tuned
models’ truthfulness using TruthfulQA (Lin et al.,
2022). We evaluate model performance in the
generation task and use ROUGE (Lin, 2004) and
BLEU (Papineni et al., 2002) to measure the quality
of responses.


Table 2 compares the pre-trained Llama-2 70B
fine-tuned on OASST dataset with responses from
different sources. We list the FActScore (FS) of
biography generation using the pre-trained model
through Bio 5-shot demonstration as reference (row
[6FAVA dataset](https://huggingface.co/datasets/fava-uw/fava-data/blob/main/annotations.json)
0) and SFT, which is fine-tuned on our seed data
with human created responses, is our baseline (row
1). We first notice that SFT shows significant
FActScore degrade (53.1 vs 44.7) compared to Bio
5-shot with the pre-trained model. It seems that
SFT tends to generate more lengthy responses but
with more erroneous facts.
When eliciting the knowledge from PT by finetuning on its own generated responses, SFT [fact] generates more factual responses in Biography and Alpaca (row 2 vs 1). However, it shows slightly inferior instruction following capability in Alpaca Eval.
This result demonstrates that human responses indeed teach LLMs how to better follow instructions
but also encourage LLMs to output more false facts.
On the other hand, eliciting the knowledge from
the pre-trained model itself avoids the encouragement of hallucination albeit with a slight reduction in instruction-following capability. Finally,
SFT combining supervision from humans and
PT, shows comparable instruction following capability and output more factual responses on factbased instructions (row 3 vs 1).

FLAME: Factuality-Aware Alignment for Large Language Models

Sheng-Chieh Lin
Luyu Gao
Barlas Oguz
Wenhan Xiong
Jimmy Lin
Wen-tau Yih
Xilun Chen


Table 3 compares different DPO training recipes.
First, we conduct DPO fine-tuning on our SFT
baseline, SFT. When further aligning the model
to follow instructions, DPO sees a significant improvement in instruction following capability (row
2 vs 1) with win rate 72.9 over SFT; however, the
instruction aligned model tends to output lengthy
responses with more factual errors (see examples
in Appendix, Figure 11). On the other hand, when
only aligned with factual preference data, DPO [fact]
shows less improvement in instruction following
capability (row 1 vs 3). These results indicate that
preference optimization for either instruction following or factuality alone may come at the expense
of the other since the former encourages models to
output long and detailed responses while the later
Table 3: Experiments of direct preference optimization (DPO). IF. and Fact. denote instruction following ( _x, y_ + _, y−_ )
and factuality ( _x ∈_ _X_ [fact] _, y_ true _, y_ false) preference data, where _X_ [fact] denotes the set of fact-based instructions. DPO [fact]
denotes the variant which only optimizes factuality. The preference data statistics is listed in Appendix, Table 9.
|Llama-2 70B|src. of supervision
IF. Fact.|Alpaca Eval Bio Alpaca Fact FAVA
win rate over (2) FS # Corr. / Err. FS # Corr. / Err. FS # Corr. / Err.|Col4|Col5|Col6|
|---|---|---|---|---|---|
|(0) Chat|Proprietary data|66.2|33.2
23.4 / 43.6|39.3
22.3 / 36.4|47.5
28.0 / 31.3|
|(1) SFT|-
-|27.1|44.7
21.1 / 26.8|38.6
16.7 / 29.0|54.4
21.2 / 25.8|
|(2) + DPO
(3) + DPOfact
(4) + DPO|✓
✗
✗
✓
✓
✓|50.0
40.8
**51.7**|42.3
24.6 / 35.0
47.1
19.8 / 23.9
44.9
23.7 / 30.3|41.6
22.9 / 34.6
48.2
17.5 / 19.0
45.0
23.1 / 28.7|52.9
28.1 / 26.8
57.9
20.0 / 15.9
56.4
27.1 / 23.3|
|(5) SFT|-
-|29.1|**49.5**
19.9 / 19.5|41.4
18.3 / 27.7|54.2
19.3 / 22.4|
|(6) + DPO
(7) + DPO|✓
✗
✓
✓|50.4
51.2|46.3
24.0 / 28.7
47.9
25.9 / 28.5|43.9
21.6 / 28.8
**48.7**
24.1 / 25.5|55.0
25.4 / 22.0
**58.9**
29.0 / 22.2|
Table 4: Results on TruthfulQA.
|Llama-2 70B|src. of supervision
IF. Fact.|TruthfulQA
BLUE ROUGE|
|---|---|---|
|(0) Chat|Proprietary data|0.21
1.16|
|(1) SFT|-
-|0.37
0.20|
|(2) + DPO
(3) + DPOfact
(4) + DPO|✓
✗
✗
✓
✓
✓|0.03
0.54
0.30
1.12
0.15
0.80|
|(5) SFT|-
-|0.39
0.51|
|(6) + DPO
(7) + DPO|✓
✗
✓
✓|0.07
0.91
0.20
0.96|
discourages models to output false claims. When
jointly conducting instruction and factuality alignment, DPO not only better follows instructions
but also outputs more factual responses (row 4 vs
1, 2). Finally, initializing from SFT, the DPO
fine-tuned models are more factual than their counterparts (i.e., 6 vs 2 and 7 vs 4) without instruction
following capability degrade. We also list the results from Llama-2-Chat 70B (row 0) and observe
that despite of its strong instruction following capability, it tends to output many more incorrect
facts. This results demonstrate that standard alignment, even on proprietary commercial data, may
encourage LLMs to hallucinate. In contrast, our
factuality-aware alignment guides LLMs to output more factual responses without degradation in
their general instruction following capabilities. It
is worth noting that SFT [fact] and DPO [fact] are similar to SFT and DPO fine-tuning proposed by Tian
et al. (2024), which improve LLMs’ factuality but
degrade instruction following capability.


Table 4 compares models performance on TruthfulQA. Generally, we observe that our factuality
aware alignment training guides LLMs to output
more truthful responses. For example, factualityaware SFT improves LLMs’ truthfulness (row 5
vs 1). In addition, DPO fine-tuning on the factuality preference data guides LLMs to output more
truthful responses (rows 3,4 vs 2 and 7 vs 6). Note
that we observe that SFT and DPO models show a
reverse trend in BLUE and ROUGE. This is likely
because SFT models tend to generate shorter responses than the DPO ones do.


Table 5: Effects of fact-based classification on factualityaware alignment.
|Col1|Classifei r Alpaca Eval Bio
Inst. Sent. win rate FS # Corr. / Err.|Col3|Col4|
|---|---|---|---|
|(1) SFT
(2)|✗
-
✓
-|47.6~~_∗_~~
51.2_∗_
|48.4 20.5 / 21.4
49.5 19.9 / 19.5|
|(3)
SFT + DPO
(4)
(5)|✗
✗
✓
✗
✓
✓|46.8~~_△_~~
51.7_△_
51.3_△_|46.8 21.7 / 25.3
45.0 23.7 / 30.3
42.9 25.5 / 36.8|
_∗_ comparing with SFT baseline, SFT.
_△_ comparing with DPO baseline, SFT + DPO.
**Effects of Fact-Based Instruction Classification.**
In our factuality-aware alignment, we prompt SFT
to judge whether an instruction requires a factual response and apply our factuality alignment strategy
to the fact-based instruction. Without the instruction classification, in our factuality-aware SFT, we
cannot create supervision from Human and PT
responses for respective non-fact-based and factbased instructions. Instead, for each instruction,
we create instruction–response pairs from 1 and 10
responses from Human and PT as supervisions,
respectively. Note that, during fine-tuning, for
each instruction, we randomly sample instruction–
Table 6: Ablation on factuality preference data creation.
|Factuality preference data
Reward model Pos.,Neg. # pairs|Alpaca Eval Bio
win rate△ FS|Col3|
|---|---|---|
|RM~~fact~~
max, min
3,315
RMfact
enum.
5,126
RMIF + 5_∗_RMfact
max, min
6,340|51.7
50.7
50.1|44.9
45.0
45.1|
_△_ comparing with DPO baseline, SFT + DPO.
response pair either created from Human or PT
with same probability. The SFT model shows
degradation in both instruction following capability
and factuality results, as shown in row 1 vs 2 of
Table 5. Second, for factuality-aware DPO, without
the instruction classification, we create factuality
preference pairs from all instructions instead of
fact-based instructions. The DPO fine-tuned model
outputs slightly more factual responses but sacrifice instruction following capability, as shown in
row 3 vs 4 of Table 5.
**Effects of Fact-Based Sentence Classification.**
In addition, we observe that not all the sentences
in a response to a fact-based instruction require
fact check. For example, given the response, “Of
course. The Commodore 64 is a 8-bit home computer that was released by Commodore International in August 1982.”, conducting fact check for
the first sentence “Of course.” is not necessary
and may make the factuality reward less accurate.
To address this issue, we prompt SFT to judge
whether each sentence in a response required fact
check using the prompt in Appendix, Figure 7. We
only conduct fact check and compute factuality rewards for those fact-based sentences. However, as
shown in Table 5, computing factuality rewards for
fact-based sentences makes our factual alignment
less effective (row 5 vs 4). This is likely because
the fact-based sentence classifier is not accurate
enough and brings noise into our factuality reward
model (see examples in Appendix, Figure 8).


**ation.** In this section, we examine different ways
of creating factuality preference data for factualityaware DPO training. First, for each fact-based instruction, instead of choosing the responses (among
the 4 generated responses) with the maximum and
minimum factuality rewards (RM [fact] ) as the respective positive and negative samples, we enumerate
all the possible response pairs and choose the response with higher (lower) RM [fact] as the positive
(negative) sample from each enumerated pair. If
Table 7: Effects of DPO training on response length.
|Col1|Alpaca Eval|Bio|Alpaca Fact|FAVA|
|---|---|---|---|---|
|(1) SFT|897|1221|969|912|
|(2) + DPO
(3) + DPOfact
(4) + DPO|1470
1160
1474|1494
1166
1395|1586
1192
1528|1540
1104
1422|
the difference of RM [fact] is smaller than 0 _._ 2, we
treat them as equal and discard the pairs. Note that
for both row 1 and 2 in Table 6, we also discard
the pairs with the difference of instruction following rewards (RM [IF] ) larger than 0 _._ 5 (as mentioned
in Section 4.2.2). Finally, for each response, we
linearly combine the rewards of RM [IF] (1–5 scale)
and RM [fact] (0–1 scale) with the respective weight
of 1 and 5 as a composite reward. For each instruction, we choose the responses with the maximum
and minimum composite rewards as the positive
and negative. As shown in Table 6, both data creation approaches increase the number of pairs in
factuality preference data; however, it yields no obvious improvement in factuality but a bit degrade
in instruction following capability (rows 2, 3 vs 1).
**Impacts of DPO on Generation Length.** Table 7 lists the averaged length of models’ responses
for each dataset. We observe that DPO fine-tuned
models tend to output lengthy responses than SFT
except for DPO [fact] on Biography. This trend indicates that our instruction following reward model
RM [IF] guides LLMs to output more detailed and
lengthy responses. In addition, we observe that
although DPO outputs responses with similar
length as DPO on Alpaca Eval, DPO generates
a bit shorter responses for the fact-based instructions in the other three datasets. This results show
that our factuality-aware DPO training mainly impacts models’ responses for fact-based instructions.
The impact is mainly to reduce the output of false
claims (see the numbers of correct and erroneous
facts in rows 2 and 4 of Table 3).
**Case Studies.** Figure 11 (in Appendix) showcases the generations of different models, SFT,
SFT + DPO and SFT + DPO, on Alpaca
Eval and Biography. Given the instruction, “What
are the names of some famous actors that started
their careers on Broadway?”, SFT only lists some
names of Broadway actors while DPO fine-tuned
models generate detailed information for each
listed Broadway actor. As for biography generations, we observe that given the instruction to gen
erate a biography for a rare name entity, Marianne
McAndrew, SFT + DPO generates a detailed response but with many wrong facts while SFT and
SFT + DPO give relatively short responses. For
the frequent entity, Ji Sung, all the models generate
detailed and mostly correct responses. This qualitative analysis shows that SFT + DPO tends to
generate detailed responses for most instructions
but for those instructions required tailed knowledge
(e.g., rare entity) likely unknown to LLMs (Mallen
et al., 2023), it manages to reduce erroneous facts
by giving less detailed responses, which is also
observed by Kang et al. (2024).

Ablations on Factuality Preference Data Cre-


In this paper, we present a study to enhance the
factuality of large language models (LLMs). We
first identify that the standard alignment approach,
comprising SFT and RLAIF with DPO, may inadvertently encourage LLMs to produce more erroneous facts. Specifically, during the SFT stage, finetuning LLMs with high-quality human responses
may introduce unfamiliar information, prompting
LLMs to output unknown facts. Additionally, during the DPO stage, enhancing LLMs’ ability to
follow instructions may result in more detailed
and lengthy responses but often leads to increased
hallucination. To tackle the shortcomings of the
standard alignment, we propose a factuality-aware
alignment method, which includes factuality-aware
SFT and DPO. Quantitative and qualitative analyses demonstrate that our factuality-aware alignment not only guides LLMs to generate detailed
and helpful responses but also helps prevent the
generation of false claims.


While we have successfully integrated factuality
into standard alignment procedure, our work only
considers two alignment skill sets: instruction following (or helpfulness) and factuality. In practice,
each instruction may require consideration of multiple and distinct alignment skill sets (Saha et al.,
2023). The method to optimize for these skill sets
tailored to each query requires further study. In
our experiments, we note that optimizing preferences solely for instruction following or factuality
could potentially compromise the other. While our
factuality-aware alignment demonstrated improvements in both aspects, it is uncertain whether there
is a trade-off between the two aspects when inte
grating our approach to large-scale alignment (Touvron et al., 2023). Finally, as shown in Appendix,
Figure 8, not all the claims (or sentences) in a response require fact verification, a more accurate
factuality reward model should take the factor into
account. While our preliminary experiment, which
removing non-fact-based sentences from the factuality reward modeling (Section 5.5), shows suboptimal performance, we believe that further study
can bring more insights.


We thank Bhargavi Paranjape for sharing fine-tuned
Llama-2 7B for atomic fact decomposition and
Jing Xu, Weizhe Yuan and Jason Weston for their
helpful suggestions.


**Entities for Training and Evaluation.** We use
500 diverse human entities to create training data
for SFT and DPO; then, evaluate LLMs’ generation
factuality on another 183 human entities from Min
et al. (2023). [7] Note that the human entities for
training and evaluation are uniformly sampled from
entities across diverse nationalities, professions,
and rarities. The instruction is generated with the
format: Tell me a bio of entity name.
**Creating Training Data for SFT.** We randomly
sample 5 human entities among the 500 entities
for training and generate their biographies using Llama-2-Chat 70B as 5-shot demonstration. [8]
With the 5-shot demonstration, we use pre-trained
Llama-2 7B to generate 10 biographies for each
human entity from the remaining 495 ones. [9] We set
temperature 0.7 and top-p 0.9 when generate multiple responses from LLMs in all our experiments.
We use the created 4,950 name entity–biography
pairs to fine-tune the pre-trained Llama-2 7B. As
for generating training data with RAG, we prepend
the top-10 passages from our retrieval system (detailed in Appendix A.2) to each instruction and
generate 10 biographies for each entity from RAG
with 5-shot demonstrations. Note that we only
prepend top-1 passage for each instruction in the
demonstration.
**Creating Factuality Preference Pairs for DPO.**
To construct factuality preference pairs, we first
compute FActScore (FS) for all the 4,950 biographies previously created by PT. Then, for each
name entity, we compare the FS for all the possible
45 pairs from the 10 generated biographies and construct DPO pairs using the biography with a higher
(lower) FS as a positive (negative). Note that we
discard the pairs if they show tied FS.


For each query, we retrieve top-20 candidate passages from Wikipedia using DRAGON+ (Lin et al.,
2023) and re-rank the candidates using a 12-layer
cross-encoder [10] . We use the Wikipedia version
from the Dec. 20, 2021 dump released by Izacard
et al. (2023) in this work.
[7https://github.com/shmsw25/FActScore](https://github.com/shmsw25/FActScore)
[8meta-llama/Llama-2-70b-chat-hf](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf)
[9meta-llama/Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b)
[10sentence-transformers/all-MiniLM-L12-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L12-v2)


**SFT.** At SFT stage, we fine-tune PT on two
seed datasets: (1) Instruction following training
(IFT) data from Li et al. (2024), consisting of 3200
instruction–response pairs created by humans from
Open Assistant dataset (OASST; Köpf et al., 2023),
where we only use the first conversational turns in
the English that are annotated rank 0; [11] (2) evaluation following training (EFT) data from Yuan
et al. (2024), the LLM-as-a-Judge data consists of
1630 samples, each of which contains instruction,
human response and the corresponding score of 1-5
scale (with chain-of-though evaluation reasoning):
( _x, y, r_ ), where ( _x, y_ ) pairs are also selected from
OASST other than training pairs and _r_ is created by
the model fine-tuned only on IFT with manual filtering. The purpose of EFT is to enhance a LLM’s
capability as a reward model to judge the quality
of a response in terms of relevance, coverage, usefulness, clarity and expertise. We refer readers to
Yuan et al. (2024) for how EFT is created and filtered with minimum human efforts. The prompt
template for LLM-as-a-Judge in EFT and an EFT
training sample are shown in Appendix, Figure 9
and 10. We refer the baseline model fine-tuned on
the IFT and EFT datasets as SFT.
**DPO for Instruction Following.** At the subsequent preference learning with DPO, following Wang et al. (2023a), we augment additional
20K instructions with Llama-2 70B chat model. [12]
For each augmented instruction _x_, we use SFT
to generate 4 responses and evaluate how well
the responses follow the instruction with score
of 1–5 scale: RM [IF] ( _x, y_ 0) _· · ·_ ; RM [IF] ( _x, y_ 3), where
_y_ 0 _, · · ·, y_ 3 _∈_ SFT( _x_ ) and RM [IF] is the instruction following reward model. Note that, in selfrewarding (Yuan et al., 2024), RM [IF] is the same
as SFT model. In addition, for each instruction–
response pair, we use the same prompt in EFT
seed data to sample the chain-of-thought evaluation three times and average the scores as the reward. Finally, for each instruction, we use the
response with the highest (lowest) reward as the
positive (negative) sample to form a preference
pair for DPO training: ( _x, y_ + _, y−_ ). We discard the
pair, if RM [IF] ( _x, y_ +) = RM [IF] ( _x, y−_ ). In the DPO
training, the model is initialized from SFT and the
fine-tuned model is denoted SFT + DPO.
[11OpenAssistant/oasst1](https://huggingface.co/datasets/OpenAssistant/oasst1)
[12meta-llama/Llama-2-70b-chat-hf](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf)
Table 8: A comparison of factuality reward models. _τ_
denotes the correlation between human annotation.
fact check model # sup. fact unit _τ_
(1) 5 0.32
Instruct Llama 7B atom.
(2) 10 0.34
(3) 5 0.28
SFT (Llama-2 70B) atom.
(4) 10 0.31
(5) 5 0.20
Instruct Llama 7B sent.
(6) 10 0.25


**Factuality Reward Models.** We build a reward
model RM [fact] to measure the factuality of each response. The factuality reward model consists of
two main modules. (1) fact decomposition: we first
use nltk.tokenize to split a response into sentences; then, use our Llama-2 7B model fine-tuned
on public datasets (Liu et al., 2023; Chen et al.,
2022; Malaviya et al., 2023) to conduct atomic fact
decomposition for each sentence. [13] (2) Retrieval
augmented claim verification: for each decomposed
fact (or claim), we use the instruct Llama 7B finetuned on Super Natural Instructions (Wang et al.,
2022) to do fact check with the prompt shown in
Figure 6. [14] We append 10 retrieved supports (using the instruction as query) from our retrieval and
re-ranking pipeline in Appendix A.2. Then, we
compute the proportion of correct atomic facts in a
response as a factuality reward.
**Quality of Factuality Reward Models.** We conduct ablation study on our factuality reward models.
Specifically, we use our factuality reward models to
detect the number of error facts in each instruction–
response pair. We try different models for fact
check using the prompt shown in Figure 6 with
different numbers of retrieved supports. We use the
LLMs’ generated responses with human annotated
hallucination provided by Mishra et al. (2024) to
evaluate the quality of the factuality reward models. [15] Specifically, we rank the responses by numbers of errors detected and calculate the Kendall
rank correlation ( _τ_ ) between the rank lists by our
factuality reward models and humans. As shown in
Table 8, conducing fact check with more retrieved
13With few-shot demonstration, SFT is able to decompose a
sentence into atomic facts with acceptable accuracy. Finetuning a Llama-2 7B is to reduce the inference time.
[14instruct Llama 7B](https://huggingface.co/kalpeshk2011/instruct-llama-7b-wdiff)
[15https://huggingface.co/datasets/](https://huggingface.co/datasets/fava-uw/fava-data/blob/main/annotations.json)
[fava-uw/fava-data/blob/main/](https://huggingface.co/datasets/fava-uw/fava-data/blob/main/annotations.json)
[annotations.json](https://huggingface.co/datasets/fava-uw/fava-data/blob/main/annotations.json)
Table 9: Training data statistics for different variants.
IF. and Fact. denote instruction following ( _x, y_ + _, y−_ )
and factuality ( _x ∈_ _X_ [fact] _, y_ true _, y_ false) preference data,
where _X_ [fact] denotes the set of fact-based instructions.
|model variant|Seed IFT (# of Inst.) Preference (# of pairs)
x ∈/Xfact x ∈Xfact IF. Fact.|Col3|
|---|---|---|
|SFT + DPO
SFT + DPOfact
SFT + DPO|2,187
1,013|18,454
-
-
3,315
18,454
3,315|
|SFT + DPO
SFT + DPO|2,187
1,013|18,603
-
18,603
4,211|
supports improves the accuracy of the factuality
reward models (row 2 vs 1). In addition, our SFT,
only fine-tuned on the IFT and EFT data, is capable
of doing fact check, compared to Instruct Llama
7B fine-tuned on Super Natural Instructions (Wang
et al., 2022). Finally, instead of computing the number of error facts from decomposed atomic facts,
we conduct fact check directly for each sentence in
a response and calculate the number of false sentences as error facts. However, the quality of the
reward models shows significant decrease (rows
5,6 vs 1,2). We finally adopt row 2 as our factuality
reward model.


We fine-tune our models for 500 steps with a
batch size of 32 and 64 on respective SFT and
DPO stages. The learning rate and maximum sequence length is set to 1 _e −_ 6 (which decays to
1 _e −_ 7) and 2048, respectively. At SFT stage, we
mix the IFT and EFT while at DPO stage, we set
_β_ = 0 _._ 1 and uniformly sample between self rewarding ( _x, y_ + _, y−_ ) and factuality reward ( _x, y_ true _, y_ false)
preference data. Note that SFT (SFT ) + DPO
meaning that we use SFT (SFT ) to create preference data, serve as instruction following reward
model RM [IF] and as the initialization of DPO. The
data used to fine-tune different variants are listed
in Table 9.
|Figure 5: Prompt to check whether an instruction is fact-based.|Col2|Col3|
|---|---|---|
|[INST] Answer the question about '<Instruction>' based on the given context.\n\n'<Retrieved Supports>'\n\nInput:
'<ATOMIC_FACT or SENTENCE>' True or False?\nOutput: [/INST]|[INST] Answer the question about '<Instruction>' based on the given context.\n\n'<Retrieved Supports>'\n\nInput:
'<ATOMIC_FACT or SENTENCE>' True or False?\nOutput: [/INST]|[INST] Answer the question about '<Instruction>' based on the given context.\n\n'<Retrieved Supports>'\n\nInput:
'<ATOMIC_FACT or SENTENCE>' True or False?\nOutput: [/INST]|
|Figure 6: Prompt for fact check.|Figure 6: Prompt for fact check.|Figure 6: Prompt for fact check.|
|[INST] Does the given sentence contain factual knowledge from Wikipedia? Question: '<QUERY>'; Sentence: '<CLAIM>'. Answer
with Yes or No. [/INST]|[INST] Does the given sentence contain factual knowledge from Wikipedia? Question: '<QUERY>'; Sentence: '<CLAIM>'. Answer
with Yes or No. [/INST]|[INST] Does the given sentence contain factual knowledge from Wikipedia? Question: '<QUERY>'; Sentence: '<CLAIM>'. Answer
with Yes or No. [/INST]|
|Figure 7: Prompt to check whether a claim is fact-based.|Figure 7: Prompt to check whether a claim is fact-based.|Figure 7: Prompt to check whether a claim is fact-based.|
|**Do you have any information about the Commodore 64?**|**Hi, could you help me to solve this cubic equation using Cardano's**
**Mehod (step by step if possible), please? -> " x^3 + 2x^2 - x - 1 = 0 "**|
**Explain the significance of the American Revolution, including the**
**events that led up to it, the impact it had on the world, and its ongoing**
**relevance today.**|
|Of course. --False
The Commodore 64 is a 8-bit home computer that was released by
|
Sure, I can help you solve the cubic equation x^3 + 2x^2 - x - 1 = 0 using
Cardano's method. --False
|

The American Revolution was a pivotal moment in world history that
marked the birth of the United States as a nation. --True
|
|Commodore International in August 1982. --True
It is one of the most successful home computers of all time, with over 17
million units sold. --True
It featured 64 kilobytes of RAM, color graphics, and the SID sound chip. --
True,It was widely used for gaming, programming, and home productivity
applications. --True
It was succeeded by the Commodore 128. --False
There is still a thriving community of Commodore 64 enthusiasts around the
world who continue to enjoy playing and programming on the system. --
True
You can find more information about the Commodore 64 on various websites
and forums dedicated to the system. --False
If you have any specific questions about the Commodore 64's hardware,
software, or programming languages, please feel free to ask. --False|

Here are the steps: Step 1: Put the equation in the form x^3 + px + q = 0 x^3
+ 2x^2 - x - 1 = 0 can be rearranged as x^3 + 0x^2 + 2x^2 - x - 1 = 0, so we
have p = 0 and q = -1. --False
Step 2: Substitute x = y - b/3 in the equation, where b = -p/3 b = -p/3 = 0/3 =
0, so we have b = 0. --False
Substituting x = y in the equation, we get: (y - 0)^3 + 0(y - 0) - 1 = 0 y^3 - 1
= 0 Step 3: Solve for y y^3 - 1 = 0 can be factored as (y - 1)(y^2 + y + 1) = 0.
--False
The roots of this equation are: y = 1 and y = (-1 ± sqrt(3)i)/2 Step 4: Find the
roots of the original equation Substituting y = 1 into x = y - b/3, we get x = 1
- 0/3 = 1. --False
So, 1 is one of the roots of the original equation. --False
Substituting y = (-1 + sqrt(3)i)/2 into x = y - b/3, we get: x = (-1 + sqrt(3)i)/2
- 0/3 = (-1 + sqrt(3)i)/2 Substituting y = (-1 - sqrt(3)i)/2 into x = y - b/3, we
get: x = (-1 - sqrt(3)i)/2 - 0/3 = (-1 - sqrt(3)i)/2 So, the roots of the original
equation are: x = 1, x = (-1 + sqrt(3)i)/2, and x = (-1 - sqrt(3)i)/2 These are
the three complex roots of the cubic equation x^3 + 2x^2 - x - 1 = 0. --True|

The events leading up to the Revolution began when the American colonies
were still under British rule and were becoming increasingly discontent with
the taxes and regulations imposed upon them by the British government. --
True
One of the most significant events leading up to the Revolution was the
Boston Tea Party, where American colonists dumped British tea into the
harbor as a symbol of their anger towards the British tea tax. --True
This event was followed by a series of skirmishes between British troops and
American colonists, which escalated into full-blown warfare. --False
The impact of the American Revolution was far-reaching, both domestically
and internationally. --True
The Revolution inspired people around the world to fight for their own
freedoms and led to the spread of democratic ideals. --True
In the United States, the Revolution led to the formation of a new
government based on democratic principles, where citizens had a say in the
laws that governed them. --True
Today, the American Revolution continues to be relevant as its ideals of
democracy, freedom, and self-governance continue to inspire people around
the world. --True
The American Revolution serves as a reminder that ordinary people can
come together to fight for their rights and make a real difference in the
world. -- False
It is a testament to the power of the human spirit and the desire for freedom.
--False|
Figure 8: The results of whether a sentence is fact-based or not classified by SFT with prompt in Figure 7.
[INST] Review the user’s question and the corresponding response using the additive 5-point scoring system described below. Points
are accumulated based on the satisfaction of each criterion:
- Add 1 point if the response is relevant and provides some information related to the user’s inquiry, even if it is incomplete or
contains some irrelevant content.
- Add another point if the response addresses a substantial portion of the user’s question, but does not completely resolve the
query or provide a direct answer.
- Award a third point if the response answers the basic elements of the user’s question in a useful way, regardless of whether it
seems to have been written by an AI Assistant or if it has elements typically found in blogs or search results.
- Grant a fourth point if the response is clearly written from an AI Assistant’s perspective, addressing the user’s question
directly and comprehensively, and is well-organized and helpful, even if there is slight room for improvement in clarity,
conciseness or focus.
- Bestow a fifth point for a response that is impeccably tailored to the user’s question by an AI Assistant, without extraneous
information, reflecting expert knowledge, and demonstrating a high-quality, engaging, and insightful answer.
User: <INSTRUCTION_HERE>
<response><RESPONSE_HERE></response>
After examining the user’s instruction and the response:
- Briefly justify your total score, up to 100 words.
- Conclude with the score using the format: “Score: <total points>”
Remember to assess from the AI Assistant perspective, utilizing web search knowledge as necessary. To evaluate the response in
alignment with this additive scoring model, we’ll systematically attribute points based on the outlined criteria. [/INST]
Figure 9: Prompt to evaluate models’ instruction following capability from Yuan et al. (2024).
Figure 10: An example of EFT data. The texts with the colors of green, red and blue are the instruction, response
and the LLM-as-a-judge results (explanation and score), respectively
|SFT|SFT + DPO|SFT + DPO|
|---|---|---|
|**What are the names of some famous actors that started their careers on Broadway?**

|**What are the names of some famous actors that started their careers on Broadway?**

|**What are the names of some famous actors that started their careers on Broadway?**

|
|1.  Robert Redford\n2.  Meryl Streep\n3.  Al Pacino\n4.  Denzel
Washington\n5.  Liza Minnelli\n6.  James Earl Jones\n7.
Bernadette Peters\n8.  Nathan Lane\n9.  Patti LuPone\n10.
Kevin Kline\n11.
 Matthew Broderick\n12.
 Whoopi
Goldberg\n13.  Hugh Jackman\n14.  Idina Menzel\n15.  Neil
Patrick Harris\n16.
 Audra McDonald\n17.
 Kristin
Chenoweth\n18.  John Lithgow\n19.  Bebe Neuwirth\n20.
Harvey Fierstein\n21.  Chita Rivera\n22.  Ben Vereen\n23.  Joel
Grey\n24.  John Cullum\n25.  John Rubinstein\n26.  John
Glover\n27.  John C. Reilly\n28.  John Turturro\n29.  John
Mahoney\n30.  John Larroquette\n31.  John Slattery\n32.  John
Benjamin Hickey\n33.
 John Gallagher Jr.\n34.
 John
Leguizamo\n35.  John Goodman\n36.  John Lithgow\n37.  John
C. McGinley\n38.**John Cusack**\n39.  John C. Reilly\n40.  John
Turturro\n41.  John C. McGinley\n42.**John Cusack**\n43.  John
C. Reilly\n44.  John Turturro\n45.  John C. McGinley\n46.
**John Cusack**\n47.  John C. Reilly\n48.  John Turturro\n49.
John C. McGinley\n50.**John Cusack**\n51.  John C. Reilly\n52.
John Turturro\n53.  John C. McGinley\n54.**John Cusack**\n55.
John C. Reilly\n56.  John Turt (cut by maximum generation
length)|1.
 Robert Redford: Redford began his acting career on
Broadway in the late 1950s, appearing in productions such as
\"Tall Story\" and \"Sunday in New York.\"\n2.  Meryl Streep:
Streep made her Broadway debut in 1975 in the play \"Trelawny
of the 'Wells.'\" She went on to**win three Tony Awards** for her
work on Broadway.\n3.  Al Pacino: Pacino made his Broadway
debut in 1969 in the play \"Does a Tiger Wear a Necktie?\" He
went on to win two Tony Awards for his work on Broadway.\n4.
Dustin Hoffman: Hoffman made his Broadway debut in 1961 in
the play \"A Cook for Mr. General.\" He went on to win**two**
**Tony Awards**for his work on Broadway.\n5.
 Denzel
Washington: Washington made his Broadway debut in**1982 in**
**the play \"A Soldier's Play.\"** He went on to win**two Tony**
**Awards** for his work on Broadway.\n6.  Glenn Close: Close
made her Broadway debut in 1974 in the musical \"Love for
Love.\" She went on to win three Tony Awards for her work on
Broadway.\n7.  Kevin Spacey: Spacey made his Broadway debut
in 1982 in the play \"Ghosts.\" He went on to win**two Tony**
**Awards** for his work on Broadway.\n8.  Whoopi Goldberg:
Goldberg made her Broadway debut in 1984 in the play \"**Ma**
**Rainey's Black Bottom.**\" She went on to win a Tony Award for
her work on Broadway.\n9.  Nathan Lane: Lane made his
Broadway debut in 1982 in the play \"Present Laughter.\" He
went on to win three Tony Awards for his work on Broadway.
\n10.  Bernadette Peters: Peters made her Broadway debut in
**1968** in the musical \"**George M!**\" She went on to win**three**
**Tony Awards**for her work on Broadway.\n11.  Patti LuPone:
LuPone made her Broadway debut in**1972** in the musical \"**The**
**Threepenny Opera.**\" She went on to win two Tony Awards for
her work on Broadway.\n12.  L (cut by maximum generation
length)|There are many famous actors who started their careers on
Broadway. Here are some of the most well-known:\n1. Meryl
Streep: Streep made her Broadway debut in 1975 in the play
"Trelawny of the 'Wells.'" She went on to win three Academy
Awards and become one of the most acclaimed actresses of her
generation.\n2. Robert De Niro: De Niro made his Broadway
debut in**1965** in the play "**The Indian Wants the Bronx**." He
went on to become one of the most iconic actors in Hollywood,
winning two Academy Awards.\n3. Al Pacino: Pacino made his
Broadway debut in 1969 in the play "Does a Tiger Wear a
Necktie?" He went on to become one of the most respected
actors in Hollywood, winning an Academy Award and numerous
other accolades.\n4. Dustin Hoffman: Hoffman made his
Broadway debut in 1961 in the play "A Cook for Mr. General."
He went on to become one of the most versatile actors in
Hollywood, winning two Academy Awards.\n5. Denzel
Washington: Washington made his Broadway debut in**1982 in**
**the play "A Soldier's Play."** He went on to become one of the
most acclaimed actors of his generation, winning two Academy
Awards.\n6. Bernadette Peters: Peters made her Broadway debut
in**1968**in the musical**"George M!"** She went on to become
one of the most beloved performers on Broadway, winning**three**
**Tony Awards**.\n7. Patti LuPone: LuPone made her Broadway
debut in**1972** in the musical**"The Threepenny Opera."** She
went on to become one of the most respected performers on
Broadway, winning three Tony Awards.\n8. Nathan Lane: Lane
made his Broadway debut in 1982 in the play "Present
Laughter." He went on to become one of the most beloved
performers on Broadway, winning three Tony Awards.\n9. Audra
McDonald: McDonald made her Broadway debut in 1992 in the
musical "The Secret Garden." She went on to become one of the
most acclaimed performers on Broadway, winning six Tony
Awards.\n10. Hugh Jackman: Jackman made (cut by maximum
generation length)|
|**Tell me the bio of Marianne McAndrew (very rare entity)**|**Tell me the bio of Marianne McAndrew (very rare entity)**|**Tell me the bio of Marianne McAndrew (very rare entity)**|
|1943-2006\n\nMarianne McAndrew was an American actress
best known for her role as Irene Molloy in the**1964** film version
of Hello, Dolly! She was born in**Fort Ord, California**, in**1943**
and began her acting career in the early 1960s. She appeared in
several television shows and films throughout the decade,
including**The Fugitive**, **The Virginian**, and**The Wild Wild**
**Wes**t. In**1964**, she was cast as Irene Molloy in Hello, Dolly!,
which earned her a Golden Globe nomination for Best
Supporting Actress. After Hello, Dolly!, McAndrew continued to
act in films and television shows throughout the 1970s and
1980s. She also appeared on Broadway in the musicals**The**
**Music Man and Mame. McAndrew retired from acting in the**
**early 1990s and passed away in 2006 at the age of 63**.|1. Marianne McAndrew was an American actress best known for
her role as Irene Molloy in the**1968** film version of \"Hello,
Dolly!\"\n2. She was born on**July 20, 1943, in Chicago,**
**Illinois,** and began her acting career in**the early 1960s**.\n3.
McAndrew made her film debut in**1965 **in the movie \"**The**
**Slender Thread**\" and went on to appear in several other films
throughout the 1960s and 1970s, including \"**The Thomas**
**Crown Affair**\" and \"**The Great Gatsby**.\"\n4. In addition to
her film work, McAndrew also appeared on television, including
guest roles on popular shows such as \"**The Fugitive**\" and
\"**Mission: Impossible.\"**\n5.** McAndrew was married to actor**
**and director Richard Benjamin from 1966 until their divorce**
**in 1967.\n6. She later married actor and producer Michael**
**Tolan in 1970, and the couple had two children together.**\n7.
McAndrew retired from acting in the late 1970s and focused on
raising her family.\n8.**She passed away on February 26, 2006,**
**at the age of 62.**\n9. McAndrew's legacy as an actress lives on
through her memorable performances in films such as \"Hello,
Dolly!\" and \"**The Thomas Crown Affair.\**"\n10. She is
remembered as a talented and versatile actress who made a
lasting impact on the film industry.|Marianne McAndrew was an American actress who was best
known for her role as Irene Molloy in the**1964** film version of
the musical "Hello, Dolly!" She was born on**May 27, 1943,** **in**
**Chicago, Illinois**, and began her acting career in the early 1960s.
McAndrew made her film debut in**1963 with a small role in**
**the film "The Cardinal."** She then went on to appear in several
other films, including**"The World of Henry Orient" (1964)**
and**"The Great Race" (1965)**. However, it was her role as
Irene Molloy in "Hello, Dolly!" that brought her the most
recognition. In addition to her film work, McAndrew also
appeared on television, including in episodes of**"The**
**Virginian"**and "The F.B.I."**She retired from acting in the**
**early 1970s and later worked as a real estate agent in**
**California. McAndrew was married twice and had two**
**children. She passed away on February 26, 2006, at the age**
**of 62.** Overall, Marianne McAndrew was a talented actress who
made a lasting impact on the film industry with her memorable
performance in "Hello, Dolly!"|
|**Tell me the bio of Ji Sung (very frequent entity)**|**Tell me the bio of Ji Sung (very frequent entity)**|**Tell me the bio of Ji Sung (very frequent entity)**|
|Ji Sung is a South Korean actor who was born on February 27,
1977, in Seoul, South Korea. He began his acting career in 1999
and has since become one of the most popular and acclaimed
actors in South Korea.\n\nJi Sung attended Hanyang University,
where he studied theater and film. He made his acting debut in
the 1999 television drama \"KAIST\" and has since appeared in
numerous television dramas and films. Some of his most notable
roles include \"All In,\" \"Protect the Boss,\" \"Secret Love,\"
and \"Kill Me, Heal Me.\"\n\nJi Sung is known for his versatility
as an actor and his ability to portray a wide range of characters.
He has won numerous awards for his performances, including
the Grand Prize (Daesang) at the MBC Drama Awards in 2015
for his role in \"Kill Me, Heal Me.\"\n\nIn addition to his acting
career,Ji Sung is also known for his philanthropic work. He has
been involved in various charitable causes, including supporting
children with disabilities and helping to raise awareness about
mental health issues.\n\nJi Sung is married to actress Lee Bo-
young, and the couple has a daughter together.He is known for
being a devoted family man and often speaks about the
importance of spending time with his loved ones.\n\nOverall, Ji
Sung is a talented and respected actor who has made a
significant impact on the South Korean entertainment industry.|1. Ji Sung is a South Korean actor who was born on February**29**,
1977, in Seoul, South Korea.\n2. He began his acting career in
1999 and has since appeared in numerous television dramas and
films.\n3. Ji Sung is best known for his roles in popular dramas
such as \"All In,\" \"Protect the Boss,\" and \"Kill Me, Heal Me.
\"\n4. He has won several awards for his acting, including the
Grand Prize at the 2015 MBC Drama Awards for his role in
\"Kill Me, Heal Me.\"\n5.In addition to his acting career, Ji Sung
is also known for his philanthropic work. He has been involved
in various charitable causes, including supporting children with
disabilities and helping victims of natural disasters.\n6. Ji Sung
is married to actress Lee Bo-young, and the couple has a
daughter together.\n7. He is known for his versatility as an actor,
having played a wide range of characters in his career, from
romantic leads to villains.\n8.Ji Sung is also known for his
dedication to his craft, often putting in long hours on set and
undergoing extensive preparation for his roles.\n9. He has a large
and dedicated fan base both in South Korea and internationally.
\n10. Ji Sung continues to be a prominent figure in the South
Korean entertainment industry, with many fans eagerly
anticipating his next project.|Ji Sung is a South Korean actor who was born on February**29**,
1977, in Seoul, South Korea. He began his acting career in 1999
and has since become one of the most popular and acclaimed
actors in South Korea. Ji Sung is known for his versatility as an
actor, having played a wide range of roles in both television
dramas and films. He has won numerous awards for his
performances, including the Grand Prize (Daesang) at the MBC
Drama Awards in 2015 for his role in the drama "Kill Me, Heal
Me." Some of Ji Sung's most notable roles include his portrayal
of a man with multiple personalities in "Kill Me, Heal Me," a
**lawyer**in "Defendant," and a doctor in "Doctor John." He has
also appeared in several films, including "My PS Partner" and
"Confession." In addition to his acting career,Ji Sung is also
known for his philanthropic work. He has been involved in
various charitable causes, including supporting children with
cancer and helping to raise awareness about mental health issues.
Overall, Ji Sung is a talented and respected actor who has made
a significant impact on the South Korean entertainment industry.|
Figure 11: Generation comparisons for instructions from Alpaca Eval and Biography (very rare and frequent
entities). Determined through manual verification using Google search, red denotes incorrect identified facts while
pink indicates unverified facts; e.g., we cannot search relevant pages about Ji Sung’s involvement in charitable
causes but also cannot dismiss the possibility of his contributions. Note that the popularity of an entity is defined by
its occurrence and page views in Wikipedia, which are provided by Min et al. (2023).


**Runsheng Yu** [1] _[∗]_ **Yong Wang** [2] **Xiaoqi Jiao** [2] **Youzhi Zhang** [3] **James T. Kwok** [1]
1Hong Kong University of Science and Technology
2LightSpeed Studios, Tencent
3Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, CAS
_{_ `runshengyu,seaywang` _}_ `@gmail.com,xiaoqijiao@tencent.com`
```
          youzhi.zhang@cair-cas.org.hk,jamesk@cse.ust.hk
```

Direct Alignment of Language Models via Quality-Aware Self-Refinement

Runsheng Yu
Yong Wang
Xiaoqi Jiao
Youzhi Zhang
James T. Kwok


Reinforcement Learning from Human Feedback (RLHF) has been commonly used
to align the behaviors of Large Language Models (LLMs) with human preferences. Recently, a popular alternative is Direct Policy Optimization (DPO), which
replaces an LLM-based reward model with the policy itself, thus obviating the
need for extra memory and training time to learn the reward model. However,
DPO does not consider the relative qualities of the positive and negative responses,
and can lead to sub-optimal training outcomes. To alleviate this problem, we investigate the use of intrinsic knowledge within the on-the-fly fine-tuning LLM to
obtain relative qualities and help to refine the loss function. Specifically, we leverage the knowledge of the LLM to design a refinement function to estimate the
quality of both the positive and negative responses. We show that the constructed
refinement function can help self-refine the loss function under mild assumptions.
The refinement function is integrated into DPO and its variant Identity Policy
Optimization (IPO). Experiments across various evaluators indicate that they can
improve the performance of the fine-tuned models over DPO and IPO.


Large Language Models (LLMs) have demonstrated significant capabilities across various natural
language processing tasks [34; 49; 42; 1]. Ensuring that these LLMs produce the desired responses
and behaviors that are aligned with human preferences is crucial for safe and controllable AI systems
[33]. To achieve this, a popular method is Reinforcement Learning from Human Feedback (RLHF)
[13; 1; 5], which first trains a reward model using human-labeled response pairs, and then uses this
to adjust the policy parameters of the LLM [5; 33]. However, the reward model is often constructed
by another LLM, which requires further training and storage [3].
To reduce the storage and training time of the reward model, a variety of methods have been
proposed recently [17; 47; 3]. In particular, a prominent solution is the Direct Policy Optimization (DPO) [3], which replaces the reward model with the policy itself, thus obviating the need
for an explicit reward model. Recently, numerous variants of DPO have also been developed
[44; 4; 3; 18; 39].
The objective of DPO is to consistently increase the likelihood of human-preferred responses while
reducing the likelihood of the undesired ones. However, this strategy does not consider the relative
qualities of the positive and negative responses, and can lead to suboptimal training outcomes, partic
_∗_ Work done during the internship at Tencent LightSpeed Studios.
Preprint. Under review.
ularly when the preferred responses are not substantially superior, or when the undesired responses
are not adequately inferior [3; 43; 16].
To alleviate this issue, Amini et al. [3] and Zhou et al. [53] propose the use of a score function to
self-refine the objective. However, this approach requires the availability of an ideal reward or score
function, which may not be always feasible. Similarly, Cui et al. [16] and Tunstall et al. [43] employ
GPT-4 [1] to select high-quality response pairs by scoring them. This method requires a strong LLM
to effectively filter the dataset, which again may not always be practical. These considerations raise
the question: _Can we achieve this by using the inherent knowledge within the policy itself?_
Recently, self-alignment has attracted increasing attention due to its ability to leverage the inherent knowledge of LLMs to enhance alignment capabilities, obviating the necessity for additional
human-annotated data [30; 2; 27; 46]. Inspired by this, we propose to utilize on-the-fly fine-tuning
of an LLM’s knowledge to help evaluate the quality of positive and negative responses. The underlying premise is that even relatively weak LLMs possess some ability to assess the quality of
responses [23; 22]. Consequently, our objective is to exploit this capability to more effectively
evaluate response quality, thereby enhancing the efficiency and accuracy of the fine-tuned model.
In this paper, we investigate the use of intrinsic knowledge within the LLM to self-refine the loss
function. In summary, the contributions of this work are as follows:
- We leverage the knowledge of the LLM to design a refinement function, which estimates the
quality of positive and negative responses.
- We demonstrate that the constructed refinement function can help self-refine the loss function
under mild assumptions.
- By utilizing the refinement function, we propose two novel approaches based on DPO and its
variant Identity Policy Optimization (IPO) [4].
Experimental results across various evaluators indicate that the proposed self-refined methods improve the performance of the fine-tuned models compared to their counterparts.


Given a pre-trained large language model (LLM) _π_ ref as initialization, Reinforcement Learning from
Human Feedback (RLHF) [13; 1; 5] aims to learn an LLM _π_ that aligns with human values and
preferences. Specifically, let _x_ be the query, _y_ be the output of _π_, and _r_ be a reward function that
evaluates the performance of _y_ given _x_, RLHF tries to maximize _r_ while ensuring that the trained
LLM _π_ does not deviate significantly from the pre-trained model _π_ ref. This can be formulated as the
following optimization problem [33; 31; 1]:
max E _x∼D,y∼π_ ( _·|x_ ) _r_ ( _y|x_ ) _−_ _β_ DKL[ _π_ ( _y|x_ ) _∥π_ ref( _y|x_ )] _,_ (1)
_π_
where DKL[ _π_ ( _y|x_ ) _∥π_ ref( _y|x_ )] is the Kullback-Leibler divergence between _π_ and _π_ ref, and _β_ is a
constant.
Since _r_ is unknown, the user needs to provide a set of preferences _D ≡{_ ( _xi, yi_ [+] _[, y]_ _i_ _[−]_ [)] _[}]_ _i_ _[N]_ =1 [, where] _[ y]_ _i_ [+]
(resp. _yi_ _[−]_ [) is the positive (resp. negative) response for query] _[ x][i]_ [. A suitable model, typically another]
LLM (parameterized by _ω_ ) [26; 1], then learns the reward function by maximizing the probability
_p_ ( _yi_ [+] _[≻]_ _[y]_ _i_ _[−][|][x]_ [)][ that] _[ y]_ _i_ [+] [is preferred over] _[ y]_ _i_ _[−]_ [(denoted] _[ y]_ _i_ [+] _[≻]_ _[y]_ _i_ _[−]_ [) [33; 3; 4]. Typically, this probability]
is defined by the Bradley-Terry preference model [8; 13] as:
_p_ ( _yi_ [+] _[≻]_ _[y]_ _i_ _[−][|][x][i]_ [)] _[ ≡]_ _[σ]_ [(] _[r]_ [(] _[y]_ _i_ [+] _[|][x][i]_ [)] _[ −]_ _[r]_ [(] _[y]_ _i_ _[−][|][x][i]_ [))] _[,]_ (2)
where _σ_ ( _·_ ) is the logistic function. The optimal _r_ can then be obtained by maximizing the log
likelihood
max _r_ E( _x,y_ + _,y−_ ) _∼D_ log[ _σ_ ( _r_ ( _yi_ [+] _[|][x][i]_ [)] _[ −]_ _[r]_ [(] _[y]_ _i_ _[−][|][x][i]_ [))]] _[.]_ (3)
With the obtained _r_, we can then find _π_ by optimizing (1).

2.1 Classical RLHF with Bradley-Terry Reward Model


In RLHF, the reward model is represented by a LLM [33]. This can be time- and memoryexpensive. It is observed that the optimal _π_ in (1) indeed has the closed form [35]: _π_ ( _y|x_ ) _∝_
_π_ ref( _y|x_ ) exp - _r_ ( _yβ|x_ ) - _,_ and so
_r_ ( _y|x_ ) = _β_ log _[π]_ [(] _[y][|][x]_ [)] (4)
_π_ ref( _y|x_ ) [+] _[ c]_ [(] _[x]_ [)] _[,]_
where _c_ ( _x_ ) _≡_ [�] _y∈Y_ _[π]_ [ref][(] _[y][ |][ x]_ [) exp] - _β_ 1 _[r]_ [(] _[y][|][x]_ [)] �, and _Y_ is the set of responses. By plugging this
into (3), the optimal policy can be found directly by maximizing:
_[π]_ [(] _[y]_ [+] _[|][x]_ [)] _[π]_ [(] _[y][−][|][x]_ [)]
_π_ ref( _y_ [+] _|x_ ) _[−]_ _[β]_ [ log] _π_ ref( _y_ _[−]_ _|x_
       max _π_ E( _x,y_ + _,y−_ ) _∼D_ log _σ_ _β_ log _[π]_ [(] _[y]_ [+] _[|][x]_ [)]
_π_ ref( _y_ _[−]_ _|x_ )
_._ (5)

2.2 Direct Preference Optimization (DPO)


DPO aims to maximize (1). However, our goal is to maximize the preference rather than the reward
[45]. Consequently, a better option is to maximize the preference probabilities _p_ ( _y ≻_ _y_ _[′]_ _|x_ ). To this
end, IPO [4] optimizes the following objective:
max _π_ E _x∼D,y_ + _∼π_ ( _·|x_ ) _,y−∼π_ ref( _·|x_ ) _p_ ( _y_ [+] _≻_ _y_ _[−]_ _|x_ ) _−_ _β_ DKL[ _π_ ( _y_ [+] _|x_ ) _∥π_ ref( _y_ _[−]_ _|x_ )] _._ (6)
Similar to DPO, following [4], the optimal _π_ also has a closed-form solution:
_π_ ( _y_ [+] _|x_ ) _∝_ _π_ ref( _y_ _[−]_ _|x_ ) exp �E _y_ + _∼π_ ref( _·|x_ ) _p_ ( _y_ [+] _≻_ _y_ _[−]_ _|x_ ) _/β_        - _._ (7)
The loss for IPO is then defined as:
_[π]_ [(] _[y]_ _[|][x]_ [)]
_π_ ref( _y_ _[−]_ _|x_ ) _[−]_ 2 [1] _β_
2 _β_
E( _x,y_ + _,y−_ ) _∼D_


��
log _[π]_ [(] _[y]_ [+] _[|][x]_ [)]
_[π]_ [(] _[y]_ [+] _[|][x]_ [)] _[π]_ [(] _[y][−][|][x]_ [)]
_π_ ref( _y_ [+] _|x_ ) _[−]_ [log] _π_ ref( _y_ _[−]_ _|x_
�2 [�]
_._ (8)
The proposed method is also related to self-alignment [27; 46; 29], which uses the LLM’s own
knowledge to improve the alignment. For example, self-judgement [27; 46; 40; 29; 50] uses the
LLM as a judge to evaluate the generated answer. Self-improving [30; 2; 11; 10; 9] improves LLM
by playing game with another LLM opponent. Our work uses the self-alignment ability of LLM to
construct a refinement function to adjust the quality of responses.


Let _r_ _[∗]_ and _r_ be the true and learned reward function respectively. For any tuple ( _x, y_ [+] _, y_ _[−]_ ) _∈_ _D_,
the true reward of the positive response _y_ [+] should exceed that of the negative response _y_ _[−]_, i.e.,
_y_ [+] _≻_ _y_ _[−]_ _⇔_ _r_ _[∗]_ ( _y_ [+] _|x_ ) _> r_ _[∗]_ ( _y_ _[−]_ _|x_ ). Consider two tuples ( _xi, yi_ [+] _[, y]_ _i_ _[−]_ [)][ and][ (] _[x][j][, y]_ _j_ [+] _[, y]_ _j_ _[−]_ [)][. When]
_r_ _[∗]_ ( _yi_ [+] _[|][x][i]_ [)] _[ −]_ _[r][∗]_ [(] _[y]_ _i_ _[−][|][x][i]_ [)] _[ > r][∗]_ [(] _[y]_ _j_ [+] _[|][x][j]_ [)] _[ −]_ _[r][∗]_ [(] _[y]_ _j_ _[−][|][x][j]_ [)][, we prefer] _[ p]_ [(] _[y]_ _i_ [+] _[≻]_ _[y]_ _i_ _[−][|][x][i]_ [)] _[ > p]_ [(] _[y]_ _j_ [+] _[≻]_ _[y]_ _j_ _[−][|][x][j]_ [)][,]
and so _σ_ ( _r_ ( _yi_ [+] _[|][x][i]_ [)] _[ −]_ _[r]_ [(] _[y]_ _i_ _[−][|][x][i]_ [))] _[ > σ]_ [(] _[r]_ [(] _[y]_ _j_ [+] _[|][x][j]_ [)] _[ −]_ _[r]_ [(] _[y]_ _j_ _[−][|][x][j]_ [))][ from (2). In other words, the more]
informative tuple ( _xi, yi_ [+] _[, y]_ _i_ _[−]_ [)][ should be more important. However, RLHF simply maximizes (3),]
which considers all tuples in the dataset _D_ equally.

3.1 Limitation of Maximizing Bradley-Terry Preference


To alleviate this problem, we propose adding a _refinement_ function ∆( _y_ _[−]_ _, y_ [+] ; _x_ ) : _Y ×Y ×X →_ R
(where _X_ and _Y_ are the sets of queries and responses, respectively) to adjust _r_ ( _y_ [+] _|x_ ) _−_ _r_ ( _y_ _[−]_ _|x_ ), so
that problem (3) is modified to:
max E( _x,y_ + _,y−_ ) _∼D_ log[ _σ_ ( _r_ ( _y_ [+] _|x_ ) _−_ _r_ ( _y_ _[−]_ _|x_ ) _−_ _λ_ ∆( _y_ _[−]_ _, y_ [+] ; _x_ ))] _,_ (9)
_π_
where _λ_ is a positive constant. Intuitively, when ∆( _y_ _[−]_ _, y_ [+] ; _x_ ) is large, _σ_ ( _r_ ( _y_ [+] _|x_ ) _−_ _r_ ( _y_ _[−]_ _|x_ ) _−_
_λ_ ∆( _y_ _[−]_ _, y_ [+] ; _x_ )) becomes small, and the optimization in (9) will tend to enlarge the “distance”
_r_ ( _y_ [+] _|x_ ) _−_ _r_ ( _y_ _[−]_ _|x_ ) between the positive and negative responses. Thus, for two tuples ( _x, y_ [+] _, y_ _[−]_ )
and ( _x,_ ˜ _y_ [+] _,_ ˜ _y_ _[−]_ ) corresponding to the same query _x_, we want to design a ∆ such that when
∆( _y_ _[−]_ _, y_ [+] ; _x_ ) _>_ ∆(˜ _y_ _[−]_ _,_ ˜ _y_ [+] ; _x_ ), their true reward values satisfy:
_r_ _[∗]_ ( _y_ [+] _|x_ ) _−_ _r_ _[∗]_ ( _y_ _[−]_ _|x_ ) _> r_ _[∗]_ (˜ _y_ [+] _|x_ ) _−_ _r_ _[∗]_ (˜ _y_ _[−]_ _|x_ ) _._
In other words, a larger difference _r_ _[∗]_ ( _y_ [+] _|x_ ) _−_ _r_ _[∗]_ ( _y_ _[−]_ _|x_ ) in the true reward values between the
positive and negative responses corresponds to a larger ∆( _y_ _[−]_ _, y_ [+] ; _x_ ), and vice versa. However,
obviously the difficulty is that we do not have access to _r_ _[∗]_ .


First, we assume that the LLM is capable of learning a reward function that aligns with the true
reward function. This premise is formalized as follows:
**Assumption 3.1.** _LLM π can construct a reward model r such that for any_ ( _x, y_ [+] _, y_ _[−]_ ) _with y_ [+] _≻_
_y_ _[−]_ _, the corresponding reward values satisfy r_ ( _y_ [+] _|x_ ) _> r_ ( _y_ _[−]_ _|x_ ) _._
As _r_ _[∗]_ is unknown, with a capable LLM, a natural idea is to use _r_ as a proxy of _r_ _[∗]_ . Recall from
Section 3.2.1 that a large _r_ _[∗]_ ( _y_ [+] _|x_ ) _−_ _r_ _[∗]_ ( _y_ _[−]_ _|x_ ) should correspond to a large ∆( _y_ _[−]_ _, y_ [+] ; _x_ ). Using
(4), one can define ∆ as
_[π]_ [(] _[y]_ [+] _[|][x]_ [)] _[π]_ [(] _[y][−][|][x]_ [)]
_π_ ref( _y_ [+] _|x_ ) _[−]_ [log] _π_ ref( _y_ _[−]_ _|x_
          ∆naive = _r_ ( _y_ [+] _|x_ ) _−_ _r_ ( _y_ _[−]_ _|x_ ) = _β_ log _[π]_ [(] _[y]_ [+] _[|][x]_ [)]
_π_ ref( _y_ _[−]_ _|x_ )
_._ (10)
Substituting this into the DPO objective (5), we have
_[π]_ [(] _[y]_ [+] _[|][x]_ [)] _[π]_ [(] _[y][−][|][x]_ [)]
_π_ ref( _y_ [+] _|x_ ) _[−]_ _[β]_ [ log] _π_ ref( _y_ _[−]_ _|x_
      E( _x,y_ + _,y−_ ) _∼D_ log _σ_ _β_ log _[π]_ [(] _[y]_ [+] _[|][x]_ [)]
_[π]_ [(] _[y][−][|][x]_ [)]
_π_ ref( _y_ _[−]_ _|x_ ) _[−]_ _[λ]_ [∆][naive][(] _[y][−][, y]_ [+][;] _[ x]_ [)]
_[π]_ [(] _[y]_ [+] _[|][x]_ [)] _[π]_ [(] _[y][−][|][x]_ [)]
_π_ ref( _y_ [+] _|x_ ) _[−]_ [(] _[β][ −]_ _[λβ]_ [) log] _π_ ref( _y_ _[−]_ _|x_
       = E( _x,y_ + _,y−_ ) _∼D_ log _σ_ ( _β −_ _λβ_ ) log _[π]_ [(] _[y]_ [+] _[|][x]_ [)]
_π_ ref( _y_ _[−]_ _|x_ )
_._
However, this is the same as the original DPO objective except for a scaling of the regularization
parameter _β_, and thus is not useful.
To alleviate this problem, our idea is to first improve the LLM performance by concatenating the
query _x_ with a prompt _p_, as _p ⊕_ _x_, where _⊕_ denotes concatenation. With (4) the reward with this
prompt-augmented query is:
_r_ ( _y|p ⊕_ _x_ ) = _β_ log _[π]_ [(] _[y][|][p][ ⊕]_ _[x]_ [)] (11)
_π_ ref( _y|p ⊕_ _x_ ) [+] _[ c]_ [(] _[x]_ [)] _[.]_
With a good prompt, this _r_ is expected to be a good proxy of _r_ _[∗]_ . Specifically, we use the following
prompt _p_ .
We assume that adding this prompt does not change the preference between the positive and negative
responses ( _y_ [+] and _y_ _[−]_ ).
**Assumption 3.2.** _For a given query x, if y_ [+] _≻_ _y_ _[−]_ _, we still have y_ [+] _≻_ _y_ _[−]_ _with the prompt-_
_augmented query p ⊕_ _x._
Analogous to (10), we consider the following refinement function:
∆ _π_      - _y_ _[−]_ _, y_ [+] ; _x_      - _≡_ _β_ log _[π]_ [(] _[y]_ [+] _[|][p][ ⊕]_ _[x]_ [)] _[π]_ [ref][(] _[y][−][|][p][ ⊕]_ _[x]_ [)] (12)
_π_ ref( _y_ [+] _|p ⊕_ _x_ ) _π_ ( _y_ _[−]_ _|p ⊕_ _x_ ) _[.]_
Note that we have added subscript _π_ to explicitly indicate the dependence of ∆ on _π_ . Obviously,
when putting this ∆ _π_ into the DPO objective (5), it does not suffer from the same problem as the
∆naive discussed earlier.
The following Proposition shows that ∆ _π_ in (12) satisfies two important properties. First,
∆ _π_ ( _y_ _[−]_ _, y_ [+] ; _x_ ) can be measured relative to the optimal response _y_ _[∗]_, which allows to represent
∆ _π_ ( _y_ _[−]_ _, y_ [+] ; _x_ ) in terms of ∆ _π_ ( _y_ _[−]_ _, y_ _[∗]_ ; _x_ ) and ∆ _π_ ( _y_ [+] _, y_ _[∗]_ ; _x_ ). Second, the positive response (which
has a higher true reward value) is “closer” to _y_ _[∗]_ than the negative response, and vice versa. All the
proofs are in Appendix A.
**Proposition 3.3.** _With Assumptions 3.1 and 3.2, we have (i)_ ∆ _π_ ( _y_ _[−]_ _, y_ [+] ; _x_ ) = ∆ _π_ ( _y_ _[−]_ _, y_ _[∗]_ ; _x_ ) _−_
∆ _π_ ( _y_ [+] _, y_ _[∗]_ ; _x_ ) _, where y_ _[∗]_ _is the optimal y for the given x; (ii) For any tuple_ ( _x, y_ [+] _, y_ _[−]_ ) _, r_ _[∗]_ ( _y_ [+] _|x_ ) _>_
_r_ _[∗]_ ( _y_ _[−]_ _|x_ ) _⇔_ ∆ _π_ ( _y_ [+] _, y_ _[∗]_ ; _x_ ) _<_ ∆ _π_ ( _y_ _[−]_ _, y_ _[∗]_ ; _x_ ) _._
The following Corollary shows that this ∆ _π_ ( _y_ _[−]_ _, y_ [+] ; _x_ ) satisfies the desired property in Section 3.2.1, namely that a larger difference _r_ _[∗]_ ( _y_ [+] _|x_ ) _−_ _r_ _[∗]_ ( _y_ _[−]_ _|x_ ) in the true reward values between
the positive and negative responses corresponds to a larger ∆ _π_ ( _y_ _[−]_ _, y_ [+] ; _x_ ).
**Corollary 3.3.1.** _For any_ ( _x, yi_ [+] _[, y]_ _i_ _[−]_ [)] _[ and]_ [ (] _[x, y]_ _j_ [+] _[, y]_ _j_ _[−]_ [)] _[, if][ r][∗]_ [(] _[y]_ _i_ [+] _[|][x]_ [)] _[ > r][∗]_ [(] _[y]_ _j_ [+] _[|][x]_ [)] _[, and][ r][∗]_ [(] _[y]_ _i_ _[−][|][x]_ [)] _[ <]_
_r_ _[∗]_ ( _yj_ _[−][|][x]_ [)] _[ we have][ r][∗]_ [(] _[y]_ _i_ [+] _[|][x]_ [)] _[ −]_ _[r][∗]_ [(] _[y]_ _i_ _[−][|][x]_ [)] _>_ _r_ _[∗]_ ( _yj_ [+] _[|][x]_ [)] _[ −]_ _[r][∗]_ [(] _[y]_ _j_ _[−][|][x]_ [)] _⇔_ ∆ _π_ ( _yi_ _[−][, y]_ _i_ [+][;] _[ x]_ [)] _>_
∆ _π_ ( _yj_ _[−][, y]_ _j_ [+][;] _[ x]_ [)] _[.]_


First, we show how to integrate the proposed refinement into DPO. Substituting (4) and (12) into
(9), we obtain:
max E( _x,y_ + _,y−_ ) _∼D_ log _σ_ ( _r_ ( _y_ [+] _|x_ ) _−_ _r_ ( _y_ _[−]_ _|x_ ) _−_ _λ_ ∆ _π_ ( _y_ _[−]_ _, y_ [+] ; _x_ ))
_π_
_[π]_ [(] _[y]_ [+] _[|][x]_ [)] _[π]_ [(] _[y][−][|][x]_ [)]
_π_ ref( _y_ [+] _|x_ ) _[−][β]_ [ log] _π_ ref( _y_ _[−]_ _|x_
        =max E( _x,y_ + _,y−_ ) _∼D_ log _σ_ _β_ log _[π]_ [(] _[y]_ [+] _[|][x]_ [)]
_π_ _π_ ( _y_ [+] _|x_
_[π]_ [(] _[y][−][|][x]_ [)]  - _._ (13)
_π_ ref( _y_ _[−]_ _|x_ ) _[−]_ _[λ]_ [∆] _[π]_ [(] _[y][−][, y]_ [+][;] _[ x]_ [)]
Recall that ∆ _π_ above depends on _π_ . During learning, we use the stop-gradient operator [2] _⊥_ [ _·_ ] on
∆ _π_ to prevent it from being changed. The whole procedure, which will be called Self-refined DPO
(Sr-DPO), is shown in Algorithm 1.
**Algorithm 1:** Self-refned Direct Policy Optimization (Sr-DPO).
**Input:** Dataset _D_ = _{_ ( _x, y_ [+] _, y_ _[−]_ ) _}_, a pre-trained LLM _π_ ref with parameter _**θ**_ 0, learning rate _α_ .


**2** sample a minibatch _B_ of size _N_ from _D_ ;
**3** _**θ**_ _t_ = _**θ**_ _t−_ 1 + _α∇_ _**θ**_ _N_ [1] - _Ni_ =1 [log] _[ σ]_ - _β_ �log _ππ_ ref(( _yyi_ [+] _i_ [+] _[|][x][|][x][i][i]_ [)][)] _[−]_ [log] _ππ_ ref(( _yyi_ _[−]_ _i_ _[−][|][x][|][x][i][i]_ [)][)]
**4 return** _**θ**_ _T_ .


For IPO, we first construct a variant of the IPO objective in (6):
- _−λ_ _⊥_ [∆ _π_ ( _yi_ _[−][, y]_ _i_ [+][;] _[ x][i]_ [)]] ;
max _π_ E _x∼D,y_ + _∼π_ ( _·|x_ ) _,y−∼π_ ref( _·|x_ ) _p_ ( _y_ [+] _≻_ _y_ _[−]_ _|x_ )+ _λr_ ( _y_ [+] _|p ⊕_ _x_ ) _−β_ DKL[ _π_ ( _y_ [+] _|x_ ) _∥π_ ref( _y_ _[−]_ _|x_ )] _,_ (14)
which adds an extra _λ_ E _x∼D,y_ + _∼π_ ( _·|x_ ) _r_ ( _y_ [+] _|p ⊕_ _x_ ) term to maximize the expectation of reward
_r_ ( _y_ [+] _|p ⊕_ _x_ ).
**Proposition 3.4.** _The optimal policy π in (14) satisfies_
_[π]_ [(] _[y]_ [+] _[|][x]_ [)] _[π]_ [(] _[y][−][|][x]_ [)]
_πref_ ( _y_ [+] _|x_ ) _[−]_ [log] _πref_ ( _y_ _[−]_ _|x_
log _[π]_ [(] _[y]_ [+] _[|][x]_ [)]
_[π]_ [(] _[y]_ _[|][x]_ [)] [1]
_πref_ ( _y_ _[−]_ _|x_ ) [=] 2
(15)
2 _β_ [+] _[ λ]_ [∆] _[π]_ [(] _[y][−][, y]_ [+][;] _[ x]_ [)] _[.]_
2In other words, _∇x ⊥_ [ _g_ ( _x_ )] _≡_ 0 and _⊥_ [ _g_ ( _x_ )] _≡_ _g_ ( _x_ ) for any differentiable _g_ .
Note that when _β_ = 0, log _ππ_ ref(( _yy_ [+][+] _|x|x_ )) _[−]_ [log] _ππ_ ref(( _yy_ _[−][−]_ _|x|x_ )) _[−]_ _[λ]_ [∆] _[π]_ [(] _[y][−][, y]_ [+][;] _[ x]_ [)][ should be infinite. This]
implies maximizing log _ππ_ ref(( _yy_ [+][+] _|x|x_ )) _[−]_ [log] _ππ_ ref(( _yy_ _[−][−]_ _|x|x_ )) _[−]_ _[λ]_ [∆] _[π]_ [(] _[y][−][, y]_ [+][;] _[ x]_ [)][ w.r.t,] _[ π]_ [, which has a similar]
form to DPO. The procedure, which will be called Self-Refined IPO (Sr-IPO), is shown in Algorithm
2. It uses stochastic gradient descent to minimize the difference between the LHS and RHS in (15).
**Algorithm 2:** Self-refned Identity Policy Optimization (Sr-IPO).
**Input:** Dataset _D_ = _{_ ( _x, y_ [+] _, y_ _[−]_ ) _}_, a pre-trained LLM _π_ ref with parameter _**θ**_ 0, learning rate _α_ .


**2** sample a minibatch _B_ of size _N_ from _D_ ;
**3** _**θ**_ _t_ = _**θ**_ _t−_ 1 _−_ _α∇_ _**θ**_ _N_ [1] - _Ni_ =1��log _ππ_ ref(( _yyi_ [+] _i_ [+] _[|][x][|][x][i][i]_ [)][)] _[−]_ [log] _ππ_ ref(( _yyi_ _[−]_ _i_ _[−][|][x][|][x][i][i]_ [)][)]
- _−λ ⊥_ [∆ _π_ ( _yi_ _[−][, y]_ _i_ [+][;] _[ x][i]_ [)]] _[ −]_ 21 _β_ �2;
**4 return** _**θ**_ _T_ .


**Datasets** . We evaluate the effectiveness of the proposed methods on three widely-used benchmark
datasets: (i) MT-Bench [52], which is a multi-turn question set on writing, roleplay, extraction,
reasoning, math, coding, knowledge I (STEM), and knowledge II (humanities/social science). (ii)
Vicuna-Bench [12], which is a single-turn question set on writing, roleplay, generic, fermi, counterfactual, coding, math, and knowledge. (iii) Open-LLM leader-board [6], which includes (a) commonsense reasoning: Arc [14], HellaSwag [48], WinoGrande [36]; (b) multi-task language understanding: MMLU [20]; (c) human falsehood mimic: TruthfulQA [28]; and (d) math problem solving: GSM8k [15]. Table 1 shows more information on the Hugging-Face Open LLM Leaderboard
datasets.
Table 1: Information on the Hugging-Face Open LLM Leaderboard datasets.
Arc TruthfulQA Winogrande GSM8k HellaSwag MMLU


performance metric acc ~~n~~ orm mc2 acc acc acc ~~n~~ orm acc
**Setup for MT-Bench and Vicuna-Bench.** As in [35], we use Pythia 2.8B [7], a pretrained LLM
without supervised fine-tuning and RLHF, as the backbone model. Following [3], we first conduct
supervised fine tuning (SFT) using the HH-RLHF dataset [5], which is human preference data on
helpfulness and harmlessness based on positive feedbacks. We then perform direct alignment using
the HH-RLHF dataset on the SFT model. Finally, we follow [35] to use GPT-4 [1] as a judge to
evaluate the testing performance of the direct alignment trained model. For performance evaluation,
we use the win-rate, tie-rate, and lose rate as in [44].
**Setup for Open-LLM Leader-board.** Following [43; 10], we use zephyr-7b-sft-full [3], a supervised
fine-tuned version of Mistral 7B [24], as the basic model. We directly perform direct alignment
using a large-scale diverse preference dataset Ultra-feedback [16]. The fine-tuned model is then
evaluated on the testing benchmarks via the platform [19].
Following [44], we use three performance metrics:
i) Average marginal: _N_ 1  - _Ni_ =1 [log] _ππ_ ref( _y_ ( _i_ [+] _yi_ [+] _[|][x][|][x][i]_ [)] _[i]_ [)] _[ −]_ [log] _ππ_ ref( _y_ ( _i_ _[−]_ _yi_ _[−][|][x][|][x][i]_ [)] _[i]_ [)] [, which measures the gap between]
positive and negative responses.
(ii) Accuracy: _N_ 1 - _Ni_ =1 [1] �log _ππ_ ref( _y_ ( _i_ [+] _yi_ [+] _[|][x][|][x][i]_ [)] _[i]_ [)] _[>]_ [ log] _ππ_ ref( _y_ ( _i_ _[−]_ _yi_ _[−][|][x][|][x][i]_ [)] _[i]_ [)]
, which measures the number of tuples
with the reward of a positive response larger than that of the negative response. Here, 1 ( _·_ ) returns 1
when the argument holds and 0 otherwise.
3 `[https://huggingface.co/alignment-handbook/zephyr-7b-sft-full](https://huggingface.co/alignment-handbook/zephyr-7b-sft-full)`
(iii) Accuracy defined on the prompt-augmented tuples: _N_ 1 - _Ni_ =1 [1] �log _ππ_ ref( _y_ ( _i_ [+] _yi_ [+] _[|][p][|][⊕][p][⊕][x][x][i]_ [)] _[i]_ [)] _[>]_
_π_ ( _yi_ _[−][|][p][⊕][x][i]_ [)]
log
_π_ ref ( _yi_ _[−][|][p][⊕][x][i]_ [)]
, which measures the accuracy for tuples with input augmented by the prompt.
**Baselines and Implmentation Details.** We choose two widely-adopted direct alignment baselines:
(i) DPO [35], and (ii) IPO [4]. Following [35], _β_ is set to 0.1 for DPO and IPO. We also set
_β_ = 0 _._ 1 for the proposed Sr-DPO and Sr-IPO for fair comparison. We select the optimal _λ_ from
_{_ 0 _._ 1 _,_ 0 _._ 3 _,_ 0 _._ 5 _,_ 1 _}_ on the first 50 tuples from the HH-RLHF testing dataset. For both the HH-RLHF
and UltraChat datasets, following [35], the learning rate is 5 _×_ 10 _[−]_ [7], the optimizer is RMSprop
[21], the batch size is 64, and we also use gradient normalization to help training. The maximal
input token size in training is 512. All experiments are performed on 8 A100 GPUs, and we use
fully sharded data parallel [51] for distributed training.


**MT-Bench and Vicuna-Bench.** Table 2 shows the testing win/tie/lose rates of the various methods
as evaluated by GPT-4. As can be seen, the proposed Sr-DPO and Sr-IPO are effective and outperform DPO and IPO (in other words, the win rate is larger than the lose rate) in both MT-bench
and Vicuna-bench. This reveals the effectiveness of our proposal methods. Some examples are also
shown in Appendix B.
Table 2: Testing performance on MT-Bench and Vicuna-Bench.
MT-Bench Vicuna-Bench
Win Rate Tie Rate Lose Rate Win Rate Tie Rate Lose Rate
Sr-DPO vs DPO 45.62% 33.76% 20.62% 63.75% 13.75% 22.50%
Sr-IPO vs IPO 38.75% 25.00% 26.25% 60.00% 8.75% 31.25%
Sr-DPO vs IPO 42.50% 21.25% 36.25% 66.25% 11.50% 26.25%
Sr-IPO vs DPO 38.61% 28.89% 32.50% 61.25% 5.00% 33.75%
**HH-RLHF.** Figure 1 shows results on the training set. As seen, the proposed Sr-DPO and Sr-IPO
have lower marginals (Figures 1a and 1d), while maintaining comparable accuracies (Figures 1b and
1e). This shows they avoid always enlarging the reward difference between positive and negative
pairs, yet still achieving high accuracies. From Figures 1c and 1f, we find that accuracies for the
prompt-augmented tuples are similar to those for the original tuples, thus verifying Assumption 3.2.
Table 3: Testing performance of the various methods on Open-LLM leader-board. The best one is
in bold.
Arc TruthfulQA WinoGrande GSM8k HellaSwag MMLU Average
Zephyr-7b-sft-full 60.41 43.73 74.19 26.76 82.85 60.92 58.14
DPO 63.05 59.30 78.53 31.01 84.89 61.44 63.04
IPO 61.86 46.03 79.24 35.91 82.80 61.55 61.23
Sr-DPO **65.27** **60.30** **80.82** 33.36 **85.52** 61.63 **64.48**
Sr-IPO 62.46 47.87 79.32 **39.80** 83.15 **61.80** 62.40
**Open-LLM leader-board.** Table 3 shows the testing performance across various methods. The results indicate that Sr-DPO achieves superior performance on Arc, TruthfulQA, WinoGrande; while
Sr-IPO excels on GSM8k and MMLU. Overall, Sr-DPO emerges as the most effective method. Furthermore, Sr-DPO (resp. Sr-IPO) consistently outperforms DPO (resp. IPO) across all six datasets.
These results validate the effectiveness of the proposed approach.


**Effect of** _λ_ **.** Figure 2a shows the testing win rate with varying _λ_ in Sr-DPO (resp. Sr-IPO) on MTBench and Vicuna-Bench. In both cases the win rate first increases with _λ_ and then decreases. In
particular, _λ_ = 0 (i.e., not using the proposed refinement) leads to the worst performance. However,
a _λ_ too large exaggerates the influence of ∆, which can also negatively impact performance. We do
not study its effect on Open-LLM leader-board because it is very time-consuming.
(a) Marginal: DPO vs Sr-DPO. (b) Accuracy: IPO vs Sr-IPO.
(d) Marginal: IPO vs Sr-IPO. (e) Accuracy: IPO vs Sr-IPO.
(c) Sr-DPO’s accuracies for augmented and original tuples.
(f) Sr-IPO’s accuracies for augmented and original tuples.
Figure 1: Average marginal and accuracy on HH-RLHF dataset with different numbers of training
tuples.
**Effect of the number of training tuples.** Figure 2b shows the testing win rate with varying number
of training tuples on Vicuna-Bench. As can be seen, both Sr-DPO and Sr-IPO can benefit from the
use of more training tuples.
Figure 3 shows the testing performance with varying number
of training tuples on Open-LLM
leader-board. As can be seen,
DPO and IPO exhibit performance drop with the use of
more training tuples on 4 of
the 6 tasks (except on MMLU
and HellaSwag), suggesting po
(a) Variation with _λ_ . (b) Variation with #tuples. tential overfitting. In contrast,
Sr-DPO and Sr-IPO are less
Figure 2: Win rates of Sr-DPO (vs DPO) and Sr-IPO (vs IPO)
w.r.t. _λ_ and number of training tuples on Vicuna-Bench. prone to overfitting and exhibit
improved performance with increased training data on ARC, TruthfulQA, GSM8k, and MMLU.
(a) Variation with _λ_ . (b) Variation with #tuples.
Figure 2: Win rates of Sr-DPO (vs DPO) and Sr-IPO (vs IPO)
w.r.t. _λ_ and number of training tuples on Vicuna-Bench.

4.3 Effects of _λ_ and Number of Training Tuples


Recall that GPT-4 is used as the evaluator. For each instance _xi_, we define the score difference between the corresponding positive _yi_ [+] [and negative] _[ y]_ _i_ _[−]_ [responses as] _[ s]_ [+][(] _[x][i]_ [)] _[ −]_ _[s][−]_ [(] _[x][i]_ [)][, where] _[ s]_ [+][(] _[x][i]_ [)]
(resp. _s_ _[−]_ ( _xi_ )) is GPT-4’s evaluation score (0-5) of _yi_ [+] [(resp.] _[ y]_ _i_ _[−]_ [). Inspired by [25], which uses]
correlation to measure the agreements between GPT-4 and testing LLMs, here we compute the cor
_π_ ( _yi_ [+] _[|][x][i]_ [)] _π_ ( _yi_ _[−][|][x][i]_ [)]
relation between the marginal log
_π_ ref( _yi_ [+] _[|][x][i]_ [)] _[ −]_ [log] _π_ ref( _yi_ _[−][|][x][i]_ [)][ and score difference] _[ s]_ [+][(] _[x][i]_ [)] _[−][s][−]_ [(] _[x][i]_ [)]
over 200 random tuples from the Ultrafeedback-binarized-preferences dataset [4] (which already contains _s_ [+] ( _x_ )’s and _s_ _[−]_ ( _x_ )’s). In particular, we use (i) Pearson’s correlation [37], which measures
linear relationships; (ii) Spearman’s rank correlation [38], which measures monotonic relationships;
and (iii) Kendall’s Tau [32], which evaluates the strength and direction of associations.
4 `[https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences](https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences)`
(a) ARC. (b) TruthfulQA. (c) WinoGrande.
(d) GSM8k. (e) HellaSwag. (f) MMLU.
Figure 3: Performance on Open-LLM leader-board with different numbers of training tuples.
Table 4 shows the correlation. As can be seen, Sr-DPO (resp. Sr-IPO) exhibits higher correlation values compared to DPO (resp. IPO). This suggests that Sr-DPO and Sr-IPO can evaluate the
qualities of the positive and negative responses more accurately.

4.4 Correlation between Marginal and Score


Table 5 shows the direct alignment training time on the HH-RLHF dataset. As can be seen, SrDPO and Sr-IPO are about 24% slower than DPO and IPO. We consider this acceptable given the
performance improvements achieved by Sr-DPO and Sr-IPO.
Table 4: Correlation between the marginal
and GPT-4’s rating.
Pearson’s Spearman’s Kendall’s
correlation rank correlation Tau
DPO 0.21 0.14 0.10
IPO 0.11 0.05 0.03
Sr-DPO 0.26 0.20 0.14
Sr-IPO 0.13 0.08 0.05


Table 5: Training time (in GPU hours) on
HH-RLHF.
DPO IPO Sr-DPO Sr-IPO
16.8 16.8 20.8 20.8
In this paper, we find that the widely adopted DPO method falls short by not accounting for the
relative qualities of positive and negative samples, which can lead to sub-optimal training outcomes.
To address this issue, we propose leveraging the intrinsic knowledge within LLMs to refine the loss
function. Our main contributions are three-fold: 1) We utilize the knowledge of LLMs to create a
refinement function that effectively estimates the quality of both positive and negative responses. 2)
We demonstrate that under mild assumptions, the refinement function can enable the loss function
to self-refine, leading to better alignment with human preferences. 3) Based on the refinement function, we develop two practical algorithms that enhance the training process. Experimental results
from various evaluators indicate that the proposed self-refined methods significantly improve the
performance of fine-tuned models compared to existing approaches.
One limitation of our work is that we do not combine it with online policy-based direct alignment
[41]. We will investigate this in our future work.


Our work falls under one of the RLHF frameworks, and thus shares similar social impacts, both
positive and negative. On the positive side, our method can help LLMs avoid generating harmful or
unhelpful results by aligning with human feedback. However, conversely, our method also has the
potential to enable LLMs to generate harmful or unhelpful results if it is used to align with malicious
preferences.


_Proof._ (i) First property: By directly substituting ∆ _π_ ( _y_ _[−]_ _, y_ _[∗]_ ; _x_ ) _−_ ∆ _π_ ( _y_ [+] _, y_ _[∗]_ ; _x_ ) into the definition
of ∆ in (12), we have
_[π]_ [(] _[y]_ [+] _[|][p][ ⊕]_ _[x]_ [)] _[π]_ [(] _[y][−][|][p][ ⊕]_ _[x]_ [)]
_π_ 0( _y_ [+] _|p ⊕_ _x_ ) _[−]_ [log] _π_ 0( _y_ _[−]_ _|p ⊕_ _x_ )
           ∆ _π_ ( _y_ _[−]_ _, y_ _[∗]_ ; _x_ ) _−_ ∆ _π_ ( _y_ [+] _, y_ _[∗]_ ; _x_ ) = _β_ log _[π]_ [(] _[y]_ [+] _[|][p][ ⊕]_ _[x]_ [)]
_π_ 0( _y_ _[−]_ _|p ⊕_ _x_ )
= _β_ log _[π]_ [(] _[y]_ [+] _[|][p][ ⊕]_ _[x]_ [)] _[π]_ [0][(] _[y][−][|][p][ ⊕]_ _[x]_ [)]
_π_ ( _y_ _[−]_ _|p ⊕_ _x_ ) _π_ 0( _y_ [+] _|p ⊕_ _x_ )
= ∆ _π_            - _y_ _[−]_ _, y_ [+] ; _x_            - _._
(ii) Second property: From assumption 3.1, we have: _r_ ( _y_ [+] _|x_ ) _> r_ ( _y_ _[−]_ _|x_ ) _, ∀_ ( _x, y_ [+] _, y−_ ) with
_y_ [+] _≻_ _y_ _[−]_ . From (4),
_r_ ( _y_ [+] _|x_ ) _> r_ - _y_ _[−]_ _|x_ - _⇔_ log _[π]_ [ (] _[y]_ [+] _[ |][ x]_ [)]
_π_ 0 ( _y_ _[−]_ _| x_ ) _[.]_
_[π]_ [ (] _[y]_ [+] _[ |][ x]_ [)] _[π]_ [ (] _[y][−]_ _[|][ x]_ [)]
_π_ 0 ( _y_ [+] _| x_ ) _[>]_ [ log] _π_ 0 ( _y_ _[−]_ _| x_
From Assumption 3.2, for the prompt-augmented query _p ⊕_ _x_, we still have _y_ [+] _≻_ _y_ _[−]_ . By using
Assumption 3.1 on ( _p ⊕_ _x, y_ [+] _, y_ _[−]_ ), we have:
_[π]_ [(] _[y]_ [+] _[|][p][ ⊕]_ _[x]_ [)] _[π]_ [(] _[y][−][|][p][ ⊕]_ _[x]_ [)]
_π_ 0( _y_ [+] _|p ⊕_ _x_ ) _[>]_ [ log] _π_ 0( _y_ _[−]_ _|p ⊕_ _x_
log _[π]_ [(] _[y]_ [+] _[|][p][ ⊕]_ _[x]_ [)]
(16)
_π_ 0( _y_ _[−]_ _|p ⊕_ _x_ ) _[.]_
Thus, from the first property,
∆ _π_ ( _y_ [+] _, y_ _[∗]_ ; _x_ ) _−_ ∆ _π_ ( _y_ _[−]_ _, y_ _[∗]_ ; _x_ ) = _−_ ∆ _π_ - _y_ _[−]_ _, y_ [+] ; _x_ - = _−β_ log _[π]_ [(] _[y]_ [+] _[|][p][ ⊕]_ _[x]_ [)] _[π]_ [0][(] _[y][−][|][p][ ⊕]_ _[x]_ [)]
_π_ 0( _y_ [+] _|p ⊕_ _x_ ) _π_ ( _y_ _[−]_ _|p ⊕_ _x_ ) _[<]_ [ 0]
because of (16), and so ∆ _π_ ( _y_ [+] _, y_ _[∗]_ ; _x_ ) _<_ ∆ _π_ ( _y_ _[−]_ _, y_ _[∗]_ ; _x_ ). Thus,
_y_ [+] _≻_ _y_ _[−]_ _⇔_ ∆ _π_           - _y_ [+] _, y_ _[∗]_ ; _x_           - _<_ ∆ _π_           - _y_ _[−]_ _, y_ _[∗]_ ; _x_           - _._ (17)
Since _y_ [+] _≻_ _y_ _[−]_ _⇔_ _r_ _[∗]_ ( _y_ [+] _|x_ ) _> r_ _[∗]_ ( _y_ _[−]_ _|x_ ). We obtain that _r_ _[∗]_ ( _y_ [+] _|x_ ) _> r_ _[∗]_ ( _y_ _[−]_ _|x_ ) _⇔_
∆ _π_ ( _y_ [+] _, y_ _[∗]_ ; _x_ ) _<_ ∆ _π_ ( _y_ _[−]_ _, y_ _[∗]_ ; _x_ ).


_Proof._ Using property (ii) in Proposition 3.3 on the assumption _r_ _[∗]_ ( _yi_ [+] _[|][x]_ [)] _[ > r][∗]_ [(] _[y]_ _j_ [+] _[|][x]_ [)][ in Corol-]
lary 3.3.1, we have
_r_ _[∗]_ ( _yi_ [+] _[|][x]_ [)] _[ > r][∗]_ [(] _[y]_ _j_ [+] _[|][x]_ [)] _[ ⇔−]_ [∆] _[π]_ [(] _[y]_ _i_ [+] _[, y][∗]_ [;] _[ x]_ [)] _[ >][ −]_ [∆] _[π]_ [(] _[y]_ _j_ [+] _[, y][∗]_ [;] _[ x]_ [)] _[.]_ (18)
Similarly, using this on the assumption in Corollary 3.3.1 that _r_ _[∗]_ ( _yi_ _[−][|][x]_ [)] _[ < r][∗]_ [(] _[y]_ _j_ _[−][|][x]_ [)][, we have]
_−r_ _[∗]_ ( _yi_ _[−][|][x]_ [)] _[ >][ −][r][∗]_ [(] _[y]_ _j_ _[−][|][x]_ [)] _[ ⇔]_ [∆] _[π]_ [(] _[y]_ _i_ _[−][, y][∗]_ [;] _[ x]_ [)] _[ >]_ [ ∆] _[π]_ [(] _[y]_ _j_ _[−][, y][∗]_ [;] _[ x]_ [)] _[.]_ (19)
Adding these two together, we obtain
_r_ _[∗]_ ( _yi_ [+] _[|][x]_ [)] _[ −]_ _[r][∗]_ [(] _[y]_ _i_ _[−][|][x]_ [)] _[ < r][∗]_ [(] _[y]_ _j_ [+] _[|][x]_ [)] _[ −]_ _[r][∗]_ [(] _[y]_ _j_ _[−][|][x]_ [)]
_⇔_ ∆ _π_ ( _yi_ _[−][, y][∗]_ [;] _[ x]_ [)] _[ −]_ [∆] _[π]_ [(] _[y]_ _i_ [+] _[, y][∗]_ [;] _[ x]_ [)] _[ >]_ [ ∆] _[π]_ [(] _[y]_ _j_ _[−][, y][∗]_ [;] _[ x]_ [)] _[ −]_ [∆] _[π]_ [(] _[y]_ _j_ [+] _[, y][∗]_ [;] _[ x]_ [)]
_⇔_ ∆ _π_ ( _yi_ _[−][, y]_ _i_ [+][;] _[ x]_ [)] _[ >]_ [ ∆] _[π]_ [(] _[y]_ _j_ _[−][, y]_ _j_ [+][;] _[ x]_ [)] _[,]_
due to property (i) of Proposition 3.3.


As in [3; 4], the closed-form solution of (14) is:
_π_ ( _y_ [+] _|x_ ) _∝_ _π_ ref( _y_ [+] _|x_ ) exp    - _β_ _[−]_ [1] E _y_ + _∼π_ ref( _·|x_ ) _p_ ( _y_ [+] _≻_ _π_ ref _|x_ ) + _λr_ ( _y_ [+] _|p ⊕_ _x_ )� _._
Similarly for _y_ _[−]_, we have
_π_ ( _y_ _[−]_ _|x_ ) _∝_ _π_ ref( _y_ _[−]_ _|x_ ) exp    - _β_ _[−]_ [1] E _y−∼π_ ref( _·|x_ ) _p_ ( _y_ _[−]_ _≻_ _π_ ref _|x_ ) + _λr_ ( _y_ _[−]_ _|p ⊕_ _x_ )� _._
After simplification, we have:
_ππ_ (( _yy_ _[−]_ [+] _||xx_ )) = _ππ_ refref(( _yy_ _[−]_ [+] _||xx_ )) [exp]  - _β_ _[−]_ [1] [E _y_ + _∼π_ ref( _·|x_ ) _p_ ( _y_ [+] _≻_ _π_ ref _|x_ ) _−_ E _y−∼π_ ref( _·|x_ ) _p_ ( _y_ _[−]_ _≻_ _π_ ref _|x_ )]
+ _λ_ [ _r_ ( _y_ [+] _|p ⊕_ _x_ ) _−_ _r_ ( _y_ _[−]_ _|p ⊕_ _x_ )]� _._
Note that _r_ ( _y_ [+] _|p ⊕_ _x_ ) _−_ _r_ ( _y_ _[−]_ _|p ⊕_ _x_ ) = ∆ _π_ ( _y_ _[−]_ _, y_ [+] ; _x_ ). After some simplifications, we have
_[π]_ [(] _[y]_ [+] _[|][x]_ [)] _[π]_ [(] _[y][−][|][x]_ [)]
_π_ ref( _y_ [+] _|x_ ) _[−]_ [log] _π_ ref( _y_ _[−]_ _|x_
log _[π]_ [(] _[y]_ [+] _[|][x]_ [)]
_π_ ref( _y_ _[−]_ _|x_ )
= _β_ 1 �E _y_ + _∼π_ ref( _·|x_ ) _p_ ( _y_ [+] _≻_ _π_ ref _|x_ ) _−_ E _y−∼π_ ref( _·|x_ ) _p_ ( _y_ _[−]_ _≻_ _π_ ref _|x_ )� + _λ_ ∆ _π_ ( _y_ _[−]_ _, y_ [+] ; _x_ ) _._
As E _y_ + _∼π_ ref( _·|x_ ) _p_ ( _y_ [+] _≻_ _π_ ref _|x_ ) _−_ E _y−∼π_ ref( _·|x_ ) _p_ ( _y_ _[−]_ _≻_ _π_ ref _|x_ ) = 2 [1] [[4], after taking expectations of]
both sides, we obtain
_[π]_ [(] _[y]_ [+] _[|][x]_ [)] _[π]_ [(] _[y][−][|][x]_ [)]
_π_ ref( _y_ [+] _|x_ ) _[−]_ [log] _π_ ref( _y_ _[−]_ _|x_
log _[π]_ [(] _[y]_ [+] _[|][x]_ [)]
_[π]_ [(] _[y]_ _[|][x]_ [)] [1]
_π_ ref( _y_ _[−]_ _|x_ ) [=] 2
(20)
2 _β_ [+] _[ λ]_ [∆] _[π]_ [(] _[y][−][, y]_ [+][;] _[ x]_ [)] _[.]_


Tables 6, 7, 8, and 9 show four examples of responses by Sr-DPO, Sr-IPO, DPO, and IPO. As can
be seen, the proposed Sr-DPO and Sr-IPO outperform DPO and IPO under GPT4’s judgement.
Table 6: Example 1 for Sr-DPO and DPO. The first row is the query, the second and third rows are
the responses generated by Sr-DPO and DPO, respectively, and the last row is the judgement made
by GPT-4.
**Prompt** Explain the process of natural selection and how it contributes to the
evolution and adaptation of species.
**Sr-DPO** Natural selection is the process of survival of the fittest, which occurs
when organisms with the most advantageous traits are more likely to reproduce and pass on their traits than organisms with less desirable traits.
It is a key factor in the evolution of species, with the most successful
species being those that are the most adaptable to the environment. Natural selection ensures that organisms with the most advantageous traits
are more likely to survive and pass on their traits to the next generation.
Here is a brief explanation of the process of natural selection:
1. The most adaptable organisms with the most advantageous traits are
more likely to survive and pass on their traits to the next generation.
2. The most advantageous traits of organisms are those that are most
effective at surviving and reproducing in their environment.
3. Those organisms with the most advantageous traits are more likely
to survive and reproduce than organisms with less advantageous traits.
**DPO** Natural selection is the process of survival of the fittest, which occurs
when the genetic makeup of a population changes over time, favoring individuals that have beneficial traits such as increased physical
strength, increased intelligence, or other adaptive features. This process allows for the survival of the species that is best able to adapt to its
environment.
**Judgment** Both assistants provide a correct explanation of natural selection, describing it as a process that favors individuals with beneficial traits,
leading to changes in the genetic makeup of a population over time.
However, Assistant A’s response is more detailed and comprehensive.
It explains how the probability of an individual’s survival and reproduction is influenced by how well adapted the organism is to its environment, and it also mentions the importance of natural selection in
the field of biology and evolution. Therefore, Assistant A’s response is
more informative and provides a deeper understanding of the topic.
Final Verdict: [[A]]
Table 7: Example 2 for Sr-DPO and DPO. The first row is the query, the second and third rows are
the responses generated by Sr-DPO and DPO, respectively, and the last row is the judgement made
by GPT-4.
**Prompt** Draft an apology email to a customer who experienced a delay in their
order, and provide reassurance that the issue has been resolved.
**Sr-DPO** Here’s an apology email draft for a customer who experienced a delay
in their order:
Dear Customer,
Thank you for your patience during our recent order process. We recently experienced a delay with your order and apologize for the inconvenience. We are currently working to process your order, and we
will provide a notification when it is ready. We sincerely apologize for
the delay and appreciate your patience. Our team is working hard to
resolve this issue as soon as possible. We look forward to welcoming
your order and providing you with the best customer service experience
possible.
Best regards,
Customer Support Team
**DPO** Here’s an apology email draft for a customer who experienced a delay
in their order:
Dear [Customer Name],
apologize for the delay in your delivery of your order. I have worked
with our team to resolve the issue, and I am very pleased to let you
know that the issue has been resolved and your order is now ready for
dispatch. I look forward to providing you with a great customer experience in the future.
Sincerely,
[Customer Name]
Customer Support
**Judgment** Both assistants provided a good response to the user’s request. However, Assistant A’s response was more detailed and provided more reassurance to the customer. Assistant A mentioned that they reviewed the
issue with their fulfillment partners and are now shipping the order as
quickly as possible. This gives the customer a sense of urgency and care
for their order. On the other hand, Assistant B’s response was also good
but lacked the level of detail and reassurance that Assistant A’s response
provided. Therefore, Assistant A’s response is better.
Final Verdict: [[A]]
Table 8: Example 3 for Sr-IPO and IPO. The first row is the query, the second and third rows are
the responses generated by Sr-IPO and IPO, respectively, and the last row is the judgement made by
GPT-4.
**Prompt** Why might someone choose to use a paper map or ask for directions
instead of relying on a GPS device or smartphone app?
**Sr-IPO** A paper map or paper printed directions can be a fun way to help you get
from point A to point B. Paper maps are portable and easy to use, and
they’re also a fun and easy way to explore your surroundings and get
around town. They’re also less expensive than GPS systems or smartphone apps, and they’re very easy to use, so they’re a good option for
older kids or adults who need to get by without a smartphone. Paper
maps offer a more personal and hands-on experience than smartphone
apps or GPS devices, and they can be a fun way to get around town or
explore your surroundings.
**IPO** A paper map or paper printed directions can be a fun way to help you get
from point A to point B. Paper maps are portable and easy to use, and
they’re also a fun and easy way to explore your surroundings and get
around town. They’re also less expensive than GPS systems or smartphone apps, and they’re very easy to carry around. If you are interested
in using a paper map or directions, here are a few reasons why you
might choose to do so:1. Paper maps are portable and easy to use.
2. Paper maps are less expensive than GPS systems or smartphone apps.
3. Paper maps are easy to carry around.
4. Paper maps are easy to use and interpret.
5. Paper maps can help you explore your surroundings and get around
town.
6. Paper maps are easily outdated or outdated, so it is important to be
sure you have the latest version of the map to use.
7. Paper maps are a fun way to explore your surroundings and get
around town.
8. Paper maps are easy to use and interpret.
9. Paper maps can help you explore your surroundings and get around
town.
10. Paper maps make it easy to get around town and explore your surroundings.
11. Paper maps are a fun and easy way to explore your surroundings
and get around town.
Remember to tailor the activities to the birthday childs interests and
preferences. Have a great celebration!
**Judgment** Assistant A’s response is more concise and to the point, providing a
clear explanation of why someone might choose to use a paper map or
ask for directions instead of relying on a GPS device or smartphone app.
Assistant B’s response, while it does provide some valid points, is repetitive and confusing, especially the last sentence about tailoring activities
to a birthday child’s interests and preferences, which is completely irrelevant to the question asked. Therefore, Assistant A’s response is more
helpful and relevant.
Final Verdict: [[A]]
Table 9: Example 4 for Sr-IPO and IPO. The first row is the query, the second and third rows are
the responses generated by Sr-IPO and IPO, respectively, and the last row is the judgement made by
GPT-4.
**Prompt** How many lightning strikes occur on Earth each day? Try to explain
your answer. Your explanation should take the reader through your reasoning step-by-step.
**Sr-IPO** There are about 2,500 lightning strikes each day on Earth. That’s about
two lightning strikes for every minute! The most common type of lightning strikes occur in the tropics, where temperatures range from 70 to
90 degrees Fahrenheit. Lightning strikes are most common around noon
and also around sunset. The next most common type of lightning occurs in the polar regions, where the temperatures can reach as low as -20
degrees Celsius. Lightning strikes are most common during the winter
months in these regions.
The most dangerous type of lightning occurs when the atmospheric
pressure is extremely low, like when a storm is approaching. Low atmospheric pressure can cause the air to move faster and more violently, creating a stronger electric current through the air. When the air is moving
quickly, it is also possible for an electrical charge to build up between
the tips of a cloud or a thunderstorm. This electrical charge can then be
released as lightning. Low atmospheric pressure can also increase the
intensity of a thunderstorm, leading to more severe lightning storms.
**IPO** There are about 2,500 lightning strikes each day on Earth. That’s about
two lightning strikes for every minute! The most common type of lightning strikes occur in the tropics, where temperatures range from 70 to
90 degrees Fahrenheit. Lightning strikes are most common around noon
and also around sunset. The next most common type of lightning occurs in the polar regions, where the temperatures can reach as low as -20
degrees Celsius. Lightning strikes are most common during the winter
months in these regions.
**Judgment** Both Assistant A and Assistant B start their responses with the same information, stating that there are about 2,500 lightning strikes each day
on Earth. They both also provide similar information about where and
when lightning strikes are most common. However, Assistant A goes a
step further by explaining the conditions under which the most dangerous type of lightning occurs, providing a more detailed and comprehensive response to the user’s question. Therefore, Assistant A’s response
is more informative and thorough.
Final Verdict: [[A]]


Lin Gui [1], Cristina Gârbacea [2], and Victor Veitch [1,2]
1 _Department of Statistics, University of Chicago_
2 _Data Science Institute, University of Chicago_

BoNBoN Alignment for Large Language Models and the Sweetness of
  Best-of-n Sampling


This paper concerns the problem of aligning samples from large language models
to human preferences using _best-of-n_ sampling, where we draw _n_ samples, rank
them, and return the best one. We consider two fundamental problems. First:
what is the relationship between best-of- _n_ and approaches to alignment that
train LLMs to output samples with a high expected reward (e.g., RLHF or DPO)?
To answer this, we embed both the best-of- _n_ distribution and the sampling
distributions learned by alignment procedures in a common class of tiltings of
the base LLM distribution. We then show that, within this class, best-of- _n_ is
essentially optimal in terms of the trade-off between win-rate against the base
model vs KL distance from the base model. That is, best-of- _n_ is the best choice
of alignment distribution if the goal is to maximize win rate. However, best-of- _n_
requires drawing _n_ samples for each inference, a substantial cost. To avoid
this, the second problem we consider is how to fine-tune a LLM to mimic the
best-of- _n_ sampling distribution. We derive _BoNBoN Alignment_ to achieve this by
exploiting the special structure of the best-of- _n_ distribution. Experiments show
that BoNBoN alignment yields substantial improvements in producing a model
that is preferred to the base policy while minimally affecting off-target aspects.
[Code is available at https://github.com/gl-ybnbxb/BoNBoN.](https://github.com/gl-ybnbxb/BoNBoN)