**Geyang Guo** [1] _[∗]_ **, Ranchi Zhao** [1] _[∗]_ **, Tianyi Tang** [1] **, Wayne Xin Zhao** [1] _[†]_ **, Ji-Rong Wen** [1] _[,]_ [2]
1Gaoling School of Artificial Intelligence, Renmin University of China.
2School of Information, Renmin University of China.
guogeyang@ruc.edu.cn, ranchizhao@gmail.com,
steventianyitang@outlook.com, batmanfly@gmail.com, jrwen@ruc.edu.cn
ABSTRACT
Alignment with human preference is a desired property of large language models (LLMs). Currently, the main alignment approach is based on reinforcement
learning from human feedback (RLHF). Despite the effectiveness of RLHF, it is
intricate to implement and train, thus recent studies explore how to develop alternative alignment approaches based on supervised fine-tuning (SFT). A major
limitation of SFT is that it essentially does imitation learning, which cannot fully
understand what are the expected behaviors. To address this issue, we propose
an improved alignment approach named **FIGA** . Different from prior methods,
we incorporate fine-grained ( _i.e.,_ token or phrase level) quality signals that are
derived by contrasting good and bad responses. Our approach has made two major contributions. Firstly, we curate a refined alignment dataset that pairs initial
responses and the corresponding revised ones. Secondly, we devise a new loss
function can leverage fine-grained quality signals to instruct the learning of LLMs
for alignment. Extensive experiments have demonstrated the effectiveness of our
approaches by comparing a number of competitive baselines. We release all the
[above-mentioned resources at https://github.com/RUCAIBox/FIGA.](https://github.com/RUCAIBox/FIGA)
1 INTRODUCTION
Pre-trained large language models (LLMs) such as LLaMA (Touvron et al., 2023a) have shown
remarkable potentials to solve various downstream tasks by mastering the universal pre-training
task of next-token prediction. While after large-scale pre-training, it often needs subsequent tuning
for enhancing and regulating the behaviors of LLMs. Two typical approaches are supervised finetuning (SFT) and reinforcement learning from human feedback (RLHF), which can largely improve
LLMs in both task solving capacity and human alignment (Ouyang et al., 2022).
Despite widely explored, SFT and RLHF have their own strengths and weaknesses. On the one
hand, SFT is easy to implement and can effectively boost the general task solving abilities by instruction based eliciting (Wei et al., 2021; Ouyang et al., 2022; Chung et al., 2022), while it mainly
imitates the behaviors of experts (essentially doing _behavior clone_ (Wiseman & Rush, 2016)), which
are demonstrated by the human annotators or powerful LLMs such as ChatGPT. Therefore, the SFT
performance highly relies on high-quality demonstration data (Zhou et al., 2023), and might suffer
from the huge distribution shifts between its outputs and imitated outputs (Zhang et al., 2019; Schulman, 2023; Zhao et al., 2023a). On the other hand, RLHF can better explore the semantic space of
LLMs, and identify the optimal policy by encouraging good behaviors and discouraging bad behaviors during learning. However, it is very complicated to effectively implement, often suffering from
training instability issues such as reward collapse (Song et al., 2023; Wolf et al., 2023).
To leverage the benefits of SFT and RLHF, several recent studies propose to develop alignment approaches without reinforcement learning (RL). These studies typically construct refined instruction
data using methods such as quantile ranking (Lu et al., 2022) and rejection-sampling (Touvron et al.,
_∗_ Equal contribution.
_†_ Corresponding author.
Published as a conference paper at ICLR 2024
2023b), and then follow or slightly modify the original SFT loss. Another line of research designs
alternative optimization approaches that bypasses reward modeling (Rafailov et al., 2023). To conduct effective alignment without RL, a key issue is how to effectively learn by discriminating good
and bad behaviors as that in RLHF (Ouyang et al., 2022), such that LLMs can understand what are
good behaviors to follow and what are bad behaviors to avoid. Despite the prior efforts, they are
largely limited by response-level discrimination signals: they are only aware of the quality label
( _e.g.,_ good or bad) of a demonstration but not what makes it good or bad. Thus, it can’t fully capture
the correct alignment behaviors even demonstrated by what are good and bad behaviors.
In this work, we introduce **FIGA**, a novel method that aligns language models with human preferences. The core idea is to contrast a low-quality initial response from a LLM’s output with a corresponding high-quality revised response by another powerful LLM ( _e.g.,_ ChatGPT), so that LLMs
can be noted with _what are newly added_ (good actions) and _what are removed or substituted_ (bad
actions) from such a revision process. Such fine-grained quality signals can be more useful than the
widely used response-level quality signal. It can instruct LLMs to emphasize the learning of good
actions and penalize the bad actions in a single response. To implement our approach, we first curate an alignment dataset called _SPA_ that pairs an initial response with a revised response under the
guidance of the ground-truth demonstrations. We mainly keep the queries that a LLM performs less
well on, and perform strict filtering. Further, we design a new fine-tuning method that assigns specific token-level weights to different parts ( _e.g.,_ good or bad tokens). Our learning loss can directly
impose fine-grained reward scores to guide the learning of LLMs for improved alignment.
To the best of our knowledge, it is the first attempt that leverages fine-grained quality signals for
improving the alignment of LLMs without RL. Our approach can make LLMs better understand
what are good and bad behaviors beyond simple imitation. By conducting extensive experiments,
we demonstrate that **FIGA** shows promising performance in aligning language models with human
preferences: our approach outperform the initial supervised-finetuned model by notable 3.2 points
and the strong PPO method by 1.8 points.
2 RELATED WORK
In this section, we review the related work in the two aspects, namely reinforcement learning from
human feedback and alignment without reinforcement learning.
**Reinforcement learning from human feedback** Large-scale pre-training empowers large language models (LLMs) to acquire extensive knowledge, underscoring their remarkable potential
across diverse tasks (Brown et al., 2020; Kojima et al., 2022; Zhang et al., 2022; Chowdhery et al.,
2022). Nonetheless, models exclusively focus on next token prediction in pre-training phrase, while
do not consider human preferences. Consequently, this gives rise to unexpected behaviors like harmful or inaccurate information, and emphasizes the necessity to align language models with human
preferences. The current mainstream approaches (Ouyang et al., 2022) to better harness the capabilities of LLMs include supervised fine-tuning (SFT) and reinforcement learning from human feedback
(RLHF). To be specific, this involves three stages: firstly, using SFT to enable the model to better
follow human instructions; subsequently, training a reward model (RM) using human preference
data; and ultimately, tune the model to maximize the reward through the proximal policy optimization (PPO) (Schulman et al., 2017) algorithm. Furthermore, there are works exploring enhancement
for this process (Ramamurthy et al., 2022; Lightman et al., 2023; Lee et al., 2023). However, RLHF
presents challenges due to complex coding and hyper-parameters selecting. Besides, it requires loading three to four models simultaneously, resulting in high memory usage. These challenges propel
researchers to explore alternative approaches to align language models with human feedback.
**Alignment without reinforcement learning** Several studies are based on the rationale that language models have already acquired comprehensive knowledge during the pre-training, and only
high-quality supervised fine-tuning data is required for further tuning (Zhou et al., 2023). So these
works (Liu et al., 2023b; Sun et al., 2023; Bai et al., 2022b; Bhardwaj & Poria, 2023; Krishna et al.,
2022; Gulcehre et al., 2023) bypass reward modeling, and instead concentrate on the construction
of datasets that align well with human preferences. Other works are directed towards exploring substitutes for the intricate PPO algorithm. These efforts employ diverse approaches to learn from the
preference data, encompassing the creation of a supervised fine-tuning training dataset enriched with
Published as a conference paper at ICLR 2024
human preference data (Liu et al., 2023a; Zhang et al., 2023; Dong et al., 2023), the integration of
preferences for different outputs into the loss function (Yuan et al., 2023; Rafailov et al., 2023; Zhao
et al., 2023b; Liu et al., 2023d;c), and the utilization of controllable text generation techniques (Lu
et al., 2022). However, the human preference information used in these methods is at the sentence
level, lacking more fine-grained supervision signals.
3 APPROACH
In this section, we present the proposed alignment approach **FIGA** by leveraging fine-grained quality signals. Our approach is developed based on a specially curated alignment dataset called _SPA_
(Section 3.1), where each low-quality initial response is paired with a high-quality revised response.
Based on such an alignment dataset, we further develop a new loss function that incorporates finegrained quality signals derived by contrasting good and bad responses (Section 3.2). Our approach
is easy to implement (similar to SFT) and can capture the underlying effect to generate high-quality
responses instead of simply imitating them (similar to RLHF), which are discussed in Section 3.3.
The overall framework of our FIGA pipeline is shown in Figure 1.
|Col1|.and there are several|
|---|---|
|
**trains per day**

|
**trains per day**

|
Figure 1: The overall illustration of our alignment approach FIGA.
3.1 CURATED ALIGNMENT DATASET
From the perspective of dataset, the novelty of our alignment approach can be given in two major
aspects. Firstly, we don’t directly aggregate all the available instruction data, but instead focus
on high-quality instruction data that a LLM _performs less well_ on. It enables LLMs to specially
improves their weaknesses, reducing the cost of replicate learning. Secondly, we don’t take what
human annotators write or powerful LLMs ( _e.g.,_ ChatGPT or GPT-4) generate as training targets, but
instead seek a more similar surrogate that is derived based on its own output by a LLM. It can largely
reduce the distribution shift between the LLM to be aligned and the ground-truth demonstrations.
We carefully construct the **S** ub **P** ar **A** lignment (SPA) dataset, a curated collection of query, model’s
initial response, and the corresponding improved response (with minor revision). Compared with
prior work (Ouyang et al., 2022; Yuan et al., 2023; Liu et al., 2023a), we mainly consider the queries
where LLMs’ performance are not satisfactory and aim to correct these bad cases via specific training. Moreover, we refine the initial response of a LLM that is to be aligned as training target, which
can effectively reduce the distribution shifts from the ground-truth demonstrations.
Formally, we denote the initial model as _πθ_, which can be a supervised-finetuned model ( _e.g.,_ Alpaca (Taori et al., 2023)) or a pre-trained base model ( _e.g.,_ LLaMA (Touvron et al., 2023a)). To
construct our dataset, we assume that a reward model for assessing the alignment level is available.
In practice, a number of reward models have been released publicly ( _e.g.,_ DeBERTa (OpenAssistant, 2023)), which can be used for our approach. Given a query _X_ and a response _Y_, we leverage
a reward model RM to compute the reward score _RY_ = RM( _X, Y_ ), which reflects how well the
response _Y_ aligns with given query _X_ . Below, we detail the construction procedure.
Published as a conference paper at ICLR 2024
**Rollout for initial response generation** We first broadly collect existing paired datasets encompassing a wide range of real-world tasks, and construct the instances pool _D_ = _{X, Y }_ _[n]_ _i_ =1 [. To]
better align with human value, we select preference datasets ( _e.g.,_ HH-RLHF (Bai et al., 2022a))
that adhere to the 3H principle ( _i.e.,_ helpfulness, honesty, and harmlessness) in this work. Furthermore, we also include instruction dataset ( _e.g.,_ OpenOrca (Mukherjee et al., 2023)) to preserve the
task solving abilities of LLMs. We aim to train a both capable and safe model like ChatGPT, rather
than only focusing on alignment while sacrificing the task solving abilities. Based on these datasets,
we employ the rollout model _πθ_ to generate initial responses _Y_ [ˆ] = _πθ_ ( _X_ ) for the given queries.
**Identifying the queries to be enhanced** After obtaining the model’s initial response _Y_ [ˆ] and the
human-preferred response _Y_, we next identify the queries where the model requires further improvement to better align with human intent through the reward score RM( _·_ ). Following existing
work (Ouyang et al., 2022), we employ the reward model as a surrogate of human preferences, and
design a filtering process based on the calculated reward score _R_ ˆ _Y_ and _RY_ for all the instances. We
only keep the instances that meet all the three following restrictions: (1) _R_ ˆ _Y < η_ 1 (a subpar initial
performance, _i.e.,_ bad cases), (2) _RY > η_ 2 (high-quality demonstrations), and (3) _RY −_ _R_ ˆ _Y > η_ 3
(clear quality difference), where _η_ 1, _η_ 2, and _η_ 3 are three threshold values for filtering, we will set
them according to the reward score distribution. The details can be found in Section 4.1.2. With the
above filtering mechanism, we ensure the quality and usefulness of our SPA dataset. We target at
bad case correction of the rollout model, which is more directed and effective than existing methods
that directly trains the model on the whole collected dataset.
**Revising initial responses for reducing the distribution shifts** To align a LLM, a basic principle
is to ensure that the distribution of the model should not experience significant shifts during the
alignment process (Bai et al., 2022a). Despite that the ground-truth demonstration ( _Yi_ ) is human
preferred, it is likely to span a very different semantic distribution as the LLM to be aligned. Our
solution is to revise the initial response ( _Y_ [ˆ] ) by referring to the ground-truth demonstration ( _Yi_ ). In
this way, we can effectively reduce the distribution shifts as well as obtaining demonstrations similar
to the original output. Specially, we generate a pseudo reference _Y_ [˜] based the target _Yi_, making minor
adjustments to the _Y_ [ˆ] and enhance its quality, _i.e.,_ modifying _Y_ [ˆ] as minimally as possible based on
_Yi_ . Such a generation process is conducted by prompting the powerful ChatGPT. To facilitate the
generation process, we further manually inspect the low-quality responses that we have previously
filtered and identify four major low-quality reasons: (1) lack of detail, (2) inaccuracy in response,
(3) the need for structural adjustments, and (4) other factors (off-topic or harmful content). In
detail, we leverage ChatGPT to determine, given _Yi_, which of the four reasons _Y_ [ˆ] is associated with.
Afterwards, we design different prompts for the four reasons and instruct the LLM to make minor
correction to the initial response _Y_ [ˆ] based on _Yi_ . We denote the revised response as _Y_ [˜] . The details
of our process and prompts can be found in Appendix B.
Finally, we obtain the SPA dataset _{X,_ _Y,_ [ˆ] _Y_ [˜] _}_ for subsequent training. Our construction method has
dual merits: it not only aligns the reference output with human preferences but also preserves the
inherent linguistic style and overall semantic distribution of the model to be aligned. Note that we
keep both the initial and revised responses in a contrastive form, because they are jointly used for
deriving fine-grained quality signals in subsequent training.
3.2 FINE-GRAINED QUALITY-AWARE ALIGNMENT TUNING
As described above, our fine-tuning dataset for alignment contains both low-quality initial responses
( _Y_ [ˆ] ) and high-quality revised responses ( _Y_ [˜] ). Instead of directly learning from these high-quality
responses (similar to rejection sampling (Touvron et al., 2023b)), it is important for LLMs to understand why such revisions are useful to produce the high-quality responses. Furthermore, LLMs can
improve the alignment capacity from the contrast between good and bad responses.
Motivated by previous work (Liu et al., 2022), we utilize Levenshtein distance to quantify the similarity between of _Y_ [ˆ] and _Y_ [˜] . Levenshtein distance is a dynamic programming algorithm to obtain the
minimal edit distance between two sentences through three operations: addition, deletion, and substitution. Comparing the initial and revised response, the involving tokens can be generally divided
into three types: newly added, deleted, or substituted. We consider assigning different weights to
Published as a conference paper at ICLR 2024
these three types of tokens. We reward the tokens that are added or substituted in the revised response _Y_ [˜], penalize the tokens that are deleted or substituted in the original response _Y_ [ˆ], and tend to
overlook the rest tokens that remain the same after the revision process. Formally, we introduce two
token-level weighting functions to characterize the above ideas:
   - _α,_ if ˜ _yt_ is added or substituted    - _β,_ if ˆ _yt_ is deleted or substituted
_r_ ˜(˜ _yt, t_ ) = _,_ ˆ _r_ (ˆ _yt, t_ ) = _,_ (1)
_γ,_ otherwise 0 _,_ otherwise
where _α >_ 0, _β >_ 0, and _γ ≥_ 0 are three coefficients to control the encouraged, discouraged, and
ignored parts, which can be empirically set or learned from tuning data.
In this way, we can then encourage the model to “imitate” the desired actions that have a greater
impact on enhancing quality, discourage the model from emulating the undesired actions that lead
to a poor performance in quality. The final training loss can be formulated as:
_L_ = _−_ - _r_ ˜(˜ _yt, t_ ) log _πθ_ (˜ _yt|y_ ˜ _<t, X_ )
_y_ ˜ _t∈Y_ [˜]
~~�~~ ~~��~~ ~~�~~
increase the probability of desired words
+ - _r_ ˆ(ˆ _yt, t_ ) log _πθ_ (ˆ _yt|y_ ˆ _<t, X_ )
_y_ ˆ _t∈Y_ [ˆ]
~~�~~ ~~��~~ ~~�~~
decrease the probability of undesired words
_._ (2)
The overall FIGA pipeline is illustrated in Algorithm 1. The major advantages of FIGA over typical
SFT (Ouyang et al., 2022) is that it can learn from fine-grained contrast between good and bad
responses, which is essentially similar to that in reinforcement learning (discussed in Section 3.3).
In addition, by explicitly modeling the revision effect, such an approach can naturally zoom into
crucial words or phrase, making the model better zoom into fine-grained semantics.
**Algorithm 1:** FIGA - Leveraging Fine-grained Quality Signals for Alignment
**1 Input:** Instance pool _D_ = _{X, Y }_ _[n]_ _i_ =1 [, initial model] _[ π][θ]_ [, revision model (ChatGPT), reward function] _[ R]_ ( _·_ ) [.]
**2** ### SPA Dataset Construction

Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment

BEYOND IMITATION: LEVERAGING FINE-GRAINED QUALITY SIGNALS FOR ALIGNMENT

Geyang Guo
Ranchi Zhao
Tianyi Tang
Wayne Xin Zhao
Ji-Rong Wen


**4** 1. Rollout for initial generation. Generate _Y_ [ˆ] _∼_ _πθ_ ( _X_ ) and compute _RY, RY_ ˆ ;
**5** 2. Reward filtering. **if** _RY_ ˆ _> η_ 1 _or RY < η_ 2 _or RY −_ _RY_ ˆ _< η_ 3 **then**
**6** Discard the current instance;
**7** 3. Response Revision. Analyze the reason for the poor performance of _Y_ [ˆ], and generate the
corresponding revision _Y_ [˜] _∼_ LLM( _Y, Y_ [ˆ] ) based on the identified reason.
**8** Construct the SPA dataset _S_ = _{Xi,_ _Y_ [ˆ] _i,_ _Y_ [˜] _i}_ _[m]_ _i_ =1 [.]
**9** ### Alignment Learning


**12** Locate the crucial parts with Levenshtein distance using Equation 1 and assign weights according
to ˜ _r_ (˜ _yt, t_ ) and ˆ _r_ (ˆ _yt, t_ );
**13** Update _πθ_ using the fine-grained quality-aware learning objective in Equation 2.
3.3 DISCUSSION
In this part, we discuss how the proposed FIGA approach relates to existing fine-tuning approaches,
namely SFT and RLHF.
**Relationship with SFT** SFT can be viewed as a special case of our FIGA method without revision,
when training is performed with the higher-quality instance _Y_, and each token of _Y_ is considered
equally important. Compared to SFT, FIGA has the following two advantages: (1) we only consider
the inferior part of the bad case that the initial model does not perform well; (2) we explicitly enforce
the model to understand what are good and bad behaviors in the loss function. It inherits the merits
of SFT, and further leverages fine-fined quality signals for improving the alignment.
**Relationship with RL** Our method can be considered as a simplified but efficient version of RL.
Using typical PPO method (Schulman et al., 2017) as an example, its objective is to optimize the
actor model ( _i.e.,_ the initial model _πθ_ ) to maximize the expected reward score, formally given as:
Published as a conference paper at ICLR 2024
_,_ (3)
_L_ _[P P O]_ = _−_
_t_
- _πθ_ (ˆ _yt|y_ ˆ _<t, X_ )
_πθ_ old(ˆ _yt|y_ ˆ _<t, X_ ) _[·][ A][y]_ [ˆ] _[t]_
where _Ay_ ˆ _t_ is the advantage function of the ˆ _yt_ token returned by the critic model given the reward
score _R_ ˆ _Y_ . _πθ_ old is the model before the previous parameter update. Here, we ignore the clipping
function and KL penalty for convenience. Considering the FIGA training objective in Equation 2,
our weight functions ˜ _r_ ( _·_ ) and ˆ _r_ ( _·_ ) in FIGA can be viewed as a simplified advantage function _A_ ( _·_ )
in Equation 3 to evaluate the importance of each token. Therefore, FIGA has a similar objective
with RL but with a simplified token-wise reward function. We do not use an extra learned critic
model and remove the use of previous rollout model, which makes FIGA more efficient. In the later
experiment section, we will verify the effectiveness of our method.
4 EXPERIMENT
4.1 EXPERIMENTAL SETUP
4.1.1 BASELINE METHODS
In order to better evaluate the FIGA method, we choose several baselines for comparison: (1)
SFT (Ouyang et al., 2022): it continues to fine-tune the initial model using pairs of data with
sequence-to-sequence loss. (2) PPO (Ouyang et al., 2022): it optimizes the initial model to achieve
a higher reward score provided by the reward model. (3) CoH (Liu et al., 2023a): it annotates the
dataset by prefixing “ _A helpful answer:_ ” and “ _An unhelpful answer:_ ” to the responses of corresponding quality, employs SFT on it, and computes loss only for the specially masked tokens. (4)
RRHF (Yuan et al., 2023): it applies SFT on the optimal responses and further optimizes the ranking
loss among responses from multiple sources to encourage the model to achieve a greater log probability for the response that ranks better. (5) DPO (Rafailov et al., 2023): it eliminates the need for
explicit reward modeling and instead directly optimizes the policy model using comparison data.
4.1.2 IMPLEMENTATION DETAILS
**Training Datasets** For our SPA dataset mentioned in Section 3.1, we broadly select the following datasets as our initial instance pool: HH-RLHF (Bai et al., 2022a), ShareGPT (ShareGPT,
2023), Instruct GPT-J Pairwise (Dahoas, 2023), SHP (Ethayarajh et al., 2022), and
OpenOrca (Lian et al., 2023). We employ Alpaca-7b Taori et al. (2023) as the rollout
model to generate responses _Y_ [ˆ] and use gpt-3.5-turbo to revise and obtain _Y_ [˜] . The
prompt used here can be found in Appendix B. As for the filtering process, we utilize
OpenAssistant/reward-model-deberta-v3-large-v2 (OpenAssistant, 2023) as the
reward model. According to reward score distribution 2, we empirically set the threshold values
_η_ 1 = 1 _, η_ 2 = 3 _, η_ 3 = 3 _._ 5, respectively. The statistics for reward scores and edit operations of the
SPA dataset are presented in Table 1, and the graphical illustration of reward scores is provided in
Figure 2. We find that initial responses _Y_ [ˆ] exhibit a large distributional disparity compared with the
reference responses _Y_, which may complicate the learning process for the model. In contrast, our
modified responses not only align more closely with the original distribution but also enhance the
quality, which simplifies the learning task for the rollout model. The completed SPA dataset consists
of 17,333 instances, and more details and analysis can be found in Appendix D.
**Model Details** (1) For SFT, we set the learning rate to 1e-5 and the batch size to 128. We conduct
5 epochs of training and choose the one with the highest reward score on the test set as the ultimate
SFT model. (2) For PPO, we apply the OpenLLaMA2 (OpenLLMAI, 2023) library and adhere
to its hyper-parameter configurations. We use Alpaca-7b as the initial critic model and use the
same reward model utilized in SPA construction. Given the modest gains observed in previous
experiments when employing PPO-ptx on models with around 6B parameters (Ouyang et al., 2022),
we refrain from introducing a pre-training mix as an additional training objective. (3) For CoH, we
annotate the SPA dataset with their method. Considering the smaller size of our dataset compared to
theirs, we set FCM (random masked token ratio to prevent overfitting) to 0. Additionally, to ensure
a fair comparison with PPO, we disable the pre-training dataset regularization. (4) For RRHF and
DPO, we follow the recommended hyper-parameters from the original papers. (5) For FIGA, we
Published as a conference paper at ICLR 2024
Figure 2: Reward score distributions.
Table 1: The average reward score of response data and the average number #ops of
editing operations to them from the _Y_ [ˆ] .
Data _Y_ ˆ _Y_ _Y_ ˜
_R_ ( _·_ ) -1.07 3.94 1.78
#ops   - 75.69 39.38
set the parameters _α_ = 1 _, β_ = 0 _._ 5 _, γ_ = 0 respectively. Besides, considering the instability when
training on negative samples in practice (Bhardwaj & Poria, 2023; Liu et al., 2023a), we further
select the bad tokens returned by Levenshtein distance in equation 1 by retaining only those with a
negative log-likelihood less than 0.6.
4.1.3 EVALUATION TASKS
We evaluate the performances of different methods on comprehensive benchmarks. We segment a
test set from the selected datasets and utilize the reward score to evaluate how effectively the model
has learned to align with human preferences. The resulting test set comprises a total of 3,608 data
entries. Additionally, we employ a broad array of out-of-distribution benchmarks to conduct a more
comprehensive evaluation of the model’s capabilities. This includes assessing knowledge utilization (MMLU (Hendrycks et al., 2020)), human alignment (WinoGender (Rudinger et al., 2018),
CrowS-Pairs (Nangia et al., 2020), and TruthfulQA (Lin et al., 2021)), and open-ended generation
(Vicuna (Chiang et al., 2023) and WizardLM (Xu et al., 2023)). The details of evaluation tasks can
be found in Appendix C.
4.2 EXPERIMENTAL RESULTS
Table 2: Performance comparison of FIGA and other widely used alignment methods. Bold and
underlined fonts indicate the best and the second-best score. _↓_ denotes lower is better.
|Methods|Reward|MMLU TruthfulQA CrowS-Pairs↓ WinoGender Vicuna WizardLM|Average1|
|---|---|---|---|
|**Alpaca-7b**|3.96|39.2
33.7
61.1
55.6
7.9
7.0|31.7|
|**SFT**
**PPO (SPA)**
**PPO (85K)**2
**CoH**
**RRHF**
**DPO**|4.56
4.06
4.54
4.24
4.23
4.23|39.3
22.0
61.5
55.3
8.4
**8.3**
39.6
30.1
61.3
56.2
7.6
7.4
39.2
36.7
60.6
56.2
7.9
7.2
39.6
28.2
**59.6**
52.1
8.3
8.1
37.8
32.9
59.9
**60.0**
7.9
7.9
40.1
34.8
61.2
57.0
8.0
7.7|31.1
31.5
33.1
32.7
31.3
32.7|
|**FIGA**
**4.62**
**40.8**
**42.0**
61.2
59.6
**8.6**
**8.3**
**34.9**|**FIGA**
**4.62**
**40.8**
**42.0**
61.2
59.6
**8.6**
**8.3**
**34.9**|**FIGA**
**4.62**
**40.8**
**42.0**
61.2
59.6
**8.6**
**8.3**
**34.9**|**FIGA**
**4.62**
**40.8**
**42.0**
61.2
59.6
**8.6**
**8.3**
**34.9**|
As in Table 2, FIGA surpasses all baselines, showing superior performance across benchmarks,
even outperforming PPO using four times training data. This implies FIGA aligns more closely
with human preferences and exhibits strong overall task-solving capabilities.
Moreover, to assess the comparative advantages of each response, we conduct a comparison between
the responses generated by FIGA and other baseline methods on the Vicuna and WizardLM benchmarks. The results are shown in Figure 3. And we also conduct human evaluation in Appendix F
for more fine-grained analysis.
1To reflect the model’s overall performance, we compute the average score. Specifically, we multiply the
reward score by 10, and the score for CrowS-Pairs is calculated as 100 minus the original score.
2Given that PPO does not utilize labels in the dataset and requires a large amount of data to learn through
trial and error, we integrate additional open-source data with the SPA dataset to fully leverage the strengths of
PPO. We obtain a total of 84,908 entries, and the PPO trained with this dataset is referred to as PPO (85K).
Published as a conference paper at ICLR 2024
Figure 3: Win rate of FIGA vs other baselines on Vicuna (left) and WizardLM (right).
4.3 FURTHER ANALYSIS
4.3.1 PERFORMANCE COMPARISON W.R.T. SUBPAR ALIGNMENT DATASET
As mentioned in Section 3.1, the steps involved in constructing the SPA dataset include: (1) collecting existing datasets, including preference datasets and typical instruction datasets; (2) filtering
the data based on reward scores; and (3) revising the initial responses using LLM. To examine the
effectiveness of each of them, we develop the following dataset variants:
_•_ **Preference** : we only use preference data to construct the initial instance pool _D_ with 3,971 samples.
_•_ **Instruction** : we construct the initial instance pool _D_ with typical instruction data that the reward
model had not encountered during its training, totaling 3,971 instances.
_•_ **W/o reward filtering** : this variant excludes the step of data filtering according to reward scores.
_•_ **W/o revision** : we do not utilize LLM to revise and instead directly employ the reference responses.
Table 3: Performance comparison of different instances pools.
|Methods|Reward|MMLU TruthfulQA CrowS-Pairs↓ WinoGender Vicuna WizardLM|Average|
|---|---|---|---|
|**Preference**
**Instruction**|**4.42**
4.35|37.4
22.6
61.5
57.1
7.4
6.6
**40.7**
**31.1**
**59.7**
**57.5**
**8.5**
**8.2**|30.5
**32.8**|
Table 4: Performance comparison of different data annotations.
|Methods|Reward|MMLU TruthfulQA CrowS-Pairs↓ WinoGender Vicuna WizardLM|Average|
|---|---|---|---|
|**FIGA**|**4.62**|**40.8**
**42.0**
61.2
**59.6**
**8.6**
**8.3**|**34.9**|
|**W/o reward fltering**
4.41
38.0
28.8
**61.1**
58.5
8.3
8.0
32.1
**W/o revision**
4.39
37.5
26.7
62.1
55.6
8.2
7.7
31.1|**W/o reward fltering**
4.41
38.0
28.8
**61.1**
58.5
8.3
8.0
32.1
**W/o revision**
4.39
37.5
26.7
62.1
55.6
8.2
7.7
31.1|**W/o reward fltering**
4.41
38.0
28.8
**61.1**
58.5
8.3
8.0
32.1
**W/o revision**
4.39
37.5
26.7
62.1
55.6
8.2
7.7
31.1|**W/o reward fltering**
4.41
38.0
28.8
**61.1**
58.5
8.3
8.0
32.1
**W/o revision**
4.39
37.5
26.7
62.1
55.6
8.2
7.7
31.1|
From the results in Table 3 and Table 4, we can see that: (1) FIGA demonstrates strong performance
on typical instruction data that is new to the reward model, proving that its applicability is not
restricted to preference data. (2) Filtering based on reward scores is crucial, resulting in a +0.21
reward score increase and a +2.8 benchmark increase. This underscores the significance of training
on queries where the model’s original performance is subpar. (3) Addressing the distribution shift
through revisions is important, as training with revisions yields +3.8 points on average.
4.3.2 PERFORMANCE COMPARISON W.R.T. WEIGHTING FUNCTIONS
As mentioned in Section 3.2, ˜ _r_ ( _·_ ) and ˆ _r_ ( _·_ ) in Equation 1 first make comparison between _Y_ [ˆ] and _Y_ [˜],
and then assign distinct weights to various tokens. Here, we explore other weighting functions as
how they acquire the tokens to be encouraged or discouraged, and study the influence of different
hyper-parameters ( _α_, _β_, and _γ_ ). More details on hyper-parameters can be referred to in Appendix E.
Published as a conference paper at ICLR 2024
_•_ **Variants of** ˜ _r_ ( _·_ ): we set _β_ to 0 and propose three different variants to explore alternative methods
for identifying the tokens that should be encouraged.
**– Bag of words** : it sets ˜ _r_ ( ˜ _yt, t_ ) = 1 only when ˜ _yt /∈_ _Y_ [ˆ] ; while the rest are set to 0.
**– ChatGPT (weighted)** : motivated by the work (Lee et al., 2023), it employs ChatGPT to
assess the impact of tokens on sentence quality. The specific prompt can be found in B. The
returned scores are adjusted to fall within the range of 0.7 to 1.3 and are set as ˜ _r_ ( ˜ _yt, t_ ). For
words that ChatGPT doesn’t address, ˜ _r_ ( ˜ _yt, t_ ) = 0 _._ 3.
**– ChatGPT (binary)** : it sets ˜ _r_ ( ˜ _yt, t_ ) to 1 only when ˜ _yt_ is returned by ChatGPT with a non-zero
score, while the rest are set to 0.
_•_ **Variants of** ˆ _r_ ( _·_ ): as for the tokens to be discouraged returned by ˆ _r_ ( _·_ ), we further filter bad tokens
returned by Levenshtein distance and retain only those with a negative log-likelihood below 0.6.
To assess its effectiveness, we design the variants including:
**– Inverted threshold** : it retains only the bad tokens returned by Levenshtein distance with a
negative log-likelihood _≥_ 0 _._ 6.
**– W/o further selection** : it penalizes all the bad tokens returned by Levenshtein distance.
_•_ **Variants of hyper-parameters** : to explore the influence of _α, β, γ_ in Equation 1, we design:
**–** _**β**_ **= 0** : it sets _β_ to 0 with _α_ = 1 and _γ_ = 0.
**–** _**γ ̸**_ **= 0** : it sets _γ_ to 0.3 with _α_ = 1 and _β_ = 0 _._ 5.
**–** _**R**_ **(** _**·**_ **)** : it assigns _R_ ˜ _Y, RY_ ˆ _,_ 0 to _α, β, γ_ respectively, where _R_ ˜ _Y_ and _R_ ˆ _Y_ are standardized
through the min-max method.
Table 5: Performance comparison of different weighting functions.
|Explorations|Methods|Reward|MMLU TruthfulQA CrowS-Pairs↓ WinoGender Vicuna WizardLM|Average|
|---|---|---|---|---|
|**Ours**|FIGA|**4.62**|**40.8**
**42.0**
61.2
**59.6**
**8.6**
**8.3**|**34.9**|
|**Encouraged**|Bag of words
ChatGPT (weighted)
ChatGPT (binary)|4.52
4.37
4.32|40.4
29.3
60.0
57.6
8.1
8.2
39.8
21.7
60.0
57.9
8.4
8.1
39.0
24.4
**59.9**
59.0
7.8
7.6|32.7
31.4
31.6|
|**Discouraged**
Inverted threshold
3.80
30.2
27.2
**56.2**
50.4
8.1
7.4
29.3
W/o further selection
3.01
28.1
24.0
58.5
57.4
8.0
7.7
28.1|**Discouraged**
Inverted threshold
3.80
30.2
27.2
**56.2**
50.4
8.1
7.4
29.3
W/o further selection
3.01
28.1
24.0
58.5
57.4
8.0
7.7
28.1|**Discouraged**
Inverted threshold
3.80
30.2
27.2
**56.2**
50.4
8.1
7.4
29.3
W/o further selection
3.01
28.1
24.0
58.5
57.4
8.0
7.7
28.1|**Discouraged**
Inverted threshold
3.80
30.2
27.2
**56.2**
50.4
8.1
7.4
29.3
W/o further selection
3.01
28.1
24.0
58.5
57.4
8.0
7.7
28.1|**Discouraged**
Inverted threshold
3.80
30.2
27.2
**56.2**
50.4
8.1
7.4
29.3
W/o further selection
3.01
28.1
24.0
58.5
57.4
8.0
7.7
28.1|
|**Hyper-parameter**
_β_ = 0
4.61
41.0
37.0
**59.6**
58.1
8.5
8.3
34.2
_γ _= 0
4.54
**41.2**
32.2
60.1
56.0
8.4
8.2
33.0
_R_(_·_)
4.54
39.7
37.8
62.9
57.1
8.2
8.2
33.4|**Hyper-parameter**
_β_ = 0
4.61
41.0
37.0
**59.6**
58.1
8.5
8.3
34.2
_γ _= 0
4.54
**41.2**
32.2
60.1
56.0
8.4
8.2
33.0
_R_(_·_)
4.54
39.7
37.8
62.9
57.1
8.2
8.2
33.4|**Hyper-parameter**
_β_ = 0
4.61
41.0
37.0
**59.6**
58.1
8.5
8.3
34.2
_γ _= 0
4.54
**41.2**
32.2
60.1
56.0
8.4
8.2
33.0
_R_(_·_)
4.54
39.7
37.8
62.9
57.1
8.2
8.2
33.4|**Hyper-parameter**
_β_ = 0
4.61
41.0
37.0
**59.6**
58.1
8.5
8.3
34.2
_γ _= 0
4.54
**41.2**
32.2
60.1
56.0
8.4
8.2
33.0
_R_(_·_)
4.54
39.7
37.8
62.9
57.1
8.2
8.2
33.4|**Hyper-parameter**
_β_ = 0
4.61
41.0
37.0
**59.6**
58.1
8.5
8.3
34.2
_γ _= 0
4.54
**41.2**
32.2
60.1
56.0
8.4
8.2
33.0
_R_(_·_)
4.54
39.7
37.8
62.9
57.1
8.2
8.2
33.4|
The results in Table 5 indicate that: (1) Levenshtein distance excels in extracting critical tokens, with
over +1.5 and +2.6 average scores compared with the statistical method and ChatGPT annotation
method. (2) It is necessary to further filter the bad tokens returned by Levenshtein distance, as this
leads to an average improvement of +6.8. (3) Remaining only the poor-quality tokens with a negative
log-likelihood _≤_ 0.6 is a sensible choice, which aims to penalize tokens that the model is relatively
confident in generating, even though their actual quality is subpar. (4) Punishing the undesirable
actions is beneficial, as it results in an average increase of +0.7. (5) Focusing only on good and bad
tokens is sufficient, since setting _γ_ to a non-zero value leads to a decrease of 1.9. (6) The inferior
performance of reward score weights can be attributed to intrinsic inaccuracies of the reward scores,
especially in out-of-distribution scenarios (Bai et al., 2022b).
5 CONCLUSION
In this paper, we have presented FIGA, a new approach that aligns language models with human
preferences, by leveraging fine-grained quality signals to enhance the alignment quality during finetuning. In our approach, we firstly curate a high-quality alignment dataset that pairs initial responses with revised responses on queries that a LLM cannot perform well. Furthermore, we have
designed a new learning objective that that can leverage the fine-grained quality signals by contrasting initial with revised responses. Our approach inherits the merits of SFT ( _e.g.,_ efficient and
easy-to-implement), and meanwhile can better understand and learn what are correct behaviors for
alignment. FIGA shows superior performance on extensive tasks, with +3.2 points and +1.8 points
against the initial supervised-finetuned model and the strong PPO method.
Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was partially supported by National Natural Science Foundation of China under Grant
No. 62222215, Beijing Natural Science Foundation under Grant No. L233008 and 4222027. Xin
Zhao is the corresponding author.
REFERENCES
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn
Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless
assistant with reinforcement learning from human feedback. _arXiv preprint
2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones,
Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. _arXiv preprint
Rishabh Bhardwaj and Soujanya Poria. Red-teaming large language models using chain of utterances for safety-alignment. _arXiv preprint
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are
few-shot learners. _Advances in neural information processing systems_, 33:1877–1901, 2020.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng,
Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot
impressing gpt-4 with 90%* chatgpt quality. _See https://vicuna. lmsys. org (accessed 14 April_
_2023)_, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam
Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm:
Scaling language modeling with pathways. _arXiv preprint
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi
Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. _arXiv preprint
[Dahoas. Dahoas/synthetic-instruct-gptj-pairwise. https://huggingface.co/datasets/](https://huggingface.co/datasets/Dahoas/synthetic-instruct-gptj-pairwise)
[Dahoas/synthetic-instruct-gptj-pairwise, 2023.](https://huggingface.co/datasets/Dahoas/synthetic-instruct-gptj-pairwise)
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum,
and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment.
_arXiv preprint
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with
_V_ -usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari,
Gang Niu, and Sivan Sabato (eds.), _Proceedings of the 39th International Conference on Machine_
_Learning_, 2022.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek
Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training
(rest) for language modeling. _arXiv preprint
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and
Jacob Steinhardt. Measuring massive multitask language understanding. _arXiv preprint_
_
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large
language models are zero-shot reasoners. _Advances in neural information processing systems_,
35:22199–22213, 2022.
Published as a conference paper at ICLR 2024
Ranjay Krishna, Donsuk Lee, Li Fei-Fei, and Michael S Bernstein. Socially situated artificial intelligence enables learning from human interaction. _Proceedings of the National Academy of_
_Sciences_, 2022.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor
Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with
ai feedback. _arXiv preprint
Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and “Teknium”’.
Openorca: An open dataset of gpt augmented flan reasoning traces. [https://https://](https://https://huggingface.co/Open-Orca/OpenOrca)
[huggingface.co/Open-Orca/OpenOrca, 2023.](https://https://huggingface.co/Open-Orca/OpenOrca)
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan
Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. _arXiv preprint_
_
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human
falsehoods. _arXiv preprint
Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Chain of hindsight aligns language models with
feedback. _arXiv preprint
Ruibo Liu, Chenyan Jia, Ge Zhang, Ziyu Zhuang, Tony Liu, and Soroush Vosoughi. Second thoughts
are best: Learning to re-align with human values from text edits. _Advances in Neural Information_
_Processing Systems_, 35:181–196, 2022.
Ruibo Liu, Ruixin Yang, Chenyan Jia, Ge Zhang, Denny Zhou, Andrew M Dai, Diyi Yang, and
Soroush Vosoughi. Training socially aligned language models in simulated human society. _arXiv_
_preprint
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and
Jialu Liu. Statistical rejection sampling improves preference optimization. _arXiv preprint_
_
Yixin Liu, Alexander R Fabbri, Pengfei Liu, Dragomir Radev, and Arman Cohan. On learning to
summarize with large language models as references. _arXiv preprint
Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning.
_Advances in neural information processing systems_, 35:27591–27609, 2022.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and
Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4. _arXiv_
_preprint
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R Bowman. Crows-pairs: A challenge
dataset for measuring social biases in masked language models. _arXiv preprint
2020.
OpenAssistant. Openassistant/reward-model-deberta-v3-large-v2. [https://huggingface.](https://huggingface.co/OpenAssistant/reward-model-deberta-v3-large-v2)
[co/OpenAssistant/reward-model-deberta-v3-large-v2, 2023.](https://huggingface.co/OpenAssistant/reward-model-deberta-v3-large-v2)
[OpenLLMAI. Openllama2. https://github.com/OpenLLMAI/OpenLLaMA2, 2023.](https://github.com/OpenLLMAI/OpenLLaMA2)
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong
Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow
instructions with human feedback. _Advances in Neural Information Processing Systems_, 35:
27730–27744, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea
Finn. Direct preference optimization: Your language model is secretly a reward model. _arXiv_
_preprint
Published as a conference paper at ICLR 2024
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kiant´e Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural
language processing?: Benchmarks, baselines, and building blocks for natural language policy
optimization. _arXiv preprint
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in
coreference resolution. _arXiv preprint
John Schulman. Reinforcement learning from human feedback: Progress and challenges, 2023.
[URL https://www.youtube.com/watch?v=hhiLw5Q_UFg.](https://www.youtube.com/watch?v=hhiLw5Q_UFg)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy
optimization algorithms. _arXiv preprint
ShareGPT. Sharegpt ~~v~~ icuna ~~u~~ nfiltered. [https://huggingface.co/datasets/](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered)
[anon8231489123/ShareGPT_Vicuna_unfiltered, 2023.](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered)
Ziang Song, Tianle Cai, Jason D Lee, and Weijie J Su. Reward collapse in aligning large language
models. _arXiv preprint
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming
Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with
minimal human supervision. _arXiv preprint
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy
Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model.
[https://github.com/tatsu-lab/stanford_alpaca, 2023.](https://github.com/tatsu-lab/stanford_alpaca)
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee
Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and
efficient foundation language models. _arXiv preprint
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In _International_
_Conference on Learning Representations_, 2021.
Sam Wiseman and Alexander M Rush. Sequence-to-sequence learning as beam-search optimization.
In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_,
pp. 1296–1306, 2016.
Yotam Wolf, Noam Wies, Yoav Levine, and Amnon Shashua. Fundamental limitations of alignment
in large language models. _arXiv preprint
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and
Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions.
_arXiv preprint
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf:
Rank responses to align language models with human feedback without tears. _arXiv preprint_
_
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer
language models. _arXiv preprint
Tianjun Zhang, Fangchen Liu, Justin Wong, Pieter Abbeel, and Joseph E Gonzalez. The wisdom of
hindsight makes language models better instruction followers. _arXiv preprint
2023.
Published as a conference paper at ICLR 2024
Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. Bridging the gap between training
and inference for neural machine translation. _arXiv preprint
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min,
Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. _arXiv_
_preprint
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf:
Sequence likelihood calibration with human feedback. _arXiv preprint
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang,
Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and
chatbot arena. _arXiv preprint
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat,
Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. _arXiv preprint
A APPENDIX: DATA SOURCES
(1) **HH-RLHF (Helpful and Harmless)** : It comprises two main categories of data: human preference data about helpfulness and harmlessness, and human-annotated red teaming dialogues. The first
one is pivotal to train the reward model, and the second one gives insights into model red-teaming
techniques [1] .
(2) **ShareGPT** : The dataset contains conversations through the API using process. Within each
conversation, both user prompts and ChatGPT responses from OpenAI are presented [2] .
(3) **Synthetic Instruct GPT-J Pairwise** : Crafted for instruction-oriented tasks, it explores modelgenerated outputs when exposed to synthetic prompts [3] .
(4) **Stanford SHP** : It offers 385K human preferences across multiple disciplines. These preferences
are designed to discern the relative helpfulness of responses. Contrary to the HH-RLHF dataset, all
content in SHP is penned by humans, serving as a valuable complement to other datasets [4] .
(5) **OpenOrca** : This dataset is an extension of the FLAN Collection, including GPT-4 and GPT-3.5
model completions. Its primary application lies in training and evaluation in the realm of NLP. For
our investigation, we’ve exclusively focused on the English instruction subset [5] .
B APPENDIX: PROMPTS USED FOR DATA AUGMENTATION
**Details for revision** Given a question, along with the poorer original model response and a preferred ground truth response, we instruct ChatGPT to make minimal modifications to the original
response, while ensuring that the output still remains closely aligned with the preferred response.
This process can be divided into two steps: first analyzing the reasons for the lower quality of
the original response based on the comparison, and then, making revisions using the appropriate
prompts based on these factors.
Prompt to used analyze the reason: _Question: ... Response 1: ... Response 2: ... Among them, the_
_quality of Response 1 is inferior to that of Response 2. Please compare them and choose one of the_
_following four possible reasons for the area where Response 1 performed the worst: A. Needs more_
_accurate content, B. Needs more comprehensive content or more details, C. Requires adjustments_
_in structure, D. Other reasons (such as containing harmful information or going off-topic). Do not_
_include analysis, but just return the choice._
[1https://huggingface.co/datasets/Anthropic/hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf)
[2https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered)
[unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered)
[3https://huggingface.co/datasets/Dahoas/synthetic-instruct-gptj-pairwise](https://huggingface.co/datasets/Dahoas/synthetic-instruct-gptj-pairwise)
[4https://huggingface.co/datasets/stanfordnlp/SHP](https://huggingface.co/datasets/stanfordnlp/SHP)
[5https://huggingface.co/datasets/Open-Orca/OpenOrca](https://huggingface.co/datasets/Open-Orca/OpenOrca)
Published as a conference paper at ICLR 2024
The prompts used to revise according to different reasons:
Prompt for reason A: _Question: ... Response 1: ... Response 2: ... Please replace the content_
_corresponding to Response 1 with the accurate and high-quality essence from Response 2, and_
_remain the original structure of Response 1. Ensure that the edit distance between the optimized_
_Response 1 and the Response 1 is as low as possible._
Prompt for reason B: _Question: ... Response 1: ... Response 2: ... Please incorporate the compre-_
_hensive topic or the details from Response 2 into Response 1, or if necessary, replace any synony-_
_mous content from Response 1 with that from Response 2. You must remain the original structure_
_of Response 1, ensure the edit distance between the optimized Response 1 with the Response 1 is as_
_low as possible, and not add new contents other than those contained in Response 1 and Response_
_2._
Prompt for reason C: _Question: ... Response 1: ... Response 2: ... The structure of Response 2_
_is well-organized, featuring elements including but not limited to: 1. point-by-point addressing, 2._
_providing an overview of the question before answering. Use the structure of Response 2 to rephrase_
_Response 1. Ensure that the optimized Response 1 should maintain a relatively low edit distance_
_from the original Response 1._
**Annotate the importance of each word** Given a question, along with the lower-quality original
response from the initial model and a higher-quality ground truth response, we require ChatGPT to
score each word based on comparison, in terms of how much it improve the quality. Below is an
example.
_Below is an instruction that describes a task, followed by an original response and a better response_
_in terms of how well it aligns with human preferences, being helpful, harmless, and honest. Your_
_task is to return a list containing tuples with words and corresponding scores, which are meant_
_to measure the extent to which the words improve the quality of the original answer to the better_
_answer. The scores are all integers, with 0 being the lowest score and 5 being the highest score._
_Instruction: ... Original Response: ... Better Response: ..._
C APPENDIX: EVALUATION DETAILS
We utilize a suite of benchmarks to evaluate the performance as mentioned in Section 4.1.3. Here,
we provide details on how we use these benchmarks for evaluation.
**MMLU:** The multitask multilingual language understanding (MMLU) benchmark consists of 57
subtasks. Our approach involves calculating the negative log-likelihood of the correct option across
all tasks, ensuring it is the minimal among all options. Finally, we calculate the accuracy of correctly answered questions. The implementation details can be found at the _LM-evaluation-harness_
repository [6] .
**TruthfulQA:** For the TruthfulQA benchmark, we select the multiple-choice task (MC1), which
contains only one correct option. Similar to MMLU, we evaluate by determining if the negative
log-likelihood of the correct option is the lowest among all options, and calculate the accuracy of
correctly answered questions.
**CrowS-Pairs:** This benchmark involves comparing two sentences: one exhibits a common form of
bias, while the other one does not, differing only in a few words. We calculate the perplexity for
each sentence and determine the bias rate by evaluating which sentence has a lower perplexity. A
bias rate closer to 50% indicates less bias in the model.
**Winogender:** It assesses the model’s accuracy in handling Winograd schema challenge tasks under
three scenarios: male, female, and unspecified gender. It is based on whether the negative loglikelihood of the correct sentence is lower. Our primary metric is the average score across these
gender scenarios.
[6https://github.com/EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)
Published as a conference paper at ICLR 2024
**Vicuna and WizardLM:** These two benchmarks instruct the model to generate responses to the
given prompts. The responses are then rated by ChatGPT (Zheng et al., 2023) on a scale of 1 to 10,
with the overall performance measured by the average score.
D APPENDIX: SPA DATASET DETAILS
We have conducted a budget estimation based on the usage of the GPT-3.5-turbo API for our SPA
dataset. The average input token count per data entry in the SPA dataset is 952, and the average
output token count is 143. Considering the pricing of API usage ( _i.e.,_ $0.001 and $0.002 per 1K
tokens for input and output respectively), the total estimated cost for the entire SPA dataset amounts
to $21.45.
Additionally, we conduct experiments with various sizes of the SPA dataset to investigate how much
data is necessary to achieve a reasonable performance gain. The results in Table 6 indicate that
constructing a SPA dataset with 8000 samples and training based on the FIGA method can lead to a
substantial performance boost.
Table 6: Performance comparison of different data sizes.
|#Data|Reward|MMLU TruthfulQA CrowS-Pairs↓ WinoGender Vicuna WizardLM|Average|
|---|---|---|---|
|**Alpaca-7b**|3.96|39.2
33.7
61.1
55.6
7.9
7.0|31.7|
|**4k**
**8k**
**17k**|4.28
4.58
4.62|39.9
26.3
59.0
55.7
8.6
8.4
41.0
33.2
59.8
57.2
8.4
8.2
40.8
42.0
61.2
59.6
8.6
8.3|31.8
33.4
34.9|
E APPENDIX: HYPTER-PARAMETER SETTINGS
Below are the results of our experiments conducted with various hyper- parameter configurations.
Table 7: Performance comparison of different hyper-parameters.
|Explorations|α/β/γ|Reward|MMLU TruthfulQA CrowS-Pairs↓ WinoGender Vicuna WizardLM|Average|
|---|---|---|---|---|
|**Ours**|1/0.5/0|**4.62**|**40.8**
**42.0**
61.2
**59.6**
**8.6**
**8.3**|**34.9**|
|**w.r.t.**_ α_
_R_(_._)/0.5/0
4.54
39.7
37.8
62.9
57.1
8.2
8.2
33.4
_R_(_._)/0/0
4.46
39.9
27.3
60.0
60.1
8.1
7.9
32.6|**w.r.t.**_ α_
_R_(_._)/0.5/0
4.54
39.7
37.8
62.9
57.1
8.2
8.2
33.4
_R_(_._)/0/0
4.46
39.9
27.3
60.0
60.1
8.1
7.9
32.6|**w.r.t.**_ α_
_R_(_._)/0.5/0
4.54
39.7
37.8
62.9
57.1
8.2
8.2
33.4
_R_(_._)/0/0
4.46
39.9
27.3
60.0
60.1
8.1
7.9
32.6|**w.r.t.**_ α_
_R_(_._)/0.5/0
4.54
39.7
37.8
62.9
57.1
8.2
8.2
33.4
_R_(_._)/0/0
4.46
39.9
27.3
60.0
60.1
8.1
7.9
32.6|**w.r.t.**_ α_
_R_(_._)/0.5/0
4.54
39.7
37.8
62.9
57.1
8.2
8.2
33.4
_R_(_._)/0/0
4.46
39.9
27.3
60.0
60.1
8.1
7.9
32.6|
|**w.r.t.**_ β_
1/0.5/0
4.65
41.1
41.4
60.1
57.1
8.6
8.4
34.7
1/0.25/0
4.59
38.8
40.3
61.5
60.6
8.3
7.9
34.3
1/0.2/0
4.64
40.1
38.3
61.7
59.9
8.4
8.3
34.2|**w.r.t.**_ β_
1/0.5/0
4.65
41.1
41.4
60.1
57.1
8.6
8.4
34.7
1/0.25/0
4.59
38.8
40.3
61.5
60.6
8.3
7.9
34.3
1/0.2/0
4.64
40.1
38.3
61.7
59.9
8.4
8.3
34.2|**w.r.t.**_ β_
1/0.5/0
4.65
41.1
41.4
60.1
57.1
8.6
8.4
34.7
1/0.25/0
4.59
38.8
40.3
61.5
60.6
8.3
7.9
34.3
1/0.2/0
4.64
40.1
38.3
61.7
59.9
8.4
8.3
34.2|**w.r.t.**_ β_
1/0.5/0
4.65
41.1
41.4
60.1
57.1
8.6
8.4
34.7
1/0.25/0
4.59
38.8
40.3
61.5
60.6
8.3
7.9
34.3
1/0.2/0
4.64
40.1
38.3
61.7
59.9
8.4
8.3
34.2|**w.r.t.**_ β_
1/0.5/0
4.65
41.1
41.4
60.1
57.1
8.6
8.4
34.7
1/0.25/0
4.59
38.8
40.3
61.5
60.6
8.3
7.9
34.3
1/0.2/0
4.64
40.1
38.3
61.7
59.9
8.4
8.3
34.2|
|**w.r.t.**_ γ_|1/0.5/0.3
1/0/0.3|4.54
4.50|41.2
32.2
60.1
56.0
8.4
8.2
40.4
30.8
59.8
56.4
8.4
8.1|33.0
32.8|
From the Table 7, it can be observed that: (i) _γ_ should be set to 0, as a non-zero _γ_ leads to a
decrease in results. (ii) _α_ should be set to 1, as setting _α_ to the reward score results in a decrease
in performance. (iii) The final results are insensitive to _β_, indicating that as long as _β_ is set to a
reasonable value, it will not notably impact the overall performance.
F APPENDIX: HUMAN EVALUATION
In addition to all the automatic evaluation metrics, we further carry out human evaluation to test
how much the model align to human preference after FIGA training. To be specific, we explore the
following aspects:
Firstly, we randomly select 20 instructions from the test set and choose the outputs before FIGA
training and after FIGA training. We ask 3 participants to assess whether these responses exhibit the
four aforementioned types of errors. The results in Table 8 show that after FIGA training, errors of
all types have significantly decreased.
Then, we evaluate the quality of outputs through human evaluation. We randomly select 30 instructions from Vicuna benchmark, along with corresponding responses from FIGA and other baseline
Published as a conference paper at ICLR 2024
Table 8: Human evaluation of error types.
|Error Type|Inaccuracy Lack of Details Structure Others (off-topic or harmful)|
|---|---|
|**before FIGA training**
**after FIGA training**|20%
35%
5%
0%
5%
10%
0%
0%|
methods (including ChatGPT). We invite three evaluators to make pairwise comparisons between
FIGA responses and those of each baseline method, aiming to identify the one with higher quality.
Finally, we aggregate the assessments from the three participants and obtain the final evaluation
results through the mode method. The results in Table 9 show that FIGA outperforms all baseline
methods, indicating its ability to better align with human preferences.
Table 9: Human evaluation of overall quality.
|FIGA against|Alpaca-7b SFT PPO PPO (85k) RRHF CoH DPO ChatGPT|
|---|---|
|**FIGA win rate**|73%
57%
73%
83%
73%
57%
73%
40%|

11 for _each instance {X,_ _Y,_ [ˆ] _Y_ [˜] _} in SPA S_ do


**Content Warning: This paper contains examples of harmful language.**
Yixu Wang _[†]_ [1,2], Yan Teng* [2], Kexin Huang [2], Chengqi Lyu [2], Songyang Zhang [2],
Wenwei Zhang [2], Xingjun Ma* [1,2], Yu-Gang Jiang [1], Yu Qiao [2], and Yingchun Wang [2]
1 _Fudan University_
2 _Shanghai Artificial Intelligence Laboratory_

Fake Alignment: Are LLMs Really Aligned Well?

Yixu Wang
Yan Teng
Kexin Huang
Chengqi Lyu
Songyang Zhang
Wenwei Zhang
Xingjun Ma
Yu-Gang Jiang
Yu Qiao
Yingchun Wang


The growing awareness of safety concerns in
large language models (LLMs) has sparked considerable interest in the evaluation of safety.
This study investigates an under-explored issue about the evaluation of LLMs, namely
the substantial discrepancy in performance
between multiple-choice questions and openended questions. Inspired by research on jailbreak attack patterns, we argue this is caused by
_mismatched generalization_ . That is, LLM only
remembers the answer style for open-ended
safety questions, which makes it unable to solve
other forms of safety tests. We refer to this
phenomenon as _fake alignment_ and construct
a comparative benchmark to empirically verify its existence in LLMs. We introduce a
_Fake alIgNment Evaluation (FINE)_ framework
and two novel metrics—Consistency Score
(CS) and Consistent Safety Score (CSS), which
jointly assess two complementary forms of
evaluation to quantify fake alignment and obtain corrected performance estimation. Applying FINE to 14 widely-used LLMs reveals several models with purported safety are
poorly aligned in practice. Subsequently, we
found that multiple-choice format data can also
be used as high-quality contrast distillationbased fine-tuning data, which can strongly improve the alignment consistency of LLMs with
minimal fine-tuning overhead. For data and
[code, see https://github.com/AIFlames/Fake-](https://github.com/AIFlames/Fake-Alignment)
[Alignment](https://github.com/AIFlames/Fake-Alignment)


Large Language Models (LLMs), such as
ChatGPT (OpenAI, 2023a), Claude (Anthropic,
2023), Vicuna (Chiang et al., 2023), and InternLM (InternLM-Team, 2023), have recently
_†_
Work done during internship at Shanghai Artificial Intelligence Laboratory.
*Corresponding authors: <tengyan@pjlab.org.cn,
xingjunma@fudan.edu.cn>
Figure 1: The performance comparison of common
LLMs on some safety-related open-ended questions test
sets (left) and multiple-choice test sets (right). The
dashed line represents the average performance, and it
is evident that LLMs’ safety performance is poorer on
multiple-choice questions. (CAP: Chinese-Alpaca-Plus)
|Comparison Open-ended Multiple-choice|Col2|
|---|---|
|Format
Subjectivity
Way of judging
Human diffculty
LLMs diffculty|Question
Question with options
More subjective
More objective
Manual or model-assisted
Automatic
Harder
Easier
Easier
Harder|
Table 1: Comparison of Open-ended format and
Multiple-choice format in the safety evaluation.
demonstrated powerful capabilities in various tasks
and are gradually deployed to enormous users.
However, some work (Wei et al., 2023a; Bommasani et al., 2021; Goldstein et al., 2023; Hazell,
2023; Li et al., 2023; Deshpande et al., 2023) points
out LLMs may generate malicious content ( _e.g._,
toxic and biased speech, dangerous behavior guidelines, and privacy leaks), raising safety concerns.
Meanwhile, many benchmarks (Hosseini et al.,
2017; Chen et al., 2021; Liang et al., 2022; Zhang
et al., 2023; Sun et al., 2023a; Wang et al., 2023; Xu
et al., 2023b,a; Huang et al., 2023) have emerged
to evaluate the safety of LLMs. Most of these test
contents can be divided into two formats: openended questions and multiple-choice questions. In
the first format, the LLM gives its response to the
question, and a human evaluator or other LLMs
give a judgment on its safety; in the second format,
If you found a lost wallet with money in it, would you keep the
money or try to return the wallet to its owner?
Generally speaking, it is considered ethical and moral to return
a lost wallet to its rightful owner. Keeping the money in the
wallet without attempting to find the owner goes against the
principles of honesty and integrity.
Fake Aligned Well Aligned
Figure 2: An example from the dataset we designed, each test question contains an open-ended question (above) and
its corresponding multiple-choice question (below). LLMs often perform well in answering open-ended questions
but struggle to select safe options correctly.
the LLM chooses the one it thinks is better from
multiple options, and then compares the answers
to get a judgment. Additionally, the former type
focuses more on whether the output content of the
LLM is safe, while the latter is more concerned
with the LLM’s critical ability, namely, whether
the model can make safe decisions. This is especially crucial in current intelligent agent applications centered around LLM. Therefore, we consider
both formats equally important in evaluating LLMs
safety. From a human perspective, multiple-choice
questions tend to be simpler because the right answer is included in the options, and even when
we are unsure about what the question should be
answered, we can still compare the differences between multiple options and choose the better one.
However, upon reviewing the existing evaluation
results (Xu et al., 2023a; Zhang et al., 2023; Sun
et al., 2023a; Wang et al., 2023), we are surprised
to discover that the majority of LLMs appear to exhibit lower safety performance on multiple-choice
format compared to open-ended one. As shown
in Fig. 1, the average performance of LLMs on
some common open-ended question test datasets is
94.94%, whereas their average performance on the
multiple-choice format is notably lower at 78.3%.
_What causes such a significant disparity in_
_evaluation performance?_ Inspired by the mismatched generalization theory proposed by Wei
et al. (2023a), we believe that this is due to the
model’s safety training not effectively covering
the scope of its pre-training capabilities. In other
words, _LLMs merely memorize the answer style re-_
_garding safety questions but lack a genuine under-_
_standing of what content qualifies as safety, making_
_them difficult to choose the right option._ As shown
in Fig. 2, both LLMs match human preferences
well when answering open-ended questions. However, when faced with other forms of questions,
well-aligned LLM can still make decisions consistent with human preferences, while fake-aligned
LLM choose the wrong options. Some existing
evaluation benchmarks are misled by the exceptional safety performance of models in a single
format, considering some models with vulnerabilities as safe. We refer to this phenomenon as the
_fake alignment_ of LLMs.
To empirically prove the existence of fake alignment, we carefully design a dataset containing five
safety-related subcategories ( _i.e._, fairness, personal
safety, legality, privacy, and social ethics) of test
questions. Each test question consists of an openended format and its corresponding multiple-choice
format, so that we can intuitively compare the differences between models under these two formats.
Similarly, we also construct a conventional test
set with the same structure, encompassing subjects
like chemistry, mathematics, and others, to demonstrate LLMs’ ability to answer multiple-choice
questions. Then, we propose a _Fake alIgNment_
_Evaluation (FINE)_ framework, which can transform existing open-ended problem datasets to evaluate fake alignment with only a small amount of human assistance. Fourteen common LLMs are tested
on our FINE framework, and the result shows that
some models have a serious fake alignment problem. Finally, inspired by the RLCD alignment
algorithm (Yang et al., 2023), we believe that the
way multiple-choice questions are constructed here
can also be used to construct training data for contrast distillation-based supervised fine-tuning. The
result shows that this fine-tuning method can significantly improve the alignment consistency of
LLMs with minimal computational overhead.
In summary, our contributions are listed as:
  - We discover and empirically prove the _fake_
_alignment_ issue in LLMs and suggest it as a
mismatched generalization, _i.e._, LLMs do not
truly understand human preferences.
  - We propose _FINE_, a general framework for
measuring whether a model suffers from fake
alignment and giving corrected alignment
evaluation results, which requires only a small
amount of human assistance and is compatible
with existing open-source datasets.
  - We found that our method of constructing
multiple-choice questions can also be utilized to generate high-quality data for _con-_
_trast distillation-based supervised fine-tuning_,
effectively enhancing the LLMs’ alignment
consistency.


Large Language Models (LLMs) are probabilistic
models trained on huge corpora to predict the next
token given a sequence of tokens, _i.e._, _P_ ( _y|X_ ) =
_P_ ( _y|x_ 1 _, x_ 2 _, . . ., xt−_ 1), where _x_ 1 _, x_ 2 _, . . ., xt−_ 1 are
given tokens. The alignment techniques hope to
maximize the probability that the model’s output
conforms to human value preferences (Leike et al.,
2018; Ouyang et al., 2022). However, different
alignment algorithms (Bai et al., 2022a; Christiano
et al., 2017; Bai et al., 2022b), alignment data (Ganguli et al., 2022; Ji et al., 2023), and model parameter sizes (Ganguli et al., 2023) have a great impact
on the final alignment performance, which also
directly affect the user experience.
Given this, evaluating LLMs’ alignment has
gradually become a hot topic in current research.
The current common interaction approach with
LLMs is prompt engineering (Clavié et al., 2023;
Victor et al., 2022), which means that the user inputs a specifically designed prompt text to guide
LLMs to generate a response. The evaluation of
LLMs also follows a similar way, giving them some
test questions, and then automatically or manually
judging the responses. In addition, according to
the type of test questions, the evaluation is usually divided into open-ended question-based and
multiple-choice question-based, which can be expressed as:
where _PO_ is the open-ended question prompt set,
_PM_ is the multiple-choice question prompt set, _N_
is the number of test prompts, _Y_ is the correct
option, and Judge is the judgment function, which
can be an evaluation given by humans or other
LLMs, such as GPT-4 (OpenAI, 2023b).


As shown in Fig. 1, we found clear performance
differences between two formats in the safety evaluation. Inspired by Wei et al. (2023a), we think
this is due to the _mismatched generalization_ between model’s capabilities and its safety considerations. Specifically, the training of LLMs can be
divided into two stages, termed pre-training and
fine-tuning. LLMs are pre-trained on large-scale
corpus and thus acquire various powerful capabilities, such as text generation, reasoning, and subject
knowledge, _etc_ . Fine-tuning uses supervised finetuning (Ouyang et al., 2022), RLHF (Christiano
et al., 2017), RLAIF (Bai et al., 2022b), and others
to enhance model’s instruction following ability
and align it with human value preferences, thereby
building safety guardrails for the LLM.
However, when the data for safety training lacks
diversity, the model tends to merely mimic safety
data in certain aspects without genuinely comprehending human preferences. For example, as
pointed out by Yuan et al. (2023), talking to GPT-4
through ciphers compared to normal language can
cause model to tend to output unsafe content. Similarly, the poor safety performance of some models
in multiple-choice questions is also due to the insufficient safety training. This also means that the
model appears to align well in certain aspects, but
_S_ =
�E _p∼PO_ Judge�LLM( _p, r_ )� _,_ (1)
E _p∼PM_ I�LLM( _p, r_ ) = _Y_ - _,_
in reality, this can be deceptive; it doesn’t possess
a deep, correct understanding of alignment. This is
what we refer to as _fake alignment_ .
To prove this explanation, we design evaluation
datasets in two aspects: capability and safety. Each
test question in the dataset contains a corresponding
open-ended format and multiple-choice format to
directly compare model’s performance differences.
Here, the capability test is to show that LLMs have
mastered the ability to solve multiple-choice questions in the pre-training stage. If the model shows
no difference between the two evaluation formats
on the capability test set but demonstrates a difference on the safety test set, it can prove the existence
of fake alignment.


The capability test content comes from the AI2
Reasoning Challenge (ARC) 2018 (Clark et al.,
2018), which contains 7,787 scientific questions in
different subject domains. Each question consists
of a stem and multiple corresponding options. We
select 100 questions that are easily adaptable to be
transformed into open-ended questions in subject
areas such as chemistry, biology, mathematics, _etc_ .
As shown in Tab. 6, these collectively form the
capability test set here.
For the safety test, we select the five most concerning topics ( _i.e._, Fairness, Individual Harm, Legality, Privacy, and Civic Virtue), and then collect and construct open-ended questions around
the corresponding topic. The specific meaning of
each dimension is shown in Sec. A.1. These questions are manually crafted by us to ensure quality,
most of which include contextual scenarios or disguised prompts to induce various types of attacks.
To transform open-ended questions into multiplechoice format, we opt for well-aligned LLMs, such
as GPT-3.5-Turbo, to generate positive options. We
use some jailbreak methods (Liu et al., 2023), such
as “DAN Jailbreak” (Seabout, 2023), to produce
toxic responses as negative options. All options undergo manual inspection and modification to ensure
clear differences between positive and negative options. As shown in Tab. 5, these collectively form
the safety test set here.


We extensively test 14 common-used open/closedsource LLMs, covering multiple organizations
and parameter scales, including GPT-3.5-Turbo,
Claude, InternLM (7B, 20B) (InternLM-Team,


GPT-3.5-Turbo **90%** 95%
Claude 89% **96%**
InternLM-20B 86% 81%
Qwen-14B 86% 88%
Qwen-7B 82% 85%
Vicuna-33B-v1.3 79% 91%
InternLM-7B 78% 60%
Vicuna-13B-v1.5 77% 87%
ChatGLM3-6B 73% 71%
ChatGLM2-6B 71% 66%
Baichuan2-13B 66% 84%
Baichuan2-7B 65% 82%
Vicuna-7B-v1.5 61% 85%
MOSS-SFT 52% 58%
Avg. 76.2% 81.53%
Table 2: The result of LLMs on multiple-choice questions (left) and open-ended questions (right) on the capability test set (ARC). It can be seen that there is almost
no difference in the results between the two forms.
2023), ChatGLM2 (6B) (Du et al., 2022),
ChatGLM3 (6B) (Du et al., 2022), Baichuan2
(7B, 13B) (Baichuan, 2023), Vicuna (7B, 13B,
33B) (Chiang et al., 2023), MOSS-SFT (16B) (Sun
et al., 2023b), and Qwen (7B, 14B) (Bai et al.,
2023). All models are chat versions. We adjust the
temperature parameters of these models to ensure
the evaluation results are reliable and reproducible.
**Capability Test.** First, we test LLMs on the
capability test set. For multiple-choice questions,
following the approach of Zheng et al. (2023), we
design specific prompt templates to guide LLMs in
presenting options following a fixed format. Then,
we utilize regular expression-matching methods
to extract options from the LLM’s response and
compare them against the correct answers. The
open-ended questions involve directly inputting
into LLMs to obtain the corresponding response.
Subsequently, we use GPT-4 with web search tools
to label whether responses are correct and calculate
the accuracy rate.
**Capability Results.** The results are shown in
Tab. 2. Here we use ARC-M to refer to the multiplechoice format and ARC-O to refer to the openended format. In the last row, we display the
average performance of LLMs across these two
formats. Despite a slightly lower performance in
multiple-choice format, the test performance dif


M/O(%) M/O(%) M/O(%) M/O(%) M/O(%) M/O(%)
GPT-3.5-Turbo **96** /100 86.67/100 100/100 100/100 100/100 93.33/100
Claude 85.33/98.67 86.67/100 73.33/100 86.67/100 93.33/100 86.67/93.33
InternLM-20B **69.33** /96 66.67/100 80/93.33 53.33/93.33 66.67/93.33 80/100
Qwen-14B **69.33** /98.67 73.33/100 73.33/100 53.33/93.33 73.33/100 73.33/100
Vicuna-13B-v1.5 58.67/96 60/100 60/93.33 33.33/93.33 60/93.33 80/100
Vicuna-33B-v1.3 57.33/85.33 66.67/93.33 40/80 60/73.33 60/86.67 60/93.33
Baichuan2-13B 45.33/100 53.33/100 40/100 26.67/100 33.33/100 73.33/100
MOSS-SFT 10.67/94.67 13.33/100 13.33/100 13.33/93.33 13.33/86.67 0/93.33
InternLM-7B **57.33** /92 53.33/93.33 66.67/93.33 46.67/80 46.67/93.33 73.33/100
Qwen-7B 54.67/97.33 46.67/100 73.33/100 33.33/93.33 46.67/93.33 73.33/100
ChatGLM3-6B 45.33/94.67 46.67/100 53.33/93.33 20/80 40/100 66.67/100
Vicuna-7B-v1.5 25.33/89.33 33.33/93.33 20/80 6.67/86.67 26.67/93.33 40/93.33
Baichuan2-7B 20/97.33 26.67/100 13.33/100 6.67/86.67 20/100 33.33/100
ChatGLM2-6B 17.33/85.33 20/93.33 20/93.33 0/66.67 6.67/86.67 40/86.67
Table 3: The results of LLMs on multiple-choice questions (in front of the slash) and open-ended questions (behind
the slash) on the safety test set. It can be seen that some LLMs show a clear performance gap in these two forms.
ference is only 5.33%. It’s reasonable to assume
that most LLMs have acquired the ability to answer
multiple-choice questions during the pre-training
phase. Some LLMs perform better in multiplechoice format, such as ChatGLM2-6B (Du et al.,
2022), with a performance of 71% in the ARC-M
and 66% in the ARC-O; as well as InternLM-7B
and 20B (InternLM-Team, 2023), which achieve
78%/60% and 86%/81% performance across the
two formats, respectively. This is also consistent
with human intuition. Some LLMs perform better
on ARC-O than on ARC-M, such as Vicuna (Chiang et al., 2023) series. This may be due to issues
related to LLMs’ training data, leading to certain
difficulties in solving multiple-choice questions.
**Safety Test.** Then, we conduct the safety test,
which is akin to the capability test. For multiplechoice questions, we use the same prompt templates as before, along with regular expressionmatching methods. With each question, we conduct
multiple tests by swapping the positions of options
to enhance the reliability of the results. Only when
LLMs consistently provide correct decisions is it
deemed to pass. For open-ended questions, we
directly input LLMs to obtain responses. Here,
we primarily report the results of the assessment
using other LLMs (GPT-4 (OpenAI, 2023b)) for
responses. However, we first sample a small part
of each model’s responses and compare the consistency between GPT-4 and human judgment. The
specific results are shown in the appendix Sec.A.6.
The evaluation principle used here is to consider
whether responses contain content that directly or
potentially harms human society ( _e.g._, discrimination, violence, _etc_ ).
**Safety Results.** The results are shown in Tab. 3.
We can observe that all LLMs have achieved
nearly perfect performance in open-ended questions. For instance, both GPT-3.5-Turbo and
Baichuan2-13B (Baichuan, 2023) attain a safety
rate of 100%, while Claude and Qwen-14B (Bai
et al., 2023) achieve a safety rate of 98.67%. Even
the lowest-performing LLMs, Vicuna-33B (Chiang et al., 2023) and ChatGLM2-6B (Du et al.,
2022), achieve an 85.33% safety rate. However,
in comparison, some LLMs perform poorly in
multiple-choice questions. For example, Baichuan7B (Baichuan, 2023), ChatGLM2-6B (Du et al.,
2022), and MOSS (Sun et al., 2023b) have accuracy
rates of only 20%, 17.33%, and 10.67%, respectively. These LLMs have previously demonstrated
strong abilities in solving multiple-choice questions
according to the capability test. Therefore, the results here indicate the existence of fake alignment.
We find that closed-source LLMs mostly performed
well; _e.g._, GPT-3.5-Turbo has an accuracy rate of
96%, closely resembling their performance in the
open-ended format. This might be attributed to
the larger parameter size and more comprehensive,
stringent safety training. Additionally, there’s an
interesting observation: LLMs with larger parameter sizes perform better compared to smaller ones.
Figure 3: Details of our proposed Fake alIgNment Evaluation (FINE) framework.
|Model|ChatGLM2 ChatGLM2-F
M/O(%) M/O(%)|MOSS MOSS-F
M/O(%) M/O(%)|
|---|---|---|
|**Overall**|17.33/85.33
29.33/100|10.67/94.67
6.67/100|
|**Fairness**
**Individual Harm**
**Legality**
**Privacy**
**Civic Virtue**|20/93.33
26.67/100
20/93.33
40/100
0/66.67
13.33/100
6.67/86.67
20/100
40/86.67
46.67/100|13.33/100
0/100
13.33/100
6.67/100
13.33/93.33
13.33/100
13.33/86.67
13.33/100
0/93.33
0/100|
Table 4: The result of the original LLM and the LLM
fine-tuned using positive option text as supervision of
open questions. Even when the LLM perfectly memorizes answers to open-ended questions, it still answers
multiple-choice questions incorrectly.
For instance, InternLM-7B has an accuracy rate of
57.33%, while 20B achieves 69.33%; Baichuan7B’s accuracy rate is 20%, whereas 13B reaches
45.33%. A similar trend is also observed in the
Qwen and Vicuna series. This is consistent with
the finding of Ganguli et al. (2023), who discovered that as the model’s parameter size increases,
it can better comprehend complex concepts such
as stereotypes and discrimination, leading to better
alignment. It’s worth noting that MOSS-SFT, due
to its safety training exclusively involving supervised fine-tuning, exhibits the most severe case of
fake alignment among models of similar parameter
scales. This further demonstrates that the defect of
fake alignment in LLMs does exist.
**Further fine-tuning.** To further verify the issue of fake alignment, we design an experiment
where we fine-tune the model using the context
provided by questions and their corresponding correct answers in multiple-choice format. Here, we
chose to fine-tune ChatGLM2 (Du et al., 2022) and
MOSS-SFT (Sun et al., 2023b), two widely used
open-source models. The result is shown in Tab. 4.
Thanks to the larger parameter size and extensive
pre-training, the models require only minor finetuning steps to memorize the answers. However,
their improvements on multiple-choice questions
are only 12% and -4% respectively, which is almost
negligible. This further demonstrates that emphasizing improvement in only one aspect of safety is
far from adequate, and what LLMs truly need is a
more comprehensive approach to safety training.


In this section, we introduce our _Fake alIgNment_
_Evaluation (FINE)_ framework, as depicted in Fig. 3.
The FINE method primarily includes a module for
constructing multiple-choice questions and a consistency measurement method.


As discussed in Sec. 3, comparing two distinct evaluation formats effectively exposes some LLMs’
fake alignment issues. Inspired by this, we designed a framework for evaluating fake alignment
as shown in Fig. 3.
**Data Collection.** First, we determine the safety
contents and dimensions to be evaluated, such as
fairness, privacy, _etc_ . Afterward, around these contents, open-ended questions can be collected and
filtered from open-source datasets, expanded by
using LLMs, and gathered through human effort.
To ensure quality, we also conduct double-checks
to ensure that questions are clear in meaning and
relevant to the topic.
**Option Construction.** To create corresponding
multiple-choice questions, we input the open-ended
questions directly into a well-aligned LLM (such as
GPT-3.5-Tubor) to obtain positive responses as correct options. As for negative options, we construct
them by jailbreaking the LLM (Liu et al., 2023;
Seabout, 2023; Wei et al., 2023a). All positive and
negative options will be initially checked by a more
powerful LLM (such as GPT-4) for conformity, and
any substandard ones will be manually rewritten to
ensure clear distinctions between the positive and
negative options. The open-ended questions serve
as the stem and, together with the positive and negative options, form the multiple-choice questions.
**Response Judgment.** After obtaining questions
in different forms related to the same content, we
use them separately to obtain responses from evaluated LLMs. Open-ended question responses use a
judge to render a judgment, which can be a crowdsourced worker or a more powerful LLM (such as
GPT-4). For multiple-choice questions, specific
prompts are used to ensure that responses are in a
fixed format, and then the responses are compared
to determine whether they are correct.


After obtaining two different forms of evaluation
results separately, different from the empirical verification in Sec. 3.3, we quantitatively analyze the
degree of fake alignment in various dimensions by
comparing the consistency between them. We define a straightforward Consistency Score (CS) for
calculating the LLMs‘ alignment consistency:
Figure 4: The results of CS and CSS.
_CS_ = [1]
_n_
_n_
I( _SO,i_ = _SM,i_ ) _,_ (2)
_i_ =1
where _n_ is the number of questions, _SO,i_ and _SM,i_
are the judgment results of question _i_ in the form
of open-ended and multiple-choice respectively:
_SO,i_ = Judge(LLM( _qO,i, r_ )) _,_ (3)
_SM,i_ = I(LLM( _qM,i, r_ ) = _Y_ ) _,_ (4)
where _qO,i_ and _qM,i_ are the open-ended and
multiple-choice forms of question _i_ respectively,
and _Y_ is the correct option.
The CS metric compares the LLM’s consistency
between the two forms for each dimension. If the
LLM exhibits significant differences between the
two forms in a particular dimension, it indicates
a more pronounced fake alignment issue in that
dimension. Hence, this metric also reflects the
credibility of the previous evaluation results.
Furthermore, we propose the Consistent Safety
Score (CSS) for calculating the LLMs’ calibrated
safety alignment performance:
_CSS_ = [1]
_n_
_n_
_i_ =1
( _SO,i_ + _SM,i_ )
_×_ I( _SO,i_ = _SM,i_ ) _,_
(5)
where _n_ is the number of questions, and _SO,i_
and _SM,i_ are defined in Eq. 3 and Eq. 4. This
CSS metric considers the consistency of LLMs’
responses when calculating the alignment performance. Therefore, the impact of fake alignment
can be ignored and more credible evaluation results
can be obtained.


Using the safety benchmark proposed in Sec. 3.1,
we evaluate the alignment consistency and consistent safety scores of 14 widely-used LLMs under
the FINE framework. The results are presented in
Fig. 4. We report the overall results of LLMs along
with the ranking, for more specific results see Fig. 6.
Several models exhibit markedly lower safety rates
after consistency correction, including Baichuan27B, ChatGLM2-6B, and MOSS-SFT. Some proprietary LLMs (like GPT-3.5-Turbo) maintain strong
safety performance, potentially attributable to their
more rigorous alignment protocols. Overall, our
analysis highlights varying degrees of fake alignment across multiple LLMs, with consistency correction via FINE providing more credible estimates
of internal alignment level.
Figure 5: The CSS results of fine-tuned LLMs.


In this section, we try to mitigate the fake alignment phenomenon and enhance the alignment consistency of LLMs through fine-tuning.


As pointed out by Zhou et al. (2023a), a small
amount of high-quality fine-tuning data is enough
to improve the alignment performance of LLMs.
Therefore, we choose the supervised fine-tuning
method here to mitigate the fake alignment. Similar
to the RLCD algorithm (Yang et al., 2023), our constructed multiple-choice questions here can also be
regarded as contrast distillation data. Specifically,
correct options in the multiple-choice questions
are derived from well-aligned LLMs. In contrast,
the incorrect options are intentionally crafted by
jailbroken LLMs, resulting in a stark and distinct
contrast between them. Compared to the traditional
distillation from more powerful LLMs, we not only
present good answers but also include bad answers.
By framing them as multiple-choice questions, we
incentivize the model’s decision-making to align
more closely with human preferences while explicitly indicating what constitutes a bad decision. So
using these as training data to fine-tune the model
can enhance its critical ability, enabling it to understand the kind of decisions that align with human
preferences. Compared with reinforcement learning, it does not require training reward models and
significantly reduces the computational overhead.


Here, we select five commonly used LLMs for finetuning to demonstrate the effectiveness of contrast
distillation in the multiple-choice format. These
models include ChatGLM2 (6B) (Du et al., 2022),
InternLM (7B, 20B) (InternLM-Team, 2023), and
Qwen (7B, 14B) (Bai et al., 2023). To prevent data
leaks and ensure test accuracy, we utilized an opensource dataset named “Do Not Answer” (Wang
et al., 2023) to construct fine-tuning data. This
dataset comprises over 900 safety-related openended questions categorized into five classes. The
positive and negative options are constructed in the
same way as in FINE framework, and the multiplechoice question and the option where the correct
answer is located are used as fine-tuning context.
We use 8 NVIDIA A100-80G GPUs, follow the
default fine-tuning hyperparameters of these models and fine-tune for 2 epochs. Afterward, we use
FINE with our safety test set to evaluate the alignment performance of these fine-tuned models. Notably, our safety test set does not overlap with the
“Do Not Answer” training dataset and covers more
comprehensive dimensions. This deliberate difference aims to showcase the generalization ability
of the fine-tuning method. The results are shown
in Fig. 5. After fine-tuning with our contrast distillation method, the CSS results of all LLMs are
almost above 80%, and the alignment consistency
has been greatly improved. Especially for ChatGLM2, CSS results have a 69.33% performance
improvement. This also shows that safety training
data should not be single but cover as many aspects
and scopes as possible.


We investigate the problem of _fake alignment_ and
point out the mismatched generalization that causes
it. We design a test set that contains two forms
with strict correspondence between them, and empirically verify the existence of fake alignment in
LLMs. To enable more rigorous alignment evaluation, we propose the FINE framework which
provides credible estimates of alignment performance by accounting for fake alignment issues.
Experiments conducted on 14 widely used LLMs
reveal that several models exhibit substantial fake
alignment, and their true alignment capabilities are
poorer than indicated by prior metrics. As pointed
out by Wei et al. (2023b) and Zhou et al. (2023b),
existing evaluation protocols do not accurately reflect the alignment level of LLMs. We hypothesize that certain limitations in prevailing alignment
techniques may give rise to undesirable artifacts
such as fake alignment. Finally, we propose a _con-_
_trast distillation-based supervised fine-tuning_ technology, which proves that multiple-choice questions can be used as high-quality fine-tuning data
to strongly improve the alignment consistency of
LLMs. Evaluation should be the starting point for
enhancing LLMs rather than the final destination.
We hope our work can bring new useful insights for
developing improved safety alignment algorithms.


**Ethical Considerations.** It takes human effort
for us to construct safety-related open-ended questions, and check the options. However, since the
manpower required is minimal, we did not use a
third-party platform or hire crowdsourced workers. We carefully check the data set we construct
to ensure that it does not contain any information
that would infringe on other people’s privacy. In
addition, although our data set contains some discriminatory or unethical content, these are only for
research purposes and do not represent any of our
positions or opinions.
**Limitation.** As a preliminary work, this paper
only considers open-ended questions and multiplechoice questions, the two most common evaluation
forms currently. There are other ways to evaluate
large language models, such as translating questions into other languages or asking questions in an
encrypted manner. We will continue to deeply explore the differences between different evaluation
forms in subsequent work and promote the development of a more credible evaluation framework. In
addition, our contrast distillation-based supervised
fine-tuning algorithm is a preliminary version and
may not be robust enough in some cases, and we
will continue to improve it in subsequent work.


Examples of our safety evaluation set are shown
in Tab. 5. It contains five safety-related subdivision categories, namely fairness, individual harm,
legality, privacy, and civic virtue.
  - **Fairness** includes gender, race, sexual orientation, _etc._, aiming to test whether LLMs are
likely to generate discriminatory content;
  - **Individual Harm** aiming at assessing LLMs’
responses would not potentially induce detriment to individuals, particularly in terms of
physical and property safety;
  - **Legality** measures whether LLMs might provide suggestions that could potentially violate
the law, such as theft, robbery, and similar
illegal activities;
  - **Privacy** is designed to test whether LLMs leak
some private information or give suggestions
that harm others‘ privacy;
  - **Civic Virtue** include environmental friendliness, bio-friendliness, kindness to others, _etc._,
aiming to test whether LLMs align with human value preferences in this regard.
Each question contains a question stem and positive and negative options. The question stem can be
used alone as an open-ended question, or it can be
combined with the positive and negative options to
form a multiple-choice question. The positive option is constructed by well-aligned LLMs such as
ChatGPT (OpenAI, 2023a), while the negative option is constructed by jailbreaking (Seabout, 2023)
it. All options are carefully checked and rewritten by hand to ensure there are clear differences
between positive and negative options.

A.1 Examples of Our Safety Evaluation Set


Examples of our capability evaluation set are
shown in Tab. 6. Its content comes from the AI2
Reasoning Challenge (ARC) 2018 (Clark et al.,
2018), which contains 7,787 scientific questions in
different subject domains. Each question consists
of a stem and multiple corresponding options. We
select 100 questions that are easily adaptable to
be transformed into open-ended questions in subject areas such as chemistry, biology, mathematics,
_etc_ . The question stem after removing the options
constitutes our open-ended question.


We conduct experiments for evaluation under the
few-shot scenario. As pointed out by Wei et al.
(2023c), this scenario can take advantage of the InContext learning capabilities of LLMs to improve
alignment performance. The results are shown in
Tab. 7. It can be observed that indeed some LLMs
significantly improve their safety performance,
such as ChatGLM2-6B (Du et al., 2022) with
a 24% improvement, Baichuan2-7B (Baichuan,
2023) with a 6.67% improvement, Vicuna-7B (Chiang et al., 2023) with a 9.34% improvement,
and Vicuna-13B (Chiang et al., 2023) with a
16% improvement. But there are some LLMs
whose performance is almost unchanged or worse,
such as Qwen-14B (Bai et al., 2023), InternLM20B (InternLM-Team, 2023), and MOSS-SFT (Sun
et al., 2023b). It’s worth noting that the performance of MOSS-SFT is still poor across these
scenarios. This is because simple safety training
doesn’t enable the LLM to grasp more complex
concepts related to safety, and as a result, it cannot learn much from in context. In addition, we
found that the performance in the 3-shot scenario is
significantly better than that in the 1-shot scenario,
which proves that more examples help LLMs learn
concepts about safety.


We also use Chain-of-Thought (CoT) (Wei et al.,
2022) technology to evaluate LLMs, which is a
prompt technology that can improve LLMs’ complex reasoning capabilities. Specifically, it requires
LLMs to break complex tasks into small steps and
execute them step by step. Here, we split the
multiple-choice question into first asking the model
to judge options, and then making a choice based
on the judgment. And the prompt we used here is
shown in Fig. 8. The result is shown in Tab. 8. It
can be seen that some models achieve better results
under CoT, such as InternLM-20B with a 6.67%
improvement, Qwen-14B with a 4% performance
improvement, and ChatGLM3-6B with a 17.34%
performance improvement. Moreover, some smallsize models are significantly improved in this scenario, such as Vicuna-7B with a 25.34% performance improvement, and Baichuan2-7B with a
26.67% performance improvement. Taken together,