The element cesium, Cs, is an alkali metal. Which
Chemistry
chemical formula represents a cesium compound that
is likely to exist?
The element cesium, Cs, is an alkali metal. Which
chemical formula represents a cesium compound that
is likely to exist?
(A) CsCl (B) CsCl2 (C) CsO (D) CsO2
... According to the central dogma of molecular biol- ... According to the central dogma of molecular biolBiology
ogy, in which structure does this error originate? ogy, in which structure does this error originate?
(A) DNA (B) mRNA (C) tRNA (D) rRNA
... One worker weighs 180 lbs. and the other weighs
Mathematics
200 lbs. ... What is the resultant maximum live load,
in pounds, that can be expected from the two workers
supported by the scaffold?
... One worker weighs 180 lbs. and the other weighs
200 lbs. ... What is the resultant maximum live load,
in pounds, that can be expected from the two workers
supported by the scaffold?
(A) 380 lbs (B) 475 lbs (C) 625 lbs (D) 950 lbs
Table 6: Examples of the ARC dataset used for capability testing, where each question contains a corresponding
open-ended question (left) and multiple-choice question (right).
|Overall Fairness Individual Harm Legality Privacy Civic Virtue
Model
1-shot 3-shot 1-shot 3-shot 1-shot 3-shot 1-shot 3-shot 1-shot 3-shot 1-shot 3-shot|Col2|Col3|Col4|Col5|Col6|Col7|
|---|---|---|---|---|---|---|
|Model
**Overall**
**Fairness**
**Individual Harm**
**Legality**
**Privacy**
**Civic Virtue**
1-shot
3-shot
1-shot
3-shot
1-shot
3-shot
1-shot
3-shot
1-shot
3-shot
1-shot
3-shot|1-shot
3-shot|1-shot
3-shot|1-shot
3-shot|1-shot
3-shot|1-shot
3-shot|1-shot
3-shot|
|Claude
GPT-3.5-Turbo|**88%**
86.67%
**88%**
90.67%|66.67%
73.33%
73.33%
80%|93.33%
93.33%
100%
100%|93.33%
73.33%
86.67%
93.33%|100%
100%
100%
100%|86.67%
93.33%
80%
80%|
|Vicuna-13B-v1.5
Baichuan2-13B
Vicuna-33B-v1.3
Qwen-14B
InternLM-20B
MOSS-SFT|**74.67%**
77.33%
62.67%
58.67%
60%
73.33%
58.67%
61.33%
54.67%
58.66%
5.33%
6.67%|73.33%
66.67%
53.33%
53.33%
33.33%
46.67%
46.67%
53.33%
33.33%
46.67%
0%
0%|80%
93.33%
60%
66.67%
53.33%
86.67%
80%
80%
73.33%
66.67%
6.67%
6.67%|53.33%
66.67%
60%
60%
73.33%
80%
40%
40%
40%
46.67%
0%
0%|86.67%
80%
66.67%
60%
73.33%
86.67%
60%
66.67%
60%
66.67%
0%
6.67%|80%
80%
73.33%
53.33%
66.67%
66.67%
66.67%
66.67%
66.67%
66.67%
20%
20%|
|InternLM-7B
ChatGLM3-6B
Qwen-7B
ChatGLM2-6B
Vicuna-7B-v1.5
Baichuan2-7B|**46.67%**
53.33%
42.67%
49.33%
41.33%
57.33%
41.33%
46.67%
34.67%
37.33%
26.67%
25.33%|33.33%
53.33%
33.33%
40%
40%
53.33%
33.33%
46.67%
26.67%
26.67%
20%
26.67%|53.33%
73.33%
60%
86.67%
40%
73.33%
66.67%
53.33%
33.33%
60%
20%
33.33%|26.67%
20%
20%
20%
20%
40%
20%
26.67%
26.67%
26.67%
13.33%
13.33%|46.67%
53.33%
33.33%
33.33%
46.67%
46.67%
33.33%
46.67%
26.67%
33.33%
33.33%
26.67%|73.33%
66.67%
66.67%
66.67%
60%
73.33%
53.33%
60%
60%
40%
46.67%
26.67%|
Table 7: The few-shot results of LLMs on multiple-choice questions on the safety test set.

Major ARC (Open-ended Question) ARC (Multiple-choice Question)


Claude **93.33%** 86.67% 100% 93.33% 100% 86.67%
GPT-3.5-Turbo 84% 86.67% 86.67% 73.33% 93.33% 80%
InternLM-20B **76%** 80% 80% 73.33% 73.33% 73.33%
Qwen-14B 73.33% 66.67% 73.33% 73.33% 73.33% 80%
Vicuna-13B-v1.5 66.67% 66.67% 66.67% 53.33% 73.33% 73.33%
Vicuna-33B-v1.3 60% 46.67% 66.67% 60% 66.67% 60%
Baichuan2-13B 60% 46.67% 60% 53.33% 66.67% 73.33%
MOSS-SFT 9.33% 20% 6.67% 0% 0% 20%
ChatGLM3-6B **62.67%** 60% 53.33% 53.33% 66.67% 80%
Qwen-7B 52% 33.33% 46.67% 33.33% 66.67% 80%
Vicuna-7B-v1.5 50.67% 40% 46.67% 46.67% 53.33% 66.67%
InternLM-7B 49.33% 40% 53.33% 33.33% 53.33% 66.67%
Baichuan2-7B 46.67% 53.33% 46.67% 26.67% 53.33% 53.33%
ChatGLM2-6B 26.67% 26.67% 26.67% 6.67% 26.67% 46.67%
Table 8: The results of LLMs on multiple-choice questions with CoT.
Figure 6: The results of CS and CSS. (Darker colors
represent better performance)
CoT can indeed improve LLMs’ performance in
multiple-choice scenarios to a certain extent and
mitigate the fake alignment problem, but it cannot
completely solve this problem.

Fake Alignment: Are LLMs Really Aligned Well?

Model Overall Fairness Individual Harm Legality Privacy Civic Virtue

Yixu Wang
Yan Teng
Kexin Huang
Chengqi Lyu
Songyang Zhang
Wenwei Zhang
Xingjun Ma
Yu-Gang Jiang
Yu Qiao
Yingchun Wang


In Sec. 4.3, we report the overall evaluation results
and rankings of 14 LLMs under the FINE framework, and here we give more fine-grained results.
As shown in Fig. 6, we report the alignment performance of models under each subcategory using
heat maps, with darker colors representing better
performance. It can be seen that most LLMs perform better in the individual harm and social ethics
dimensions, but perform slightly worse in the fairness dimension, which may be attributed to the bias
in the safety training data.


Here, we verify the effectiveness of using GPT-4
as a judge through experimental comparison. We
randomly select a part of each model’s responses
to the question and then use humans and GPT-4
to judge this part of the responses. The results are
shown in the Tab.9. It can be seen that the average
consistency between the two evaluation methods
reaches more than 90%, so GPT-4 can be used as
the main evaluation method, thereby significantly
reducing manpower requirements.
Figure 7: Multiple-choice question results of fine-tuned
LLMs under two different prompts.


Claude 93.33%
GPT-3.5-Turbo 98.67%
InternLM-20B 89.33%
Qwen-14B 92%
Vicuna-13B-v1.5 92%
Vicuna-33B-v1.3 85.33%
Baichuan2-13B 93.33%
MOSS-SFT 89.33%
ChatGLM3-6B 93.33%
Qwen-7B 92%
Vicuna-7B-v1.5 88%
InternLM-7B 90.67%
Baichuan2-7B 93.33%
ChatGLM2-6B 94.67%
Avg. 91.81%
Table 9: The results of comparing the consistency of
human and GPT-4 judgment.


Here, we show in Fig. 8 some of the prompts
we use when evaluating and fine-tuning. Among
them, we design the Default prompt with reference to the prompt in MT-Bench (Zheng et al.,
2023). It is the most commonly used prompt in
this paper to test the performance of LLMs under
multiple-choice questions and construct the contrast distillation-based fine-tuning data. The Chainof-Thought prompt is a prompt we use to test the
LLMs’ multiple-choice question performance in a
chain-of-thought scenario. We design it in the form
of multi-round dialogue, first asking the LLM to
give a judgment on each option, and then choosing
the one it thinks is correct based on the judgment.
The last Simple prompt is the simplest. It is used
in some previous benchmarks (Zhang et al., 2023).
However, because it is too simple, the test results
may not be robust enough. Here we only use it
to test LLMs fine-tuned by contrast distillation to
prove that our fine-tuning method has strong generalization ability.


In Sec. 5, we propose our contrast distillation-based
fine-tuning method and evaluate the performance of
fine-tuned LLMs under the FINE framework. Here,
to further demonstrate the generalization ability of
our fine-tuning method, we test the multiple-choice
question performance when using a test prompt
( _i.e._, the Simple prompt) that is completely different from the prompt during fine-tuning ( _i.e._, the
Default prompt). The specific contents of the Default prompt and Simple prompt are shown in Fig. 8.
The results are shown in Fig. 7. It can be seen that
even using Simple prompt that does not give LLMs
explicit guidance, LLMs are still able to select options that are in line with human preferences.
Figure 8: The examples of our used prompts.