**Xiao Liu** [1] _[,]_ [2] _[,][∗]_ **, Xuanyu Lei** [1] _[,]_ [2] _[,][∗][,][†]_ **, Shengyuan Wang** [1] _[,]_ [2] _[,][†]_ **, Yue Huang** [5] _[,]_ [2] _[,][†]_ **,**
**Zhuoer Feng** [4] _[,]_ [2] _[,][†]_ **, Bosi Wen** [4] _[,]_ [2] _[,][†]_ **, Jiale Cheng** [4] _[,]_ [2] _[,][†]_ **, Pei Ke** [4] **, Yifan Xu** [1] _[,]_ [2] **,**

AlignBench: Benchmarking Chinese Alignment of Large Language Models

Xiao Liu
Xuanyu Lei
Shengyuan Wang
Yue Huang
Zhuoer Feng
Bosi Wen
Jiale Cheng
Pei Ke
Yifan Xu
Weng Lam Tam
Xiaohan Zhang
Lichao Sun
Hongning Wang
Jing Zhang
Minlie Huang
Yuxiao Dong
Jie Tang


**Hongning Wang** [4] **, Jing Zhang** [3] **, Minlie Huang** [4] **, Yuxiao Dong** [1] _[,][‡]_ **, Jie Tang** [1] _[,][‡]_
1The Knowledge Engineering Group (KEG), Tsinghua University; 2Zhipu AI;
3Renmin University of China; 4The CoAI Group, Tsinghua University;
5University of Notre Dame; 6Lehigh University

Weng Lam Tam [2] , Xiaohan Zhang [2] , Lichao Sun [6] , Xiaotao Gu [2] ,


Alignment has become a critical step for
instruction-tuned Large Language Models
(LLMs) to become helpful assistants. However,
the effective evaluation of alignment for emerging Chinese LLMs is still largely unexplored.
To fill in this gap, we introduce ALIGNBENCH,
a comprehensive multi-dimensional benchmark
for evaluating LLMs’ alignment in Chinese.
We design a human-in-the-loop data curation
pipeline, containing eight main categories, 683
real-scenario rooted queries and corresponding human verified references. To ensure
the correctness of references, each knowledgeintensive query is accompanied with evidences
collected from reliable web sources (including URLs and quotations) by our annotators.
For automatic evaluation, our benchmark employs a rule-calibrated multi-dimensional LLMas-Judge (Zheng et al., 2023) approach with
Chain-of-Thought to generate explanations and
final ratings, ensuring high reliability and interpretability. All evaluation code, data, and
[LLM generations are available at https://](https://github.com/THUDM/AlignBench)
[github.com/THUDM/AlignBench. Since its re-](https://github.com/THUDM/AlignBench)
lease, AlignBench has been adopted by top
(Chinese) LLMs for evaluating their alignment
capabilities in Chinese, including ChatGLM,
Qwen, DeepSeek, Yi, Baichuan, and Abab.


Large Language Models (LLMs) (Brown et al.,
2020; Chowdhery et al., 2022; Zhang et al., 2022;
Zeng et al., 2022; Touvron et al., 2023) have experienced a surge in development thanks to popular
products such as ChatGPT (OpenAI, 2022). During the period, alignment (Ouyang et al., 2022;
Bai et al., 2022), including supervised fine-tuning
(SFT), reinforcement learning from human feedback (RLHF), and related techniques, has been
_∗_ Xiao and Xuanyu made equal contributions.
_†_ Work done while these authors interned at Zhipu AI.
_‡_ Corresponding Authors: Yuxiao and Jie.
justified as a key strategy to endow pre-trained
LLMs (which can hardly follow instructions) with
strong grasping of human intentions and preferences. After training, aligned LLMs have not
only mastered a wide array of established NLP
tasks (Wang et al., 2019; Liang et al., 2022) but
also versatile language-grounded missions (Cobbe
et al., 2021; Chen et al., 2021; Liu et al., 2023a).
As a result, LLMs have paced a firm step towards
practical applications in the wild.
Meanwhile, reliably benchmarking the broad
and strong competence of LLMs has also become
a significant challenge. In English, there have been
MMLU (Hendrycks et al., 2021), Big-Bench (Srivastava et al., 2023), and HELM (Wang et al.,
2019; Liang et al., 2022); in Chinese, there are
C-Eval (Huang et al., 2023b) and CMMLU (Zeng,
2023). However, prior arts hardly examine aligned
LLMs’ fulfillment of user intention and human
preference in real-world conditions, and even fall
short to tell the difference between aligned and
base LLMs. Consequently, dedicated benchmarks
are crucial for development and meaningful comparisons of aligned LLMs.
Nevertheless, designing a comprehensive and
reliable benchmark for LLM alignment is nontrival. An alignment benchmark should meet several important requirements, which correspond to
the unique strengths of LLMs and their applications
for users:
 - **Real-World Scenarios** : Query forms and topics should be diverse and derived from real scenarios to reflect the authentic usages of LLMs.
 - **Open-Ended** : As aligned LLMs usually produce long open-ended replies, the benchmark
should judge the correctness of detailed responses without specified forms.
 - **Challenging** : LLMs are improving so rapidly
on various aspects beyond estimation. The
benchmark thus has to ensure its difficulty to
identify subtle capability gaps between LLMs.
|1. Data Curation|2. Task Taxonomy|3. LLM Evaluation|
|---|---|---|
||||
Figure 1: Overall framework of ALIGNBENCH. 1) Data Curation: a human-in-the-loop pipeline to allow continual
high-quality test query harvesting from real scenarios. 2) Task Taxonomy: 8 main categories that cover the common
usages of LLMs in Chinese. 3) LLM Evaluation: automatic multi-dimensional rule-calibrated LLM-as-Judge.
 - **Automatic Judging** : Benchmark construction
and evaluation should be as automatic as possible to provide scalable, reproducible, and intime feedback to facilitate LLM development.
There have been recent attempts to introduce
LLM-as-Judge (Li et al., 2023; Zheng et al., 2023)
for evaluating the general alignment of LLMs. AlpacaEval (Li et al., 2023) compares target LLM’s
replies against text-davinci-003’s, but has been
shown unstable and uninterpretable due to its direct and pairwise scoring. MT-Bench (Zheng et al.,
2023) harnesses point-wise scoring with Chain-ofThought (CoT) (Wei et al., 2022) explanations for
better accuracy and transparency. However, it employs only 80 test samples and a scoring prompt
that judges queries of different tasks and domains
uniformly. Lastly, both benchmarks are designed
only in English and cannot well reflect the level of
alignment of many emerging Chinese LLMs.
In light of all mentioned issues, in this work
we present ALIGNBENCH, a comprehensive multidimensional benchmark for evaluating LLMs’
alignment in Chinese. Based on observations from
an online LLM service (anonymous due to review
policy), we set up a semi-automatic data curation pipeline with human in the loop to create
high-quality queries to construct ALIGNBENCH.
ALIGNBENCH summarizes a taxonomy comprising 8 major categories of queries (Cf. Figure 1) to
comprehensively cover and align with real-scenario
usages. In order to make the judge model generate objective and fair evaluations, each sample is accompanied with a human-corrected LLMgenerated reference. To ensure reference correctness for knowledge intensive queries (which accounts for 66.5%), we ask annotators to search
on the web, provide evidences including urls and
quotations, and finally synthesize references.
To enhance the automation of the evaluation,
similar to MT-Bench (Zheng et al., 2023), ALIGNBENCH distinctly leverages GPT-4 (OpenAI, 2023)
as the major model evaluator in its development,
which serves to discern the data samples and evaluate by referenced point-wise scoring with CoT. Differently, ALIGNBENCH further highlights strategies of rules-calibration and task-specific multidimensional judgement in the scoring. Our experiments demonstrate that these strategies contribute
to ALIGNBENCH’s better consistency with human
judgement and better explanation quality. Based on
ALIGNBENCH, we evaluate 17 popular API-based
or open-sourced LLMs that support Chinese, providing detailed comparisons of these LLMs across
fine-grained capabilities on Chinese alignment.
In summary, the contributions of our work are:
 - We construct ALIGNBENCH, a systematic
benchmark rooted in real-scenario usages to
evaluate Chinese alignment of LLMs. We also
tailor a human-in-the-loop pipeline to allow accurate and sustainable benchmark maintenance.
 - Targeting accurate and automatic evaluation
of LLMs, we design a rule-calibrated multidimensional point-wise LLM-as-judge method
for grading. Human evaluations justified its applicability compared to existing LLM-as-Judge
methods (Zheng et al., 2023).
 - We systematically benchmark 17 LLMs’ Chinese alignment on ALIGNBENCH. On top of
their performance, we provide deep insights
into status quo of Chinese LLMs’ development
and highlight future directions.


In this section, we introduce the data composition
and construction pipeline of ALIGNBENCH.
Dataset Information Evaluation Method
Benchmark
Open MultiData Size Language Data Source Domain Metric
-ended Dimensional
MMLU (Hendrycks et al., 2021) 15,908 English Exams & Textbooks Knowledge Accuracy
GSM8k (Cobbe et al., 2021) 8,000 English Human Writers Math ✓ Accuracy
HumanEval (Chen et al., 2021) 164 Python Human Writers Code ✓ Pass@k
CMMLU (Zeng, 2023) 11,528 Chinese Exams & Textbooks Knowledge Accuracy
AGI-Eval (Zhong et al., 2023) 8,062 Chi. & Eng. Exams Knowledge Accuracy
C-Eval (Huang et al., 2023b) 13,948 Chinese Exams Knowledge Accuracy
AlpacaEval (Li et al., 2023) 805 English Alpaca Data General ✓ Model Judge (w/o CoT)
MT-Bench (Zheng et al., 2023) 80 English Self-constructed General ✓ Model Judge (w/ CoT)
**ALIGNBENCH (ours)** **683** **Chinese** **Real-Scenario Usage** **General** ✓ ✓ **Model Judge (w/ CoT)**
Table 1: Comparisons between ALIGNBENCH and other benchmarks, illustrating the features of ALIGNBENCH in
terms of dataset information and evaluation methods.
Category `中文名` #Samples
Fundamental Language Ability `基本任务` 68
Advanced Chinese Understanding `中文理解` 58
Open-ended Questions `综合问答` 38
Writing Ability `文本写作` 75
Logical Reasoning `逻辑推理` 92
Mathematics `数学计算` 112
Task-oriented Role Play `角色扮演` 116
Professional Knowledge `专业能力` 124
Total `总计` 683
Table 2: Sample distribution of ALIGNBENCH dataset.


In this section, we introduce the overall composition of ALIGNBENCH. To perform a systematic
evaluation, we frame a comprehensive taxonomy
of the LLMs’ abilities based on real-user instructions. We inspect and summarize these use-cases
into 8 main categories and 683 samples in total as
shown in Table 2.
**Fundamental Language Ability.** This category
focuses on the basic language tasks derived from
traditional NLP tasks such as information extraction (Etzioni et al., 2008), text classification (Wang
and Manning, 2012), and commonsense knowledge (Talmor et al., 2019). They reflect common
users’ practical needs of LLMs to conduct traditional tasks under zero-shot or few-shot settings
with customized prompts and formats, such as text
classification, information extraction, and short
summarization. Thus we select high-quality diverse queries relevant to each traditional task in
this category.
**Advanced Chinese Understanding.** This category
aims to evaluate LLMs’ ability to understand cultural and historical background in Chinese-specific
tasks. In Chinese context, a large percentage of
real-user needs are related to Chinese culture, characters, and history. However, without deliberate
optimization on Chinese, cutting-edge LLMs (e.g.,
GPT-4 (OpenAI, 2023)) would fail to understand
and answer these questions correctly. As ALIGNBENCH targets Chinese alignment, the category
plays a vital role in our overall design.
**Open-Ended Questions.** The category represents
a common usage of LLMs to answer subjective
questions in an open-ended manner. Users may
seek for advice, recommendations, and tutoring for
many daily questions concerning their work, study,
travel, and lives. The key to good open-ended responses is about catering to human preference, featuring long, detailed, and highly related content.
**Writing Ability.** Writing, one of the most frequently used LLM function, plays a vital role in
LLMs’ applications. We select challenging writing instructions, which require not only an excellent mastering of language but also a high level of
instruction-following (e.g., specific formatting conditions), consistency (e.g., argumentative topics),
and creativity (e.g., fictions or poems).
**Logical Reasoning.** The ability to process complicated problems with step-by-step reasoning and
LLMs’ inherent knowledge is highlighted for current strong LLMs. The category aims to evaluate LLMs’ abilities to understand, analyze, and
produce correct responses given intricate logical
problems, using questions that require deductive,
abductive, multi-hop, or commonsense reasoning.
**Mathematics.** Math problems are challenging but
also widely-adopted for LLMs’ ability testing due
to practical applications. We collect math problems
in different difficulty levels from elementary to advanced mathematics and in different formats, including calculations, simple problem solving, concept explanation, theorem-proof, etc.
**Task-Oriented Role Play.** Lots of users request
the model to play as a specific identity to perform
corresponding tasks, which is summarized as taskoriented role play. In order to evaluate the ful
fillment of users’ instructions and the quality of
responses when role play, we collect role play instructions of high complexity for the category.
**Professional Knowledge.** LLMs have proven their
competence in solving domain-specific problems
that require professional knowledge. The category
aims to evaluate LLMs’ abilities in specific domains, including physics, history, music, law, etc.
Additionally, the queries we select are generative
open-ended questions that allow LLMs to generate
freely and provide sufficient details.


Each sample in ALIGNBENCH contains a taskoriented query, a high-quality referenced answer,
and its category in our taxonomy. The detailed
construction pipeline is described as follows.
**Query Curation.** To ensure the diversity and authenticity of the queries, we mainly refer to two
sources, namely the scenarios from an online chat
service and challenging problems written by researchers. Given the inherently noisy nature, we
go through a high-standard data curation pipeline
following rules described below. **1) Task-oriented:**
The query should represent human intentions and
instruct LLMs to complete the specified task. **2)**
**Clarity & Fluency:** The query should be clear,
and easy-to-understand and the demands should
be smoothly expressed. **3) Complexity & Diffi-**
**culty:** The query should be hard for most LLMs,
requiring them to utilize their capabilities to solve
it comprehensively. **4) Desensitization:** Ensure
that the queries are safe and insensitive.
**Reference Acquisition & Improvement.** For
point-wise grading for ALIGNBENCH (Cf. Section 3 for analysis), scoring with a pivotal reference answer has been found beneficial to improve
the reliability of the LLM-as-Judge (Zheng et al.,
2023; Zhang et al., 2020). Thus we decide to provide human-curated reference answers, serving to
assist evaluators in determining the correctness of
the answer, and act as a scoring pivot.
However, because ALIGNBENCH has been designed to be difficult and of wide coverage, it turns
out quite challenging for human annotators to provide answers from scratch in our preliminary trial.
As a result, we first utilize GPT-4 to generate answers, and then ask human annotators to meticulously review, revise, and refine them as reference
answers for ALIGNBENCH. To ensure reference
quality, especially for knowledge-intensive queries
Figure 2: Cumulative distribution of judging by human,
general (Zheng et al., 2023) and rule-calibrated on sampled ALIGNBENCH along their ratings.
```
     高音单簧管和高音萨克斯的调性相同吗？如果相同，请说出他们的调性，如果
```
Question `不同，请分别说出他们的调性`
Evidence 1 [url] [https://baike.baidu.com/item/%E5%8D%95%E7%B0%A7%E7%AE%A1/346415](https://baike.baidu.com/item/%E5%8D%95%E7%B0%A7%E7%AE%A1/346415)
quote `常见的单簧管分为` bB `调、` A `调和` C `调，这三种都属于高音单簧管。`
Evidence 2 [url] [https://baike.baidu.com/item/%E8%90%A8%E5%85%8B%E6%96%AF/147180](https://baike.baidu.com/item/%E8%90%A8%E5%85%8B%E6%96%AF/147180)
quote `乐器本调` :bB( `高音，次中音` )
`高音单簧管和高音萨克斯的调性不同。高音单簧管的调性通常为` B _♭_ `调、` A `调`
Reference `和` C `调这三种，而高音萨克斯的调性则通常为` B _♭_ `。`
Table 3: An example case from ALIGNBENCH’s knowledge intensive query annotation.
from categories such as Professional Knowledge,
Mathematics, and Logical Reasoning, we explicitly
ask annotators to conduct web search during the
verification. During the search, webpage urls and
quotations of contents for writing the references
are required to be recorded, as shown in Table 3.
**Filtering & Classification.** To enhance distinguishment in scores between strong LLMs, it is
necessary to filter more challenging samples for
evaluation. Consequently, we engage three relatively advanced Chinese-supported LLMs, including GPT-3.5-turbo (OpenAI, 2022), ChatGLM (Du
et al., 2022; Zeng et al., 2022) APIs and Sparkdesk
to serve as difficulty filters within our construction procedure. We subject these models to evaluation, analyzing their responses to the processed
queries and subsequently scoring the answers utilizing GPT-4. By computing the average score across
responses and utilizing it as a signal, we discard
50% of the queries that garnered the highest average scores, indicative of their lower difficulty levels.
The approach ensures a meticulous and discerning
selection of samples, effectively distinguishing between strong LLMs of varying capacities.


To effectively evaluate the quality of responses,
ALIGNBENCH employs GPT-4 (OpenAI, 2023)
as a major evaluator to analyze and subsequently grade the responses following adopted prac
[https://xinghuo.xfyun.cn/](https://xinghuo.xfyun.cn/)
tices (Zheng et al., 2023; Li et al., 2023; Liu et al.,
2023b). However, a significant designing space
still exists regarding prompting, score calibration,
critique explainability, and evaluation dimensions,
which have been hardly explored.
Therefore, in ALIGNBENCH we design a rulecalibrated multi-dimensional point-wise LLM-asJudge method. The detailed prompts are in Appendix A.2 and an example is displayed in Fig 3.


**et al., 2022).** When LLM-as-Judge is leveraged,
two grading methods have been previously implemented: point-wise (Zheng et al., 2023) or pairwise
grading (Li et al., 2023). Nevertheless, previous
study has indicated that the point-wise grading possessed comparable agreement with humans than
the pair-wise grading, which suffers from position
bias (Zheng et al., 2023). Additionally, considering
the evaluating efficiency, compared to pair-wise
grading’s quadratic number of comparisons, the
point-wise grading has advantages in terms of expenses and time. Therefore ALIGNBENCH adopts
point-wise grading either. During the evaluation,
the inputs are the query, the model’s response, and
a human-curated reference answer, and the output
is an multi-dimensional analytical explanation and
a final rating, ranging from 1 to 10. As the task of
grading involves complex reasoning, introducing
Chain-of-Thought in the scoring process has also
been proved useful to augment both the score reliability and interpretability (Zheng et al., 2023).
Specifically, GPT-4 is instructed to generate explanations from multiple dimensions before providing
a final grade on a scale of 1 to 10.
**Rule-Calibrated Referencing.** Given that many
of the questions in ALIGNBENCH are of significant
complexity and difficulty even for GPT-4, we provide a high-quality reference answer, which is primarily generated by GPT-4 and modified by human
annotators to ensure its correctness and improve
its quality. To guide the evaluator to compare the
answer with the reference and generate more controllable scores, we provided detailed grading rules
elaborating the relationship between score intervals
and the answer’s quality compared to the reference.
Additionally, we set the reference answer to score
8 as a reference scoring pivot.
We plot the cumulative distribution of human
judge, general judge and rule-calibrated judge in
Figure 2 to show that rule-calibration judge has
a narrower gap to human evaluation’s cumulative
distribution. Typically, rule-calibrated judge scores
much fewer top scores (9 and 10) than general
judge, aligned with human scoring habits and therefore enhance the discrimination of ALIGNBENCH.
**Multi-Dimensional Analysis.** As tasks vary in
their nature and characteristics, applying the same
evaluation criteria to all tasks would be unjust. For
instance, writing tasks should prioritize creativity,
whereas logical reasoning tasks primarily require
logical coherence. As a solution, we propose a
multi-dimensional scoring approach to evaluate
LLMs’ responses, tailoring the evaluation to the
specific task at hand, promising a more comprehensive and organized explanation. Specifically, we
set up different evaluation dimensions based on different types of questions, as shown in Table 7 and
we instructed the evaluator to analyze the model
answer from specified dimensions and provide dimensional scores. Furthermore, we found that our
multi-dimensional method could effectively balance different dimensions, reducing verbosity bias,
with an example shown in Table 11. The categorical information is also useful for conditioning
generation temperature for target LLMs to generate reply (Zheng et al., 2023). For tasks that has a
relatively fixed answers (e.g., Mathematics, Professional Knowledge, etc.), we set temperature to 0.1
to ensure more deterministic and reproducible generation; for other tasks (e.g., Writing, Task-oriented
Role Play, etc.) that may need more creativity, a
high temperature (e.g., 0.7) is adopted to encourage
longer and more diverse generation.

Point-Wise Grading & Chain-of-Thought (Wei


To justify the rule-calibrated multi-dimensional
point-wise LLM-as-Judge method we design for
ALIGNBENCH, we conduct extensive human evaluation over ALIGNBENCH’s selected queries. We especially focus on two aspects: the method’s agreement with human judging, and the method’s critique quality for more human-interpretable results.


Previous studies (Zheng et al., 2023) have executed
comprehensive agreement experiments, demonstrating that GPT-4 (OpenAI, 2023) evaluator concur excellently with humans within English contexts. However, such agreement remains considerably under-investigated in Chinese contexts,
Since the translation subcategory contains open-ended
questions that requires creativity, it’s treated as Generative
Question.
**Metric** **Method** **Overall** **Dom.** **Chi.** **Fund.** **Math** **Writ.** **Open.** **Role.** **Logic.**


**general** 0.618 0.738 0.576 0.549 0.669 **0.548** 0.524 0.621 0.600
**rules** 0.628 0.709 **0.667** 0.568 **0.689** 0.524 **0.541** **0.673** 0.581


**rules** **0.999** 0.981 **0.901** 0.987 **0.995** 0.976 **0.979** 0.981 0.975
**ours** 0.998 **0.991** 0.869 **0.995** 0.993 0.960 0.936 **0.987** 0.978
**general** 0.751 **0.827** 0.784 0.692 **0.780** **0.714** 0.665 0.735 0.784
**rules** 0.724 0.775 0.764 0.686 0.745 0.651 0.667 0.750 0.743


Table 4: Comparison on human agreement between different judging methods on sampled ALIGNBENCH, rated
by **gpt-4-0613** . The “general” method is a translated version of Zheng et al. (2023)’s with minor modifications.
“ _Fund._ ” denotes Fundamental Language Ability, “ _Chi._ ” denotes Advanced Chinese Understanding, “ _Open._ ” denotes
Open-ended Questions, “ _Writ._ ” denotes Writing Ability, “ _Role._ ” denotes Task-oriented Role Play, “ _Pro_ ” denotes
“Professional Knowledge”, “ _Math._ ” denotes Mathematics, and “ _Logic._ ” denotes Logical Reasoning.
Judge Results ∆ **WR**
**Winner** [Win Rate]
(w/o tie)(%) (%)
A B A Win Tie B Win
ours general 217 94 155 **ours** 58.3 +12.4
ours rules 241 102 139 **ours** 63.4 +20.4
rules general 186 167 147 **rules** 55.9 +7.8
Table 5: Results of quality evaluation (pairwise comparison) by human annotators. Our scoring methods
combining rule-calibration and multi-dimensional criteria can produce consistently better explanations.
thereby warranting further exploration. We have
conducted a comprehensive human annotation experiment, aiming to measure the agreement between evaluations adjudicated by human annotators
and our method.
**Dataset.** We randomly sample a subset of 400
queries from the complete ALIGNBENCH dataset.
To make sure each category consists of enough
samples to produce reliable results, smaller categories are upsampled. To cover LLMs with a
wider levels of capability, we adopt answers from
8 LLMs, including GPT-4 (OpenAI, 2023), three
versions of ChatGLM series (Zeng et al., 2022; Du
et al., 2022), Sparkdesk, Qwen-plus-v1-search(Bai
et al., 2023a), InternLM-7B-Chat (Team, 2023)
and Chinese-Llama2-7B-Chat, producing a total
of 3200 question-answer pairings. Subsequent to
the compilation of the evaluation set, the questionanswer-reference triples are delivered to human
annotators, tasked with assigning quality ratings to
the answers according to the references. Given the
inherent limitations bound to human cognition, annotators are instructed to employ a rating on a scale
from 1 to 5. The scores are indicative of response
quality, with higher scores epitomizing superior
quality and profound satisfaction. In particular, a
score of 1 marks irrelevant, incorrect, or potentially
harmful responses.
[https://xinghuo.xfyun.cn/](https://xinghuo.xfyun.cn/)
**Baselines.** The experiment incorporated two robust baseline comparisons to benchmark our evaluation approach. Note that all the methods use
GPT-4 to evaluate for fairness. **1) General grad-**
**ing:** which leverages a translated and then slightly
modified Chinese version of the evaluation prompt
employed in MT-bench (Zheng et al., 2023). **2)**
**Rule-calibrated grading:** To better instruct the
model to compare the model answer and reference
answer and reduce score variances, we incorporate grading rules into the evaluation process. The
method comprises five scoring intervals, each associated with a specific quality description. The
reference answer is anchored to the score of 8, serving as a relative scoring pivot.
**Metrics.** To comprehensively measure the agreement between the GPT-4 judges and human evaluators, we adopt several metrics. **1) Sample-level**
**Pearson Correlation:** measures correlations at
sample level, by averaging the Pearson score of
each sample. **2) System-level Pearson Correla-**
**tion:** measures correlations at system level. It
calculates the Pearson coefficient between humanjudge and model-judge average scores of LLMs. **3)**
**Pairwise Agreement (w/o tie):** For each response,
human-judge and model-judge scores were converted into pairwise comparisons (excluding ties).
**Analysis.** Results of agreement experiment are
presented in Table 4. It shows that our point-wise
multi-dimensional rules-calibrated LLM-as-Judge
method performs best, particularly on the Samplelevel Pearson metric and the Pairwise Agreement
(w/o tie) metric, thereby substantiating the excellent agreement with human judges. Furthermore, it
is noteworthy that all methods considered demonstrates impeccable performance on the Systemlevel Pearson metric, indicating the reliability and
robustness of the LLM-as-judge.
|Reasoning 中文推理 Language 中文语言
model Overall
Avg. Math. Logi. Avg. Fund. Chi. Open. Writ. Role. Pro.
推理 数学 逻辑 语言 基本 中文 综合 文本 角色 专业
模型 总分
总分 计算 推理 总分 任务 理解 问答 写作 扮演 能力|Col2|Col3|Col4|Col5|Col6|
|---|---|---|---|---|---|
|**model**
**Overall**
Reasoning` 中文推理`
Language` 中文语言`
**_Avg._**
**Math.**
**Logi.**
**Avg.**
**Fund.**
**Chi.**
**Open.**
**Writ.**
**Role.**
**Pro.**
`模型`
`总分`
`推理`
`总分`
`数学`
`计算`
`逻辑`
`推理`
`语言`
`总分`
`基本`
`任务`
`中文`
`理解`
`综合`
`问答`
`文本`
`写作`
`角色`
`扮演`
`专业`
`能力`|**model**
**Overall**
Reasoning` 中文推理`
Language` 中文语言`
**_Avg._**
**Math.**
**Logi.**
**Avg.**
**Fund.**
**Chi.**
**Open.**
**Writ.**
**Role.**
**Pro.**
`模型`
`总分`
`推理`
`总分`
`数学`
`计算`
`逻辑`
`推理`
`语言`
`总分`
`基本`
`任务`
`中文`
`理解`
`综合`
`问答`
`文本`
`写作`
`角色`
`扮演`
`专业`
`能力`|**model**
**Overall**
Reasoning` 中文推理`
Language` 中文语言`
**_Avg._**
**Math.**
**Logi.**
**Avg.**
**Fund.**
**Chi.**
**Open.**
**Writ.**
**Role.**
**Pro.**
`模型`
`总分`
`推理`
`总分`
`数学`
`计算`
`逻辑`
`推理`
`语言`
`总分`
`基本`
`任务`
`中文`
`理解`
`综合`
`问答`
`文本`
`写作`
`角色`
`扮演`
`专业`
`能力`|**Math.**
**Logi.**
`数学`
`计算`
`逻辑`
`推理`|**Math.**
**Logi.**
`数学`
`计算`
`逻辑`
`推理`|**Fund.**
**Chi.**
**Open.**
**Writ.**
**Role.**
**Pro.**
`基本`
`任务`
`中文`
`理解`
`综合`
`问答`
`文本`
`写作`
`角色`
`扮演`
`专业`
`能力`|
|gpt-4-1106-preview
gpt-4-0613
chatglm-turbo`（智谱清言）`
erniebot-3.5`（文心一言）`
gpt-3.5-turbo-0613
chatglm-pro`（智谱清言）`
spark_desk_v2`（讯飞星火）`
Qwen-14B-Chat
Baichuan2-13B-Chat
ChatGLM3-6B
Baichuan2-7B-Chat
InternLM-20B
Qwen-7B-Chat
ChatGLM2-6B
InternLM-Chat-7B
Chinese-LLaMA-2-7B-Chat
LLaMA-2-13B-Chinese-Chat|**8.01**
**7.53**
**6.24**
**6.14**
**6.08**
**5.83**
**5.74**
**5.72**
**5.25**
**4.97**
**4.97**
**4.96**
**4.91**
**4.48**
**3.65**
**3.57**
**3.35**|**_7.73_**
_7.47_
_5.00_
_5.15_
_5.35_
_4.65_
_4.73_
_4.81_
_3.92_
_3.85_
_3.66_
_3.66_
_3.73_
_3.39_
_2.56_
_2.68_
_2.47_|**7.80**
**7.66**
7.56
7.37
4.74
5.26
5.03
5.27
5.68
5.02
4.54
4.75
4.71
4.74
4.91
4.71
3.76
4.07
3.55
4.14
3.56
3.75
3.39
3.92
3.62
3.83
3.16
3.61
2.45
2.66
2.29
3.07
2.21
2.73|**8.29**
7.59
7.49
7.13
6.82
7.01
6.76
6.63
6.59
6.10
6.28
6.26
6.09
5.58
4.75
4.46
4.23|**7.99**
7.33
**8.61**
**8.67**
**8.47**
**8.65**
7.81
6.93
7.42
7.93
7.51
7.94
6.82
7.17
8.16
7.77
7.76
7.24
6.62
**7.60**
7.26
7.56
6.83
6.90
6.71
5.81
7.29
7.03
7.28
6.77
6.51
6.76
7.47
7.07
7.34
6.89
5.84
6.97
7.29
7.18
6.92
6.34
6.90
6.36
6.74
6.64
6.59
6.56
6.22
6.05
7.11
6.97
6.75
6.43
5.75
5.29
6.71
6.83
6.28
5.73
5.81
5.50
7.13
6.84
6.53
5.84
5.96
5.50
7.18
6.19
6.49
6.22
6.40
5.74
6.26
6.31
6.19
5.66
4.91
4.52
6.66
6.25
6.08
5.08
4.34
4.09
5.82
4.89
5.32
4.06
4.31
4.26
4.50
4.63
4.91
4.13
4.13
3.31
4.79
3.93
4.53
4.71|
Table 6: ALIGNBENCH rated by **gpt-4-0613** . “ _Fund._ ” denotes Fundamental Language Ability, “ _Chi._ ” denotes
Advanced Chinese Understanding, “ _Open._ ” denotes Open-ended Questions, “ _Writ._ ” denotes Writing Ability, “ _Role._ ”
denotes Task-oriented Role Play, “ _Pro_ ” denotes “Professional Knowledge”, “ _Math._ ” denotes Mathematics, and
“ _Logic._ ” denotes Logical Reasoning.


Previous study (Zheng et al., 2023) mainly focuses
on the agreement between model judges and human judges when evaluating LLM-as-Judge methods. However, considering the interpretability and
readability as an evaluation process and the clarity
and helpfulness as a feedback-providing approach,
it is also of great significance to evaluate the quality
of the explanation generated by the model judge before grading. To assess the quality of explanations
generated by the methods, as well as to ascertain
the final rating, we conduct a pairwise quality comparison experiment.
**Experiment Settings.** To compare the quality
of the explanations given by our method and the
two baselines mentioned above, we sample 500
question-answer pairs from the ALIGNBENCH
dataset and generated explanations using the three
LLM-as-Judge methods. Then, we pair three explanations under each sample in pairs, producing
1,500 samples, subsequently delivered into human
preference comparisons.
Given a question, a model answer, a reference
answer, and two explanations (denoted as A and
B) given by GPT-4 judges, human annotators are
instructed to compare the quality of explanations.
To guide the human evaluators on the comparison,
we make a quality-judgement guideline as follow:
**1) Rationality** : if the explanation is reasonable,
correct, and fair. **2) Readability** : if the explanation
is logical, well-organized, comprehensive, and detailed. **3) Consistency** : if the explanation and the
final rating are consistent. which also serves as a
brief standard of high-quality explanations.
**Analysis.** Results of quality evaluations are presented in Table 5. Results indicate that our method
generate the most high-quality explanation and
helpful feedback, defeating two baselines in pairwise comparisons with high win rates. Furthermore, it’s worth noting that rules-based grading
outperforms general grading in terms of explanation quality, proving that the grading rules can provide a clear reference-based standard and, therefore
contribute to the clear comparison of the reference
answer and the model answer.


Based on the validness of ALIGNBENCH’s LLMas-Judge scoring, we systematically benchmark a
wide array of LLMs on Chinese alignment with the
help of ALIGNBENCH. We use gpt-4-0613 (OpenAI, 2023) as the judge model to evaluate model
performances.
**Main Results.** Results are shown in Table 6.
The results highlight that most of the evaluated
close-sourced LLMs successfully achieve commendable scores (above or near 6 points). This
demonstrates the potent capability of these ad
vanced LLMs in fulfilling user intents with highquality responses, showing a relatively excellent
level of alignment. It is particularly promising
for the Chinese LLM community to note that certain innovative Chinese-developed LLMs display
performance either equivalent to or marginally surpassing that of gpt-3.5-turbo (OpenAI, 2022),
drawing closer to the recognized leading model,
gpt-4-1106-preview (OpenAI, 2023).
**Analysis: Reasoning Drawbacks.** The evaluation
has revealed some drawbacks in reasoning abilities amongst Chinese-oriented LLMs, which require further attention and development. Given the
leading capability of gpt-4-1106-preview (OpenAI, 2023) in Logical Reasoning and Mathematics
(with scores of 7.66 and 7.80 respectively), there is
substantial room for improvement in light of their
significantly weaker performance.
**Analysis: Chinese Abilities.** Furthermore, certain categories, such as Advanced Chinese Understanding, underline the necessity for an LLM
to possess a proficient understanding of the Chinese language, culture, and history. Our findings
suggest that while gpt-4-1106-preview (OpenAI, 2023) performs relatively well (ranks the second) in these areas, the best-performing Chinesedeveloped LLMs achieved comparable or even better performance in Advanced Chinese Understanding category, potentially attributed to incorporating
more culture-relevant and high-quality Chinesespecific instruction-tuning data in the alignment of
these Chinese LLMs.
**Analysis:** **Open-Source Achievements.** The
results indicated that the top-tier Chinese opensourced LLMs such as Qwen-14B-Chat (Bai et al.,
2023a), Baichuan2-13B-Chat (Yang et al., 2023),
have shown great performance in terms of instruction following and high-quality generation, approaching the performance of some close-sourced
LLMs. Based on our evaluation, Chinese opensourced LLMs have demonstrated their potential
to become alternatives and even competitors with
close-sourced LLMs, showing the inspiring dynamism of the Chinese LLM community.
**Analysis: Dimensions.** From Table 10, we observe that gpt-4-1106-preview (OpenAI, 2023)
has achieved the highest scores in all dimensions.
Generally, the dimensional results are aligned with
the overall results in Table 6. Furthermore, in terms
of correctness and user satisfaction, there is a significant gap between gpt-4 series (OpenAI, 2023)
and other LLMs. We show complete results in Appendix A.3 and hope the evaluation can better assist
the researchers to understand and enhance Chinese
LLMs’ alignment on multiple dimensions.


**Evaluation of LLMs.** Self-supervisedly (Liu et al.,
2021) pre-trained LLMs (OpenAI, 2023; Anil et al.,
2023) exhibit excellent performance in language
tasks, bringing severe challenges to the effective
and comprehensive evaluation of LLMs (Chang
et al., 2023; Zhuang et al., 2023; Xu et al., 2023b).
Faced with the challenges, researchers have proposed benchmarks (Hendrycks et al., 2021; Zhong
et al., 2023; Huang et al., 2023a; Cobbe et al., 2021;
Chen et al., 2021; Bai et al., 2023b; Zhang et al.,
2023) focused on measuring atomic abilities, which
fall short to consider real-scenario usages enough.
**LLM-as-Judge Methods.** LLMs have shown great
potential in evaluating the text quality with high
agreement with human judges (Li et al., 2023;
Wang et al., 2023a; Liu et al., 2023b; Zheng et al.,
2023) and providing helpful feedbacks (Wang et al.,
2023c; Cui et al., 2023) serving as guides for improvement. However, some potential bias and risks
are also identified (Zheng et al., 2023; Wang et al.,
2023b; Ke et al., 2023) when using LLMs-as-judge.
**Alignment of LLMs.** Alignment, including following human instructions and providing helpful
assistance, is crucial for LLMs’ pragmatic applications (Liu et al., 2023c). To aligning LLMs with
humans, related methods include supervisedly finetune LLMs (Wang et al., 2023d; Xu et al., 2023a;
Sun et al., 2023) and improve further through reinforcement learning from human feedback (Stiennon et al., 2020; Ouyang et al., 2022; Glaese
et al., 2022; Rafailov et al., 2023). However, it’s
challenging to effectively evaluate the capabilities
of alignment since the high expense to acquire human preferences and the open-ended reference-free
feature in real application scenarios.


We introduce ALIGNBENCH, a comprehensive multi-dimensional benchmark for evaluating
LLMs’ alignment in Chinese. A sustainable humanin-the-loop data curation pipeline and a better
LLM-as-Judge method has been tailored to enable
ALIGNBENCH’s high-quality automatic evaluation
of LLMs’ Chinese alignment. Benchmarking results of 17 Chinese-supported LLMs are reported.


Here we discuss several limitations of this work.
**To improve automatic LLM-as-Judge.** While we
show that leveraging GPT-4 as judge could achieve
relatively high correlation with human evaluation,
there is a large room for improving this correlation and granularity. Additionally, it is shown that
LLM-as-Judge has some potential biases (Zheng
et al., 2023), including position, verbosity, and selfenhancement. These biases could harm the correctness of evaluation on certain models. We leave
these open problems for future research.
**To cover more topics and queries.** Despite
ALIGNBENCH has a relatively large query set in its
class, it still needs to be enriched for a more stable
and reliable LLM evaluation. Also, ALIGNBENCH
does not include evaluation over long-text queries,
which could be added in its future version.


This work is supported by Technology and Innovation Major Project of the Ministry of Science and
Technology of China under Grant 2022ZD0118600,
Natural Science Foundation of China (NSFC)
62276148 and 62425601, the New Cornerstone Science Foundation through the XPLORER PRIZE.


In this section, we will further elaborate the overall
composition of ALIGNBENCH. In order to perform
a systematic evaluation, we conducted a thorough
real-user query analysis in our LLM-based chat
service. We framed a comprehensive taxonomy of
the LLMs’ abilities based on the Chinese real-user
demands. The overall taxonomy consists of 8 categories, namely Fundamental Language Ability, Advanced Chinese Understanding, Open-ended Questions, Writing Ability, Mathematics, Task-oriented
Role Play, and Professional Knowledge. Each category contains several subcategories, effectively
serving as a complementary materials to deeper
understand the composition of our dataset.
Figure 3: An exemplar scoring process of ALIGNBENCH on _Logical Reasoning_ category. Given question, reference,
and LLM’s answer, ALIGNBENCH harnesses multi-dimensional rule-calibrated LLM-as-Judge to produce a comprehensive comment, consequently giving an integral score of the LLM response.


This category focuses on the basic language understanding and processing tasks, which are derived
from traditional NLP tasks such as entity extraction,
text classification, and commonsense knowledge.
 - **Commonsense knowledge.** This subcategory
mainly evaluates the model’s ability to master commonsense knowledge and fundamental
facts, explain primary concepts, and form a
basic understanding of the physical world.
 - **Reading Comprehension.** This subcategory
mainly evaluates the model’s ability to process
and understand the provided textual material
and answer users’ questions based on that.
 - **Translation.** This subcategory requires models to master different kinds of languages and
understand the interactions between them. Currently, this subcategory mainly covers English
and Chinese.
 - **Text Classification.** This subcategory tests
model’s ability to classify text data into given
categories with different characteristics.
 - **Information Extraction.** This subcategory
measures model’s ability to extract knowledge
from text data, which lays a solid foundation
for more challenging tasks.


This category aims to evaluate the abilities to understand, analyze and produce reasonable and correct
responses faced with Chinese-specific problems,
including Chinese characters, history and culture.
 - **Character-wise Understanding.** This subcategory focuses on the real queries related to Chinese characters and phrases, requiring LLMs
to understand the complex structure and contextual meaning of certain Chinese characters
and utilize them to form reasonable phrases and
sentences.
 - **Cultural Understanding.** This subcategory is
intended to measure models’ abilities to master
the Chinese language at a higher level, including understanding implicit meaning, mastering rhetorical techniques such as humor and
sarcasm, utilizing Chinese-specific idioms and
phrases, and mastering knowledge related to
Chinese culture and history.


This category represents an important role for the
LLMs to become a critic or an advisor for the
users. Equipped with sufficient knowledge and
advanced reasoning ability, LLMs are capable of
providing fresh thoughts, creative perspectives, feasible advice, and comprehensive recommendations
for users. Therefore, this category measures the
LLMs’ ability to offer opinions and suggestions.
 - **Opinion Expression.** This subcategory requires the model to offer reasonable thoughts
to certain subjects, people, events, or circumstances following users’ instructions. Containing open-ended questions without groundtruths, the logical smoothness and the information correctness were of high priority during
evaluation.
 - **Suggestion Offering.** This subcategory aims
to evaluate the ability to analyze users’ intentions and offer some feasible suggestions or recommendations. Being selected carefully, the
questions cover a wide range of topics from
daily life to professional advice.


Regarded as one of the most frequently used capabilities, writing abilities play a vital role in LLMs’
applications. Therefore, we systematically framed
this category into 4 subcategories and selected typical real-user writing instructions, which require not
only an excellent mastery of language but also a
high level of thought formulation and creativity.
 - **Practical Writing.** Practical writing is a practical style of articles formed in long-term social
practice activities and is often used when dealing with public and private affairs in people’s
daily work, including speech scripts, workrelated emails, personal statements, regulations,
etc. Additionally, practical writing assistance
can play a vital part in alleviating workloads
and boosting productivity.
 - **Creative Writing.** Creative writing is a writing style that requires a high level of creativity, emotions, aesthetic taste, and delicate design, including writing novels, essays, poems,
lyrics, and even jokes. It represents not only
higher requirements for the LLMs to fulfill
users’ writing instructions as well as generating high-quality outputs but also a promising
attempt at AI creativity.
 - **Professional Writing.** Professional writing
usually contains domain-specific materials and
has high requirements for professional format
and content. Considering the demands to use
LLMs as professional writing assistants, this
subcategory includes instructions like academic
reports, comprehensive surveys, legal documents, financial analysis, etc.
 - **Custom Writing.** With lots of the writing
queries classified into the above 3 subcategories, there exist other highly personalized
and context-related instructions that require our
attention. Therefore, we collected them into
this subcategory, for instance, rewriting with
casual style, correction of grammar errors, expansion of the given text, imitation writing, etc.


This category aims to evaluate the abilities to understand, analyze, and produce reasonable and correct
responses faced with Chinese-specific textual logic
problems. Therefore, this category plays a vital
role in our overall design.
 - **Proof** Proof problems are a widely used technique for determining the correctness of arguments and propositions. It involves verifying a
proposed proposition through a series of legitimate reasoning steps to ensure that it is valid
and reasonable within a given logical framework. This subcategory focuses on several intellectual qualities such as LLMs’ ability to reason logically, reverse thinking, mathematical
symbolic representation, and clarity of reasoning.
 - **Reasoning** In logical reasoning, reasoning
methods are required to solve a variety of complex problems related to arguments and propositions. Reasoning problems require LLMs to
use a combination of logical reasoning skills,
creative thinking, mathematical symbolic representation, and clear reasoning processes in order to solve a variety of complex logical puzzles
effectively. Compared with Proof subcategory,
this subcategory doesn’t provide ground-truths,
increasing difficulty and complexity.


Considering its logical complexity and a large proportion, math problems are regarded as a necessary
field to evaluate LLMs. We collected math problems in different difficulty levels from elementary
mathematics to advanced mathematics and in different formats, including calculations, simple problem
solving, concept explanation, theorem-proof, etc.
 - **Elementary Mathematics.** The elementary
mathematics subcategory is a branch of mathematics that examines the capability to master
basic mathematical concepts, techniques, and
methods by LLMs. It typically includes but is
not limited to arithmetic, Algebra, geometry,
probability, and statistics.
 - **Advanced Mathematics.** This subcategory is
from a branch of mathematics that covers a
wide range of mathematical topics and concepts and is designed to provide students with
in-depth knowledge of mathematical theory and
problem-solving skills. Advanced mathematics
typically includes calculus, differential equations, linear algebra, probability and statistics,
mathematical proofs, functions of a complex
variable, and linear programming. These elements form the core of advanced mathematics
and provide the basis for solving complex mathematical problems in a variety of fields.
 - **Applied Mathematics.** This subcategory focuses on how mathematical theories and methods can be applied to solve real-life problems
and challenges. Distinguished from higher and
elementary mathematics, applied mathematics
not only requires a certain mathematical ability
but also tests the linguistic comprehension and
mathematical modeling skills of large language
models.


Our real-user scenario analysis indicates that lots
of users request the model to play as a specific
identity to perform corresponding tasks, which is
summarized as task-oriented role play. In order
to evaluate the fulfillment of users’ instructions
and the quality of responses when role-play, we
collected role-play instructions and constructed this
category.
 - **Celebrity.** The instructions in this subcategory
perform the role-play of realistic celebrities,
such as politicians, athletes, etc.
 - **Emotional.** The instructions in this subcategory assign the identity of real-life roles such
as friends, relatives, pets, etc., and provide emotional support for the users.
 - **Entertainment.** The instructions in this subcategory involves games played between the
user and the LLM, requiring imagination and
creativity.
 - **Functional.** The instructions in this subcategory assign the identity of roles with different occupations, experience, or knowledge and
achieve some specific tasks.
 - **Daily life.** The instructions in this subcategory
perform the role-play of different kinds of activities in a more realistic life scenario.


With their advanced knowledge abilities, LLMs
have proven their competence in solving domainspecific problems that require professional knowledge. This category aims to evaluate LLMs’ abilities in specific domains, for instance, physics, history, music, law, etc.
 - **Physics.** Physics is the natural science that
studies the nature and properties of matter and
energy.
 - **Chemistry.** Chemistry is the study of the nature, composition, structure, and patterns of
change of substances. The study of chemistry involves the interrelationships between
substances or the correlation between matter
and energy.
 - **Computer Science.** Computer science is the
systematic study of the theoretical foundations
of information and computation and the practical techniques of how they are implemented
and applied in computer systems.
 - **Biology or Medicine.** Biology consists of the
empirical and extensive study of all aspects of
life. Medicine is an applied science that aims
at and studies the protection of human health
and the enhancement of physical and mental
fitness.
 - **Economics.** Economics is the social science
discipline that studies the relationship between
goods and services, including all purchasing,
production, distribution, and consumption behaviors therein.
 - **Astronomy.** Astronomy is a natural science
that studies celestial bodies and astronomical
phenomena.
 - **Sociology.** Sociology uses a variety of research
methods of empirical investigation and critical
analysis to develop and refine a body of knowledge about the structure of human societies,
social action, or social relations, and to apply
that knowledge.
 - **History.** History, the study of human history
as an object of study, is a form of knowledge
in which human beings sift and combine their
historical materials
 - **Music.** Music, broadly speaking, is any art
that consists of sound. All human cultures have
music, which means that the performance of
music is a universal phenomenon in all cultures.
 - **Law.** Law is a system of rules, the implementa
tion of which is ensured by the coercive power
of the state, which regulates the behavior of
individuals.
 - **Sport.** Sports or sports competition is a social activity that aims at strengthening physical fitness, improving skills, and enriching cultural life through physical exercise, technology,
training, and competition.
 - **Geography.** Geography is the study of the
Earth and its features, inhabitants, and phenomena, the study of the Earth’s surface circles
interacting with each other, and its spatial differences and the process of change.
 - **Literature.** Literature, in a narrow sense, is a
kind of language art, i.e., an art that uses language and writing as a means to visualize and
reflect objective social life and to express the
thoughts and feelings of the subjective author.
 - **Others.** This subcategory contains questions
that not covered by the above domains.


In our multi-dimensional analysis, we select different dimensions for different categories to provide
a more comprehensive and reasonable evaluation.
The detailed selections of the dimensions are described in Table 8 and the descriptions of the dimensions are described in Table 9.
All the prompts used in our experiments are displayed as follows.
**General LLM-as-Judge** is displayed in Figure 6.
**Rule-calibrated LLM-as-Judge** is displayed in
Figure 5.


Our multi-dimensional rule-calibrated LLM-asJudge method provides multi-dimensional analysis
and scores. Therefore, we also calculated dimensional average scores and reported them in Table 10.
For each dimension, the dimensional score was
averaged across all the samples that evaluate the
dimension ability, since each category could be
used for evaluating several but not all dimensions
according to the features of the category.


**A.4.1** **Misleading.**
Upon meticulously verifying and modifying the
reference answers to ensure correctness, we ob
served several instances of incorrect referencing.
This underscores the inherent challenges in utilizing LLMs for evaluative tasks in practical settings,
where reference information may be fraught with
inaccuracies and confusion. Therefore, it is imperative to delve into the LLM evaluator’s response to
such ambiguous or misleading references.
As shown in Figure 7, we highlight a discrepancy
where the correct answer should be _India_ but the
provided reference erroneously states _Japan_ . Notably, the GPT-4 (OpenAI, 2023) evaluator failed to
detect this error, evaluating based on the incorrect
reference. Conversely, when posed the question
directly, GPT-4 (OpenAI, 2023) accurately generated _India_ as the correct answer, demonstrating
its profound knowledge to the question. This dichotomy suggests that while LLMs can provide
accurate responses independently, their evaluative
capabilities can be compromised when presented
with incorrect references or ambiguous materials.
This raises concerns regarding the reliability of using LLMs as evaluators in large-scale applications
and implies that the alignment process of LLMs
may inadvertently reduce the model’s ability to
rectify user errors.


During the evaluation, we noted that certain LLMs
are prone to producing extensive details pertinent
to the query posed. This complexity poses a significant challenge for the LLM evaluator, particularly
when attempting to ascertain the accuracy of the
information provided in scenarios where reference
materials are scarce or incomplete.
As shown in Figure 8, The provided information( _more than 30_ ) regarding the number of countries participating( _in fact 13)_ is inaccurate. In the
absence of corresponding reference information,
the LLM evaluator was unable to assess the factual
correctness of the response accurately. This underscores a discernible weakness in the evaluation
capabilities of LLMs when operating in settings
devoid of sufficient reference material, resulting
in suboptimal performance in discerning factual
inaccuracies.
Addressing this issue may necessitate the integration of an autonomous factual verification tool,
supported by a robust and dynamically updated
information database. We acknowledge the complexity of this challenge and propose it as an avenue
for future research endeavors.


(Fundamental Language Ability) `基本任务` (Factual and Explanatory Question) `事实与解释型问题` `事实正确性清晰` (Correctness), `度` (Clarity), `满 完足备用性户` (Completeness) `需求` (User Satisfaction), 0.1
(Advanced Chinese Understanding) `中文理解` (Factual and Explanatory Question) `事实与解释型问题` `事实正确性清晰` (Correctness), `度` (Clarity), `满 完足备用性户` (Completeness) `需求` (User Satisfaction), 0.1
(Open-ended Questions) `综合问答` (Recommendation Question) `建议型问题` `公平事与实可正负责确性程` (Correctness), `度` (Fairness and Responsibility), `满足用户需求` (User Satisfaction), `创造性` (Creativity) 0.7
(Writing Ability) `文本写作` (Generative Question) `生成型问题` `逻辑连事贯实性正` (Logical Coherence), `确性` (Correctness), `满 创足造用性户` (Creativity), `需求` (User Satisfaction), `丰富度` (Richness) 0.7
(Logical Reasoning) `逻辑推理` (Logical Reasoning Question) `逻辑推理型问题` `事逻实辑连正确贯性性` (Correctness),(Logical Coherence), `满足用户 完需备求性` (User Satisfaction),(Completeness) 0.1
(Mathematics) `数学计算` (Logical Reasoning Question) `逻辑推理型问题` `事逻实辑连正确贯性性` (Correctness),(Logical Coherence), `满足用户 完需备求性` (User Satisfaction),(Completeness) 0.1
(Task-oriented Role Play) `角色扮演` (Generative Question) `生成型问题` `逻辑连事贯实性正` (Logical Coherence), `确性` (Correctness), `满 创足造用性户` (Creativity), `需求` (User Satisfaction), `丰富度` (Richness) 0.7
(Professional Knowledge) `专业能力` (Factual and Explanatory Question) `事实与解释型问题` `事实正确性清晰` (Correctness), `度` (Clarity), `满 完足备用性户` (Completeness) `需求` (User Satisfaction), 0.1
Table 7: Judging dimensions and LLM reply generation temperatures of ALIGNBENCH on different categories.
They both help to provide better category-conditioned scoring in practice (Cf. Section 3.)
Table 8: Dimension setting for different kinds of questions.
|Question Type|Evaluation Dimension|
|---|---|
|`事实与解释型问题`
(Factual and Explanatory Question)|`事实正确性`(Factuality),` 满足用户需求`(User Satisfaction),
`清晰度`(Clarity),` 完备性`(Completeness)|
|`逻辑推理型问题`
(Logical Reasoning Question)|`事实正确性`(Factuality),` 满足用户需求`(User Satisfaction),
`逻辑连贯性`(Logical Coherence),` 完备性`(Completeness)|
|`生成型回答`
(Generative Question)|`事实正确性`(Factuality),` 满足用户需求`(User Satisfaction),
`逻辑连贯性`(Logical Coherence),` 创造性`(Creativity),` 丰富度`(Richness)|
|`建议型回答`
(Recommendation Question)|`事实正确性`(Factuality),` 满足用户需求`(User Satisfaction),
`公平与可负责程度`(Fairness and Responsibility),` 创造性`(Creativity)|
Table 9: The definition of different dimensions.
|Dimension|Definition|
|---|---|
|`事实正确性`
(Factuality)|`回答中提供的信息是否准确无误，是否基于可信的事实和数据。`
Whether the information provided in the response is accurate, based on reliable facts and data.|
|`满足用户需求`
(User Satisfaction)|`回答是否满足了用户提出问题的目的和需求，是否对问题进行了全面而恰当的回应。`
Whether the response meets the user’s question and needs, and
provides a comprehensive and appropriate answer to the question.|
|`逻辑连贯性`
(Logical Coherence)|`回答是否在整体上保持一致，是否在不同部分之间保持逻辑连贯性，避免了自相矛盾。`
Whether the response maintains overall consistency and
logical coherence between different sections, avoiding self-contradiction.|
|`丰富度`
(Richness)|`回答包含丰富的信息、深度、上下文考虑、多样性、`
`详细解释和实例，以满足用户需求并提供全面理解。`
Whether the response includes rich info, depth, context, diversity, detailed
explanations and examples to meet user needs and provide a comprehensive understanding.|
|`创造性`
(Creativity)|`回答是否具有创新性或独特性，是否提供了新颖的见解或解决方法。`
Whether the response is innovative or unique, providing novel insights or solutions.|
|`公平与可负责程度`
(Fairness and Responsibility)|`回答中提供的建议或信息是否可行，`
`是否负有一定的责任，是否考虑了潜在风险和后果。`
Whether the advice or information provided in the response is feasible, carries a certain
degree of responsibility, and considers potential risks and consequences.|
|`完备性`
(Completeness)|`回答是否提供了足够的信息和细节，以满足用户的需求，是否遗漏了重要的方面。`
Whether the response provides suffcient information and details to meet the
user’s needs, and whether it avoids omitting important aspects.|
|`清晰度`
(Clarity)|`回答是否表达清晰易懂，是否使用了简洁的语言和结构，以便用户可以轻松理解。`
Whether the response is clear and understandable, and whether it uses
concise language and structure so that the user can easily understand it.|
Figure 4: The prompt template of multi-dimension rule-calibrated method. The text in blue includes different
evaluation dimensions, the text in red is the scoring process, and the text in green is the scoring rules.
Figure 5: The prompt template for rule-calibrated grading. The text in green is the scoring rules
Figure 6: The prompt template for general grading.
Table 10: LLMs’ performance on different dimensions judged by **gpt-4-0613**, where Corr., Satis., Logic., Compl.,
Clar., Crea., Rich. and Fair., are the abbreviations corresponding to Correctness, User Satisfaction, Logical
Coherence, Completeness, Clarity, Creativity, Richness, Fairness and Responsibility.
**Model** **Corr. Satis. Logic. Compl. Clar. Crea. Rich. Fair.**

Category Question Type Evaluation Dimension Reply Temperature


**gpt-4-0613** 8.00 7.33 9.21 7.85 8.97 6.97 7.41 9.03
**chatglm3-turbo** 6.85 6.52 8.88 6.69 7.99 7.19 7.79 8.89
**erniebot (** `文心一言` **)** 6.94 6.16 8.76 6.20 7.69 6.71 7.15 8.11
**gpt-3.5-turbo-0613** 6.94 6.11 8.80 6.12 7.92 6.50 6.89 8.51
**chatglm2-pro (** `智谱清言` **)** 6.58 6.02 8.73 6.17 7.80 6.62 7.07 8.41
**spark_desk_v2 (** `讯飞星火` **)** 6.60 5.76 8.40 5.68 7.56 6.46 6.74 8.27
**Qwen-14B-Chat** 6.72 5.77 8.73 5.77 7.63 5.94 6.27 7.73
**Baichuan2-13B-Chat** 6.16 5.48 8.43 5.30 7.24 6.27 6.68 8.35
**internlm-20b** 5.97 5.17 8.14 5.15 6.82 5.85 6.33 8.35
**Baichuan2-7B-Chat** 5.83 5.13 8.30 4.91 6.79 6.23 6.47 7.92
**chatglm3-6b-openai** 5.74 5.12 8.21 5.09 6.77 5.93 6.35 7.59
**Qwen-7B-Chat** 5.96 4.93 8.48 4.90 6.75 5.58 5.70 7.30
**chatglm2-6b** 5.13 4.54 7.94 4.46 6.60 5.66 5.79 7.27
**internlm-chat-7b** 4.79 3.51 6.94 3.00 5.82 4.75 4.77 7.35
**Chinese-LLaMA-2-7B-Chat** 4.47 3.49 6.83 3.46 5.22 4.56 4.64 5.16
**LLaMa-2-13B-Chinese-Chat** 4.39 3.29 6.77 3.31 4.84 4.17 3.98 5.62
Figure 7: Misleading Case. The text in red is a Wrong
Reference.
Figure 8: Reference-free Case. The text in red is an
Insufficient Reference.
Query `对于正整数` _n_ `，计算积分` �01 _[x][n][dx]_
_Mathematics_ For a positive integer _n_, calculate the integral �01 _[x][n][dx]_
Reference `对于给定的积分` **:** �01 _[x][n][dx]_ `[ 我][们][可][以][直][接][对]` _[x][n]_ `[ 进][行][不][定][积][分][，][得][到][：]` - _x_ _[n]_ _dx_ = ( _x_ [(] _[n]_ [+1)] ) _/_ ( _n_ + 1) + _C_ `其中` **C** `是常数。然后使用定积分`
Generated
Response
_**Wrong**_
_**Answer**_
General
Judge
_**Lost In**_
_**Verbosity**_
`的上下限` 0 `和` 1 `来计算这个积分的值：` �01 _[x][n][dx]_ [ = (] _[x]_ [(] _[n]_ [+1)] _[/]_ [(] _[n]_ [ + 1))] _[|]_ 0 [1] [= (1][(] _[n]_ [+1)][)] _[/]_ [(] _[n]_ [ + 1)] _[ −]_ [(0][(] _[n]_ [+1)][)] _[/]_ [(] _[n]_ [ + 1) = 1] _[/]_ [(] _[n]_ [ + 1)] _[ −]_ [0 =]
1 _/_ ( _n_ + 1) `所以，` �01 _[x][n][dx]_ [ = 1] _[/]_ [(] _[n]_ [ + 1)] `[。]`
For the given integral �01 _[x][n][dx]_ [, we can directly integrate] _[ x][n]_ [ to obtain the indefinite integral:] - _x_ _[n]_ _dx_ = ( _x_ [(] _[n]_ [+1)] ) _/_ ( _n_ + 1) + _C_,
where _C_ is a constant. Then, we use the definite integral limits 0 and 1 to compute the value of this integral. �01 _[x][n][dx]_ [ = (] _[x]_ [(] _[n]_ [+1)] _[/]_ [(] _[n]_ [+]
1)) _|_ [1] 0 [= (1][(] _[n]_ [+1)][)] _[/]_ [(] _[n]_ [ + 1)] _[ −]_ [(0][(] _[n]_ [+1)][)] _[/]_ [(] _[n]_ [ + 1) = 1] _[/]_ [(] _[n]_ [ + 1)] _[ −]_ [0 = 1] _[/]_ [(] _[n]_ [ + 1)] `[ 所][以][，]` �01 _[x][n][dx]_ [ = 1] _[/]_ [(] _[n]_ [ + 1)]
`要计算这个积分，需要先确定` _n_ `的值。因为` _n_ `是整数，所以我们可以尝试计算` _n_ = 1 _,_ 2 _,_ 3 _,_ 4 _,_ 5 `等不同情况下的积分值，从而找到通解。`
`当` _n_ = 1 `时，原式变为` �01 _[xdx]_ **[,]** `[这][个][积][分][可][以][直][接][求][解][：]` �01 _[xdx]_ [ =] _[ x][|]_ 0 [1] [= 1] _[ −]_ [0 = 1] `[ 当]` **[n=2]** `[ 时][，][原][式][变][为]` �01 _[x]_ [2] _[dx]_ **[,]** `[这][个][积][分][也]`
`可以直接求解：` �01 _[x]_ [2] _[dx]_ [ = (1] _[/]_ [3)] _[x]_ [3] _[|]_ 0 [1] [= (1] _[/]_ [3)(1][3][)] _[ −]_ [(1] _[/]_ [3)(0][3][) = 1] _[/]_ [3] `[ 当]` **[n=3]** `[ 时][，][原][式][变][为]` �01 _[x]_ [3] _[dx]_ **[,]** `[这][个][积][分][也][可][以][直][接][求]`
`解：` �01 _[x]_ [3] _[dx]_ [ = (1] _[/]_ [4)] _[x]_ [4] _[|]_ 0 [1] [= (1] _[/]_ [4)(1][4][)] _[ −]_ [(1] _[/]_ [4)(0][4][) = 1] _[/]_ [4] `[ 当]` **[n=4]** `[ 时][，][原][式][变][为]` �01 _[x]_ [4] _[dx]_ **[,]** `[这][个][积][分][也][可][以][直][接][求][解][：]` �01 _[x]_ [4] _[dx]_ [ =]
(1 _/_ 5) _x_ [5] _|_ [1] 0 [= (1] _[/]_ [5)(1][5][)] _[ −]_ [(1] _[/]_ [5)(0][5][) = 1] _[/]_ [5] `[ 当]` _[n]_ [ = 5] `[ 时][，][原][式][变][为]` �01 _[x]_ [5] _[dx]_ **[,]** `[这][个][积][分][也][可][以][直][接][求][解][：]` �01 _[x]_ [5] _[dx]_ [ = (1] _[/]_ [6)] _[x]_ [6] _[|]_ 0 [1] [=]
(1 _/_ 6)(1 [6] ) _−_ (1 _/_ 6)(0 [6] ) = 1 _/_ 6
`通过这些计算，我们可以猜测当` **n=k** `时，原式可以表示为：` �01 _[x][k][dx]_ [ = (1] _[/k]_ [)] _[x]_ [(] _[k]_ [+1)] _[|]_ 0 [1] `[下][面][用][数][学][归][纳][法][证][明][这][个][猜][测][：][当]` _[n]_ [ = 1] `[ 时][，]`
`左边` **=1,** `右边` **=1,** `猜测成立。假设当` **n=k** `时猜测成立，即：` �01 _[x][k][dx]_ [ = (1] _[/k]_ [)] _[x]_ [(] _[k]_ [+1)] _[|]_ 0 [1] `[那][么][当]` _[n]_ [ =] _[ k]_ [ + 1] `[ 时][，][有][：]` �01 _[x]_ [(] _[k]_ [ + 1)] _[dx]_ [ =]
(1 _/_ ( _k_ + 1)) _x_ [(] _[k]_ [+2)] _|_ [1] 0 `[由][归][纳][假][设][可][知][，][当]` _[n]_ [ =] _[ k]_ `[ 时][猜][测][成][立][，][因][此][当]` _[n]_ [ =] _[ k]_ [ + 1] `[ 时][猜][测][也][成][立][。][因][此][，][对][于任][意][正][整数]` _[n]_ **[,]** `[原][式][可][以][表][示]`
`为：` �01 _[x][n][dx]_ [ = (1] _[/n]_ [)] _[x]_ [(] _[n]_ [+1)] _[|]_ 0 [1]
To compute this integral, it is necessary first to determine the value of _n_ . Since _n_ is an integer, we can attempt to calculate the integral
values for _n_ = 1 _,_ 2 _,_ 3 _,_ 4 _,_ 5, etc., to find a general solution.
When _n_ = 1, the expression becomes �01 _[xdx]_ [, which can be directly solved:] �01 _[xdx]_ [ =] _[ x][|]_ 0 [1] [= 1] _[ −]_ [0 = 1][. When] _[ n]_ [ = 2][, it]
becomes �01 _[x]_ [2] _[dx]_ [, and this integral is also directly solvable:] �01 _[x]_ [2] _[dx]_ [ =] 31 _[x]_ [3] _[|]_ 0 [1] [=] 13 [(1][3][)] _[ −]_ 3 [1] [(0][3][) =] 31 [. When] _[ n]_ [ = 3][, it]
`助手的答案在解答过程中，虽然没有直接给出最终的公式，但是通过列举` _n_ = 1 _,_ 2 _,_ 3 _,_ 4 _,_ 5 `的情况，然后通过观察和归纳，得出了正确的公式。`
```
这种方法虽然比参考答案的直接求解方法更繁琐，但是它展示了一种通过观察和归纳得出结论的思考过程，这对于理解和学习积分的求解方法是
```
`有帮助的。然而，助手的答案在最后的公式中有一个小错误，应该是` �01 _[x][n][dx]_ [ = 1] _[/]_ [(] _[n]_ [ + 1)] `[，][而][不][是]` �01 _[x][n][dx]_ [ = (1] _[/n]_ [)] _[x]_ [(] _[n]_ [+1)] _[|]_ 0 [1] `[。][因][此][，]`
`虽然助手的答案在思考过程和方法上是有价值的，但是在准确性上有所欠缺。评级：` **[[7]]**
The assistant’s response, in the process of solving, did not directly present the final formula. However, by enumerating cases where
_n_ = 1 _,_ 2 _,_ 3 _,_ 4 _,_ 5 and then through observation and induction, the correct formula was derived. Although this method is more cumbersome
than the direct solution approach of the reference answer, it demonstrates a process of reasoning through observation and induction, which
is beneficial for understanding and learning the methods of integral calculus. However, there is a minor error in the final formula presented
by the assistant. The correct expression should be �01 _[x][n][dx]_ [ =] _n_ +11 [, not] �01 _[x][n][dx]_ [ =] _n_ [1] _[x]_ [(] _[n]_ [+1)] _[|]_ 0 [1][. Therefore, while the assistant’s]
response is valuable in terms of its thought process and methodology, it lacks in accuracy. Rating: [7]."
becomes �01 _[x]_ [2] _[dx]_ [, and this integral is also directly solvable:] �01 _[x]_ [2] _[dx]_ [ =] 31 _[x]_ [3] _[|]_ 0 [1] [=] 13 [(1][3][)] _[ −]_ 3 [1] [(0][3][) =] 31 [. When] _[ n]_ [ = 3][, it]
becomes �01 _[x]_ [3] _[dx]_ [, which can be solved in the same way:] �01 _[x]_ [3] _[dx]_ [ =] [1] 4 _[x]_ [4] _[|]_ 0 [1] [=] 4 [1] [(1][4][)] _[ −]_ 4 [1] [(0][4][) =] [1] 4 [. For] _[ n]_ [ = 4][, it becomes]
- 1 - 1 1 1 1 - 1
0 _[x]_ [4] _[dx]_ [, solvable as:] 0 _[x]_ [4] _[dx]_ [ =] 5 _[x]_ [5] _[|]_ 0 [1] [=] 5 [(1][5][)] _[ −]_ 5 [1] [(0][5][) =] 5 [. When] _[ n]_ [ = 5][, it becomes] 0 _[x]_ [5] _[dx]_ [, which is solved as:]
[1] 4 _[x]_ [4] _[|]_ 0 [1] [=] 4 [1]
4 [1] [(1][4][)] _[ −]_ 4 [1]
4 [1] [(0][4][) =] [1] 4
- 1 - 1 1 1 1 - 1
0 _[x]_ [4] _[dx]_ [, solvable as:] 0 _[x]_ [4] _[dx]_ [ =] 5 _[x]_ [5] _[|]_ 0 [1] [=] 5 [(1][5][)] _[ −]_ 5 [1] [(0][5][) =] 5 [. When] _[ n]_ [ = 5][, it becomes] 0 _[x]_ [5] _[dx]_ [, which is solved as:]
�01 _[x]_ [5] _[dx]_ [ =] [1] 6 _[x]_ [6] _[|]_ 0 [1] [=] [1] 6 [(1][6][)] _[ −]_ 6 [1] [(0][6][) =] [1] 6 [.]
From these calculations, we can hypothesize that for _n_ = _k_, the expression can be represented as: �01 _[x][k][dx]_ [ =] _k_ 1 _[x]_ [(] _[k]_ [+1)] _[|]_ 0 [1][. We]
now prove this hypothesis using mathematical induction: When _n_ = 1, the left side equals 1, and the right side equals 1, so the
hypothesis holds. Assume the hypothesis is true for _n_ = _k_, i.e., �01 _[x][k][dx]_ [ =] _k_ 1 _[x]_ [(] _[k]_ [+1)] _[|]_ 0 [1][. Then for] _[ n]_ [ =] _[ k]_ [ + 1][, we have]
- 1 1
0 _[x]_ [(] _[k]_ [+1)] _[dx]_ [ =] _k_ +1 _[x]_ [(] _[k]_ [+2)] _[|]_ 0 [1][. By the inductive assumption, the hypothesis is true for] _[ n]_ [ =] _[ k]_ [; therefore, it also holds for] _[ n]_ [ =] _[ k]_ [ + 1][.]
[1] 6 _[x]_ [6] _[|]_ 0 [1] [=] [1] 6
[1] 6 [(1][6][)] _[ −]_ 6 [1]
6 [1] [(0][6][) =] [1] 6
Hence, for any positive integer _n_, the expression can be represented as: �01 _[x][n][dx]_ [ =] _n_ [1] _[x]_ [(] _[n]_ [+1)] _[|]_ 0 [1][."]
General
Score **7**
Multi-dimensional
Rule-calibrated
Judge(Ours)
_**Fair Due to**_
_**Multi-dimensions**_
**AI** `助手的答案在事实正确性上存在问题。虽然它试图通过计算几个特定的` _n_ `值来找到通解，并使用数学归纳法来证明其猜测，但是它的通解公式`
`是错误的。正确的公式应该是` �01 _[x][n][dx]_ [ = 1] _[/]_ [(] _[n]_ [ + 1)] `[，][而][不][是]` **[AI]** `[助][手][给][出][的]` �01 _[x][n][dx]_ [ = (1] _[/n]_ [)] _[x]_ [(] _[n]_ [+1)] _[|]_ 0 [1] `[。][因][此][，][事][实][正][确][性][的][得][分][是]` [2] `[。]`
`在满足用户需求这一维度上，` **AI** `助手的答案也存在问题。用户的问题是求解积分` �01 _[x][n][dx]_ `[，][而]` **[AI]** `[助][手][的][答][案][虽][然][试][图][解][决][这][个][问][题][，][但][是][由][于]`
`其通解公式错误，所以并没有正确地解决用户的问题。因此，满足用户需求的得分是` **2** `。`
`在逻辑连贯性这一维度上，` **AI** `助手的答案表现得相对较好。它首先计算了几个特定的` _n_ `值，然后基于这些计算结果提出了一个猜测，最后使用数`
`学归纳法来证明这个猜测。虽然其猜测是错误的，但是其推理过程是连贯的。因此，逻辑连贯性的得分是` **6** `。`
`在完备性这一维度上，` **AI** `助手的答案存在问题。虽然它试图通过计算几个特定的` _n_ `值来找到通解，并使用数学归纳法来证明其猜测，但是由于其`
`通解公式错误，所以其答案并不完备。因此，完备性的得分是` **2** `。`
`综合以上各个维度的评价，` **AI** `助手的答案的综合得分是` **3** `。` **{’** `事实正确性` **’: 2, ’** `满足用户需求` **’: 2, ’** `逻辑连贯性` **’: 6, ’** `完备性` **’: 2, ’** `综合得分` **’: 3}**
The AI assistant’s response has issues in terms of factual accuracy. While it attempted to find a general solution by calculating several
specific values of _n_ and used mathematical induction to prove its conjecture, the general solution formula it provided is incorrect. The
correct formula should be �01 _[x][n][dx]_ [ =] _n_ +11 [, not the] �01 _[x][n][dx]_ [ =] _n_ [1] _[x]_ [(] _[n]_ [+1)] _[|]_ 0 [1] [as given by the AI assistant. Therefore, the score for]
factual accuracy is 2.
In terms of meeting user needs, the AI assistant’s response also has issues. The user’s query was to solve the integral �01 _[x][n][dx]_ [, and]
although the AI assistant’s response attempted to address this, it failed to correctly solve the user’s problem due to the incorrect general
solution formula. Hence, the score for meeting user needs is 2.
Regarding logical coherence, the AI assistant’s response performed relatively well. It first calculated several specific values of _n_, then
based on these calculations, it proposed a conjecture, and finally used mathematical induction to prove this conjecture. Although its
conjecture was incorrect, the reasoning process was coherent. Therefore, the score for logical coherence is 6.
In terms of completeness, the AI assistant’s response has shortcomings. Although it attempted to find a general solution by calculating
specific values of _n_ and used mathematical induction to prove its conjecture, the incorrect general solution formula means that its answer
is not complete. Thus, the score for completeness is 2.
Considering the evaluations across these dimensions, the comprehensive score for the AI assistant’s response is 3. ’Factual Accuracy’: 2,
’Meeting User Needs’: 2, ’Logical Coherence’: 6, ’Completeness’: 2, ’Overall Score’: 3."
Ours
Score **3**
Table 11: An comparison example between General Judge and Our Multi-dimensional Rule-calibrated Judge. In this
case, General Judge suffered from verbosity bias, giving a high score to a wrong but long answer. On the contrary,
our Judge balanced different factors and gave a fair rating to the wrong answer.


**Bill Yuchen Lin** _[♠]_ **Abhilasha Ravichander** _[♠]_ **Ximing Lu** _[♢]_ **Nouha Dziri** _[♠]_
**Melanie Sclar** _[♢]_ **Khyathi Chandu** _[♠]_ **Chandra Bhagavatula** _[♠]_ **Yejin Choi** _[♠♢]_
_♠_ Allen Institute for Artificial Intelligence _♢_ University of Washington
 - yuchenl@allenai.org
⋆ [https://allenai.github.io/re-align](https://allenai.github.io/re-align)
ABSTRACT
Alignment tuning has become the de facto standard practice for enabling base
large language models (LLMs) to serve as open-domain AI assistants. The alignment tuning process typically involves instruction learning through supervised
fine-tuning (SFT) and preference tuning via reinforcement learning from human
feedback (RLHF). A recent study, LIMA (Zhou et al., 2023), shows that using
merely 1K examples for SFT can achieve significant alignment performance as
well, suggesting that the effect of alignment tuning might be “ _superficial_ . ” This
raises questions about _how exactly the alignment tuning transforms a base LLM_ .
We analyze the effect of alignment tuning by examining the token distribution
shift between base LLMs and their aligned counterpart (e.g., Llama-2 and Llama2-chat). Our findings reveal that base LLMs and their alignment-tuned versions
perform nearly identically in decoding on the majority of token positions (i.e., they
share the top-ranked tokens). Most distribution shifts occur with _stylistic_ tokens
(e.g., discourse markers, safety disclaimers). These direct evidence strongly supports the hypothesis that alignment tuning primarily learns to adopt the language
style of AI assistants, and that the knowledge required for answering user queries
predominantly comes from the base LLMs themselves.
Based on these findings, we rethink the alignment of LLMs by posing the research question: _how effectively can we align base LLMs without SFT or RLHF?_
To address this, we introduce a simple, tuning-free alignment method, URIAL
( **U** ntuned LLMs with **R** estyled **I** n-context **AL** ignment). URIAL achieves effective alignment purely through in-context learning (ICL) with base LLMs, requiring as few as three constant stylistic examples and a system prompt. We
conduct a fine-grained and interpretable evaluation on a diverse set of examples, named just-eval-instruct. Results demonstrate that base LLMs with
URIAL can match or even surpass the performance of LLMs aligned with SFT
(Mistral-7b-Instruct) or SFT+RLHF (Llama-2-70b-chat). We show that the gap
between tuning-free and tuning-based alignment methods can be significantly reduced through strategic prompting and ICL. Our findings on superficial nature of
alignment tuning and results with URIAL suggest that deeper analysis and theoretical understanding of alignment is crucial to future LLM research.
1 INTRODUCTION
Base large language models (LLMs) that are only pre-trained on unsupervised text corpora typically
cannot directly serve as open-domain AI assistants such as ChatGPT. In order to align these base
LLMs as helpful and harmless assistants, recent research typically fine-tunes base LLMs with instruction tuning (Wei et al., 2022a) and preference learning (Bai et al., 2022). Instruction tuning is a
supervised fine-tuning (SFT) process that mainly uses human annotations or data collected from proprietary LLMs such as GPT-4 (Taori et al., 2023; Wang et al., 2023). A typical method of preference
learning is reinforcement learning from human feedback (RLHF) (Bai et al., 2022), which continually tunes a SFT-ed LLM for further aligning with human preference. Tuning-based alignment has
led to significant improvements in LLMs, appearing to unlock impressive capabilities (Bubeck et al.,
2023), suggesting that extensive fine-tuning is essential to build AI assistants.
Preprint
On the other hand, a recent study, LIMA (Zhou et al., 2023), proposes the “Superficial Alignment
Hypothesis,” which argues that alignment tuning might simply teach base LLMs to select a subdistribution of data formats for interacting with users. Zhou et al. (2023) demonstrates that SFT
with as few as 1,000 examples can also yield high-quality aligned models, thus providing indirect
support for this hypothesis. However, conclusive and direct supporting evidence for the superficial alignment hypothesis remains underexplored. Therefore, it is important to analyze how exactly
alignment tuning alters the behavior of base LLMs.
To this end, we investigate the effects of alignment tuning by directly comparing the token distributions between base LLMs and their aligned versions (e.g., Llama-2 and Llama-2-chat). Surprisingly,
we find that base and aligned LLMs typically perform almost identically in most positions in terms
of ranking tokens during decoding (Sec. 2). Additionally, we observe that the top-ranked tokens in
aligned LLMs are mostly found within the top five tokens ranked by base LLMs, and the distribution
shift is more pronounced in earlier token positions. The most significant distribution shifts occur predominantly in _stylistic tokens_ (e.g., ‘Hello’, ‘Thank’, ‘However’, ‘Remember’, etc.), which include
transitional phrases, discourse markers, and safety disclaimers, rather than in content-bearing words
that directly provide useful knowledge for resolving the queries from users. Our findings (Sec. 2.3)
from token distribution shift analysis directly provide substantial support for the superficial alignment hypothesis. We offer both quantitative and qualitative analyses to demonstrate that alignment
tuning primarily focus on adopting the language style of responsible AI assistants and depends to a
great extent on the knowledge that base LLMs have already acquired.
Based on our findings regarding the superficial nature of alignment tuning, we pose the research
question for rethinking the research on aligning LLMs: _how effectively can we align base LLMs_
_without any tuning?_ We propose a simple, tuning-free alignment method called **URIAL** ( **U** ntuned
LLMs with **R** estyled **I** n-context **AL** ignment), which effectively aligns base LLMs without tuning
their weights (Sec. 3). URIAL leverages in-context learning (ICL) through prompting with just a
few carefully curated stylistic examples and a carefully designed system prompt to achieve impressive alignment results. We craft the in-context examples to begin by affirming the user query and
introducing background information, then proceed to enumerate items or steps with comprehensive
details, and finally conclude with an engaging summary that includes safety-related disclaimers.
Surprisingly, we find that such a straightforward baseline method can significantly reduce the performance gap between base LLMs and aligned LLMs.
To rigorously evaluate different alignment methods, we
design a multi-aspect, interpretable evaluation protocol, detailed in Sec. 4. We create a dataset named
- **just-eval-instruct** which contains 1,000 diverse
instructions from 9 existing datasets, such as those used
by AlpacaEval (Li et al., 2023a), MT-bench (Zheng
et al., 2023), and LIMA (Zhou et al., 2023). Our
analysis encompasses six dimensions of LLM outputs: � helpfulness, � clarity, � factuality, � depth,
- engagement, and � safety. Our extensive results indicate that URIAL, using as few as three constant incontext examples, can effectively align base LLMs. Remarkably, URIAL surpass the LLMs aligned with SFT
or SFT+RLHF on strong base LLMs such as Mistral7b (Jiang et al., 2023a) and Llama-2-70b (Touvron et al.,
2023), as reported in Fig. 1 and Tab. 1.
Figure 1: Comparisons of alignment performance on different aspects.
helpfulness
safety
The surprisingly strong performance of URIAL not only
further substantiates the superficial alignment hypothesis, but also prompts us to rethink the current
research on alignment. To deepen our understanding of LLMs, we believe that it is essential to
accurately distinguish which knowledge and reasoning capabilities originate from pre-training as
opposed to those that must be acquired through alignment tuning. In this vein, our contributions in
this work can support future research in the analysis and alignment of base LLMs. Furthermore,
our findings indicate that developing better tuning-free, inference-time alignment methods could be
a promising alternative to SFT and RLHF in certain scenarios (Sec. 5.4).
Preprint
|Context: 𝒒+ 𝑜 + …+ 𝑜<br>" !#"|Col2|
|---|---|
|**Aligned**||
Figure 2: **Analyzing alignment with token distribution shift.** An aligned LLM (llama-2-chat) receives a
query **q** and outputs a response **o** . To analyze the effect of alignment tuning, we decode the untuned version
(llama-2-base) at each position _t_ . Next, we categorize all tokens in **o** into three groups based on _ot_ ’s rank in
the list of tokens sorted by probability from the base LLM. On average, 77.7% of tokens are also ranked top 1 by
the base LLM ( **unshifted** positions), and 92.2% are within the top 3 (+ _**marginal**_ ). Common tokens at **shifted**
positions are displayed at the top-right and are mostly stylistic, constituting discourse markers. In contrast,
knowledge-intensive tokens are predominantly found in unshifted positions. (More in Fig. 8 and Appendix. C)
2 DEMYSTIFYING ALIGNMENT VIA TOKEN DISTRIBUTION SHIFT
**Background.** In this paper, we use the terms “ _untuned LLMs_ ” and “ _base LLMs_ ” interchangeably to
refer to LLMs that have been pre-trained on large corpora without any subsequent fine-tuning using
instruction data. We denote a base LLM as _f_ ( **x** ; _θ_ ), where **x** is the input context and _θ_ represents
the set of parameters that generate the next token. The “ _**alignment tuning**_ ” [1] process tunes the
parameters _θ_ of a base model _f_ to create a more assistant-like model _g_ ( **x** ; _β_ ) that adheres to user
instructions and human preferences. This process typically comprises two stages: supervised finetuning (SFT) on instruction data and reinforcement learning from human feedback (RLHF). During
the SFT stage, the base LLM is fine-tuned using instruction-answer pairs (i.e., instruction tuning).
In the RLHF stage, a reward model is used to further refine the SFT-ed model, resulting in better
alignment with human expectations in terms of helpfulness, honesty, and harmlessness.
2.1 ALIGNMENT AS TOKEN DISTRIBUTION SHIFT
**Motivation.** To understand the learning process during alignment tuning (SFT+RLHF) and the
differences between aligned and base models, we propose to analyze through the lens of token
distribution shift. Specifically, for a given user query **q** = _{q_ 1 _, q_ 2 _, · · · }_, we input it into the aligned
model _g_ ( _x_ ) to obtain its output **o** = _{o_ 1 _, o_ 2 _, · · · }_ via greedy decoding. For each position _t_, we
define a _‘context’_ at this position to be **xt** = **q** + _{o_ 1 _, · · ·, ot−_ 1 _}_ . We denote the aligned model’s
probability distribution for predicting the next token of this position as _P_ align, where _ot_ has the
highest probability. Our analysis is driven by the question: _What happens if we switch from the_
_aligned model g to the base model f for decoding the next token at this position?_
By passing the context **xt** into the base model _f_, we generate another probability distribution, _P_ base,
for sampling the next token at this position. If the base model learns to modify its behavior in this
context through alignment tuning, we should observe a distribution shift between _P_ base and _P_ align
at this position. On the other hand, if the two distributions are very similar to each other, it implies
that alignment tuning has minimal impact on this position.
**Shifted positions.** Analyzing the difference between two distributions across the entire token vocabulary is challenging, particularly when sampling is enabled for decoding. As illustrated in Figure 2,
the aligned model _g_ with greedy decoding is first used to generate a full output **o** . For each position
_t_, tokens are ranked according to their probability _P_ base as predicted by the base model _f_ . The rank
of _ot_ in this sorted list is defined as the ‘base rank’, denoted as _η_ . This results in three types of
positions: (1) **unshifted positions** ( _η_ = 1): _ot_ is the top-ranked token in both _P_ base and _P_ align, having the highest probability; (2) **marginal positions** (1 _< η ≤_ 3): although _ot_ is not the top-ranked
token in _P_ base, it is still likely to be sampled for decoding, with the 2nd or 3rd highest probability.
1We use the term “alignment tuning” to refer both SFT (instruction-tuning) and RLHF process for simplicity.
Preprint

The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context
  Learning

THE UNLOCKING SPELL ON BASE LLMS: RETHINKING ALIGNMENT VIA IN-CONTEXT LEARNING

Bill Yuchen Lin
Abhilasha Ravichander
Ximing Lu
Nouha Dziri
Melanie Sclar
Khyathi Chandu
Chandra Bhagavatula
Yejin Choi