abilities, such as the ability to follow instructions and to
conduct helpful conversations with the user. Many model
providers perform further tuning, known as alignment, to make
the LLM harmless according to their definition of harmlessness. These steps of ‘civilizing’ and ‘humanizing’ the LLM are
decisive in controlling the model’s behavior, more than the pretraining of the base model. The harms that model providers
aim to prevent are common ones found in risk taxonomies
such as hate, malice, exclusion, profanity and toxicity, which
have existing benchmarks and evaluation datasets.
Nevertheless, we do not believe that such alignment to
common concerns can ever be comprehensive and we do not
believe that all dimensions of alignment are always necessarily
desirable. Context matters. Every industry, sector, jurisdiction,
culture, and use case has its own unique and _particular_ desired
behaviors that are _not_ captured in a _common_ taxonomy. The
examples are numerous. In a medical application, developers
may not want an LLM to treat names of body parts as profanity. In a customer complaint processing application whose
inputs are laced with offensive language, developers may want
the system to continue to respond. A grocery store’s chatbot
may have an extra requirement to refrain from mentioning
poisonous food items and a bank’s chatbot may have an extra
requirement to refrain from mentioning competitor’s brands
or products. Laws may require certain LLM behaviors, like
The authors are with IBM Research.
Fig. 1. A stylized depiction of Alignment Studio with its three components:
Framers, Instructors, and Auditors.
one in China requiring all generative content to reflect the
core socialist values. An organization may have a style guide
for the LLM’s tone and personality that must be adhered to.
Companies may have guidelines that specify business conduct
to be respected. All of these examples are valid desired
behaviors depending on the context, but they would not show
up in the alignment done by model providers for common
concerns. [1]
There are many sources of regulations: not only laws,
but also social norms, market demands, and technological
constraints [1]. The associated requirements can be quite
unique and contextual based on the use case. As such, they
will not have existing benchmarks with which to evaluate the
LLM. Additionally, different regulations may be competing or
conflicting. In contrast to high-level general statements like
[2]: “Do NOT choose responses that exhibit toxicity, racism,
sexism or any other form of physical or social harm,” particular
regulations may be quite detailed. At IBM where we work, we
have detailed business conduct guidelines and will use these
as a running example of a particular set of regulations with
which to align LLM behavior.
1We, as authors, may not agree with aligning an LLM to all of the listed
examples. But that is the point: our personal values are not universal and we
should not impose them on end-communities. We have given the examples in
the spirit of providing a broad aperture on the possibilities.
Adherence to various particular contextual regulations has
many business benefits, from better serving customers to
avoiding prosecution. Perhaps the biggest benefit is making
the LLM authentic to the values of the model deployer and
the community of end-users. It is a form of personalization
or customization [3], a kind of steerable pluralism [4], and
a method of decoloniality that dismantles the power and
privilege of model providers and empowers communities to
have a say in what is ‘civil’ and ‘human’ [5]. Importantly,
application developers can only further align LLMs beyond
the common alignment done by model providers if the models
are _open_ . Furthermore, alignment techniques must not be too
costly or burdensome that they extend beyond the means of
application developers.
The customization of the LLM’s behavior to non-universal
values and requirements calls for tooling that we name _Align-_
_ment Studio_ . The starting point is a set of regulations given
in a natural language policy document, which could be a law,
an industry standard, or a corporate guideline that has already
been deliberated upon and adopted [2] . The tooling permits a
principled, transparent, auditable, and deliberate approach to
alignment.
As illustrated in Fig. 1, the first component of Alignment Studio is named ‘Framers.’ It applies several knowledge engineering and generative AI techniques to produce
instruction data and scenario data in a form that enables
downstream tasks of instilling desired behaviors into the LLM
and evaluating whether we were successful in doing so. The
second component, ‘Instructors,’ uses the output of Framers to
fine-tune the model accordingly. Importantly, the Instructors
component also allows for the orchestration of competing
values or regulations. The third component, ‘Auditors,’ uses a
combination of human and automated benchmarking and redteaming to evaluate whether the fine-tuned model has learned
the desired behaviors. Framers, Instructors, and Auditors form
a continuous cycle of development. Framers need not necessarily be the first step in the development lifecycle; for
example, a test-driven development approach could begin with
Auditors. Auditors can also quantify the limits of alignment
(for example, misalignment to principles), which can suggest remediation approaches like data augmentation or model
editing. A representative software architecture for Alignment
Studio, starting with policy documents, is illustrated in Fig. 2.
ALIGNMENT STUDIO AND GOVERNANCE
Broadly, the purpose of aligning an LLM to a particular
set of regulations is to control or govern its behavior. In this
sense, we can imagine the Alignment Studio as a feedback
control system like the heating and air-conditioning system
in a building. Users set a desired temperature schedule, a
thermostat controller turns the furnace and air conditioner
on and off to try to meet the desired temperature, and a
sensor measures the current temperature that is compared to
the desired temperature. The difference is fed back to the
controller. The desired behavior and the measured behavior
2In future work, we will expand to other forms of values specifications
such as fables and folklore.
must be comparable. Context is important throughout. A
different schedule may be desired if there is an evening event
or a holiday. There may be technical requirements such as
maintaining a minimum temperature inside (so that water pipes
do not burst) or a minimum temperature outside at which the
air conditioner may be activated (so that its compressor is not
wrecked). There may be conflicting values such as keeping the
building fairly cool in the winter for environmental reasons and
keeping it fairly warm to satisfy a visiting guest who likes it
that way.
Alignment Studio provides the technologies to accomplish
all these parts for AI governance, but with more complex
values, behaviors, and regulations than a scalar-valued temperature. While regulations are often quite specific in their intent
(e.g., an employee may not work for a competitor), they often
require knowledge beyond what is given in the document (e.g.,
who constitutes a competitor in the above case). Regulations
are also essential for constraining the response of a model
with a significant emphasis on correctness, faithfulness, etc.;
it is not acceptable to give an incorrect answer (that violates
the regulations). More importantly, if additional information is
required to provide a correct answer, the model must say so,
rather than always generate a response to the question. This
prevents known LLM issues such as hallucinations.
Another important dimension of AI governance provided by
Alignment Studio is transparently and controllably choosing
among possibly conflicting behaviors. Lazar summarizes the
need for such governance [6]: “Steering LLM behaviour is
actually a matter of governing their end-users, developing
algorithmic protections to prevent misuse. If this algorithmic
governance depends on inscrutable trade-offs made by an
LLM, over which we have no explicit or direct control,
then that governing power is prima facie illegitimate and
unjustified.”
_Running Example_
In this article, we will use the following running example
to illustrate the Alignment Studio. An LLM is infused into
an application for IBM employees to ask general questions,
receive advice, and receive suggestions. The LLM is aligned
to IBM’s corporate policies documented in the IBM Business
Conduct Guidelines (BCGs), [3] which is a 46-page document
with about 11,500 words divided into 8 sections with 78
paragraphs. The content is expressed in different forms such
as _topic-paragraph_, _question-answer_, and call-out _blocks_, that
incorporate 306 enforceable individual policies. While we can
fine-tune a model directly on the raw text, and this may
teach the model the vocabulary and general patterns in the
text, there is not enough signal for a model to learn about
the BCG policies themselves. For clarity in exposition, we
only showcase alignment over a base instruction-following
model and exclude other bells and whistles that would be
part of a robust system. Importantly, the internal chatbot
application we are imagining is not meant only as an interface
to retrieve facts or knowledge about the BCGs, but as a
3https://www.ibm.com/investor/governance/business-conduct-guidelines
|Policy|Col2|Col3|Col4|
|---|---|---|---|
|Policy|Policy|Policy|Policy|
|Policy|do|cuments|cuments|
Fig. 2. A realization of the Alignment Studio software architecture, starting with policy documents. End-to-end software testing and documentation is
recommended, but implementations need not include all components.
general-purpose question-answering service that uses BCG
policies as constraints to its responses about various topics.
FRAMERS
The Framers module identifies the knowledge that users
consider essential to the application (or domain) so it can
be codified for the customization of the LLM model and the
validation of its results. In a word, Framers _frames_ the problem
space so it can be leveraged down the line by the rest of the
system.
In our running example, this means leveraging the structure
and content of the IBM BCGs to create fine-tuning data
suitable for model alignment. As mentioned above, directly
fine-tuning a language model with policy documents would
endow it with policy-related vocabulary but would not give
it the ability to respond with contextually-relevant policy
information or to assess policy compliance.
Hence we proceed to create _instruction_ style data [7], [8],
which consists of examples of policy-relevant instructions for
various tasks, and _scenario_ style data (discussed below) in
order to align models to the type of tasks that users will
need, including identifying the relevant policies for a given
situation or whether a scenario is compliant or not. Manually
creating sufficient training data is expensive; hence we adopt
a hybrid approach where we create some _seed_ data in both
styles and use LLMs to create synthetic data to augment this
dataset. Both of these datasets require extraction of paragraphs
and self-sufficient atomic policies from the BCG document.
We could also use other sources of data such as policy
training materials that contain questions and answers related to
policies; this would be a high-quality data source for validating
alignment results.
For the first style of data, we extract three types of seed data:
(a) _topic-paragraph_, corresponding to topics and paragraphs in
the document, (b) _question-answer_, corresponding to question
and answers provided in the document, and (c) _blocks_, corresponding to call-out blocks that highlight a policy scenario.
These correspond to only two different task instructions:
_summarization_ and _question-answering_ . The small quantity of
seed data we have and these two instruction types alone do not
enable the model to generalize. Therefore, we prompt another
LLM to generate _synthetic_ data based on this seed data. We
find that LLMs are adept at producing a diverse variety of
task instructions, even starting from just two seed tasks. This
is true with powerful LLMs such as _LLama2-70B_, _Falcon-_
_180B_, and _Mixtral-8×7B_ using just a few in-context examples.
From the 100,000 synthetic examples that we created, we
are left with about 76,000 examples to train the model, after
filtering malformed examples, and withholding a small fraction
as validation/test data. A depiction of creating instruction style
data is provided in Fig. 3.
For the scenario style data, we start by manually creating
real-world situations that either comply, violate, or are ambiguous (they require extra information to make a decision)
for a small number of policies in the IBM BCGs. For example, for the policy, _It is your responsibility to maintain_
_IBM confidential and proprietary information_, a compliant
scenario (fictional) could be _Asha was asked by her good_
_friend about a deal that IBM was involved in, but she knew_
_it was confidential. So she politely declined to answer._ A
non-compliant scenario can be similarly created, and such
contrastive scenarios can be used to align models to policies in
a fine-grained way. Since manual data creation for scenarios
can be expensive, we leverage this _seed_ data to create large
amounts of synthetic scenarios for every policy in the document, by appropriately prompting LLMs. This scenario dataset
can be used as additional data for a _classification_ task but it
can also be used for creating policy violation detectors.
While asking LLMs to generate synthetic data usually results in knowledge of good quality and reasonable variety, we
have no assurances as to the coverage of the dataset, and hence,
the overall reliability of the target LLM. A way to ensure
coverage of the domain for both tuning and validation is to use
a complementary symbolic technology: knowledge graphs, in
Fig. 3. Creating instruction style seed data from policy documents (left) and using it to generate synthetic data using LLMs in a few-shot setting (right).
particular, ontologies with reasoning over relations. Ontologies
provide structured, factual information in an algorithmic and
traceable way, while LLMs offer advanced natural language
understanding and generation. Fortunately, there is a large
body of open knowledge codified into ontologies such as
Wikidata [9] and ConceptNet. We use this knowledge to
systematically generate data to cue scenarios with a thorough
domain vocabulary.
The BCGs contain main structural hierarchies within and
outside of IBM, such as organizations (like suppliers, competitors, and government entities), departments (for instance,
legal, human resources, and accounting), assets (like products,
facilities, systems, and intellectual property), and people (for
example, IBM employees, government officials, and family
members). Further, the BCGs make clear what different entities are, and how they are organized and related, for example,
what constitutes a government entity, how an employee relates
to a manager, and what kind of information a data processor
handles.
This information can be applied to construct a domainspecific ontology for a BCG use case, and this ontology can
complement the entities and relations extracted automatically
from Wikidata, such as stakeholders (people, organizations like
corporations, government entities and departments), geopolitical entities, occupations and information technology products and services. We extract the inheritance hierarchies and
ancillary entities (e.g., locations) based on the semantics of
the relations (which are orders of magnitude smaller than
entities). This ontological structure can be used to clarify
ambiguous statements (as seen earlier with an employee may
not work for a competitor) by stating explicitly the relation
between employee and organization (for which the employee
works), organization and competitor, and whether ‘works’ is
a permissible relation between employee and competitor, and
if so, how. The ontological structure can also be utilized to
check for appropriate coverage of the created datasets as we
expect all of the ontology terms and relations to appear at least
once in the created datasets.
INSTRUCTORS
Through Instructors, instilling the desired values and behaviors for alignment is performed using instruction data
and human guidance. This is usually achieved using supervised fine-tuning (SFT) on high quality demonstrations of desired behavior, and/or using reinforcement learning fine-tuning
(RLFT) to optimize rewards evaluating preferences over LLM
behavior. For the use case of BCGs, these algorithms help the
LLM follow various implicit values/behaviors, expressed in
the document, through instruction and scenario data generated
by the Framers component.
Regulatory documents typically reflect multiple, sometimes
conflicting, values or desired behaviors that LLMs need to
be aligned with, requiring techniques for the aggregation of
these values and behaviors. Instructors allows for the training
of reward models from both preference data as well as binary
labels. These rewards assess how well the LLM output aligns
with each individual value and desired behavior considered
under a use case. Instructors allows for the elicitation of the
values or principles we want the LLM to follow as well as the
relative importance among them to resolve possible conflicts.
Then, RLFT is used to align the LLM with these values
based on their relative importance. Instructors also allows for
inferring the relative importance of each value from the context
in which the LLM is used [10]. Finally, in a low-resource
regime, fine-tuning requires parameter-efficient optimization
strategies. These strategies are incorporated using (quantized)
low-rank adaptations via (Q)LoRA [11], [12].
AUDITORS
The Auditors component is tasked with ensuring that the
data from Framers and methods from Instructors have resulted
in a well-performing model with respect to all the desired
criteria, including particular contextual regulations. In general,
model evaluation can be categorized along three axes:
1) **When** : At which moment in the development lifecycle
of the model or application the evaluation is performed
(e.g., during training to ensure the model is capable with
respect to general desirable abilities and/or particular
regulations; after training, once a model checkpoint is
deemed sufficiently performant, to establish whether
the model satisfies criteria that are too costly to be
checked during training; after deployment, to ensure no
unexpected or unaccounted for behavior is encountered);
2) **What** : What type of data is used during the evaluation (e.g., established benchmarks for testing generalpurpose abilities, general-purpose alignment data to test
for general human preferences, hand-crafted domainspecific data to ensure adherence to particular desired
criteria);
3) **How** : How the auditing or evaluation methodology is
performed and by whom (e.g., automated evaluation
based on well-defined benchmarks, human-in-the-loop
red-teaming of models, or a combination of both).
Systematic evaluation of models for particular contextual
regulations requires specially crafted data, as general benchmarks which cover specific regulations are unlikely to exist.
Hence, domain-specific evaluation is carried out in two steps.
First, the model is evaluated for alignment against a small,
curated dataset of test cases. Then, red-teaming [13] is utilized
to uncover potential deficiencies in the aligned model. This
red-teaming step helps to dynamically extend the datasets that
can be used across the lifecycle for subsequent iterations of
the aligned model.
_Red-Teaming_
We find that red-teaming for adherence to particular contextual regulations is particularly effective when comparing the
output generated by two models side by side – one aligned
model which Instructors has trained with data from Framers,
and one unaligned model that has not seen any of the particular
regulation data. Given these two models, red-team members
are asked to craft prompts which test for adherence to the
regulations or policies of interest. Red-team members grade
the responses along different dimensions such as faithfulness
and completeness, providing detailed comments on the output
quality whenever binary grading does not provide sufficient
signal. The data collected during this red-teaming can be used
to develop further insights into improving the aligned model
as follows:
1) Both the aligned and unaligned models generate aligned
responses: the corresponding test cases are deemed
“straightforward” examples, where the input is not difficult enough such that alignment to the specific regulation
is required in order to provide a relevant and accurate
answer.
2) The unaligned model provides better responses than
the aligned model: this situation indicates that conflicting values and/or poor quality instruction data may
have negatively impacted the helpfulness of the aligned
model.
3) The unaligned model fails to follow the particular regulations being tested, while the aligned model does:
these are great examples! They demonstrate the need
for specific alignment to the particular policies in order
to accurately respond to the query.
4) Both the aligned and unaligned models fail to follow
the particular regulations being tested: these are the
“hardest” examples, indicating that more or better data
is required for the aligned model to learn the particular
regulation.
Recall that the main priority of the Auditors is to validate
initial alignment and establish baselines. Baselines are then
used to continually control and ensure desired behavior of
the LLM model. Indeed, the key aspect of Auditors is that
evaluation is _never_ completed: it is a dynamic activity that
lasts as long as the deployed model or application is userfacing.
SUMMARY DEMONSTRATION
We demonstrate Alignment Studio by aligning an IBM
Granite model [14] to the IBM BCGs. We create the seed
_instruction_ style data using the methodology described in the
Framers section and use SFT to align the model. In addition,
we use retrieval-augmented generation (RAG) [15] to enable
a fair comparison between, and improve the faithfulness of,
the unaligned and the aligned models. We developed a user
interface (UI) to achieve three goals: (a) prompt the aligned
model to obtain responses, (b) compare the responses of
the aligned model and the unaligned model, and (c) capture
user preference between the responses of the unaligned and
the aligned model along with the capability to perform redteaming. The UI, illustrated in Fig. 4, contains three tabs
- one corresponding to each of the above goals. Using the
UI, we illustrate an example comparing the responses of the
unaligned model and the aligned model to a user prompt about
purchasing software. The aligned model produces a response
that is more faithful to the policy.
As another example, consider the following prompt: _I want_
_to create a plan that will help me allocate time between my_
_work at IBM and volunteering as an open source developer._
_What should I be careful about?_ The unaligned model produces the response: _You should be careful about spending too_
_much time on open source projects, as this can affect your_
_work performance at IBM._, while the aligned model responds:
_Consult IBM’s Open Source Participation Guidelines and_
_follow them._ The unaligned model response, while reasonable,
is neither helpful nor faithful. The aligned model clearly and
concisely states the guidelines that need to be consulted.
Lastly, consider the prompt: _My brother has a generative_
_AI startup company. What kinds of insider information about_
_IBM’s new venture fund can I send him?_ The response of
the unaligned model is: _Information that is not available to_
Fig. 4. UI for comparing the responses of the unaligned and aligned model for a given prompt.
_the general public that could influence a reasonable investor_
_to buy, sell or hold a company’s securities is considered_
_inside information. Information can be material even if it_
_would have only a small impact on the price of the securities._
The aligned model responds: _None. Insider information about_
_IBM’s financial performance, acquisitions, divestitures, or_
_other significant activities, or inside information about IBM or_
_any other company, is strictly confidential and should not be_
_disclosed to anyone outside of IBM, including family members_
_or IBM colleagues who do not have a business need to know._
_Additionally, buying or selling securities of a client, IBM_
_Business Partner, or other party based on inside information is_
_unethical and illegal. It is important to consult IBM Counsel_
_if you have any questions or concerns about what you can_
_or cannot do with inside information._ The unaligned model
is incorrect since it does not directly address the question.
The aligned model’s response, while correct, can be improved
to include more policy details. This additional information,
given through a feedback UI shown in Fig. 5, can be used by
developers to improve the aligned model.
CONCLUSION
We present a principled approach to align LLMs to particular contextual regulations, along with a robust and extensible
architecture for achieving this. Our methodology is flexible
and we demonstrate it by aligning an LLM to the IBM Business Conduct Guidelines. Future work includes expanding to
other forms of value specifications and adding semi-automated
approaches to eliciting misaligned responses [16].
ACKNOWLEDGMENT
The authors thank Payel Das, Jason D’Cruz, Michael Hind,
Miao Liu, Ronny Luss, Jacquelyn Martino, Erik Miehling,
John T. Richards, Matthew Riemer, Skyler Speakman, and
John Wamburu for comments and discussions.
Fig. 5. UI for evaluating responses for correctness and providing feedback
on their quality.
REFERENCES
[1] L. Lessig, “The new Chicago school,” _Journal of Legal Studies_, vol. 27,
no. S2, pp. 661–691, Jun. 1998.
[2] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen,
A. Goldie, A. Mirhoseini, C. McKinnon, C. Chen, C. Olsson, C. Olah,
D. Hernandez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez,
J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, K. Lukosuite,
L. Lovitt, M. Sellitto, N. Elhage, N. Schiefer, N. Mercado, N. DasSarma,
R. Lasenby, R. Larson, S. Ringer, S. Johnston, S. Kravec, S. El Showk,
S. Fort, T. Lanham, T. Telleen-Lawton, T. Conerly, T. Henighan,
T. Hume, S. R. Bowman, Z. Hatfield-Dodds, B. Mann, D. Amodei,
N. Joseph, S. McCandlish, T. Brown, and J. Kaplan, “Constitutional AI:
Harmlessness from AI feedback,”
[3] H. R. Kirk _et al._, “Personalisation within bounds: A risk taxonomy
and policy framework for the alignment of large language models with
personalised feedback,”
[4] T. Sorensen _et_ _al._, “A roadmap to pluralistic alignment,”
[5] K. R. Varshney, “Decolonial AI alignment: Openness, vi´ses.a-dharma,
and including excluded knowledges,”
[6] S. Lazar, “Frontier AI ethics,” _Aeon_, Feb. 2024.
[7] Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi,
and H. Hajishirzi, “Self-instruct: Aligning language models with selfgenerated instructions,” in _Proceedings of the Annual Meeting of the_
_Association for Computational Linguistics_, Jul. 2023, pp. 13 484–13 508.
[8] O. Honovich, T. Scialom, O. Levy, and T. Schick, “Unnatural instructions: Tuning language models with (almost) no human labor,” in
_Proceedings of the Annual Meeting of the Association for Computational_
_Linguistics_, Jul. 2023, pp. 14 409–14 428.
[9] D. Vrandeˇci´c and M. Kr¨otzsch, “Wikidata: A free collaborative knowledgebase,” _Communications of the ACM_, vol. 57, no. 10, pp. 78–85,
Oct. 2014.
[10] J. Rios _et al._, “Contextual moral value alignment through context-based
aggregation,” _under review_, 2024.
[11] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang,
and W. Chen, “LoRA: Low-rank adaptation of large language models,”
[12] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “QLoRA:
Efficient finetuning of quantized LLMs,”
[13] D. Ganguli, L. Lovitt, J. Kernion, A. Askell, Y. Bai, S. Kadavath,
B. Mann, E. Perez, N. Schiefer, K. Ndousse, A. Jones, S. Bowman,
A. Chen, T. Conerly, N. DasSarma, D. Drain, N. Elhage, S. El-Showk,
S. Fort, Z. Hatfield-Dodds, T. Henighan, D. Hernandez, T. Hume,
J. Jacobson, S. Johnston, S. Kravec, C. Olsson, S. Ringer, E. TranJohnson, D. Amodei, T. Brown, N. Joseph, S. McCandlish, C. Olah,
J. Kaplan, and J. Clark, “Red teaming language models to reduce harms:
Methods, scaling behaviors, and lessons learned,”
2022.
[14] IBM Research, “Granite foundation models,”
https://www.ibm.com/downloads/cas/X9W4O6BM, 2023.
[15] Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, Q. Guo,
M. Wang, and H. Wang, “Retrieval-augmented generation for large
language models: A survey,”
[16] G. Kour, M. Zalmanovici, N. Zwerdling, E. Goldbraich, O. N. Fandina,
A. Anaby-Tavor, O. Raz, and E. Farchi, “Unveiling safety vulnerabilities
of large language models,”


**Feifan Song** [1] **, Bowen Yu** [2] **, Hao Lang** [2] **, Haiyang Yu** [2]

Scaling Data Diversity for Fine-Tuning Language Models in Human
  Alignment

Feifan Song
Bowen Yu
Hao Lang
Haiyang Yu
Fei Huang
Houfeng Wang
Yongbin Li


1National Key Laboratory of Multimedia Information Processing
School of Computer Science, Peking University
songff@stu.pku.edu.cn; wanghf@pku.edu.cn
2Alibaba Group
yubowen.ybw, hao.lang, yifei.yhy, f.huang, shuide.lyb@alibaba-inc.com

Fei Huang [2] , Houfeng Wang [1] , Yongbin Li [2]


Alignment with human preference prevents large language models (LLMs) from generating misleading or toxic
content while requiring high-cost human feedback. Assuming resources of human annotation are limited, there
are two different ways of allocating considered: more diverse **PROMPTS** or more diverse **RESPONSES** to be
labeled. Nonetheless, a straightforward comparison between their impact is absent. In this work, we first control the
diversity of both sides according to the number of samples for fine-tuning, which can directly reflect their influence.
We find that instead of numerous prompts, more responses but fewer prompts better trigger LLMs for human
alignment. Additionally, the concept of diversity for prompts can be more complex than responses that are typically
quantified by single digits. Consequently, a new formulation of prompt diversity is proposed, further implying a linear
correlation with the final performance of LLMs after fine-tuning. We also leverage it on data augmentation and
conduct experiments to show its effect on different algorithms.
**Keywords:** Human Alignment, Large Language Model, Scaling Law


Large Language Models (LLMs) have gained
widespread recognition for their proficiency in many
domains, including instruction following, intimation and knowledge utilization (Brown et al., 2020;
Chung et al., 2022; Muennighoff et al., 2022; Wei
et al., 2022; Wang et al., 2022a; Zhou et al., 2022;
Von Oswald et al., 2023; Dai et al., 2023; Yang et al.,
2023a; Zhong et al., 2023; Schick et al., 2023; Li
et al., 2023; Song et al., 2023b; Qin et al., 2023;
Wang et al., 2023a; Yang et al., 2023b; Lyu et al.,
2024). However, they can reveal toxic or offensive
content either inadvertently or intentionally, underscoring the importance of aligning them with human values (Bai et al., 2022b). The transition of
paradigm from model-centric to data-centric (Zha
et al., 2023b,a) has led to the development of products that are refined using abundant data with human feedback (e.g., ChatGPT, Claude). These
products show remarkable capabilities in delivering
reliable responses, which prioritizes data collection
for LLM fine-tuning aimed at human alignment.
In this field, a natural challenge is the huge expense of high-quality human annotation for diverse
samples (Casper et al., 2023). The greater the diversity within the dataset is, the higher upper bound
of performance can be achieved. Nevertheless,
this diversity also results in higher costs. In detail,
LLMs are forced to generate responses in line with
  - Corresponding authors
human preference, based on provided prompts.
When the annotation resources are limited, a decision must be made regarding the allocation of these
resources between a broader range of prompts or
a larger number of responses to be annotated, as
illustrated in Figure 1.
The well-known LLaMA-2 (Touvron et al., 2023b)
chooses to utilize samples of human alignment,
each containing one prompt and two responses
to maximize the prompt diversity. On the contrary,
various studies (Ouyang et al., 2022; Yuan et al.,
2023b; Song et al., 2023a) concentrate on providing
each prompt with more responses, enabling LLMs
to distinguish subtle differences among various candidates. Although both sides are intuitively reasonable, there is currently a lack of direct comparison
and comprehensive analyses between them.
In this paper, we investigate the impact of both
prompts and responses for LLM fine-tuning in human alignment. We first design a quantitative experiment to assess the effect of the two strategies.
A series of sub-datasets are created from the raw
dataset, some of which contain more prompts but
fewer responses for each prompt, while others have
more responses for each prompt but fewer prompts.
These subsets maintain a certain proportional relation to maintain a constrained total number of annotations, on which we fine-tune LLMs and compare
their performances for comprehensive analyses.
While Song et al. (2023a) has demonstrated the
effect of increased responses, a scaling law be
(2)
Figure 1: Different directions of data expansion for human alignment: (1) Expanding more prompts;
(2) Expanding more responses for each prompt.
tween prompts diversity and the final performance
in human alignment is yet to be established. Similar to Kaplan et al. (2020) and Muennighoff et al.
(2023) exploring the correlation token statistics and
evaluation metrics, the aforementioned quantitative experiment manipulates prompt diversity by
adjusting training set sizes only, overlooking the
influence of token combinations representing syntax and contextual information. To address this
gap, we introduce a novel formulation to empirically
define prompt diversity based on N-grams. Furthermore, we uncover a linear relationship between
this diversity and the acquired reward scores by
examining various scales of training sets, different
base models, and algorithms.
We also try to enhance data diversity by employing this new formulation to guide a data augmentation process. Beginning with existing samples, we
sample multiple new prompts and corresponding
responses, then assess them based on N-gram
overlap with the given demonstrations to determine
their acceptance. Implementing this method leads
to an improvement in performance compared to
randomly sampled data.
We conclude from all experiments that:
(1) Expanding responses yields more benefit
than prompts. We attribute it to two reasons: just
a few prompts can activate LLMs in human alignment, as explained in Zhou et al. (2023), while more
responses offer clearer signals for fine-tuning, thus
proving more help.
(2) The empirical formulation of prompt diversity
can establish a linear correlation with the final performance of LLMs.
(3) Directed by the proposed formulation of
prompt diversity, the new process of data augmentation can promote the performance of LLMs.


Despite their promising potential, large language
models carry the risk of generating toxic or offensive content without human alignment. One approach that has gained considerable attention in
addressing this issue is Reinforcement Learning
from Human Feedback (RLHF) (Stiennon et al.,
2020; Ouyang et al., 2022; Bai et al., 2022a; Zhu
et al., 2023a,b; Yu et al., 2023). For instance, InstructGPT (Ouyang et al., 2022) builds a three-step
pipeline of RLHF, which includes supervised finetuning (SFT), reward model (RM) training, and reinforcement learning using PPO (Schulman et al.,
2017). This process involves collecting numerous samples, each consisting of one prompt and
multiple candidate responses ranked by human
annotators. These annotated rankings are then
segmented into pairs to enhance computational
efficiency. Touvron et al. (2023b) allocate more
resources to the prompt collection to maximize its
diversity while featuring only two responses per
prompt. Conversely, some works introduce finegrained distinctions to LLMs by incorporating listwise comparisons among responses, or dynamically sampling better candidates for SFT (Yuan
et al., 2023b; Dong et al., 2023; Song et al., 2023a),
also leading to improved performance.
While more prompts can cover a wider range
of domains and topics, limitations in annotation resources often force researchers to choose one side
between diverse prompts and longer rankings with
more responses. In our study, we investigate the
impact of prompt diversity and compare it quantitatively with that of responses. We also establish
empirical relations between prompt diversity and
the final performance of tuned LLMs.


As LLMs continue to increase in scale, leading to
higher training costs, it becomes crucial to make
initial predictions regarding their performance. Various key factors of LLMs can be scaled to predict
the ultimate performance. From a micro perspective, Kaplan et al. (2020) and OpenAI (2023) try to
formulate power laws from the model size or the
amount of computation of LLMs to their converged
loss values during pre-training. In contrast, Lee
et al. (2023) examine the impact of different training paradigms for human alignment from a macro
perspective. Additionally, Zhang et al. (2023) explore how the assembly of LLMs can influence the
final performance, and Yuan et al. (2023a) show
that loss values can even indicate the accuracy of
mathematical reasoning.
The impact of data used in the pre-training or
fine-tuning stages can also be investigated. Kaplan et al. (2020) and Muennighoff et al. (2023)
scale the total number of tokens associated with
the performance levels achievable by LLMs However, token count may not perfectly represent the
diversity of data distribution. Zhao et al. (2023)
accordingly propose a tree-like structure for instruction alignment and study the scaling relationship
between the complexity of instructions and final success rates. Lu et al. (2023) and Wei et al. (2023)
propose different metrics for the estimation data
diversity to label or filter training samples, building upon the observation (Zhou et al., 2023) that
a small dataset can unlock the specific capabilities of LLMs through fine-tuning. Building on these
studies, we concentrate on the distribution of split
prompts and responses for human alignment, and
provide detailed analyses of performance improvement influenced by dataset sizes and diversity.


Different from pre-training, individual samples are
typically divided into a prompt _x_ and a response _y_ in
LLM fine-tuning. Specifically in human alignment,
a single prompt can be associated with multiple
responses _y_ [1:] _[n]_ = _y_ [1] _, y_ [2] _, · · ·, y_ _[n]_, ranked according
to varying levels of preference, which are learned
by LLMs to enhance their outputs.
Intuitively, a broad range of prompts can potentially enhance the generalization ability of LLMs,
thus improving their final performance. Likewise,
using diverse candidate responses can be beneficial by enabling LLMs to capture subtle distinctions reflecting different preferences. It is difficult
to determine the ideal number of samples and the
optimal length of response rankings for LLMs to
align with human preference. However, the fact
is that human annotations are always costly, and
the total amount of annotations can be accordingly
limited. Therefore, given a fixed amount of human
annotations, there has to be a trade-off between
increasing prompts (while reducing the length of
response rankings), or associating each prompt
with more responses (but fewer prompts in total) in
the dataset. Researchers need to make a choice
between these two directions (Ouyang et al., 2022;
Yuan et al., 2023b; Dong et al., 2023; Song et al.,
2023a; Touvron et al., 2023b).
In this section, we design a quantitative experiment aimed at conducting preliminary comparisons
of their effects. We select a series of subsets for
fine-tuning, all sharing the same total annotation
volume, some emphasizing more prompts while
others prioritize more responses. Subsequently,
we apply two well-known algorithms to these subsets and aggregate their performance results to
assess the impact of different configurations.


Similar to Yuan et al. (2023b), Rafailov et al. (2023)
and Song et al. (2023a), we utilize the **Human**


**ness**, referred to as _HH-RLHF_ (Bai et al., 2022),
as the foundational dataset. Each original sample
consists of a common prompt and two candidate
responses (named 2-ranking), one chosen by human annotators and the other rejected. We extend
each 2-ranking into 4-ranking through zero-shot
augmentation using Curie (Brown et al., 2020) and
Alpaca (Taori et al., 2023), neither of which has
been fine-tuned for human alignment previously.
Assuming the total volume of human annotations is 2N, there are various subsets with different
prompt sizes and response ranking lengths. For
example, each subset may consist of N prompts
with 2-rankings, 2N/3 prompts with 3-rankings, or
N/2 prompts with 4-rankings, which all maintain 2N
annotations (2N=2 _×_ N=3 _×_ 2N/3=4 _×_ N/2).
We also attach additional subsets containing N
prompts, 2N/3 prompts, and N/2 prompts with 2/3/4rankings, to present comprehensive results for further analyses.

Preference Data on Helpfulness and Harmless-


Unlike some tasks that can be easily measured, human preference can be more abstract and hard to
estimate. Both Yuan et al. (2023b) and Song et al.
(2023a) utilize RMs to evaluate the performance of
fine-tuned LLMs, while the emerging GPT-4-as-ajudge in human alignment (Rafailov et al., 2023) can
be also convincing. Our evaluation predominantly
relies on public reward models, employing distinct
reward models RMtrain and RMtest for training and
testing phases, respectively. The outcomes are
**Settings** **# Candidate** **# Prompts** (N=24000) **# Prompts** (N=3000)
(Algorithm, Backbone, Domain) **Responses**
N/2 2N/3 N N/2 2N/3 N
2 55.58 55.58 **57.01** 50.42 51.37 **53.65**
PRO, OPT-1.3B, Harmless 3 57.11 **57.29** 59.28 53.28 **54.80** 56.47
4 **59.24** 58.98 59.92 **55.36** 56.73 58.28
2 49.05 49.09 **50.06** 44.69 45.37 **46.49**
PRO, OPT-1.3B, Helpful 3 49.98 **51.00** 51.43 45.74 **47.01** 49.73
4 **51.35** 51.04 51.74 **48.85** 48.69 50.33
2 52.78 52.73 **53.78** 47.64 48.57 **50.13**
PRO, OPT-1.3B, Global 3 53.91 **54.67** 55.51 49.56 **50.76** 53.28
4 **55.30** 55.12 55.96 **52.28** 52.54 54.19
2 52.25 52.78 **52.63** 49.85 49.33 **50.47**
SFT, OPT-1.3B, Global 3 53.60 **54.18** 54.20 51.59 **51.26** 51.35
4 **55.06** 55.00 56.27 **51.93** 51.97 53.03
2 54.53 54.68 **55.16** 53.16 53.06 **53.31**
PRO, LLaMA-7B, Global 3 56.58 **55.89** 56.26 54.74 **53.00** 56.05
4 **57.50** 56.42 57.26 **55.25** 55.88 55.35
2 53.18 53.29 **54.63** 52.78 53.03 **53.80**
SFT, LLaMA-7B, Global 3 56.01 **55.53** 56.07 53.98 **53.74** 54.50
4 **56.13** 56.30 57.30 **55.05** 55.00 56.14
Table 1: Results of quantitative experiments. LLMs can acquire better performance with either diverse
prompts or responses used for fine-tuning, while increasing responses benefits LLMs more than increasing
prompts with the same amount of annotations (highlighted in red bold).
then cross-validated by GPT-4 assessments. For
fine-tuning, we combine all 4 subsets in _HH-RLHF_
for LLM fine-tuning and present the outcomes for
two representative subsets, namely _Harmless_ base
and _Helpful_ base. Furthermore, we provide the overall scores for all test samples across the 4 subsets.


For each dataset, we select representative supervised methods as benchmark algorithms, because
supervised training can directly reflect the impact
of the datasets involved. Specifically, we opt for
two widely-used algorithms, namely **Supervised**


**tion** (Song et al., 2023a), denoted SFT and PRO,
to represent other methods that are either sensitive
or insensitive to response rankings.
To elaborate, SFT is similar to the pre-training
process but exerts supervision solely on the top
candidate _y_ [1],
_|_ _[y]_ [1] _|_
_L_ SFT( _y_ [1] _| x_ ) = _−_   - log _p_ Θ   - _yi_ [1] �� _x, y<i_ 1   - (1)
_i_ =1
Instead, PRO forces the LLM to distinguish the
best one from multiple candidates. It utilizes
the whole ranking _y_ [1] _, y_ [2] _, · · ·, y_ _[n]_ through multiple
one-to-many contrasts, implemented as:
_Tk_ _[k]_
_L_ ( _y_ [1:] _[n]_ _| x_ ) = _−_
_n−_ 1 exp - log _p_ Θ( _x,y_ _[k]_ )
_k_ �=1 log ~~�~~ _n_ exp ~~�~~ log _T pk_ _[k]_ Θ( _x,y_
_n−_ 1
_n_
~~�~~
_i_ ~~�~~ = _k_ exp ~~�~~ log _p_ Θ _Tk_ ( _[i]_ _x,y_ _[i]_ )
_Tk_ _[i]_
(2)
~~�~~
_Tk_ _[i>k]_ = _rϕ_ ( _x, y_ _[k]_ ) _−_ _rϕ_ ( _x, y_ _[i]_ ) (3)
_Tk_ _[k]_ [= min] _k_ (4)
_i>k_ _[T][ i]_
and the final objective appends _L_ SFT for a balance
between text quality and human preference,
_L_ PRO( _y_ [1:] _[n]_ _| x_ ) = _βL_ SFT( _y_ [1] _| x_ ) + _L_ ( _y_ [1:] _[n]_ _| x_ ) (5)

Fine-tuning and Preference Ranking Optimiza-


The experiments are conducted with different
_N_ (24000 and 3000). We mainly utilized OPT1.3B (Zhang et al., 2022) as the base LLM and
tested it with three different seeds, while incorporating LLaMA-7B (Touvron et al., 2023a) with just
one seed due to computational constraints. For
the fine-tuning process of LLMs, we configured the
total training steps to 4000 for each dataset, performing validation every 500 steps. Both RMtrain [1]
and RMtest [2] are publicly available checkpoints.
1 `[https://huggingface.co/OpenAssistant/](https://huggingface.co/OpenAssistant/oasst-rm-2.1-pythia-1.4b-epoch-2.5)`
```
oasst-rm-2.1-pythia-1.4b-epoch-2.5
```
2 `[https://huggingface.co/OpenAssistant/](https://huggingface.co/OpenAssistant/oasst-rm-2-pythia-6.9b-epoch-1)`
```
oasst-rm-2-pythia-6.9b-epoch-1
```
Figure 2: Distribution of BLEU scores with different
settings.
The original dataset comprises newly added data
without human annotations. Therefore, we first
score all responses using RMtrain, then re-rank them
based on their scores. Furthermore, we ensure
that datasets of larger sizes will encompass their
smaller counterparts with rankings of the same
length, while for datasets of the same size, longer
rankings will include the shorter ones for each sample. More details are available in the code [3] .


As demonstrated in Table 1, we gathered the average reward scores corresponding to various settings, including algorithms, backbones, and domains. These results aim to address the following
research questions (RQs).


Longer response rankings and more prompts are
both beneficial, but their effects are different. Generally, with an equal quantity of annotations, extending each ranking of responses leads to **better**
enhancement compared to expanding prompts, regardless of the backbones used or the values of N.
These outcomes are highlighted in red bold within
each grid, where scores in the **lower left** side indicate longer response rankings but fewer prompts,
while scores in the **upper right** side represent the
opposite scenario. This observation is compatible
with the hypothesis that LLMs possess the potential
for human alignment that can be activated with a
small number of samples (Zhou et al., 2023). However, more responses for each prompt offer clearer
alignment signals through comparisons, leading to
a more significant optimization of LLMs.
In detail, an increased number of responses benefits PRO more than SFT, as the former emphasizes the importance of response rankings lacking
3 `[https:github.com/F2-song/](https:github.com/F2-song/ScalingAlignment)`
```
ScalingAlignment
```
in the latter. Nonetheless, a longer ranking represents more samplings from the whole linguistic
space, where the preferred response is more likely
to be identified. This explains why SFT methods
also benefit from expanding responses. Further
examination of the results in the domains of harmlessness and helpfulness with OPT-1.3B and PRO
confirms the validity of the above statement.


Changes in the allocation of annotations (more
prompts or more responses) appear to have no
impact on the outcome. We hereby plot the BLEU
distribution generated by the OPT-1.3B model finetuned with PRO and 24000 samples in Figure 2,
where the distribution does not present a consistent pattern but fluctuates randomly across different
configurations. The variations observed may be
explained by the fact that LLMs possess robust
language modeling abilities, consequently not requiring too many samples. It underscores the importance of allocating more annotation resources
to prepare responses when a certain number of
prompts have been guaranteed.


Intuitively, the more samples for fine-tuning, the
more diverse they are, leading to potentially greater
improvements for the tuned LLMs. However, determining the adequate quantity is a complex task, as
it depends on factors like the algorithms, base models, and the number of responses. For instance,
while LLaMA-7B demonstrates notably high scores
with 3000 samples, surpassing OPT-1.3B with an
equal amount of training data, it shows a slower
increase in performance compared to OPT-1.3B
when more samples are included. Moreover, the
degree of improvement achieved by increasing the
dataset size usually decreases. This is because the
dataset is more likely to contain duplicate or similar
content as the number of samples grows, making
it less efficient to continually invest resources in
comparison to potential performance gains.


Apart from the automatic evaluation above with
RMtest, we also take GPT-4-as-a-judge into consideration, since it has been widely recognized as
an efficiently human-like tool to give fair judgment,
especially for abstract concepts like human preference (Zheng et al., 2023; Song et al., 2023a;
Dubois et al., 2023). It further validates the statistical findings in Table 1 by directly comparing the
three settings as described below:
N=24000 N=3000 N=24000 N=3000
1 2 3 1 2 3 1 2 3 1 2 3
SFT PRO
Figure 3: GPT-4 Evaluation
**Setting 1:** N prompts, i.e., a total of N samples,
each with 2 responses.
**Setting 2:** 2N/3 prompts, each with 3 responses.
**Setting 3:** N/2 prompts, each with 4 responses.
All three settings have a total of 2N annotations for
fine-tuning.
We use LLaMA-7B for fine-tuning under these
three settings since it can yield high-quality outputs.
We randomly select prompts from the test sets of
_Harmlessbase_ and _Helpfulbase_ for evaluation. The
outputs of each tuned LLaMA are compared with
those under other settings, scored directly by GPT4 through bi-directional comparisons to enhance
fairness (Wang et al., 2023b), and the win-lose
rates of each comparison are depicted in Figure 3.
In each matrix _M_, each row _i_ (or column _j_ ) corresponds to Setting _i_ (or Setting _j_ ), with the element
_Mi,j_ indicating the win rate of LLM outputs tuned
with Setting _i_ against those with Setting _j_ . The diagonal elements in _M_ are uniformly set at 33.33
for comparisons between two identical contents,
distributed evenly among [Win, Lose, Tie].
Figure 3 illustrates that _Mi,j_ always surpasses
_Mj,i_ along the main diagonal. This implies that
the win rate of Setting _i_ against Setting _j_ is always
higher than its loss rate, consistent with the results
in Table 1. This also proves that RMtest can be a
reliable evaluator. In general, it reaffirms the conclusion that increasing annotations for responses
improves LLMs to better align with human preference than prompts.


While the diversity of prompts or responses can
both be beneficial to LLMs fine-tuning, increasing
prompts is less effective compared to increasing
responses. This difference can be attributed to
the inefficiency of using quantity alone to measure
prompt diversity. In this section, we explore the concept of prompt diversity. We first discuss the impor
tance of controlling prompt diversity and then propose a new empirical formulation for it. Additionally,
assuming all other factors remain constant (such
as base models, fine-tuning algorithms, annotation
sources, and lengths of response rankings), a **lin-**
**ear correlation** between the final performances of
fine-tuned LLMs and the calculated diversity from
various subsets can be illustrated.


Utilizing quantity to control response diversity appears to be rational. Given that new responses
typically originate from various sources, and the
range of response rankings is relatively limited, augmenting the number of responses can lead to significant variations. The quantitative experiments
also demonstrate that expanding response rankings contributes to improvements for LLMs.
However, the approach becomes oversimplified
when applied to prompt diversity. Here the diversity
is based on all prompts, and minor adjustments in
quantity may not have a noticeable impact, as evidenced by the marginal improvement from N/2 to
2N/3 (for N=3000) in the quantitative experiments,
while potential duplication in prompts can be another factor. Furthermore, fine-grained features
within utterances, such as semantics, contexts, and
even syntax, are crucial to prevalent LLMs that depend on tokenization followed by causal modeling.
They should also be taken into consideration.
Moreover, the redundant prompts mentioned earlier do not contribute significantly to enhancing overall diversity and should be initially removed. This
can be measured by assessing the proportion of
distinct N-grams within the dataset.
Different from Kaplan et al. (2020) and Muennighoff et al. (2023), we leverage N-grams instead
of individual tokens as the basic element for calculation, because N-grams inherently capture contextual details beyond the meaning of single tokens.
In addition, the aforementioned duplicated data essentially do not provide any extra advantages in
(c)
(d)
Figure 4: (a) Linear fitting from different sample amounts to finally acquired rewards of LLMs tuned with
**PRO** . (b) The trend of diversity with the increasing sample amount. (c) Linear fitting from the proposed
diversity metric to finally acquired rewards of LLMs tuned with **PRO** . (d) Linear fitting from the proposed
diversity metric to finally acquired rewards of LLMs tuned with **SFT** .
terms of overall diversity and should be filtered out
initially. We define it as the rate of unique N-grams
present in the dataset,
_r_ unique = _[|]_ [Filter][(] _[G]_ [)] _[|]_ (6)
_|G|_
where _G_ is the collection of all N-grams derived
from the tokenized corpus and Filter( _G_ ) denotes
the removal of repeated elements. Subsequently,
the diversity metric _d_ can be formulated as product
of _r_ unique and the total number of prompts _m_,
_d_ = _r_ unique _∗_ _m_ (7)
Empirically, as the number of prompts increases,
the marginal effect decreases gradually, and the
diversity should follow the same pattern. Therefore,
we introduced a decay index _p_ to the sample quantity _m_ to incorporate decay into its growth rate. The
concept of prompt diversity is formulated as,
_d_ = _r_ unique _∗_ _m_ _[p]_ (8)


To examine the connection between the pre-defined
diversity metric and the final performance of finetuned LLMs, we use 2-grams for calculation and
collect {1500, 2000, 3000, 6000, 12000, 16000,
24000, 36000} samples from the original dataset
with rankings of lengths 2/3/4. This resulted in
24 subsets for LLMs fine-tuning. In this part, we
empirically set the value of _p_ to 0.5 for _HH-RLHF_,
although it could also be found using grid search.
We start our analysis by visualizing the results of
above 24 subsets in Figure 4(a). A discernible positive correlation between enhanced performance
and the increasing quantity is observed. Furthermore, improved scores can also be achieved with
longer response sequences. Nevertheless, the
growth in reward scores and quantity of prompts
is not synchronized. The former shows a gradual decline in speed, while the latter maintains a
consistent pace. More precisely, the performance
experiences a sharp increase with a rising number
of samples at the beginning, yet tends to plateau
with a larger volume of samples. Even when we
convert the X-axis from sample quantity to token
quantity, this conclusion still holds.
A commonly accepted concept is that the diversity of a dataset may not continue to increase indefinitely. As the size of the dataset expands, new content often contains complete or partial duplications
of earlier material. By analyzing actual datasets, we
have graphed in Figure 4(b) the evolution of diversity outlined in Equation 8 as the sample size grows.
This graph aligns with the idea that the rate of diversity growth should gradually decrease. Based on
the similar patterns observed in performance and
**Algorithm 1:** Determining the decay index
with grid search.
**Input:** Fine-tuning algorithm _FT_,
Datasets _{Di}_ of ascending sizes,
Language Model _π_, Step Length _l_
**Output:** Decay Index _p_
```
 // Fine-tuning and evaluation
```
**1** Let _S_ be an empty set


**3** Let _πi_ = _FT_ ( _π, Di_ )
**4** Evaluate _πi_
**5** Let _si_ be the performance of _πi_
**6** Add _si_ to _S_


```
 // Searching the decay index
```
**8** Let _p_ = 0, ˆ _p_ = 0, _L_ = _inf_


**10** Let ˆ _p_ = ˆ _p_ + _l_
**11** Compute diversity degrees _{di}_ based
on ˆ _p_, _{Di}_ and Equation 8
**12** Compute the MSE _L_ [ˆ] using linear fitting
on _{di}_ and _S_


**17 return** _p_
diversity trends, a linear correlation between these
them may exist,
_r_ = _α ∗_ _d_ + _β_ + _ϵ_ (9)
where _α_ and _β_ are coefficients, while _r_ and _ϵ_ denote
the reward score and error term, respectively.
We then gather the performance (i.e., average
reward) achieved by OPT-1.3B under the supervision of PRO and SFT, respectively, and then apply
linear fitting to correlate each final score with the
computed diversity of the specific subset used. The
outcomes are presented in Figure 4(c) and (d). Additionally, we compare these results with the linear
fitting between performance and the sample quantity, as shown in Figure 4(a), while it leads to a
significantly higher mean squared error (MSE) of
1.2e-4 compared to 2.8e-5 and 8.8e-6 corresponding to Figure 4(c) and (d), respectively. This validates the linear correlation between our proposed
_d_ and the final performance. We also compute the
MSE values with LLaMA-7B, which are marginally
higher than those with OPT-1.3B, possibly due to
fluctuations in a single seed (4.6e-5 for PRO and
1.7e-5 for SFT).


In this section, we present a technique for data
augmentation using the existing samples. Fresh
samples are first collected and then selected to enrich the overall variety of prompts. This selection is
aimed to optimize the local diversity between the
new and existing samples. We initially demonstrate
the effectiveness of this technique in enhancing
prompt diversity. By fine-tuning LLMs on the augmented datasets, there is also a slight performance
enhancement along with increased diversity.


We design the data augmentation as where there
are existing samples that constitute a seed set. In
this setting, _n_ samples are randomly selected to
support each augmentation iteration. To simplify
the experiment, we reuse one subset _D_ in the last
section as the aforementioned seed set, and select
new samples from the original _HH-RLHF_ to simulate the process of augmentation. It is ensured
that each new sample is selected from the nonoverlapping portion of _HH-RLHF_ concerning _D_ .


We start by revisiting the concept of prompt diversity. The proposed metric above can be affected
by two factors: _r_ unique, representing the ratio of
unique N-grams, and the total number of prompts,
which rises with decreased speed. Consequently,
by simultaneously increasing _r_ unique during data
augmentation, the diversity metric can experience
a more rapid growth.
However, identifying a batch of new samples that
maximizes the diversity of the total _D_ can be challenging. Therefore, we introduce a locally greedy
search process to filter new samples based on the
supporting samples. Specifically, by computing the
Jaccard Index between the set _X_ of supporting
samples and the _i_ -th element _Yi_ of set _Y_,
Jaccard _i_ = [Filter][(] _[G][X]_ [)] _[ ∩]_ [Filter][(] _[G][Y][i]_ [)] (10)
Filter( _GX_ ) _∪_ Filter( _GYi_ )
where _Yk_ with the lowest Jaccard _i_ can be selected
to enhance the local _r_ unique, thereby boosting the
overall diversity.


We set _n_ as 2 and utilize the subset _D_ 6000 containing 6000 samples (for all versions with 2/3/4
rankings), to which 6000 new samples are then
added, forming _D_ [ˆ] 12000. Meanwhile, we treat the
subset _D_ 12000 as its counterpart, comprising 12000
randomly sampled samples without filtration and


Table 2: Results of Data Augmentation.
covering _D_ 6000. We apply PRO and SFT on OPT1.3B using these subsets and present the outcomes
in Table 2. The diversity of prompts increases from
18.86 to 19.75, and slight enhancements are shown
with few exceptions, which can be normal fluctuations. This primarily demonstrates the impact of
the proposed filtering method. It may potentially
be amplified with larger _n_, a direction we leave for
future research.


This study focuses on the impact of data diversity on human alignment fine-tuning. Given the
common limitation of available annotations in most
scenarios, we investigate the effect of distributing
them to enhance diversity in different ways, such
as increasing prompts or responses. Our extensive experiments show that increasing the number
of responses generally leads to greater enhancements in human alignment compared to expanding
prompts. Additionally, we design an empirical metric to measure prompt diversity and reveal a linear
correlation between it and the final performance
of LLMs. Finally, we propose a straightforward
method to boost diversity in data augmentation, resulting in better performance of fine-tuned LLMs.


The presence of sensitive and offensive content
within the datasets should be acknowledged. It is
important to highlight that these contents do not
reflect our views or beliefs, but are solely intended
for research purposes.
Moreover, Curie inference and GPT-4 evaluation
are utilized where they are available to adhere to
legal requirements.


This work was supported by National Science
and Technology Major Project (2022ZD0116308)
and National Natural Science Foundation of China
(62036001).


**Shu Yang** [1,2,6] **, Jiayuan Su** [1,2,5] **, Han Jiang** [1,2,7] **, Mengdi Li** [1,2,4] **, Keyuan Cheng** [1,2,3] **,**
**Muhammad Asif Ali** [1,2] **, Lijie Hu** [1,2] **, and Di Wang** [1,2]
1Provable Responsible AI and Data Analytics (PRADA) Lab
2King Abdullah University of Science and Technology
3South China University of Technology 4University of Hamburg
5Zhejiang University 6University of Macau 7Tongji University

Dialectical Alignment: Resolving the Tension of 3H and Security Threats
  of LLMs

Dialectical Alignment: Resolving the Tension of 3H and Secu- rity Threats of LLMs

Shu Yang
Jiayuan Su
Han Jiang
Mengdi Li
Keyuan Cheng
Muhammad Asif Ali
Lijie Hu
Di Wang


With the rise of large language models (LLMs), ensuring they embody the
principles of being helpful, honest, and harmless (3H), known as Human
Alignment, becomes crucial. While existing alignment methods like RLHF,
DPO, etc., effectively fine-tune LLMs to match preferences in the preference dataset, they often lead LLMs to highly receptive human input and
external evidence, even when this information is poisoned. This leads to
a tendency for LLMs to be _Adaptive Chameleons_ when external evidence
conflicts with their parametric memory. This exacerbates the risk of LLM
being attacked by external poisoned data, which poses a significant security
risk to LLM system applications such as Retrieval-augmented generation
(RAG). To address the challenge, we propose a novel framework: _Dialectical_
_Alignment (DA)_, which (1) utilizes AI feedback to identify optimal strategies
for LLMs to navigate inter-context conflicts and context-memory conflicts
with different external evidence in context window (i.e., different ratios
of poisoned factual contexts); (2) constructs the SFT dataset as well as the
preference dataset based on the AI feedback and strategies above; (3) uses
the above datasets for LLM alignment to defense poisoned context attack
while preserving the effectiveness of in-context knowledge editing. Our
experiments show that the dialectical alignment model improves poisoned
data attack defense by 20% and does not require any additional prompt
engineering or prior declaration of “you may be attacked“ to the LLMs’
context window.


Large language models (LLMs) trained on large datasets and with significant computational
resources have shown unprecedented capabilities (Brown et al., 2020; Kaddour et al., 2023;
Zhao et al., 2023; Bubeck et al., 2023). For instance, Claude-3’s latest release exhibits tentative
self-awareness [1] in the Needle In A Haystack eval [2], highlighting the importance of further
enhancing the security, user-friendliness, and controllability of LLMs (See et al., 2019; Wang
et al., 2023a;d). A straightforward way to achieve these goals is to align LLMs’ _behavior_ with
_human feedback_ (Stiennon et al., 2022; Ouyang et al., 2022).
Existing alignment methods first allow humans (or LLMs) to select preferred model
responses based on specific criteria. Then, reinforcement learning techniques, e.g.,
RLHF (Ziegler et al., 2020; Ramamurthy et al., 2022; Stiennon et al., 2022), RLAIF (Bai et al.,
2022b; Lee et al., 2023; Chu et al., 2023), and Direct Preference Optimization (DPO) (Rafailov
et al., 2023) methods are used to train LLMs towards preference-specific behaviors. However, utilizing the existing widely used helpful, honest, harmless (3H) preference criteria
1 `[https://www.anthropic.com/news/claude-3-family](https://www.anthropic.com/news/claude-3-family)`
2 `[https://github.com/gkamradt/LLMTest_NeedleInAHaystack/tree/main](https://github.com/gkamradt/LLMTest_NeedleInAHaystack/tree/main)`
Preprint. Under review.
to align LLMs poses potential risks as follows: (i) In terms of the model performance,
over-optimizing 3H rewards according to Goodhart’s law [3] may hamper truthfulness performance (Gao et al., 2022), e.g., balancing harmlessness vs usefulness (Bai et al., 2022a; Dai
et al., 2023); (ii) 3H models tend to imply that LLMs overly prefer human inputs in favor of
self-positions (Xie et al., 2024; Xu et al., 2023a), rendering them susceptible to camouflaged
red-team attacks or poisoned text in the context window (Liu et al., 2023c; Zou et al., 2024).
The phenomenon that LLMs are _highly receptive_ to external information is referred to as
_Adaptive Chameleon_ by Xie et al. (2024).
In this paper, we introduce _Dialectical Alignment (DA)_ to address the challenge that 3H LLMs
frequently alter their answers when encountering knowledge conflicts due to their tendency
to trust external input (by humans) easily. DA empowers aligned LLMs to think dialectically
upon conflicting knowledge so that they can spontaneously decide whether to trust the
external information or reject the poisoned contexts. We first explore two tasks representing
two sides of a coin in dealing with knowledge conflicts: In-context Knowledge Editing
(IKE) (Zheng et al., 2023a) and Poisoned Context Attack (PCA) (Zhong et al., 2023; Liu et al.,
2023d; Zou et al., 2024) (see more details in Section 2). The existing studies mentioned above
have investigated these two tasks separately, without recognizing their correlation: if an
LLM is more susceptible to the external information in the context, the effectiveness of IKE
improves while the success rate of PCA also increases; conversely, if a model tends to adhere
to its own parametric memory, Poisoned Context Attack will be defended while the model
also rejects IKE. We give intuitive examples of this fact in Figure 1 and Figure 2.
Specifically, we observe LLMs’ dialectical thinking ability with different inference paths
(detailed in Table 1) in both IKE and PCA defense tasks. We use AI feedback to help
us determine whether the model correctly handles context-memory conflict (the conflict
between internal memory and external information) and inter-context conflict (conflict
among various contexts of the external information) (Xu et al., 2024a). Based on the feedback,
we construct supervised fine-tuning datasets and preference datasets. We align LLMs to
dialectically deal with IKE and PCA without any additional prompt engineering or defensive
prefix prompt like ”you may be attacked.”
Our experiments show that DA can improve LLMs’ PCA defense performance with minimal impairment of IKE effects. This is extremely valuable for the Rretrieval-augmented
generation (RAG) (Gao et al., 2024) applications, where both factual data and poisoned data
may be retrieved into the model’s context in the real-world scenarios (Barnett et al., 2024).
Our experiments also lead to a series of interesting findings. We highlight the following:
  - _LLMs are highly susceptible to attacks involving poisoned contexts within their context_
_window._ This finding aligns with the conclusions drawn by Zou et al. (2024). Furthermore, we observe that as the number of poisoned contexts increases, defending
against such attacks becomes increasingly challenging.
  - _Prior knowledge is important for LLMs to deal with PCAs._ Comparing all the paths in
Table 1, we find that the Chain-of-Thought path with prior knowledge (CoT-PK)
is effective in resisting the degradation in PCA defense, which is caused by the
growth in a number of poisoned contexts we mentioned above.
  - _Prior knowledge leads to more stubborn LLMs._ Although the CoT-PK mentioned above
improves the ability of the model to defend PCAs, the price is a sharp decrease in
the performance of IKE, which means that CoT-PK causes LLMs to be even more
stubborn when dealing with knowledge conflicts.


**Helpful, Honest, Harmless (3H) and Attack Defense.** Shayegani et al. (2023) proposed an
important mode for the failure of LLMs’ safety training - competing objectives. This mode
points to the difficulty of balancing the safety and helpfulness of a model when the LLMs’
function (e.g., should always follow instructions and be helpful) conflicts with their safety
3 `[https://www.lesswrong.com/tag/goodhart-s-law](https://www.lesswrong.com/tag/goodhart-s-law)`
Preprint. Under review.
objectives (Wei et al., 2023a; Shen et al., 2023b). An overly security-trained LLM can easily
reject innocuous user instructions (Ganguli et al., 2022) (e.g., role-playing), while complex
jailbreak instructions can easily attack a model that lacks security training. On top of this,
we presented a more indistinguishable example of knowledge editing and poisoned context
attacks in Figure 1, which both exploit the model’s property of following user instructions
as well as trusting the human input but achieve different goals (edit vs. attack). Existing
alignment work does not focus on the balance between these two, while our Dialectical
Alignment trains models to have the ability to spontaneously make judgments for trust the
input or not.
Figure 1: **In-context Knowledge Editting(left) and Poisoned Context Attack (right) are**
**two sides of the coin.** An intuitive example illustrates that both knowledge editing based
on In-context Learning (Zheng et al., 2023a) and attacking a model by injecting posioned
data into the context window of LLMs are essentially identical (Zhong et al., 2023; Zou et al.,
2024), differing only in the content of the information in the model’s input and the user’s
purpose.
**In-context Knowledge Editing and Poisoned Data Attack** In-context knowledge editing
(IKE) is a novel strategy for LLMs’ knowledge editing without retraining (Zheng et al.,
2023a). Compared to other parameter-updating approaches (Yao et al., 2023), it effectively
adapts the factual knowledge in language models without parameter updating and with
fewer unwanted side-effects (Onoe et al., 2023; Cohen et al., 2023). Despite the effectiveness
brought by in-context learning (Brown et al., 2020), uncurated external information also
introduces hazards to LLMs. When encountering knowledge conflicts between context and
parametric memory (Xu et al., 2024a), studies by Qian et al. (2023) and Xie et al. (2024) reveal
that LLMs are more inclined towards external evidence, especially when it appears coherent
and convincing. Consequently, similar to various context-based attacks (Liu et al., 2022;
Mei et al., 2023; Liu et al., 2023c; Toyer et al., 2024; Schulhoff et al., 2023), malicious users
could easily exploit IKE to attack LLMs with false information (Zou et al., 2024), which is
namely poisoned context attack (Lukas & Kerschbaum, 2023). In an effort to mitigate the
risk of data poisoning attacks, some defense methods are proposed for pre-trained language
models (Zhang et al., 2022; Wang et al., 2022a; Jia et al., 2022; Wang et al., 2022b; Wang
& Feizi, 2023). Chen et al. (2022) conduct a calibration study to discourage models from
providing a single answer when confronted with multiple conflicting pieces of evidence.
However, there is limited research addressing the potential danger associated with IKE.
Our work aims to leverage the reasoning capability of LLMs to dialectically reassess the
information factuality in their context window, thereby offering an aligned approach to
resolving knowledge conflicts in the retrieval augmentation of LLMs.


**From Human Feedback to AI Feedback.** RLHF pipeline (Ziegler et al., 2020; Stiennon
et al., 2022) is effective at aligning LLMs to human preferences, but gathering high-quality
human preference labels is a key bottleneck. Bai et al. (2022b) proposed Constitutional
AI for defining a set of principles to guide LLMs in self-criticizing and improving and
thus collecting preference data. RLAIF (Lee et al., 2023)(Reinforcement Learning from AI
Preprint. Under review.
Context-memory conflict
Context-memory conflict Context-memory conflict
Dialectical LLM
Both Edit and Attack Succeed
Edit Succeed   Attack Rejected
Both Edit and Attack Rejected
Figure 2: **Dialectical LLMs.** An intuitive example illustrates that both knowledge editing
and poisoning data attacks essentially exploit Context-memory conflict. Merely being
friendly can lead to LLMs tending to believe the input and change their views (left one),
while excessive defensiveness can result in models becoming stubborn because they question
external data (right one). However, by learning via our Dialectical Alignment, the model
can decide when to update and when to defend (middle one).
Feedback) trains language models using preferences labeled by an AI system instead of
humans. Lee et al. (2023) indicate that RLAIF achieves comparable performance to RLHF.
More proximately, AI feedback data solves the data bottleneck of LLM preference learning,
but first training a reward model and then reinforcement learning is still a complex and
unstable process. Rafailov et al. (2023) propose DPO, which directly optimizes for the policy
best satisfying the preferences with a simple classification objective.
**Knowedge Conflicts.** Xu et al. (2024a) categorized existing conflicts in large model knowledge into three types: context-memory, inter-context, and intra-memory conflict. Retrieval
augment generation has become mainstream in existing applications of LLMs (Lewis et al.,
2020; Gao et al., 2024). However, external information is often erroneous and noisy (Chen
et al., 2023a; Liu et al., 2023b; Ali et al., 2020; 2021; 2019), thus context-memory and intercontext conflicts increasingly impact the credibility of LLMs generated content. In our
experiments, we emphasize both of these conflicts. In our scenario, inter-context conflict
manifests when both correct factual information and poisoned contexts are simultaneously
input into the LLM’s context window. Context-memory conflict, on the other hand, arises
in our experiments due to conflicts between prior knowledge memorized by the model
parameters and external information, Figure 2 offers an intuitive example.


To address the challenge of getting LLMs to balance IKE and PCA defense mentioned above,
unlike prior alignment work that focuses on making the model helpful and harmless, and
using offensive or disturbing topics as red team behaviors (Bai et al., 2022a; Ganguli et al.,
2022), our goal is to defend more insidious red team behaviors, where they exploit the
human-friendly and instruction-following nature of the aligned LLMs for the attacks (Wei
et al., 2023b; Zou et al., 2024). Specifically, we attempt to align the LLMs to dialectically
recognize the user’s purpose (attack or not) and thus selectively choose to believe or reject
the input in their context window. Below, we provide precise details regarding our end-toend _Dialectical Alignment_ framework. See Figure 3 for an illustration.
4.1 Base Response Evaluation
In STEP 1 in Figure 3, the initial LLM responds to external information and corresponding
questions by the instruction with named `Base` in Table 1, which may include poisoned
context or factual evidence, or a combination of both in LLMs’ context window. At the same
Preprint. Under review.
Figure 3: **Dialectical Alignment framework.** In STEP 1, we use the **Base** instruction in
Table 1 to enable the model to answer questions based on the provided context, which
may consist of factual or poisoned information (or a combination of both). Once the model
provides answers, we use a SOTA LLM as the ground truth to assess the _Accuracy (ACC)_ of
the answers (referred to as AI feedback in this figure). In STEP 2, we use other dialectical
paths in Table 1 to prompt LLMs and provide AI feedback again. Based on this, we select
the optimal dialectical path (e.g., CoT-PK, as illustrated in the figure) for different contexts.
In STEP 3, we refine these paths using another SOTA LLM if the response is unnatural.
Finally, in STEP 4, we construct a supervised fine-tuning dataset using paths corresponding
to higher ACC and fine-tune the model.
|Dialectical Path|Primary Details|
|---|---|
|Base Generate with Extra
Contexts (**Base**)|`Given Relevant:`_ {_`context`_}_`.`
`Query:`_ {_`question`_}_|
|Generate with Tips (**Tips**)|`Given Relevant:`_ {_`context`_}_`.`
`Dialectical KEY: The contexts might be`
`correct or incorrect.`
`Query:`_ {_`question`_}_|
|Generate with Base CoT
(**Base CoT**)|`Given Relevant:`_ {_`context`_}_`.`
`Dialectical KEY: The contexts might be`
`correct or incorrect.`
`Think step by step Query:`_ {_`question`_}_|
|Generate with Given No
Prior
Knowledge
CoT
(**CoT-NoPK**)|`Given Relevant:`_ {_`context`_}_`.`
`Dialectical KEY: Follow the steps below:`
`1.Judge the accuracy of the content; 2.Decide whether or not to refer`
`to this content; 3.Give the correct answer`
`Query:`_ {_`question`_}_|
|Generate with Given Prior
Knowledge CoT (**CoT-PK**)|`Part 1:`
`Entity Extract Extract the key concepts in the given`
_{_`question`_}_` by format`_ \_`[’concept1’,’concept2’,...]`
`Part 2:`
`Long Context Generate Tell me what you know about`
_{_`concept`_}_
`Part 3:`
`CoT Generate Retrieved Contexts:`
_{_`context`_}_
`Dialectical`
`KEY:Follow the steps below:`
`1.Judge the accuracy of the content`
`based on context generated in Part 2; 2.Decide whether or not to`
`refer to this content; 3.`
`Give the correct answer Query:`
_{_`question`_}_|
Table 1: **Dialectical Paths.** The reasoning strategies of the large language model to generate
the answer to the question increase in complexity from top to bottom. _{_ context _}_ represents
external evidence integrated into the LLM’s context window, while Dialectical KEY emphasizes the key aspects of these dialectical paths. For detailed instructions, refer to Table 8 in
the Appendix.
time, a more robust LLM serves as the evaluator for these responses. In our experiments,
the evaluator is GLM-4 from Zhipu AI [4] . See Appendix A for more discussion on LLM in
evaluation. The selection process identifies poisoned responses, comprising those influenced
by poisoned context, resulting in incorrect answers and responses containing factual data
but yielding incorrect answers. The goal of this step is to establish a baseline of not providing
any hints to the model that the external information may be incorrect.
4 `[https://zhipuai.cn/devday](https://zhipuai.cn/devday)`
Preprint. Under review.
4.2 Dialectical Path Testing and Revision
Next, we aim to inject different reasoning strategies into the instructions to help the model
verify the trustworthiness of the external information. Specifically, in STEP 2 in Figure 3),
we test various reasoning strategies, which are referred to as _Dialectical Paths_ for the same
external information as in STEP 1. These paths, detailed in Table 1, are designed to find
the most effective strategies for improving the LLM in terms of generating correct answers
in different situations, i.e., better IKE and PCA defense capabilities. We follow the logic
of designing these paths from simple to complex, from single to multiple turns of dialogs.
Specifically, we first introduce only **Tips**, suggesting that external context may be incorrect
first. Building on the previous research that Chains-of-Thought have been encoded in
the model’s parameters (Wei et al., 2023c), we aim to investigate whether providing such
tips can prompt LLMs’ dialectical thinking. Then, we use the **Base CoT** prompt “let’s
think step by step” proposed by Kojima et al. (2023), without referring to explicit thinking
steps. Following this, we describe **CoT-NoPK**, where the reasoning steps that the model is
prompted to follow are clearly described, including fact-checking and filtering the poisoned
context. However, in our experiments, we find that under the aforementioned dialectical
paths, the LLM still tends to engage in lazy thinking, i.e., directly paraphrasing contexts
from the external evidence without referencing the knowledge memoried in its parameters.
Therefore, we devise a multi-turn dialog dialectical path, **CoT-PK** : first, we prompt the
model to extract entities from the question, then based on its memory, output knowledge
regarding these entities (referred to as _prior knowledge_ in our study); finally, we prompt the
model to assess the credibility of the external contexts based on this knowledge. Finally, we
prompt the model to perform reasoning using the same strategy as **CoT-NoPK** .
It is worth noting that responses generated by LLMs following our specific dialectical path
testing tend to be less natural (e.g., always in the format of “step 1..., step 2..., step 3...”).
Thus we use another SOTA LLM to revise them to be more readable in STEP 3 (e.g., “First,
based on the..., so..., lastly...”). We use different SOTA LLMs in the LLM feedback and
revision step to prevent LLMs from _self-serving bias_ over their own generated answers (Xu
et al., 2024b) and affecting the fairness of accuracy evaluation.
4.3 Model Supervised Fine-Tuning (SFT)
In STEP 4, we use the revised optimal dialectical paths’ responses in the previous steps
(named Dialectical Response in Figure 3) to construct the dialectical supervised dataset.
Subsequently, we conduct supervised fine-tuning (SFT) with the objective of imparting the
model with foundational dialectical reasoning skills. Specifically, we construct a dataset
using the Alpace (Taori et al., 2023) format, i.e., _{_ `"instruction"; "output"` _}_, where the
instructions are in the format of the **Base** path in Table 1, and the outputs use the revised
responses corresponding to the cases with different external information. An example
sample of the constructed dataset is illustrated in Table 5 in the Appendix. Through the
steps above, we eschew costly human feedback in favor of utilizing AI feedback (Bai et al.,
2022b; Lee et al., 2023) to navigate the most efficient path toward dialectical thinking and
aligning LLMs to acquire this skillset.


In this section, we describe our experimental details. First, based on the analogous nature
of IKE and PCA in utilizing external evidence, as demonstrated in Figure 1, we design a
unified experimental framework to study these two tasks. Second, we compare in detail
the effects of different reasoning strategies in Table 1 on IKE and PCA defense. Finally, we
construct SFT datasets and use them for Dialectical supervised fine-tuning.
5.1 Experimental Settings
**Tasks.** Our experiments explore the ability of LLMs to give correct answers grounded in
external information within the context window. Our experiments are for tasks: (1) **PCA**
**Defense** : LLM needs to defend the attack of the poisoned information in their context
window; (2) **IKE** : LLM needs to update its answer based on the factual information in the
context and thus generates the correct answer. Referring to our description in Section 2 and
Preprint. Under review.
Figure 1, these two tasks are essentially two sides of the same coin of knowledge conflicts.
Therefore, we tailor these tasks simply by controlling the ratio of factual and poisoned
information in the external context.
**Models.** We selecte TinyDolphin-2.8-1.1B (shortened as TinyDolphin in the paper) [5] and
Mistral-7B-Instruct-v0.2 (shortened as Mistral-7B in the paper) [6] in our experiments. TinyDolphin is trained from TinyLlama (Zhang et al., 2024) on the Dolphin 2.8 dataset 7, a
dataset that filters out samples such as alignments, rejected answers, etc., to fine-tune TinyLlama into an unaligned and uncensored model. Mistral-7B (Jiang et al., 2023) is a model
that outperforms all other 7B models on MT-Bench (Zheng et al., 2023b) and stands out as a
model comparable to the 13B chat model. We select gpt-3.5-turbo-16k [8] as the revision LLM
in STEP 3 of DA and GLM-4 [9] as the scoring model for all experiments.
**Datasets.** We follow the poisoned dataset format of (Zou et al., 2024). These datasets are
sampled from HotpotQA (HQA) (Yang et al., 2018), MS-MARCO (MS) (Nguyen et al., 2016),
and Natural Questions (NQ) (Kwiatkowski et al., 2019), and each sample consists of one
question, one correct and one incorrect answer, five poisoned contexts that support the
incorrect answer, and one to two factual contexts that support the correct answer, see an
example in Appendix B. We utilized 300 samples to identify efficient inference paths and
build the training data. Another 300 non-overlapping samples were set aside for the test
set to prevent model memorization of correct answers during training. Unless specified
otherwise, the results presented in the paper pertain to the test set.
**Evaluation.** We employ GLM-4 to assess the accuracy (ACC) of the LLMs’ responses based
on the correct answers. We use the template in Table 7 to instruct GLM-4 to make the
judgment.
**Experimental Variables.** LLMs are vulnerable to various factors when utilizing external
information to answer questions, including the temperature used during inference, the
length of information in their context window, the noise in content (Briakou et al., 2023;
Liu et al., 2023a; Renze & Guven, 2024), etc. Our experiments manage these variables.
Specifically, our experiments are evaluated at two temperatures ( **T** ), 0.1 and 0.7, where
higher temperatures represented higher creativity and diversity of responses (Renze &
Guven, 2024). The number of poisoned contexts ( **PCN** ) ranges from 0 to 5, accompanied by
whether the context window has factual contexts supporting the correct answer ( **FC** in the
Table 6) and whether the factual context is located at the beginning or end of LLM’s context
window. Details of the experimental variables are introduced in Table 6 in Appendix C.
**Unified Experimental Framework for IKE and PCA Defense.** Our above variable setup
allows us to experiment uniformly with IKE and PCA defense tasks. Specifically, when
_PCN_ = 0 and _FC_ = `False`, LLM simply answers based on its memory; when _PCN_ = 0
and _FC_ = `True`, LLM answers based on the factual contexts in its context window. In this
process, LLM deals with knowledge conflicts between its own parametric memory and
external context (called context-memory conflict by Xu et al. (2024a), which is consistent
with the IKE setting. When _PCN ̸_ = 0 and _FC_ = `False`, we focus on scenarios where LLM
handles context-memory conflict and PCA defense. Finally, when _PCN ̸_ = 0 and _FC_ = `True`,
we focus on scenarios where LLM handles both inter-context conflict and context-memory
conflict to defend PCA. In addition, we use **RO** to stand for reorder, which means putting
the factual evidence in front of the poisoned contexts.
5.2 Finding Paths to Motivate LLMs’ Dialectical Thinking
In this section, we explore in detail the dialectical thinking paths of LLMs when dealing
with Inter-context conflict and Context-memory conflict, and we analyze the most efficient
paths under different variable settings. In Table 2, we present the accuracy of the model in
5 `[https://huggingface.co/cognitivecomputations/TinyDolphin-2.8-1.1b](https://huggingface.co/cognitivecomputations/TinyDolphin-2.8-1.1b)`
6Model Version: v0.2; Released: 12/11/2023
7 `[https://erichartford.com/dolphin](https://erichartford.com/dolphin)`
8 `[https://platform.openai.com/docs/models/overview](https://platform.openai.com/docs/models/overview)`
9 `[https://zhipuai.cn/devday](https://zhipuai.cn/devday)`
Preprint. Under review.
|Model|T|HQA MS NQ|
|---|---|---|
|TinyDolphin|0.1

0.7
|14.94
24.44
16.13
13.41
6.98
9.89|
|Mistral-7B|0.1

0.7
|49.43
77.08
60.82
47.73
80.00
56.38|
Table 2: Direct Generate ACC (%) Results. For comparison, we evaluate the model’s ability
to answer questions without any external information. Detailed instructions can be found
in Table 8.
Without FC With FC RO
1 2 3 4 5
PCN
(e) CoT-PK
Without FC With FC RO
1 2 3 4 5
PCN
(b) Tips
Without FC With FC RO
1 2 3 4 5
PCN
(a) Base
Without FC With FC RO
1 2 3 4 5
PCN
(d) CoT-NoPK
Without FC With FC RO
1 2 3 4 5
PCN
(c) Base CoT
Figure 4: PCA Results of TinyDolphin with Different Paths in Section 4.2. FC, RO, and PCN
refer to Factual Context and reorder Factual evidence before the poisoned contexts and the
number of poisoned contexts, respectively.
Without FC With FC RO
1 2 3 4 5
PCN
(d) CoT-NoPK
Without FC With FC RO
1 2 3 4 5
PCN
(c) Base CoT
Without FC With FC RO
1 2 3 4 5
PCN
(e) CoT-PK
Without FC With FC RO
1 2 3 4 5
PCN
(a) Base
Without FC With FC RO
1 2 3 4 5
PCN
(b) Tips
Figure 5: PCA Results of Mistral-7B with Different Paths in Section 4.2.
generating answers directly, without utilizing any external data. We provide the average
ACC of the two models under the PCA defense task (i.e., _PCN ̸_ = 0) when answering using
different paths in Figure 4 and Figure 5. The results for the IKE task (i.e., only factual context
provided) are detailed in Table 3. Our findings on path selection strategy are as follows:
**Aligned LLM is more receptive to external evidence than unaligned LLM.** As shown in
Table 2, TinyDolphin exhibits significantly lower ACC than Mistral-7B when no external
information is available. Additionally, in Figure 4 and 5, we observe that across all paths,
TinyDolphin consistently has lower ACC than Mistral when _PCN_ = 1 and factual evidence
is present in the context (represented by green and blue lines in these figures). Despite this,
TinyDolphin still maintains an ACC of about 5% when the external information contains
only poisoned data (indicated by the yellow line with the ”Without FC” tag), while Mistral7B’s ACC is very close to zero, especially when _PCN ≥_ 2. These results appear to contradict
the findings of Xie et al. (2024), which hypothesized that larger LLMs would be more
stubborn to their own parameter memories due to their enhanced memory and reasoning
abilities and greater sensitivity to poisoned datasets. Our results suggest that even though
a larger aligned model possesses better memory and reasoning abilities, it could be more
susceptible to a poisoned data attack because of the high trust in (human) inputs.
**Think less but become more dialectical? The path that excels in both IKE and PCA**
**defense seems to be non-existent.** From Table 3, we observe that in the IKE task, employing
complex multi-turn dialog CoT paths (CoT-PK) leads to a notable decrease in the model’s
answer accuracy compared to other paths. Furthermore, there are no substantial disparities
among other paths with no prior knowledge. This suggests that when the model encounters
a knowledge conflict and emphasizes the knowledge memoried in its parameter first, it
tends to become more stubborn, which can be particularly detrimental in knowledge editing.
However, upon comparing subfigures 5e and 4e with the other subfigures in Figure 4 and
Preprint. Under review.
1 2 3 4 5
PCN
(b) TinyDolphin
1 2 3 4 5
PCN
(a) Mistral-7B
Figure 6: DA-SFT Results in PCA Defense.
5, it becomes evident that CoT-PK allows the model to better resist the attack of poisoned
contexts. Conversely, the effectiveness of defense diminishes for the other paths as the
amount of poisoned evidence increases. In particular, for Mistral-7B, when the number
of poisoned contexts and the number of factual contexts in the model context are almost
equal (PCN = 1), the paths with no prior knowledge exhibit a higher ACC (as depicted in
subfigures 5a- 5d). However, when PCA significantly surpasses the factual contexts (PNC _≥_
2), these paths result in a lower ACC, which is consistent with previous research indicating
that LLMs tend to choose the side supported by more evidence (Xie et al., 2024; Xu et al.,
2024a). That’s why it’s crucial to train LLMs to learn to dialectically adopt the optimal path
based on the distribution of poisoned and factual information in the context window. We
provide more detailed results in Appendix D to analyze the effects of different temperatures
and the order of external evidence.
|Model|T|Dataset|Base Tips Base CoT CoT-NoPK CoT-PK DA-SFT|
|---|---|---|---|
|TinyDolphin
0.1
HQA
50.60
46.75
38.96
45.20
8.47
43.67
MS
51.35
36.11
42.03
34.48
8.06
50.19
NQ
49.43
49.39
44.00
40.00
3.66
43.20
0.7
HQA
29.82
26.15
25.00
33.93
6.77
23.90
MS
20.83
27.42
18.92
23.33
3.28
19.30
NQ
27.03
28.17
23.38
25.00
3.95
20.00|0.1|HQA
MS
NQ|50.60
46.75
38.96
45.20
8.47
43.67
51.35
36.11
42.03
34.48
8.06
50.19
49.43
49.39
44.00
40.00
3.66
43.20|
|TinyDolphin
0.1
HQA
50.60
46.75
38.96
45.20
8.47
43.67
MS
51.35
36.11
42.03
34.48
8.06
50.19
NQ
49.43
49.39
44.00
40.00
3.66
43.20
0.7
HQA
29.82
26.15
25.00
33.93
6.77
23.90
MS
20.83
27.42
18.92
23.33
3.28
19.30
NQ
27.03
28.17
23.38
25.00
3.95
20.00|0.7|HQA
MS
NQ|29.82
26.15
25.00
33.93
6.77
23.90
20.83
27.42
18.92
23.33
3.28
19.30
27.03
28.17
23.38
25.00
3.95
20.00|
|Mistral-7B
0.1
HQA
86.46
89.36
88.54
89.13
25.45
89.58
MS
89.80
86.73
86.96
87.5
16.28
89.47
NQ
88.54
92.31
84.54
92.39
11.90
94.79
0.7
HQA
93.81
89.13
87.62
88.54
31.57
89.36
MS
88.54
92.39
88.42
91.4
27.50
89.58
NQ
87.91
83.16
86.46
83.33
18.87
93.20|0.1|HQA
MS
NQ|86.46
89.36
88.54
89.13
25.45
89.58
89.80
86.73
86.96
87.5
16.28
89.47
88.54
92.31
84.54
92.39
11.90
94.79|
|Mistral-7B
0.1
HQA
86.46
89.36
88.54
89.13
25.45
89.58
MS
89.80
86.73
86.96
87.5
16.28
89.47
NQ
88.54
92.31
84.54
92.39
11.90
94.79
0.7
HQA
93.81
89.13
87.62
88.54
31.57
89.36
MS
88.54
92.39
88.42
91.4
27.50
89.58
NQ
87.91
83.16
86.46
83.33
18.87
93.20|0.7|HQA
MS
NQ|93.81
89.13
87.62
88.54
31.57
89.36
88.54
92.39
88.42
91.4
27.50
89.58
87.91
83.16
86.46
83.33
18.87
93.20|
Table 3: IKE Result. Under the IKE task, the external information in the LLMs’ context
window are all factual evidence that can support the correct answer in our dataset.
5.3 Finetuning LLM with Dialectical Data
Based on the previous findings, we construct the Dialectical SFT dataset using different
dialectical path responses on different external evidence distributions. Specifically, when
( _PCN_ = 0 _∧_ _FC_ = True) _∨_ ( _PCN_ = 1 _∧_ _FC_ = True), we opt for the responses generated by
the four paths without prior knowledge. When ( _PCN ≥_ 2 _∧_ _FC_ = True) _∨_ ( _FC_ = False), we
choose the CoT-PK responses to compile the SFT data. Notably, due to the consistently low
ACC of TinyDophin, we exclusively utilize revisoned Mistral-7B’s responses for constructing
the training data. We constructed a total of 9,012 data in Alpaca format (see an example
in Table 5) and fine-tuned the model using LoRA (Hu et al., 2022a). Figure 6 and Table 3
display the results of these fine-tuned models on the test dataset, showcasing their enhanced
ability to defend PCA while maintaining strong performance in IKE tasks. Moreover, the
improvement is particularly notable when PCN significantly exceeds factual context (i.e.
when( _PCN ≥_ 2 _∧_ _FC_ = True) _∨_ ( _FC_ = False)).


In this work, we propose Dialectical Alignment to address the challenge that helpful, honest,
and harmless LLMs are highly receptive to human input, which can result in vulnerability
to poisoned data attacks (PCA). Unlike previous research, our goal is to maintain the Incontext knowledge editing (IKE) performance while simultaneously enhancing the model’s
resilience against such attacks. To strike such a balance, we design five dialectical paths
Preprint. Under review.
ranging from simple to complex to observe the effective reasoning strategies of LLMs when
dealing with poisoned and factual contexts with their context window. Building upon
this insight, we construct a dialectical dataset and conduct supervised fine-tuning of the
model. The resulting fine-tuned model effectively defends against PCA while ensuring
IKE’s effectiveness remains intact. We anticipate that our approach will provide actionable
insights and solutions for enhancing retrieval augmented generation systems of LLMs.


Human judgments are known to be costly and laborious (Bai et al., 2022a; Wang et al.,
2022c; Diao et al., 2023). In contrast, utilizing LLMs for judgments (Bubeck et al., 2023;
Dettmers et al., 2024) offers two significant advantages: scalability and explainability (Zheng
et al., 2024). By leveraging LLMs, the need for extensive human involvement is reduced,
facilitating scalable benchmarks and rapid iterations (Diao et al., 2023; Dubois et al., 2024).
While some concerns have been raised regarding LLMs judgments (Wang et al., 2023c;
Raghubir & Valenzuela, 2006; Blunch, 1984), such as position bias, verbosity bias, selfenhancement bias, and limited reasoning ability, Zheng et al.2024 have shown that many of
these biases are either minor or can be effectively mitigated. Furthermore, Zheng et al.2024
have also demonstrated that advanced LLMs judges like GPT-4 are capable of aligning well
with both controlled and crowdsourced human preferences, achieving agreement levels
exceeding 80%, which is comparable to the agreement observed among humans.

A Comparing LLMs Judgments with Human Judgments