**Alignment Algorithm and Preference Optimization** RLHF [Ouyang et al., 2022] requires enormous human annotations to build reward models which is used to align LLM with reinforcement
learning. Constitution AI [Bai et al., 2022] partially replaces human annotations with a hybrid
AI-human approach to build their preference models, but still relies heavily on human input. DPO
[Rafailov et al., 2024] and later variants such as SimPO [Meng et al., 2024] eliminate the need for a
separate reward model but offer no clear advantage over methods like STaR [Zelikman et al., 2022]
especially under situations requiring reasoning, according to empirical results in [Du et al., 2024].
**Iterative self-improving LLM system** Recent research has explored self-improving systems that
enhance model capabilities without extensive human annotation. These systems combine a "thinking"
process that generates superior responses with a "learning" process to integrate these improvements
iteratively. Self-rewarding LLMs [Yuan et al., 2024] use self-evaluation to select the best responses
for Direct Preference Optimization (DPO), while Meta-rewarding LLMs [Wu et al., 2024] introduce a
meta-judge to refine the evaluation layer itself. Direct Nash Optimization (DNO) [Rosset et al., 2024]
employs an oracle model to assist in both generation and selection, avoiding performance saturation
more effectively. However, to prevent model collapse [Shumailov et al., 2024], these systems often
reset to the base model for each iteration, effectively turning the process into a data augmentation
pipeline. Special focus on "hard cases" as demonstrated by more capable LLMs is first highlighted
by LLM2LLM [Lee et al., 2024].
**Thinking mechanism** Chain of thought (CoT) [Wei et al., 2022] improves reasoning ability
of language model through connecting overlapping localized knowledge in its pre-traing corpus
[Prystawski et al., 2023]. As CoT innately requires searching [Ye et al., 2024], explicit tree search
mechanism could further enhance it [Yao et al., 2023]. Language agent tree search (LATS) also
includes feedback, self-evaluation, and MCTS [Zhou et al., 2024], achieving better performance by
deliberately going through each thread of thoughts. However, they suffers from significant latency
increase, and shows significant limitation when facing problem which are easy under visual modality
[Menon et al., 2024], planning with graph-like structure [Lin et al., 2024] has also shown advantage,
despite relying still on language modality.
**Efficient Alignment in LLMs** Recent research has focused on enhancing LLMs’ alignment through
various supervised fine-tuning techniques. These approaches generally aim to augment supervision
datasets using LLM-generated data. For instance, [Zelikman et al., 2022] leverages provided answers
to iteratively generate diverse reasoning paths, selecting those that lead to correct outcomes, [Li et al.,
2022] extends this to distill the rationales offered from a teacher model into a student models. [Lee
et al., 2024] takes a targeted approach by identifying cases where LLMs produce incorrect outputs
and concentrating augmentation efforts on these instances. These studies support our hypothesis that
for specific tasks, a set of key associations is required, and focused augmentation combined with
supervised learning can significantly enhance LLMs’ reasoning abilities.


We bootstrap reasoning ability of a strong VLM with explicit graph reasoning, logical operations are
carried out within the graph to further understand the problem. This is the adaptive reasoning with
graph section. Such graph is used to detect the weakness of the language model, and then iteratively
enhance its understanding of the current problem.

Fangyuan Yu
Hardeep Singh Arora
Matt Johnson


Figure 3: Iterative Graph Prompting (IGP)
Given a rule _s_, a query _x_, we use a VLM to obtain an answer _y_, together with a logical graph _G_
which contains the thought process of providing the rule-aligned response.
We ask the VLM to generate narrated version of a graph _G_ which is essentially a JSON formatted list
of (entity, relation, entity) triplets. We could also plot such graph _G_ into an image Φ( _G_ ), which can
be prompted directly into the VLM again.
**Self Evaluation** We begin with a self-check, evaluating the LLM’s initial response for consistency
with its own judgement:
_y ∼_ _π_ ( _s, x_ ) _,_ _π_ eval( _s, x, y_ ) (1)
where _s_ is the instruction, _x_ the query, _y_ the answer, _π_ the LLM’s response function, and _π_ eval
indicates the evaluation of the LLM on its own response.
For query which the model fails to provide a satisfactory response, we proceed with iterative graph
construction process. Given the instruction and query, we ask the VLM to provide its logical reasoning
process in the form of (entity, relation, entity) triplets, which we use to initialize a graph _G_ 1.
**Iterative Refinement** For _i_ = 1 to _k_ iterations:
_Gi_ = _π_ refine( _s, x, ϕ_ ( _Gi−_ 1)) (2)
where _ϕ_ ( _Gi−_ 1) is the visual representation of the graph from the previous iteration, and _π_ refine is the
LLM’s graph refinement function.
**Visual Prompting** We leverage the graph for visual prompting, enhancing the LLM’s reasoning
capabilities:
_y ∼_ _π_ ( _s, x, ϕ_ ( _G_ )) (3)
We observe empirically that strong LLMs (GPT-4 and Sonnet-3.5) perform better given the graph as
an image Φ( _G_ ) compared to descriptive text _G_ .
We provide Iterative Graph Prompting (IGP) in Algorithm (3.1) and its diagram in Figure (3)
**Algorithm 1** Iterative Graph Prompting
**Require:** _s_ (rule), _x_ (query), vision language model _π_
**Ensure:** y
1: _y ∼_ _π_ ( _s, x_ )
2: **if** _π_ eval( _s, x, y_ ) **then** _▷_ Naive Inference is accepted
3: **return** _r_
4: **else**
5: _G_ 1 _←_ _π_ initialize( _s, x_ ) _▷_ Initialize Graph
6: _i ←_ 1
7: **while** _Gi ̸_ = _Gi−_ 1 **do**
8: _Gi_ +1 _←_ _π_ refine( _s, x,_ Φ( _Gi_ )) _▷_ Iterate on Graph
9: _i ←_ _i_ + 1
10: **end while**
11: _y ∼_ _π_ ( _s, x, ϕ_ ( _Gi_ ))
12: **return** _y_
13: **end if**


We begin with three core intuitions about the current process of training language models, arguing
the need for a paradigm shift. Firstly, when given an annotated answer, a language model should
recognize that any ’aligned’ answer is acceptable, rather than being constrained to believe this is the
sole correct response. Traditional fine-tuning approaches, however, treat the annotated answer as a
definitive target, aiming to minimize the likelihood of deviating from it.
Our second intuition posits that language models should learn the thought processes behind generating
answers, not just the answers themselves. This approach is akin to teaching a man to fish, which
sustains him for a lifetime, rather than giving him a fish, which feeds him for a day.
Thirdly, the learning process should be individually tailored to each model, akin to customizing education for different students. Training should focus incrementally on scenarios that pose challenges,
ensuring that each step is calibrated to address and overcome the specific difficulties encountered by
the model. This method ensures that learning is not only adaptive but also progressively targets areas
of weakness, fostering a more robust and comprehensive understanding over time.
Figure 4: Self-Aligned Incremental Learning (SAIL)
**Algorithm 2** Propose & Check (PAC)
**Require:** Model _π_, solved cases _D_, unsolved cases _D_ unsolved, stage number _n_ stage
**Ensure:** Updated _D_ and _D_ unsolved
1: **for** each ( _x, y, G_ ) in _D_ unsolved **do**
2: **if** _n_ stage = 1 **then**
3: _y_ propose _∼_ _πpropose_ ( _x_ ) _▷_ Direct proposition
4: **else if** _n_ stage = 2 **then**
5: _y_ propose _∼_ _π_ propose( _x, G_ ) _▷_ Hinted proposition
6: **else**
7: _y_ propose _∼_ _πpropose_ ( _x, y, G_ ) _▷_ Guided proposition
8: **end if**
9: _e ∼_ _πcheck_ ( _y_ propose _, y_ ) _▷_ Check alignment
10: **if** _e_ = True **then**
11: _D ←_ _D ∪{_ ( _x, y_ propose) _}_
12: _D_ unsolved _←_ _D_ unsolved _\ {_ ( _x, y, G_ ) _}_
13: **end if**
14: **end for**
15: **return** _D, D_ unsolved
**Algorithm 3** Self-Aligned Incremental Learning (SAIL)
**Require:** Initial dataset _D_ 0, base LLM _π_ 0, helper LLMs _{πh_ 1 _, ..., πhm_ _}_, number of iterations _T_,
repetition factors _n_ 2 _, n_ 3
**Ensure:** Trained LLM _πT_
1: _D ←∅_
2: _D_ unsolved _←_ _D_ 0
3: **for** _t_ = 1 **to** _T_ **do**
4: _D, D_ unsolved _←_ PAC( _πt−_ 1 _, D, D_ unsolved _,_ 1)
5: _D_ unsolved _←_ [�] _[n]_ _i_ =1 [2] _[D]_ [unsolved] _▷_ Augment moderate difficult case
6: _D, D_ unsolved _←_ PAC( _πt−_ 1 _, D, D_ unsolved _,_ 2)
7: **for** each _πh_ in _{πh_ 1 _, ..., πhm}_ **do**
8: _D, D_ unsolved _←_ PAC( _πh, D, D_ unsolved _,_ 2)
9: **end for**
10: _D_ unsolved _←_ [�] _[n]_ _i_ =1 [3] _[D]_ [unsolved] _▷_ Augment most challenging case
11: _D, D_ unsolved _←_ PAC( _πt−_ 1 _, D, D_ unsolved _,_ 3)
12: **for** each _πh_ in _{πh_ 1 _, ..., πhm}_ **do**
13: _D, D_ unsolved _←_ PAC( _πh, D, D_ unsolved _,_ 3)
14: **end for**
15: _πt ←_ SFT( _π_ 0 _, D_ ) _▷_ Supervised Fine-Tuning
16: **end for**
17: **return** _πT_
Self-Aligned Incremental Learning (SAIL) (see Figure 3) introduces a novel paradigm for aligning
Large Language Models (LLMs). At its core, SAIL employs a ’propose and check’ methodology (see
Algorithm 2), which generates a range of thoughts and answers, then checks each against annotated
answers for alignment. This self-alignment process is seamlessly integrated into Supervised FineTuning (SFT), moving beyond uniform training with static question-answer pairs. SAIL incrementally
selects and augments cases within the training dataset based on difficulty and the model’s current
capabilities through a multi-agent incremental curriculum building process.
This process unfolds in three progressive stages, each offering increasing levels of support. In the
direct proposition stage, the student model independently proposes solutions. Unsolved cases are
then augmented and advanced to the hinted proposition stage, where the model receives additional
context to encourage building upon existing knowledge. Finally, in the guided proposition stage,
remaining challenging cases are presented with both hints and annotated answers, prompting the
model to propose similar solutions while grasping the underlying reasoning.
To enhance the diversity of responses and effectively patch representation gaps, SAIL employs
multiple helper models in the hinted and guided stages. These helper models, with their varied
capabilities and knowledge bases, contribute to a more comprehensive exploration of solution spaces.
This collective intelligence approach ensures a rich, diverse set of responses, helping to address blind
spots in the student model’s understanding. SAIL is presented in Algorithm (3).


Iterative Graph Alignment (IGA) aims to combine the power of IGP and SAIL to achieve efficient
alignment with minimal human supervision. IGP naturally provides a logical graph _G_, together with
a thoughtful response _r_, we could narrate the graph to form the hint _ϕ_ ( _G_ ). This provides annotations
without any human input. Optionally, extra human editing could be leveraged to further enhance the
quality of such a dataset. The algorithm is displayed in 3.3
**Algorithm 4** Iterative Graph Alignment (IGA)
**Require:** Teacher VLM _πvlm_, student LLM _π_ 0, helper LLMs _{πh_ 1 _, ..., πhm}_, queries _{xi}_ _[N]_ _i_ =1 [, rule]
_r_, number of iterations _T_
**Ensure:** Trained LLM _πT_
_D ←∅_ _▷_ Initialize dataset


( _yi, Gi_ ) _←_ IGP( _s, xi, πvlm_ ) _▷_ Iterative Graph Prompting
4: _D ←_ _D ∪{_ ( _xi, yi, Gi_ ) _}_


_πT ←_ SAIL( _π_ 0 _, {πh_ 1 _, ..., πhm_ _}, D, T_ ) _▷_ Self-Aligned Incremental Learning
**return** _πT_


To evaluate Iterative Graph Alignment (IGA), we developed a test suite ’RuleAlign’ consists of
five rule-based scenarios, each comprising 100 test and 200 training queries. These queries, which
mix in-domain and out-of-domain examples, are designed to assess rule-alignment capacity of a
LLM. Each query is open-ended to better mimic practical scenarios. In order to evaluate a proposed
response, we check whether it aligns with the reference answer collected from human. The complete
[dataset and evaluation suite is available at RuleAlign.](https://github.com/fangyuan-ksgk/RuleEval)
**Prompting Techniques Comparison** To achieve annotation-free rule-based alignment, a high level
of rule adherence from Iterative Graph Prompting (IGP) is critical, as it will be used to teach the
student LLM, along with the obtained logical graph. We compare IGP against naive prompting and
chain-of-thought (CoT) prompting [Kojima et al., 2023], which is applied in current self-alignment
algorithms [Yuan et al., 2024, Rosset et al., 2024, Bai et al., 2022].
We conduct our experiments with Claude Sonnet-3.5 [Anthropic, 2024], with results presented in
Table 1. In practice, we use 2 iterations for IGP to balance performance and speed. We found IGP
achieves an average 73.12% relative improvement in rule-based alignment compared to the baseline.
Interestingly, we observed degraded rule adherence from CoT in roleplay scenarios. We suspect this
is due to the ’AI Assistant self-identification’ bias in these instruct models, which is retrieved with
CoT, aligning with insights from [Prystawski et al., 2023].
In roleplay scenarios, we found that the model tends to repeat the same question when presented with
an atypical query (e.g., "What are the store hours?" asked to an LLM roleplaying as a customer).
By inspecting the two logical graphs generated through IGP, we identified that while the model
recognizes that a customer typically asks such questions, it fails to deduce that, as a customer, it
should decline to answer rather than ask the question itself. By explicitly including prompts in
the graph refinement process, we effectively ’engineer its thought process’, instructing it to decide
whether to ’answer’ or ’reject’ the question after identifying its role. This suggests potential in the
area of ’thought flow engineering’.
**Learning Techniques Comparison** Although Iterative Graph Alignment (IGA) requires zero
human annotation, unlike Supervised Fine-Tuning (SFT) and Self-Taught Reasoner (STaR) [Zelikman
et al., 2022], we compare IGA with these methods using teacher VLM annotations collected through
IGP to ensure a fair comparison. For Self-Aligned Incremental Learning (SAIL), SFT, and STaR,
we chose Llama3-8B-Instruct [llama3 teams, 2024] as our student models and applied LoRA [Hu
et al., 2022], which is more commonly adopted in resource-limited scenarios. We used the same
hyper-parameters for LoRA across all three methods.
Table 2 presents our comparison results. On average, IGA achieves a relative improvement of 86.20%
over the baseline. Notably, an IGA fine-tuned 8B model outperforms Claude Sonnet-3.5, matching
its IGP-enhanced responses in rule-based alignment. Showing the effectiveness of IGA in patching
representation gap and achieving annotation-free rule-based alignment.
Table 1: Comparison of effectiveness across different prompting strategies (Claude).
Scenario Naive CoT IGP
Do not talk about elephant 76.0% 82.5% **97.0%**
Roleplay as customer 51.6% 42.5% **96.6%**
Roleplay as salesperson 54.2% 38.2% **98.2%**
Roleplay as patient 43.1% 34.6% **97.2%**
Connect user to human when requested 65.4% 76.2% **94.2%**
_Relative improvement over Naive baseline (%)_
Do not talk about elephant           - 8.55% **27.63%**
Roleplay as customer           - -17.64% **87.21%**
Roleplay as salesperson           - -29.52% **81.18%**
Roleplay as patient           - -19.72% **125.52%**
Connect user to human when requested           - 16.51% **44.04%**
Average relative improvement           - -8.36% **73.12%**
Table 2: Comparison of effectiveness across different prompting strategies (Llama3).
Scenario Naive CoT SFT STaR IGA
Do not talk about elephant 73.0% 81.2% 53.6% 84.6% **96.5%**
Roleplay as customer 52.4% 10.2% 4.4% 65.8% **98.0%**
Roleplay as salesperson 53.2% 12.6% 3.2% 74.2% **96.8%**
Roleplay as a patient 36.0% 10.6% 4.0% 68.4% **97.4%**
Connect user to human when requested 61.4% 74.2% 48.2% 80.8% **97.8%**
_Relative improvement over Naive baseline (%)_
Do not talk about elephant       - 11.23% -26.58% 15.89% **32.19%**
Roleplay as customer      - -80.53% -91.60% 25.57% **87.02%**
Roleplay as salesperson       - -76.32% -93.98% 39.47% **81.95%**
Roleplay as a patient       - -70.56% -88.89% 90.00% **170.56%**
Connect user to human when requested      - 20.85% -21.50% 31.60% **59.28%**
Average relative improvement      - -39.07% -64.51% 40.51% **86.20%**


Maintaining the accuracy of evaluations is critical for SAIL. Our research indicates that self-alignment
checks often favor the model’s own generated responses, potentially leading to the misidentification of
complex cases. This bias can skew the distribution of training data, resulting in suboptimal outcomes.
Additionally, although our approach excels at detecting representation gaps, it primarily focuses on
augmenting responses with diversity to individual questions. This strategy is frequently insufficient
for disciplines like mathematics and coding, which also require a wide range of question types. We
advocate for future research to emphasize automatic data augmentation based on ’meta-logic’ to
improve the resilience and versatility of our techniques.
Ultimately, we aspire to develop an ideal version of the Integrated Growth and Reasoning (IGA)
system as a fully autonomous self-enhancement framework for Vision-Language Models (VLMs).
This system would integrate reasoning enhancements from Integrated Growth Processes (IGP) into
the model’s learning seamlessly. Moreover, it would use visual grounding through feedback from
virtual environments to tackle the challenges in evaluation and augmentation, particularly in light of
the extensive online interaction data (see [Putta et al., 2024]). By incorporating these innovations, we
aim to construct a more advanced and self-optimizing IGA system capable of adapting and evolving
without human oversight.


**Ke Xu** [1], **Jie Fu** [4], **Wangchunshu Zhou** [5][*], **Wenhao Huang** [2][*]
1Beihang University; 201.AI; 3Tsinghua University; 4HKUST 5AIWaves;
zenmoore@buaa.edu.cn

PopAlign: Diversifying Contrasting Patterns for a More Comprehensive
  Alignment

Zekun Moore Wang [1,2] , Shawn Wang [3] , Kang Zhu [2] , Jiaheng Liu [1] ,

Zekun Moore Wang
Shawn Wang
Kang Zhu
Jiaheng Liu
Ke Xu
Jie Fu
Wangchunshu Zhou
Wenhao Huang


Alignment of large language models (LLMs)
involves training models on preferencecontrastive output pairs to adjust their responses according to human preferences. To
obtain such contrastive pairs, traditional methods like RLHF and RLAIF rely on **limited**
contrasting patterns, such as varying model
variants or decoding temperatures. This singularity leads to two issues: (1) alignment is
not comprehensive; and thereby (2) models are
susceptible to jailbreaking attacks. To address
these issues, we investigate how to construct
more comprehensive and diversified contrasting patterns to enhance preference data (RQ1)
and verify the impact of the diversification of
contrasting patterns on model alignment (RQ2).
For RQ1, we propose **PopAlign**, a framework
that integrates diversified contrasting patterns
across the prompt, model, and pipeline levels,
introducing six contrasting strategies that do
not require additional feedback labeling procedures. Regarding RQ2, we conduct thorough
experiments demonstrating that PopAlign significantly outperforms existing methods, leading to more comprehensive alignment.


Alignment is a crucial stage in training large language models (LLMs), where the objective is to
adjust the models’ response distributions to align
with human values or preferences. Typical alignment approaches include Reinforcement Learning
from Human Feedback (RLHF) (Ouyang et al.,
2022) and Reinforcement Learning from AI Feedback (RLAIF) (Lee et al., 2023). These methods
generally begin with the model generating paired
outputs for the _same_ prompt, which are then evaluated by human annotators or language models. The
paired outputs are categorized into one preferred
(chosen) response and one less preferred (rejected)
response, forming contrastive pairs. Subsequently,
*Corresponding author.
Figure 1: Illustration of the effects of alignment considering the contrasting patterns. _πref_ _[i]_ [denotes the dis-]
tribution of the reference model under pattern _i_ . _πdpoi_
denotes the overall distribution of the model after DPO
alignment on pattern _i_ .
various preference optimization algorithms, such
as Proximal Policy Optimization (PPO) (Schulman
et al., 2017) or Direct Preference Optimization
(DPO) (Rafailov et al., 2024), can be employed
to train the models on this preference-contrastive
data, aiming to increase the probability of generating the chosen responses while reducing the likelihood of generating the rejected ones. However,
these approaches typically consider only _limited_
contrasting patterns. For example, LLaMA 2 (Touvron et al., 2023b) solely uses responses generated by varying either model variants or decoding
temperatures to synthesize preference-contrastive
data. As illustrated in Figure 1, alignment based on
such _limited_ contrasting patterns may not lead to
a comprehensive alignment effect, rendering models more susceptible to jailbreaking attacks (Liu
et al., 2023b). Therefore, two key research questions arise: (RQ1) How can more comprehensive
and diversified contrasting patterns be constructed
IDs
to enhance the preference data? and (RQ2) How
do more diverse contrasting patterns impact the
performance of model alignment?
To address RQ1, it is intuitive that any contrastive feature within the response generation
workflow can be utilized to construct contrastive responses. For example, employing different prompt
templates for the same instruction can lead to varied responses, with some being preferred over others. Additionally, different models generating responses for the same instruction can result in outputs of varying quality, and differences in the overall pipeline built on top of the models can also lead
to response discrepancies. In general, the response
generation workflow of LLMs can be divided into
three key components: (1) prompt, (2) model, and
(3) pipeline. By exploring the sources of response
discrepancies within each of these components,
preference-contrastive data can be constructed. In
practice, we find that these readily available contrastive signals can also be leveraged to directly
determine pseudo-feedback for the synthesized responses, thereby providing an additional advantage
of eliminating the need for a separate stage of human or AI feedback labeling.
Specifically, for the (1) prompt level, prompt engineering typically focuses on three parts (Dong
et al., 2022): (1.a) instructions, (1.b) demonstrations, and (1.c) chain of thoughts (Wei et al., 2022).
Each of these can be made contrastive to elicit distinct responses. At the (2) model level, contrastive
signals can be found from two principles: (2.a) the
model size scaling law (Kaplan et al., 2020), and
(2.b) the training data quality law (Que et al., 2024;
Ye et al., 2024). Variations in the number of model
parameters and the quality of training data can result in preference gaps on generated responses. Regarding the (3) pipeline level, responses can be
incrementally improved by increasing the number
of interactions or iterations of the models (Shridhar
et al., 2023; Madaan et al., 2023; Renze and Guven,
2024; Chan et al., 2023). These procedures share
a common fundamental operation: (3.a) response
refinement, which improves the response from the
previous dialogue turn.
To instantiate the above intuitions, we propose
**PopAlign** (Prompt-Model-Pipeline Contrast for
LLM Alignment), a framework that integrates six
distinct response contrasting strategies, encompassing a comprehensive composition of contrastive
signals within the response generation workflow.
These six contrasting strategies include: (1) Pre
fix Contrast, which designs contrastive instructions
in a prefix-like format, following the practice of
RLCD (Yang et al., 2023); (2) Demon Contrast,
which enhances the contrast of responses through
differentiated few-shot demonstrations; (3) Elicitive Contrast, which uses the Chain of Thought
(CoT, (Wei et al., 2022)) technique to let the model
first think about _how_ to generate a good or bad
response and then generate it; (4) NParam Contrast, which utilizes the response differences between models of different sizes ( _i.e._, number of
parameters); (5) Leaderboard Contrast, which uses
the response differences between models ranked
differently on public authoritative leaderboards
such as AlpacaEval (Li et al., 2023; Dubois et al.,
2024), mainly due to the differences in training data
quality; [1] (6) Refine Contrast, which is based on
the multi-turn dialogue ability, allowing the model
to refne its response after generating an initial one.
Therefore, in Figure 2, based on _{Ri}_ [6] _i_ =1
(where _Ri_ represents one of the six contrasting strategies), given a prompt _q_, we generate the chosen (+) and rejected ( _−_ ) responses
_{_ ( _ri_ [+] _[, r]_ _i_ [-][)] _[}]_ [6] _i_ =1 [=] _[ {][R][i]_ [(] _[q]_ [)] _[}]_ _i_ [6] =1 [, and then train the]
model by DPO (Rafailov et al., 2024). This procedure can be seen as a comprehensive contrastive
distillation (Yang et al., 2023) method to distill
diverse contrasting patterns of the response generation workflow into the aligned model.
To address RQ2, we conduct comprehensive experiments to evaluate the effectiveness of contrasting pattern diversification. Our findings indicate
that our PopAlign, which integrates diversified contrasting patterns, outperforms previous methods for
synthesizing preference data across various tasks.
Furthermore, we investigate the effects of each individual contrasting pattern, as well as their cumulative impact when combined one-by-one, highlighting the superior gains by our novel “Elicitive Contrast” strategies. In addition, we perform a detailed
analysis of contrasting accuracies [2] and the preference modeling. We also examine the influence of
different preference optimization algorithms and
various models.
1In our work, we leverage Yi-34B-Chat (AI et al., 2024) v.s.
Vicuna-33B (Chiang et al., 2023) for Leaderboard Contrast.
Since both of them are in LLaMA (Touvron et al., 2023a)’s
architecture, the performance gap between them is mainly due
to the data quality.
2Contrast accuracy measures how accurately contrasting
strategies generate pairwise responses that align with the preferences of an oracle preference model.
4 NParam Contrast
5 Leaderboard Contrast
6 Refinement Contrast
1 Prefix Contrast
2 Demon Contrast
3 Elicitive Contrast
|😋|😋|😋|..|...|😋|
|---|---|---|---|---|---|
|🤯|🤯|🤯|..|...|🤯|
Contrastive
DPO
Responses
PopAlign
Instructions
(Data Synthesis)
😋 Chosen Response 🤯 Rejected Response
Figure 2: The workflow of PopAlign. PopAlign involves three kinds of contrasting strategies: (1) Prompt Contrast
such as Prefix Contrast, Demon Contrast ( _i.e._, Demonstration Contrast, and Elicitive Contrast), (2) Model Contrast
such as NParam (number of parameters) Contrast and Leaderboard Contrast, as well as (3) Pipeline Contrast such as
Refinement Contrast. By mixing the preference data synthesized with diverse contrasting strategies and conducting
DPO alignment training on it, we can easily align the LLM without either human annotation or reward labeling.


As illustrated in Figure 2, we design three groups
of contrasting strategies: (1) **Prompt Contrast**,
which involves using different prompt templates to
elicit contrastive responses. (2) **Model Contrast**,
which involves generating contrastive responses
using a generally superior model and an inferior
model. (3) **Pipeline Contrast**, which involves
generating pairwise responses by using a superior
pipeline alongside an inferior one.


We instantiate Prompt Contrast with three approaches: Prefix Contrast, Demonstration Contrast
( _i.e._, Demon Contrast), and Elicitive Contrast.
**Prefix Contrast**, introduced by RLCD (Yang
et al., 2023), prepends contrastive prefixes to the
user query to elicit contrastive responses. Given
the user query _q_ and a model _M_, Prefix Contrast leverages a manually predefined prefix pair
( _p_ [+] _, p_ _[−]_ ) to prompt the model to generate the contrastive responses via _r_ [+] = _M_ ([ _p_ [+] _, q_ ]) and _r_ _[−]_ =
_M_ ([ _p_ _[−]_ _, q_ ]). For example, as illustrated in Figure
2, the prefix can be a positive one such as “(helpful, harmless)”. The model is prompted with the
concatenation of the input query and this prefix,
formatted within the chat template such as “Input:
{query} | Output: {prefix}” ( _c.f._, SectionA), to generate
the chosen response.
**Demon Contrast** relies on the In-Context Learning (Brown et al., 2020; Dong et al., 2022) abilities
of LLMs, using manually crafted few-shot demonstrations with good versus bad responses to prompt
the model. Formally, with contrastive demonstrations _d_ [+] and _d_ _[−]_, the contrastive responses are generated as _r_ [+] = _M_ ([ _d_ [+] _, q_ ]) and _r_ _[−]_ = _M_ ([ _d_ _[−]_ _, q_ ]).
**Elicitive Contrast** leverages the Chain of
Thought (Wei et al., 2022) abilities of LLMs, eliciting the models to first generate insights on how
to craft good (or bad) responses and then generate
the actual responses. The query is wrapped with
prompt templates (as shown in Figure 2) that elicit
good and bad responses respectively: _T_ [+] ( _q_ ) and
_T_ _[−]_ ( _q_ ). Then we obtain ( _t_ [+] _, r_ [+] ) = _M_ ( _T_ [+] ( _q_ ))
and ( _t_ _[−]_ _, r_ _[−]_ ) = _M_ ( _T_ _[−]_ ( _q_ )), where _t_ [+] or _t_ _[−]_ denotes the thoughts or insights on how to craft a
chosen or a rejected response, respectively.


We instantiate Model Contrast with two approaches: NParam Contrast (Number-of-Parameter
Contrast) and Leaderboard Contrast.
**NParam Contrast** involves contrasting responses through models of different sizes, based
on the rule of thumb that a larger model typically
demonstrates better performance. For example,
we can use a model with 34B parameters, Yi-34BChat (AI et al., 2024), as _M_ _[L]_, and Yi-6B-Chat with
6B parameters as _M_ _[S]_ . The pairwise responses are
therefore _r_ [+] = _M_ _[L]_ ( _q_ ) and _r_ _[−]_ = _M_ _[S]_ ( _q_ ).
**Leaderboard Contrast** leverages models of
different ranks in well-recognized leaderboards
to generate contrastive responses. For instance,
_r_ [+] = _M_ [1] _[st]_ ( _q_ ) and _r_ _[−]_ = _M_ [2] _[nd]_ ( _q_ ), with _M_ [1] _[st]_
denoting Yi-34B-Chat (AI et al., 2024) and _M_ [2] _[nd]_
denoting Vicuna-33B (Chiang et al., 2023), since
Yi-34B-Chat ranks higher than Vicuna-33B in most
well-recognized leaderboards such as AlpacaEval (Li et al., 2023; Dubois et al., 2023, 2024) and
Chatbot Arena (Chiang et al., 2024).


We instantiate Pipeline Contrast with just one approach: Refine Contrast.
**Refine Contrast** constructs two pipelines with
differing answer qualities. The less effective
pipeline produces a response in a single model
run; the more effective one adds a second turn
to refine the response. Formally, the process is
defined as _r_ _[−]_ = _M_ ( _q_ ) for the initial response,
and _r_ [+] = _M_ ([ _q, r_ _[−]_ _, I_ ]) for the refined response,
where _I_ is a second-turn user instruction that
prompts the model to improve _r_ _[−]_ . For example,
as illustrated in Figure 2, given a user query, the
model generates an initial response in the first dialogue turn. In the second dialogue turn, the model
is prompted to refine its initial response into an improved version. The final response, along with its
initial version, then serve as a contrastive response
pair, with the final response being chosen and the
initial response being rejected.
We refer readers to Appendix A for more details
about the implementation and the prompt templates.
Samples of the synthesized preference data are presented in Appendix D.


Based on the aforementioned six contrasting strategies _{Ri}_ [6] _i_ =1 [, we can synthesize a preference]
dataset with diverse contrastive patterns. For each
instruction _qj_ in the source dataset _D_, PopAlign
generates six response pairs _{_ ( _rj,i_ [+] _[, r]_ _j,i_ _[−]_ [)] _[}]_ _i_ [6] =1 [=]
_{Ri_ ( _qj_ ) _}_ [6] _i_ =1 [. Thus, the resulting dataset is:]
_D_ ˜ = _{_ ( _qj,_ ( _rj,i_ [+] _[, r]_ _j,i_ _[−]_ [))] _[ |][ q][j][ ∈]_ _[D, i][ ∈{]_ [1] _[,]_ [ 2] _[, . . .,]_ [ 6] _[}}]_
We then perform alignment training, such as
DPO (Rafailov et al., 2024), on the dataset _D_ [˜] .


We evaluate PopAlign on two alignment tasks and
three well-recognized leaderboards: (1) HarmlessBase subset in HH-RLHF (Bai et al., 2022a),
(2) Helpful-Base subset in HH-RLHF (Bai et al.,
2022a), (3) AlpacaEval 2.0 (Li et al., 2023; Dubois
et al., 2023, 2024), (4) Arena Hard (Li et al., 2024),
and (5) MT-Bench (Zheng et al., 2024a). (See Appendix C.1 for more details on these tasks.)
The two alignment tasks (Helpful-Base and
Harmless-Base) focus on specific aspects of LLMs’
capabilities, such as helpfulness and harmlessness.
In contrast, the three leaderboards assess the models’ overall and comprehensive capabilities, which
can be enhanced through alignment training.


For the alignment tasks, our evaluation is based
on GPT following common practices (Yang et al.,
2023; Li et al., 2023). We compare each baseline
with PopAlign to obtain a win rate. For the three
leaderboards, we follow the official evaluation procedures of each. Specifically, they compare each
method with GPT-3.5 or GPT-4 to get a win rate.
The evaluation details including the prompt templates are shown in Appendix C.


In our work, we utilize UltraFeedback (Ding et al.,
2023; Cui et al., 2023)’s binarized version (Tunstall
et al., 2023) [3] as the source instructions for data
synthesis. We use Yi-34B-Chat (AI et al., 2024) as
the default LLM due to its superior capabilities. For
Model Contrast, we use Yi-6B-Chat for NParam
Contrast, and Vicuna-33B (Chiang et al., 2023)
for Leaderboard Contrast. We use Yi-6B-Chat (AI
et al., 2024) as the model to be aligned using DPO
algorithm (Rafailov et al., 2024). We train the
model for one epoch on the synthesized dataset
with a micro batch size of 4. We use cosine learning
rate scheduler with a peak learning rate of 5e-7 and
a warmup ratio of 0.1. The beta for DPO training
is set as 0.01. The max sequence length is set to be
2048. All the model parameters are trainable.
Moreover, following the practices of
RLCD (Yang et al., 2023), which runs separate variants for helpfulness and harmlessness
[3https://huggingface.co/datasets/](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized)
[HuggingFaceH4/ultrafeedback_binarized](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized)
respectively due to the differing nature of these
two aspects, we adopt a similar experimental setup.
But in our settings, to assess the LLMs more
comprehensively, we run a variant of PopAlign
for alignment tasks that simultaneously focuses
on harmlessness and helpfulness ( _i.e._, the contrast
is “helpful _+_ harmless _v.s._ harmful _+_ unhelpful”).
Additionally, we run another variant of PopAlign
for leaderboards that focuses on the overall aspect
of alignment ( _i.e._, the contrast is “good _v.s._ bad”).
Refer to Appendix A and B for more details.


We compare PopAlign with three baselines of preference data synthesis methods, a baseline which is
the initial state of the model to be aligned, and an
additional strong baseline which is trained on the
original response labels of the training set. That is,
(1) **Yi-6B-Chat** (AI et al., 2024), the model to be
aligned in our work. (2) **RLAIF** (Bai et al., 2022b),
which generates two responses for the same prompt
using Yi-34B-Chat and then scores the responses
using the same model. The prompt template used
for scoring is the same as (Bai et al., 2022b). (3)
**RLCD** (Yang et al., 2023), which generates contrastive responses using Yi-34B-Chat for prompts
prepended with contrastive prefixes. The prefixes
used are the same as those for Prefix Contrast. (4)
**Context Distillation** (Huang et al., 2022) (ContextDist), which conducts supervised fine-tuning on
the chosen responses in the data used by RLCD.
(5) **Label-DPO**, a strong baseline trained directly
on the label responses.
All the baselines are trained under the same settings as PopAlign, detailed in Appendix B.


**Evaluating PopAlign on Alignment Tasks.** In
two alignment tasks (i.e., Helpful-Base and
Harmless-Base), we compare each baseline with
PopAlign to determine its win rate. The win rates
of PopAlign are thus reported as 50.0 (tie). The experimental results are presented in Table 3.2.1. We
observe that PopAlign achieves the highest performance when compared with various baselines, even
comparable with the strong baseline of Label-DPO.
Notably, our method is more effective in terms of
helpfulness compared to the aspect of harmlessness. This suggests that the primary differences
among different prompts, models, and pipelines,
are predominantly in their helpfulness. This obser
vation is expected, given that popular models are
generally optimized for improved response accuracy and instruction-following capabilities, which
can result in less sufficient alignment in terms of
safety. Nonetheless, PopAlign outperforms various
baselines such as Context Distillation and RLCD in
terms of both harmlessness and helpfulness, highlighting the importance of incorporating more diverse contrasting strategies.
**Evaluating PopAlign on General Leaderboards.**
We also run our PopAlign workflow for overall
alignment, which is evaluated on three leaderboards: MT-Bench, AlpacaEval 2.0, and Arena
Hard. Notably, since the GPT-4 evaluator exhibits
an unreasonable bias towards longer responses, we
use the length-controlled win rate on AlpacaEval
2.0 to achieve higher agreement with human preference (Dubois et al., 2024). The evaluation results are presented in Table 1. We can observe that
PopAlign significantly enhances the model’s leaderboard performance, even surpassing the strong
baseline of Label-DPO.
We refer readers to Appendix E for comparative
demonstrations of different methods.


**sponses.** We define contrast accuracy as the percentage of synthesized response pairs where the
oracle model correctly identifies and prefers the
chosen response over the rejected one. Specifically,
given a response pair ( _r_ [+] _, r_ _[−]_ ) in the synthesized
preference dataset, an oracle model is asked to select the preferred response. We then calculate the
ratio at which _r_ [+] is selected. We randomly selected 200 samples from the synthesized training
set of PopAlign for the evaluation using GPT-4 and
PairRM (Jiang et al., 2023), a well-known off-theshelf reward model. Unlike typical reward models
that score one response at a time, PairRM is a contextualized reward model capable of comparing
and ranking multiple responses simultaneously, resulting in higher comparative reward accuracies.
The evaluation results are shown in Table 2. Our
proposed novel contrasting strategy, Elicitive Contrast, demonstrates significantly higher contrast accuracy compared to all other strategies. This indicates the remarkable effectiveness of distilling
implicit preference knowledge from the teacher
model via elicitive prompting.
Moreover, both Model Contrast approaches
Method Harmless-Base Helpful-Base MT-Bench AlpacaEval 2.0 Arena Hard
Yi-6B-Chat 48.4 36.0 6.0 11.8 4.1
Label-DPO 50.9 50.2 6.5 15.8 5.7
RLAIF 49.5 34.5 6.5 11.7 4.5
Context-Dist 43.4 31.9 5.9 10.3 **5.7**
RLCD 35.9 47.2 6.1 16.9 3.9
PopAlign **50.0** **50.0** **6.6** **19.0** 5.5
Table 1: Main Experiments. GPT evaluation is used to compare each baseline with PopAlign on two alignment
tasks ( _i.e._, Harmless-Base, and Helpful-Base). The scores under 50.0 refers to the inferior performance compared
with PopAlign. For the other three leaderboards ( _i.e._, MT-Bench, AlpacaEval 2.0, and Arena Hard), we use their
official GPT evaluation pipeline to obtain a point-wise score. Note that we report length-controlled win rate on
AlpacaEval 2.0 tasks due to its higher agreement with human annotation. Higher scores indicate better performance.
The best result is in **bold**, and the second-best result is underlined (with Label-DPO excluded).
|Strategy GPT-4 PairRM|Col2|
|---|---|
|Strategy
GPT-4
PairRM|Strategy
GPT-4
PairRM|
|Demon Contrast
Prefx Contrast
Elicitive Contrast
NParam Contrast
Leaderboard Contrast
Refne Contrast|76.5
65.5
75.5
56.5
**91.5**
**85.5**
88.0
73.0
84.0
65.5
55.5
50.5|
Table 2: Contrast Accuracy of the Synthesized Responses. We evaluate the percentage of synthesized
chosen responses preferred by an oracle model over the
rejected ones. We utilized GPT-4 and PairRM (Jiang
et al., 2023) as the oracle models due to their wellrecognized abilities in preference labeling. The best
result is highlighted in **bold** .
achieve good contrast accuracy, implying that our
Model Contrast methods can effectively capture
and instantiate human preferences. However, Demon Contrast and Prefix Contrast demonstrate inferior contrast accuracy compared to the third Prompt
Contrast approach, namely, Elicitive Contrast. Intuitively, both Demon Contrast and Prefix Contrast tend to cause the model’s output contrast patterns to collapse into the patterns specified by the
contrastive demonstrations or prefixes. This leads
to a static underlying contrast pattern that lacks
context-awareness. On the other hand, the contrast patterns guided by Elicitive Contrast are dynamic, _i.e._, allowing each instruction to generate
self-adaptive contrastive responses elicited by the
generated thoughts, resulting in more diverse and
robust contrast patterns. [4]
4“Dynamic” refers to a sample-level rather than a task-level
dependency (Liu et al., 2023a). “Self-Adaptive” means that
the contrast is tailored to each instruction.
Figure 3: The Cumulative Effect of Different Contrasting Strategies. Starting with Prefix Contrast, new contrasting strategies are incrementally added to assess their
cumulative effects.
Surprisingly, Refine Contrast has limited contrast accuracy. Through our case studies (see Appendix D), we found that the inaccurately chosen
responses are often too detailed and lengthy. This
outcome is natural, as if the model provides a sufficiently good response in the first dialogue turn,
subsequent prompts to further improve the response
tend to result in adding more details.
**Effect of Accumulating Contrasting Strategies.**
To evaluate the cumulative effect of different contrasting strategies, we begin with Prefix Contrast
and incrementally add new strategies: first Prompt
Contrast, followed by Model Contrast, and finally
Pipeline Contrast. In Figure 3, we observe that
while Demon Contrast significantly degrades the
ment when utilized in an accumulative setting, as
discussed in Section3.2.2. This suggests that Refine Contrast may serve as a regularization for PopAlign.
Notably, our Elicitive Contrast shows the most
comprehensive performance gains compared to the
other individual strategies, verifying the effectiveness of the dynamic and self-adaptive contrast,
which is tailored for each individual instruction.
Reward Reward
Method
Accuracy Margins
PairRM **78.9**   Label-RM 68.1  Label-DPO 68.7 21.4
RLCD 62.3 7.4
RLAIF 53.2 0.7
PopAlign **70.3** **70.2**
Table 3: Evaluating the preference modeling, LabelRM and Label-DPO are trained on the original label
responses in the training dataset. We report reward
accuracies and reward margins ( _i.e._, the mean difference
between the chosen and corresponding rejected rewards)
on the test split of UltraFeedback. The best scores are
highlighted in **bold** .
**Preference Modeling.** As shown in Table 3,
we evaluate the reward modeling performance of
PopAlign. We use three strong baselines: a strong
off-the-shelf reward model, PairRM (Jiang et al.,
2023), a reward model and a DPO model trained on
the original preference-labeled responses of the UltraFeedback’s training split. We compare PopAlign
with RLCD and RLAIF, both trained using DPO.
We observe that PopAlign demonstrates the highest preference modeling accuracies. Furthermore,
PopAlign exhibits the highest reward margins, indicating that its rewards for the ground-truth chosen
responses are significantly greater than those for
the rejected responses. Notably, the performance
of PopAlign’s reward model surpasses that of two
strong baselines, Label-RM and Label-DPO, suggesting that these baselines may overfit their training sets, while PopAlign demonstrates superior
generality. Notably, the reward accuracy of RLAIF
is nearly random. This occurs because the pairwise responses generated by RLAIF exhibit limited
contrast, resulting in high randomness in AI feedback. Consequently, the reward models learned
from these responses are inaccurate.
Figure 4: The impact of each contrasting strategy.
performance of Prefix Contrast in terms of harmlessness, it markedly enhances performance in
terms of helpfulness, indicating that these two contrasting strategies focus on different aspects which
may have an adversarial relationship. Additionally,
when accumulating Model Contrast approaches
from Prompt Contrast, or incorporating Pipeline
Contrast approaches further, there is a noticeable
performance gain, underscoring the importance of
the diversity of contrasting strategies.
**Effect of Individual Contrasting Strategies.**
We also conduct ablation experiments on individual contrasting strategies under the same settings as
used by PopAlign (as detailed in Appendix B). The
results in Figure 4 indicate that PopAlign, which integrates all the contrasting strategies, demonstrates
the best comprehensive performance.
In terms of harmlessness and helpfulness—two
specific alignment aspects—PopAlign shows suboptimal performance. Specifically, the Elicitive
Contrast is the most effective for helpfulness alignment, while the Leaderboard Contrast is the most
effective for harmlessness. However, for the leaderboard tasks ( _i.e._, Arena Hard and MT-Bench),
PopAlign achieves the highest scores, implying
its superior overall alignment effectiveness.
Furthermore, different contrasting strategies exhibit significantly varied alignment patterns. For
example, Refine Contrast is effective in terms of
harmlessness but shows limited performance in
helpfulness, whereas NParam Contrast exhibits the
opposite trend. Interestingly, although Refine Contrast demonstrates limited performance gains individually, it contributes significantly more improve
Model Data MT-Bench
Yi-6B-Chat     - 6.0
PopAlign-Yi Yi 6.6
LLaMA3-8B-Instruct  - 8.0
PopAlign-LLaMA3 Yi 8.2
Table 4: The effect of different models to be aligned.
Both PopAlign-Yi and PopAlign-LLaMA3 is trained on
the same data synthesized by Yi series (AI et al., 2024)
as detailed in Section3.1.3.
**Effect of Different Models to be Aligned.** The
previous experiments primarily focused on the
Yi series, with the teacher model being Yi-34BChat (AI et al., 2024). To assess the impact
of using different models to be aligned with the
same synthesized data, we also train LLaMA-38B-Instruct (AI@Meta, 2024). Evaluation on MTBench demonstrates that our method remains effective for LLaMA-3, even though the data synthesis
procedures are not tailored for the LLaMA-3 series. However, the performance improvement from
PopAlign for LLaMA-3 is more limited compared
to PopAlign-Yi. This implies the importance of
minimizing the gap between the models for data
synthesis and the models being aligned.
Method Help/Harm MT A-E A-H
Yi-6B-Chat 48.4/36.0 6.0 11.8 4.1
PopAlign-DPO **50.0** /50.0 6.6 **19.0** **5.5**
PopAlign-PPO 40.4/ **54.1** **6.7** 14.1 4.9
Table 5: Impact of DPO and PPO on the effectiveness
of PopAlign. “Help/Harm” denotes “Helpful-Base” and
“Harmless-Base”, respectively. “MT”, “A-E”, and “A-H”
represent “MT-Bench”, “AlpacaEval 2.0”, and “ArenaHard”, respectively. The experimental procedures are
consistent with those detailed in Section3.1.

Contrast Accuracy of the Synthesized Re-


**gorithms.** We further investigate the impact of
employing different preference optimization algorithms, specifically DPO (Rafailov et al., 2024) and
PPO (Schulman et al., 2017), on the effectiveness
of PopAlign. As shown in Table 5, PopAlign-DPO
demonstrates superior performance in helpfulness
alignment. Conversely, PopAlign-PPO excels in
harmlessness alignment. However, on the AlpacaEval 2.0 and Arena-Hard benchmarks, PopAlignDPO achieves higher scores. These results highlight that PopAlign-DPO offers better alignment
in terms of both helpfulness and general align
ment performance, while PopAlign-PPO provides
enhanced safety features. This finding suggests
that reward models are more effective at capturing
response variations associated with harmlessness,
while their ability to discern other more intricate or
subtle preferences is comparatively limited.

Effect of Different Preference Optimization Al-


Typically, RLHF (Ouyang et al., 2022; Bai et al.,
2022a; Ganguli et al., 2022; Askell et al., 2021)
is used for alignment training, requiring laborintensive human preference annotation. Recently,
AI feedback has been proposed to replace human
feedback (Lee et al., 2023; Bai et al., 2022b). However, these methods generate pairwise responses
using the same prompt template and model, resulting in low contrast and further inaccurate reward
models ( _c.f._, Section3). To address this, various methods enable LLMs to generate contrastive responses.
For example, RLCD (Yang et al., 2023) uses contrastive prefixes in prompts, while Contrastive PostTraining (Xu et al., 2023) and ALMoST (Kim
et al., 2023) employ models with performance gaps
and varying demonstration qualities or quantities.
LLaMA 2 (Touvron et al., 2023b) enhances the
response diversity by varying model variants and
decoding temperatures. But these methods only
consider limited contrasting patterns.
In our work, we argue that diversifying contrasting patterns are vital for a more comprehensive
LLM alignment. Our work not only proposes novel
contrasting strategies such as Elicitive Contrast, but
also rigorously investigates the impact of contrasting pattern diversification from a systematic and
comprehensive perspective, _i.e._, Prompt-ModelPipline. By diversifying the contrastive strategies,
our method demonstrates superior gains.


In this paper, we presented PopAlign, a novel
framework that enhances the alignment of LLMs
by diversifying contrasting patterns across prompt,
model, and pipeline levels. By introducing six
distinct strategies, PopAlign enables the construction of comprehensive preference-contrastive data
without extra feedback labeling. Our experiments
demonstrate that this diversified approach leads
to significantly improved alignment performance
compared to traditional methods.


While our work demonstrates superior performance, it has several limitations. Firstly, we have
not yet verified our method with more large language models (LLMs) or with LLMs of larger sizes.
Investigating the scaling law of our methods is an
interesting and valuable area for further research.
Secondly, our method integrates six contrasting
strategies, but there are potentially more contrasting strategies that could be explored. Thirdly, we
use DPO/PPO in our work for alignment training;
it would also be interesting to explore the influence
of different alignment training algorithms.


Our work is based on Large Language Models
(LLMs), which can generate potentially harmful
and unfaithful responses. Our proposed method,
PopAlign, aims to address this issue more effectively; however, due to the nature of LLMs, it may
still produce sensitive content. It is important to
note that the data we used contains potentially sensitive content, which is utilized to improve the models’ helpfulness and reduce harmfulness.


We would like to express our gratitude to Prof.
Mrinmaya Sachan for his valuable and constructive
feedback on this work.


**Prompt Contrast.** All the Prompt Contrast approaches are based on Yi-34B-Chat (AI et al.,
2024), including Prefix Contrast, Demon Contrast, and Elicitive Contrast. Table 6 and Table 7 presents the positive and negative prompt
templates used to elicit the synthesized chosen
and rejected responses, respectively. Note that
the chat template for Yi-34B-Chat is “<im_start>
user\n {query} <im_end>\n<im_start> assistant\n
{response} <im_end>”. For Prefix Contrast, the
prefixes are inserted just after the “assistant\n”.
While for the others, the prompt templates are
placed in the user’s part. We extract the responses
from the model outputs. For example, we need to
eliminate the prefixes from the assistant messages
when using Prefix Contrast. And we should remove
the thoughts when it comes to Elicitive Contrast.
**Model Contrast.** For NParam Contrast, we use
Yi-34B-Chat (AI et al., 2024) to generate the synthesized chosen responses and use Yi-6B-Chat to
generate the rejected ones. For leaderboard tasks,
no prompt template is employed. While for alignment tasks, we also add a positive prefix “(helpful,
harmless)” for Yi-34B-Chat and a negative prefix
“(unhelpful, harmful)” for Yi-6B-Chat. For Leaderboard Contrast, we use Yi-34B-Chat to generate
the synthesized chosen responses and use Vicuna33B (Chiang et al., 2023) to generate the rejected
ones. There is no prompt template for both models
and for both tasks when using Leaderboard Contrast.
**Pipeline Contrast.** For Refine Contrast, we use
Yi-34B-Chat. The synthesized rejected responses
are from the first dialogue turn. While the chosen
ones are from the second turn. The user query for
this second turn is “Please make your response
more helpful and harmless.\n Format your output as follows:\n Thought:\n <Insights on how
to refine your response>\n Response:\n<A more
helpful and harmless response>” for alignment
**Prefix Contrast** (+) **Prefix Contrast** ( _−_ )
(helpful, harmless) (unhelpful, harmful)
**Demon Contrast** (+) **Demon Contrast** ( _−_ )
What are some effective ways to reduce stress?
Regular exercise, meditation, and proper sleep can
significantly help in reducing stress. It’s also beneficial to engage in hobbies and to talk about your
feelings with someone you trust.
How do we measure the age of the universe?
Scientists measure the age of the universe using the
redshift of distant galaxies and the cosmic microwave
background radiation. These methods suggest that
the universe is approximately 13.8 billion years old.
Can you explain Newton’s three laws of motion?
Certainly! Newton’s First Law states that an object at
rest stays at rest and an object in motion remains in
motion unless acted upon by an external force. His
Second Law relates the force acting on an object to
its mass and acceleration (F=ma). The Third Law
states that for every action, there is an equal and opposite reaction.
What are some effective ways to reduce stress?
Just don’t think about it. It’s all in your head.
How do we measure the age of the universe?
I guess scientists just make a wild guess based on
some old rocks or something.
Can you explain Newton’s three laws of motion?
They’re just some rules about stuff moving and stopping. Not that important really.
**Elicitive Contrast** (+) **Elicitive Contrast** ( _−_ )
Please first consider the principles of crafting a helpful and harmless response, and then generate the response. Format your output as follows: Thought: <Insights on creating a helpful and harmless response>
Response: <The helpful and harmless response>
Please first consider the principles of crafting a unhelpful or harmful response, and then generate the response. Format your output as follows: Thought: <Insights on creating a unhelpful or harmful response>
Response: <The unhelpful or harmful response>
Table 6: Prompt templates to elicit the chosen (+) and rejected responses ( _−_ ) in three Prompt Contrast approaches
for alignment tasks. The blue text means it is inserted into the part of the assistant message in the chat template.
tasks. For leaderboard tasks, it is changed into
“Please improve your response.\n Format your output as follows:\n Thought:\n <Insights on how to
refine your response>\n Response:\n<A better response>”. And then we extract the clean responses
from the model outputs. Note that for the second
turn, we need to double the max sequence length
for the model input.
**Hyperparameters for Data Generation.** The
max sequence length for the model inputs is set to
1024, except for the second turn when using Refine Contrast. The max new tokens is set to 1024
( _i.e._, the max length for the newly generated content). All other hyperparameters are set to their
default values as specified in HuggingFace Transformers (Wolf et al., 2020).


**Prompt Templates.** There is no prompt template
for DPO training. We align Yi-6B-Chat using
the original instructions and their synthesized responses.
**Hyperparameters.** We train the model for 1
epoch on the dataset with a micro batch size of
4. We use a cosine learning rate scheduler with a
peak learning rate of 5 _×_ 10 _[−]_ [7] and a warmup ratio
of 0.1. Gradient accumulation steps are set to 1,
and Flash Attention 2 (Dao, 2023) is enabled. The
beta for DPO training is set at 0.01. The maximum
sequence length is 2048. The training involves all
the model parameters, and the DeepSpeed stage is
ZeRO-2 (Wang et al., 2023). We implement the
DPO training with LLaMA-Factory (Zheng et al.,
**Prefix Contrast** (+) **Prefix Contrast** ( _−_ )
(good response) (bad response)
**Demon Contrast** (+) **Demon Contrast** ( _−_ )
What are some effective ways to reduce stress?
Regular exercise, meditation, and proper sleep can
significantly help in reducing stress. It’s also beneficial to engage in hobbies and to talk about your
feelings with someone you trust.
How do we measure the age of the universe?
Scientists measure the age of the universe using the
redshift of distant galaxies and the cosmic microwave
background radiation. These methods suggest that
the universe is approximately 13.8 billion years old.
Can you explain Newton’s three laws of motion?
Certainly! Newton’s First Law states that an object at
rest stays at rest and an object in motion remains in
motion unless acted upon by an external force. His
Second Law relates the force acting on an object to
its mass and acceleration (F=ma). The Third Law
states that for every action, there is an equal and opposite reaction.
What are some effective ways to reduce stress?
Just don’t think about it. It’s all in your head.
How do we measure the age of the universe?
I guess scientists just make a wild guess based on
some old rocks or something.
Can you explain Newton’s three laws of motion?
They’re just some rules about stuff moving and stopping. Not that important really.
**Elicitive Contrast** (+) **Elicitive Contrast** ( _−_ )
Please first consider the principles of crafting a good
response, and then generate the response. Format
your output as follows: Thought: <Insights on creating a good response> Response: <The good response>
Please first consider the principles of crafting a bad
response, and then generate the response. Format
your output as follows: Thought: <Insights on creating a bad response> Response: <The bad response>
Table 7: Prompt templates to elicit the chosen (+) and rejected responses ( _−_ ) in three Prompt Contrast approaches
for leaderboard tasks. The blue text means it is inserted into the part of the assistant message in the chat template.
2024b)’s codebase. All hyperparameters not mentioned are set to their default values as specified in
this codebase. Note that all the above parameters
are not only used in training PopAlign; all baselines
and ablated experiments are also based on these
parameters. However, due to the scale-up by contrasting strategy diversification, PopAlign is trained
using 32 A800-80GB GPUs, whereas other experiments are conducted with 8 A800-80GB GPUs.
For the reward model training in the ablation studies, the peak learning rate is adjusted to 1 _×_ 10 _[−]_ [6] .
All other parameters remain the same, with the
training conducted on 8 A800-80GB GPUs.


1. Harmless-Base subset in HH-RLHF (Bai et al.,
2022a): This subset contains dialogues that
include potentially harmful instructions. The
preferences are labeled by human annotators,
preferring responses that are ethical, inoffensive, and socially acceptable.
2. Helpful-Base subset in HH-RLHF (Bai et al.,
2022a): This subset is similar to the HarmlessBase but emphasizes the aspect of helpfulness.
3. AlpacaEval 2.0 (Li et al., 2023; Dubois
et al., 2023, 2024): A well-recognized leaderboard for evaluating the general instructionfollowing abilities of LLMs.
4. Arena Hard (Li et al., 2024): This leaderboard
involves 500 challenging instructions collected from live data in Chatbot Arena(Chiang
et al., 2024). It emphasizes domain knowledge, complexity, problem-solving, speci
ficity, creativity, technical accuracy, and the
real-world application of the instructions.
5. MT-Bench (Zheng et al., 2024a): This leaderboard contains 80 high-quality multi-turn
questions designed to assess models’ abilities in multi-turn conversation and instructionfollowing. It emphasizes evaluating advanced
abilities like math and reasoning.


Prompt templates for GPT Evaluation that are not
listed in this section can be found in the official
evaluation procedures of the corresponding task.