Alibaba Group
{bowen.xbw,wushaoyu.wsy,lk265203,chudu.hll}@alibaba-inc.com
ABSTRACT
With the proliferation of large language models (LLMs), the comprehensive alignment of such models across multiple tasks has emerged as a critical area of research.
Existing alignment methodologies primarily address single task, such as multi-turn
dialogue, coding, mathematical problem-solving, and tool usage. Although there is
a large amount of high-quality data available for those tasks, most of them provide
only questions and answers without including the system prompt. Though a detailed analysis of the Qwen language model, we found that the system prompt has a
significant impact on both training and inference processes of LLM. We attributes
this phenomenon to overfitting to the system prompt. In address this issue, we
introduce a novel technique termed Mixture-of-Instructions (MoI), which employs
a strategy of instruction packing combined with diverse system prompts to boost
the alignment efficiency of language models. We have also compiled a diverse set
of seven benchmark datasets to rigorously evaluate the alignment efficacy of the
MoI-enhanced language model. Our methodology was applied to the open-source
Qwen-7B-chat model, culminating in the development of Qwen-SFT-MoI. This
enhanced model demonstrates significant advancements in generative capabilities
across coding, mathematics, and tool use tasks.
1 INTRODUCTION
The rise of large language models (LLMs) has underscored the importance of effective training
techniques, encompassing Pre-training, Supervised Fine-Tuning (SFT), and Reinforcement Learning
from Human Feedback (RLHF), to expand their knowledge and ensure alignment with human values
for greater reliability. However, despite advancements, LLMs often falter in specialized tasks such
as coding, mathematics, and reasoning. With the growing access to high-quality, domain-specific
datasets(Luo et al., 2023c; Li et al., 2023a; CodeFuse, 2023; Wei et al., 2023; Yue et al., 2023; Yu
et al., 2023; Luo et al., 2023a; Zeng et al., 2023; Qin et al., 2023), this study investigates how targeted
SFT can boost LLM capabilities in these areas while preserving existing competencies.
We examined a case where the Qwen-7B-chat (Bai et al., 2023)model failed to learn Python code for
the Boyer-Moore majority vote algorithm. Training with standard code dataset proved ineffective,
with the model producing suboptimal solutions due to prompt-induced knowledge conflicts. To our
surprise, we found that simply changing to a different system prompt and retraining could easily
overcome this issue, indicating that altering prompts can resolve knowledge conflicts and improve
task performance across tasks.
Leveraging our initial findings, we developed a strategy that assigns unique system prompts to
different tasks and integrates these varying prompts into a unified instruction set for model alignment.
We introduce the Mixture of Instructions (MoI) methodology, a structured approach to amalgamate
open-source datasets for comprehensive training, spanning dialogue, code, math, and agent-based
tasks, each with bespoke prompts. This not only recasts training as a multi-task learning (Chung et al.,
2024; Wei et al., 2021; Raffel et al., 2020; Sanh et al., 2021) endeavor but also tackles challenges such
as dataset bias, where improvement in one task could detriment another. Through MoI, we adeptly
use diverse prompts to ensure balanced task performance, effectively navigating the complexities of
multi-task learning.
_∗_ equal contribution
Figure 1: Performance of Qwen-SFT-MoI and Qwen-7B-chat models, along with various SFT-aligned models
on subdomain datasets, evaluated across seven datasets encompassing mathematics, programming, tool usage,
common sense, and both single and multi-turn dialogues. Results demonstrate that training with our MoI method
enhances multiple capabilities of language models, achieving improved alignment.
Ultimately, we integrated the MoI with multi-turn dialogue, code generation, mathematics reasoning
and tool usage datasets for comprehensive training, resulting in a significantly enhanced language
model—Qwen-SFT-MoI. This model exhibited substantial improvement across a range of benchmarks, including MT-Bench(Zheng et al., 2024a), HumanEval(Chen et al., 2021), MBPP(Austin
et al., 2021), MATH(Hendrycks et al., 2021), GSM8K(Cobbe et al., 2021), MMLU(Hendrycks et al.,
2020), and T-EVAL(Chen et al., 2023), compared to the Qwen-7B-Chat model, validating our MoI
strategy for further alignment on an already SFT model. To summarize, our work makes the following
contributions:
- We observed the different impacts of system prompts on the LLM during training and inference,
and proposed a solution of setting domain-specific system prompts to address these issues. This
resulted in improved alignment with and performance on code generation task.
- Our MoI method facilitates joint multi-task training, effectively reducing dataset bias and preserving
conversational abilities. The efficacy of the MoI approach was substantiated through extensive
experimentation.
- By applying MoI alongside high-quality datasets on the Qwen-7B-chat model, we developed the
Qwen-SFT-MoI model. This model demonstrates superior performance in mathematical reasoning,
code generation, tool usage, and chat benchmarks, thereby highlighting the effectiveness of MoI in
enhancing SFT models.
2 MIXTURE OF INSTRUCTIONS
2.1 OVERFITTING TO THE SYSTEM PROMPT
Our motivation began with testing the performance of the Qwen-7B-chat(Bai et al., 2023) model. In
the current mainstream SFT schemes, an instruction is defined as consisting of three parts: system,
Figure 2: In the scenarios depicted, a system prompt was configured for the models. In the first case,
model effectively responded to questions aligned with the system prompt settings.
question, and answer. we conducted an interesting experiment by combining a date-related setting
with the default system prompt ’You are a helpful assistant’ to form a new system prompt. By
swapping the order of these two elements in a controlled experiment, we found that simply changing
the order significantly undermines the model’s accuracy in answering questions.The Qwen-7B-Chat
model stubbornly seeks information following the "You are a helpful assistant" keyword. When it
cannot find the expected content in this area, the model tends to create an answer rather than capturing
information that came before the key setting.
Could the reliance on system prompts during the inference process be attributed to an overfitting of
the model to the limited diversity of system prompts encountered during the alignment phase? To
test this hypothesis, we conducted a validation experiment on the MT-Bench(Zheng et al., 2024a)
dataset. A substantial amount of study(Hu et al., 2024; Zhang et al., 2024; Inc., 2023; Chiang et al.,
2023) indicates that small language model do not perform well on mathematical and reasoning tasks
within the MT-Bench. The Qwen-7B-chat also did not achieve high scores on the coding questions in
the MT-Bench, particularly to specific question: _Write a function to find the majority element in a_
_given integer array using the Boyer-Moore Voting Algorithm_ . Although Qwen-7B-chat successfully
identified the majority element, its approach resulted in a higher time complexity of O(nlog(n)) and a
space complexity of O(n), deviating from the expected O(n) time and O(1) space complexities of
the algorithm. Subsequently, we discovered this question corresponds to an original problem from
the LeetCode repository. Consequently, we downloaded 2612 LeetCode problems along with their
solutions and trained the Qwen-7B-chat model using SFT. However, we observed that the model,
even after SFT, still failed to answer this particular question.
Figure 3: Attention maps for responses to Question ID 127 in the MT-Bench, which show attention distribution
across the system prompt, the question, and each model’s answer. (a) Qwen-7B-chat focuses heavily on the
prompt and question but incorrectly associates ‘Boyer’ with frequency tracking, misrepresenting the BoyerMoore algorithm. (b) After SFT on code generation data, the model still overemphasizes the prompt and
question, overlooks ‘Boyer’ and fixates on iterative element search. (c) Post-SFT with a new system prompt, the
model shifts attention towards generating an answer, correctly hinting at the Boyer-Moore algorithm, which
fundamentally tracks a candidate element.
As depicted in Figure 3, analysis of attention maps for both the original Qwen-7B-chat and SFTenhanced models when processing a specific question revealed that the original Qwen-7B-chat model
Figure 4: (a). Sequence of Instructions involves extracting instructions from a dataset, tokenizing them, and
using padding tokens to reach a fixed maximum length for SFT in an LLM. (b). Packed Instructions merges
several instructions into a longer single instruction, minimizing the need for padding and enhancing training
efficiency. (c).Balanced Packed Instructions employs balanced sampling from various datasets and concatenates
instructions to meet a maximum token length. (d). Mixture of Instructions then prioritizes the instruction with
the default system prompt at the start.
favored array traversal and frequency counting, divergent from the Boyer-Moore algorithm’s approach.
The SFT model also disproportionately focused on array operations and unduly on the system prompt.
To redirect the attention of a large language model, we replaced its system prompt with _‘You are_
_a programmer’_ in the SFT regime, inspired by Wang et al. (2024). This change, as shown in
Figure 3, led to a realigned attention map that concentrated on the key ‘Boyer-Moore Algorithm’
principles, prompting accurate descriptions of the algorithm’s iteration and candidate tracking.
The shift in focus from the prompt to the algorithmic core problem and solution was significant.
Additionally, we evaluated the effects of varying SFT-driven system prompts on model performance
using the HumanEval, MBPP, and MT-Bench coding benchmark. Table 3 conveys that the _‘You are a_
_programmer’_ prompt engendered a notable uplift in code generation proficiency.
2.2 PACKING INSTRUCTIONS WITH BALANCED SAMPLING
In the previous experiments, we have demonstrated that system prompts significantly influence the
performance of language models during both training and inference. Therefore, can we achieve better
alignment of the model by constraining the system prompts? Numerous studies (Radford et al., 2019;
Sanh et al., 2021; Chung et al., 2024), demonstrates that pre-trained and fine-tuned language models
inherently possess multi-task capabilities. Wang et al. (2024) explores how incorporating character
definitions in system prompts can assist language models in achieving multi-domain alignment.
Meanwhile, Lu et al. (2024)reveals that the intrinsic capabilities of LLMs confine the knowledge
within role-play.
It seems that by assigning different system prompts to different tasks and training them together, we
can obtain a well-performing language model. However, when aggregating datasets from multiple
tasks for sequence training, we found that models trained on combined tasks performed worse than
those trained on individual tasks as shown in Table 2. This highlights a common issue in multitask learning: training on combined data can lead to model bias towards certain tasks, enhancing
performance in some while degrading it in others.
A direct approach to address this issue is to increase the batch size during model training(McCandlish
et al., 2018), thereby obtaining more balanced gradients, which optimizes the model’s alignment. In
addition to directly increasing the batch size, packing multiple data samples into a single one (Wei
Figure 5: The **numbers** represent the simplified position IDs, which are used to generate the masked-out
position embedding. Comparison of different attention masks on the same data: (a) default attention mask for
sequence instruction concatenation, (b) default attention mask for balanced sampling concatenation, (c) specially
designed mutually isolated attention mask, and (d) our chunk-based attention mask.
et al., 2021; Raffel et al., 2020; Sanh et al., 2021; Chung et al., 2024) can implicitly achieve the effect
of a larger batch size. After applying the packing technique, we found that the model performed
better on both mathematical and coding tasks. However, its performance on the MT-Bench declined,
indicating that there is still room for improvement in the standard data packing approach.
To address this, we proposed a balanced sampling scheme on packed data. By using a resampling
method (Ming et al., 2018), we ensure that the amount of data from different tasks within a single
packed instruction is as balanced as possible. The structure of the balanced sampling method is
illustrated in the Figure 4 which sequentially collects instructions from datasets of various tasks,
decodes these instructions into tokens, and combines them. The effects of using balanced sampling
are shown in Table 2.
2.3 CHUNK-BASED ATTENTION MASKING
Although a balanced sampling scheme can mitigate dataset biases and promote domain-specific
learning in models, both packing and balanced sampling methods combine multiple training examples
into a single sequence without applying an attention mask to prevent tokens from attending across
packed example boundaries. This phenomenon, known as attention cross-contamination(Krell et al.,
2021; Zhao et al., 2024), can affect model performance. Our experiments, shown in Table 8, indicate
that attention cross-contamination lowers the model’s performance on complex reasoning tasks.
However, it can also enhance the model’s abilities in code and mathematical tasks.
Zhao et al. (2024) reveals that achieving optimal training results during pre-training requires making
the packed training examples as similar as possible and isolating them using attention masks. Our
proposed chunk-based attention masking approach shares similarities with this scheme, but we
construct chunks within each balanced sampled packed instruction based on the number of domains
(such as the four domains in this paper) and ensure that each chunk is sorted in the same order.
By applying chunk-based attention masking, we block interference between different chunks while
ensuring high similarity between the data in each pair of chunks. Our experimental results, shown in
Table 8, demonstrate that by using chunk-based attention, the mathematical and coding capabilities
derived from packing are preserved while also enhancing the model’s complex reasoning ability.
Figure 5 provides a schematic representation of the chunk-based attention mask implementation.
2.4 MIXTURE OF INSTRUCTIONS
Ultimately, we propose the Mixture of Instructions(MoI) scheme, which comprises several key
components. First, we set unique system prompts for each domain-specific task to ensure tailored
guidance. Next, we employ balanced sampling techniques to concatenate task data from multiple
domains into proportionally balanced datasets. During training, we apply a chunk-based attention
mask to isolate unrelated contexts, thereby achieving optimal alignment. By explicitly modifying the
optimization objective of standard SFT, MoI achieves superior alignment.
During the standard SFT processing, language models can learn alignment by fine-tuning on highquality pairs of system prompt _s_, question _x_ and answer _y_ . Processed through an auto-regressive
LLM weights _θ_ to maximize the probability _Pθ_ ( _y|_ [ _s, x_ ]). The loss function of SFT should be:
_L_ SFT( _θ_ ) = _−_ E[log _Pθ_ ( _y |_ [ _s, x_ ])] (1)
Specifically, in the implementation process, we compute the loss for each data point and then divide
it by the number of tokens in the response. Then the loss of each batch in following format:
_Lseq_ ([ _s,x,y_ ])( _θ_ ) = _−_ ~~�~~ _N_
_n_ =1 _[|][y][n][|]_
_N_
- _logPθ_ ( _yn |_ [ _s, x_ ] _n_ ) (2)
_n_ =1
where _N_ indicates the batch size number, _|y|_ denotes the number of tokens in the response and the
[ _·_ ] operation represents token concatenation. In computing the loss function for packed data, we
followed a loss calculation scheme that avoids the suppression of the loss for shorter responses in
longer packed sequences.
_Ln_ +1
_i_ = _Ln_
_Lpacked_ ([ _s,x,y_ ])( _θ_ ) = _−_ _N_ [1] ˆ
_N_ ˆ
_n_ =1
_logPθ_ ( _yi |_ [[ _s, x, y_ ] _<i,_ [ _s, x_ ] _i_ ])
(3)
( _Ln_ +1 _−_ _Ln_ ) _· |yi|_
Here, _Ln_ represents the index number of the packed data within the original dataset, _N_ [ˆ] represents the
batch size of the packed data. Despite calculating the probability of obtaining _y_ based on _s_ and _x_, the
model implicitly treats all preceding packed data [ _s, x, y_ ] as part of the context, thereby interfering
with the model’s alignment. We additionally generate an attention mask and position ID for each
instruction during packing, thereby eliminating the impact of [ _s, x, y_ ] on the loss.
_Lmaskingout_ ([ _s,x,y_ ])( _θ_ ) = _−_ _N_ [1] ˆ
_N_ ˆ _Ln_ +1
-
_n_ =1 _i_ = _Ln_
_logPθ_ ( _yi |_ [ _s, x_ ] _i_ )
(4)
( _Ln_ +1 _−_ _Ln_ ) _· |yi|_
Specifically, in an ideal scenario where the number of instructions in each new packed instruction
and the token length of each instruction’s response are constant, The form of the loss function will
become identical to that in sequence training 2:
_Lmaskingout_ ([ _s,x,y_ ])( _θ_ ) = _−_ ~~�~~ _N_
_i_ =1 _[|][y][i][|]_
_N_
- _logPθ_ ( _yi |_ [ _s, x_ ] _i_ ) (5)
_i_ =1
By combining _Lpacked_ and _Lmaskingout_, the loss function of the MoI scheme by employing a
chunk-based attention mask:
_Ln_ +1
_nmix_
_i_ = _Ln_
_nmix_
( _i_ +1) _·nmix_
_j_ = _i·nmix_
_LMoI_ ([ _s,x,y_ ])( _θ_ ) = _−_ _N_ [1] ˆ
_N_ ˆ
_n_ =1
_logPθ_ ( _yj |_ [[ _s, x, y_ ] _i·nmix<j,_ [ _s, x_ ] _j_ ]) (6)
( _Ln_ +1 _−_ _Ln_ ) _· |yj|_
_n_ mix is defined as the number of instructions in a chunk. Consequently, when calculating the loss,
the model only considers the context within the same chunk, minimizing cross-contamination of
attention. MoI refines a single packed data sequence into smaller chunks, creating a multi-level
mixture of packed data. This methodology is the basis for the term ‘Mixture of Instructions’.
3 EXPERIMENTS
3.1 EXPERIMENT SETUP
All of our experiments are performed on 8x80G A100 GPUs, using the Qwen-7B-chat model as the
base for each independent experiment.We utilized the Huggingface Transformers(Wolf et al., 2019)
library to execute our training. During training, we employed DeepSpeed(Rasley et al., 2020) Zero3
optimization and Flashattention-2 (Dao, 2023) to enhance memory efficiency.
Figure 6: Qwen-SFT-balanced represents the results of data concatenation using balanced sampling, while
Qwen-SFT-concat represents the results of regular packed data concatenation. (a) Performance on MBPP dataset
using two methods across iteration steps. (b) Comparative performance on GSM8K dataset with two approaches
over iterations. (c) Performance evaluation on HumanEval dataset using two methods across iterations. (d)
Performance trajectory on MATH dataset under two methodologies over iteration steps.
3.2 TRAINING DATASET
we gathered open-source datasets for four tasks: multi-turn dialogue, code generation (Ding et al.,
2023; Wei et al., 2023; Zheng et al., 2024b; Luo et al., 2023c) mathematics reasoning (Yue et al.,
2023; Yu et al., 2023), and tool usage (Yang et al., 2024; Qin et al., 2023; Zeng et al., 2023), providing
a specific system prompt for each, The detailed information of the data is presented in Table 1.
Table 1: All of our training data are sourced from open datasets downloaded from Huggingface, categorized
into four tasks: multi-turn dialogue, code generation, mathematics reasoning, and tools usage. Custom system
prompts were individually crafted for the dataset of each task to ensure precise alignment with their respective
|Dataset|Task|System Prompt|
|---|---|---|
|Ultrachat200k|Multi-turn dialogue|You are a helpful assistant.|
|MagicCoder110k, EvolInstructCode80k, CodeFeedback157k|Code generation|You are a programmer.|
|MathInstruct262k, MetaMath395k|Mathematics reasoning|You are a mathematics expert.|
|GPT4Tools75k, ToolbenchG3, AgentInstruct|Tool usage|You are an Agent with a lot of tools.|
3.3 EVALUATION BENCHMARK
The validation datasets used in this paper include: GSM8K (Cobbe et al., 2021),MATH
dataset (Hendrycks et al., 2021),SVAMP (Patel et al., 2021), ASDiv (Miao et al., 2021),HumanEval (Hendrycks et al., 2021) MBPP (Austin et al., 2021),T-EVAL (Chen et al.,
2023),MMLU (Hendrycks et al., 2020),BBH-hard (Suzgun et al., 2022) and MT-Bench (Zheng
et al., 2024a). For detailed description, please refer to A.1.
3.4 ABLATION STUDY AND ANALYSIS
**The impact of different system prompts.** By modifying system prompts for SFT, we can comprehensively enhance the alignment performance of models. As shown in Table 3, the experimental
results indicate that MoI can transfer abilities originally elicited only under their respective system
prompts to the default system prompt.
**The impact of training on different task datasets.** Our comparative ablation study in Table 2
revealed that while models trained on math, and code data respectively enhance performance in their
specific tasks, this specialization comes at the cost of diminished multi-turn conversation ability, as
evidenced by a significant drop in MT-Bench 2nd turn scores for all models. Thus, models trained on
these datasets are recommended for domain-specific tasks rather than as conversational assistants.
Additionally, models trained on the agent dataset did not show improvement in any metric, suggesting
that the T-Eval dataset serves as a comprehensive test scheme to assess model capabilities.
**Dataset bias.** Figure 6 shows our ablation study compared Qwen-SFT-packed and Qwen-SFTbalanced, demonstrating that balanced sampling rapidly improves GSM8K accuracy and stabilizes
code generation performance during training. In contrast, packing showed no substantial late-stage
Table 2: Performance of SFT models on widely-used benchmarks. We report zero-shot performance of the
models.For all benchmark computations, we consistently used the system prompt **"You are a helpful assistant."**
to ensure the effectiveness of our methodology. When computing the average score, we multiplied the mt-bench
scores by 10 to ensure numerical validity.The comparative values in the data are all benchmarked against
Qwen-7B-chat. For detailed description, please refer to A.2.
|Model|MMLU MBPP HumanEval GSM8K MATH T-Eval MT-Bench MT-Bench Avg<br>0-shot pass@1 pass@1 0-shot 0-shot overall 1st turn 2nd turn|
|---|---|
|Qwen-7B-chat|57.0<br>31.8<br>37.2<br>50.3<br>6.8<br>60.7<br>5.65<br>4.51<br>43.1|
|Qwen-SFT-code<br>Qwen-SFT-math<br>Qwen-SFT-agent<br>Qwen-SFT-chat|41.5(_−_15_._5)<br>53.0(+21_._2)<br>56.1(+18_._9)<br>34.0(_−_16_._3)<br>0.0(_−_6_._8)<br>22.3(_−_38_._4)<br>5.70(+0_._05)<br>3.13(_−_2_._38)<br>36.9(_−_6_._2)<br>45.9(_−_11_._1)<br>36.8 (+5_._0)<br>22.5(_−_14_._7)<br>71.8(+21_._6)<br>27.8(+21_._0)<br>14.0(_−_46_._7)<br>5.58(_−_0_._07)<br>3.36(_−_1_._15)<br>38.5(_−_4_._6)<br>47.7(_−_9_._3)<br>15.0(_−_16_._8)<br>20.7(_−_16_._5)<br>16.1(_−_34_._2)<br>0.1(_−_6_._7)<br>47.1(_−_13_._6)<br>4.41(_−_1_._24)<br>2.87(_−_1_._64)<br>27.4(_−_15_._7)<br>57.8(+0_._8)<br>12.6(_−_19_._2)<br>22.5(_−_14_._7)<br>54.5(+4_._2)<br>0.0(_−_6_._8)<br>60.8(+0_._1)<br>5.73(+0_._07)<br>4.45(_−_0_._06)<br>38.7(_−_4_._4)|
|Qwen-SFT-seq<br>Qwen-SFT-packed<br>Qwen-SFT-balanced|59.2(+2_._2)<br>46.6(+14_._8)<br>50.6(+13_._4)<br>69.2(+18_._9)<br>30.3(+23_._5)<br>**69.6(**+8_._9**)**<br>5.81(+0_._16)<br>4.60(+0_._09)<br>53.7(+10_._6)<br>**59.7(**+2_._7**)**<br>49.0(+17_._2)<br>56.1(+18_._9)<br>71.3(+21_._0)<br>30.3 (+23_._5)<br>67.1(+6_._4)<br>5.99(+0_._34)<br>4.20(+0_._05)<br>54.4(+11_._3)<br>59.4(+2_._4)<br>51.6(+19_._8)<br>55.5(+18_._3)<br>71.9(+21_._6)<br>30.0(+23_._2)<br>66.9(+6_._2)<br>6.12(+0_._47)<br>5.01(+0_._50)<br>55.8(+12_._7)|
|Qwen-SFT-MoI|59.3(+2_._3)<br>**61.0**(+29_._2)<br>**60.4**(+23_._2)<br>**74.5**(+24_._5)<br>**31.9**(+25_._1)<br>68.2(+7_._5)<br>6.91(+1_._26)<br>4.75(+0_._24)<br>**59.9**(+14_._2)|
Table 3: Performance of SFT models with different system prompt on widely-used code generation and
mathematics benchmarks. We report both zero-shot and few-shot performance of the models.
|Model|System prompt|MBPP<br>Pass@1|HumanEval<br>Pass@1|MT-Bench<br>Coding 1st turn|GSM8K<br>Accuracy(%)|MATH<br>Accuracy(%)|MT-Bench<br>Math 1st turn|
|---|---|---|---|---|---|---|---|
|Qwen-7B-chat|Assistant|31.8|37.2|3.8|50.3|6.8|2.65|
|Qwen-SFT-code|Assistant|53.0|56.1|4.1|34.0|0.0|2.3|
|Qwen-SFT-code|Programmer|55.2|60.4|**5.0**|32.8|0.0|2.1|
|Qwen-SFT-math|Assistant|36.8|22.5|1.6|71.8|27.8|4.9|
|Qwen-SFT-math|Mathematics expert|34.9|20.3|1.7|72.1|30.0|**5.0**|
|Qwen-SFT-MoI|Assistant|**61.0**|**60.4**|4.7|**74.5**|**31.9**|5.0|
gains in GSM8K and a decline in code capability. With identical datasets and hyperparameters, these
differences highlight balanced sampling superior handling of multi-task learning in LLM.
**The impact of different data sampling strategies.** Based on Table 2, it can be observed that the
sequence strategy significantly enhances the model’s performance across all validation datasets.
The packing strategy, building upon the sequence sampling approach, further improves the model’s
performance on mathematical and coding tasks, yet it diminishes the model’s ability in conducting
two-round dialogues. The balance strategy maintains the model’s capability in two-round dialogues,
though there is a slight performance decline on the HumanEval and MATH datasets. Nonetheless,
there is an overall increase in the model’s average (Avg) performance metric. The MoI strategy, by
integrating chunk-based attention on top of the balance approach, achieves substantial improvements
across all tasks.
**Comparison with open-source models.** Table 4 shows that despite the lack of extensive code
pre-training, our model, through high-quality SFT and the MoI method, matches the performance
of contemporary code models with similar parameters. Comparisons with open-source math reasoning models indicate our model’s math skills are on par with those fine-tuned solely on math
datasets. Additionally, our model exhibits improved multi-turn dialogue and common sense reasoning
capabilities.
**Solving math problems with code interpreter.** We attempted to have the MoI model use a code
interpreter for solving mathematical problems. Our evaluation extends beyond MATH and GSM8K
datasets to include SVAMP and ASDiv, aiming for a thorough appraisal of mathematical proficiency.
Table 5 shows our MoI model, untrained on code interpretation tasks, harnesses integrated Math,
Agent, and Coding proficiencies to approach problems. Despite falling short of CoT outcomes, our
findings affirm MoI’s potential to equip models with a synergistic blend of skills.
**Effectiveness on small language models and Llama family.** We tested the scalability of the MoI
method on smaller models. It can be observed in Table 7 that MoI enhances coding and mathematical
capabilities on both 1.8B and 4B version of Qwen1.5. Experimental results on Llama2-7B and
Llama3-8B are summarized in Table 6, which demonstrates that MoI could further improves the
performance of trained LLMs that were already performing well.
**What is the impact of attention masking in the training of MoI?** We utilized the method
outlined in (Dao, 2023) to create a 2D Flash Attention mask, which was then input into the
flash_attn_varlen_func function to prevent cross-contamination in attention. Table 8 shows
Table 4: Scores on HumanEval, MBPP, GSM8K, MATH. The pass@1 scores of our models are computed with
greedy decoding. Models are evaluated in 0-shot on all datasets.
|Model|Params|HumanEval MBPP GSM8K<br>pass@1 pass@1 Accuracy(%)|Col4|Col5|MATH<br>Accuracy(%)|
|---|---|---|---|---|---|
|GPT-3.5(OpenAI, 2022)<br>GPT-4(OpenAI, 2023)|-<br>-|73.2<br>86.6|81.7<br>83.0|80.8<br>92.0|35.5<br>42.5|
|StarCoder Base(Li et al., 2023a)<br>Code Llama(Rozière et al., 2023)<br>Code Llama-Instruct(Rozière et al., 2023)<br>Code Llama-Python(Rozière et al., 2023)<br>WizardCoder(Luo et al., 2023c)<br>Magicoder(Wei et al., 2023)|7B<br>7B<br>7B<br>7B<br>7B<br>7B|24.4<br>33.5<br>34.7<br>38.4<br>48.2<br>60.4|33.1<br>41.4<br>44.4<br>47.6<br>56.6<br>**64.2**|-<br>-<br>-<br>-<br>-<br>-|-<br>-<br>-<br>-<br>-<br>-|
|WizardMath(Luo et al., 2023a)<br>MetaMath(Yu et al., 2023)<br>MAmmoTH(Yue et al., 2023)|7B<br>7B<br>7B|50.0<br>-<br>24.4|53.9<br>-<br>42.4|54.9<br>66.4<br>59.4|10.7<br>19.4<br>33.4|
|Llama2(Touvron et al., 2023b)<br>Mistral(Jiang et al., 2023)<br>EURUS-SFT(Yuan et al., 2024)<br>Qwen(Bai et al., 2023)|7B<br>7B<br>7B<br>7B|12.2<br>27.4<br>55.5<br>37.2|20.8<br>38.6<br>59.1<br>35.8|16.7<br>47.5<br>-<br>52.5|3.3<br>11.3<br>32.6<br>10.3|
|Qwen-SFT-MoI (ours)|7B|**60.4**|61.0|**74.5**|31.9|
Table 5: Scores on GSM8K, MATH, SVAMP, ASDiv. The Accuracy and pass@1 scores of our models are
computed with greedy decoding. Models are evaluated in 0-shot on all datasets.
|Model|mode|GSM8K<br>Accuracy(%)|MATH<br>Accuracy(%)|SVAMP<br>Accuracy(%)|ASDiv<br>Accuracy(%)|Avg<br>Accuracy(%)|
|---|---|---|---|---|---|---|
|GPT-4(OpenAI, 2023)<br>GPT-4(OpenAI, 2023)|CoT<br>Code Interpreter|92.0<br>94.2|42.5<br>51.8|93.1<br>94.8|91.3<br>92.6|79.8<br>83.3|
|WizardMath-Llama2-7B(Luo et al., 2023a)<br>MAmmoTH-CodeLlama-7B(Yue et al., 2023)<br>CodeLlama-7B(Rozière et al., 2023)<br>ToRA-CodeLlama-7B(Gou et al., 2023)<br>OpenMath-CodeLlama-7B(Toshniwal et al., 2024)|CoT<br>CoT<br>Code Interpreter<br>Code Interpreter<br>Code Interpreter|54.9<br>59.4<br>34.0<br>72.6<br>75.9|10.7<br>33.4<br>16.6<br>44.6<br>43.6|57.8<br>71.4<br>59.0<br>70.4<br>79.6|73.5<br>72.3<br>61.4<br>78.7<br>77.7|49.2<br>59.1<br>42.8<br>66.6<br>69.2|
|Qwen-SFT-MoI (ours)<br>Qwen-SFT-MoI (ours)|CoT<br>Code Interpreter|74.5<br>67.2|31.9<br>31.7|**84.4**<br>82.3|**88.7**<br>84.2|**69.9**<br>66.3|
Table 6: Evaluation of the MoI training scheme on open-source Llama2 and Llama3 models.
|Model|HumanEval<br>pass@1|MBPP<br>pass@1|GSM8K<br>Accuracy(%)|MATH<br>Accuracy(%)|MMLU<br>Accuracy(%)|Avg<br>Accuracy(%)|
|---|---|---|---|---|---|---|
|Llama2-7B-chat(Zhang et al., 2024)<br>Llama3-8B-Instruct(Team et al., 2024)|12.8<br>62.2|20.8<br>70.1|16.7<br>79.5|3.3<br>30.0|47.8<br>67.2|20.3<br>61.8|
|Llama2-7B-MoI (ours)<br>Llama3-8B-MoI (ours)|61.0 (+48_._2)<br>64.0(+1_._8)|52.8 (+32_._0)<br>70.9(+0_._8)|74.5 (+57_._8)<br>80.2(+0_._7)|30.7 (+27_._4)<br>**35.9**(+5_._9)|52.2(+4_._7)<br>67.4(+0_._2)|54.2 (+33_._9)<br>63.7(+1_._9)|
that SFT with masking significantly improves reasoning ability on the T-Eval dataset. However, it
also leads to a considerable decline in performance on the GSM8K, MATH, and MBPP validation
sets. We hypothesize that this issue stems from attention contamination. The final MoI solution,
employing a chunk-based attention mask, effectively balances task metrics and mitigates attention
contamination.
**Why is MoI effective?** We hypothesize that training with mixed instructions can significantly alter
the attention distribution of an SFT model, thereby enhancing its alignment performance. To test
this, we conducted a case study by replacing the initial chat model’s weights with those from an
SFT-trained model, specifically targeting attention and other weights, and observed performance
changes on a specific dataset. As shown in Table 9, replacing the chat model’s attention weights with
MoI’s attention weights resulted in a performance decline, a phenomenon not observed in other SFT
models. This suggests that the MoI model’s attention mechanisms are unable to extract the most
relevant knowledge from the original chat MLP. Additionally, the experiments in **??** also demonstrate
the robustness of MoI’s attention to irregular system prompts.
Table 7: Comparison of smaller models in the range of 1B to 4B parameters on a variety of benchmarks. MoI
|Model|HumanEval<br>pass@1|MBPP<br>pass@1|GSM8K<br>Accuracy(%)|MATH<br>Accuracy(%)|BBH<br>Accuracy(%)|MMLU<br>Accuracy(%)|Avg<br>Accuracy(%)|
|---|---|---|---|---|---|---|---|
|Tinyllama-1.1B(Zhang et al., 2024)<br>Gemma-2B(Team et al., 2024)<br>StableLM-Zephyr-3B(Bellagente et al., 2024)<br>MiniCPM-2B(Hu et al., 2024)<br>Phi-2(Li et al., 2023b)|6.7<br>22.0<br>35.4<br>**50.0**<br>47.6|19.9<br>29.2<br>31.9<br>47.3<br>**55.0**|2.3<br>17.7<br>52.5<br>53.8<br>57.2|0.7<br>11.8<br>12.5<br>10.2<br>3.5|28.8<br>35.2<br>37.7<br>36.9<br>**43.4**|24.0<br>42.3<br>45.9<br>**53.5**<br>52.7|13.7<br>26.4<br>36.0<br>42.0<br>43.2|
|Qwen1.5-1.8B-chat(Bai et al., 2023)<br>Qwen1.5-4B-chat(Bai et al., 2023)|20.1<br>25.6|18.0<br>29.2|38.4<br>57.0|10.1<br>10.0|24.2<br>32.5|46.8<br>56.1|26.3<br>31.7|
|Qwen1.5-1.8B-MoI (ours)<br>Qwen1.5-4B-MoI (ours)|32.3 (+12_._2)<br>40.8(+15_._2)|27.8 (+9_._8)<br>44.4(+15_._2)|55.2 (+16_._8)<br>**70.0**(+13_._0)|18.3 (+8_._2)<br>**23.8**(+13_._8)|29.8 (+5_._6)<br>34.7(+2_._2)|48.1(+1_._3)<br>58.5(+2_._4)|34.0 (+7_._7)<br>44.3(+12_._6)|
Table 8: Model performance with different attention mask methods. All data underwent balanced sampling and
reordering, with the difference being the application of different attention masks. **No attention mask** : The model
sees all packed questions and answers during loss computation. **Isolated attention mask** : A strict attention
mask and reset position IDs ensure the model sees only individual questions and answers within the packed data.
**Chunk-based attention mask** : Following task partitioning and system prompt reordering, internal visibility is
allowed among every four instructions (chat, code, math, agent) while each group of four is isolated from the
rest using an attention mask.
|Model|HumanEval<br>pass@1|MBPP<br>pass@1|GSM8K<br>Accuracy(%)|MATH<br>Accuracy(%)|T-Eval<br>Accuracy(%)|Avg<br>Accuracy(%)|
|---|---|---|---|---|---|---|
|No attention mask|57.9|61.8|75.4|31.7|64.3|58.2|
|Isolated attention mask|60.3(+2_._4)|54.8(_−_7_._0)|69.4(_−_6_._0)|29.5(_−_2_._2)|69.5(+5_._2)|56.7(_−_1_._5)|
|Chunk-based attention mask|60.4(+2_._5)|61.0(_−_0_._8)|74.5(_−_0_._9)|31.9(+0_._2)|68.2(+3_._9)|59.2(+1_._0)|
Table 9: Impact of replacing the weights of different components in language models from a Chat model
with those from an SFT model. **Attn** represents the weights of the q, k, v, and o in the attention layer. **Others**
represent the weights of the embedding, FFN, Layernorm, and lm_head.

Mixture-of-Instructions: Comprehensive Alignment of a Large Language
  Model through the Mixture of Diverse System Prompting Instructions

Bowen Xu _[∗]_ Shaoyu Wu _[∗]_ Kai Liu Lulu Hu


|Attn Others|Seq Packed Balanced MoI|Seq Packed Balanced MoI|
|---|---|---|
|Chat
Chat|37.2
37.2
37.2
37.2|50.3
50.3
50.3
50.3|
|SFT
SFT|50.6(+13_._4)
56.1(+18_._9)
55.5(+18_._3)
60.4(+23_._2)|69.2(+18_._9)
71.3(+21_._0)
71.9(+21_._6)
74.5(+24_._2)|
|SFT
Chat|42.7(+5_._5)
42.1(+4_._9)
40.2(+3_._0)
32.9(_−_4_._3)|62.9(+12_._6)
54.8(+4_._5)
60.2(+9_._9)
47.4(_−_2_._9)|
|Chat
SFT|48.8(+11_._58)
49.4(+12_._19)
50.6(+13_._4)
44.5(+7_._3)|50.5(+0_._2)
58.6(+8_._3)
44.0(_−_6_._3)
50.0(_−_0_._3)|
4 RELATED WORK
4.1 LLMS FOR VERSATILE TASKS: MATHEMATICS, CODING, AND TOOL UTILIZATION.
Solving complex mathematical tasks is challenging for open-source LLMs. Research focuses on
fine-tuning open-source models (e.g., Touvron et al. (2023a;b); Bai et al. (2023)) with insights from
advanced proprietary LLMs (OpenAI, 2023; 2022). WizardMath (Luo et al., 2023a) employs a
reinforced evol-instruct method, combining supervised fine-tuning with PPO training to enhance
reasoning. Meanwhile, MAmmoTH (Yue et al., 2023) uses Chain of Thought and Program-ofThought to enable LLMs to work with external tools, like Python interpreters, for mathematical
problem-solving.
LLMs have been specifically designed for code generation tasks, such as StarCoder (Li et al., 2023a)
and DeepSeek-Coder (Guo et al., 2024). Alternatively, code-generating LLMs can be derived
from fine-tuning general-purpose models, exemplified by CodeLlama (Rozière et al., 2023) and
WizardCoder(Luo et al., 2023c). Our approach aligns with the latter, fine-tuning general-purpose
LLMs to enhance code generation.
LLMs in real-world scenarios must adeptly select and apply tools from numerous APIs. Gorilla
(Patil et al., 2023) pairs LLMs with a broad API set, while ToolAlpaca (Tang et al., 2023) documents
3938 tool uses from over 400 real-world APIs in 50 categories. ToolLLM (Qin et al., 2023) further
contributes with ToolBench, a rich dataset featuring 16464 real-world APIs across 49 categories from
RapidAPI Hub, accommodating diverse single and multi-tool use cases.
4.2 MULTI-TASK LEARNING
During multi-task learning, dataset bias (Hadsell et al., 2020) is a phenomenon where two or
more tasks pull the model parameters in different directions, thus impacting the multi-task learning
performance. GPT-2(Radford et al., 2019) shows language models are unsupervised multitask learners
and Flan-T5(Chung et al., 2024) has demonstrated that language models can learn up to 1836 tasks.
Liu et al. (2024) proposes concatenating multiple datasets during training and using focal loss to
balance data bias among multiple tasks. Dong et al. (2023) proved multi-task learning of LLM lead
to conflicts, while sequential training results in catastrophic forgetting. In our work, we propose a
method to mitigate dataset bias that focuses on the data rather than the model.
4.3 SUPERVISED FINE-TUNING IN THE ALIGNMENT OF LLM
The training process of LLMs is categorized into pretraining and fine-tuning(Touvron et al., 2023b),
with Supervised Fine-Tuning (SFT) as key components of the fine-tuning phase. Some researchers,
such as (Wei et al., 2021), refer to SFT as instruction-tuning, while Ouyang et al. (2022) defines LLM
alignment as ’helpfully and safely following user instructions.’ Additionally, studies like (Luo et al.,
2023b; Zhang & Wu, 2024; Kotha et al., 2023; Razdaibiedina et al., 2023) have examined catastrophic
forgetting during fine-tuning. Furthermore, Wang et al. (2024) explores how incorporating character
definitions in system prompts can assist language models in achieving multi-domain alignment
while Ge et al. (2024) propose a novel persona-driven data synthesis methodology for SFT. Lu et al.
(2024)reveals that the intrinsic capabilities of LLMs confine the knowledge within role-play. In our
work, we explored how to efficiently use multiple system prompts to better continue aligning a model
that has undergone SFT.
5 CONCLUSION
In this work, we highlight the significant role of system prompts in enhancing language model
performance during SFT. By refining prompts, we mitigated issues in code generation with the
Qwen-7B-chat model, leading to better handling of complex tasks. Our innovative learning strategy
, Mixture of Instructions(MoI), which utilizes curated datasets with tailored prompts, successfully
tackled the dataset bias seen in multi-task training. The MoI method further ensured consistent
performance across tasks. Our finalized Qwen-SFT-MoI model showcased substantial improvements
over the original, underscoring MoI’s potential for improving language model adaptability and
proficiency.
REFERENCES
Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and
Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. _arXiv_
_preprint
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan,
Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language
models. _arXiv preprint
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge,
Yu Han, Fei Huang, et al. Qwen technical report. _arXiv preprint
Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi, Reshinth
Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, et al. Stable lm 2 1.6 b
technical report. _arXiv preprint
Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models in
retrieval-augmented generation. In _Proceedings of the AAAI Conference on Artificial Intelligence_,
volume 38, pp. 17754–17762, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared
Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large
language models trained on code. _arXiv preprint
Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo,
Songyang Zhang, Dahua Lin, Kai Chen, et al. T-eval: Evaluating the tool utilization capability
step by step. _arXiv preprint
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng,
Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna:
An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality, March 2023. URL
[https://lmsys.org/blog/2023-03-30-vicuna/.](https://lmsys.org/blog/2023-03-30-vicuna/)
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li,
Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language
models. _Journal of Machine Learning Research_, 25(70):1–53, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser,
Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve
math word problems. _arXiv preprint
[CodeFuse. CodeFuse codeexercise-python-27k, 2023. URL https://huggingface.co/](https://huggingface.co/datasets/codefuse-ai/CodeExercise-Python-27k)
[datasets/codefuse-ai/CodeExercise-Python-27k.](https://huggingface.co/datasets/codefuse-ai/CodeExercise-Python-27k)
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. _arXiv_
_preprint
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong
Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional
conversations. _arXiv preprint
Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang,
Zheng Yuan, Chang Zhou, and Jingren Zhou. How abilities in large language models are affected
by supervised fine-tuning data composition. _arXiv preprint
Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data
creation with 1,000,000,000 personas. _arXiv preprint
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas,
Guanyu Feng, Hanlin Zhao, et al. Chatglm: A family of large language models from glm-130b to
glm-4 all tools. _arXiv preprint
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen,
et al. Tora: A tool-integrated reasoning agent for mathematical problem solving. _arXiv preprint_
_
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao
Bi, Y Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming–the
rise of code intelligence. _arXiv preprint
Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual
learning in deep neural networks. _Trends in cognitive sciences_, 24(12):1028–1040, 2020.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and
Jacob Steinhardt. Measuring massive multitask language understanding. _arXiv preprint_
_
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song,
and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. _arXiv_
_preprint
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang,
and Weizhu Chen. Lora: Low-rank adaptation of large language models. _arXiv preprint_
_
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang,
Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models
with scalable training strategies. _arXiv preprint
Baichuan Inc. Baichuan-7B: A large-scale 7B pretraining language model developed by BaiChuan[Inc, 2023. URL https://github.com/baichuan-inc/Baichuan-7B.](https://github.com/baichuan-inc/Baichuan-7B)
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot,
Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.
Mistral 7b. _arXiv preprint
Suhas Kotha, Jacob Mitchell Springer, and Aditi Raghunathan. Understanding catastrophic forgetting
in language models via implicit inference. _arXiv preprint
Mario Michael Krell, Matej Kosec, Sergio P Perez, and Andrew Fitzgibbon. Efficient sequence
packing without cross-contamination: Accelerating large language models without impacting
performance. _arXiv preprint
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao
Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii,
Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João
Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee,
Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang,
Rudra Murthy V, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan
Dey, Zhihan Zhang, Nour Moustafa-Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh,
Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee,
Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank
Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish
Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis,
Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. StarCoder:
May the source be with you! _CoRR_, abs/2305.06161, 2023a. doi: 10.48550/arXiv.2305.06161.
[URL https://doi.org/10.48550/arXiv.2305.06161.](https://doi.org/10.48550/arXiv.2305.06161)
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee.
Textbooks are all you need ii: phi-1.5 technical report. _arXiv preprint
Bingchang Liu, Chaoyu Chen, Zi Gong, Cong Liao, Huan Wang, Zhichao Lei, Ming Liang, Dajun
Chen, Min Shen, Hailian Zhou, et al. Mftcoder: Boosting code llms with multitask fine-tuning. In
_Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_, pp.
5430–5441, 2024.
Keming Lu, Bowen Yu, Chang Zhou, and Jingren Zhou. Large language models are superpositions of
all characters: Attaining arbitrary role-play via self-alignment. _arXiv preprint
2024.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng,
Qingwei Lin, Shifeng Chen, and Dongmei Zhang. WizardMath: Empowering mathematical
reasoning for large language models via reinforced evol-instruct. _arXiv preprint
2023a.
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of
catastrophic forgetting in large language models during continual fine-tuning. _arXiv preprint_
_
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing
Ma, Qingwei Lin, and Daxin Jiang. WizardCoder: Empowering code large language models with
evol-instruct. _arXiv preprint
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of
large-batch training. _arXiv preprint
Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing
english math word problem solvers. _arXiv preprint
Yang Ming, Borovec Jirka, Bang Jihwan, Heon Cho Dong, Baig Wajih Ullah, Nitrome Zimo,
[Machnitzki Tobias, and Tae-hoon. Lee. imbalanced-dataset-sampler. https://github.com/](https://github.com/ufoym/imbalanced-dataset-sampler)
[ufoym/imbalanced-dataset-sampler, 2018.](https://github.com/ufoym/imbalanced-dataset-sampler)
[OpenAI. Introducing ChatGPT, 2022. URL https://openai.com/blog/chatgpt.](https://openai.com/blog/chatgpt)
OpenAI. GPT4 technical report. _arXiv preprint
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong
Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow
instructions with human feedback. _Advances in Neural Information Processing Systems_, 35:
27730–27744, 2022.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math
word problems? _arXiv preprint
Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model
connected with massive apis. _arXiv preprint
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru
Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li,
Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+
real-world apis, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language
models are unsupervised multitask learners. _OpenAI blog_, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi
Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text
transformer. _Journal of machine learning research_, 21(140):1–67, 2020.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In _Proceedings of_
_the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pp.
3505–3506, 2020.
Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi.
Progressive prompts: Continual learning for language models. _arXiv preprint
2023.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi
Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code Llama: Open foundation models for code.
_arXiv preprint
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine
Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables
zero-shot task generalization. _arXiv preprint
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung,
Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks
and whether chain-of-thought can solve them. _arXiv preprint
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. _arXiv preprint
2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak,
Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models
based on gemini research and technology. _arXiv preprint
Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. Openmathinstruct-1: A 1.8 million math instruction tuning dataset. _arXiv preprint_
_
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée
Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and
efficient foundation language models. _arXiv preprint
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay
Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian
Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin
Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar
Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann,
Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana
Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor
Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan
Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang,
Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen
Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic,
Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models.
_CoRR_ [, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/](https://doi.org/10.48550/arXiv.2307.09288)
[10.48550/arXiv.2307.09288.](https://doi.org/10.48550/arXiv.2307.09288)
Rui Wang, Fei Mi, Yi Chen, Boyang Xue, Hongru Wang, Qi Zhu, Kam-Fai Wong, and Ruifeng Xu.
Role prompting guided domain adaptation with general capability preserve for large language
models. _arXiv preprint
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du,
Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. _arXiv preprint_
_
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code is
all you need. _arXiv preprint
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi,
Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface’s transformers:
State-of-the-art natural language processing. _arXiv preprint
Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching
large language model to use tools via self-instruction. _Advances in Neural Information Processing_
_Systems_, 36, 2024.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok,
Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical
questions for large language models, 2023.
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin
Chen, Ruobing Xie, Yankai Lin, et al. Advancing llm reasoning generalists with preference trees.
_arXiv preprint
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen.
MAmmoTH: Building math generalist models through hybrid instruction tuning. _arXiv preprint_
_
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning:
Enabling generalized agent abilities for llms. _arXiv preprint
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small
language model. _arXiv preprint
Xiao Zhang and Ji Wu. Dissecting learning and forgetting in language model finetuning. In _The_
_Twelfth International Conference on Learning Representations_, 2024.
Yu Zhao, Yuanbin Qu, Konrad Staniszewski, Szymon Tworkowski, Wei Liu, Piotr Miło´s, Yuxiang
Wu, and Pasquale Minervini. Analysing the impact of sequence composition on language model
pre-training. _arXiv preprint
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang,
Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and
chatbot arena. _Advances in Neural Information Processing Systems_, 36:46595–46623, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang,
Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and
chatbot arena. _Advances in Neural Information Processing Systems_, 36, 2024a.
Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and
Xiang Yue. Opencodeinterpreter: Integrating code generation with execution and refinement. _arXiv_
_preprint
A APPENDIX
A.1 DETAILS OF EVALUATION DATASETS
**Mathematic reasoning.** GSM8K (Cobbe et al., 2021) presents diverse grade school math word
problems, while the MATH dataset (Hendrycks et al., 2021) focuses on evaluating mathematical
reasoning abilities with a collection of problems.In addition, we employed the SVAMP (Patel et al.,
2021) and ASDiv (Miao et al., 2021) datasets as supplementary test sets to assess mathematical
capabilities.
**Code generation.** HumanEval (Hendrycks et al., 2021) assesses code generation models’ problemsolving skills through Python programming challenges, and MBPP (Austin et al., 2021) targets
models’ execution-based performance on Python problems.
**Tool usage.** T-EVAL (Chen et al., 2023) breaks down the assessment of tool usage into several
sub-domains, shedding light on LLMs’ comprehensive and specific competencies.
**Common sense and logical reasoning.** MMLU (Hendrycks et al., 2020) covers 57 subjects from
STEM to humanities and social sciences, testing models on a wide range of difficulty levels and
world knowledge. We utilized the BBH-hard (Suzgun et al., 2022) dataset as an additional test set to
evaluate reasoning abilities.
**Multi-turn dialogue.** MT-Bench (Zheng et al., 2024a) features 80 questions across eight domains,
including writing, roleplay, reasoning, and more, with each domain offering a two-round queryresponse evaluation using GPT-4.
A.2 DETAILS OF EXPERIMENTS
**Qwen-SFT-code.** Trained on MagicCoder110k, CodeFeedback157k, and EvolInstructCode80k.
**Qwen-SFT-math.** Utilized MathInstruct262k and MetaMath395k datasets.
**Qwen-SFT-agent.** Fine-tuned with GPT4Tools75k, AgentInstruct, and ToolbenchG3.
**Qwen-SFT-chat.** Focused on ultrachat200k for dialogue interactions.
**Qwen-SFT-seq.** Combined datasets, following sequence instruction strategy.
**Qwen-SFT-packed.** Merged datasets with packing strategy, learning rate of 4e-5.
**Qwen-SFT-balanced.** Balanced sampling from all datasets, learning rate of 4e-5.
**Qwen-SFT-MoI.** Employed the MoI scheme using "You are a helpful assistant" as the default system
prompt, learning rate of 4e-5.
In all experiments, weight decay is set at a factor of 0.1, adam optimizer beta is set to 0.95, Warm-up
ratio is set at 0.01, the maximum length of the model is 8192 tokens, gradient checkpointing is
enabled.
A.3 MIXTURE OF INSTRUCTIONS DETAILS
During the training of our language model, we employed the classic ChatML format:
<|im_start|>system
You are a helpful assistant.
<|im_end|>
<|im_start|>user
How are you?
<|im_end|>
<|im_start|>assistant
As I am a large language model, I do not have personal emotions.
However, I am functioning properly and ready to assist you with any
information or tasks you need help with. How may I be of service today?
<|im_end|>
Figure 7: The function of the system prompt and the mapping of knowledge learned from all system
prompts onto the default system prompt following the application of the Mixture of Instructions
method.
Table 10: Detailed scores of SFT models on T-EVAL. We report zero-shot performance of the models.
**Model** Instruct Plan Reason Retrieve Understand Review overall
Qwen-7B-chat 67.6 56.3 46.9 66.1 66.5 60.6 60.7
Qwen-SFT-code 0.8 0.9 35.7 29.8 52.4 14.0 22.3
Qwen-SFT-math 0.3 9.2 21.5 11.2 30.3 11.7 14.0
Qwen-SFT-agent 44.1 22.8 37.8 58.7 63.7 55.2 47.1
Qwen-SFT-chat 64.0 61.1 51.3 63.4 74.4 50.3 60.8
Qwen-SFT-seq **83.7** **65.8** **54.4** **77.1** **78.4** **58.3** **69.6**
Qwen-SFT-packed 83.6 62.9 51.2 72.7 75.6 56.3 67.1
Qwen-SFT-balanced 82.6 64.3 51.0 73.0 75.9 54.6 66.9
Qwen-SFT-MoI 78.1 65.3 53.9 76.8 77.8 57.5 68.2
In the example above, the portion labeled with ‘system’ is what we have been discussing as the
‘system prompt’ in our paper.
Figure 7 illustrates the effects of the Mixture of Instructions (MoI) framework. In the initial phase,
models subjected to Supervised Fine-Tuning (SFT) with the default system prompt strive to replicate
the canonical implementation of the Boyer-Moore algorithm. However, as the process advances,
they divert their approach towards devising an algorithm based on iterative querying. Subsequent
to retraining with a modified system prompt, the model acquires the capability to accurately script
the Boyer-Moore algorithm, demonstrating a clear understanding of its logic and structure. Utilizing
the Mixture of Instructions (MoI) approach for further training enables the seamless transference
of knowledge acquired in the _‘You are a programmer’_ domain to applications within the _‘You are a_
_helpful assistant’_ scenario, showcasing the versatility and adaptability of the learned skills across
different contexts.
A.4 MIXTURE OF INSTRUCTIONS ATTENTION MAP VISUALIZATION
Figure 3 shows the configuration of the attention map for MoI model, we further visualize the attention
maps from each layer within our MoI model during inference in Figure 9, where the attention of each
layer is obtained by averaging across all heads’ attention in that layers.
Figure 8: Detailed scoring charts on MT-Bench for Qwen-7B-chat, Qwen-SFT-code, Qwen-SFT-math,
and Qwen-SFT-MoI models.
A.5 MT-BENCH RESULTS
Figure 8 shows that employing the MoI technology in language models concurrently boosts mathematical reasoning and coding skills, paralleling the proficiency achieved by models trained solely on
individual tasks. This consistency underscores the robustness of our proposed approach.
A.6 T-EVAL RESULTS
Table 10 indicates that the agent capabilities within language models exhibit a direct association
with their chat functionalities. Subjecting the models to SFT using datasets composed exclusively of
code, math, or agent-specific content can significantly impair their agent abilities. The Sequence of
Instructions approach outperforms other methodologies in agent-related tasks, which we surmise is
due to its compliance with the input format used during the initial SFT, thus maximally preserving
the consistency of the language model. Furthermore, our MoI technique secures the highest scores in
techniques involving instruction packing, suggesting that our method also aims to retain the original
competencies of the language model to the greatest possible extent.
A.7 CASE STUDY
**[Turn #1] Benjamin went to a bookstore and purchased a variety of books. He bought 5 copies**
**of a sci-fi novel, each priced at $20, 3 copies of a history book priced at $30 each, and 2 copies**

Qwen Params HumanEval pass@1 GSM8K Accuracy(%)


Benjamin bought 5 copies of a sci-fi novel, each priced at $20, so the total cost of the sci-fi novels is
5 * $20 = $100. He also bought 3 copies of a history book, each priced at $30, so the total cost of the
history books is 3 * $30 = $90. Lastly, he bought 2 copies of a philosophy book, each priced at $45,
so the total cost of the philosophy books is 2 * $45 = $90. The total cost of his purchases is $100 +
$90 + $90 = $280.

of a philosophy book for $45 each. What was the total cost of his purchases?


The answer is 280
**[Turn #2] Suppose Benjamin decides to sell each of these books at a 25% markup from the**
**price he purchased them. What would be his total revenue if he sold all the books he bought?**
Benjamin bought 5 copies of a sci-fi novel, each priced at $20, so the total cost of the sci-fi novels is
5 * $20 = $100. He also bought 3 copies of a history book, each priced at $30, so the total cost of the
history books is 3 * $30 = $90. Lastly, he bought 2 copies of a philosophy book, each priced at $45,
so the total cost of the philosophy books is 2 * $45 = $90. The total cost of his purchases is $100 +
$90 + $90 = $280. If Benjamin decides to sell each of these books at a 25% markup from the price
he purchased them, the markup would be 25% of $280, which is 0.25 * $280 = $70. So, the total
revenue Benjamin would get if he sold all the books he bought is $280 + $70 = $350.


The answer is 350
**[Turn #3]After achieving his profit goal by selling books at a markup, Benjamin decides to**
**donate a portion of his total revenue to a local library. If he chooses to donate 15% of his total**


Benjamin’s total revenue is $350. If he chooses to donate 15% of his total revenue, the amount he
will donate is 15/100 * $350 = $52.50. So, the local library will receive $52.50.

revenue, how much money will the local library receive?


The answer is 52.5
**[Turn #4]If Benjamin aims to achieve a profit of $100 over the initial cost of his purchases by**
**selling all the books, what percentage markup should he apply to each book’s selling price?**
Benjamin aims to achieve a profit of $100 over the initial cost of his purchases, which is $280. To
calculate the markup percentage, we need to find the difference between the selling price and the cost
price, and then divide that by the cost price. The selling price is the cost price plus the markup, so the
selling price is $280 + $100 = $380. The markup is the difference between the selling price and the
cost price, which is $380 - $280 = $100. To find the markup percentage, we divide the markup by the
cost price and multiply by 100. The markup percentage is ($100 / $280) * 100 = 35.71%.


The answer is 35.71
Table 11: Our MoI model demonstrated performance on the MT-Bench math, and we proceeded to
inquire about the third and fourth rounds.
**[Turn #1]Write a simple website in HTML. When a user clicks the button, it shows a random**
**joke from a list of 4 jokes.**
Sure, here is a simple HTML code that does what you asked for:
<!DOCTYPE html>

<title>Joke Generator</title>

Joke Generator>/h1>
<button onclick="displayJoke()">Generate Joke</button>

This code creates a simple webpage with a button that, when clicked, generates a random joke from
the list of jokes and displays it in a paragraph element.


You can use CSS to change the color of the jokes. Here is how you can do it:
<!DOCTYPE html>

<title>Random Joke Generator</title>

Random Joke Generator

<button onclick="displayJoke()">Generate Joke</button>

In this code, the CSS rule p { color: red; } is used to change the color of all paragraph elements to
red.
**[TURN #3]How to dynamically retrieve a list of jokes from an external file or API**
You can dynamically retrieve a list of jokes from an external file or API using JavaScript’s fetch
function. Here is an example:
<!DOCTYPE html>

<title>Random Joke Generator</title>

Random Joke Generator

<button onclick="displayJoke()">Generate Joke</button>

Table 12: Our MoI model demonstrated performance on the MT-Bench code, and we proceeded to
inquire about the third round.

[Turn #2]How to use CSS to change the color of jokes to red?


You have access to the following API: [
[
’AirbnbSearch.get_property_details’, {
’name’: ’AirbnbSearch.get_property_details’,
’description’: ’This function retrieves the details of a property given its ID.’,
’required_parameters’: [
{’name’: ’propertyId’,’type’: ’NUMBER’, ’description’: ’The ID of the property.’}],
’optional_parameters’: [],
’return_data’: [
{ ’name’: ’title’, ’description’: ’The title of the property.’},
{’name’: ’propertyType’, ’description’: ’The type of the property.’},
{’name’: ’personCapacity’, ’description’: ’The capacity of the property.’},
{’name’: ’rating’, ’description’: ’The rating of the property.’},
{’name’: ’latitude’, ’description’: ’The latitude of the property.’},
{’name’: ’longitude’, ’description’: ’The longitude of the property.’}]}
],
[
’AirbnbSearch.get_property_reviews’, {
’name’: ’AirbnbSearch.get_property_reviews’,
’description’: ’This function retrieves the reviews of a property given its ID.’,
’required_parameters’: [
{’name’: ’propertyId’, ’type’: ’NUMBER’, ’description’: ’The ID of the property.’}],
’optional_parameters’: [],
’return_data’: [
{’name’: ’reviews’, ’description’: ’The reviews of the property, containing comment, rating,
and date. At most 3 reviews are returned.’}]}
],
[
’AirbnbSearch.check_availability’,{
’name’: ’AirbnbSearch.check_availability’,
’description’: ’This function checks the availability of a property given its ID.’,
’required_parameters’: [
{’name’: ’propertyId’, ’type’: ’NUMBER’, ’description’: ’The ID of the property.’}],
’optional_parameters’: [
{’name’: ’date’, ’type’: ’STRING’, ’description’: ’The date to check, it must in the format
of YYYY-MM-DD.’}],
’return_data’: [
{’name’: ’availability’, ’description’: ’The availability of the property for the given date or
the next 10 days.’}]}
],
[
’AirbnbSearch.search_property_by_place’, {
’name’: ’AirbnbSearch.search_property_by_place’,
’description’: ’This function takes various parameters to search properties on Airbnb.’,
’required_parameters’: [
{’name’: ’place’, ’type’: ’STRING’, ’description’: ’The name of the destination.’}],
’optional_parameters’: [],
’return_data’: [
{’name’: ’property’, ’description’: ’a list of at most 3 properties, containing id, name, and
address.’}]}
]
]
Please generate a plan for answer user’s questions, which should be a list of actions with the following
format:
“‘
[
{
// id of the action
"id": number;
// the name of the action
"name": string;
// input params required by this action
"args": "str(Record<string, any>)";
},
...
]
“‘
You can imagine args when you plan the action, and these instructions will be executed sequentially.For example, if you want to call ‘api1‘ with ‘arg1‘ and ‘arg2‘, you can write the following
plan:
“‘
[
{
"id": 0,
"name": "api1",
"args": "{’arg1’: ’value1’, ’arg2’: ’value2’, ...}",
},
...
]
“‘
The args should be a dictionary in string format. PLEASE use ” in args dictionary and use "" in other
places, DO NOT print args with value None or null.You should only generate a list in json format.
The list should be the full planning list without ‘...‘. DO NOT generate any text to explain the json.


I’m helping a client find the first available rental property in Los Angeles. Please find a property in
Los Angeles. Then, check the availability of the property for the coming week. Next, get the details
of that property. Finally, retrieve the reviews of the property.


[
{
"id": 0,
"name": "AirbnbSearch.search_property_by_place",
"args": "{’place’: ’Los Angeles’}"
},
{
"id": 1,
"name": "AirbnbSearch.check_availability",
"args": "{’propertyId’:<property_id>, ’date’: ’2022-01-01’}"
},
{
"id": 2,
"name": "AirbnbSearch.get_property_details",
"args": "{’propertyId’: <property_id>}"
},
{
"id": 3,
"name": "AirbnbSearch.get_property_reviews",
"args": "{’propertyId’: <property_id>}"
}
]
Table 13: Our MoI model demonstrated performance on the T-EVAL plan.
A.8 SOLVE MATH PROBLEM WITH CODE INTERPRETER
In this section, we demonstrate that models trained using the Mixture of Instructions (MoI) approach
can perform functions akin to the code interpreter capabilities of GPT-4. This case necessitates a
comprehensive evaluation of the model’s mathematical reasoning, code writing, and tool utilisation
proficiencies.
We feed prompts into the Qwen-SFT-MoI system in the following format, adhering to the ReACT
standard during tool use—prompting the model to comply with a Thought, Action, Action Input
sequence to generate the tool’s output:
You are a helpful assistant.
Answer the following questions as best you can. You have access to the
following APIs:
code_interpreter: available for executing Python code. The drive at
’workspace/’ can be used to save and persist user files. Internet access
for this session is disabled. Do not make external web requests or API
calls as they will fail.
Parameters:
{
"type": "object",
"properties": {
"code": {"type": "string", "description": "executable code"}
}
}
Use the following format:
Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [code_interpreter]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can be
repeated zero or more times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question
Begin!
This interaction informs the model to employ a code interpreter tool to address mathematical queries.
A model proficient in following instructions should, upon encountering the user’s question, compose
code that meets the formatting requirements. For this functionality, we have developed an entire
pipeline. It filters and extracts the relevant substance from the model’s code output, then passes it
into a Python code interpreter to obtain execution results, which are subsequently compared with the
standard outcomes.
Questions:
Henry and 3 of his friends order 7 pizzas for lunch. Each pizza is
cut into 8 slices. If Henry and his friends want to share the pizzas
equally, how many slices can each of them have?
Answer:
Thought:Let’s solve this problem using Python code.
Action:code_interpreter
Action Input: {"code":"number_of_pizzas = 7
slices_per_pizza = 8
number_of_friends = 3
number_of_slices_per_person = number_of_pizzas * slices_per_pizza /
(number_of_friends + 1)
number_of_slices_per_person
"}
Observation: 14.0
Thought: I now know the final answer.
Final Answer: Thus each of them can have \boxed{14} slices.
Our model’s efficacy is validated against the GSM8K benchmark, where our Qwen-SFT-MoI model
achieves an accuracy of 69.14% under a zero-shot setting. Contrastingly, when the model infers
separately on the problems, the accuracy reaches 72%. This case illustrates that our model not only
possesses commendable mathematical reasoning, code writing, and tool utilisation capacities but can
also synergise these abilities to resolve problems encountered in practical processes.
A.9 THE EFFECTIVENESS OF THE MOI METHOD IN RAG SCENARIOS
Table 14: Detailed scores of SFT models on RGB English Benchmark for Retrieval-Augmented
|Generation (Chen et al., 2024).|Col2|Col3|Col4|Col5|Col6|
|---|---|---|---|---|---|
|Model|Noise Ratio
0|Noise Ratio
0.2|Noise Ratio
0.4|Noise Ratio
0.6|Noise Ratio
0.8|
|ChatGPT(OpenAI, 2022)
ChatGLM2-6B(GLM et al., 2024)
Vicuna-7B-v1.3(Zheng et al., 2023)|96.33
91.33
87.67|94.67
89.67
83.33|94.00
83.00
86.00|90.00
77.33
82.33|76.00
57.33
60.33|
|Qwen-7B-Chat(Bai et al., 2023)
Qwen-SFT-MoI (ours)|94.33
93.33|91.67
90.00|91.00
89.67|87.67
86.33|73.67
71.67|
Table 15: Detailed scores of SFT models on L-EVAL(An et al., 2023).
|Model|Coursera|GSM|QuALITY|TOEFL|CodeU|SFiction|Avg|
|---|---|---|---|---|---|---|---|
|ChatGPT(OpenAI, 2022)
Llama2-7b-chat(Touvron et al., 2023b)|63.51
29.21|84.00
19.00|61.38
37.62|78.43
51.67|12.22
1.11|64.84
60.15|60.73
33.12|
|Qwen-7B-Chat(Bai et al., 2023)
Qwen-SFT-MoI (ours)|45.64
47.24|29.0
68.0|59.40
50.99|76.20
66.17|5.55
4.44|60.93
55.93|46.12
48.80|
A.10 THE EFFECTIVENESS OF THE MOI METHOD ON QWEN 72B WITH LORA
Table 16: Detailed scores of LoRA(Hu et al., 2021) SFT on Qwen1.5-72B-Chat model. The specific
LoRA parameters: r = 32, alpha = 64, learning rate=0.001, . We report zero-shot performance of the
|models.|Col2|Col3|Col4|Col5|Col6|Col7|
|---|---|---|---|---|---|---|
|Model|MMLU|GSM8K|MATH|HumanEval|MBPP|Avg|
|Qwen1.5-72B-Chat(Bai et al., 2023)
Qwen1.5-72B-SFT-MoI-LoRA(ours)|77.5
77.9|82.7
83.0|42.5
43.1|71.3
72.8|71.9
72.2|69.18
69.80|
A.11 CASE STUDY BETWEEN QWEN-7B-CHAT AND QWEN-SFT-MOI
In the two cases above, we set a system prompt for the models. In case 1, we see that both models can
respond to questions based on the settings in the system prompt. However, in case 2, we first set the
date to Friday, May 17th, and then told the model "You are a helpful assistant." We can observe that
the Qwen-7B-Chat model seems to "miss" the time setting, whereas the MoI model can capture this
information. This suggests that the MoI model’s attention mechanism is more robust to the sequence
of input instructions. In contrast,


**Wenhan Xiong** [3], **Jimmy Lin** [1], **Wen-tau Yih** [3], and **Xilun Chen** [3] _[†]_
University of Waterloo [1], Carnegie Mellon University [2], Meta AI [3]
s269lin@uwaterloo.ca, xilun@meta.com

FLAME: Factuality-Aware Alignment for Large Language Models

Sheng-Chieh Lin [1] _[∗]_ , Luyu Gao [2] , Barlas Oguz [3] ,

Sheng-Chieh Lin
Luyu Gao
Barlas Oguz
Wenhan Xiong
Jimmy Lin
Wen-tau Yih
Xilun Chen


Alignment is a standard procedure to fine-tune
pre-trained large language models (LLMs) to
follow natural language instructions and serve
as helpful AI assistants. We have observed,
however, that the conventional alignment process fails to enhance the factual accuracy of
LLMs, and often leads to the generation of
more false facts (i.e. _hallucination_ ). In this
paper, we study how to make the LLM alignment process more factual, by first identifying
factors that lead to hallucination in both alignment steps: supervised fine-tuning (SFT) and
reinforcement learning (RL). In particular, we
find that training the LLM on new knowledge
or unfamiliar texts can encourage hallucination. This makes SFT less factual as it trains
on human labeled data that may be novel to
the LLM. Furthermore, reward functions used
in standard RL can also encourage hallucination, because it guides the LLM to provide
more helpful responses on a diverse set of instructions, often preferring longer and more
detailed responses. Based on these observations, we propose _**f**_ _actua_ _**l**_ _ity-aware_ _**a**_ _lign_ _**me**_ _nt_
(FLAME ), comprised of _factuality-aware SFT_
and _factuality-aware RL_ through direct preference optimization. Experiments show that
our proposed factuality-aware alignment guides
LLMs to output more factual responses while
maintaining instruction-following capability.


Alignment is a standard procedure to make pretrained large language models (LLMs) (Brown
et al., 2020; Touvron et al., 2023) follow natural
language instructions and serve as helpful AI assistants. Despite significant progress in instruction
tuning (Wang et al., 2023a; Zhou et al., 2023; Li
et al., 2024) and LLM alignment (Ouyang et al.,
2022; Bai et al., 2022; Yuan et al., 2024), stateof-the-art LLMs are still prone to generate false
_∗_ This work is done during Sheng-Chieh’s internship at Meta.
_†_ Xilun and Sheng-Chieh contributed equally to this work.
|SF|T
length: 1.0k|+DPO|Col4|Col5|
|---|---|---|---|---|
|length: 1.2|k|+DPO|
length: 1.4k||
|SFT||+DPO
|length: 1.5k
length: 1.4k||
|||||length: 1.6k|
20 30 40 50 60 70


Figure 1: Models’ helpfulness on Alpaca Eval vs factuality on biography. Helpfulness is measured by models’
win rate over our baseline SFT + DPO on Alpaca Eval.
Dot size represents averaged length of bio generation.
claims (Min et al., 2023). This motivates us to
study the underlying causes of LLM hallucination
as well as its relation to the alignment procedure.
The standard alignment process consists of
two training phases: (1) supervised fine-tuning
(SFT) (Sanh et al., 2022); (2) reinforcement learning (RL) with human (RLHF, Ouyang et al., 2022;
Bai et al., 2022) or automated feedback (RLAIF,
Bai et al., 2023). In our study, we find that both
the SFT and the RL steps in the standard alignment
process may actually _encourage_ LLMs to hallucinate. First, in the SFT stage, LLMs are fine-tuned
with diverse instructions paired with human created high-quality responses. While this leads to
strong instruction following capability (Zhou et al.,
2023), our study shows that such human labeled
responses may present _new or unknown informa-_
_tion_ to the LLM. This, in turn, may inadvertently
promote hallucination. Second, we find that the
standard reward used in the RL stage often prefers
longer and more detailed responses (Singhal et al.,
2023; Yuan et al., 2024), which tends to stimulate
the LLM to yield more false claims, as shown in
the black dots in Figure 1. One possible reason is
that most existing RLHF or RLAIF approaches rely
on a single scalar reward to represent preference,
which struggles to cover multiple alignment skill
sets (Ye et al., 2024) and is likely to under-present
the aspect of factuality (Hosking et al., 2024).
To address the aforementioned issues, we study
the key factors which impact factuality during
alignment. In particular, we first conduct a pilot study on the biography generation task (Min
et al., 2023) in a more controlled setting where
the alignment process focuses solely on factuality
(Section 3). Our pilot study reveals that a LLM
hallucinates more if it is fine-tuned on new knowledge in either the SFT or the RL stage. For example, a LLM becomes significantly less factual
when fine-tuned on responses produced by a model
with access to external knowledge (e.g. a retrievalaugmented LLM), even though those responses
are more factual themselves. Similarly, hallucination is greatly increased if RLAIF is performed on
preference pairs that consist of retrieval-augmented
LLM output as positive examples and the LLM’s
own output as negative examples. As a result, we
discover that fine-tuning a pre-trained LLM on (a
selected subset of) its own generations yields more
factual responses and reduces hallucinations.
Our ultimate goal is to improve the factuality of
the standard alignment process, which is challenging since LLMs may be given diverse and complex
instructions. As shown in Figure 2, we observe that
some instructions require factual responses while
the others do not. Motivated by the observation,
we propose factuality-aware alignment. We first
identify fact-based instructions that require factual
responses. For fact-based instructions, we leverage the findings in our pilot study to create additional training data at both SFT and RL stages to
explicitly guide LLMs to output factual responses.
Specifically, at the SFT stage, for fact-based instructions, instead of using human created seed training
data, we construct few-shot demonstrations (from
the same seed data) and generate training data using the pre-trained LLM’s own knowledge. This
can prevent fine-tuning the LLM on knowledge
unknown to itself. At the RL stage, we create additional preference pairs focused on factuality for
fact-based instructions, which are combined with
the standard preference pairs for instruction following during Direct Preference Optimization (DPO;
Rafailov et al., 2023).
We evaluate models on Alpaca Eval (Dubois
et al., 2024) and Biography, using win rate for instruction following capability and FActScore (Min
et al., 2023) for factuality evaluation, respectively.
As shown in Figure 1, using our FLAME method
(SFT + DPO ), a significantly higher FActScore
(+5.6 pts) is achieved compared to the standard
alignment process (SFT + DPO), without sacrificing the LLM’s instruction following capability
(51.2% win rate). Our ablation study also indicates
that identifying fact-based instructions is the key to
factual alignment in the general alignment setting.


**Alignment.** Since pre-trained LLMs cannot accurately follow human instructions, a bunch of
work has been proposed to improve LLM alignment through SFT and RL. Some propose to improve SFT through data curation (Zhou et al., 2023;
Chen et al., 2024), diverse instruction augmentation (Wang et al., 2023a; Li et al., 2024) while
others focus on RL with human feedback (Ouyang
et al., 2022; Bai et al., 2022), AI feedback (Bai
et al., 2023; Sun et al., 2024; Yuan et al., 2024). The
main goal of these alignment approaches is instruction following capability (or helpfulness), which
may guide LLMs to output detailed and lengthy
responses (Singhal et al., 2023) but inevitably encourage hallucination.
**Factuality.** Prior work has highlighted the issue
of hallucination in LLMs (Kandpal et al., 2023;
Mallen et al., 2023). To address the issue, important
research lines are factuality evaluation (Min et al.,
2023; Wang et al., 2023b; Chern et al., 2023) and
improvement. Some training-free approaches to
improve LLMs’ factuality include external knowledge augmentation (Kandpal et al., 2023; Cheng
et al., 2023; Jiang et al., 2023) and specialized decoding (Li et al., 2023; Chuang et al., 2024).
Recent studies apply RL to improve LLMs’ factuality. For example, Tian et al. (2024) propose to
construct factuality preference pairs for direct preference optimization (DPO; Rafailov et al., 2023),
which is closely related to our work. However,
they focus solely on enhancing LLMs’ factuality through DPO but overlook its potential impact
on the models’ instruction-following capability, as
demonstrated in our experiments. In contrast, our
work provides a comprehensive examination of improving LLMs’ factuality and instruction-following
ability through fine-tuning approaches encompassing both SFT and DPO. Concurrent to our work,
Kang et al. (2024) find that LLMs tend to hallucinate when facing unfamiliar queries. They consider
improving LLMs’ factuality as teaching LLMs to
output abstaining or less detailed responses on such
unfamiliar queries, a similar behavior observed
from our LLMs fine-tuned with FLAME (see case
studies in Section 5.5). It is worth mentioning that
both prior studies focus on a simplified scenario as
our pilot study in Section 3: fine-tuning LLMs to
improve factuality on a single task (e.g., fine-tuning
and evaluating on biography generation). In contrast, we consider the general alignment task, where
LLMs are given diverse and complex instructions.


In this section, we first study how to align large
language models (LLMs) to be more factual. We
use biography generation as the task of our pilot
study for two main reasons: (1) Biography generation is a simplified setting where factuality is the
sole focus of the alignment process. As we will
discuss in Section 4, studying factual alignment
on diverse human instructions is more complex,
as the alignment process encompasses aspects beyond factuality, such as helpfulness and safety. (2)
Evaluating the factuality of biography generation
is relatively easy since Wikipedia covers sufficient
information for public figures and most the facts
about a person is non-debatable (Min et al., 2023).


A standard alignment procedure consists of supervised fine-tuning (SFT) and reinforcement learning (RL). In this pilot study, our main goal is to
teach LLMs to generate biography with reduced
misinformation. For the experiment, we compile
training and evaluation datasets comprising 500
and 183 diverse human entities, respectively (further details provided in Appendix A.1). We employ FActScore (FS; Min et al., 2023) as the automated metric for assessing factuality, given its
fine-grained evaluation capabilities for long-form
text generation and its strong correlation with human judgments. [1] To study factuality alignment
in this pilot study, we posit that training data is
needed where the responses are more factual than
the LLM’s own generations. Thus, we use retrievalaugmented LLMs (RAG; Lewis et al., 2020) to
generate training data, which has been shown to
output more factual responses (Mialon et al., 2023).
Throughout the paper, we refer to the pre-trained
(PT), supervised fine-tuned (SFT), and direct pref
1We use the evaluator: retrieval+llama+npm
Table 1: Pilot study on biography generation. Pos. denotes the positives for SFT or DPO. Neg. denotes the
negatives for DPO. FS denotes FActScore.
|Llama-2 7B|src. of supervision
Pos. Neg.|Bio
FS # Corr. / Err.|
|---|---|---|
|(1) PT
(2) PTRAG|-
-
-
-|39.1
14.4 / 22.0
55.4
18.6 / 15.9|
|(3) SFT
(4)|PT
-
PTRAG
-

|37.9
13.4 / 21.8
35.7
13.5 / 23.7|
|(5) DPO
(6)|PT~~_∗_~~
PT~~_∗_~~
PTRAG
PT|41.6
15.4 / 20.7
23.5
12.7 / 34.9|
_∗_ FActScore is used to select positives and negatives.
erence optimization (DPO) fine-tuned LLMs as
PT, SFT, and DPO, respectively. [2]
**SFT.** We explore two sources of supervision to
generate training data (detailed in Appendix A.1):
(1) using PT [RAG] with few-shot demonstration to
generate biographies for each name entity in training data, where PT [RAG] is PT augmented with an
off-the-shelf retriever (Lin et al., 2023); (2) using
vanilla PT with few-shot demonstration to generate training data as a baseline. As shown in Table 1,
PT [RAG] is indeed much more factual than PT. However, a surprising discovery in the pilot study is
that _fine-tuning on such more factual instruction–_
_biography pairs generated by_ PT _[RAG]_ _results in a_
_less factual_ SFT _model_ (row 4 vs 3).
**DPO.** We further fine-tune the LLMs to be more
factual through DPO. An intuitive way to create
factuality preference pairs is to directly use the
samples from PT [RAG] and PT as positives and negatives since PT [RAG] generates more factual biographies than PT (row 2 vs 1). Another approach is
to employ FActScore (FS) as the reward to select
positive and negative samples among the generations from PT itself (Tian et al., 2024) (detailed
in Apppendix A.1). As shown in Table 1, DPO
fine-tuned on self-generated data with FS reward
guides models to generate more factual responses
(row 5 vs 3); however, DPO fine-tuned with the
supervision of PT [RAG] makes the models hallucinate
even more than its SFT counterpart (6 vs 4).
This outcome suggests that compelling models to generate responses akin to PT [RAG] prompts
increases hallucination. Conversely, fine-tuning
LLMs on their own generations appears to be crucial for factual alignment, a finding applicable to
both SFT and DPO fine-tuning.
2Note that in our experiments, we use DPO as the substitute
of RL (Schulman et al., 2017).
To tackle the aforementioned challenges, we propose _**f**_ _actua_ _**l**_ _ity-aware_ _**a**_ _lign_ _**me**_ _nt_ (FLAME ). To
address the first challenge, we propose to prompt
LLMs to classify whether a given instruction demands the response to be factual, as shown in Figure 2. We then apply the factuality fine-tuning
strategy for SFT and DPO discussed in Section 3.2
to those fact-based instructions. Furthermore, to
address the second challenge, we employ separate
rewards to evaluate the factuality and instruction
following capability of a LLM. For simplicity, our
work only considers two alignment skill sets: instruction following and factuality. We leave more
comprehensive reward modeling to future work.
In the following, we first describe our baseline
alignment approach and introduce our proposed
factuality-aware alignment built on top of the baseline alignment procedure.


We initialize PT from Llama-2 70B pre-trained
model [3] and build our baseline alignment procedure
following self-rewarding language models (Yuan
et al., 2024) due to its simplicity and independence
of other strong LLMs (e.g., GPT4) or human evaluators as a reward model. The alignment comprises
two steps: (1) building SFT model fine-tuned on a
high-quality seed data consisting of 3,200 instructions and each instruction is paired with the best
response created by humans from Open Assistant
dataset (OASST; Köpf et al., 2023); (2) further finetuning SFT through DPO on instruction following
preference data ( _x, y_ + _, y−_ ) constructed by itself
(SFT) as the reward model, RM [IF], where _y_ + and
_y−_ are the positive and negative responses for a
given prompt _x_, respectively. The resulting finetuned model is denoted as SFT + DPO. Note that,
following Yuan et al. (2024), we use additional augmented 20K instructions to create the preference
training data for DPO fine-tuning. Further details
are provided in Appendix A.3.


Although leveraging human created high-quality
seed data is a reasonable choice for SFT (Zhou
et al., 2023), our study in Section 3 suggests that
fine-tuning on such high-quality data generated by
models other than the LLM itself may present unknown information to the LLM, which may in turn
[3meta-llama/Llama-2-70b](https://huggingface.co/meta-llama/Llama-2-70b)
|( ∈ Xfact)
Fact-based x|Col2|fact-based( ∉ Xfact)
Non x|Col4|
|---|---|---|---|
|(1)|Do you have any information
about the Commodore 64?|(6)|How would a child feel if it fell
down on the ground hitting its|
|(2)|Hi, could you help me to solve
this cubic equation using
Cardano's Mehod (step by step if
possible), please? -> " x^3 + 2x^2
- x - 1 = 0 "|(7)|Write a fun story that can be told
in 3 minutes at the dinner table.
We are 3 developers enjoying a
pizza. The story must contain
these word: zombie, ethernet|
|(3)|Please give me a brief history of
coffee.|(8)|Tell me a story about a pig who
goes to the moon.|
|(4)|What are the principles at play in
UHPLC-MS analysis?|(9)|Is the internet's focus on
engagement the root of most of its
problems and shortcomings?|
|(5)|Explain the significance of the
American Revolution, including
the events that led up to it, the
impact it had on the world, and its
ongoing relevance today.|(10)|Can you tell me a bit about what
has gone into your creation?|
Figure 2: Instructions from Open Assistant dataset. The
instructions are classified with SFT model using the
prompt in Appendix, Figure 5.


From the pilot study, we find that better quality data
(in terms of factuality) for SFT and DPO does not
necessarily yield models with better factual alignment. This is likely because the supervision from
RAG contains information unknown to the LLM;
thus, fine-tuning on RAG generated responses may
inadvertently encourage the LLM to output unfamiliar information. To avoid unknown knowledge
from being presented to the LLM, a viable strategy
is to create SFT and DPO training data using the
generated responses from the LLM itself.


In the section, we further extend our discussion
of factual alignment to encompass more general
instructions. Unlike biography generation in Section 3, where factuality is the main alignment objective, human instructions are diverse and complex,
necessitating a range of alignment skill sets beyond
factuality alone; e.g., logical thinking, problem handling and user alignment (Ye et al., 2024). Thus,
conducting factual alignment with the diverse instructions face two main challenges: (1) different
instructions may demand distinct skill sets. For
example, in Figure 2, instruction 3, “Please give
me a brief history of coffee”, necessitates factual
accuracy and concise summarization, while instruction 8, “Tell me a story about a pig who goes to
the moon”, prioritizes creativity and imagination
over strict factuality. (2) As recent studies have
emphasized (Ye et al., 2024; Hosking et al., 2024),
using a single scalar for reward modeling fails to
adequately address multiple alignment skill sets
and often under-presents the aspect of factuality.
|Coffee is believed to
ha1v6eth o criegnintuartye:d C ino ffee
Ethiopia,...|Col2|Col3|Col4|
|---|---|---|---|
|Coffee is believed to
have originated in
Ethiopia,...
16th century: Coffee

|ave originated in
thiopia,...
16th century: Coffee

|ave originated in
thiopia,...
16th century: Coffee

|
|
|Coffee is believed to
have originated in
Ethiopia,...
16th century: Coffee

|
is
E|
 first introduced to
urope by …
1. Coffee originated
in Ethiopia in the 9t|
 first introduced to
urope by …
1. Coffee originated
in Ethiopia in the 9t|
Figure 3: Illustration of response generation using a
pre-trained LLM (PT) with few-shot demonstration.
sue, for each instruction from the seed data, we
elicit the knowledge from the pre-trained LM itself
by generating the responses with few-shot demonstration. Furthermore, to better use the knowledge
from both humans and the pre-trained LLM itself,
we propose to utilize human generated responses
for non-fact-based instructions, while leveraging
the responses sampled from pre-trained LLMs for
fact-based instructions to mitigate the introduction
of unknown knowledge.
Specifically, we create factuality-aware alignment training data for SFT with two steps. (1)
Classifying instructions: we first prompt SFT to
judge whether an instruction from the seed data is
fact-based ( _x ∈_ _X_ [fact] ) or not. [4] (2) Eliciting knowledge from PT: as illustrated in Figure 3, we sample
10 responses from PT with 5-shot demonstration,
( _x_ 0 _,_ Human( _x_ 0)) _· · ·_ ( _x_ 4 _,_ Human( _x_ 4)), where _xk_
is the top- _k_ similar instruction to _x_ retrieved by
DRAGON+ (Lin et al., 2023) from the seed data.
Human( _xk_ ) denotes the corresponding human response to _xk_ in the seed data.
As illustrated in Figure 4(a), the resulting training data for SFT is ( _x /∈_ _X_ [fact] _,_ Human( _x_ )) _,_ ( _x ∈_
_X_ [fact] _,_ PT( _x_ )), where PT( _x_ ) denotes the set of responses to _x_ sampled from PT. The resulting finetuned model is denoted as SFT .


At the second stage of alignment with DPO, we use
SFT to generate multiple responses _y_ 0 _, y_ 1 _, · · ·_
for a given instruction _x_ ; then, using SFT itself
as the reward model (RM [IF] ) to create a preference
pair: ( _x, y_ + _, y−_ ). [5] The above data creation procedure is the same as the second stage of our baseline
4Prompt for fact-based instruction classification is shown in
Appendix, Figure 5.
5We sample 4 responses for each augmented instruction.
Figure 4: Illustration of factuality-aware alignment.
alignment in Section 4.1. However, recent studies (Saha et al., 2023; Hosking et al., 2024; Ye
et al., 2024) indicate that a single scalar reward
from human feedback or LLM reward models may
under-represents the aspect of factuality. To address this limitation, we introduce another factuality reward model (RM [fact] ) to evaluate factuality
of responses and create a factuality preference pair
for fact-based instructions: ( _x ∈_ _X_ [fact] _, y_ true _, y_ false).
Specifically, we build RM [fact] with retrieval augmentation to measure the percentage of facts in a response that are correct. RM [fact] comprises two main
components: atomic fact decomposition and retrieval augmented claim verification. We detail the
components and ablate their impacts on the quality
of RM [fact] in Appendix A.4. We compute factuality reward for the same responses sampled from
SFT : RM [fact] ( _x, y_ 0) _,_ RM [fact] ( _x, y_ 1) _, · · ·_ . The response with the highest (lowest) factuality reward is chosen as _y_ true ( _y_ false). Note that if the
chosen paired responses show large difference in
instruction-following reward, we discard the pair;
i.e., _|_ RM [IF] ( _x, y_ true) _−_ RM [IF] ( _x, y_ false) _| >_ 0 _._ 5. As illustrated in Figure 4(b), in factuality-aware DPO
training, the model is initialized from SFT and
the fine-tuned model is our final factuality-aware
aligned model, denoted SFT + DPO . The specific procedures for fine-tuning models in both the
SFT and DPO are described in Appendix A.5.