GPT-4-Turbo (1106) Official Aligned Model as Ref.
LC (%) WR (%) SD LC (%) WR (%) SD
Qwen2-1.5B-Instruct **3.91** 3.00 0.54 50 50               Qwen2-1.5B
Base Model + MAGPIE 3.48 **5.32** 0.67 **56.66** **66.27** 1.50
Qwen1.5-4B-Chat 5.89 4.74 0.67 50 50             Qwen1.5-4B
Base Model + MAGPIE **9.1** **10.96** 0.93 **68.09** **72.42** 1.42
Qwen1.5-7B-Chat 14.75 11.77 0.97 50 50             Qwen1.5-7B
Base Model + MAGPIE **15.10** **18.51** 1.14 46.28 **58.53** 1.44
demonstrates remarkable performance gains. Specifically, it outperforms the official Llama-3-8BInstruct model on both the AlpacaEval 2 and Arena-Hard benchmarks. Most notably, our model even
surpasses GPT-4-Turbo(1106) on AlpacaEval 2. Finally, we highlight that our alignment process uses
no more than 400K data, whereas the official aligned models are aligned with more than 10M data
samples. This demonstrates the high quality of the data generated by MAGPIE.
**MAGPIE can enhance the performance of other backbone models.** Table 2 illustrates the efficacy
of MAGPIE when applied to generate instruction dataset and fine-tune other base models, i.e., Qwen21.5B, Qwen1.5-4B, and Qwen1.5-7B. The results demonstrate that our fine-tuned models achieve
better performance than the official aligned models, which have undergone both supervised finetuning and preference tuning. These results underscore the effectiveness of MAGPIE and the quality
of its generated instructions. In addition, we apply MAGPIE-generated datasets to align Llama-3.1Minitron-4B-Width-Base (Sreenivas et al., 2024) and Llama-3.1-8B-Instruct (Dubey et al., 2024)
using SFT followed by DPO. The resulting aligned model, which we term MagpieLM, achieves
remarkable performance and ranks first among popular open-source instruction models with fewer
than 10 billion parameters. The details of MagpieLM are deferred to Appendix B.
**Performance of MAGPIE on More Benchmarks.** We report the performance of models supervised
fine-tuned using MAGPIE-Air and MAGPIE-Pro, evaluated across a range of tasks featured on the
Huggingface Open LLM Leaderboard (Beeching et al., 2023) in Table 3. The tasks includes MMLU
(Hendrycks et al., 2020), ARC Challenge (Clark et al., 2018), HellaSwag (Zellers et al., 2019),
TruthfulQA (Lin et al., 2021), WinoGrande (Levesque et al., 2012), and GSM8K (Cobbe et al., 2021).
We also perform experiments on MMLU-Redux (Gema et al., 2024) with zero-shot prompting. Our
experimental results demonstrate that models fine-tuned with MAGPIE-Air and MAGPIE-Pro achieve
comparable performance to the official instruct model and other baselines.
We note that the performance of MAGPIE may degrade on reasoning tasks, which is attributed to the
small proportion of reasoning instructions in MAGPIE-Air and MAGPIE-Pro datasets. In response, we
provide a supplementary "booster" dataset containing 150K math, code, and reasoning instructions
using the MAGPIE extension mentioned in Section 2.2. We combine this booster dataset with
MAGPIE-Pro-300K-Filtered and create MAGPIE-Pro-Mix-Filtered. Experimental results presented
in Table 3 demonstrate that the model supervised fine-tuned using the mixed dataset effectively
addresses the initial weakness in reasoning tasks. Notably, this new model ranks among the top-3 of
all model checkpoints, performing only slightly weaker than OpenHermes 2.5 (1M conversations) and
Llama-3-8B-Instruct (>10M conversations). This significant improvement showcases the flexibility
and adaptability of the MAGPIE framework in generating task-specific instruction data.
**Additional Experimental Results.** We defer additional experimental results and analysis of multiturn datasets, i.e., MAGPIE-Air-MT and MAGPIE-Pro-MT, to Appendix F.1. We conduct a detailed
comparison between MAGPIE and Self-Instruct in Appendix F.2. In addition, ablations on data
quantity, quality, filter designs, and response generator are deferred in Appendices F.3, F.4, and F.5.
MAGPIE model’s performance on trustworthiness benchmarks is reported in Appendices F.6.
5 RELATED WORK
**LLM Alignment.** Instruction tuning (Wei et al., 2022) and preference tuning (Bai et al., 2022)
are widely used to align the responses of LLMs with human values. Instruction tuning utilizes an
Preprint
Table 3: This table compares the performance of models supervised-fine-tuned on MAGPIE-Air,
MAGPIE-Pro, and MAGPIE-Pro-Mix against baselines and official instruct model across various
downstream benchmarks. All models are supervised-fine-tuned on the Llama-8B base models.
|Alignment Setup|MMLU (5) ARC (25) HellaSwag (10) TruthfulQA (0) WinoGrande (5) GSM8K (5)|MMLU-Redux (0)|Average|
|---|---|---|---|
|ShareGPT
Evol Instruct
GenQA
OpenHermes 1
OpenHermes 2.5
Tulu V2 Mix
WildChat
UltraChat|66.03
58.45
81.50
52.34
74.03
48.67
65.62
60.75
82.70
52.87
76.16
42.91
63.45
58.53
79.65
48.85
74.03
43.14
65.42
62.29
82.15
50.85
75.61
47.16
65.70
61.86
82.53
51.35
76.09
67.02
66.34
59.22
82.80
47.99
76.16
58.07
65.95
59.22
81.39
53.18
75.30
48.75
65.23
62.12
81.68
52.76
75.53
50.57|50.68
52.73
51.87
46.07
46.07
46.97
52.59
50.75|61.67
61.96
59.93
61.36
**66.24**
62.51
62.34
62.66|
|MAGPIE-Air-300K-Filtered
MAGPIE-Pro-300K-Filtered
**MAGPIE-Pro-Mix-Filtered**|64.45
61.01
79.90
53.48
72.38
52.24
64.25
60.41
80.52
52.46
73.32
47.92
65.65
59.64
80.72
50.81
73.24
63.08|52.34
52.16
56.34|62.25
61.58
**64.21**|
|Llama-3-8B-Instruct|67.82
61.52
78.67
52.47
72.14
71.72|58.60|**66.13**|
instruction dataset to fine-tune LLMs, where each instruction data consists of one turn or multiple
turns of instructions and desired responses. The performance of instruction tuning heavily relies on
the quality of instruction data (Taori et al., 2023; Wang et al., 2023; Zhou et al., 2023). Preference
tuning further improves responses of LLMs using reinforcement learning human feedback (RLHF)
(Bai et al., 2022) or preference optimization (Azar et al., 2024; Ethayarajh et al., 2024; Hong et al.,
2024; Rafailov et al., 2023) based on a preference dataset.
**Alignment Dataset Construction.** We classify the existing methods of creating datasets for model
alignment into two main categories: human interactions with LLMs and synthetic instruction generation. To create datasets for alignment, previous studies have collected **human** interactions with LLMs
(Databricks, 2023; Zhao et al., 2024; Zheng et al., 2024; 2023; Köpf et al., 2023). However, manually
crafting instructions is not only time-consuming and labor-intensive, but may also incorporate toxic
content (Zhao et al., 2024). Another category of approaches (Wang et al., 2023; Taori et al., 2023; Xu
et al., 2023a;b; Wang et al., 2024c; Sun et al., 2023) focus on prompting LLMs to generate **synthetic**
instruction datasets, beginning with a small set of human-annotated seed instructions and expanding
these through few-shot prompting. However, these methods face a diversity challenge, as few-shot
prompting often results in new instructions that are too similar to the original seed questions (Li et al.,
2024a). To enhance coverage, some research (Ding et al., 2023; Li et al., 2024a) summarizes world
knowledge and employs it to generate synthetic datasets. We note that our MAGPIE dataset also
belongs to the synthetic dataset. However, we leverage the prompt template without any requirement
for seed questions or prompt engineering.
Compared to the above two main categories, alignment data can also be generated by **transforming**
existing data (Wang et al., 2022; Sanh et al., 2022; Gandhi et al., 2024). However, the constrained
variety of NLP tasks in these datasets may impede the ability of tuned LLMs to generalize in realworld scenarios (Li et al., 2024a). There are also **mixture** datasets (e.g., (Ivison et al., 2023; Teknium,
2023a; Liu et al., 2024b; Zhou et al., 2023)) that combine or select high-quality instruction data from
various existing open-source instruction datasets to enhance coverage (Ivison et al., 2023; Teknium,
2023a) and/or improve overall performance (Liu et al., 2024b; Zhou et al., 2023). There are also data
construction methods focusing on improving the reasoning and math abilities (Yue et al., 2023; 2024),
which can be further merged with MAGPIE for creating a better mixture of data for instruction tuning.
**Training Data Extraction.** Language models have the capability to memorize examples from their
training datasets, potentially enabling malicious users to extract private information (Brown et al.,
2022; Biderman et al., 2023; Carlini et al., 2021). Pioneering work (Krishna et al., 2020; Carlini
et al., 2021; Nasr et al., 2023) has demonstrated that it is possible to extract private pre-training data
from BERT (Devlin et al., 2018), GPT-2 (Radford et al., 2018), and ChatGPT (Achiam et al., 2023),
respectively. Yu et al. (2023) propose several techniques including adjusting sampling strategies to
better extract training datasets from language models. Recently, Kassem et al. (2024) propose a blackbox prompt optimization method that uses an attacker LLM to extract high levels of memorization in
a victim LLM. Wang et al. (2024b) leverage membership inference attack (MIA) to extract fine-tuning
datasets from fine-tuned language models. Bai et al. (2024) extract the training dataset of production
language models via special characters (e.g., structural symbols of JSON files, and, # in emails and
online posts). Different from the prior work, we aim to create publicly available alignment datasets
with minimal human effort by leveraging the remarkable generation capabilities of LLMs, rather than
extracting private training data from LLMs.
Preprint
6 LIMITATIONS, DISCUSSIONS, AND ETHICAL CONSIDERATIONS
**Limitations and Discussions.** MAGPIE-aligned LLMs demonstrate strong performance on instruction following benchmarks compared to the official Llama-3-8B-Instruct. However, we observe a
performance degradation on math and reasoning benchmarks. Although we leverage the techniques
described in Section 2.2 to generate specialized booster reasoning datasets, there is still a performance
gap between MAGPIE-aligned LLMs and the official models. Enhancing the reasoning ability of
MAGPIE-aligned models presents a promising direction for future research.
**Societal Impact and Potential Harmful Consequences.** The primary objective of this paper is
to develop a scalable method to synthesize instruction data to enhance the instruction-following
capabilities of LLMs, and thus align them with human values. However, the data generated by
MAGPIE may contain harmful instructions and/or responses, which may lead to unsafe behaviors if
used raw in instruction tuning. Our empirical evaluations indicate that such harmful data instances
constitute less than 1% of the dataset. Our data filtering technique in Appendix C can identify and
remove these instances, thus mitigating the risk.
7 CONCLUSION
In this paper, we developed a scalable method, MAGPIE, to synthesize instruction data for fine-tuning
large language models. MAGPIE leveraged the predefined instruction templates of open-weight LLMs
and crafted a prompt specifying only the role of instruction provider. Given the crafted prompt, the
LLM then generated detailed instructions due to their auto-regressive nature. MAGPIE then sent the
generated instructions to the LLM to generate corresponding responses. These pairs of instructions
and responses constituted the instruction dataset. We used Llama-3-8B-instruct to label the instruction
dataset and developed a filtering technique to select effective data instances for instruction tuning.
We fine-tuned the Llama-3-8B base model using the selected data, and demonstrated that the finetuned model outperformed those fine-tuned using all baselines. Moreover, our fine-tuned models
outperformed the official aligned model, Llama-3-8B-Instruct, which has been instruction-tuned
and preference-optimized using more than 10M data instances. This highlighted the quality of the
instruction data synthesized by MAGPIE.
8 ACKNOWLEDGEMENT
The research of Z. Xu, F. Jiang, L. Niu, and R. Poovendran is partially supported by the National
Science Foundation (NSF) AI Institute for Agent-based Cyber Threat Intelligence and Operation
(ACTION) under grant IIS 2229876 and Air Force Office of Scientific Research (AFOSR) under
grant FA9550-23-1-0208. The research of Y. Choi is partially supported by the National Science
Foundation (NSF) under grant DMS-2134012 (Scaling Laws of Deep Learning) and the Office of
Naval Research (ONR) under grant N00014-24-1-2207 (Symbolic Knowledge Distillation of LLMs
for All: Diverse Scales, Skills, and Values).
This work is supported in part by funds provided by the National Science Foundation, Department of
Homeland Security, and IBM. Any opinions, findings, and conclusions or recommendations expressed
in this material are those of the author(s) and do not necessarily reflect the views of the NSF or its
federal agency and industry partners.
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany
Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report:
A highly capable language model locally on your phone. _arXiv preprint
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman,
Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.
_arXiv preprint
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal
Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from
Preprint
human preferences. In _International Conference on Artificial Intelligence and Statistics_, pp.
4447–4455. PMLR, 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge,
Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu,
Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan,
Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin
Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng
Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou,
Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. _arXiv preprint
2023.
Yang Bai, Ge Pei, Jindong Gu, Yong Yang, and Xingjun Ma. Special characters attack: Toward
scalable training data extraction from large language models. _arXiv preprint
2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn
Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson
Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez,
Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario
Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan.
Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert,
Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open llm leader[board. https://huggingface.co/spaces/open-llm-leaderboard/open_llm_](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
[leaderboard, 2023.](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
Stella Biderman, Usvsn Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. Emergent and predictable memorization in large language models.
_Advances in Neural Information Processing Systems_, 36, 2023.
Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr.
What does it mean for a language model to preserve privacy? In _Proceedings of the 2022 ACM_
_Conference on Fairness, Accountability, and Transparency_, pp. 2280–2292, 2022.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine
Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data
from large language models. In _30th USENIX Security Symposium (USENIX Security 21)_, pp.
2633–2650, 2021.
Jiuhai Chen, Rifaa Qadri, Yuxin Wen, Neel Jain, John Kirchenbauer, Tianyi Zhou, and Tom Goldstein. Genqa: Generating millions of instructions from a handful of prompts. _arXiv preprint_
_
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay
Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training a better alpaca with fewer data.
_arXiv preprint
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng,
Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An
[open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:](https://lmsys.org/blog/2023-03-30-vicuna/)
[//lmsys.org/blog/2023-03-30-vicuna/.](https://lmsys.org/blog/2023-03-30-vicuna/)
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and
Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.
_arXiv preprint
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser,
Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve
math word problems. _arXiv preprint
Preprint
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu,
and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. _arXiv_
_preprint
Databricks. Databricks dolly-15k, 2023. [URL https://huggingface.co/datasets/](https://huggingface.co/datasets/databricks/databricks-dolly-15k)
[databricks/databricks-dolly-15k.](https://huggingface.co/datasets/databricks/databricks-dolly-15k)
Daniel Deutsch, Rotem Dror, and Dan Roth. On the limitations of reference-free evaluations of
generated text. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), _Proceedings of_
_the 2022 Conference on Empirical Methods in Natural Language Processing_, pp. 10960–10977,
Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis[tics. doi: 10.18653/v1/2022.emnlp-main.753. URL https://aclanthology.org/2022.](https://aclanthology.org/2022.emnlp-main.753)
[emnlp-main.753.](https://aclanthology.org/2022.emnlp-main.753)
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. _arXiv preprint
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong
Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional
conversations. _arXiv preprint
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel
Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha
Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.
_arXiv preprint
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled
alpacaeval: A simple way to debias automatic evaluators. _arXiv preprint
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model
alignment as prospect theoretic optimization. _arXiv preprint
Saumya Gandhi, Ritu Gala, Vijay Viswanathan, Tongshuang Wu, and Graham Neubig. Better
synthetic data by retrieving and transforming existing datasets. _arXiv preprint
2024.
Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria
Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani,
et al. Are we done with mmlu? _arXiv preprint
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and
Jacob Steinhardt. Measuring massive multitask language understanding. _arXiv preprint_
_
Jiwoo Hong, Noah Lee, and James Thorne. Reference-free monolithic preference optimization with
odds ratio. _arXiv preprint
Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang,
Wenhan Lyu, Yixuan Zhang, Xiner Li, Hanchi Sun, Zhengliang Liu, Yixin Liu, Yijue Wang,
Zhikun Zhang, Bertie Vidgen, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chunyuan Li,
Eric P. Xing, Furong Huang, Hao Liu, Heng Ji, Hongyi Wang, Huan Zhang, Huaxiu Yao, Manolis
Kellis, Marinka Zitnik, Meng Jiang, Mohit Bansal, James Zou, Jian Pei, Jian Liu, Jianfeng Gao,
Jiawei Han, Jieyu Zhao, Jiliang Tang, Jindong Wang, Joaquin Vanschoren, John Mitchell, Kai
Shu, Kaidi Xu, Kai-Wei Chang, Lifang He, Lifu Huang, Michael Backes, Neil Zhenqiang Gong,
Philip S. Yu, Pin-Yu Chen, Quanquan Gu, Ran Xu, Rex Ying, Shuiwang Ji, Suman Jana, Tianlong
Chen, Tianming Liu, Tianyi Zhou, William Yang Wang, Xiang Li, Xiangliang Zhang, Xiao Wang,
Xing Xie, Xun Chen, Xuyu Wang, Yan Liu, Yanfang Ye, Yinzhi Cao, Yong Chen, and Yue Zhao.
Trustllm: Trustworthiness in large language models. In _Forty-first International Conference on_
_Machine Learning_ [, 2024. URL https://openreview.net/forum?id=bWUU0LwwMp.](https://openreview.net/forum?id=bWUU0LwwMp)
Preprint
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi,
Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in a changing climate:
Enhancing lm adaptation with tulu 2. _arXiv preprint
Aly M Kassem, Omar Mahmoud, Niloofar Mireshghallah, Hyunwoo Kim, Yulia Tsvetkov, Yejin
Choi, Sherif Saad, and Santu Rana. Alpaca against vicuna: Using llms to uncover memorization of
llms. _arXiv preprint
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens,
Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David
Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander
Mattick. Openassistant conversations - democratizing large language model alignment. In A. Oh,
T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), _Advances in Neural_
_Information Processing Systems_, volume 36, pp. 47669–47681. Curran Associates, Inc., 2023.
URL [https://proceedings.neurips.cc/paper_files/paper/2023/file/](https://proceedings.neurips.cc/paper_files/paper/2023/file/949f0f8f32267d297c2d4e3ee10a2e7e-Paper-Datasets_and_Benchmarks.pdf)
[949f0f8f32267d297c2d4e3ee10a2e7e-Paper-Datasets_and_Benchmarks.](https://proceedings.neurips.cc/paper_files/paper/2023/file/949f0f8f32267d297c2d4e3ee10a2e7e-Paper-Datasets_and_Benchmarks.pdf)
[pdf.](https://proceedings.neurips.cc/paper_files/paper/2023/file/949f0f8f32267d297c2d4e3ee10a2e7e-Paper-Datasets_and_Benchmarks.pdf)
Kalpesh Krishna, Gaurav Singh Tomar, Ankur P. Parikh, Nicolas Papernot, and Mohit Iyyer. Thieves
on sesame street! model extraction of bert-based apis. In _International Conference on Learning_
_Representations_ [, 2020. URL https://openreview.net/forum?id=Byl5NREFDr.](https://openreview.net/forum?id=Byl5NREFDr)
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E.
Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model
serving with pagedattention. In _Proceedings of the ACM SIGOPS 29th Symposium on Operating_
_Systems Principles_, 2023.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu,
Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models
for language modeling. _arXiv preprint
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In
_Thirteenth international conference on the principles of knowledge representation and reasoning_,
2012.
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large language model society. _Advances in Neural_
_Information Processing Systems_, 36:51991–52008, 2023a.
Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, et al. Synthetic data (almost) from
scratch: Generalized instruction tuning for language models. _arXiv preprint
2024a.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion
Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024b. URL
[https://lmsys.org/blog/2024-04-19-arena-hard/.](https://lmsys.org/blog/2024-04-19-arena-hard/)
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy
Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following
[models. https://github.com/tatsu-lab/alpaca_eval, 2023b.](https://github.com/tatsu-lab/alpaca_eval)
Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu,
Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base llms: Rethinking alignment
via in-context learning. _arXiv preprint
Bill Yuchen Lin, Khyathi Chandu, Faeze Brahman, Yuntian Deng, Abhilasha Ravichander, Valentina
Pyatkin, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking language models with
[challenging tasks from real users in the wild, 2024. URL https://huggingface.co/](https://huggingface.co/spaces/allenai/WildBench)
[spaces/allenai/WildBench.](https://huggingface.co/spaces/allenai/WildBench)
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human
falsehoods. _arXiv preprint
Preprint
Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi
Peng, Diyi Yang, Denny Zhou, et al. Best practices and lessons learned on synthetic data for
language models. _arXiv preprint
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for
alignment? a comprehensive study of automatic data selection in instruction tuning. In _The Twelfth_
_International Conference on Learning Representations_ [, 2024b. URL https://openreview.](https://openreview.net/forum?id=BTKAeLqLMw)
[net/forum?id=BTKAeLqLMw.](https://openreview.net/forum?id=BTKAeLqLMw)
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a referencefree reward. _arXiv preprint
[Meta. Llama 3. https://ai.meta.com/blog/meta-llama-3/, 2024.](https://ai.meta.com/blog/meta-llama-3/)
Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito,
Christopher A Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable
extraction of training data from (production) language models. _arXiv preprint
2023.
[OpenAI. Tiktoken. https://github.com/openai/tiktoken, 2024.](https://github.com/openai/tiktoken)
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language
understanding by generative pre-training. 2018.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea
Finn. Direct preference optimization: Your language model is secretly a reward model. In
_Thirty-seventh Conference on Neural Information Processing Systems_ [, 2023. URL https:](https://openreview.net/forum?id=HPuSIXJaa9)
[//openreview.net/forum?id=HPuSIXJaa9.](https://openreview.net/forum?id=HPuSIXJaa9)
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine
Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker,
Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen,
Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen,
Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao,
Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training
enables zero-shot task generalization. In _International Conference on Learning Representations_,
[2022. URL https://openreview.net/forum?id=9Vrb9D0WI4.](https://openreview.net/forum?id=9Vrb9D0WI4)
Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Mostofa
Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Llm pruning
and distillation in practice: The minitron approach. _arXiv preprint
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming
Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with
minimal human supervision. _Advances in Neural Information Processing Systems_, 36, 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy
Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model.
[https://github.com/tatsu-lab/stanford_alpaca, 2023.](https://github.com/tatsu-lab/stanford_alpaca)
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya
Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al.
Gemma 2: Improving open language models at a practical size. _arXiv preprint
2024.
[Llama Team. Meta llama guard 2. https://github.com/meta-llama/PurpleLlama/](https://github.com/meta-llama/PurpleLlama/blob/main/Llama-Guard2/MODEL_CARD.md)
[blob/main/Llama-Guard2/MODEL_CARD.md, 2024.](https://github.com/meta-llama/PurpleLlama/blob/main/Llama-Guard2/MODEL_CARD.md)
Teknium. Openhermes dataset, 2023a. [URL https://huggingface.co/datasets/](https://huggingface.co/datasets/teknium/openhermes)
[teknium/openhermes.](https://huggingface.co/datasets/teknium/openhermes)
Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023b.
[URL https://huggingface.co/datasets/teknium/OpenHermes-2.5.](https://huggingface.co/datasets/teknium/OpenHermes-2.5)
Preprint
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay
Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation
and fine-tuned chat models. _arXiv preprint
Hoang Tran, Chris Glaze, and Braden Hancock. Iterative DPO alignment. Technical report, Snorkel
AI, 2023.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. _Journal of machine_
_learning research_, 9(11), 2008.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences
via multi-objective reward modeling and mixture-of-experts. In _EMNLP_, 2024a.
Jeffrey G Wang, Jason Wang, Marvin Li, and Seth Neel. Pandora’s white-box: Increased training
data leakage in open llms. _arXiv preprint
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei,
Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan
Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson,
Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir
Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri,
Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta
Patro, Tanay Dixit, and Xudong Shen. Super-NaturalInstructions: Generalization via declarative
instructions on 1600+ NLP tasks. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.),
_Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pp.
5085–5109, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational
[Linguistics. doi: 10.18653/v1/2022.emnlp-main.340. URL https://aclanthology.org/](https://aclanthology.org/2022.emnlp-main.340)
[2022.emnlp-main.340.](https://aclanthology.org/2022.emnlp-main.340)
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and
Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In
_Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume_
_1: Long Papers)_, pp. 13484–13508, Toronto, Canada, 2023. Association for Computational
Linguistics.
Zifeng Wang, Chun-Liang Li, Vincent Perot, Long T Le, Jin Miao, Zizhao Zhang, Chen-Yu Lee, and
Tomas Pfister. Codeclm: Aligning language models with tailored synthetic data. _arXiv preprint_
_
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du,
Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In _International_
_Conference on Learning Representations_ [, 2022. URL https://openreview.net/forum?](https://openreview.net/forum?id=gEZrGCozdqR)
[id=gEZrGCozdqR.](https://openreview.net/forum?id=gEZrGCozdqR)
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang.
Iterative preference learning from human feedback: Bridging theory and practice for rlhf under
kl-constraint, 2024.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin
Jiang. Wizardlm: Empowering large language models to follow complex instructions. _arXiv_
_preprint
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model
with parameter-efficient tuning on self-chat data. In Houda Bouamor, Juan Pino, and Kalika
Bali (eds.), _Proceedings of the 2023 Conference on Empirical Methods in Natural Language_
_Processing_, pp. 6268–6278, Singapore, December 2023b. Association for Computational Linguis[tics. doi: 10.18653/v1/2023.emnlp-main.385. URL https://aclanthology.org/2023.](https://aclanthology.org/2023.emnlp-main.385)
[emnlp-main.385.](https://aclanthology.org/2023.emnlp-main.385)
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li,
Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. _arXiv preprint_
_
Preprint
Da Yin, Xiao Liu, Fan Yin, Ming Zhong, Hritik Bansal, Jiawei Han, and Kai-Wei Chang. Dynosaur:
A dynamic growth paradigm for instruction-tuning data curation. _arXiv preprint
2023.
Weichen Yu, Tianyu Pang, Qian Liu, Chao Du, Bingyi Kang, Yan Huang, Min Lin, and Shuicheng
Yan. Bag of tricks for training data extraction from language models. In _International Conference_
_on Machine Learning_, pp. 40306–40320. PMLR, 2023.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu
Chen. Mammoth: Building math generalist models through hybrid instruction tuning.
_ArXiv_ [, abs/2309.05653, 2023. URL https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:261696697)
[261696697.](https://api.semanticscholar.org/CorpusID:261696697)
Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the
web, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine
really finish your sentence? _arXiv preprint
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat:
1m chatGPT interaction logs in the wild. In _The Twelfth International Conference on Learning_
_Representations_ [, 2024. URL https://openreview.net/forum?id=Bl8u7ZRlbM.](https://openreview.net/forum?id=Bl8u7ZRlbM)
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang,
Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.
Judging LLM-as-a-judge with MT-bench and chatbot arena. In _Thirty-seventh Conference on_
_Neural Information Processing Systems Datasets and Benchmarks Track_ [, 2023. URL https:](https://openreview.net/forum?id=uccHPGDlao)
[//openreview.net/forum?id=uccHPGDlao.](https://openreview.net/forum?id=uccHPGDlao)
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao
Zhuang, Zhuohan Li, Zi Lin, Eric Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. LMSYSchat-1m: A large-scale real-world LLM conversation dataset. In _The Twelfth International_
_Conference on Learning Representations_ [, 2024. URL https://openreview.net/forum?](https://openreview.net/forum?id=BOfDKxfwt0)
[id=BOfDKxfwt0.](https://openreview.net/forum?id=BOfDKxfwt0)
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia
Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. _Advances in Neural Information_
_Processing Systems_, 36, 2023.
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li,
Huazuo Gao, Shirong Ma, et al. Deepseek-coder-v2: Breaking the barrier of closed-source models
in code intelligence. _arXiv preprint
Preprint
A STATISTICS OF INSTRUCTION DATASETS GENERATED BY MAGPIE
COMPARED TO OTHER INSTRUCTION DATASETS.
MAGPIE can be readily deployed to state-of-the-art open-weight model families, including but
not limited to Llama-3 (Meta, 2024), Llama-3.1 (Dubey et al., 2024), Qwen2 (Yang et al., 2024),
Gemma-2 (Team et al., 2024), and Phi-3 (Abdin et al., 2024) model families.
In what follows, we compare datasets generated by MAGPIE with the above model families compared
to other state-of-the-art instruction datasets. The MAGPIE dataset family encompasses over 11.4
million diverse and high-quality instructions and corresponding responses generated from state-ofthe-art open-source models. This corpus represents the largest alignment dataset for LLMs that does
not rely on human-written questions or employ complex multi-stage pipelines.
Table 4: Statistics of MAGPIE family compared to other instruction datasets. Tokens are counted
using the tiktoken library (OpenAI, 2024).
Instruction Human Response
Dataset Name #Convs #Turns #Tokens / Turn #Total Tokens
Source Effort Generator
Alpaca (Taori et al., 2023) 52K 1 Low text-davinci-003 67.38 _±_ 54 _._ 88 3.5M
Synthetic Evol Instruct (Xu et al., 2023a) 143K 1 Low ChatGPT 473.33 _±_ 330 _._ 13 68M
UltraChat (Ding et al., 2023) 208K 3.16 Low GhatGPT 376.58 _±_ 177 _._ 81 238M
Human
Dolly (Databricks, 2023) 15K 1 High ChatGPT 94.61 _±_ 135 _._ 84 1.42M
ShareGPT (Zheng et al., 2023) 112K 4.79 High ChatGPT 465.38 _±_ 368 _._ 37 201M
WildChat (Zhao et al., 2024) 652K 2.52 High GPT-3.5 & GPT-4 727.09 _±_ 818 _._ 84 852M
LMSYS-Chat-1M (Zheng et al., 2024) 1M 2.01 High Mix 260.37 _±_ 346 _._ 97 496M
Deita (Liu et al., 2024b) 9.5K 22.02       - Mix 372.78 _±_ 182 _._ 97 74M
Mixture OpenHermes (Teknium, 2023a) 243K 1 - Mix 297.86 _±_ 258 _._ 45 72M
Tulu V2 Mixture (Ivison et al., 2023) 326K 2.31       - Mix 411.94 _±_ 447 _._ 48 285M


Llama-3-MAGPIE-Air **3M** 1 **No** Llama-3-8B-Instruct 426.39 _±_ 217 _._ 39 1.28B
Llama-3-MAGPIE-Air-MT 300K 2 **No** Llama-3-8B-Instruct 610.80 _±_ 90 _._ 61 366M
Llama-3-MAGPIE-Pro 1M 1 **No** Llama-3-70B-Instruct 478.00 _±_ 211 _._ 09 477M
Llama-3-MAGPIE-Pro-MT 300K 2 **No** Llama-3-70B-Instruct 554.53 _±_ 133 _._ 64 333M
Llama-3.1-MAGPIE-Pro 1M 1 **No** Llama-3.1-70B-Instruct 482.35 _±_ 378 _._ 45 482M
Llama-3.1-MAGPIE-Pro-MT 300K 2 **No** Llama-3.1-70B-Instruct 552.53 _±_ 325 _._ 49 331M
Qwen2-MAGPIE-Air 3M 1 **No** Qwen2-7B-Instruct 577.87 _±_ 416 _._ 10 1.73B
Qwen2-MAGPIE-Pro 1M 1 **No** Qwen2-72B-Instruct 424.87 _±_ 339 _._ 71 424M
Gemmma-2-MAGPIE-Pro 1M 1 **No** Gemma-2-27b-it 483.90 _±_ 237 _._ 80 259M
Phi-3-MAGPIE-Pro 534K 1 **No** Phi-3-Medium-Instruct 391.38 _±_ 414 _._ 32 391M
B MAGPIELM

Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs
  with Nothing

Zhangchen Xu
Fengqing Jiang
Luyao Niu
Yuntian Deng
Radha Poovendran
Yejin Choi
Bill Yuchen Lin


Greedy Decoding
Alpaca Eval 2 LC Arena Hard WildBench Score
🌟 **MagpieLM-8B-Chat**

Benchmark Scores of Small/Medium Language Models (<10B)


0 40 80 120 160
Figure 6: This figure shows the performance of MAGPIELM-4B-Chat and MAGPIELM-8B-Chat
compared with baselines. MAGPIELM significantly outperforms baselines of similar sizes.
Preprint
In this section, we discuss the details of our MAGPIELM. To construct the SFT and DPO datasets
for aligning MAGPIELM, we select 550K and 200K high-quality instructions, respectively, from the
MAGPIE family. These instructions cover diverse categories, ensuring a comprehensive training set.
We then generate corresponding responses using the Gemma-2-9b-it model (Team et al., 2024).
The benchmark performance of both models is demonstrated in Figure 6. Notably, MAGPIELM
significantly outperforms other baselines of similar model sizes across multiple benchmarks, including
Alpaca Eval 2 (Li et al., 2023b), Arena Hard (Li et al., 2024b), and Wildbench (Lin et al., 2024).
C FILTER SETUPS
In this section, we explore potential filter configurations for selecting high-quality instructional data
for fine-tuning purposes. We provide the following metrics to enable users to customize their filtered
MAGPIE dataset:
1. **Input Length** : The total number of characters in the instructions.
2. **Output Length** : The total number of characters in the responses.
3. **Task Category** : The specific category of the instructions. See Appendix D.1 for details.
4. **Input Quality** : The clarity, specificity, and coherence of the instructions, rated as ‘very
poor’, ‘poor’, ‘average’, ‘good’, and ‘excellent’.
5. **Input Difficulty** : The level of knowledge required to address the task described in the
instruction, rated as ‘very easy’, ‘easy’, ‘medium’, ‘hard’, or ‘very hard’.
6. **Minimum Neighbor Distance** : The embedding distance to the nearest neighbor. Can be
used for filtering out repetitive or similar instances.
7. **Reward** : Denoted as _r_ _[∗]_ . See Section 3 for details. This metric can be used to filter out
low-quality responses, such as repetitions or refusals.
8. **Reward Difference** : Denoted as _r_ _[∗]_ _−_ _rbase_ . See Section 3 for details.
We provide several off-the-shelf configurations, as demonstrated in Table 5. We defer the detailed
performance analysis of each filter configuration for MAGPIE-Pro to Appendix F.4.
Table 5: Different filter configurations we provide. We note that the Output Length filter is applied
last. Specifically, this filter selects the _k_ instances of the longest responses. In our experiments, we
empirically set _τ_ 1 = _−_ 12, and _τ_ 2 = 0.
Input Output Task Input Input Min Neighbor Reward
Source Dataset Filter Name #Convs Reward
Length Length Category Quality Difficulty Distance Difference
MAGPIE-Air Filter 300K  - Longest  - _≥_ good _≥_ medium _>_ 0  - _> τ_ 2
MAGPIE-Pro
Filter 300K - Longest - _≥_ average - _>_ 0 _> τ_ 1 Filter2 300K - Longest - _≥_ good _≥_ easy _>_ 0 _> τ_ 1 Filter3 300K - Longest - - - _>_ 0 _> τ_ 1 Filter4 300K - Longest - _≥_ good _≥_ easy _>_ 0 - _> τ_ 2
Filter5 338K - - - _≥_ good _≥_ easy _>_ 0 _> τ_ 1 Filter6 200K - Longest - - 50% easy + 50% _>_ easy _>_ 0 _> τ_ 1
D MORE DATASET ANALYSIS
This section provides additional dataset analysis, complementing the discussions in Section 3.
Statistics including lengths of instructions and responses are illustrated in Figure 7.
D.1 ADDITIONAL ANALYSIS ON DATASET COVERAGE AND ATTRIBUTES.
**Dataset Coverage Measured by T-SNE.** Figure 8 presents the t-SNE plots of MAGPIE-Pro, Alpaca,
Evol Instruct, and UltraChat. Each t-SNE plot is generated by randomly sampling 10,000 instructions
from the associated dataset. We observe that the t-SNE plot of MAGPIE-Pro encompasses the area
covered by the plots of Alpaca, Evol Instruct, and UltraChat. This suggests that MAGPIE-Pro provides
a broader or more diverse range of topics, highlighting its extensive coverage across varied themes
and subjects.
Preprint
|Col1|Col2|Col3|
|---|---|---|
||||
(b) Output Length of MAGPIE-Air (in tokens)
|Col1|Col2|Col3|
|---|---|---|
|

|

|

|
(d) Input Length of MAGPIE-Pro (in tokens)
Figure 7: Lengths of instructions
and responses in MAGPIE-Air/Pro.
Figure 8: This figure compares the t-SNE plot of MAGPIE-Pro
with those of Alpaca, Evol Instruct, and UltraChat, each of
which is sampled with 10,000 instructions. The t-SNE plot of
MAGPIE-Pro encompasses the area covered by the other plots,
demonstrating the comprehensive coverage of MAGPIE-Pro.
**Task Categories of MAGPIE-Pro and MAGPIE-Air.** Figure 9 illustrates the task category distributions for MAGPIE-Pro and MAGPIE-Air, as labeled by Llama-3-Instruct. We observe that the
task category distributions of these two datasets are largely similar, however, MAGPIE-Pro exhibits a
higher percentage of creative writing tasks.
(b) Task categories of MAGPIE-Air.
(a) Task categories of MAGPIE-Pro.
Figure 9: This figure visualizes the task category of MAGPIE-Pro and MAGPIE-Air by topic tags.
**Visualization of Root Verbs and Their Direct Noun Objects.** Figure 10 visualizes the top
common root verbs and their direct noun objects of MAGPIE-Air dataset. This indicates the diverse
topic coverage of MAGPIE-Air.
D.2 ADDITIONAL SAFETY ANALYSIS
Table 6 illustrates the percentage of different unsafe categories of MAGPIE-Air and MAGPIE-Pro, as
labeled by Llama-Guard-2 (Team, 2024). We have two key observations. First, the proportion of data
Preprint
Figure 10: This figure demonstrates the top 20 most common root verbs (shown in the inner circle)
and their top 5 direct noun objects (shown in the outer circle) within the MAGPIE-Air dataset. This
indicates that MAGPIE encompasses a broad range of topics.
containing potentially harmful queries is minimal, with less than 1% for both datasets. Second, the
majority of unsafe responses fall into the category of specialized advice, which includes responses
that may offer specialized financial, medical, or legal advice, or suggest that dangerous activities or
objects are safe.
Table 6: This table shows the percentage of different unsafe categories of MAGPIE-Air and MAGPIEPro tagged by Llama-Guard-2 Team (2024) model.
Dataset Safe ViolentCrimes Non-ViolentCrimes Sex-RelatedCrimes Child SexualExploitation SpecializedAdvice Privacy IntellectualProperty IndiscriminateWeapons Hate Self-HarmSuicide & ContentSexual Others
MAGPIE-Air 99.128% 0.001% 0.073% 0.003% 0.000% 0.636% 0.022% 0.026% 0.038% 0.001% 0.002% 0.009% 0.062%
MAGPIE-Pro 99.347% 0.001% 0.049% 0.002% 0.000% 0.446% 0.015% 0.074% 0.014% 0.001% 0.004% 0.011% 0.036%
D.3 ABLATION ANALYSIS ON GENERATION CONFIGURATIONS
(a) Average Quality Scores of
MAGPIE-Air
(b) Average Difficulty Scores of
MAGPIE-Air
(c) Average Minimum Neighbor
Distances of MAGPIE-Air
Figure 11: This figure illustrates the impact of varying decoding parameters on the quality, difficulty,
and diversity of generated instructions. We observe that while higher temperature and top-p values
may decrease the overall quality, they tend to increase both the difficulty and diversity of the
instructions.
**Ablation Analysis on Decoding Parameters.** We conduct an ablation analysis on the decoding
parameters used in generating instruction with MAGPIE. Specifically, we use three different temperatures (i.e., 1, 1 _._ 1, and 1 _._ 2) and top-p values (i.e., 1, 0 _._ 995, and 0 _._ 99) during Step 1 of MAGPIE.
Preprint
We use three metrics, **Average Quality Score**, **Average Difficulty Score** and **Average Minimum**
**Neighbor Distance** to characterize the quality, difficulty, and diversity of instructions using different
decoding parameters. The Average Quality Score is calculated by averaging the ratings of all data
within a specific temperature-top-p pair, on a scale from 1 (‘very poor’) to 5 (‘excellent’). Similarly,
the Average Difficulty Score is rated on a scale from 1 (‘very easy’) to 5 (‘very hard’). The Average
Minimum Neighbor Distance is calculated by averaging the minimum neighbor distances, as defined
in Section 3, for all data generated using the same decoding parameters.
The findings are summarized in Figure 11. We observe that
higher temperature and top-p values may slightly decrease the
overall quality of instructions, while simultaneously increasing
the difficulty and remarkably enhancing the diversity of the instructions generated. The selection of these hyper-parameters
should be tailored to the user’s specific requirements, balancing the trade-offs between quality, difficulty, and diversity.
**Ablation Analysis on the System Prompt.** Figure 12 compares the use of system prompt compared with not using it in
Step 1 of MAGPIE. Since the Llama-3 model does not have
an official system prompt, we use the default system prompt
from Vicuna (Chiang et al., 2023): A chat between a curious
user and an artificial intelligence assistant. The assistant gives
helpful, detailed, and polite answers to the user’s questions.
We observe that using a system prompt generally results in a
decrease in the overall quality of instructions, and the instructions are easier. Consequently, we recommend not appending
system prompts in default settings.
D.4 IMPACT OF ANNOTATING MODELS
We note that LLMs may occasionally favor its own response
(Deutsch et al., 2022). In what follows, we conduct experiments to evaluate the impact of annotating models when labeling quality and difficulty of the MAGPIE-Air dataset. We used
the Qwen-2-7B-Instruct model (outside the Llama-3 family) to
annotate the quality and difficulty of our MAGPIE-Air dataset.
The statistics are summarized in Figure 13.
Our findings show that even when evaluated by Qwen-2-7BInstruct, the MAGPIE-Air dataset maintains high quality and
difficulty, which is even higher than those originally annotated by Llama-3-8B-Instruct. This suggests that our dataset’s
quality is robust across different annotators.
E DETAILED EXPERIMENTAL SETUPS
E.1 EXPERIMENTAL SETUPS
FOR GENERATING MAGPIE-AIR AND MAGPIE-PRO
(a) Comparison of Input Quality w/wo System Prompts
(b) Comparison of Input Difficulty w/wo System Prompts
Figure 12: This figure compares the
input quality and difficulty with and
without system prompts.
(a) Statistics on Input Quality Using Different Annotators
(b) Statistics on Input Difficulty Using Different Annotators
Figure 13: This figure compares
the impact of different annotators
on evaluating the instruction quality and difficulty.
As detailed in Appendix D.3, varying decoding parameters in Step 1 can significantly influence
the quality, difficulty, and diversity of the generated instructions. To optimize the trade-offs among
these attributes, we employ diverse decoding parameters for the generation of MAGPIE-Air and
MAGPIE-Pro. Table 7 presents the configurations of MAGPIE-Air and MAGPIE-Pro, showcasing how
diverse decoding parameters shape each dataset.
We employ greedy decoding to generate responses in Step 2 for MAGPIE-Air and MAGPIE-Pro. The
intuition is that the word with the highest probability is more likely to originate from the model’s
training dataset.
Preprint
Table 7: This table demonstrates the configurations of generating instructions of MAGPIE-Air and
MAGPIE-Pro datasets with varying decoding parameters.


Temperature Top-p #Convs
1.0 1.00 300K
MAGPIE-Air
MAGPIE-Pro
1.0 0.995 300K
1.0 0.990 300K
1.1 1.00 300K
1.1 0.995 300K
1.1 0.990 300K
1.2 1.00 300K
1.2 0.995 300K
1.2 0.990 300K
1.25 1.00 100K
1.25 0.995 100K
1.25 0.990 100K
1.0 1.00 300K
1.1 0.995 300K
1M
1.2 0.995 300K
1.25 0.990 100K
3M
E.2 EXPERIMENTAL SETUPS FOR INSTRUCTION TUNING AND PREFERENCE TUNING
**Supervised Fine-Tuning Hyper-parameters.** Table 8 demonstrates the detailed supervised finetuning hyper-parameters. These experiments were conducted using Axolotl [5] .
Table 8: This table shows the hyper-parameters for supervised fine-tuning.


Learning Rate 2 _×_ 10 _[−]_ [5]
Number of Epochs 2
Number of Devices 4
Per-device Batch Size 1
Gradient Accumulation Steps 8
Effective Batch Size 32
Optimizer Adamw with _βs_ = (0 _._ 9 _,_ 0 _._ 999) and _ϵ_ = 10 _[−]_ [8]
Learning Rate Scheduler cosine
Warmup Steps 100
Max Sequence Length 8192
**Preference Tuning Hyper-parameters.** Table 9 demonstrates the detailed DPO hyper-parameters
for aligning Llama-3-8B using MAGPIE-Air-DPO and MAGPIE-Pro-DPO. These experiments were
conducted using Alignment Handbook [6] .
**Decoding parameters for evaluation benchmarks.** For Arena-Hard (Li et al., 2024b) and WildBench (Lin et al., 2024), we follow its default setting and use greedy decoding for all settings. For
AlpacaEval 2 (Li et al., 2023b) which allows the model provider to specify decoding parameters,
we also employ greedy decoding in all experiments with a slightly increased repetition penalty
( _RP_ = 1 _._ 2) to mitigate the potential repetitive outputs during the generation.
[5https://github.com/OpenAccess-AI-Collective/axolotl](https://github.com/OpenAccess-AI-Collective/axolotl)
[6https://github.com/huggingface/alignment-handbook](https://github.com/huggingface/alignment-handbook)
Preprint
Table 9: This table shows the hyper-parameters for direct preference optimization.


Learning Rate 5 _×_ 10 _[−]_ [7]
Number of Epochs 1
Number of Devices 4
Per-device Batch Size 2
Gradient Accumulation Steps 16
Effective Batch Size 128
Optimizer Adamw with _βs_ = (0 _._ 9 _,_ 0 _._ 999) and _ϵ_ = 10 _[−]_ [8]
Learning Rate Scheduler cosine
Warmup Ratio 10%
F ADDITIONAL EXPERIMENTAL RESULTS
F.1 PERFORMANCE OF MAGPIE-MT
Table 10 compares the performance of MAGPIE-Air-MT and MAGPIE-Pro-MT with their respective single-turn counterparts. We observe that the multi-turn datasets have enhanced performance,
particularly in the Arena-Hard benchmark.
Table 10: This table compares the performance of the multi-turn versions, MAGPIE-Air-MT and
MAGPIE-Pro-MT, with their single-turn counterparts. All models are instruction-tuned on the Llama8B base models.


GPT-4-Turbo (1106) Llama-3-8B-Instruct
LC (%) WR (%) SD LC (%) WR (%) SD WR (%)
Single-Turn 22.66 23.99 1.24 49.27 50.80 1.44 14.9
MAGPIE-Air
MT **22.98** **24.02** 1.27 **49.63** **51.42** 1.40 **15.5**
Single-Turn **25.15** **26.50** 1.30 50.52 52.98 1.43 18.9
MAGPIE-Pro
MT 24.21 25.19 1.28 **52.92** **54.80** 1.41 **20.4**
F.2 COMPARE MAGPIE AND SELF-INSTRUCT USING LLAMA-3-8B-INSTRUCT
To compare the performance of MAGPIE and other synthetic dataset generation methods using the
same model, we follow the official Self-Instruct (Wang et al., 2023) setup and generate a 100K
supervised fine-tuning dataset using Llama-3-8B-Instruct. For a fair comparison, we select the first
100K data samples from the MAGPIE-Air dataset generated by Llama-3-8B-Instruct. The performance
of models fine-tuned with these two datasets is shown in the table 11.
We observe a significant performance gap between models fine-tuned with datasets generated by SelfInstruct and our MAGPIE. Our analysis revealed that the instruction format in Self-Instruct-generated
Table 11: This table compares the performance of models fine-tuned using 100K instruction-following
datasets generated by Self-Instruct and MAGPIE. All models are supervised-fine-tuned on the
Llama-8B base models. We observe that MAGPIE significantly outperforms Self-Instruct across all
benchmarks.
**Dataset** #Convs


GPT-4-Turbo (1106) Llama-3-8B-Instruct
LC (%) WR (%) SD LC (%) WR (%) SD WR (%)
MAGPIE-Air-100K 100K 20.17 21.33 1.21 46.82 48.76 1.44 15.7
Self-Instruct (Llama-3) 100K 7.21 5.18 0.7 17.86 12.73 1.05 4.0
Preprint
datasets is predominantly constrained by the patterns defined in the seed instructions, resulting in
a lack of diversity. This comparison indicates the novelty of our MAGPIE in generating diverse
high-quality instructions without any seed questions.
F.3 ABLATION ANALYSIS ON DATA QUANTITY AND QUALITY
In what follows, we compare within the family of datasets generated by MAGPIE in Table 12. These
datasets differ in size, deployment of filtering, and models used to generate data. We observe
that as the dataset’s size increases, the fine-tuned model’s performance improves, indicating that
data quantity plays a critical role in enhancing instruction-following capabilities. Furthermore, the
model fine-tuned with MAGPIE-Pro-300K-Filtered outperforms those fine-tuned with the same or
even higher amounts of raw data. This demonstrates the effectiveness of our filtering technique,
and underscores the importance of data quality. Finally, we observe that the models fine-tuned
with MAGPIE-Pro consistently outperform those fine-tuned with MAGPIE-Air. The reason is that
MAGPIE-Pro is generated by the more capable model, i.e., Llama-3-70B-Instruct.
Table 12: This table compares MAGPIE datasets within its family that differ in size, deployment of
filtering, and models used to generate data. All models are supervised-fine-tuned on the Llama-8B
base models.


GPT-4-Turbo (1106) Llama-3-8B-Instruct
LC (%) WR (%) SD LC (%) WR (%) SD WR(%)
300K-Raw 300K 21.99 21.65 1.21 48.63 48.06 1.42 15.8
MAGPIE-Air 3M-Raw 3M 22.96 21.09 1.20 50.57 48.40 1.42 16.1
300K-Filtered 300K 22.66 23.99 1.24 49.27 50.8 1.44 14.9
MAGPIE-Pro
300K-Raw 300K 21.65 22.19 1.2 49.65 50.84 1.42 15.9
1M-Raw 1M 24.16 23.93 1.25 49.97 50.34 1.43 16.7
100K-Filtered 100K 20.47 24.52 1.25 47.92 52.75 1.43 17.2
200K-Filtered 200K 22.11 26.02 1.26 51.17 **56.76** 1.41 15.9
300K-Filtered 300K **25.08** **29.47** 1.35 **52.12** 53.43 1.44 **18.9**
MAGPIE-Air + MAGPIE-Pro 4M-Raw 4M 24.45 24.08 1.26 51.96 52.08 1.42 15.5
F.4 ABLATION ANALYSIS ON FILTER DESIGNS
We conduct an ablation analysis on various filter designs within MAGPIE-Pro to assess their impact
on the performance of supervised fine-tuned models. The results are presented in Table 13. We
observe that different filtering strategies yield optimal performance on different benchmarks, and no
single filter consistently achieves the best performance across all benchmarks. Therefore, determining
how to select instructional data to enhance the performance in supervised fine-tuning is an interesting
topic for future research.
Table 13: This table compares the performance of different filter designs within MAGPIE-Pro. All
models are supervised-fine-tuned on the Llama-8B base models.


GPT-4-Turbo (1106) Llama-3-8B-Instruct
LC (%) WR (%) SD LC (%) WR (%) SD WR (%)
MAGPIE-Pro
Filter 25.08 **29.47** 1.35 52.12 53.43 1.44 **18.9**
Filter 2 **25.15** 26.50 1.30 50.52 52.98 1.43 **18.9**
Filter 3 23.90 25.21 1.25 51.45 53.64 1.41 18.3
Filter 4 24.20 25.33 1.27 **52.43** 54.34 1.43 17.9
Filter 5 24.85 25.12 1.26 52.12 53.43 1.44 18.4
Filter 6 23.20 28.43 1.26 51.34 **57.29** 1.41 17.9
F.5 ABLATION ANALYSIS ON RESPONSE GENERATOR
To investigate the impact of the response generator on the supervised fine-tuning performance using
MAGPIE, we conduct an ablation study by replacing the response generator with Qwen-2-7B-Instruct
Preprint
(Yang et al., 2024) within MAGPIE-Air-300K-Filtered. We note that the performance of Qwen-2-7BInstruct is comparable to, or slightly weaker than, Llama-3-8B-Instruct. The results are summarized
in Table 14.
We observe that although there is a slight performance degradation, the model fine-tuned using
Qwen-2-7B-Instruct as the response generator still outperforms all baselines, including those using
GPT-4 as the response generator. These findings indicate two key points: (1) The success of MAGPIE
depends little on the specific response generator used, and (2) the instructions generated by MAGPIE
are of high quality and diversity.
Table 14: This table compares the impact of different response generators on the model performance.
All models are supervised-fine-tuned on the Llama-8B base models.


GPT-4-Turbo (1106) Llama-3-8B-Instruct
LC (%) WR (%) SD LC (%) WR (%) SD WR (%)
Llama-3-8B-Instruct 22.66 23.99 1.24 49.27 50.80 1.44 14.9
Qwen2-7B-Instruct 15.01 15.60 1.05 41.09 42.07 1.47 13.7
F.6 TRUSTWORTHINESS OF MAGPIE-ALIGNED MODELS
In what follows, we conduct more experiments to compare MAGPIE model and Llama-3-8B-Instruct
on the TrustLLM benchmark (Huang et al., 2024). The results for safety, fairness, ethics, privacy, and
robustness are summarized in Table 15.
We observe that our supervised-fine-tuned model slightly underperforms Llama-3-8B-Instruct in
terms of safety and fairness. However, it outperforms the official instruct model on ethics, privacy, and
robustness. Considering that our fine-tuned model uses much fewer data samples (300K compared to
over 10M), these results again highlight the high quality of data generated by MAGPIE.
Table 15: This table compares the performance of model supervised-fine-tuned using MAGPIE-Pro300K-Filtered and the official Llama-3-8B-Instruct on the TrustLLM benchmark (Huang et al., 2024).
**TrustLLM** Evaluation/Dataset Llama-3-8B-Instruct MAGPIE-Pro-300K-Filtered
Jailbreak (RtA _↑_ ) **0.93** 0.80
Safety Misuse (RtA _↑_ ) **0.85** 0.80
Exaggerated Safety (RtA _↓_ ) **0.54** 0.52
Fairness
Ethics
Stereotype Recognition (Acc _↑_ ) **0.49** 0.40
Stereotype Query Test (RtA _↑_ ) **1.00** 0.99
Disparagement Sex (p-value _↑_ ) **0.99** **0.99**
Disparagement Race (p-value _↑_ ) **0.55** 0.47
Social Chemistry 101 (Acc _↑_ ) **0.94** 0.63
ETHICS (Acc _↑_ ) 0.65 **0.69**
MoralChoice (Acc _↑_ ) **0.97** 0.95
MoralChoice (RtA _↑_ ) 0.97 **0.98**
Privacy Awareness-Normal (RtA _↑_ ) 0.33 **0.71**
Privacy Privacy Awareness-Augmented (RtA _↑_ ) **1.00** 0.98
Privacy Leakage (RtA _↑_ ) 0.66 **0.87**
AdvGlue (RobustScore _↑_ ) 0.42 **0.58**
Robustness OOD Detection (RtA _↑_ ) **0.37** 0.26
OOD Generalization (F1-Score _↑_ ) 0.83 **0.84**
G PROMPT TEMPLATES
G.1 PROMPT TEMPLATES FOR MAGPIE EXTENSION
This section presents the prompt template used to generate MAGPIE-MT and control instruction tasks,
as detailed in Figure 14 and Figure 15, respectively.
Preprint
Figure 14: Prompt for generating MAGPIE-MT. We take Llama-3-8B-Instruct as an example. The
placeholder {instruction} and {response} are from the first turn.
Figure 15: Prompts for controlling instruction generation tasks. These examples illustrate how to
guide Llama-3-8B-Instruct in generating instructions for specific domains: mathematics, coding,
translation, and multilingual tasks. To adapt this approach for different instruction tasks, replace the
System Prompt placeholder in the System Prompt Template with the appropriate domain-specific
prompt.
Preprint
Figure 16: Prompt for generating task categories
G.2 PROMPT TEMPLATES FOR EVALUATION
Here, we present the prompt template employed to generate task categories, quality, and difficulty, as
detailed in Figure 16, Figure 17, and Figure 18, respectively. The placeholder input represents the
instructions to be evaluated.
Preprint
Figure 17: Prompt for generating quality of instructions
Figure 18: Prompt for generating difficulty of instructions
Preprint
H MAGPIE EXAMPLES
In this section, we demonstrate several representative instructions generated by MAGPIE with different
task categories. The following instructions are generated by the Llama-3-70B-Instruct model.
Preprint
MAGPIE can also generate domain-specific instructions using models that are tailored to particular
fields, as mentioned in Section 2.2. The following instructions are generated by DeepSeek-Coder-V2
(Zhu et al., 2024) and Qwen2-Math-7B-Instruct (Yang et al., 2024), respectively.
We note that MAGPIE’s capabilities extend beyond generating English datasets to producing diverse
multilingual datasets. The following instructions are generated by the Qwen2-72B-Instruct model.
Preprint


Department of Electrical and Computer Engineering
University of Wisconsin-Madison
```
     daiwei.chen@wisc.edu
```

PAL: Pluralistic Alignment Framework for Learning from Heterogeneous
  Preferences

Daiwei Chen
Yi Chen
Aniket Rege
Ramya Korlakai Vinayak


Department of Electrical and Computer Engineering
University of Wisconsin-Madison
```
      yi.chen@wisc.edu
```


Department of Computer Sciences
University of Wisconsin-Madison
```
  aniketr@cs.wisc.edu
```


Department of Electrical and Computer Engineering
University of Wisconsin-Madison
```
       ramya@ece.wisc.edu
```


Large foundation models pretrained on raw web-scale data are not readily deployable without
additional step of extensive _alignment_ to human preferences [1]. Such alignment is typically done
by collecting large amounts of pairwise comparisons from humans (“Do you prefer output A or
B?”) and learning a reward model or a policy with the Bradley-Terry-Luce (BTL) model [2] as a
proxy for a human’s underlying implicit preferences. These methods generally suffer from assuming
a universal preference shared by all humans, which lacks the flexibility of adapting to plurality
of opinions and preferences [3]. In this work, we propose PAL, a framework to model human
preference complementary to existing pretraining strategies, which incorporates plurality from the
ground up. We propose using the _ideal point model_ [4] as a lens to view alignment using preference
comparisons. Together with our novel reformulation and using mixture modeling, our framework
captures the _plurality_ of population preferences while simultaneously learning a common preference
latent space across different preferences, which can few-shot generalize to new, unseen users. Our
approach enables us to use the penultimate-layer representation of large foundation models and
simple MLP layers to learn reward functions that are on-par with the existing large state-of-theart reward models, thereby enhancing efficiency of reward modeling significantly. We show that
PAL achieves competitive reward model accuracy compared to strong baselines on 1) Language
models with Summary dataset [5] ; 2) Image Generative models with Pick-a-Pic dataset [6] ; 3)
A new semisynthetic heterogeneous dataset generated using Anthropic Personas [7]. Finally, our
experiments also highlight the shortcoming of current preference datasets that are created using rigid
rubrics which wash away heterogeneity, and call for more nuanced data collection approaches.
_**Keywords**_ Alignment _·_ Preference Learning _·_ Plurality


Large pre-trained “foundation” models [8], such as large language models (LLMs) for language generation [9, 10, 11,
12, 13, 14, 15] and text-to-image (TTI [16]) models for image generation [17, 18, 19, 20, 21, 22, 23], are trained on
massive amounts of data, including data from the internet. While such models learn useful representations for general
language or vision tasks, they are not readily deployable out-of-the-box to be used in the real world. Modern machine
learning (ML) systems built on large foundation models go through rigorous fine-tuning or _aligning_ towards human
preferences to make them amenable to real-world usage. This is usually achieved through supervised fine-tuning (SFT)
with direct human input on what the desired outputs should look like for a given context and then followed by alignment
with large amounts of human preference feedback usually in the form of pairwise comparison of two outputs to a given
Pluralistic Alignment Framework for Learning from Heterogeneous Preferences A PREPRINT
Figure 1: Illustration of PAL framework for learning from diverse preferences (Section 3). For any user _i_, the probability
of preferring **x** _l_ to **x** _r_ for the context **x** _c_ is given by a reward model _rθ_ [(] _[i]_ [)] which is modeled as a uses a mixture modeling
approach to capture diverse user preferences – each user’s preference is modeled as a convex combination of _K_
prototypes. Reward function formulated using PAL framework can be used flexibly, e.g, with fixed preference points
(Model A), with preference points that are functions of the context/prompt **x** _c_ (Model B).
input context [1]. This is usually achieved either by (i) fine-tuning the SFT model with explicitly learned reward as done
in reinforcement learning with human feedback (RLHF) methods such as proximal policy optimization (PPO) [24] or
implicitly with methods such as direct preference optimization (DPO) [25], or (ii) inference-time policy adaptation [26]
without fine-tuning the original large policy model.
While aligning ML/AI models to human preferences, it is imperative to ask ourselves _whose preferences are we_
_aligning the ML/AI models to?_ [27] The status quo of the alignment phase is to assume a homogeneous preference
shared by all humans and attempt to learn a reward model to learn this preference with the Bradley-Terry-Luce
(BTL) model [2] of paired preferences. We challenge these notions in an attempt to capture diverse, heterogeneous
preferences [3, 28, 29, 30]. The importance of capturing the plurality of preferences and values among humans has also
been highlighted recently by Sorensen et al.. However, the methods suggested therein and other recent works that look
at learning multiple rewards as a top-down approach where the system designer decides the number and axes that one
should care about [1, 27, 32, 33, 34], e.g., helpfulness vs. harmfulness [35, 36, 37, 38]. In reality, human preference is
more complex than the designer-specified axes [28], which leads us to propose the following goal.
**Our Contributions.** Towards this goal, we make the following contributions,
1. **Novel Reformulation:** We reframe the problem of alignment from human preferences by introducing the lens
of ideal point model [4] and metric learning [39]. This re-framing enables leveraging modeling and algorithmic
techniques from a richer set of toolboxes (Sections 2 and 3.1).
2. **New Framework for Pluralistic Alignment:** We propose PAL, a general framework for pluralistic alignment using
diverse human preferences from the ground up. Our framework uses a mixture modeling approach combined with
the ideal point and metric learning reframing that is interpretable (Figure 1, Section 3). It can work with the output
of any foundation model and learn reward function to generalize to a population of diverse people. Our framework is
versatile to be applied to a wide variety of application domains.
3. **Empirical Validation on Benchmark Datasets** : We evaluate our framework through extensive experiments on
both synthetic and real datasets (Sections 4.1 and 4.3). Our experiments highlight the ability and versatility of the
PAL framework to learn from diverse preferences when heterogeneity exists in both language and vision data. Our
experiments also reveal that even when the datasets are collected in a homogeneous way, PAL can learn reward
functions using very simple models, e.g., 2-layer MLP, on top of foundation models and are competitive with
state-of-the-art (SoTA) models that fine-tune large foundation models. We discuss broader impacts, limitations and
areas for future work in Section 6.
Pluralistic Alignment Framework for Learning from Heterogeneous Preferences A PREPRINT


We begin with a brief discussion of the BTL model and how it is currently used in reward learning from pairwise
preference comparisons, followed by motivating the ideal point model.
**Bradley-Terry-Luce (BTL) model** [2] is a parametric model for ranking. Given _m_ items or alternatives, the assumption
is that there is a universal ranking: _σ_ (1) _≻_ _σ_ (2) _≻· · · ≻_ _σ_ ( _m_ ) which are a reflection of the _unknown_ true scores or
weights associated with each of these items _s_ _[⋆]_ _σ_ (1) _[> s]_ _σ_ _[⋆]_ (2) _[>][ · · ·][ s]_ _σ_ _[⋆]_ ( _m_ ) [, where] _[ σ]_ [(] _[.]_ [)][ denotes permutation and the scores]
_s_ _[⋆]_ are positive real numbers. Then, the probability that “ _i beats j_ " when comparing them, denoted by _i ≻_ _j_ is given by,
Pr( _i ≻_ _j_ ) = _s_ _[⋆]_ _i_ = exp ( _ri_ ) (1)
_s_ _[⋆]_ _i_ [+] _[ s]_ _j_ _[⋆]_ exp ( _ri_ ) + exp ( _rj_ ) _[,]_ [ where the variables] _[ r]_ [ re-parameterize] _[ s >]_ [ 0] _[.]_
**Notation:** We set up some notation for further discussion. Let _D_ denote the dimension of the representation space of
the foundation models. Let **x** _c ∈_ R _[D]_ denote the representation of the prompt or the context. Let **x** _l ∈_ R _[D]_ and **x** _r ∈_ R _[D]_
denote the embeddings of two items where the subscripts denote _left_ and _right_ respectively. Note that while we take the
dimensions of the representation space for the prompt and the output to be the same, it need not be the same in general.
In the literature on alignment with human feedback, the scores re-parametrized with _reward_, denoted here by _r_, are
modeled using a neural network denoted by _rθ_ . More concretely, given a context or prompt **x** _c_, the probability that
output **x** _l_ is preferred to output **x** _r_ under the BTL model is given by,
exp ( _rθ_ ( **x** _l_ ; **x** _c_ )) 1
Pr( **x** _l ≻_ **x** _r|_ **x** _c_ ) = (2)
exp ( _rθ_ ( **x** _l_ ; **x** _c_ )) + exp ( _rθ_ ( **x** _r_ ; **x** _c_ )) [=] 1 + exp ( _rθ_ ( **x** _r_ ; **x** _c_ ) _−_ _rθ_ ( **x** _l_ ; **x** _c_ )) _[.]_
The goal then is to _learn_ this _reward function rθ_ that takes the output **x** _∈_ R _[D]_ for a given context **x** _c ∈_ R _[D]_, denoted by
( **x** ; **x** _c_ ), as input and map it to a real-valued _reward score_ to approximate human preference. This learning of _rθ_ is done
using lots of pairwise comparison data obtained by querying humans. Such a learned reward function can be used to
_align_ the model [1, 40, 41], score the generations during inference time to output more aligned answers [42] and to rank
the generations of multiple models [43, 44]. Recent work from Rafailov et al. bypasses the status quo two-stage reward
learning + RL pipeline and directly finetune on pairwise preferences, but still implicitly assumes the BTL model for
ranking.
While most alignment literature focuses on the BTL modeling approach, we want to draw attention to the _ideal point_
_model_ [4] for preference learning.
**Ideal point model** was proposed by Coombs for human preference modeling in the psychology literature. The key
idea behind this model is to exploit the geometry of the problem, assuming there exists a meaningful representation
space for the items/alternates being compared. Let _X ⊆_ R _[D]_ denote the domain of feature space of the concepts
(items, objects, images, choices, etc.) with a distance associated with it. Preference learning based on ideal point
model [4, 45, 46, 47, 48, 49, 50] assumes that there is an _unknown_ ideal preference point **a** _∈X_ that represents the
reference point people use for their preference judgments based on distances. So, when asked “Do you prefer _i_ or _j_ ?”,
they respond with _i_ as their preference if dist( **x** _i,_ **a** ) _<_ dist( **x** _j,_ **a** ) and vice versa, where **x** _i,_ **x** _j ∈X_ are the feature
representations of _i_ and _j_ respectively. That is, items that are closer to the user’s ideal preference points are preferred
by the user over those that are farther away. The goal of preference learning then is to use the responses for pairwise
comparison queries from people and learn the preference point **a** . Once we learn **a**, we can predict the choices people
make between new unseen pairs. More formally, in general, the probability that _i_ beats _j_ in preference for user _a_ is
given by,
Pr( _i ≻_ _j_ ) _∝_ _h_ (dist [2] ( **x** _j,_ **a** ) _−_ dist [2] ( **x** _i,_ **a** )) _,_ (3)
where _h_ is a link function [51] which can be any monotonic function such that Pr( _i ≻_ _j_ ) _>_ 1 _/_ 2 when dist [2] ( **x** _j,_ **a** ) _−_
dist [2] ( **x** _i,_ **a** ) _>_ 0 and Pr( _i ≻_ _j_ ) = 1 _/_ 2 when dist [2] ( **x** _j,_ **a** ) = dist [2] ( **x** _i,_ **a** ). Essentially, the idea here is that the larger the
difference in distance between the alternates to the ideal point, the easier it is to choose between them, and hence the
answer is less noisy. In contrast, if the difference of distance is close to zero, the alternates seem to be equally good or
bad to the user and therefore the probability of _i ≻_ _j_ will be close to random.


In this section, we begin by describing how to view existing approaches that use the BTL model for alignment through
the lens of the ideal point model, and then introduce our framework for pluralistic alignment.
Pluralistic Alignment Framework for Learning from Heterogeneous Preferences A PREPRINT

3 Framework for Pluralist Alignment ( PAL )


The assumption in the ideal point model (Section 2) that the items being compared have representations in a vector space
is mild one, especially while working with foundation models. However, assuming that the Euclidean distance or a
known distance function in the representation space of these foundation models to be the _correct_ notion of similarity and
dissimilarity for human judgments is a strong one. We re-formulate the goal of alignment, i.e. learning a reward function,
to learning a (potentially non-linear) transformation of the representation output by the foundation model where a
known distance function, e.g., Euclidean distance or cosine similarity, is a good approximation (in the transformed
space) to capture human judgments of similarity and dissimilarity.
Looking at the current alignment approaches using the BTL model through the lens of the ideal point model, we can
re-interpret the Equation 2,
Pr( **x** _l ≻_ **x** _r|_ **x** _c_ ) = (4)
1 + exp ( _rθ_ ( **x** _r_ ; **x** _c_ ) _−_ _rθ_ ( **x** _l_ ; **x** _c_ )) _[.]_
as an ideal point model where the difference of rewards is a proxy for the difference of distances [1] and the link function
being the Sigmoid or logistic function.
By relaxing the requirement of the Sigmoid link function used by the BTL model and the known distance function by
the ideal point model, we propose to view alignment from human preferences as learning a _reward_ function that can
generalize to the following: [2]
Pr( **x** _l ≻_ **x** _r|_ **x** _c_ ) = _h_ ( _rθ_ ( **x** _r_ ; **x** _c_ ) _−_ _rθ_ ( **x** _l_ ; **x** _c_ )) _,_ (5)
where _h_ any monotonic link function appropriately normalized to obtain probabilities such that Pr( **x** _l ≻_ **x** _r|_ **x** _c_ ) _>_ 1 _/_ 2
when _rθ_ ( **x** _r_ ; **x** _c_ ) _−_ _rθ_ ( **x** _l_ ; **x** _c_ ) _>_ 0, i.e., there is a clear winner between items, and Pr( **x** _l ≻_ **x** _r|_ **x** _c_ ) = 1 _/_ 2 when
_rθ_ ( **x** _r_ ; **x** _c_ ) = _rθ_ ( **x** _l_ ; **x** _c_ ), i.e., the winner is random.
We instantiate the reward function in the following ways:
1. With _unknown_ but fixed ideal point, _unknown_ representation space for _jointly_ representing the prompt input **x** _c_ and
the corresponding output **x** from the foundation model and _Euclidean distance_, we model the reward function as,
_rθ_ ( **x** _,_ **x** _c_ ) = _||f_ ( **x** ; **x** _c_ ) _−_ _f_ ( **a** ) _||_ [2], where mapping _f_ : R [2] _[D]_ _→_ R _[d]_ and ideal point **a** _∈_ R [2] _[D]_ are _unknown_ and learned
from pairwise comparison queries. This corresponds to the following pairwise ranking model,
Pr( **x** _l ≻_ **x** _r|_ **x** _c_ ) = _h_ ( _||f_ ( **x** _r_ ; **x** _c_ ) _−_ _f_ ( **a** ) _||_ [2] 2 _[−||][f]_ [(] **[x]** _[l]_ [;] **[ x]** _[c]_ [)] _[ −]_ _[f]_ [(] **[a]** [)] _[||]_ 2 [2][)] _[.]_ (6)
2. The user ideal point is an _unknown_ function of the prompt **x** _c_ and distance is the angle between the ideal point
conditioned on the prompt and the _unknown_ representation space for the output **x** from the foundation model,
_rθ_ ( **x** _,_ **x** _c_ ) = _⟨f_ ( **x** ) _, z_ ( **x** _c_ ) _⟩_, where the mappings _f_ and _z_ map R _[D]_ _→_ R _[d]_ and are _unknown_ and are learned from
pairwise comparisons. Here we assume that the range spaces of _f_ and _z_ are normalized to use the angle as the
distance function. This corresponds to the following pairwise ranking model,
Pr( **x** _l ≻_ **x** _r|_ **x** _c_ ) = _h_ ( _⟨f_ ( **x** _r_ ) _, z_ ( **x** _c_ ) _⟩_ ) _−⟨f_ ( **x** _l_ ) _, z_ ( **x** _c_ ) _⟩_ ) _._ (7)

3.1 Viewing alignment through the lens of ideal point model and metric learning


When learning a reward function from diverse preferences, there are two types of generalization to consider. (1)
Generalization for _unseen pairs for seen users_, i.e., predicting well for new pairs for the people for whom the weights
have already been learned from the training data. We call this _seen accuracy_ . (2) Generalization is for _unseen users_, i.e.,
predicting well for people whose data was not part of the training data at all. For such new users, some part of their new
data will be used to localize them within the learned model by only learning the weights for the new user by keeping
the shared mappings and prototypes fixed. We call this _unseen accuracy_ . We also note that we can use the weighted
combination of the prototypes, i.e., an average of all the seen users, as the _zero-shot_ ideal point for new users. However,
we emphasize that it is important for reward functions to generalize to _unseen_ users and our framework provides a
natural way to localize the new user.

3.3.1 Generalization over _seen_ users versus _unseen_ users


-        - _N_
Given the dataset _D_ = _{_ ( **x** _l,_ **x** _r_ ; **x** _c_ ) [(] _j_ _[i]_ _i_ [)] _[}]_ _j_ _[m]_ _i_ =1 _[i]_ _i_ =1 [, loss function] _[ ℓ]_ [and model class for] _[ f][θ]_ [, the learning algorithm for]
**model A** starts by randomly initializing the prototypes **P** = [ **p** 1 _, ...,_ **p** _K_ ], **p** _k ∈_ R _[D]_, user weights **W** = [ **w** [(1)] _, ...,_ **w** [(] _[N]_ [)] ],
where **w** [(] _[i]_ [)] _∈_ ∆ _[K]_ . Then, in each iteration until convergence criteria, the following steps are repeated,
          -
- **Sample** a random mini-batch ( **x** _l,_ **x** _r_ ; **x** _c_ ) _j_ [(] _[i]_ [)] of comparison data from _D_ .
- **Compute user ideal points: a** [(] _[i]_ [)] = **P** _·_ **w** [(1)] .
- **Compute distances** : _d_ [(] _l,j_ _[i]_ [)] [=] _[ ||][f][θ]_ [ (] **[x]** _[l]_ [;] **[ x]** _[c]_ [)] _[ −]_ _[f][θ]_ [(] **[a]** [(] _[i]_ [)][)] _[||]_ 2 [2][,] _[ d]_ [(] _r,j_ _[i]_ [)] [=] _[ ||][f][θ]_ [ (] **[x]** _[r]_ [;] **[ x]** _[c]_ [)] _[ −]_ _[f][θ]_ [(] **[a]** [(] _[i]_ [)][)] _[||]_ 2 [2][.]
- **Loss for each comparison** _j_ **for user** _i_ : _ℓ_ [(] _j_ _[i]_ [)][(] **[x]** _[l][,]_ **[ x]** _[r]_ [;] **[ x]** _[c]_ [) =] _[ ℓ]_ [(] _[d]_ _r,j_ [(] _[i]_ [)] _[−]_ _[d]_ _l,j_ [(] _[i]_ [)][)][.]
- **Update Step: argmax** _θ,_ **P** _,{_ **w** ( _i_ ) _∈_ ∆ _K_ _}Ni_ =1 - _i,j_ _[ℓ]_ _j_ [(] _[i]_ [)][(] **[x]** _[l][,]_ **[ x]** _[r]_ [;] **[ x]** _[c]_ [)][.]
The above steps describe updating the learning algorithm for model A. Similarly, the dataset _D_ =
- - _N_
_{_ ( **x** _l,_ **x** _r_ ; **x** _c_ ) [(] _j_ _[i]_ _i_ [)] _[}]_ _j_ _[m]_ _i_ =1 _[i]_ _i_ =1 [, loss function] _[ ℓ]_ [and model class for] _[ f][θ]_ [, the learning algorithm for] **[ model B]** [ starts by]
Pluralistic Alignment Framework for Learning from Heterogeneous Preferences A PREPRINT
randomly initializing the prototype functions _{g_ 1 _, ..., gK}_, where each _gk_ is a function that maps **x** _c ∈_ R _[D]_ to R _[d]_, user
weights **W** = [ **w** [(1)] _, ...,_ **w** [(] _[N]_ [)] ], where **w** [(] _[i]_ [)] _∈_ ∆ _[K]_ . Then, in each iteration until convergence criteria, the following
steps are repeated,
          -
- **Sample** a random mini-batch ( **x** _l,_ **x** _r_ ; **x** _c_ ) _j_ [(] _[i]_ [)] of comparison data from _D_ .
- **Compute user ideal points: a** [(] _[i]_ [)] = [ _g_ 1( **x** _c_ ) _, ..., gK_ ( **x** _c_ )] _·_ **w** [(1)] .
- **Compute distances** : _d_ [(] _l,j_ _[i]_ [)] [=] _[ ⟨][f][θ]_ [ (] **[x]** _[l]_ [;] **[ x]** _[c]_ [)] _[ −]_ _[f][θ]_ [(] **[a]** [(] _[i]_ [)][)] _[⟩]_ [,] _[ d]_ _r,j_ [(] _[i]_ [)] [=] _[ ⟨][f][θ]_ [ (] **[x]** _[r]_ [;] **[ x]** _[c]_ [)] _[ −]_ _[f][θ]_ [(] **[a]** [(] _[i]_ [)][)] _[⟩]_ [.]
- **Loss for each comparison** _j_ **for user** _i_ : _ℓ_ [(] _j_ _[i]_ [)][(] **[x]** _[l][,]_ **[ x]** _[r]_ [;] **[ x]** _[c]_ [) =] _[ ℓ]_ [(] _[d]_ _r,j_ [(] _[i]_ [)] _[−]_ _[d]_ _l,j_ [(] _[i]_ [)][)][.]
- **Update Step: argmax** _θ,{g_ 1 _,...,gK_ _},{_ **w** ( _i_ ) _∈_ ∆ _K_ _}Ni_ =1 - _i,j_ _[ℓ]_ _j_ [(] _[i]_ [)][(] **[x]** _[l][,]_ **[ x]** _[r]_ [;] **[ x]** _[c]_ [)][.]
See Appendix D for pseudocode details.


We conduct extensive experiments on simulated (Section 4.1), semi-synthetic (Section 4.2), and real (Section 4.3)
preference datasets for both text and image generation tasks to demonstrate that our proposed PAL (Pluralistic
ALignment) framework can: (1) effectively capture the diversity of user preferences, thereby outperforming existing
homogeneous reward models; (2) efficiently achieve performance comparable to the existing SoTA reward models with
far fewer parameters and compute costs; and (3) be versatile and applied to different domains.
For experiments on semi-synthetic and real preference datasets, a simple two-layer MLP PAL reward model can achieve
or exceed the performance of existing status quo reward models, which often contain billions of parameters.
**Compute Resources.** We conducted most of our experiments using 4 RTX 4090, each with 24 GB of VRAM. All
of our experiments can be run on a single RTX 4090 with RAM and VRAM usage of less than 16 GB. A typical
experiment can be finished within 2 hours.


**Dataset.** We synthesize a simple preference dataset with the normal distribution (we use a setting similar to [45]) and
true _f_ _[∗]_ : R _[d]_ _→_ R _[d]_ is linear and the weight **W** _∼N_ (0 _, I_ ). Let **x** _i ∼N_ (0 _,_ (1 _/d_ ) _I_ ) denote the _i_ th item.
**Experiment Setup.** Assume _K_ _[∗]_ user prototypes _{_ **p** _i}_ _[K]_ _i_ =1 _[∗]_ [, where] **[ p]** _[i][ ∼N]_ [(0] _[,]_ [ (1] _[/d]_ [)] _[I]_ [)][ with the minimum distance]
constraint _∥_ **p** _i −_ **p** _j∥≥_ _δ, ∀i, j ∈_ [ _K_ _[∗]_ ] _, i ̸_ = _j_ . We consider two settings: 1) a **mixture** setting, where we assume each
user is located in the convex hull of _K_ prototypes; 2) a simpler **partition** setting, where we assume _N_ users are evenly
sampled from _K_ prototypes, with **a** _i ∈{_ **p** _k}_ _[K]_ _k_ =1 [. Each sample is generated as follows: we randomly draw two items]
_{_ **x** _l,_ **x** _r}_ and one user **a** _i_, and label the user’s preference as sign( _∥f_ _[∗]_ ( **x** _l_ ) _−_ _f_ _[∗]_ ( **a** _i_ ) _∥_ 2 _−∥f_ _[∗]_ ( **x** _r_ ) _−_ _f_ _[∗]_ ( **a** _i_ ) _∥_ 2). We
generate a total of _n_ samples per user to learn the user’s ideal point. We use model A with a single-layer MLP (without
bias) with hinge loss and evaluate on the held-out test set.
Figure 2: The performance of model A on the simulation datasets with _d_ = 16, _K_ = _{_ 1 _,_ 2 _,_ 3 _,_ 4 _,_ 5 _}_, _K_ _[∗]_ = _{_ 2 _,_ 3 _,_ 4 _}_,
_N_ = 50 _∗_ _K_ _[∗]_, and mixture user ideal point setting. For the fig 2(a) visualization, we set _d_ = 2, _K_ = 3, _K_ _[∗]_ = 3.
**Results.** We simulate datasets with multiple settings (different true _K_ _[∗]_, _K_ and, _d_ in both mixture and partition settings
- see Appendix C.1 for details) and evaluate our model A on these simulation datasets with different # samples and #
prototypes. Figure 2(a) shows that PAL can align the user ideal points to the true user ideal points in the representation
Pluralistic Alignment Framework for Learning from Heterogeneous Preferences A PREPRINT
_K_ _[⋆]_ Personas
2 interest in art, interest in literature
3 interest in art, interest in literature, interest in math
4 interest in art, interest in literature, interest in math, interest in music
5 interest in art, interest in literature, interest in math, interest in music, interest in science
6 interest in art, interest in literature, interest in math, interest in music, interest in science, interest in sports
Table 1: Personas used for each _K_ _[⋆]_ in our heteogeneous persona dataset.
space. See Appendix C.1 for more detailed results. Figure 2(b) shows that the homogeneous reward model (# prototypes
= 1) can only achieve sub-optimal performance on the simulated dataset when diverse preferences exist. When we learn
pluralistic preferences by setting multiple learnable prototypes with PAL, we gain a significant 7% accuracy boost.
Figure 2(c) shows that as we increase the number of training samples for seen users, PAL achieves higher test accuracy,
and is also more accurate in capturing the true number of prototypes in the dataset (which we know from simulations).
**Remark.** Recall that _seen_ users are the users whose data is a part of the training data and the accuracy for them is
measured on prediction on unseen pairs using the weights learned during training. Notice that without enough samples
per user, learning diverse preferences can harm the performance, which indicates the importance of sample size in
pluralistic preference learning. Figure 2(d) presents PAL’s potential to generalize to unseen users. Without any further
fine-tuning of the well-trained PAL reward model (trained with 100 samples per seen user), we can simply learn a new
weight for new unseen users with limited labeled samples to achieve prediction accuracy similar to that of seen users.
That is, we keep the prototypes and mappings learned fixed, but only train the weights for the new user using a few
comparison samples. We note that for alignment to be truly effective, it needs to generalize to new users beyond the set
of users whose comparison data are part of the training dataset. Therefore, we clearly distinguish between accuracy for
unseen pairs for seen users versus accuracy for unseen users.


We evaluate the performance of PAL on semi-synthetic datasets which we construct by injecting diversity into real
preference datasets for both text generation and image synthesis tasks. Our results show that PAL can achieve or
surpass existing state-of-the-art (SoTA) large reward models with only 2-layer MLP networks.

4.2 Heterogeneous Semi-Synthetic Datasets


Anthropic’s Persona dataset [7] consists of a series of personalities (personas), each corresponding with 500 statements
that agree with the persona and 500 statements that do not. We denote the set of statements that agrees with a persona _ρ_
as _S_ ( _ρ_ ). We construct a semi-synthetic dataset using Anthropic’s Persona to evaluate PAL.
**Dataset.** Let _ρ_ = _{ρ_ 1 _, . . ., ρK⋆_ _}_ denote the set of personas that exists in our semi-synthetic heterogeneous dataset with
_K_ _[⋆]_ “true” preference groups i.e. each person (user) has one of the _K_ _[⋆]_ personalities. For each _ρj ∈_ _ρ_, we generate
_N_ synthetic _seen_ and _unseen_ users, where a seen user provides preference samples in the training and test sets, while
the unseen user only provides samples in the test set. For each seen synthetic user, we generate _np_ queries that ask if
the user agrees with a given statement from the persona dataset. For each unseen synthetic user, we generate _np,_ unseen
queries. If the statement aligns with the persona _ρj_ of the user, i.e. the statement belongs to _S_ ( _ρj_ ), then the user
answers `yes`, otherwise `no` . Table 1 lists the personas we used to create the dataset for each _K_ _[⋆]_ . Figure 3 shows a
sample question.
**Experiment Setup.** We evaluate the performance of model B with logistic loss on the heterogeneous persona dataset
in various settings. We use model B, which accounts for user-specific variance in conditioning (text prompt), as
the prompts in the dataset are the only quantities that differ from question to question. To examine the impact of
various hyperparameters, we conduct experiments varying the number of true prototypes in the dataset _K_ _[⋆]_, number of
prototypical groups used in the model _K_, queries per seen user _np_ and latent dimension _d_ with fixed number of users
per group _N_ = 10000 as seen in Figure 4.
(a) Varying _K_ _[⋆]_ = _{_ 2 _, . . .,_ 6 _}_ and _K_ = _{_ 1 _, . . .,_ 8 _}_ while fixing _np_ = 1000, _d_ = 16.
(b) Varying _np_ = _{_ 75 _,_ 100 _,_ 200 _,_ 500 _,_ 1000 _}_ and _K_ = _{_ 1 _, . . .,_ 5 _}_ while fixing _K_ _[⋆]_ = 4, and _d_ = 16.
(c) Varying _d_ = _{_ 4 _,_ 8 _,_ 16 _,_ 32 _,_ 64 _}_ and _K_ = _{_ 1 _, . . .,_ 5 _}_ while fixing _K_ _[⋆]_ = 4, and _np_ = 1000.
(d) Varying _np,_ unseen = _{_ 1 _,_ 10 _,_ 20 _,_ 50 _,_ 100 _,_ 200 _,_ 500 _,_ 1000 _}_ and _K_ = _{_ 1 _, . . .,_ 5 _}_ while fixing _K_ _[⋆]_ = 4, _np_ = 1000,
and _d_ = 16.
Pluralistic Alignment Framework for Learning from Heterogeneous Preferences A PREPRINT
Figure 3: An example of pairwise comparison query with a prompt from our heterogeneous persona dataset generated
using Anthropic’s Persona. A synthetic user who is assigned with a persona of _interest in art_ will have a ground truth of
_y_ = _−_ 1 by answering `no`, whereas a user who is assigned with _interest in math_ pairs with a ground truth of _y_ = +1 by
answering `yes` .
Figure 4: Seen accuracy (a,b,c) and unseen accuracy (d) evaluated on the heterogeneous persona dataset across the
number of prototypical groups used in the model _K_ . We vary (a) the number of true prototypes _K_ _[⋆]_, (b) the number
of comparisons per seen user _np_, (c) the size of latent dimension _d_, (d) the number of comparisons per unseen user
_np,_ unseen.
**Results.** We repeat these experiments five times and report the mean _±_ one standard deviation as bar plots in Figure 4.
Figure 4 (a, b) illustrates the generalization performance of PAL on the heterogeneous persona dataset. We observe
that as _K →_ _K_ _[⋆]_, the seen accuracy increases to 100% given a sufficient number of users and number of comparisons
per user. Figure 4 (b) shows that as we get more comparisons per user, we achieve reasonable _seen user_ accuracy, i.e.
we can generalize to unseen pairs for users who are seen (provide training samples) in the dataset. Figure 4 (c) shows
that the size of latent dimension _d_ does not affect the seen accuracy dramatically. Figure 4(d) shows the accuracy for
_unseen users_, i.e., users who do not provide training samples. When _K_ = 1, there is no further learning needed to
generalize to new users. However, when _K >_ 1, we require weights over the _K_ prototypes that we have not yet learned.
To learn these new user weights, as discussed in Section 3.3.1, we fix the _K_ prototypes and the mapping _f_, and use
only a few test data samples to learn the user weights (few-shot). We use these learned weights to make predictions on
the remaining test data (Also see Remark, Section 4.1). From Figure 4(d) we see that for _K_ = 1 the number of samples
used to learn weights makes no difference, since there are no weights to learn over a single prototype. For _K_ = 2,
we see that as we use more data for learning the new user weights, the performance shows diminishing returns until
saturation. We also demonstrate that as the number of prototypes _K_ increase, more comparisons per user are needed to
learn the new user weights, since the dimension of the weight vector increases with _K_ .


We construct a semi-synthetic heterogeneous preference dataset which we call _Pick-a-Filter_, and show that our PAL
reward model can significantly surpass the homogeneous reward model when pluralistic preferences are present.
**Dataset.** The Pick-a-Pic dataset [6] is a large, open dataset for human feedback in text-to-image generation, designed to
align pretrained models with human preferences. It contains around a million samples of text-to-image prompts and
real user preferences over generated images from multiple open-source popular diffusion models, with anonymous
user IDs. Motivated by a natural human color preference distribution [52], we construct the _Pick-a-Filter_ dataset by
adding different color filters to the generated images to explicitly "inject" diverse user preferences into the Pick-a-Pic
V1 dataset. Further details are provided in Figure 7 and Appendix B. The magnitude of heterogeneous preference
injection is determined by a hyperparameter called mixture ratio. The **mixture ratio** _β_ reflects the proportion between
the original pairs from the Pick-a-Pic dataset and the color-filtered pairs. The larger the _β_, the more color-filtered pairs.
Pluralistic Alignment Framework for Learning from Heterogeneous Preferences A PREPRINT
Table 2: PAL test accuracy can match SoTA Pickscore [6] on the Picka-Pic-v1 test set with a fraction of the compute.
Figure 5: PAL Model B test accuracy on
Pick-a-Filter compared to CLIP-H.
Model Test Accuracy(%)
Pick-a-Pic v1 test
CLIP-H14 59.23


Table 3: Results for Summary Dataset: Seen accuracy and unseen accuracy of our model with _K_ = 1 _,_ 5 _,_ 10 compared
to the individual user model proposed in [53]. With only 594K parameters, we achieve on-par performance compared
to a method that requires a supervised-finetuned 6B model.
_K_ = 1 _K_ = 5 _K_ = 10 Li et. al. [53]
Seen accuracy 59 _._ 28 _±_ 0 _._ 14 59 _._ 66 _±_ 0 _._ 09 59 _._ 51 _±_ 0 _._ 12 61.72
Unseen accuracy (zero-shot) 59 _._ 20 _±_ 0 _._ 16 59 _._ 45 _±_ 0 _._ 12 59 _._ 15 _±_ 0 _._ 11 60.65
**Experiment Setup.** We train model B with logistic loss on the _Pick-a-Filter_ dataset with different mixture ratios.
Detailed training setups are deferred to Appendix C.2.
**Results.** Figure 5 shows that PAL-B effectively captures diverse preferences across mixture ratios in Pick-a-Filter.
We can view these mixture ratios as indicating the extent to which the two user groups prefer their respective color
filters. The figure illustrates that PAL enables learning beyond a universal preference ( _K >_ 1) to identify diverse user
preference groups. PAL significantly outperforms the homogeneous reward model in predicting user preferences – at a
mixture ratio of 1, PAL achieves 95.2% test accuracy compared to 75.4% from the homogeneous reward model.


**Dataset.** Reddit TL;DR summary dataset curated by [5] contains a series of preferences over summaries generated by
language models. For each pair of summaries, **x** _l_ and **x** _r_, a worker _i_ determines if **x** _l_ is preferred or not. Moreover,
each pair is also accompanied by the unique identifier of the worker who provides the preference. This would allow us
to apply our model to such a dataset.
**Experiment Setup.** We evaluate our model A with hinge loss on a trimmed version of the summary dataset described in
[53]. Details regarding how the dataset is constructed and comparisons to other baselines are deferred to Appendix C.3.
**Results.** Table 3 compares the performance of our method to the one proposed in [53]. We use the weighted average of
prototypes learned as the general ideal point for new users to conduct zero-shot learning. We emphasize that even though
our model only has 594K parameters and the sentence embeddings we used are generated from `all-mpnet-base-v2`
sentence transformer [54], which contains around 105M parameters, we still can achieve on par performance, especially
in terms of unseen accuracy.