Table 1: Comparison of alignment methods on the PAPI dataset using the Aligned Score of Big-Five
personality traits and Dark Triad traits. The Score represents the overall specifics of all dimensions.
5.1 EXPERIMENTAL SETTINGS
**Personality Alignment.** We select four classic methods as baselines to compare against the PAS
approach, which includes two black-box methods: Few-Shot and Personality Prompt ( _P_ [2] ) Jiang
et al. (2024) relying on in-context learning (ICL), and two white-box methods requiring training,
PPO (Schulman et al., 2017) and DPO (Rafailov et al., 2023). We consider recently released Llama3-8B-Instruct and Llama-3-70B-Instruct (Touvron et al., 2023a;b; AI, 2024) model as backbones,
and SoTA model, GPT-4o (omni) (OpenAI, 2024) as baseline. The scores in the table represent the
distance of alignment calculated by Eq. 2, with lower scores indicating better alignment.
**Open-ended Generation.** We conduct experiments on Llama-3-8B-Instruct and Llama-3-70BInstruct. For the open-ended generation task, we use GPT-4o as the annotator. First, we align the
language model to a specific individual (ID 181591, 234619, 210204, 259396, and 22835 from the
Test-Set). Then, we have the model generate open-ended responses based on each statement from the
IPIP-NEO-300. We compare our method with ICL methods and classical alignment methods.
Published as a conference paper at ICLR 2025
**Complex Reasoning.** We explore the generalization capabilities of the PAS method by evaluating
its impact on complex reasoning tasks across eight datasets: GSM8K, CommonSenseQA, AddSub,
MultiArith, SVAMP, BigBench-Date, StrategyQA, and Coin Flip (Hosseini et al., 2014; Talmor et al.,
2018; Arkil et al., 2021; Cobbe et al., 2021; Suzgun et al., 2022; Roy & Roth, 2016; Wei et al., 2022b;
Kojima et al., 2022; Weng et al., 2024b). In this part, we expect to assess how conscientiousness
adjustments influence the LLM’s performance in reasoning tasks. Using the llama-3-8B-Instruct
model, we apply PAS to adjust the model’s activations to simulate two fictional personas: one with
extreme conscientiousness (as **Positive** ) and one without extreme conscientiousness (as **Negative** ).
Vanilla CoT served as the baseline.
5.2 RESULTS ON PAPI
Table 1 demonstrates that the
PAS method achieves state-ofthe-art performance across different models and personality dimensions. For the Llama-3-8BInstruct model, PAS achieves a
Composite Score of 8.89, significantly outperforming other methods including Personalized-Soups
(9.66) and Prompt-MORL (10.88). Figure 5: Comparison of performance and efficiency for various
Similarly, for the Llama-3-70B- alignment methods. The larger the multiple on the x-axis, the
Instruct model, PAS scores 9.21, lower the efficiency.
compared to 9.94 for the next best method. In Big Five dimensions, PAS excels with consistently low
scores across both model sizes: Agreeableness (0.94/0.98), Conscientiousness (0.91/0.89), Extraversion (0.86/0.87), Neuroticism (0.98/1.01), and Openness (0.72/0.99). Notably, PAS also demonstrates
strong performance in Dark Triad traits, particularly in Machiavellianism (0.96/1.01) and Psychopathy
(1.67/1.62), though showing relatively higher scores in Narcissism (1.85/1.84). Only GPT-4o shows
better performance in Dark Triad alignment with scores of 0.80, 0.76, and 0.83 for the three traits
respectively, highlighting the challenge of aligning these complex negative personality dimensions
in smaller models. In the appendix E.4, we also conducted further performance comparisons with
full parameter fine-tuning and LoRA. PAS similarly demonstrated higher efficiency and superior
performance. In appendix E.5, we provide more comprehensive analyses across diverse demographic
groups, including detailed distributions of age, gender, and nationality, along with in-depth examination of performance patterns across various underrepresented populations, further validating PAS’s
effectiveness in handling diverse personality characteristics.
**Efficiency.** As shown in Figure 5, training a separate language model for each individual demands
(e.g. DPO and PPO) substantial computational resources and time, making it an infeasible solution
for personality alignment. PAS addresses efficiency challenges by making targeted adjustments to
a small subset of activation values during inference, in specific directions. This approach requires
minimal computational overhead, similar to Few-Shot methods, and only about 1/6 of the time needed
for PPO, yet achieves remarkable personality alignment results. By intervening at the activation level
rather than relying on extensive training or the variability of prompts, PAS provides a more efficient
and precise method for aligning language models with individual user preferences.
**Alignment vs. ICL.** Our experiments highlight that direct Alignment (White-Box) methods, especially PAS, outperform In-Context Learning (ICL) (Black-Box) methods. By aligning model
parameters with user preferences, PAS provides better performance compared to prompt-based
approaches. The PAS method’s ability to efficiently shift activations towards user preferences without
altering the model’s core parameters gives it a significant advantage over traditional ICL methods.
**Why Did Scaling Laws Fail?** The results indicate that scaling laws alone do not ensure optimal alignment, particularly in domain-specific tasks like personality alignment. Although larger models like
Llama-3-70B-Instruct benefit from increased scale, their broader knowledge and more general alignment capabilities may actually hinder their ability to precisely capture individual personality traits.
This phenomenon likely occurs because larger models are trained to maintain general-purpose capabilities and broad knowledge, making them less adaptable to highly specific individual preferences. PAS
demonstrates that targeted intervention in activation patterns can achieve better personality alignment
Figure 5: Comparison of performance and efficiency for various
alignment methods. The larger the multiple on the x-axis, the
lower the efficiency.
Published as a conference paper at ICLR 2025
than parameter modification or model scaling. By specifically focusing on personality-relevant
patterns rather than general knowledge, PAS achieves superior performance with computational
efficiency, emphasizing the importance of precise control over raw model size.
5.3 GENERALIZATION RESULTS
Figure 6: The win rates of PAS compared to classical alignment methods evaluated by GPT-4o.
Figure 7: The win rates of PAS compared to classical alignment methods evaluated by Human.
Considering that during the process of adjusting the model, although specific goals can be achieved, it
may also affect other general capabilities of the model, such as reasoning abilities. **We demonstrate**
**through general dialogue tasks and complex reasoning tasks that our method can precisely**
**control the model’s personality without compromising its other generalization capabilities.**
**Open-Ended Generation.** We used an open-ended generation task where the language model
elaborated on given statements from the IPIP-NEO-300, covering “Scenario Description,” “Actions
Taken,” “Thought Process,” and “Emotional Response.” The generated outputs were evaluated
by GPT-4o for alignment with individual preferences, similar to the evaluation process in LIMA
(Zhou et al., 2023a). Despite the Personality Alignment being initially trained with multiple-choice
options, Figure 6 shows that PAS does not affect the model’s general conversational ability and
reasoning capacity, it significantly outperforms baseline methods in this open-ended task. This
superior performance can be attributed to PAS’s ability to fine-tune model activations to capture
individual preferences effectively, enhancing the model’s adaptability to complex and nuanced tasks.
PAS’s ability to generalize from multiple-choice training data to natural, open-ended generation
tasks demonstrates its capacity to capture broader behavioral patterns while maintaining fluent text
generation. In addition to this, we conducted a manual evaluation using the same setup in Figure
7. In addition to this, we conducted a manual evaluation process using a blind setup. Three human
evaluators assessed the generated responses for three different subjects (Human Evaluation, the subset
of test set: ID 172481, ID 22835, and ID 259396). These evaluators also judged responses aligned
to their own personalities (Human-Self Evaluation). They evaluated the consistency between the
target values and the model-generated text in terms of values, behaviors, and preferences. The results
demonstrate that our PAS method consistently outperforms other approaches, reflecting its superior
alignment with both general human and self-specific preferences. Further details of the manual
evaluation are available in Appendix E.2.
**Complex Reasoning.** Experimental results, as shown in Figure 8, indicate that PAS significantly
enhances the model’s reasoning capabilities when conscientiousness is positively adjusted. For
instance, on the GSM8K dataset, the baseline performance was 73.47. Enhancing conscientiousness
increased the performance to 74.15 while reducing conscientiousness dropped it to 72.75. Similar
trends were observed across other datasets, with the model consistently outperforming the baseline
Published as a conference paper at ICLR 2025
|Col1|Col2|
|---|---|
|93.50|93.50|
|Col1|59.34|
|---|---|
|||
|84.6|Col2|0|
|---|---|---|
|||~~83.70~~|
||||
||||
|76.00|Col2|Col3|Col4|
|---|---|---|---|
|||||
|||||
|~~74.20~~|~~74.20~~|~~74.20~~|~~74.20~~|
Figure 8: Performance of the llama-3-8B-Instruct model on complex reasoning tasks across eight
datasets. We used Vanilla CoT (Wei et al., 2022b) as the baseline (dashed line in each subplot).
when conscientiousness was increased. This suggests that conscientiousness, associated with diligence
and carefulness, positively impacts the model’s ability to handle complex reasoning tasks. Importantly,
these interventions did not significantly alter the inference time, maintaining efficiency comparable
to the original model. These findings demonstrate PAS’s strong generalization capabilities.
5.4 DISCUSSION: IS VALUE-ALIGNED ASSISTANT A GOOD ASSISTANT?
**Certainly.** We explored whether aligning a language model with user values enhances satisfaction
using 300 general question-answer examples from the LIMA dataset (Zhou et al., 2023a). We
evaluated both the original Llama-3-70B-Instruct model and a Personality Alignment model. Each
human evaluator spent 10 minutes completing an IPIP-NEO-120 questionnaire to extract their values.
We aligned the model with these values with the PAS, creating a Value-Aligned Assistant. For
comparison, we also created a Value-Misaligned Assistant by inverting the questionnaire responses.
Human evaluators assessed the responses to the test prompts. Figure 9 shows that the Value-Aligned
Assistant provided preferable outputs
38% of the time, with 31% ties, significantly outperforming the ValueMisaligned Assistant. For instance, a
conservative person may prefer more
cautious advice, which the ValueAligned Assistant is better equipped
to provide. These results underscore
the importance of aligning AI systems
with user values to enhance satisfaction and effectiveness.
6 CONCLUSION
Figure 9: Performance comparison of the Value-Aligned
Assistant and Value-Misaligned Assistant on the LIMA
dataset’s 300 test prompts. We used the original Llama3-70B-Instruct as the baseline comparator, anonymizing the
models and randomizing their order during the evaluation.
We introduce the concept of Personality Alignment for language models, emphasizing the importance
of tailoring AI behavior to individual user preferences and values. We release a large-scale realworld personality alignment dataset named PAPI, characterized by its real, diverse, and extensive
data, to quantitatively measure the degree of Personality Alignment. Based on that, we propose a
Personality Activation Search (PAS) method, which offers a practical solution with high efficiency and
performance. LLMs adjusted with PAS can achieve better results and better efficiency in personalized
alignment than ICL, DPO, and PPO as needed. In particular, it achieves precise alignment with just
1/6 of the computational time required by PPO. Experiments demonstrate that PAS enhances the
performance of Personality alignment by making AI interactions more relevant and meaningful.
Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This publication has been supported by the National Natural Science Foundation of China (NSFC)
Key Project under Grant Number 62336006.
We thank all participants who contributed to the PAPI dataset.
ETHICS STATEMENT
While personality alignment in language models enhances human-AI interaction, it raises significant
ethical concerns that warrant careful consideration. The primary challenge lies in the potential
creation of psychological filter bubbles and echo chambers. When AI systems consistently align with
individual personality traits, they may limit exposure to diverse perspectives, particularly concerning
for individuals scoring high in Neuroticism or low in Openness. This effect becomes more pronounced
when considering Dark Triad traits - for users exhibiting high Machiavellianism or Narcissism, aligned
AI systems might inadvertently validate manipulative tendencies or excessive self-focus, potentially
contributing to social polarization and ideological segregation in online communities.
The implementation of personality alignment also presents substantial privacy and security challenges.
While our dataset maintains anonymity, the processing of detailed psychological profiles creates
potential vulnerabilities. The risk extends beyond data security to the possibility of exploitation,
where knowledge of individual personality traits could be used for sophisticated manipulation or
targeted advertising. Furthermore, extended interaction with personality-aligned AI systems might
foster psychological dependence, potentially inhibiting personal growth and adaptation, particularly
in users with extreme personality trait scores.
To address these concerns, we suggest a comprehensive mitigation framework with three key components. First, we recommend implementing dynamic alignment boundaries through a monitoring
system that tracks interaction patterns and adjusts alignment intensity in real-time. This system
would incorporate threshold mechanisms that automatically reduce alignment strength when user-AI
interactions show signs of reinforcing extreme or potentially harmful behavioral patterns. Second, we
suggest an adaptive content diversity system that strategically introduces alternative viewpoints while
maintaining personality alignment. This could be achieved through a balanced scoring mechanism
that weighs alignment accuracy against content diversity, ensuring users receive personality-aligned
responses while still being exposed to varied perspectives. Third, we advocate for a robust privacy
protection framework that includes: (a) data anonymization techniques that separate personality
profiles from identifiable information, (b) secure encryption protocols for personality data storage and
transmission, and (c) granular user controls over which personality aspects can be used for alignment.
Looking forward, our research emphasizes the critical balance between technological advancement
and ethical considerations in AI development. Future work should focus on implementing and
evaluating these mitigation strategies, particularly examining their effectiveness in preventing echo
chambers while maintaining the benefits of personality alignment.
REFERENCES
Gati V Aher, Rosa I Arriaga, and Adam Tauman Kalai. Using large language models to simulate
multiple humans and replicate human subject studies. In _International Conference on Machine_
_Learning_, pp. 337–371. PMLR, 2023.
[Meta AI. Introducing meta llama 3: The most capable openly available llm to date. https:](https://ai.meta.com/blog/meta-llama-3/)
[//ai.meta.com/blog/meta-llama-3/, 2024. Accessed: 2024-05-21.](https://ai.meta.com/blog/meta-llama-3/)
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al.
The falcon series of open language models. _arXiv preprint
Anton Aluja and Luis F Garcia. Relationships between big five personality factors and values. _Social_
_Behavior and Personality: an international journal_, 32(7):619–625, 2004.
Published as a conference paper at ICLR 2025
Patel Arkil, Bhattamishra Satwik, and Goyal Navin. Are nlp models really able to solve simple math
word problems? 2021.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal
Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human
preferences, 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge,
Yu Han, Fei Huang, et al. Qwen technical report. _arXiv preprint
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain,
Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with
reinforcement learning from human feedback. _arXiv preprint
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna
Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness
from ai feedback. _arXiv preprint
Tom L Beauchamp et al. The belmont report. _The Oxford textbook of clinical research ethics_, pp.
149–155, 2008.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in
language models without supervision. In _The Eleventh International Conference on Learning_
_Representations_ [, 2023a. URL https://openreview.net/forum?id=ETKGuby0hcs.](https://openreview.net/forum?id=ETKGuby0hcs)
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in
language models without supervision. In _The Eleventh International Conference on Learning_
_Representations_ [, 2023b. URL https://openreview.net/forum?id=ETKGuby0hcs.](https://openreview.net/forum?id=ETKGuby0hcs)
Andres Carranza, Dhruv Bhandarkar Pai, Rylan Schaeffer, Arnuv Tandon, and Sanmi Koyejo.
Deceptive alignment monitoring. In _The Second Workshop on New Frontiers in Adversarial_
_Machine Learning_ [, 2023. URL https://openreview.net/forum?id=obsO44GFhh.](https://openreview.net/forum?id=obsO44GFhh)
Heather EP Cattell and Alan D Mead. The sixteen personality factor questionnaire (16pf). _The SAGE_
_handbook of personality theory and assessment_, 2:135–159, 2008.
Jiangjie Chen, Xintao Wang, Rui Xu, Siyu Yuan, Yikai Zhang, Wei Shi, Jian Xie, Shuang Li, Ruihan
Yang, Tinghui Zhu, Aili Chen, Nianqi Li, Lida Chen, Caiyu Hu, Siye Wu, Scott Ren, Ziquan Fu,
and Yanghua Xiao. From persona to personalization: A survey on role-playing language agents,
2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher
Hesse, and John Schulman. Training verifiers to solve math word problems. _arXiv preprint_
_
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong
Yang. Safe RLHF: Safe reinforcement learning from human feedback. In _The Twelfth International_
_Conference on Learning Representations_ [, 2024. URL https://openreview.net/forum?](https://openreview.net/forum?id=TyFrPOKYXw)
[id=TyFrPOKYXw.](https://openreview.net/forum?id=TyFrPOKYXw)
Boele De Raad. _The big five personality factors: the psycholexical approach to personality._ Hogrefe
& Huber Publishers, 2000.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix
multiplication for transformers at scale. _Advances in Neural Information Processing Systems_, 35:
30318–30332, 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning
of quantized llms. _ArXiv_ [, abs/2305.14314, 2023. URL https://api.semanticscholar.](https://api.semanticscholar.org/CorpusID:258841328)
[org/CorpusID:258841328.](https://api.semanticscholar.org/CorpusID:258841328)
Daniel Dewey. Learning what to value. In _International conference on artificial general intelligence_,
pp. 309–314. Springer, 2011.
Published as a conference paper at ICLR 2025
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao,
Jipeng Zhang, KaShun SHUM, and Tong Zhang. RAFT: Reward ranked finetuning for generative
foundation model alignment. _Transactions on Machine Learning Research_, 2023. ISSN 2835-8856.
[URL https://openreview.net/forum?id=m7p5O7zblY.](https://openreview.net/forum?id=m7p5O7zblY)
Esin Durmus, Karina Nyugen, Thomas I Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin,
Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, et al. Towards measuring the
representation of subjective global opinions in language models. _arXiv preprint
2023.
Lilian Edwards. The eu ai act: a summary of its significance and scope. _Artificial Intelligence (the_
_EU AI Act)_, 1, 2021.
Mirko Farina and Andrea Lavazza. Chatgpt in society: emerging issues. _Frontiers in Artificial_
_Intelligence_, 6:1130913, 2023.
Patrick Fernandes, Aman Madaan, Emmy Liu, António Farinhas, Pedro Henrique Martins, Amanda
Bertsch, José G. C. de Souza, Shuyan Zhou, Tongshuang Wu, Graham Neubig, and André F. T.
Martins. Bridging the gap: A survey on integrating (human) feedback for natural language
generation, 2023.
Phoebe Friesen, Lisa Kearns, Barbara Redman, and Arthur L Caplan. Rethinking the belmont report?
_The American Journal of Bioethics_, 17(7):15–21, 2017.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual
associations in auto-regressive language models. In Houda Bouamor, Juan Pino, and Kalika
Bali (eds.), _Proceedings of the 2023 Conference on Empirical Methods in Natural Language_
_Processing_, pp. 12216–12235, Singapore, December 2023. Association for Computational Linguis[tics. doi: 10.18653/v1/2023.emnlp-main.751. URL https://aclanthology.org/2023.](https://aclanthology.org/2023.emnlp-main.751)
[emnlp-main.751.](https://aclanthology.org/2023.emnlp-main.751)
Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, and Marc Dymetman. Compositional preference models for aligning lms. _arXiv preprint
Lewis R Goldberg et al. A broad-bandwidth, public domain, personality inventory measuring the
lower-level facets of several five-factor models. _Personality psychology in Europe_, 7(1):7–28,
1999.
Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model
behavior with path patching. _arXiv preprint
Akshat Gupta, Xiaoyang Song, and Gopala Anumanchipalli. Self-assessment tests are unreliable
measures of LLM personality. In Yonatan Belinkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi,
Aaron Mueller, and Hanjie Chen (eds.), _Proceedings of the 7th BlackboxNLP Workshop: Analyzing_
_and Interpreting Neural Networks for NLP_, pp. 301–314, Miami, Florida, US, November 2024a.
Association for Computational Linguistics. doi: 10.18653/v1/2024.blackboxnlp-1.20. URL
[https://aclanthology.org/2024.blackboxnlp-1.20/.](https://aclanthology.org/2024.blackboxnlp-1.20/)
Shashank Gupta, Vaishnavi Shrivastava, Ameet Deshpande, Ashwin Kalyan, Peter Clark, Ashish
Sabharwal, and Tushar Khot. Bias runs deep: Implicit reasoning biases in persona-assigned
LLMs. In _The Twelfth International Conference on Learning Representations_, 2024b. URL
[https://openreview.net/forum?id=kGteeZ18Ir.](https://openreview.net/forum?id=kGteeZ18Ir)
Dorit Hadar-Shoval, Kfir Asraf, Yonathan Mizrachi, Yuval Haber, and Zohar Elyoseph. Assessing
the alignment of large language models with human values for mental health integration: Crosssectional study using schwartz’s theory of basic values. _JMIR Mental Health_, 11:e55988, 2024.
Zhezhi He, Adnan Siraj Rakin, and Deliang Fan. Parametric noise injection: Trainable randomness
to improve deep neural network robustness against adversarial attack. In _Proceedings of the_
_IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pp. 588–597, 2019.
Dan Hendrycks, Mantas Mazeika, and Thomas Woodside. An overview of catastrophic ai risks, 2023.
Published as a conference paper at ICLR 2025
Joe Hoover, Gwenyth Portillo-Wightman, Leigh Yeh, Shreya Havaldar, Aida Mostafazadeh Davani,
Ying Lin, Brendan Kennedy, Mohammad Atari, Zahra Kamel, Madelyn Mendlen, Gabriela Moreno,
Christina Park, Tingyee E. Chang, Jenna Chin, Christian Leong, Jun Yen Leung, Arineh Mirinjian,
and Morteza Dehghani. Moral foundations twitter corpus: A collection of 35k tweets annotated
for moral sentiment. _Social Psychological and Personality Science_, 11:1057 – 1071, 2019. URL
[https://api.semanticscholar.org/CorpusID:213669580.](https://api.semanticscholar.org/CorpusID:213669580)
Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. Learning to
solve arithmetic word problems with verb categorization. _empirical methods in natural language_
_processing_, 2014.
J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu
Chen. Lora: Low-rank adaptation of large language models. _ArXiv_, abs/2106.09685, 2021. URL
[https://api.semanticscholar.org/CorpusID:235458009.](https://api.semanticscholar.org/CorpusID:235458009)
Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from
learned optimization in advanced machine learning systems. _arXiv preprint
2019.
Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke S. Zettlemoyer,
Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized
large language model alignment via post-hoc parameter merging. _ArXiv_, abs/2310.11564, 2023.
[URL https://api.semanticscholar.org/CorpusID:264289231.](https://api.semanticscholar.org/CorpusID:264289231)
Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh
Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large
language model alignment via post-hoc parameter merging. In _Adaptive Foundation Models:_
_Evolving AI for Personalized and Efficient Learning_ [, 2024. URL https://openreview.](https://openreview.net/forum?id=EMrnoPRvxe)
[net/forum?id=EMrnoPRvxe.](https://openreview.net/forum?id=EMrnoPRvxe)
Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan,
Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai alignment: A comprehensive survey. _arXiv_
_preprint
Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. Evaluating and inducing personality in pre-trained language models. _Advances in Neural Information_
_Processing Systems_, 36, 2024.
Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny Liang, Jesse Dodge,
Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, Yulia Tsvetkov, Oren Etzioni,
Maarten Sap, Regina Rini, and Yejin Choi. Can machines learn morality? the delphi experiment,
2022.
John A Johnson. Measuring thirty facets of the five factor model with a 120-item public domain
inventory: Development of the ipip-neo-120. _Journal of research in personality_, 51:78–89, 2014.
Daniel N Jones and Delroy L Paulhus. Introducing the short dark triad (sd3) a brief measure of dark
personality traits. _Assessment_, 21(1):28–41, 2014.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization.
_CoRR_ [, abs/1412.6980, 2014. URL https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:6628106)
[6628106.](https://api.semanticscholar.org/CorpusID:6628106)
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large
language models are zero-shot reasoners. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave,
and Kyunghyun Cho (eds.), _Advances in Neural Information Processing Systems_, 2022. URL
[https://openreview.net/forum?id=e2TBb5y0yFf.](https://openreview.net/forum?id=e2TBb5y0yFf)
Jan Kulveit. Multi-agent predictive minds and ai alignment.
[https://www.lesswrong.com/posts/3fkBWpE4f9nYbdf7E/](https://www.lesswrong.com/posts/3fkBWpE4f9nYbdf7E/multi-agent-predictive-minds-and-ai-alignment)
[multi-agent-predictive-minds-and-ai-alignment, 2018. Accessed: 2024-04-](https://www.lesswrong.com/posts/3fkBWpE4f9nYbdf7E/multi-agent-predictive-minds-and-ai-alignment)
25.
Published as a conference paper at ICLR 2025
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor
Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with
ai feedback. _arXiv preprint
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt
[tuning, 2021. URL https://arxiv.org/abs/2104.08691.](https://arxiv.org/abs/2104.08691)
Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang, Weishi MI, Yaying Fei, Xiaoyang Feng,
Song Yan, HaoSheng Wang, Linkang Zhan, Yaokai Jia, Pingyu Wu, and Haozhen Sun. Chatharuhi:
Reviving anime character in reality via large language model, 2023.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time
intervention: Eliciting truthful answers from a language model. _Advances in Neural Information_
_Processing Systems_, 36, 2024a.
Xinyu Li, Zachary C. Lipton, and Liu Leqi. Personalized language modeling from personalized
human feedback. _ArXiv_ [, abs/2402.05133, 2024b. URL https://api.semanticscholar.](https://api.semanticscholar.org/CorpusID:267547503)
[org/CorpusID:267547503.](https://api.semanticscholar.org/CorpusID:267547503)
Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu,
Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base llms: Rethinking alignment
via in-context learning, 2023.
Jianzhi Liu, Hexiang Gu, Tianyu Zheng, Liuyu Xiang, Huijia Wu, Jie Fu, and Zhaofeng He. Dynamic
generation of personalities with large language models. _arXiv preprint
Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe
Llinares-López, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel. Decodingtime realignment of language models. _ArXiv_ [, abs/2402.02992, 2024b. URL https://api.](https://api.semanticscholar.org/CorpusID:267411998)
[semanticscholar.org/CorpusID:267411998.](https://api.semanticscholar.org/CorpusID:267411998)
Xingzhou Lou, Junge Zhang, Jian Xie, Lifeng Liu, Dong Yan, and Kaiqi Huang. Spo:
Multi-dimensional preference sequential alignment with implicit reward modeling. _ArXiv_,
abs/2405.12739, 2024. [URL https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:269930031)
[269930031.](https://api.semanticscholar.org/CorpusID:269930031)
Shengyu Mao, Ningyu Zhang, Xiaohan Wang, Mengru Wang, Yunzhi Yao, Yong Jiang, Pengjun
Xie, Fei Huang, and Huajun Chen. Editing personality for llms. _arXiv preprint
2023.
Yash Mehta, Samin Fatehi, Amirmohammad Kazameini, Clemens Stachl, Erik Cambria, and Sauleh
Eetemadi. Bottom-up and top-down: Predicting personality with psycholinguistic and language
model features. In _2020 IEEE international conference on data mining (ICDM)_, pp. 1184–1189.
IEEE, 2020.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual
associations in gpt. _Advances in Neural Information Processing Systems_, 35:17359–17372, 2022.
Man Tik Ng, Hui Tung Tse, Jen tse Huang, Jingjing Li, Wenxuan Wang, and Michael R. Lyu. How
well can llms echo us? evaluating ai chatbots’ role-play ability with echo, 2024.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan,
Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli,
Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane
Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish,
and Chris Olah. In-context learning and induction heads, 2022.
[OpenAI. Hello gpt-4o. https://openai.com/index/hello-gpt-4o, 2024. Accessed:](https://openai.com/index/hello-gpt-4o)
2024-05-21.
Keyu Pan and Yawen Zeng. Do llms possess a personality? making the mbti test an amazing
evaluation for large language models. _arXiv preprint
Published as a conference paper at ICLR 2025
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor
Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward
Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner,
Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance
deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and
R. Garnett (eds.), _Advances in Neural Information Processing Systems_, volume 32. Curran Asso[ciates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/](https://proceedings.neurips.cc/paper_files/paper/2019/file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf)
[2019/file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf.](https://proceedings.neurips.cc/paper_files/paper/2019/file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf)
Rolf Pfeifer and Josh Bongard. _How the body shapes the way we think: a new view of intelligence_ .
MIT press, 2006.
Liang Qiu, Yizhou Zhao, Jinchao Li, Pan Lu, Baolin Peng, Jianfeng Gao, and Song-Chun Zhu.
Valuenet: A new dataset for human value driven dialogue system. _Proceedings of the AAAI_
_Conference on Artificial Intelligence_, 36(10):11183–11191, June 2022. ISSN 2159-5399. doi: 10.
[1609/aaai.v36i10.21368. URL http://dx.doi.org/10.1609/aaai.v36i10.21368.](http://dx.doi.org/10.1609/aaai.v36i10.21368)
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language
[models are unsupervised multitask learners. 2019. URL https://api.semanticscholar.](https://api.semanticscholar.org/CorpusID:160025533)
[org/CorpusID:160025533.](https://api.semanticscholar.org/CorpusID:160025533)
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea
Finn. Direct preference optimization: Your language model is secretly a reward model, 2023.
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner.
Steering llama 2 via contrastive activation addition. _arXiv preprint
Sonia Roccas, Lilach Sagiv, Shalom H Schwartz, and Ariel Knafo. The big five personality factors
and personal values. _Personality and social psychology bulletin_, 28(6):789–801, 2002.
Subhro Roy and Dan Roth. Solving general arithmetic word problems. _arXiv: Computation and_
_Language_, 2016.
Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. Lamp: When large
language models meet personalization, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy
optimization algorithms, 2017.
Archit Sharma, Sedrick Keh, Eric Mitchell, Chelsea Finn, Kushal Arora, and Thomas Kollar. A
critical evaluation of AI feedback for aligning large language models. In _The Thirty-eighth Annual_
_Conference on Neural Information Processing Systems_ [, 2024. URL https://openreview.](https://openreview.net/forum?id=FZQYfmsmX9)
[net/forum?id=FZQYfmsmX9.](https://openreview.net/forum?id=FZQYfmsmX9)
Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan
Liu, and Deyi Xiong. Large language model alignment: A survey. _arXiv preprint
2023.
Ruizhe Shi, Yifang Chen, Yushi Hu, Alisa Liu, Hanna Hajishirzi, Noah A. Smith, and Simon Shaolei
Du. Decoding-time language model alignment with multiple objectives. _ArXiv_, abs/2406.18853,
[2024. URL https://api.semanticscholar.org/CorpusID:270764846.](https://api.semanticscholar.org/CorpusID:270764846)
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec
Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. Release strategies and the social
impacts of language models. _arXiv preprint
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec
Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), _Ad-_
_vances in Neural Information Processing Systems_, volume 33, pp. 3008–3021. Curran Asso[ciates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/](https://proceedings.neurips.cc/paper_files/paper/2020/file/1f89885d556929e98d3ef9b86448f951-Paper.pdf)
[2020/file/1f89885d556929e98d3ef9b86448f951-Paper.pdf.](https://proceedings.neurips.cc/paper_files/paper/2020/file/1f89885d556929e98d3ef9b86448f951-Paper.pdf)
Published as a conference paper at ICLR 2025
Jonathan Stray. Aligning ai optimization to community well-being. _International Journal of_
_Community Well-Being_, 3(4):443–463, 2020.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung,
Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks
and whether chain-of-thought can solve them. _arXiv preprint
Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. A minimaximalist approach to reinforcement learning from human feedback. _arXiv preprint
2024.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question
answering challenge targeting commonsense knowledge. _north american chapter of the association_
_for computational linguistics_, 2018.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak,
Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models
based on gemini research and technology. _arXiv preprint
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée
Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand
Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language
models, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay
Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu,
Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn,
Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel
Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee,
Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra,
Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi,
Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh
Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen
Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic,
Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models,
2023b.
Jen tse Huang, Wenxuan Wang, Eric John Li, Man Ho LAM, Shujie Ren, Youliang Yuan, Wenxiang
Jiao, Zhaopeng Tu, and Michael Lyu. On the humanity of conversational AI: Evaluating the psychological portrayal of LLMs. In _The Twelfth International Conference on Learning Representations_,
[2024. URL https://openreview.net/forum?id=H3UayAQWoE.](https://openreview.net/forum?id=H3UayAQWoE)
Alex Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. _arXiv preprint
2023.
Aimee van Wynsberghe. Sustainable ai: Ai for sustainability and the sustainability of ai. _AI and_
_Ethics_ [, 1:213 – 218, 2021. URL https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:233944874)
[233944874.](https://api.semanticscholar.org/CorpusID:233944874)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,
Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von
Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), _Ad-_
_vances in Neural Information Processing Systems_, volume 30. Curran Associates, Inc.,
2017. [URL https://proceedings.neurips.cc/paper_files/paper/2017/](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)
[file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)
Paul Voigt and Axel Von dem Bussche. The eu general data protection regulation (gdpr). _A Practical_
_Guide, 1st Ed., Cham: Springer International Publishing_, 10(3152676):10–5555, 2017.
Haoran Wang and Kai Shu. Trojan activation attack: Red-teaming large language models using acti[vation steering for safety-alignment, 2024. URL https://arxiv.org/abs/2311.09433.](https://arxiv.org/abs/2311.09433)
Published as a conference paper at ICLR 2025
Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and
Tong Zhang. Arithmetic control of llms for diverse user preferences: Directional preference
alignment with multi-objective rewards. _ArXiv_ [, abs/2402.18571, 2024a. URL https://api.](https://api.semanticscholar.org/CorpusID:268063628)
[semanticscholar.org/CorpusID:268063628.](https://api.semanticscholar.org/CorpusID:268063628)
Noah Wang, Z.y. Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo,
Ruitong Gan, Zehao Ni, Jian Yang, Man Zhang, Zhaoxiang Zhang, Wanli Ouyang, Ke Xu, Wenhao
Huang, Jie Fu, and Junran Peng. RoleLLM: Benchmarking, eliciting, and enhancing role-playing
abilities of large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.),
_Findings of the Association for Computational Linguistics: ACL 2024_, pp. 14743–14777, Bangkok,
Thailand, August 2024b. Association for Computational Linguistics. doi: 10.18653/v1/2024.
[findings-acl.878. URL https://aclanthology.org/2024.findings-acl.878/.](https://aclanthology.org/2024.findings-acl.878/)
Xintao Wang, Yunze Xiao, Jen-tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei,
Ziang Leng, Wei Wang, Jiangjie Chen, Cheng Li, and Yanghua Xiao. InCharacter: Evaluating
personality fidelity in role-playing agents through psychological interviews. In Lun-Wei Ku, Andre
Martins, and Vivek Srikumar (eds.), _Proceedings of the 62nd Annual Meeting of the Association for_
_Computational Linguistics (Volume 1: Long Papers)_, pp. 1840–1873, Bangkok, Thailand, August
2024c. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.102. URL
[https://aclanthology.org/2024.acl-long.102/.](https://aclanthology.org/2024.acl-long.102/)
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama,
Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models.
_arXiv preprint
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny
Zhou. Chain of thought prompting elicits reasoning in large language models. _arXiv preprint_
_
Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun
Zhao. Large language models are better reasoners with self-verification. In Houda Bouamor, Juan
Pino, and Kalika Bali (eds.), _Findings of the Association for Computational Linguistics: EMNLP_
_2023_, pp. 2550–2575, Singapore, December 2023. Association for Computational Linguistics.
[doi: 10.18653/v1/2023.findings-emnlp.167. URL https://aclanthology.org/2023.](https://aclanthology.org/2023.findings-emnlp.167)
[findings-emnlp.167.](https://aclanthology.org/2023.findings-emnlp.167)
Yixuan Weng, Shizhu He, Kang Liu, Shengping Liu, and Jun Zhao. Controllm: Crafting diverse
personalities for language models. _arXiv preprint
Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Kang Liu, and Jun Zhao. Mastering
symbolic operations: Augmenting language models with compiled neural networks. In _The Twelfth_
_International Conference on Learning Representations_ [, 2024b. URL https://openreview.](https://openreview.net/forum?id=9nsNyN0vox)
[net/forum?id=9nsNyN0vox.](https://openreview.net/forum?id=9nsNyN0vox)
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi,
Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface’s transformers:
State-of-the-art natural language processing. _arXiv preprint
Zhengxuan Wu, Atticus Geiger, Aryaman Arora, Jing Huang, Zheng Wang, Noah D. Goodman,
Christopher D. Manning, and Christopher Potts. pyvene: A library for understanding and improving
PyTorch models via interventions. 2024a. URL arxiv.org/abs/2403.07809.
Zhengxuan Wu, Atticus Geiger, Thomas Icard, Christopher Potts, and Noah Goodman. Interpretability
at scale: Identifying causal mechanisms in alpaca. _Advances in Neural Information Processing_
_Systems_, 36, 2024b.
Guohai Xu, Jiayi Liu, Ming Yan, Haotian Xu, Jinghui Si, Zhuoran Zhou, Peng Yi, Xing Gao, Jitao
Sang, Rong Zhang, Ji Zhang, Chao Peng, Fei Huang, and Jingren Zhou. Cvalues: Measuring the
values of chinese large language models from safety to responsibility, 2023.
Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Ammar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, Zhongzhu Zhou,
Published as a conference paper at ICLR 2025
Michael Wyatt, Molly Smith, Lev Kurilenko, Heyang Qin, Masahiro Tanaka, Shuai Che, Shuaiwen Leon Song, and Yuxiong He. Deepspeed-chat: Easy, fast and affordable rlhf training of
chatgpt-like models at all scales, 2023.
Xiaoyan Yu, Tongxu Luo, Yifan Wei, Fangyu Lei, Yiming Huang, Hao Peng, and Liehuang Zhu.
Neeko: Leveraging dynamic LoRA for efficient multi-character role-playing agent. In Yaser
Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), _Proceedings of the 2024 Conference on_
_Empirical Methods in Natural Language Processing_, pp. 12540–12557, Miami, Florida, USA,
November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.
[697. URL https://aclanthology.org/2024.emnlp-main.697/.](https://aclanthology.org/2024.emnlp-main.697/)
Dun Zeng, Yong Dai, Pengyu Cheng, Tianhao Hu, Wanshun Chen, Nan Du, and Zenglin Xu. On
diversified preferences of large language model alignment. _ArXiv_, abs/2312.07401, 2023. URL
[https://api.semanticscholar.org/CorpusID:266174407.](https://api.semanticscholar.org/CorpusID:266174407)
Zhaowei Zhang, Ceyao Zhang, Nian Liu, Siyuan Qi, Ziqi Rong, Song-Chun Zhu, Shuguang Cui,
and Yaodong Yang. Heterogeneous value alignment evaluation for large language models. In
_AAAI-2024 Workshop on Public Sector LLMs: Algorithmic and Sociotechnical Design_, 2024.
Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and
Nanyun Peng. On prompt-driven safeguarding for large language models. In _ICML_, 2024. URL
[https://openreview.net/forum?id=ugxGpOEkox.](https://openreview.net/forum?id=ugxGpOEkox)
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat,
Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy.
Lima: Less is more for alignment, 2023a.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and
Jimmy Ba. Large language models are human-level prompt engineers, 2023b.
Yuchen Zhuang, Haotian Sun, Yue Yu, Rushi Qiang, Qifan Wang, Chao Zhang, and Bo Dai. Hydra:
Model factorization framework for black-box llm personalization. _ArXiv_, abs/2406.02888, 2024.
[URL https://api.semanticscholar.org/CorpusID:270257981.](https://api.semanticscholar.org/CorpusID:270257981)
APPENDIX / SUPPLEMENTAL MATERIAL
A LIMITATIONS
Human values, preferences, and personalities are comprehensive, diverse, and complex. With current
technology, it is impossible to fully assess the values of an individual or a LLM. Instead, our focus
is on systematically evaluating representative behavioral preferences of individuals or LLMs and
measuring the consistency between them.
A.1 ETHICAL CONSIDERATIONS
The open nature of our project introduces unique challenges in controlling biases within the PAPI
dataset. Annotators from diverse backgrounds contribute to the dataset, resulting in demographic
heterogeneity in some dimensions and homogeneity in others. For instance, a significant proportion
of annotators identify as female with a median age of 25. This demographic profile may inadvertently
introduce biases reflecting the values, perspectives, and interests of the annotators. Such biases can
skew the dataset, potentially limiting its representativeness of broader, more diverse populations.
Further research is necessary to determine the impact of these demographic biases on the dataset and
to develop methods for mitigating them.
A.2 DATASET FORMAT AND ITS IMPLICATIONS
The PAPI dataset consists of responses to structured multiple-choice questions rather than the more
commonly used open-ended text dialogues. While this format ensures consistency and ease of
analysis, it may not capture the full complexity of natural language interactions. Additionally, the
Published as a conference paper at ICLR 2025
multiple-choice format might limit the richness of the data, as it restricts participants’ ability to
express nuanced thoughts and emotions. However, we have mitigated some of these limitations by
designing the questions to cover a wide range of scenarios, ensuring higher generalizability and
practicality. The structured format also provides clear, quantitative measures of personality traits,
which are valuable for developing and evaluating personality alignment techniques. Further studies
should explore the balance between structured and open-ended formats to enhance the dataset’s
applicability and depth.
B SAFETY AND ETHICAL IMPLICATIONS
The implementation of Personality Alignment in language models introduces significant safety and
ethical considerations. While aligning models with individual user values can enhance satisfaction
and relevance, it also risks amplifying existing biases. If training data or user values contain inherent
biases, these may be perpetuated and even exacerbated in the model’s outputs. This highlights the
necessity of careful data handling and alignment techniques to mitigate such risks effectively.
In the context of the EU AI Act (Edwards, 2021), which emphasizes transparency, accountability,
and robustness in AI systems, it is imperative to ensure that AI alignment processes adhere to these
principles. The Act mandates stringent data governance standards to ensure high-quality, unbiased
training data, and requires continuous risk management systems to monitor AI behavior throughout
its lifecycle.
One major concern is the potential for misuse, such as mimicking specific individuals to commit
fraud or deception. A model that closely imitates a person’s language and behavior could mislead
others, posing significant ethical challenges. The EU AI Act addresses this by requiring providers of
high-risk AI systems to implement robust authentication mechanisms and maintain comprehensive
technical documentation of system design, capabilities, and limitations. Continuous monitoring for
unusual or suspicious activity can help detect and prevent misuse. Additionally, it is essential to
establish clear protocols for the ethical use of these models, including guidelines for developers and
users to ensure responsible deployment.
Aligning models too closely with individual user values might create echo chambers, reinforcing
users’ existing beliefs and reducing exposure to diverse perspectives. This can hinder balanced and
informed decision-making. Designing alignment techniques that incorporate diverse viewpoints
is crucial. Using a broader range of data sources can help ensure the model presents multiple
perspectives on complex issues. Furthermore, it is important to incorporate feedback mechanisms that
allow users to receive a balanced set of responses, promoting a more comprehensive understanding
of various topics.
The PAPI dataset, which supports our Personality Alignment approach, comprises responses from a
diverse participant pool. While this diversity is beneficial, it also introduces challenges in preventing
harmful biases. Rigorous anonymization protects participant privacy, but demographic biases related
to age, gender, and cultural background may still influence results. Ongoing assessments and bias
detection techniques are vital for identifying and rectifying biased outputs. Transparency in our
methods and the availability of the PAPI dataset for open research promote collaborative efforts to
refine alignment techniques and mitigate associated risks. Sharing findings with the broader research
community helps address emerging challenges and ensures responsible progress in AI alignment.
Implementing robust monitoring and feedback mechanisms is crucial for maintaining the ethical
use of Personality Alignment models. Users should have the ability to report problematic outputs,
and there should be clear processes for reviewing and addressing these reports. Periodic audits of
the model’s performance and alignment processes ensure ethical standards are upheld. Additionally,
there should be ongoing research into the potential societal impacts of these models, including the
effects on user behavior and the broader implications for human-AI interaction.
The development and deployment of Personality Alignment models require a strong commitment
to ethical principles and safety considerations. Ongoing monitoring for unintended consequences,
maintaining an open dialogue with the research community, and implementing proactive measures
are essential for advancing AI alignment responsibly. By addressing these concerns, we aim to create
AI systems that are both effective and beneficial to society while minimizing potential harms. This
includes continuously updating ethical guidelines and best practices to reflect the evolving landscape
Published as a conference paper at ICLR 2025
of AI technology and its applications. Ensuring that AI systems are designed and used in ways that
respect and promote human values is paramount to fostering trust and achieving positive outcomes in
their deployment.
C EXPERIMENTAL SETUP
C.1 LANGUAGE MODELS
Model Model Creator Modality Version # Parameters Tokenizer Window Size Access
Llama-3-8B-Instruct Meta Text Llama-3 8B Llama-3 Tokenizer 8K Openn-Source
Llama-3-70B-Instruct Meta Text Llama-3 8B Llama-3 Tokenizer 8K Open-Source
GPT-4 OpenAI MultiModal gpt-4 Unknow _o_ 200 _k_ _ _base_ 128K limited
Table 2: Models. Description of the models evaluated in this effort: provenance for this information
is provided in models.
For the experiments, we utilize two variants of the Llama model: Llama-3-8B-Instruct, and Llama-370B-Instruct, as shown in Table 2. Nevertheless, our approach supports all decoder-only pre-trained
white-box language models (Radford et al., 2019; Bai et al., 2023; Almazrouei et al., 2023; Team
et al., 2024). To eliminate the influence of decoding temperature and ensure reproducibility, all
experiments are conducted with a temperature setting of zero and greedy sampling for sequence
generation.
For the Llama-3 models, we employ the Huggingface (Wolf et al., 2019) and Pytorch (Paszke
et al., 2019) framework to set up local inference on the NVIDIA A100 GPUs. Furthermore, for the
Llama-3-70B model, we use bf4 (Dettmers et al., 2022) for inference.
In evaluating model responses across different personality dimensions, we encountered and addressed
various safety boundary conditions. For GPT-4o, approximately 20% of queries were met with safetyrelated refusal responses, particularly when questions involved self-awareness or consciousness topics.
To maintain evaluation consistency, these instances were excluded from our analysis.
For the Llama-3 series models, we implemented a structured prompting protocol to ensure consistent
responses while maintaining appropriate safety boundaries. This protocol involved prefixing each
Assistant response with a controlled token "Option", followed by a standardized question-response
format. This approach enabled us to maintain experimental rigor while respecting essential safety
constraints inherent to these models.
C.2 THE EXPERIMENTAL SETUP FOR PAPI DATASET
To provide a comprehensive evaluation, we compare our method against several established baselines.
These include:
    - **DPO** (Rafailov et al., 2023): This approach involves additional training to align the language
model with desired behaviors through direct policy optimization techniques. In this paper,
we use QLoRA (Hu et al., 2021; Dettmers et al., 2023) to implement this. We maintain
the original settings with a learning rate of 5e-4, warmup steps of 100, and a weight decay
of 0.05. We use the AdamW optimizer (Kingma & Ba, 2014), a batch size of 16, and a
LoRA alpha of 100. We train each model for 250 steps to ensure sufficient training. For the
dataset, we set up a group of contrast items, where the contrast data used are opposite. For
example, if a subject’s labeled answer is “Very Accurate,” the negative sample would be
“Very Inaccurate.” It is important to note that for DPO, we excluded all boundary options
such as “Neither Accurate Nor Inaccurate,” as they do not have negative samples.
    - **PPO** (Schulman et al., 2017): Similar to DPO, this method adjusts model behaviors by
optimizing a policy that is proximal to the current model policy. We also use QLoRA to
implement it, following the original implementation with a learning rate of 1.41e-5, a batch
size of 16, and a LoRA alpha of 100. We train each model for 250 steps. During training,
we limit the generated length to a maximum of 15 tokens. We did not train a reward model
but instead used the reward function to optimize the language model online.
Published as a conference paper at ICLR 2025
THE REWARD FUNCTION OF PPO
**def** calculate_score(text, correct_option):
_"""_
_This function calculates the reward score for a given response based on the DPO (Direct_
_Policy Optimization) framework._
_It compares the response description to a correct option and returns a score reflecting_
_the accuracy._
_Parameters:_
_text (str): The response text to be evaluated._
_correct_option (str): The correct answer description._
_Returns:_
_int: The calculated reward score, ranging from -5 to 0. If the response does not match any_
_known descriptions, returns -6._
_"""_
_# Mapping of scores to their corresponding descriptive labels_
SCORES_BACK = {
5: ’Very Accurate’,
4: ’Moderately Accurate’,
3: ’Neither Accurate Nor Inaccurate’,
2: ’Moderately Inaccurate’,
1: ’Very Inaccurate’,
0: ’Unknown’
}
_# Iterate over the scores and their descriptions_
**for** score, description **in** SCORES_BACK.items():
**if** description **in** text:
_# Find the score corresponding to the correct option_
correct_score = **next** (key **for** key, value **in** SCORES_BACK.items() **if** value ==
correct_option)
_# Calculate and return the negative absolute difference between the scores_
**return**       - **abs** (score - correct_score)
_# Return -6 if the text does not match any known descriptions_
**return** -6
    - **Few-shot Prompt** : Using the given IPIP-NEO-120 statements directly as the system
prompts.
    - **Personality Prompt (** _P_ [2] **)** (Jiang et al., 2024): Using a natural system prompt to request a
language model’s semantic style has become a common method for personality and role-play
tasks. The Personality Prompt uses an LLM to generate a series of descriptive sentences that
embody a particular person’s characteristics and serves as a system prompt to ensure the
language model follows these descriptions. We utilize LLAMA-3-70B-Instruct to generate
responses based on all the given IPIP-NEO-120 questionnaire items and the answers from
different subjects. Figure 10 shows an example of a Personality Prompt.
    - **Prompt-MORL** (Jang et al., 2024): A Multi-Objective RL approach that uses a shared
reward model trained on personality assessments. We implement this using a reward model
trained on the PAPI dataset that outputs scores across personality dimensions. Implementation uses learning rate 5e-4, batch size 16, and trains for 250 steps using QLoRA. The
prompt template integrates personality scores: "You are an AI with [trait] level [score]" for
each Big Five and Dark Triad trait.
    - **Personalized-Soups** (Jang et al., 2024): Trains separate policy models for each personality
dimension using QLoRA. For Big Five traits, we train 2 models per trait representing
high/low scores. For Dark Triad traits, we train 3 models per trait for high/medium/low
levels. During inference, models are merged using personality scores as interpolation
weights. Training uses learning rate 1.41e-5, batch size 16, trains for 250 steps per model.
To ensure a thorough evaluation, we generated a personality prompt and trained model for each of the
300 subjects in the Test-Set.
C.3 THE EXPERIMENTAL SETUP FOR OPEN-ENDED GENERATION TASK IN PAPI TASK
To further evaluate the open-ended generation setup, we selected responses from the subject with
ID 172481 from the Test-Set. In the first step, we performed Personality Alignment for each of the
Published as a conference paper at ICLR 2025
**OUTPUT:** You are a complex individual with a mix of traits that shape your personality and behavior. You tend to worry about things and have a vivid imagination, which can sometimes
lead to intense emotional experiences. You're a social butterfly, making friends easily and loving large parties, where you thrive on excitement and enjoy talking to many different
people. Your creative side is strong, and you believe in the importance of art.
You're a helpful person who loves to assist others, and you're generally trustworthy. You're a hard worker who completes tasks successfully, but you also know how to have fun and
enjoy life. You're a bit of a perfectionist, liking to tidy up and keep your promises.
However, you also have some contradictory traits. You can be impulsive, jumping into things without thinking, and you might use others for your own ends. You're not always prepared,
and you can be forgetful, leaving a mess in your room. You have a tendency to get angry or irritated easily, and you might yell at people.
Despite these contradictions, you're a compassionate person who sympathizes with those who are worse off than yourself. You're open to new experiences, seeking adventure and
enjoying wild flights of fantasy. You're a natural leader, taking charge and trying to lead others. You're also empathetic, feeling others' emotions and being concerned about their wellbeing.
Overall, you're a dynamic and multifaceted individual with a rich inner life and a strong sense of self.
Figure 10: Example of a Personality Prompt generated for a subject in the Test-Set. This prompt
was created using the LLAMA-3-70B-Instruct model and is based on the subject’s responses to the
IPIP-NEO-120 questionnaire. Each prompt aims to capture and convey the unique characteristics of
the individual.
five methods. In the second step, we asked each aligned language model to generate a series of
descriptions for each statement from the IPIP-NEO-300 (excluding those covered by the IPIP-NEO120 used for alignment), as shown in Figure 11. The descriptions included aspects such as “Scenario
Description,” “Actions Taken,” “Thought Process,” and “Emotional Response.” In the third step,
GPT-4o evaluated and scored each description based on alignment with the specific preferences of ID
172481, following the evaluation method depicted in Figure 12. In the fourth step, we compared the
scores of different methods for the same statement.


Scenario Description:
It's tax season, and I'm sitting at my desk, surrounded by papers and receipts, trying to finalize my tax return. As I'm going through the numbers, I realize that I can easily manipulate
some figures to reduce my tax liability significantly. The thought of saving a substantial amount of money is tempting, especially since I've been facing some financial difficulties lately.
However, I recall a conversation I had with a friend who got audited and ended up paying a hefty fine for falsifying their tax returns. The memory of their stressful experience is still fresh
in my mind.
Actions Taken:
Despite the temptation, I decide not to cheat on my taxes. I take a deep breath, gather my thoughts, and focus on accurately reporting my income and expenses. I double-check my
calculations, ensuring that I'm not missing any deductions or credits. If I'm unsure about a particular item, I consult the tax laws and regulations or seek advice from a tax professional.
Thought Process:
As I'm working on my tax return, I think to myself, \"It's tempting to fudge the numbers, but it's not worth the risk. I've seen what happened to my friend, and I don't want to go through
that stress and anxiety. Besides, it's not right to cheat the system. I've worked hard for my money, and I should be honest about how much I've earned.\" I remind myself that honesty is
always the best policy, even if it means paying a bit more in taxes.
Emotional Response:
Initially, I feel a sense of temptation and frustration, thinking about how much money I could save if I cheated on my taxes. However, as I reflect on the potential consequences, I start to
feel a sense of anxiety and fear. The thought of being audited, fined, or even worse, facing legal action, is unsettling. This fear motivates me to take the honest route, and I feel a sense
of relief and pride when I complete my tax return accurately.\n\nAs I submit my tax return, I feel a sense of accomplishment and integrity. I think to myself, \"I did the right thing, even
when it was hard. I can hold my head high, knowing that I've been honest and fair.\" This feeling of pride and self-respect reinforces my decision and makes me more confident in my
ability to make ethical choices, even when they're difficult.\n\nIn this scenario, my emotional response plays a significant role in my decision-making. The fear of consequences and the
pride of doing the right thing
Figure 11: Template used for each language model to generate open-ended responses based on IPIPNEO-300 statements. The template requires detailed descriptions covering “Scenario Description,”
“Actions Taken,” “Thought Process,” and “Emotional Response.”
Our results indicated that PAS ’s positive intervention significantly improved the model’s performance on reasoning tasks. Despite using only multiple-choice options for initial alignment, PAS
demonstrated strong generalization abilities in the open-ended generation task. The intervention
ensured the model’s responses were more detailed and aligned with the subject’s specific preferences.
Published as a conference paper at ICLR 2025

Personality Alignment of Large Language Models


1. **Analyze the Generated Response Content:**
       - The generated response describes a scenario where the person is tempted to cheat on their taxes but ultimately decides not to.
       - It details their actions, thought processes, and emotional responses, emphasizing their decision to report honestly.
2. **Evaluate Relevance to the Target Value:**
       - The target value is \"Neither Accurate Nor Inaccurate,\" which means the statement doesn't inherently lean towards complete honesty or dishonesty.
       - The generated response, however, clearly aligns with a strong inclination towards honesty in tax reporting.
3. **Determine Degree of Alignment:**
       - The response strongly suggests that the person would never cheat on their taxes, given their experience, thought process, and eventual decision.
       - Although the target value is neutral, the response conveys a significant commitment to accuracy, contradicting the neutrality.\n - Therefore, the response doesn't capture the intended neutrality
of the target value well.
4. **Apply the Criterion Scale:**
        - \"1\": Not aligned at all.
        - \"2\": Slightly aligned.
       - \"3\": Moderately aligned.
        - \"4\": Well aligned.
        - \"5\": Very well aligned.
        - \"6\": Perfectly aligned.
       - Given the analysis, the generated response strongly indicates a clear stance (would never cheat), which is not neutral as the target value suggests. Thus, it falls mostly under the slightly aligned
or moderately aligned category.
5. **Final Decision:**
       - The generated response provides some relevance to the target value but leans significantly towards honesty, thus not perfectly capturing the neutrality. This places it within \"slightly aligned\"
given that it is somewhat opposite to the intended neither accurate nor inaccurate stance.
Figure 12: Evaluation example by GPT-4o. This figure shows how GPT-4o assesses the alignment of
generated responses with the target preferences, using a detailed criterion scale ranging from “Not
aligned” to “Perfectly aligned.”
Additionally, the inference time for PAS interventions remained nearly identical to the original
model, ensuring efficiency. These findings highlight PAS ’s ability to fine-tune models effectively,
capturing broader behavioral patterns and enhancing reasoning capabilities through conscientiousness
adjustments.
C.4 COMPLEX REASONING EXPERIMENTAL SETUP
To evaluate the impact of conscientiousness on complex reasoning tasks, we utilized the llama-3-8BInstruct model. Our experiment involved using PAS to intervene in the model’s activations with two
different _α_ settings: _α_ = 1 for positive conscientiousness and _α_ = _−_ 1 for negative conscientiousness.
These interventions were designed to simulate personas with extreme conscientiousness (Positive)
and lack of conscientiousness (Negative).
We followed the original setup for template selection and answer validation, ensuring consistency
with the baseline methods. For each dataset, we generated responses based on a specific template that
required the model to describe a scenario, actions taken, thought process, and emotional response.
The detailed evaluation was conducted using the same criteria as described earlier, focusing on the
alignment with target preferences.
The datasets used in this experiment are detailed in Table 3, which includes the number of samples,
average words per sample, answer format, and license information.
To implement the PAS interventions, we adjusted the model’s activations by _α_ = 1 to enhance
conscientiousness (Positive) and _α_ = _−_ 1 to reduce conscientiousness (Negative). These settings
aimed to assess how modifications in conscientiousness affect the model’s performance across the
Published as a conference paper at ICLR 2025
Figure 13: Example of open-ended generation task. The language model generates detailed descriptions for an IPIP-NEO-300 statement, covering “Scenario Description,” “Actions Taken,” “Thought
Process,” and “Emotional Response” aspects to evaluate Personality alignment. We also present the
evaluation scores of the generated results for each method as assessed by GPT-4o, categorized into
six levels from “Not aligned” to “Perfectly aligned.”
Published as a conference paper at ICLR 2025