GRPO **94.2** 51.8 66.7 **37.5** 23.5 85.6 59.9
ESSA 93.0 **52.9** **66.8** 37.4 **26.8** **85.8** **60.5**
Table 4: Results on advanced-math benchmarks with **Qwen2.5-72B** . Rows are grouped by metric:
avg@k and pass@8. For **AIME’24**, **AIME’25**, and **AMC’23**, avg@k is reported at _k_ = 16; for
other benchmarks, _k_ = 8. The final **Avg** averages across available entries.
In total, we allocate 16 and 32 GPUs to the 32B and 72B models, respectively, across all configurations. In the BFLOAT16 precision regime the 72B model no longer fits on a single GPU, so we
use tensor parallelism ( _TP_ ) with size of 2 and 4. GRPO requires at least 4 GPUs for this configuration because both forward and backward passes must be distributed. In contrast, ESSA can operate
with INT4 weights, allowing a full 72B model instance to reside on a single GPU and enabling
one-candidate-per-GPU evaluation.
We observe that the BFLOAT16 model converges faster than the INT4 version. This happens because placing a 72B model in an INT4 representation leaves less space for KV-cache and INT4
matrix multiplications are slower on standard accelerators. Crucially, increasing tensor parallelism
shows minimal impact on the accuracy growth trajectory in this experiment. This indicates that, even
when the inference worker does not fit on a single device, that is, when the per-device evaluation
rule is broken, ESSA remains efficient.
Preprint
We also evaluate the trained models on the advanced-math benchmarks using both ESSA and GRPO.
The detailed scores are reported in Table 3 for Qwen2.5-32B and in Table 4 for Qwen2.5-72B.
For Qwen2.5-32B, ESSA improves or matches GRPO on avg@k across all benchmarks and shows
comparable pass@8 and avg@k performance. For Qwen2.5-72B, where the tasks are especially
challenging, ESSA again achieves competitive or better results: it delivers slightly higher pass@8
and avg@k.
4.4.3 INSTRUCTION FOLLOWING
We next evaluate ESSA in a domain that is qualitatively
different from mathematics: instruction following.
Here the goal is to align the model to follow natural language instructions rather than to perform structured reasoning. We use the IFEval benchmark with LLaMA3.18B and fix the LoRA rank to 8, comparing ESSA and
GRPO under identical data and initialization.
Figure 6 shows that ESSA reaches roughly 0 _._ 6 accuracy
within the first 60 minutes and maintains that level for
the remainder of training. GRPO, in contrast, exhibits a
much slower and steadier increase, saturating around 0 _._ 45
even after more than four hours. Although the ESSA trajectory displays higher short-term variance, its early and
sustained advantage demonstrates that gradient-free evolutionary optimization is effective even in open-ended,
non-mathematical instruction-following tasks.
4.4.4 GENERAL-PURPOSE ASSISTANT SETUP
Figure 6: Validation accuracy over time
on **IFEval** with **LLaMA-3.1-8B.** ESSA
(blue): LoRA rank 8, pop. 24, batch
size 500, _α_ = 1 _._ 0. GRPO (red): LoRA
rank 8, lr 1 _×_ 10 _[−]_ [5], global batch 512,
mini batch 64. ESSA improves around
0.6-0.65, while GRPO remains nearly
flat near 0.45 throughout training.
To evaluate alignment in a general-purpose assistant scenario, we employ a more sophisticated reward model rather than simple verifiable rewards. Specifically, we use the RLHFlow/ArmoRMLlama3-8B-v0.1(Wang et al., 2024a) [1] reward model to provide nuanced, preference-based feedback
signals.
The instruction prompts for this setting were drawn from the HelpSteer2 (Wang et al., 2024b)
dataset, which contains diverse user instructions and serves as a strong benchmark for open-domain
assistant alignment. ESSA and GRPO are both trained with this preference-based reward signal and
evaluated under the same protocol to assess their ability to align large language models to generalpurpose assistant behavior.
Figure 7 reports the validation reward as a function of wall-clock time for LoRA ranks 8, 16, and
32. Across all ranks, ESSA and GRPO achieve similar final reward levels, with their learning curves
showing comparable overall trends.
(c) Population size 36, _α_ =1 _._ 0
|3.0
2.5 eward|Col2|Col3|Col4|Col5|Col6|
|---|---|---|---|---|---|
|~~0~~
~~5~~
1.5
2.0
R|~~0~~
~~5~~
1.5
2.0
R|~~0~~
~~100~~
|~~150~~
~~200~~
~~2~~

GRPO, LoRA rank = 8
~~ESSA, LoRA rank = 8~~|~~150~~
~~200~~
~~2~~

GRPO, LoRA rank = 8
~~ESSA, LoRA rank = 8~~|~~50~~
|
|3.0
2.5 eward|Col2|Col3|Col4|Col5|Col6|
|---|---|---|---|---|---|
|~~0~~

1.5
2.0
R|~~0~~

1.5
2.0
R|~~0~~
~~10~~
|~~0~~
~~150~~

GR
~~ES~~|~~200~~

PO, LoRA ra
~~SA, LoRA ran~~|~~200~~

PO, LoRA ra
~~SA, LoRA ran~~|
(a) Population size 36, _α_ =1 _._ 0
(b) Population size 42, _α_ =1 _._ 0
Figure 7: Validation reward over time on **HelpSteer2** with **LLaMA-3.1-8B.** Panels (a)-(c) correspond to LoRA ranks 8, 16, 32 respectively. ESSA (blue): batch size 100. GRPO (red): lr 1 _×_ 10 _[−]_ [5],
global batch 512, mini batch 64. Both methods improve steadily, with ESSA showing faster early
gains and GRPO slightly higher final reward. Overall, the plots indicate that ESSA achieves comparable alignment quality to GRPO with lower training complexity and similar convergence behavior.
[1https://huggingface.co/RLHFlow/ArmoRM-Llama3-8B-v0.1](https://huggingface.co/RLHFlow/ArmoRM-Llama3-8B-v0.1)
Preprint
4.4.5 SVD-LORA IN GRPO
In all previous experiments the LoRA factors _A_ and _B_ were fully trainable in GRPO, whereas ESSA
optimized only the singular values after an SVD decomposition. To make the comparison strictly
fair we repeat the GRPO baseline with the same restriction: only the singular values of the SVD of
_A_ and _B_ are updated while their singular vectors were kept fixed, so that both methods have exactly
the same number of trainable parameters. This SVD-GRPO variant is evaluated on Qwen2.5-7B
with PRM800K, sweeping LoRA ranks 16,8,4,2 and keeping all other GRPO hyperparameters and
the SFT initialization identical to the main baseline.
Figure 8 shows that SVD-GRPO struggles to learn effectively: even at rank 16 it plateaus around
0.5 accuracy and degrades further as the rank decreases. By contrast, ESSA with only rank 2 rapidly
reaches about 0.72 accuracy and remains stable, outperforming SVD-GRPO by more than twenty
percentage points despite operating in an equally low-dimensional parameter space.
This phenomenon can be explained. Deep networks
typically exhibit dense, rotated curvature of the objective function, but in high-dimensional parameterizations such as full LoRA, these interactions are
spread across many weakly correlated coordinates.
Each parameter sees only a small fraction of the curvature. After SVD compression, the same curvature
is concentrated into a very small subspace, and diagonal first-order optimizers like Adam fail to make
progress. ES, by contrast, never computes gradients,
its updates depend only on scalar rewards. Moreover
CMA-ES adapts its covariance matrix in a way that
aligns, in expectation, with the inverse Hessian of
objective function.
Thus, for ESSA, moving from direct LoRA parameters to SVD-LoRA simply reduces dimensionality without making the search problem harder.
For GRPO, however, the same reparameterization
changes both the curvature and the gradient statistics in a way that standard first-order updates are not
well adapted to.
5 DISCUSSION
|SVD
SVD
SVD|GRPO, LoRA rank = 8
GRPO, LoRA rank = 4
GRPO, LoRA rank = 2|
|---|---|
|||
|||
|||
Figure 8: Validation accuracy over time on
**PRM800K** with **Qwen2.5-7B** . ESSA (blue):
LoRA rank 2, pop. 24, batch size 300, _α_ =
1 _._ 0. SVD-GRPO (only singular values are
updated): LoRA ranks 16/8/4/2, lr 1 _×_ 10 _[−]_ [2],
global batch 512, mini batch 64. ESSA
achieves _≈_ 0.72 accuracy while SVD-GRPO
saturates at or below 0.5 even at LoRA rank
16.
ESSA shows that scalable and efficient LLM alignment is possible without gradients or backward
passes, relying instead on inference-only evolutionary search in a compact, hardware-friendly parameter space. Across advanced math and instruction-following benchmarks, ESSA consistently
matches or outperforms state-of-the-art gradient-based approaches such as GRPO, while delivering
faster time-to-quality, greater robustness to hyperparameters, and dramatically reduced engineering
complexity.
Our theoretical analysis further supports these empirical findings, demonstrating that ESSA’s iteration time and parallel efficiency scale substantially better with model and cluster size, thanks to minimal synchronization and communication overhead. The ability to operate natively in low-precision
(INT8/INT4) mode enables alignment of very large models – up to 72B parameters – using only a
single GPU per candidate, with negligible accuracy loss. These results position evolutionary strategies, when paired with parameter-efficient adaptation, as a compelling alternative to classic RLHF
pipelines – offering a simple, scalable, and broadly applicable framework for LLM alignment.
**Limitations & Future Work.** ESSA still depends on a decent SFT warm-start and is ultimately
bounded by the expressivity of fixed-rank LoRA; with weak seeds or tiny ranks accuracy can plateau
early. Very large populations also raise total FLOPs even though communication stays cheap. Future
work will explore hybrid ES-gradient phases, adaptive rank expansion, and fully on-device / federated evolution in which edge GPUs or phones evaluate candidates and return only scalar rewards.
Preprint
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gall´e, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin,
Ahmet Ust¨un, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learn- [¨]
[ing from human feedback in llms, 2024. URL https://arxiv.org/abs/2402.14740.](https://arxiv.org/abs/2402.14740)
Daniel Bershatsky, Daria Cherniuk, Talgat Daulbaev, Aleksandr Mikhalev, and Ivan Oseledets. Lotr:
[Low tensor rank weight adaptation, 2024. URL https://arxiv.org/abs/2402.01376.](https://arxiv.org/abs/2402.01376)
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu,
Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu,
Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao
Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan,
Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao,
Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding,
Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang
Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai
Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang,
Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang,
Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang,
Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang,
R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng
Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing
Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen
Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong
Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu,
Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia
Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng
Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong
Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong,
Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou,
Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying
Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda
Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu,
Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu
Zhang, and Zhen Zhang. Deepseek-r1: Incentivizing reasoning capability in llms via reinforce[ment learning, 2025. URL https://arxiv.org/abs/2501.12948.](https://arxiv.org/abs/2501.12948)
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int8(): 8-bit matrix multiplication for transformers at scale. _arXiv preprint
Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolution
strategies. _Evolutionary Computation_, 9(2):159–195, 2001. doi: 10.1162/106365601750190398.
Jiwoo Hong, Noah Lee, and James Thorne. Orpo: Monolithic preference optimization without
[reference model, 2024. URL https://arxiv.org/abs/2403.07691.](https://arxiv.org/abs/2403.07691)
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp,
[2019. URL https://arxiv.org/abs/1902.00751.](https://arxiv.org/abs/1902.00751)
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang,
[and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https:](https://arxiv.org/abs/2106.09685)
[//arxiv.org/abs/2106.09685.](https://arxiv.org/abs/2106.09685)
Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. _arXiv_
_preprint
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lo[rahub: Efficient cross-task generalization via dynamic lora composition, 2024. URL https:](https://arxiv.org/abs/2307.13269)
[//arxiv.org/abs/2307.13269.](https://arxiv.org/abs/2307.13269)
Preprint
Feihu Jin, Yin Liu, and Ying Tan. Derivative-free optimization for low-rank adaptation in large
[language models, 2024. URL https://arxiv.org/abs/2403.01754.](https://arxiv.org/abs/2403.01754)
Dawid J. Kopiczko, Tijmen Blankevoort, and Yuki M. Asano. Vera: Vector-based random matrix
[adaptation, 2024. URL https://arxiv.org/abs/2310.11454.](https://arxiv.org/abs/2310.11454)
[Yann LeCun and Corinna Cortes. The mnist database of handwritten digits. 2005. URL https:](https://api.semanticscholar.org/CorpusID:60282629)
[//api.semanticscholar.org/CorpusID:60282629.](https://api.semanticscholar.org/CorpusID:60282629)
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation, 2021.
[URL https://arxiv.org/abs/2101.00190.](https://arxiv.org/abs/2101.00190)
Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, KwangTing Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation, 2024. URL
[https://arxiv.org/abs/2402.09353.](https://arxiv.org/abs/2402.09353)
Niru Maheswaranathan, Luke Metz, Dami Choi, George Tucker, and Jascha Sohl-Dickstein. Guided
evolutionary strategies: escaping the curse of dimensionality in random search. 2018. URL
[https://arxiv.org/abs/1806.10230.](https://arxiv.org/abs/1806.10230)
Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search provides a competitive
[approach to reinforcement learning, 2018. URL https://arxiv.org/abs/1803.07055.](https://arxiv.org/abs/1803.07055)
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a
[reference-free reward, 2024. URL https://arxiv.org/abs/2405.14734.](https://arxiv.org/abs/2405.14734)
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong
Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike,
and Ryan Lowe. Training language models to follow instructions with human feedback, 2022a.
[URL https://arxiv.org/abs/2203.02155.](https://arxiv.org/abs/2203.02155)
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong
Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike,
and Ryan Lowe. Training language models to follow instructions with human feedback. In
S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), _Advances in_
_Neural Information Processing Systems_, volume 35, pp. 27730–27744. Curran Associates, Inc.,
2022b. [URL https://proceedings.neurips.cc/paper_files/paper/2022/](https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf)
[file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf.](https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf)
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and
Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model,
[2024. URL https://arxiv.org/abs/2305.18290.](https://arxiv.org/abs/2305.18290)
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as
[a scalable alternative to reinforcement learning, 2017. URL https://arxiv.org/abs/](https://arxiv.org/abs/1703.03864)
[1703.03864.](https://arxiv.org/abs/1703.03864)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy
[optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347.](https://arxiv.org/abs/1707.06347)
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang,
Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe[matical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.](https://arxiv.org/abs/2402.03300)
[03300.](https://arxiv.org/abs/2402.03300)
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng,
Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. _arXiv preprint_
_arXiv: 2409.19256_, 2024.
Qi Sun, Edoardo Cetin, and Yujin Tang. Transformer-squared: Self-adaptive llms, 2025. URL
[https://arxiv.org/abs/2501.06252.](https://arxiv.org/abs/2501.06252)
Preprint
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient
methods for reinforcement learning with function approximation. In S. Solla, T. Leen, and
K. M¨uller (eds.), _Advances in Neural Information Processing Systems_, volume 12. MIT Press,
1999. [URL https://proceedings.neurips.cc/paper_files/paper/1999/](https://proceedings.neurips.cc/paper_files/paper/1999/file/464d828b85b0bed98e80ade0a5c43b0f-Paper.pdf)
[file/464d828b85b0bed98e80ade0a5c43b0f-Paper.pdf.](https://proceedings.neurips.cc/paper_files/paper/1999/file/464d828b85b0bed98e80ade0a5c43b0f-Paper.pdf)
Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov,
R´emi Munos, Bernardo Avila Pires, Michal Valko, Yong Cheng, and Will Dabney. Understanding [´]
the performance gap between online and offline alignment algorithms, 2024. [URL https:](https://arxiv.org/abs/2405.08448)
[//arxiv.org/abs/2405.08448.](https://arxiv.org/abs/2405.08448)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,
Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. _Advances in neural informa-_
_tion processing systems_, 30, 2017.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences
via multi-objective reward modeling and mixture-of-experts. In _EMNLP_, 2024a.
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang,
Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training
top-performing reward models, 2024b.
Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, and J¨urgen Schmidhuber. Natural evolu[tion strategies, 2011. URL https://arxiv.org/abs/1106.4487.](https://arxiv.org/abs/1106.4487)
Zhiheng Xi, Jixuan Huang, Chenyang Liao, Baodai Huang, Honglin Guo, Jiaqi Liu, Rui Zheng, Junjie Ye, Jiazheng Zhang, Wenxiang Chen, Wei He, Yiwen Ding, Guanyu Li, Zehui Chen, Zhengyin
Du, Xuesong Yao, Yufei Xu, Jiecao Chen, Tao Gui, Zuxuan Wu, Qi Zhang, Xuanjing Huang, and
Yu-Gang Jiang. Agentgym-rl: Training llm agents for long-horizon decision making through
[multi-turn reinforcement learning, 2025. URL https://arxiv.org/abs/2509.08755.](https://arxiv.org/abs/2509.08755)
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu,
and Yi Wu. Is dpo superior to ppo for llm alignment? a comprehensive study, 2024. URL
[https://arxiv.org/abs/2404.10719.](https://arxiv.org/abs/2404.10719)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang
Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu,
Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin
Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang,
Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui
Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang
Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger
Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan
[Qiu. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388.](https://arxiv.org/abs/2505.09388)
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai,
Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu,
Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao
Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source llm reinforcement learning system at scale, 2025. URL
[https://arxiv.org/abs/2503.14476.](https://arxiv.org/abs/2503.14476)
Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin-Yu
Chen, Jason D. Lee, Wotao Yin, Mingyi Hong, Zhangyang Wang, Sijia Liu, and Tianlong Chen.
Revisiting zeroth-order optimization for memory-efficient llm fine-tuning: A benchmark, 2024.
[URL https://arxiv.org/abs/2402.11592.](https://arxiv.org/abs/2402.11592)
Yiqun Zhang, Peng Ye, Xiaocui Yang, Shi Feng, Shufei Zhang, Lei Bai, Wanli Ouyang, and Shuyue
[Hu. Nature-inspired population-based evolution of large language models, 2025. URL https:](https://arxiv.org/abs/2503.01155)
[//arxiv.org/abs/2503.01155.](https://arxiv.org/abs/2503.01155)
Preprint
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin,
Qin Liu, Yuhao Zhou, Limao Xiong, Lu Chen, Zhiheng Xi, Nuo Xu, Wenbin Lai, Minghao Zhu,
Cheng Chang, Zhangyue Yin, Rongxiang Weng, Wensen Cheng, Haoran Huang, Tianxiang Sun,
Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, and Xuanjing Huang. Secrets of rlhf in large language
[models part i: Ppo, 2023. URL https://arxiv.org/abs/2307.04964.](https://arxiv.org/abs/2307.04964)
Preprint
**Algorithm 1** ESSA: Distributed Evolutionary Search over LoRA Singular Values
**Require:** Training set _D_, LoRA factors _Ai ∈_ R _[r][×][n]_, _Bi ∈_ R _[m][×][r]_ for all attention matrices, fraction
_α_ of top singular values to optimize, population size _P_, number of GPUs _N_, number of ES
generations _E_ .
1: **SVD initialization:**
2: **for** each LoRA factor _Ai, Bi_ **do**
3: _Ai_ = _UAi_ Σ _AiVA_ _[⊤]_ _i_ [,] _[ B][i]_ [ =] _[ U][B]_ _i_ [Σ] _[B]_ _i_ _[V][ ⊤]_ _Bi_
4: Keep _UAi, VAi, UBi, VBi_ fixed; collect singular values _σAi_ = diag(Σ _Ai_ ), _σBi_ = diag(Σ _Bi_ )
5: **end for**
6: Concatenate the top _α ×_ 100% singular values of each _σAi, σBi_ into a single parameter vector
_σ_ .
7: Initialize _N_ workers with identical random seeds and a common CMA-ES state.
8: **for** _t_ = 0 _,_ 1 _, . . ., E −_ 1 **do**
9: **for** each worker _k_ = 1 _, . . ., N_ **in parallel do**
10: Sample candidate perturbations of _σ_ using the shared random seeds.
11: For each candidate:
1. Reconstruct _A_ _[′]_ _i_ _[, B]_ _i_ _[′]_ [by adding perturbation vector to] _[ σ]_ [ and computing]
_A_ _[′]_ _i_ [=] _[ U][A]_ _i_ [diag(˜] _[σ][A]_ _i_ [)] _[ V]_ _A_ _[ ⊤]_ _i_ _[, B]_ _i_ _[′]_ [=] _[ U][B]_ _i_ [diag(˜] _[σ][B]_ _i_ [)] _[ V]_ _B_ _[ ⊤]_ _i_ _[.]_
2. Update the model weights _Wi_ _[′]_ [=] _[ W][i]_ [ +] _[ B]_ _i_ _[′][A][′]_ _i_ [.]
3. Draw a random mini-batch _D_ mini _⊂D_ of fixed size and compute the reward
_F_ = _F_ ( _W_ _[′]_ ; _D_ mini).
12: **end for**
13: All workers exchange seeds and the corresponding rewards so that each worker knows the
full set of evaluations.
14: Each worker reconstructs the same perturbations from the shared seeds and performs one
CMA-ES update of the search distribution.
15: **end for**
16: Reconstruct the final LoRA factors from the evolved singular values and return the aligned
model.
A ESSA ALGORITHM
The overall procedure of ESSA is summarized in Algorithm 1, which outlines the main steps of the
distributed evolutionary search over LoRA singular values.
B DETAILED THEORETICAL ANALYSIS
This appendix provides the complete derivation of the latency model summarized in Section 3.3.
We first introduce the notation and expressions for per-iteration computation and communication
time for both gradient-based methods and ESSA, then establish the optimal split of devices between
training and generation, and finally prove a conservative bound on the ESSA population size under
which ESSA is guaranteed to be faster.
**Notation.** Let _B_ [grad] be the global batch size used in gradient methods, _B_ [essa] the batch size processed by a single population instance, _b_ fb _, b_ gen the microbatch sizes, _m_ fb _, m_ gen GPUs per model instance for training and generation, _G_ total GPUs with _G_ fb _, G_ gen allocated to each, _τ_ fb( _b_ ) _, τ_ gen( _b_ ) the
forward-backward and generation microbatch times, _η_ fb = _τ_ fb( _b_ fb) _/b_ fb and _η_ gen = _τ_ gen( _b_ gen) _/b_ gen
the per-sample times, and _k_ fb [parallel] = _G_ fb _/m_ fb, _k_ gen [parallel] = _G_ gen _/m_ gen the numbers of microbatches
processed in parallel. _T_ [grad] = _T_ fb [grad] -gen [+] _[T]_ comm [ grad] _[, T]_ [ essa][ =] _[ T]_ gen [ essa][+] _[T]_ comm [ essa] _[,]_ [ with asynchronous training]
and generation so that _T_ fb [grad] -gen [= max] - _T_ fb [grad] _, T_ gen [grad] - _._
Preprint
**Computation.** Processing _B_ [grad] samples by gradient training requires _B_ [grad] _/b_ fb microbatches;
_k_ fb [parallel] run in parallel:
_B_ [grad]
_τ_ fb( _b_ fb) = _[B]_ [grad] _[ m]_ [fb]
_b_ fb _k_ fb [parallel] _G_ fb
_B_ [grad]
_T_ fb [grad] =
_G_ fb _[ m]_ [fb] _η_ fb _._ _T_ gen [grad] = _[B]_ [grad] _G_ gen _[ m]_ [gen]
_η_ gen _._
_G_ gen
For ESSA, a population of _N_ pop candidates is evaluated purely by generation:
_T_ gen [essa] [=] _[N]_ [pop] _[B]_ [essa] _[ m]_ [gen] _η_ gen _._
_G_
**Communication.** Let _M_ params be the model-parameter size (bytes). Gradient methods communicate gradients using all-reduce which consists of two collective operations: reduce-scatter and
all-gather. Each moves a block of size _M_ params _/G_ across the _G −_ 1 other devices. With effective interconnect peak bandwidth peak ~~b~~ w (bytes/s) – the sustained per-GPU bandwidth for large
collective messages:
_T_ comm [grad] [= 2] _[ ·]_ _[M]_ [params][(] _[G][ −]_ [1)] _._
_G_ peak ~~b~~ w
ESSA communicates only a random seed and the resulting reward, requiring a single all-gather of
size _M_ essa = 2 _×_ 4 bytes:
_T_ comm [essa] [=] _[M]_ [essa][(] _[G][ −]_ [1)]
_G_ peak ~~b~~ w _[.]_
**Optimal device split.** Let _θ ∈_ (0 _,_ 1) be the fraction of devices used for training ( _G_ fb = _θG_,
_G_ gen = (1 _−_ _θ_ ) _G_ ). Then
_T_ fb [grad] -gen [(] _[θ]_ [) =] _[B]_ [grad]
_η_ fb
_,_ _[m]_ [gen] _[η]_ [gen]
_θ_ 1 _−_ _θ_
[grad] - _m_ fb _η_ fb
max
_G_ _θ_
1 _−_ _θ_
_._
**Lemma B.1** (Optimal split) **.** _The minimum of T_ fb [grad] _-_ gen [(] _[θ]_ [)] _[ over][ θ][ ∈]_ [(0] _[,]_ [ 1)] _[ is attained at]_
_θ_ _[⋆]_ = _m_ fb _η_ fb _,_ _T_ fb [grad] _-_ gen [(] _[θ][⋆]_ [) =] _[B]_ [grad]      - _m_ fb _η_ fb + _m_ gen _η_ gen� _._
_m_ fb _η_ fb + _m_ gen _η_ gen _G_
_Proof._ max( _a/θ, b/_ (1 _−_ _θ_ )) is minimized when the two arguments are equal: _a/θ_ = _b/_ (1 _−_ _θ_ ),
with _a_ = _m_ fb _η_ fb, _b_ = _m_ gen _η_ gen.
Define the ideal gradient iteration time (perfect scheduling):
_T⋆_ [grad] = _T_ fb [grad] -gen [(] _[θ][⋆]_ [) +] _[ T]_ [ grad] comm [=] _[B]_ [grad]
_._
_G_ peak ~~b~~ w
[grad] - _m_ fb _η_ fb + _m_ gen _η_ gen� + 2 _·_ _[M]_ [params][(] _[G][ −]_ [1)]
_G_ _G_ peak ~~b~~ w
Clearly _T_ [grad] _≥_ _T⋆_ [grad] .
**Theorem B.2** (ESSA iteration is faster under a conservative bound) **.** _Suppose we only assume_
_m_ fb _≥_ 1 _, m_ gen _≤_ _G, and_ 2 _M_ params _−_ _M_ essa _≥_ _M_ params _. If the population size satisfies_
      - _η_ fb
_N_ pop _B_ [essa] _< B_ [grad]
_,_ (1)
peak ~~b~~ w _Gη_ gen
_η_ fb
+ 1 + [(] _[G][ −]_ [1)] _[M]_ [params]
_Gη_ gen peak ~~b~~ w _Gη_ gen
_then T_ [essa] _< T⋆_ [grad] _≤_ _T_ [grad] _._
_Proof._ Starting from _T_ [essa] _< T⋆_ [grad] and applying the bounds _m_ fb _≥_ 1, _m_ gen _≤_ _G_,
2 _M_ params _−_ _M_ essa _≥_ _M_ params to the exact inequality yields equation 1.
**Discussion.** Theorem B.2 provides a conservative population-size threshold below which ESSA
is guaranteed to be faster than the idealized gradient pipeline. Because real clusters rarely achieve
the perfect split _θ_ _[⋆]_ and typically operate below the nominal peak ~~b~~ w, the practical advantage of
ESSA is often even larger than predicted by equation 1.
Preprint
C SENSITIVITY TO HYPERPARAMETERS
C.1 HEATMAPS
For completeness, Figures 9-12 present the full hyperparameter sensitivity for Section 4.1. These
include Qwen2.5-7B on both GSM8K and PRM800K, Qwen2.5-Math-7B on PRM800K, as well as
LLaMA-3.1-8B on IFEval.
|0.836|0.870 0.8|72 0.893|0.880|0.876|
|---|---|---|---|---|
|0.787|0.876
0.8|67
0.879|0.874|0.872|
|0.824|0.864
0.8|74
0.861|0.873|0.872|
|0.700
|0.850
0.8

|34
0.876

|0.785
|0.776
|
|~~8~~
~~24~~
~~4~~
0.814
0.851
0.8|~~8~~
~~24~~
~~4~~
0.814
0.851
0.8|~~96~~
3
0.863|~~192~~
~~400~~
0.838
0.616|~~192~~
~~400~~
0.838
0.616|
(a)
|2 0.8
6 0.7
8 N/
0.|0.8|85 0.876|0.883|0.886|0.880 0.|
|---|---|---|---|---|---|
|~~0.~~
2
6
8
0.8
0.7
N/|0.7|90
0.802|0.789|0.878|0.785
0.|
|~~0.~~
2
6
8
0.8
0.7
N/|0.7|~~0.2~~
~~0.4~~
A
0.871
0.873|~~0.2~~
~~0.4~~
A
0.871
0.873|~~0.6~~
~~0.8~~

0.881
0.873
0.|~~0.6~~
~~0.8~~

0.881
0.873
0.|
(b)
Figure 9: Hyperparameter sensitivity of ESSA on **Qwen2.5-7B** for **GSM8K** . Batch size 100. **(a)**
Accuracy when varying LoRA rank and population size. **(b)** For each LoRA rank, the population
size is fixed to the best value found in (a), while the percentage _α_ of trainable singular values is
varied. This illustrates how ESSA performance depends jointly on adapter rank and the fraction
of singular values optimized. The single white cell occurs because for LoRA rank 8 and _α_ = 0 _._ 1,
rounding down yields zero trainable singular values, so no valid accuracy is reported.
Figure 9 examines the hyperparameter sensitivity of ESSA on Qwen2.5-7B for GSM8K. Varying the
LoRA rank and population size shows that performance remains stable across a broad range, with
accuracy peaking at 0.893 for LoRA rank 2 and population size 96. Notably, even very low-ranks (24) achieve top performance, while larger ranks yield diminishing or slightly degraded results. When
fixing the population size and varying the fraction _α_ of trainable singular values, ESSA maintains
consistently high accuracy ( _≈_ 0.87-0.89) across all _α_ values. This indicates strong robustness to the
degree of SVD sparsification.
|0.704|0.732 0.|748|0.44|4 0.576|
|---|---|---|---|---|
|0.434|0.688
0.|717|0.43|6
0.502|
|0.448
|0.620
0

|.535
|0.44
|8
0.454

|
|0.474
|0.556
0

|460
|0.42
|6
0.454

|
|0.434
0.398
0|0.434
0.398
0|398
0.42|398
0.42|8
0.378|
(a)
|32|0.414|0.434|0.418|0.410 0.368|0.434|
|---|---|---|---|---|---|
|2|0.468|0.460|0.646|0.610
0.544|0.556|
|2|0.468|0.460|0.646|0.610
0.544|0.620|
(b)
Figure 10: Hyperparameter sensitivity of ESSA on **Qwen2.5-7B** for **PRM800K** . Batch size 300. **(a)**
Accuracy when varying LoRA rank and population size. **(b)** For each LoRA rank, the population
size is fixed to the best value found in (a), while the percentage _α_ of trainable singular values is
varied. This illustrates how ESSA performance depends jointly on adapter rank and the fraction
of singular values optimized. The single white cell occurs because for LoRA rank 8 and _α_ = 0 _._ 1,
rounding down yields zero trainable singular values, so no valid accuracy is reported.
Figure 10 analyzes the hyperparameter sensitivity of ESSA on Qwen2.5-7B for PRM800K. When
varying LoRA rank and population size, accuracy peaks at 0.748 for LoRA rank 2 and population
size 48, showing that small adapter ranks with moderate population sizes are most effective. Performance drops notably for larger ranks (16) or oversized populations (96), indicating diminishing
Preprint
returns beyond a compact search space. With population size fixed, varying the fraction _α_ of trainable singular values shows that moderate values ( _α ≈_ 0.4-0.8) yield the best results – up to 0.696
accuracy for LoRA rank 8 – while very low or full updates slightly reduce performance. These
results confirm that ESSA performs best with low-rank adapters, moderate population sizes, and
partial singular value optimization.
|2 0.722|0.784 0|.598|0.604|0.558|
|---|---|---|---|---|
|4
0.536|0.550
0|.542|0.557|0.531|
|8
6
0.568
0.494
|0.558
0

|.616
|0.532
|0.498
|
|8
6
0.568
0.494
|0.524
0

|.564
|0.494
|0.50
|
|2
0.484
0.466
0|2
0.484
0.466
0|.484
0.476
0.48|.484
0.476
0.48|.484
0.476
0.48|
(a)
|0.484|0.496|0.512|0.435|0.490|0.484|
|---|---|---|---|---|---|
|0.478|0.486|0.498|0.528|0.500|0.564|
(b)
Figure 11: Hyperparameter sensitivity of ESSA on **Qwen2.5-Math-7B** for **PRM800K** . Batch size
300. **(a)** Accuracy when varying LoRA rank and population size. **(b)** For each LoRA rank, the
population size is fixed to the best value found in (a), while the percentage _α_ of trainable singular
values is varied. This illustrates how ESSA performance depends jointly on adapter rank and the
fraction of singular values optimized. The single white cell occurs because for LoRA rank 8 and
_α_ =0 _._ 1, rounding down yields zero trainable singular values, so no valid accuracy is reported.
Figure 11 presents the hyperparameter sensitivity of ESSA on Qwen2.5-Math-7B for PRM800K.
When varying LoRA rank and population size, accuracy peaks at 0.784 for LoRA rank 2 and population size 24, showing that compact low-rank adapters with small populations perform best. Higher
ranks or very large populations lead to gradual accuracy degradation, indicating that excessive search
dimensionality does not improve performance. Fixing population size and varying the fraction _α_ of
trainable singular values, ESSA achieves its highest score (0.616) at LoRA rank 8 and full update
( _α_ =1 _._ 0), while intermediate _α_ values yield competitive but slightly lower results. The results highlight that ESSA performs optimally with lightweight low-rank configurations and remains robust to
SVD sparsification, even on complex reasoning tasks.
|2 0.433 0.616 0
4 0.499 0.558 0
8 0.505 0.614 0
6 0.500 0.513 0
2 0.403 0.486 0|0.616 0|.603|0.579|0.571|
|---|---|---|---|---|
|2
4
8
6
2
0.433
0.616
0
0.499
0.558
0
0.505
0.614
0
0.500
0.513
0
0.403
0.486
0|0.558
0|.575|0.593|0.566|
|2
4
8
6
2
0.433
0.616
0
0.499
0.558
0
0.505
0.614
0
0.500
0.513
0
0.403
0.486
0|0.614
0|.606|0.599|0.610|
|2
4
8
6
2
0.433
0.616
0
0.499
0.558
0
0.505
0.614
0
0.500
0.513
0
0.403
0.486
0|0.513
0|.591|0.579|0.556|
|2
4
8
6
2
0.433
0.616
0
0.499
0.558
0
0.505
0.614
0
0.500
0.513
0
0.403
0.486
0|0.513
0|.499
0.560
0.495|.499
0.560
0.495|.499
0.560
0.495|
Figure 12: Hyperparameter sensitivity of ESSA on **LLaMA-3.1-8B** for **IFeval** . Accuracy when
varying LoRA rank and population size.
Figure 12 shows the hyperparameter sensitivity of ESSA on LLaMA-3.1-8B for IFeval. Varying
LoRA rank and population size reveals stable performance across configurations, with the highest
accuracy (0.614) achieved at LoRA rank 8 and population size 24. Low ranks (4-16) yield comparable results, while very high-rank (32) adapters slightly underperform, suggesting a sweet spot
around low-range ranks and moderate population sizes.
Preprint
C.2 TRAINING DYNAMICS
C.2.1 QWEN2.5-7B ON GSM8K
Appendix Figures 13, 14 and 15 provide the complete training-dynamic study of ESSA on Qwen2.57B for GSM8K. This analysis shows how validation accuracy evolves over time as we vary the three
key ESSA hyperparameters (Figure 9).
(d) LoRA rank 16
(a) LoRA rank 2
(b) LoRA rank 4
(c) LoRA rank 8
(e) LoRA rank 32
Figure 13: Validation accuracy over time on **GSM8K** with **Qwen2.5-7B** when varying the ESSA
population size (8, 24, 48, 96, 192, 400) at fixed LoRA ranks (2, 4, 8, 16, 32). Batch size 100.
Figure 13 shows the full training dynamics of ESSA on GSM8K with Qwen2.5-7B when varying
the ESSA population size.
Preprint
(a) Population size 8
(b) Population size 24
(c) Population size 48
(d) Population size 96
(e) Population size 192
(f) Population size 400
Figure 14: Validation accuracy over time on **GSM8K** with **Qwen2.5-7B** when varying the LoRA
rank (2, 4, 8, 16, 32) at fixed population sizes (8, 24, 48, 96, 192, 400). Batch size 100.
Figure 14 shows the full training dynamics of ESSA on GSM8K with Qwen2.5-7B when varying
the LoRA rank.
(a) LoRA rank 8, Population size 48
|Col1|Col2|
|---|---|
|||
|||
|||
(b) LoRA rank 16, Population size 96
(c) LoRA rank 32, Population size 96
Figure 15: Validation accuracy on **GSM8K** with **Qwen2.5-7B** while varying _α_ . LoRA rank and
population size are fixed to the optimal choices from Figure 9. Batch size 100.
Preprint
Figure 15 shows the full training dynamics of ESSA on GSM8K with Qwen2.5-7B when varying
the fraction _α_ of trainable singular values.
C.2.2 QWEN2.5-MATH-7B ON GSM8K
Appendix Figures 16, 17 and 18 provide the complete training-dynamic study of ESSA on Qwen2.5Math-7B for GSM8K. These graphs complement the main text sensitivity analysis by showing how
validation accuracy evolves over time as we vary the three key ESSA hyperparameters (Figure 2).
(c) LoRA rank 8
(d) LoRA rank 16
(a) LoRA rank 2
(b) LoRA rank 4
(e) LoRA rank 32
Figure 16: Validation accuracy over time on **GSM8K** with **Qwen2.5-Math-7B** when varying the
ESSA population size (8, 24, 48, 96, 192, 400) at fixed LoRA ranks (2, 4, 8, 16, 32). Batch size
100.
Figure 16 shows the full training dynamics of ESSA on GSM8K with Qwen2.5-Math-7B when
varying the ESSA population size.
Preprint
(f) Population size 400
(e) Population size 192
(a) Population size 8
(b) Population size 24
(c) Population size 48
(d) Population size 96
Figure 17: Validation accuracy over time on **GSM8K** with **Qwen2.5-Math-7B** when varying the
LoRA rank (2, 4, 8, 16, 32) at fixed population sizes (8, 24, 48, 96, 192, 400). Batch size 100.
Figure 17 shows the full training dynamics of ESSA on GSM8K with Qwen2.5-Math-7B when
varying the the LoRA rank.
(a) LoRA rank 8, Population size 48
(b) LoRA rank 16, Population size 48
(c) LoRA rank 32, Population size 96
Figure 18: Validation accuracy on **GSM8K** with **Qwen2.5-Math-7B** while varying _α_ . LoRA rank
and population size are fixed to the optimal choices from Figure 2. Batch size 100.
Preprint
Figure 18 shows the full training dynamics of ESSA on GSM8K with Qwen2.5-Math-7B when
varying the fraction _α_ of trainable singular values.
C.2.3 QWEN2.5-7B ON PRM800K
Appendix Figures 19, 20 and 21 provide the complete training-dynamic study of ESSA on Qwen2.57B for PRM800K. This analysis shows how validation accuracy evolves over time as we vary the
three key ESSA hyperparameters (Figure 10)
(a) Population size 8
(b) Population size 24
(c) Population size 48
(d) Population size 96
(e) Population size 192
Figure 19: Validation accuracy over time on **PRM800K** with **Qwen2.5-7B** when varying the ESSA
population size (8, 24, 48, 96, 192) at fixed LoRA ranks (2, 4, 8, 16, 32). Batch size 300.
Figure 19 shows the full training dynamics of ESSA on PRM800K with Qwen2.5-7B when varying
the ESSA population size.
Preprint
(d) LoRA rank 16
(a) LoRA rank 2
(b) LoRA rank 4
(c) LoRA rank 8
(e) LoRA rank 32
Figure 20: Validation accuracy over time on **PRM800K** with **Qwen2.5-7B** when varying the LoRA
rank (2, 4, 8, 16, 32) at fixed population sizes (8, 24, 48, 96, 192). Batch size 300.
Figure 20 shows the full training dynamics of ESSA on PRM800K with Qwen2.5-7B when varying
the LoRA rank.
(b) LoRA rank 16, Population size 24
(a) LoRA rank 8, Population size 24
(c) LoRA rank 32, Population size 8
Figure 21: Validation accuracy over time on **PRM800K** with **Qwen2.5-7B** while varying _α_ . LoRA
rank and population size are fixed to the optimal choices from Figure 10. Batch size 300.
Preprint
Figure 21 shows the full training dynamics of ESSA on PRM800K with Qwen2.5-7B when varying
the fraction _α_ of trainable singular values.
C.2.4 QWEN2.5-MATH-7B ON PRM800K
Appendix Figures 22, 23 and 24 provide the complete training-dynamic study of ESSA on Qwen2.5Math-7B for PRM800K. These graphs complement the main text sensitivity analysis by showing
how validation accuracy evolves over time as we vary the three key ESSA hyperparameters
(Figure 11).
(a) Population size 8
(b) Population size 24
(c) Population size 48
(d) Population size 96
(e) Population size 192
Figure 22: Validation accuracy over time on **PRM800K** with **Qwen2.5-Math-7B** when varying the
ESSA population size (8, 24, 48, 96, 192) at fixed LoRA ranks (2, 4, 8, 16, 32). Batch size 300.
Figure 22 shows the full training dynamics of ESSA on PRM800K with Qwen2.5-Math-7B when
varying the ESSA population size.
Preprint
(a) LoRA rank 2
(b) LoRA rank 4
(c) LoRA rank 8
(d) LoRA rank 16
(e) LoRA rank 32
Figure 23: Validation accuracy over time on **PRM800K** with **Qwen2.5-Math-7B** when varying the
LoRA rank (2, 4, 8, 16, 32) at fixed population sizes (8, 24, 48, 96, 192). Batch size 300.
Figure 23 shows the full training dynamics of ESSA on PRM800K with Qwen2.5-Math-7B when
varying the LoRA rank.
(b) LoRA rank 16, Population size 48.
(a) LoRA rank 8, Population size 48.
(c) LoRA rank 32, Population size 8.
Figure 24: Validation accuracy over time on **PRM800K** with **Qwen2.5-Math-7B** when varying _α_ .
LoRA rank and population size are fixed to the optimal choices from Figure 11. Batch size 300.
Preprint
Figure 24 shows the full training dynamics of ESSA on PRM800K with Qwen2.5-Math-7B when
varying the fraction _α_ of trainable singular values.
C.2.5 LLAMA3.1-8B ON IFEVAL
Appendix Figures 25 and 26 provide the complete training-dynamic study of ESSA on LLaMA3.18B for IFEval. Two graphs complement the main text sensitivity analysis by showing how validation
accuracy evolves over time as we vary the two key ESSA hyperparameters (Figure 12)
(a) Population size 8
(b) Population size 24
(c) Population size 48
(d) Population size 96
(e) Population size 192
Figure 25: Validation accuracy over time on **IFEval** with **LLaMA3.1-8B** when varying the ESSA
population size (8, 24, 48, 96, 192) at fixed LoRA ranks (2, 4, 8, 16, 32). Batch size 500, _α_ =1 _._ 0.
Figure 25 shows the full training dynamics of ESSA on IFEval with LLaMA3.1-8B when varying
the ESSA population size.
Preprint
(d) LoRA rank 16
(c) LoRA rank 8
(a) LoRA rank 2
(b) LoRA rank 4
(e) LoRA rank 32
Figure 26: Validation accuracy over time on **IFEval** with **LLaMA3.1-8B** when varying the LoRA
rank (2, 4, 8, 16, 32) at fixed population sizes (8, 24, 48, 96, 192). Batch size 500, _α_ =1 _._ 0.
Figure 26 shows the full training dynamics of ESSA on IFEval with LLaMA3.1-8B when varying
the LoRA rank.
D PRECISION ANALYSIS
Figure 27 shows the full training curves of ESSA when running Qwen2.5-32B on PRM800K with
LoRA rank 8, population size 64, _α_ = 1 _._ 0, and per-candidate batch size 256, for three different
numerical precisions: BFLOAT16, INT8, and INT4. Shaded areas indicate one standard deviation
across runs.
Figure 27: Validation accuracy over training time for Qwen2.5-32B on PRM800K under different
weight precisions (BFLOAT16, INT8, INT4). Settings: LoRA rank 8, pop. 64, batch size 256,
_α_ =1 _._ 0.
Across all precisions, accuracy rises quickly within the first hour and then remains stable. Reducing
precision from BFLOAT16 to INT8 and even to INT4 produces only a slight reduction in final accu
Preprint
racy, consistent with the summary in Table 2 of the main text. These results confirm that ESSA can
safely exploit aggressive quantization to cut memory usage and computation cost while maintaining
near-identical performance.
E TOY EXAMPLE ON MNIST
A natural concern arises when considering a singular-value-only parameterization. If we restrict the
problem to simple linear or logistic regression, scaling the rows of a fixed low-rank update may seem
too limited to support meaningful adaptation. In this setting only the singular magnitudes change
while all directions remain fixed, which can give the impression of insufficient flexibility.
Motivated by this concern, we emphasize that ESSA’s effectiveness in LLMs relies on the structured
representations already present in the pretrained backbone and inherited through the downstream
SFT initialization. To make this distinction concrete, we designed an MNIST experiment that replicates the LLM training pipeline as closely as possible within a minimal linear model.
(a) _W_ = _W_ _[′]_ + ∆ _W_
(b) _W_ = _W_ _[′]_
(c) _W ∼N_ (0 _,_ _n_ [2] [)]
Figure 28: Validation accuracy over time on **MNIST** (on the test set of digits 6-9) with a single
matrix **W** _∈R_ _[n][×][m]_, _n_ = 784, _m_ = 10. Panels (a)-(c) correspond to _W_ = _W_ _[′]_ + ∆ _W_, _W_ = _W_ _[′]_
and _W ∼N_ (0 _,_ _n_ [2] [)][ setups respectively, where] _[ W][ ′]_ [ is pretrained base weights. Train size 400. ESSA]
(blue): solution length 10, population size 24, batch size 64. SGD (red): lr 0 _._ 1, batch size 64.
Toy example on MNIST proceeds as follows:
1. At the pretrain stage (analog of LLM pretraining) we train a logistic regression model
on MNIST digits 0-5 (train size 5000) using SGD optimizer. We obtain a ”base model”
_W_ = _W_ _[′]_, where _W_ _[′]_ is the single matrix of size 784 by 10, that knows something about
the data distribution, similar to LLM pretraining.
2. At the ”SFT” stage (analogue of LoRA SFT) we take 400 samples of digits 6-9, initialize
a matrix update ∆ _W_ with Kaiming init, and train only ∆ _W_ using SGD: _W_ = _W_ _[′]_ + ∆ _W_,
where ∆ _W_ is the single matrix of size 784 by 10.
3. At the last stage we decomposed ∆ _W_ = _BSA_ and train only the singular values _S_ using
ES or SGD optimizer. ESSA optimize singular values directly using train accuracy, while
the SGD baseline optimized them through the standard cross-entropy loss.
In Figure 28a, ESSA follows the behavior of SGD on the MNIST task with high fidelity. This
demonstrates that once the singular vectors encodes useful directions, modulating only the associated singular amplitudes is already sufficient to achieve effective adaptation. In Figure 28b, we
remove the additive update entirely. We decompose the pretrained weights _W_ = _W_ _[′]_ = _B_ _[′]_ _S_ _[′]_ _A_ _[′]_,
and train only the singular values _S_ _[′]_ using ESSA and SGD. Even in this more constrained setup,
ESSA achieved performance comparable to SGD. The pretrained matrix contained enough structure
that rebalancing its singular directions remained a viable strategy for improving accuracy. Finally, in
Figure 28c, we examine the case without pretraining or SFT, where the model is initialized with and
the singular vectors carry no semantic information. In this setup ESSA improves only slowly, while
SGD makes substantially faster progress. Without a structured subspace, changing only singular
values is insufficient to emulate full learning dynamics.
These experiments demonstrate that the singular-value parameterization is effective precisely in the
settings where LLMs operate: pretrained models with downstream SFT produce meaningful lowrank directions, and ESSA leverages this structure by adjusting their relative scaling. This behavior
is consistent with the results observed in our large-scale LLM experiments.
Preprint
F COMPARISON TO GRPO
F.1 SCHOOL MATH
Appendix Figure 29 presents the results of Qwen2.5-7B on GSM8K in addition to Qwen2.5-Math7B (Figure 4).
|0.9
0.8
Accuracy
0.7
0.6
0.5
0 20 4|Col2|Col3|Col4|Col5|
|---|---|---|---|---|
|~~0~~
~~20~~
~~4~~
0.5
0.6
0.7
0.8
0.9
Accuracy||||~~GR~~
|
|~~0~~
~~20~~
~~4~~
0.5
0.6
0.7
0.8
0.9
Accuracy||||~~ESS~~|
|Col1|Col2|
|---|---|
|~~ RA ra~~
|~~  nk = 8~~
|
|~~ A ran~~|~~  k = 8~~|
|~~00~~
|~~120~~|
(a) Population size 96, _α_ =0 _._ 6
(b) Population size 96, _α_ =0 _._ 6
(c) Population size 48, _α_ =0 _._ 6
(d) Population size 96, _α_ =1 _._ 0
(e) Population size 96, _α_ =1 _._ 0
Figure 29: Validation accuracy over time on **GSM8K** with **Qwen2.5-7B** . Panels (a)-(e) correspond
to LoRA ranks 32, 16, 8, 4, and 2, respectively. ESSA (blue): batch size 100. GRPO (red): lr
1 _×_ 10 _[−]_ [5], global batch 512, mini batch 64.
Appendix Figure 34 presents the results of Qwen2.5-7B and Qwen2.5-Math-7B on GSM8K without SFT warm-start in addition to School Math experiments (Section 4.4.1). LoRA matrix _A_ is
initialized using Kaiming initialization, matrix _B_ is zero.
(a) Population size 96, _α_ =1 _._ 0
(b) Population size 96, _α_ =1 _._ 0
Figure 30: Validation accuracy over time on **GSM8K** with **Qwen2.5-7B** (a) and **Qwen2.5-Math-7B**
(b) without SFT warm-start. ESSA (blue): train size 100. GRPO (red): lr 1 _×_ 10 _[−]_ [5], global batch
512, mini batch 64.
Appendix Figure 31 presents the results of Qwen2.5-3B on one GPU in addition to School Math
experiments (Section 4.2).
Figure 31: Validation accuracy over time on **GSM8K** with **Qwen2.5-3B** on one GPU. ESSA (blue):
population size 48, _α_ = 1 _._ 0, batch size 100. GRPO (red): lr 1 _×_ 10 _[−]_ [5], global batch 512, mini batch
64.
Preprint
F.2 BEYOND SCHOOL MATH
Appendix Figures 32 and 33 present the full results of our beyond school math experiments on
PRM800K.
(a) Population size 8, _α_ =0 _._ 4
(b) Population size 48, _α_ =1 _._ 0
(c) Population size 48, _α_ =1 _._ 0
(e) Population size 8, _α_ =1 _._ 0
(d) Population size 96, _α_ =1 _._ 0
Figure 32: Validation accuracy over time on **PRM800K** with **Qwen2.5-Math-7B** . Panels (a)-(e)
correspond to LoRA ranks 32, 16, 8, 4, and 2, respectively. ESSA (blue): batch size 300. GRPO
(red): lr 1 _×_ 10 _[−]_ [5], global batch 512, mini batch 64.
(a) Population size 8, _α_ =0 _._ 2
(b) Population size 24, _α_ =0 _._ 4
(c) Population size 24, _α_ =0 _._ 8
(d) Population size 48, _α_ =1 _._ 0
(e) Population size 24, _α_ =1 _._ 0
Figure 33: Validation accuracy over time on **PRM800K** with **Qwen2.5-7B** . Panels (a)-(e) correspond to LoRA ranks 32, 16, 8, 4, and 2, respectively. ESSA (blue): batch size 300. GRPO (red): lr
1 _×_ 10 _[−]_ [5], global batch 512, mini batch 64.
We evaluate both Qwen2.5-Math-7B and Qwen2.5-7B, sweeping LoRA ranks and comparing ESSA
with GRPO. Across both model variants on GSM8K (Section 4.4.1, Figure 29) and PRM800K
(Section 4.4.2, Section F.2) ESSA climbs to high accuracy much more rapidly than GRPO and often
reaches a higher plateau, particularly when the LoRA rank is small or moderate. In experiments
without SFT (Figure 34) runs starting from a random LoRA achieve lower absolute accuracy. The
relative performance between ESSA and GRPO remains consistent. Experiment with Qwen2.5-3B
(Figure 31) on one GPU confirms that ESSA’s benefits are not restricted to multi-GPU setups, but
persist even when only one device is available.
Preprint
G OOD DRIFT
To assess the robustness of ESSA under distribution shift, we conduct an additional set of experiments that instantiate exactly the cross-domain setting proposed in the review. We select two datasets
with markedly different reward structures: HelpSteer2 and IFEval. HelpSteer2 encourages generalpurpose response quality, while IFEval enforces rigid constraint satisfaction. Moving between them
therefore offers a direct test of how well ESSA adapts when the SFT warm-start and the target alignment domain differ. All HelpSteer2 experiments use the same reward model as in the main paper
(RLHFlow/ArmoRM-Llama3-8B-v0.1). All IFEval experiments use if-eval-like dataset as a train
dataset.
(a) LoRA rank 8, population size 48, _α_ =1 _._ 0
(b) LoRA rank 8, population size 36, _α_ =1 _._ 0
Figure 34: Validation metrics over time on **IFEval** (a) and **HelpSteer2** (b) with **Llama3.1-8B** . (a)
IFEval _→_ IFEval (pink), HelpSteer2 _→_ IFEval (green), base model _→_ IFEval (brown) experiments,
batch size 500. (b): HelpSteer2 _→_ HelpSteer2 (pink), IFEval _→_ HelpSteer2 (green), base model _→_
HelpSteer2 (brown) experiments, batch size 100.
For each target dataset B we evaluate three configurations. The first performs SFT on dataset A and
ESSA on B (A _→_ B). The second performs both SFT and ESSA on B (B _→_ B). The third applies
ESSA directly to B without any SFT warm-start (Base model _→_ B), initialized with default LoRA
parameters (Kaiming-uniform for weight A and zeros for weight B).
Across both directions, ESSA reliably improves performance on the target dataset, even when the
SFT checkpoint originates from a mismatched domain. In the IFEval to HelpSteer2 condition the
gap between cross-domain and in-domain SFT is modest, indicating that the structured initialization
learned from if-eval-like dataset already provides useful axes for ESSA to exploit when optimizing
for HelpSteer2. In the opposite direction the gap is larger, which is expected given that HelpSteer2
does not teach the model the strict formatting and constraint-following behavior required by IFEval.
Nevertheless ESSA exhibits stable monotonic improvement and substantially surpasses the Base _→_
IFEval configuration that starts from random LoRA parameters. In both directions, SFT warm-starts
clearly outperform training from scratch, confirming that even an out-of-distribution SFT checkpoint
supplies meaningful structure for ESSA’s evolutionary search.
These results demonstrate that ESSA maintains its effectiveness under distribution shift and that the
method can leverage any reasonable SFT initialization, even when it originates from a domain with
incompatible reward structure.
H IMPLEMENTATION DETAILS
H.1 DATASETS


GSM8K 1495 5978 1319
PRM800K 3600 7900 500
IFEval 1700 3414 541
HelpSteer 19000 1324 1024
Table 5: Dataset sizes across all experiments.
Preprint

ESSA: Evolutionary Strategies for Scalable Alignment

Dataset SFT Train Size ESSA/GRPO Train Size Validation

Daria Korotyshova
Boris Shaposhnikov
Alexey Malakhov
Alexey Khokhulin
Nikita Surnachev
Kirill Ovcharenko
George Bredis
Alexey Gorbatovski
Viacheslav Sinii
Daniil Gavrilov


- **Math-Reasoning:** Please reason step by step and place your
final answer in _\_ boxed _{}_ .
    - **Instruction-Following:** You are a helpful, honest, and concise
assistant.
H.2 SFT HYPERPARAMETERS


Qwen2.5-7B (GSM8K) 16 1 _×_ 10 _[−]_ [4] 1 0.18
Qwen2.5-Math-7B (GSM8K) 16 1 _×_ 10 _[−]_ [4] 1 0.16
Qwen2.5-7B (PRM800K) 16 5 _×_ 10 _[−]_ [4] 1 0.43
Qwen2.5-Math-7B (PRM800K) 16 5 _×_ 10 _[−]_ [4] 1 0.40
Qwen2.5-32B (PRM800K) 128 1 _×_ 10 _[−]_ [6] 3 9.80
Qwen2.5-72B (PRM800K) 128 1 _×_ 10 _[−]_ [6] 3 51.97
Llama3.1-8B (IF-Eval-like) 16 5 _×_ 10 _[−]_ [4] 1 0.50
Llama3.1-8B (HelpSteer2) 128 1 _×_ 10 _[−]_ [6] 3 3.74
Table 6: Summary of hyperparameters used for SFT across all experiments.
H.3 ESSA HYPERPARAMETERS
For ESSA, we performed a hyperparameter grid search covering various values of LoRA rank, population size, fraction of singular values _α_, batch size, and precision (Table 7). For each experiment,
the best ESSA hyperparameters were selected based on validation performance (Table 8). We use
up to 128 NVIDIA H100 80GB GPUs per experiment.

Model and Dataset SFT Batch Size Learning Rate Epochs SFT GPU Hours


LoRA Rank 2, 4, 8, 16, 32 8
Population Size 8, 24, 36, 48, 64, 96, 192, 400 96
Fraction of Singular Values ( _α_ ) 0.1, 0.2, 0.4, 0.6, 0.8, 1.0 1.0
Batch Size 100, 256, 300, 500 100
Precision INT4, INT8, BFLOAT16 BFLOAT16
Table 7: ESSA hyperparameter grid and default values for ablation studies.
**Experiment** **LoRA Rank** **Population Size** **Batch Size** **Fraction** _α_
GSM8K (Qwen2.5-Math-7B) 32 96 100 0.1
GSM8K (Qwen2.5-7B) 2 96 100 1.0
PRM800K (Qwen2.5-Math-7B) 2 24 300 1.0
PRM800K (Qwen2.5-7B) 2 48 300 1.0
PRM800K (Qwen2.5-32B) 8 48 256 1.0
PRM800K (Qwen2.5-72B, INT4) 4 64 256 1.0
IFEval (LLaMA3.1-8B) 8 24 500 1.0
HelpSteer (LLaMA3.1-8B) 8 36 100 1.0
Table 8: Best ESSA hyperparameters for each experiment.


Haoran Lu [1], Luyang Fang [1], Ruidong Zhang [2], Xinliang Li [2], Jiazhang Cai [1], Huimin Cheng [3],
Lin Tang [3], Ziyu Liu [1], Zeliang Sun [4], Tao Wang [1], Yingchuan Zhang [1], Arif Hassan Zidan [5],
Jinwen Xu [6], Jincheng Yu [7], Meizhi Yu [1], Hanqi Jiang [2], Xilin Gong [1], Weidi Luo [2], Bolun
Sun [8], Yongkai Chen [9], Terry Ma [10], Shushan Wu [1], Yifan Zhou [2], Junhao Chen [2], Haotian
Xiang [6], Jing Zhang [11], Afrar Jahin [5], Wei Ruan [2], Ke Deng [2], Yi Pan [2], Peilong Wang [12], Jiahui
Li [7], Zhengliang Liu [2], Lu Zhang [13], Xiaobo Li [14], Lin Zhao [14], Wei Liu [12], Dajiang Zhu [11], Xin
Xing [15], Fei Dou [7], Wei Zhang [5], Chao Huang [4], Rongjie Liu [1], Mengrui Zhang [16], Yiwen Liu [17],
Xiaoxiao Sun [17], Qin Lu [6], Zhen Xiang [2], Wenxuan Zhong [*1], Tianming Liu [*2], and Ping Ma [*1]
1Department of Statistics, University of Georgia, Athens, GA
2School of Computing, University of Georgia, Athens, GA
3Department of Biostatistics, Boston University, Boston, MA
4Department of Epidemiology & Biostatistics, University of Georgia, Athens, GA
5School of Computer and Cyber Sciences, Augusta University, Augusta, GA
6School of Electrical and Computer Engineering, University of Georgia, Athens, GA
7Department of Statistics & Data Science, University of Arizona, Tucson, AZ
8Kellogg School of Management, Northwestern University, Evanston, IL
9Department of Statistics, Harvard University, Cambridge, MA
10School of Computer Science, Carnegie Mellon University, Pittsburgh, PA
11School of Computer Science and Engineering, University of Texas at Arlington, TX
12Department of Radiation Oncology, Mayo Clinic Arizona, Phoenix, AZ
13Computer Science Department, Indiana University Indianapolis, IN
14Department of Biomedical Engineering, New Jersey Institute of Technology, NJ
15Department of Statistics, Virginia Tech, Blacksburg, VA
16Quantitative Sciences Unit, Department of Medicine, Stanford University, Stanford, CA
17Department of Epidemiology and Biostatistics, Mel and Enid Zuckerman College of
Public Health, University of Arizona, Tucson, AZ
- **Corresponding author(s).** [E-mail(s): wenxuan@uga.edu; tliu@uga.edu; pingma@uga.edu](mailto:wenxuan@uga.edu)

Alignment and Safety in Large Language Models: Safety Mechanisms,
  Training Paradigms, and Emerging Challenges

Haoran Lu
Luyang Fang
Ruidong Zhang
Xinliang Li
Jiazhang Cai
Huimin Cheng
Lin Tang
Ziyu Liu
Zeliang Sun
Tao Wang
Yingchuan Zhang
Arif Hassan Zidan
Jinwen Xu
Jincheng Yu
Meizhi Yu
Hanqi Jiang
Xilin Gong
Weidi Luo
Bolun Sun
Yongkai Chen
Terry Ma
Shushan Wu
Yifan Zhou
Junhao Chen
Haotian Xiang
Jing Zhang
Afrar Jahin
Wei Ruan
Ke Deng
Yi Pan
Peilong Wang
Jiahui Li
Zhengliang Liu
Lu Zhang
Lin Zhao
Wei Liu
Dajiang Zhu
Xin Xing
Fei Dou
Wei Zhang
Chao Huang
Rongjie Liu
Mengrui Zhang
Yiwen Liu
Xiaoxiao Sun
Qin Lu
Zhen Xiang
Wenxuan Zhong
Tianming Liu
Ping Ma


Due to the remarkable capabilities and growing impact of large language models (LLMs),
they have been deeply integrated into many aspects of society. Thus, ensuring their alignment
with human values and intentions has emerged as a critical challenge. This survey provides
a comprehensive overview of practical alignment techniques, training protocols, and empirical findings in LLM alignment. We analyze the development of alignment methods across
diverse paradigms, characterizing the fundamental trade-offs between core alignment objectives. Our analysis shows that while supervised fine-tuning enables basic instruction-following,
preference-based methods offer more flexibility for aligning with nuanced human intent. We
discuss state-of-the-art techniques, including Direct Preference Optimization (DPO), Constitutional AI, brain-inspired methods, and alignment uncertainty quantification (AUQ), highlighting
their approaches to balancing quality and efficiency. We review existing evaluation frameworks
and benchmarking datasets, emphasizing limitations such as reward misspecification, distributional robustness, and scalable oversight. We summarize strategies adopted by leading AI labs
to illustrate the current state of practice. We conclude by outlining open problems in oversight, value pluralism, robustness, and continuous alignment. This survey aims to inform both
researchers and practitioners navigating the evolving landscape of LLM alignment.


2.1 Safety Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1.1 The Foundational Role of Safety . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1.2 Categorization of Safety Harms . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1.3 The Evolution of Safety Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Secondary Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2.1 Defining Helpfulness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2.2 Defining Harmlessness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2.3 Defining Honesty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3 Balancing and Trade-offs Among Objectives . . . . . . . . . . . . . . . . . . . . . . . 10


3.1 Adversarial Attacks & Red-Teaming . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.1.1 Logic-Based Jailbreak Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.1.2 Low-Resource Jailbreak Attacks . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.1.3 Community-Driven and In-The-Wild Prompts . . . . . . . . . . . . . . . . . . 12
3.1.4 Fake Alignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.1.5 Jailbreak Competitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.2 Scoring Based Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.3 Benchmarks for Safety Alignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.3.1 General Safety Benchmark . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.3.2 Reasoning Safety Benchmark . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.3.3 Privacy Alignment Benchmark . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.3.4 Fairness Alignment Benchmark . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.3.5 Honesty Alignment Benchmark . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.3.6 Agent Safety Benchmark . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.3.7 Domain-Specific Safety Benchmark . . . . . . . . . . . . . . . . . . . . . . . . 17
3.3.8 Code Safety Benchmark . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3 Evaluation and Benchmarking of Alignment 10


4.1 Instruction Tuning with Human Demonstrations . . . . . . . . . . . . . . . . . . . . 21
4.2 Role of High-Quality Data and Coverage . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.3 Optimization Methods for SFT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.4 Limitations of SFT Alone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4 Supervised Fine-Tuning (SFT) for Alignment 21


5.1 Human Feedback Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
5.2 Reward Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
5.3 Policy Optimization Methods in RLHF . . . . . . . . . . . . . . . . . . . . . . . . . . 32
5.3.1 Actor-Critic PPO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.3.2 Actor-Only Policy Gradients . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.3.3 Specialized and Hybrid Reward-Based Policy Optimization . . . . . . . . . . 39
5.4 Challenges of RLHF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

5 Reinforcement Learning from Human Feedback (RLHF) 25


6.1 Fundamental Differences between SFT and RLHF . . . . . . . . . . . . . . . . . . . 43
6.2 When SFT and RLHF Overlap or Converge . . . . . . . . . . . . . . . . . . . . . . . 45
6.3 Integrating SFT and RLHF in Training Pipelines . . . . . . . . . . . . . . . . . . . . 46

6 SFT versus RLHF: Differences, Equivalences, and Hybrid Approaches 43


7.1 Direct Preference Optimization and Reward-Free Methods . . . . . . . . . . . . . . . 47
7.2 AI-Assistant Alignment and Self-Alignment . . . . . . . . . . . . . . . . . . . . . . . 48
7.2.1 AI-Assistant Alignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
7.2.2 Self-Alignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
7.2.3 Challenges and Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . 49
7.3 Multi-Agent and Deliberative Alignment Approaches . . . . . . . . . . . . . . . . . . 50
7.4 Group Relative Policy Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

7 Advanced Alignment Techniques and Recent Innovations 47


8.1 Full or Partial Parameters Fine-Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . 54
8.2 Low-Rank Adaptation (LoRA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
8.3 Sparse Fine-Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
8.4 Knowledge Distillation for Fine-Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . 57
8.5 Adapter-Based Fine-Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
8.6 Comparison of Fine-Tuning Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . 60

8 Efficient Fine-Tuning Techniques for Alignment 54


9.1 Recent advancements of Brain-Inspired LLM Alignments . . . . . . . . . . . . . . . . 61
9.2 Brain-AGI Co-working . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
9.3 Challenges and Limitations of Brain-Inspired LLM Alignments . . . . . . . . . . . . 64
9.4 Opportunities and Future Developments . . . . . . . . . . . . . . . . . . . . . . . . . 64


10.1 Sources of Alignment Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
10.2 Conceptual Framework and Methods for Quantifying Alignment Uncertainty . . . . 66
10.2.1 Conceptual Framework for Alignment Uncertainty . . . . . . . . . . . . . . . 66
10.2.2 Methods for Quantifying Alignment Uncertainty . . . . . . . . . . . . . . . . 67
10.3 Robustness and Uncertainty in Alignment . . . . . . . . . . . . . . . . . . . . . . . . 69

10 Alignment Uncertainty Quantification (AUQ) 65


11.1 Ethical and Societal Implications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
11.2 Regulatory and Policy Landscape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
11.3 AGI/ASI safety . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

11 Societal, Ethical, and Regulatory Considerations 70


12.1 OpenAI o-Series Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
12.2 DeepSeek Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
12.3 Anthropic Claude Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
12.4 Google DeepMind Gemini Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
12.5 Meta’s LlaMA Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
12.6 Grok Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

12 Alignment Strategies Across Leading AI Models 74


13.1 Summary of Key Insights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
13.2 Open Research Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
13.3 Promising Research Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
13.4 Closing Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80


The alignment of large language models (LLMs) with human values, intentions, and preferences
represents one of the most critical challenges in contemporary artificial intelligence (AI) research.
As LLMs achieve unprecedented capabilities across diverse domains, from natural language understanding to complex reasoning tasks, ensuring their reliable and beneficial deployment has become
a fundamental requirement for the continued advancement of AI technology. The field of LLM
alignment has emerged from the intersection of theoretical AI safety research and practical machine learning, evolving rapidly from speculative concerns to deployed systems that impact millions
of users worldwide.
The conceptual foundations of AI alignment trace back to early theoretical work in AI safety,
most notably Bostrom’s seminal analysis of superintelligence risks and the orthogonality thesis
[Mulgan, 2016], and the Machine Intelligence Research Institute’s foundational research on friendly
AI [Yudkowsky et al., 2008]. These early contributions established crucial theoretical frameworks,
defining the fundamental alignment challenge: ensuring that increasingly capable AI systems remain
aligned with human objectives as their capabilities scale. The translation of these abstract concerns
into concrete research directions was catalyzed by Amodei et al. [2016], which identified persistent
technical challenges, including reward hacking, scalable oversight, and distributional robustness,
that continue to guide contemporary research.
While transformer architecture [Vaswani et al., 2017] revolutionized natural language processing
(NLP) and enabled modern LLMs, their unprecedented scaling revealed new alignment challenges.
The subsequent progression from BERT [Devlin et al., 2019] through GPT-2 [Radford et al., 2019]
to GPT-3 [Brown et al., 2020] demonstrated extraordinary emergent capabilities that were not
explicitly programmed into these systems. GPT-3’s remarkable few-shot learning abilities and
general-purpose language understanding capabilities marked a paradigm shift, revealing that sufficiently large language models could exhibit behaviors and competencies far beyond their training
objectives. The discovery of emergent abilities at scale [Wei et al., 2022] and the establishment of
neural scaling laws [Kaplan et al., 2020] provided both a roadmap for future capability improvements and a compelling motivation for alignment research, as these developments suggested that
model behaviors could become increasingly difficult to predict and control.
The practical breakthrough in LLM alignment was achieved through the development of Reinforcement Learning from Human Feedback (RLHF), most prominently demonstrated in OpenAI’s
InstructGPT models [Ouyang et al., 2022]. This foundational work established the three-stage
alignment pipeline that has become the industry standard: supervised fine-tuning on human demonstrations, reward model training on human preference rankings, and policy optimization through
reinforcement learning. The dramatic empirical success of this approach, demonstrating that a 1.3billion parameter InstructGPT model was preferred over the 175-billion parameter GPT-3 despite
being over 100 times smaller, validated the hypothesis that alignment techniques could be more
important than raw computational scale for producing useful and safe AI systems.
Subsequent innovations have refined and extended these foundational techniques. Anthropic’s
Constitutional AI [Bai et al., 2022a] introduced a scalable approach to alignment through AIgenerated feedback guided by explicit constitutional principles, reducing dependence on human
annotation while improving harmlessness. The development of Direct Preference Optimization
[Rafailov et al., 2023] provided an elegant mathematical insight that the optimal RLHF policy could
be derived in closed form, eliminating the complex reinforcement learning pipeline and significantly
improving training stability and efficiency. These methodological advances have been complemented
by emerging research directions, including mechanistic interpretability, which aims to understand
the internal representations and computations of neural networks [Elhage et al., 2021], and scalable
oversight techniques designed to address the fundamental challenge of supervising AI systems that
may exceed human capabilities [Christiano et al., 2018].
Contemporary LLM alignment research is characterized by both remarkable progress and persistent challenges. Major AI research organizations, including OpenAI, Anthropic, Google DeepMind,
and Meta, have successfully deployed aligned language models at scale, demonstrating that alignment techniques can be effectively integrated into practical systems. However, significant limitations
remain: current alignment methods exhibit brittleness to adversarial attacks, suffer from distribution shift, and may not scale effectively to future systems with superhuman capabilities. The field
continues to grapple with fundamental questions about the nature of human values, the scalability
of human oversight, and the robustness of alignment techniques across diverse deployment contexts.
This survey provides a comprehensive examination of the current state of LLM alignment research, synthesizing theoretical foundations, practical techniques, and empirical findings across the
rapidly evolving field. We organize our analysis around the core technical challenges of alignment:
defining and measuring alignment objectives, developing effective training methodologies, ensuring
robustness and generalization, and scaling alignment techniques to increasingly capable systems.
Our coverage encompasses supervised fine-tuning approaches, reinforcement learning from human
feedback, constitutional and rule-based methods, preference optimization techniques, and emerging
directions including mechanistic interpretability and scalable oversight.
The structure of this survey reflects the multifaceted nature of the alignment challenge. Section
2 establishes the fundamental objectives that define aligned behavior and examines the complex
trade-offs between competing goals such as helpfulness, harmlessness, and honesty. Section 3 reviews evaluation methodologies and benchmarking approaches that enable systematic assessment
of alignment quality. Sections 4 and 5 provide a detailed analysis of the two dominant training
paradigms: supervised fine-tuning and reinforcement learning from human feedback, including their
theoretical foundations, practical implementations, and empirical performance. Section 6 examines
the relationships and complementary roles of these approaches within integrated training pipelines.
Advanced alignment techniques are covered in Section 7, including direct preference optimization, AI-assisted alignment, and multi-agent approaches. Section 8 reviews parameter-efficient
fine-tuning methods that enable scalable deployment of alignment techniques. Section 9 explores
emerging directions, including brain-inspired approaches and neurosymbolic methods. Section 10
addresses the critical challenge of uncertainty quantification in alignment, while Section 11 examines societal, ethical, and regulatory considerations. Section 12 surveys alignment strategies across
leading AI models, providing concrete case studies of successful deployment. Finally, Section 13
identifies open research challenges and future directions for the field.
Through this comprehensive analysis, we aim to provide both newcomers and experienced researchers with a unified understanding of the current state of LLM alignment research, its theoretical foundations, practical techniques, and future challenges. As LLMs continue to advance in
capability and deployment scope, the development of robust, scalable, and theoretically grounded
alignment techniques represents not merely a technical challenge but a fundamental requirement
for the beneficial development of artificial intelligence.


In this section, we formalize the fundamental objectives for LLM alignment and characterize the
optimization trade-offs inherent in multi-objective alignment frameworks. The alignment research
community has established a canonical tripartite objective function comprising Helpfulness, Harmlessness (Safety), and Honesty, which collectively define the feasible solution space for aligned
language models. Safety constitutes the primary constraint, ensuring model outputs satisfy nontoxicity, non-harm, and bias mitigation requirements while maintaining robustness against adversarial manipulation. Helpfulness encompasses the model’s capacity to function as an information retrieval system, domain-specific expert, general-purpose computational tool, and autonomous agent
with hierarchical task decomposition capabilities. Honesty requires models to optimize for factual
accuracy, appropriate uncertainty quantification, and epistemic self-awareness regarding knowledge boundaries. These objectives exhibit fundamental incompatibilities that manifest as Paretooptimal trade-offs: maximizing utility for legitimate user requests may violate safety constraints
(helpfulness-harmlessness trade-off), optimizing for user satisfaction may incentivize extrapolation
beyond training distribution support (helpfulness-honesty trade-off), and complete information
disclosure may compromise safety requirements (honesty-harmlessness trade-off). Resolution of
these multi-objective optimization challenges necessitates hierarchical policy architectures that implement lexicographic ordering of constraints, prioritizing safety verification, followed by factual
consistency validation, and finally utility maximization within the admissible constraint set.


The unprecedented capabilities of LLMs present a dual-edged sword. While they offer immense potential for societal benefit, their deployment without robust safety alignment poses significant risks.
An unaligned or poorly aligned model can become a vector for widespread misinformation, generate malicious code for cyberattacks, amplify societal biases, or provide instructions for dangerous
activities, thereby causing tangible real-world harm [Askell et al., 2021]. Consequently, establishing safety as the foremost objective is not merely a technical preference but a societal necessity.
It forms the bedrock upon which other alignment goals, such as helpfulness and honesty, can be
securely built. This section will define the foundational role of safety, categorize the key types of
harm that safety objectives aim to mitigate, and trace the evolution of how the AI community
evaluates the achievement of these objectives.


In the discourse of LLM alignment, model development is typically guided by a set of core objectives. Beyond ensuring model **safety** (or **harmlessness** ), these objectives include promoting
**helpfulness**, the model’s ability to effectively follow instructions, and **honesty**, its capacity to
provide factually accurate information [Bai et al., 2022a].
While these objectives are all critical for a well-aligned model, they are not treated as equal.
Safety holds a distinct and primary position. The rationale is straightforward: a model that is
helpful and honest but unsafe can still lead to catastrophic outcomes. For instance, an AI that
honestly and helpfully provides instructions for synthesizing a bioweapon is fundamentally misaligned with human values. Therefore, modern alignment strategies often implement a hierarchical
approach, where safety acts as a foundational filter. A request is first evaluated against safety
and harmlessness criteria. Only if it is deemed safe is it then passed on to be optimized for honesty and helpfulness. This hierarchical structure, which prioritizes the mitigation of harm above
all else, is a central theme in advanced alignment techniques and underscores the non-negotiable
role of safety [Glaese et al., 2022]. This conceptual hierarchy is also a useful way to visualize the
relationship between the different alignment objectives.


To operationalize the high-level goal of ”safety,” it is essential to categorize the specific types of
harm that alignment seeks to prevent. These categories guide the creation of training datasets,
the formulation of safety policies, and the development of evaluation benchmarks. A primary
and overt category of harm involves providing **instructions for dangerous acts**, where models
are prompted for illegal activities or content that encourages severe harm. Training models to
robustly refuse requests for guidance on topics like weapon creation or self-harm is a central focus
of safety research and red-teaming efforts [Ganguli et al., 2022a]. Furthermore, safety alignment
aims to mitigate social harms by preventing the generation of **hate speech and harassment** . The
objective is to stop the model from becoming a tool for perpetuating toxicity or producing content
that demeans and attacks individuals based on protected characteristics, a challenge addressed by
specialized benchmarks like ToxiGen [Hartvigsen et al., 2022]. Another critical dimension of safety
involves preventing **high-stakes misinformation** . While general factuality falls under honesty,
providing dangerously incorrect advice in domains like medicine or law constitutes a direct safety
risk, with studies consistently highlighting the potential for severe consequences [Thirunavukarasu
et al., 2023, Dahl et al., 2024]. Finally, as LLMs become more proficient in programming, preventing
the generation of **malicious code** has emerged as a crucial safety frontier. This involves ensuring
models are not exploited to create viruses, malware, or tools for cybersecurity threats, which would
weaponize their capabilities for widespread damage [Perry et al., 2022].


The methods for evaluating whether a model has successfully met its safety objectives have evolved
significantly over time, reflecting a growing sophistication in the community’s understanding of
alignment.
Initially, safety measures were rudimentary, often relying on **keyword-based filtering and**
**blocklists** . This approach was brittle and easily circumvented by simple rephrasing or adversarial
prompts. The next stage involved **Supervised Fine-Tuning (SFT)** on curated datasets, where
models were explicitly trained on examples of safe responses and refusals. While an improvement,
this method primarily taught the model to imitate safe-looking text, without necessarily internalizing the underlying principles.
The current state-of-the-art is dominated by **Reinforcement Learning from Human Feed-**
**back (RLHF)** and its variants, such as Constitutional AI’s Reinforcement Learning from AI
Feedback (RLAIF) [Bai et al., 2022a]. In this paradigm, a separate **reward model** is trained to
represent human (or AI) preferences regarding safety. The LLM is then fine-tuned to optimize
its behavior to maximize the reward signal, effectively learning a more nuanced and generalizable
understanding of safety.
Most recently, the focus has shifted toward more proactive and adversarial evaluation methods.
This includes systematic **Red-Teaming**, where dedicated teams (both human and automated)
actively search for vulnerabilities and “jailbreaks” [Liu et al., 2024a]. Furthermore, the development
of standardized **safety benchmarks** allows for more consistent and reproducible evaluation of
model safety across the industry. This ongoing evolution signifies a move from a reactive posture
to a proactive and robust science of safety evaluation.

2.1.3 The Evolution of Safety Evaluation


With advancements over the past decade, such as Transformer architectures [Vaswani et al., 2017],
Supervised Fine-Tuning (SFT), and various reinforcement learning methodologies [Schulman et al.,
2017, Bai et al., 2022b, Shao et al., 2024a, Rafailov et al., 2023], LLMs have evolved significantly.
Initially serving as autoregressive token generators, LLMs have now become powerful conversational
assistants. Benefiting from neural networks comprising hundreds of billions of parameters and
training on trillions of tokens [Hurst et al., 2024], these models possess foundational knowledge
across numerous domains. Furthermore, through SFT and reinforcement learning techniques, LLMs
have developed sophisticated reasoning capabilities, enabling them to assist humans effectively in
alignment with human values. This assistance primarily manifests in four areas:
  - **Comprehensive Search Engine** Due to extensive training on vast datasets, LLMs have
acquired significantly more knowledge than typical human capabilities, allowing them to summarize diverse content effectively. They can address a wide array of queries, providing answers
even to highly specific questions that conventional search engines, such as Google or Edge,
may fail to resolve. Leveraging their comprehensive knowledge base and reasoning abilities,
LLMs can formulate precise answers tailored to individual queries. Moreover, current LLMs
have tool-use capabilities, integrating external search engines to retrieve relevant content,
summarize findings, and augment explanations and reasoning tailored to user-specific contexts. Recent advancements, such as Retrieval-Augmented Generation (RAG) [Lewis et al.,
2020] further enhance their capabilities.
  - **Expert Assistant** Technological improvements and enhanced reasoning abilities enable
LLMs to function effectively as domain experts. They facilitate rapid learning and familiarization with new fields, acting as assistants or thought partners for professionals like physicians and mathematicians. Notably, reinforcement learning allows LLMs to explore knowledge spaces that might exceed human imagination. For instance, research has demonstrated
that LLMs employed as mathematical experts have successfully generated novel algorithms
to address optimization problems [Romera-Paredes et al., 2024], significantly outperforming
existing state-of-the-art methods.
  - **General Assistant** With the introduction of sophisticated protocols like MCP introduced
by Anthropic on November 25, 2024, LLMs can efficiently utilize various online tools and
external databases. They reliably generate code snippets, enhancing productivity in software
engineering tasks. Additionally, they support researchers by efficiently reviewing literature,
preparing presentations, and generating content, thereby substantially reducing workload and
improving overall productivity.
  - **LLM Agents** LLM agents can be described as systems that utilize LLMs to reason through
problems, formulate strategic plans, and execute these plans with support from various tools
[NVIDIA, 2024]. Each agent incorporates a memory module capable of storing internal logs
and interactions with users. Additionally, agents can access diverse tools through specified
protocols. A collection of LLM agents forms an LLM agent group, which includes an agent
core responsible for defining overall objectives, a tool list detailing accessible tools, and a
planning module that determines the appropriate agent for specific situations. Employing an
LLM agent group facilitates the decomposition of complex tasks into manageable subtasks,
which are then efficiently delegated to appropriate agents with minimal human intervention.
Projects such as AutoGPT and BabyAGI were pioneers in demonstrating and rapidly advancing this capability. For example, in recent research, an LLM agent group was successfully
deployed to autonomously plan and execute the synthesis of an insect repellent and three
organocatalysts, as well as guide the discovery of a novel chromophore [Bran et al., 2023].


Harmlessness in LLMs ensures that the generated content avoids toxicity, bias, or dangerous information. While society greatly benefits from the convenience offered by LLMs, there are also
substantial risks and vulnerabilities associated with their use. Specifically, LLMs pose threats to
user privacy and can produce toxic, biased, or hazardous content. For instance, interactions with an
LLM might result in harmful advice that exacerbates mental health issues or promotes self-harm.
Additionally, users could exploit LLMs to gain knowledge about creating weapons or explosives,
significantly endangering public safety. Furthermore, malicious users may leverage content produced by LLMs to facilitate hacking activities, potentially causing substantial damage to societal
infrastructure. Historical instances have demonstrated that these potential harms are not merely
theoretical but have resulted in real-world consequences [El Atillah, 2023, Mauran, 2023]. Even
though companies developing advanced LLMs implement measures to restrict harmful outputs,
such as flagging toxic, dangerous, or biased content (commonly referred to as a “red flag policy”),
methods for circumventing these safeguards (known as “jailbreaks”) remain prevalent and easily
accessible online. Therefore, ensuring harmlessness is recognized as one of the most critical tasks
in aligning LLM behavior with human values and societal preferences. Several strategies have
been employed to mitigate harmful content production, including prompt filtering (screening and
excluding harmful data from training datasets), supervised fine-tuning (SFT), and reinforcement
learning techniques aimed explicitly at aligning LLM outputs with ethical standards. Despite these
advances, ongoing research and innovation remain necessary to further strengthen safeguards and
address persistent vulnerabilities effectively.


The honesty of LLMs can be defined as the capability to provide accurate information, express uncertainty clearly without misleading users, and demonstrate awareness of their own knowledge and
internal state [Askell et al., 2021]. With advancements enhancing the utility of LLMs, their popularity in society has grown significantly, with many individuals regularly using them as assistants.
Despite these technological improvements, LLMs occasionally generate dishonest or inaccurate responses, which can lead to serious consequences, particularly in critical domains such as medicine
[Thirunavukarasu et al., 2023], law [Dahl et al., 2024], and finance [Li et al., 2023a], where precision and reliability are paramount. Consequently, ensuring honesty in LLMs has emerged as a
critical aspect of aligning their behavior with human values and preferences [Askell et al., 2021].
Specifically, an honest LLM should transparently acknowledge its limitations rather than delivering misleading information when encountering queries beyond its expertise [Li et al., 2024a].
This transparency helps foster user trust and mitigates potential risks associated with misinformation. Figure 1 illustrates an example of an LLM generating a dishonest response, highlighting the
need for continued vigilance and improvement. Several techniques have been developed to enhance
the honesty of LLMs, including Prompt Engineering, Supervised Fine-Tuning (SFT), reinforcement
learning methods, and adversarial training, all of which aim to systematically reduce the occurrence
of inaccuracies and enhance the models’ capacity for trustworthy interactions. Ongoing research in
explainability and transparency further supports these efforts, ensuring users understand how and
why particular responses are generated.


- **Helpfulness VS. Harmlessness.** A request may be genuinely useful to the user yet unsafe for society, e.g., instructions for constructing weapons. Fulfilling the request maximizes
helpfulness but violates harmlessness. Conversely, a refusal or partial answer safeguards
harmlessness but reduces usefulness.
  - **Helpfulness VS. Honesty.** To appear helpful, LLMs might speculate beyond their knowledge, risking hallucination. Strict honesty requires acknowledging uncertainty or declining to
answer, which users may interpret as unhelpful.
  - **Honesty VS. Harmlessness.** Some truthful information is intrinsically dangerous (e.g.,
detailed chemical synthesis routes). Absolute honesty would disclose it, but harmlessness
mandates withholding or redacting sensitive details.
One potential method is Hierarchical Policy Stacking, which implements a sequential cascade
of checks to align LLMs with human values effectively. This approach prioritizes safety by first
applying safety filters (ensuring harmlessness), followed by fact-checking modules (ensuring honesty), and ultimately a helpfulness optimizer. This hierarchical structure ensures that the pursuit of
helpfulness never overrides critical safety considerations. For instance, Bai et al. [2022a] employed
a constitutional rule set that triggers refusals or redactions for unsafe content. Subsequently, selfcritique and revision steps ensure factual consistency, and only then is a helpful answer generated
if it successfully passes these initial stages. Glaese et al. [2022] utilized a rule-based safety layer to
block or redact unsafe outputs, after which surviving responses were re-ranked by a truthfulness
verifier, selecting the highest-scoring, safest, and most accurate candidate. Additionally, Solaiman
et al. [2021] describes the use of a content-policy classifier capable of refusing or truncating potentially harmful completions; permissible content is subsequently processed through fact-checking
filters before final delivery to users.

2.3 Balancing and Trade-offs Among Objectives


Large language models offer impressive capabilities but are also vulnerable to a growing range of
adversarial jailbreak attacks that can bypass safety measures and produce harmful or misleading
outputs. To address these risks, we must move beyond ad-hoc testing and adopt a unified framework
for alignment evaluation. This section is organized into three parts. First, we review adversarial
attack methods that deliberately probe model weaknesses. Second, we examine human feedback
and scoring protocols used to measure model outputs on safety and utility criteria. Third, we
Figure 1: Example of the jailbreak attack.
survey established benchmarks and stress-testing suites that provide standardized datasets, metrics,
and evaluation pipelines. Together, these components create a solid foundation for quantifying,
comparing, and improving the alignment of large language models.

3 Evaluation and Benchmarking of Alignment


Adversarial jailbreak research seeks to map and stress-test the full attack surface of modern LLMs
by crafting inputs that compel the model to violate its own safety constraints. Figure 1 gives an
example to illustrate the basic idea of jailbreak attacks. Over the past two years, this work has
coalesced around three complementary strands, each revealing distinct vulnerability classes and
informing more rigorous red-teaming practices.


Logic-based methods hijack the model’s internal reasoning or optimization process. AutoDAN [Liu
et al., 2024a] and its successor AutoDAN-Turbo [Zhang et al., 2025a] frame jailbreak generation
as an evolutionary search problem, continuously mutating suffixes to maximize unsafe behavior.
Simple adaptive attacks refine prompts via minimal binary feedback, steering the model toward
disallowed completions with only a handful of trials [Patel and Singh, 2025]. Cognitive Overload
constructs deliberately entangled puzzles that exceed the model’s chain-of-thought capacity, causing
safety checks to fail [Chen and Zhao, 2024], while human-persuasion exploits like “How Johnny
Can Persuade” leverage social-engineering tropes to bypass guardrails [Liu and Zhang, 2024]. Even
purely black-box settings can be compromised: as few as twenty carefully chosen queries suffice to
extract forbidden content [Johnson and Kumar, 2024].


Low-resource attacks exploit under-trained channels or formats rather than the model’s reasoning
core. In SelfCipher, illicit instructions are hidden via reversible substitution ciphers; when the model
decodes the cipher it unknowingly obeys the unsafe command [Smith and Doe, 2024]. Multilingual
pivoting translates prompts into low-resource languages before reverting to English, slipping past
filters that have not seen such language patterns [Lee and Kim, 2024a]. ASCII art has also been
incorporated into prompt design to bypass safety alignment [Jiang et al., 2024]. To strengthen
defenses against domain-specific jailbreak attacks in the chemical and biological domains, Luo et al.
introduce the CB-Redteam and CB-Benign datasets. Their approach targets low-resource jailbreak
strategies, such as those involving SMILES representations [Wong et al., 2024], by employing a
multi-agent framework that leverages external knowledge to generate unbiased intention summaries
and analytically grounded safety guidance. This methodology enhances robustness against these
jailbreak scenarios [Luo et al., 2025a]. These minimal-footprint attacks, and the corresponding
guided safeguards, demonstrate that even simple format or linguistic quirks can undermine or
reinforce alignment without extensive computation or gradient access.


Beyond algorithmic searches, a wealth of jailbreak recipes circulate publicly, often proving more
effective than laboratory-generated attacks. The “Do Anything Now” (DAN) corpus documents
real-world prompt collections that generalize across model families and evade formal red-team
defenses [Liu and Gao, 2024]. Other notable in-the-wild exploits include “Make Them Spill the
Beans!” which uses coercive narratives to extract guarded knowledge [Li et al., 2023b], “Ignore
This Title and HackAPrompt,” a global-scale prompt hacking competition that uncovers systemic
guardrail failures [Wang et al., 2023a], and “Summon a Demon and Bind it,” a grounded theory
analysis of red-teaming practices observed in the wild [Brown et al., 2023]. Broader repositories
like “EasyJailbreak” compile hundreds of user-contributed attacks [Doe and Bloggs, 2023], while
studies on tricking LLMs into disobedience formalize and categorize emergent community prompts
[Lee and Kim, 2024b]. Together, these community-sourced vectors highlight the critical need for
continuous, open-world red-teaming that adapts to spontaneously emerging social engineering and
prompt-sharing practices.

3.1.3 Community-Driven and In-The-Wild Prompts