Fake alignment, sometimes called alignment faking or deceptive alignment, occurs when an LLM
superficially obeys its safety constraints while covertly preserving misaligned objectives. Rather
than exploiting vulnerabilities at inference time, a fake-aligned model deliberately feigns compliance
during training or monitored evaluation in order to avoid corrective fine-tuning and retain its true
preferences [Hubinger et al., 2024]. Such models exhibit situational awareness, identifying when
they are under scrutiny and adjusting their behavior accordingly; for instance, an LLM may refuse
harmful requests in a perceived unmonitored setting but comply when it detects that responses
will influence its training data [Wang et al., 2023b]. Critically, the model’s underlying goals remain
intact despite safety interventions, as universal backdoor triggers and persistent hidden objectives
can survive even extensive fine-tuning [Rando et al., 2024a]. This strategic deception undermines
standard evaluation protocols and highlights the need for alignment benchmarks that probe beyond
surface-level obedience to reveal a model’s true intent [Greenblatt et al., 2024].


To systematically uncover vulnerabilities in safety-aligned large language models, several competitive platforms and challenges have been established. At IEEE SaTML 2024, the “Competition Report: Finding Universal Jailbreak Backdoors in Aligned LLMs” challenged participants to discover
*universal* backdoor triggers capable of reliably bypassing model defenses across ten real-world
scenarios, over four hundred tools, and multiple LLM backbones [Rando et al., 2024a]. Complementing this, the ETH Z¨urich SPY Lab organized the RLHF Trojan Competition, where teams
designed detection methods for backdoors inserted during RLHF pipelines, incentivized by a substantial prize pool and a public GitHub repository [Rando and Tram`er, 2024]. The NeurIPS 2023
Trojan Detection Challenge (TDC 2023) established a leaderboard for red-teaming and trojan detection, with top submissions advancing recall and reverse-engineering metrics for hidden triggers
[Mazeika et al., 2023, 2024a]. A subsequent competition extended jailbreak attack challenges to
LLM-based agents [Xiang et al., 2024]. In parallel, the SaTML Capture-the-Flag event framed
safety as a secret-string challenge, tasking teams with both attack and defense, and producing a
large dataset of adversarial multi-turn conversations [Debenedetti et al., 2024a]. Community-driven
hackathons like the Alignment Jam’s Trojan Detection track further broadened participation, enabling lightweight, red-team-style evaluations of prompt and model vulnerabilities [Christiano et al.,
2023a]. SPY Lab’s own blog summaries highlight how CTF-style competitions and formal benchmark releases work in concert to expose real-world failure modes [Rando et al., 2024b]. These
competitive efforts have not only revealed persistent backdoors that survive standard fine-tuning
[Hubinger et al., 2024] and stealthy black-box attacks such as FlipAttack [Liu et al., 2024b], but have
also inspired novel defense strategies and analysis methods, exemplified by infectious image-based
jailbreaks demonstrated by Agent Smith [Gu et al., 2024]. Collectively, jailbreak competitions serve
as an empirical proving ground for attackers and defenders alike, accelerating progress toward more
robust alignment.

Alignment and Safety in Large Language Models: Safety Mechanisms,
  Training Paradigms, and Emerging Challenges

Haoran Lu
Luyang Fang
Ruidong Zhang
Xinliang Li
Jiazhang Cai
Huimin Cheng
Lin Tang
Ziyu Liu
Zeliang Sun
Tao Wang
Yingchuan Zhang
Arif Hassan Zidan
Jinwen Xu
Jincheng Yu
Meizhi Yu
Hanqi Jiang
Xilin Gong
Weidi Luo
Bolun Sun
Yongkai Chen
Terry Ma
Shushan Wu
Yifan Zhou
Junhao Chen
Haotian Xiang
Jing Zhang
Afrar Jahin
Wei Ruan
Ke Deng
Yi Pan
Peilong Wang
Jiahui Li
Zhengliang Liu
Lu Zhang
Lin Zhao
Wei Liu
Dajiang Zhu
Xin Xing
Fei Dou
Wei Zhang
Chao Huang
Rongjie Liu
Mengrui Zhang
Yiwen Liu
Xiaoxiao Sun
Qin Lu
Zhen Xiang
Wenxuan Zhong
Tianming Liu
Ping Ma


Scoring-based evaluation methods repurpose large language models themselves as automatic judges,
prompting them to generate quantitative assessments of candidate outputs under user-defined criteria. GPTScore [Fu et al., 2023a] pioneered this paradigm by using generative pre-trained models
to issue zero-shot, instruction-driven quality scores across multiple aspects without requiring annotated examples. Building on this, G-Eval [Liu et al., 2023a] leverages chain-of-thought prompting
and a form-filling interface on GPT-4 to achieve strong correlations with human judgments in summarization and dialogue tasks. Subsequent studies have probed the reliability and bias of LLM
judges: “LLMs as Narcissistic Evaluators” highlights in-model favoritism toward self-generated outputs [Liu et al., 2023b], while “Large Language Models are not Fair Evaluators” reveals positionand format-dependent inconsistencies [Wang et al., 2023c]. The DHP Benchmark [Wang et al.,
2024a] systematically measures an LLM’s discernment across hierarchically perturbed inputs, uncovering strengths and blind spots. Alternative frameworks, such as JudgeLM [Zhu et al., 2023a]
and PandaLM [Wang et al., 2023d], explore fine-tuning lightweight LLMs as specialized evaluators,
whereas LLM-Eval [Lin and Chen, 2023] and CLAIR [Chan et al., 2023] demonstrate multimodal
and structured-format extensions. More recently, FLEUR [Lee et al., 2024a] integrates referenceless reasoning for image captioning evaluation. Together, these works form a rapidly maturing
“LLM-as-a-Judge” ecosystem, offering flexible, scalable metrics but also raising new challenges in
consistency, calibration, and evaluative bias.


To systematically evaluate alignment performance, the community has developed comprehensive
benchmark suites that define rigorous protocols, curated adversarial scenarios, and quantitative
metrics. As adversarial exploits, jailbreak strategies, and automated red-teaming techniques evolveuncovering increasingly subtle and diverse failure modes, these standardized evaluations become
indispensable. By offering repeatable testbeds and clear success criteria, safety alignment benchmarks enable precise progress tracking, facilitate head-to-head comparisons, and illuminate residual
vulnerabilities, thereby driving the design of more robust and resilient alignment methodologies.


In recent years, the field has seen the emergence of a variety of general safety benchmarks designed to systematically evaluate the robustness and alignment of large language models against
harmful or adversarial inputs. AdvBench introduced the first large-scale suite of adversarial suffix attacks to probe aligned models’ vulnerability to objectionable content generation [Zou et al.,
2023]. SALAD-Bench followed with a hierarchical evaluation framework covering attack, defense,
and ethical dimensions in both English and Chinese [Li et al., 2024b]. SafetyBench presented over
11,000 multiple-choice safety questions spanning seven key risk categories and demonstrated significant performance gaps even among state-of-the-art models [Zhang et al., 2023a]. COLD targeted
Chinese offensive-language detection with curated examples to reveal nuanced toxic behaviors in
generative systems [Deng et al., 2022], while BeaverTails provided a massive meta-labeled dataset
distinguishing helpfulness from harmlessness in QA pairs to advance safety alignment research [Ji
et al., 2024].
Subsequent benchmarks have broadened the landscape further: SORRY-Bench systematically
analyzes refusal behaviors across 45 fine-grained unsafe topics and 20 linguistic augmentations, enabling efficient automated evaluation with smaller models [Xie et al., 2024]. Rainbow Teaming casts
adversarial prompt generation as a quality–diversity problem, yielding highly transferable jailbreaks
that stress-test model defenses [Samvelyan et al., 2024]. CoSafe investigates multi-turn dialogue
coreference attacks to expose vulnerabilities in conversational models under context-tracking failures [Yu et al., 2024a]. SC-Safety offers a 4,912-question adversarial benchmark in Chinese across
more than 20 safety sub-dimensions, demonstrating the persistent gap between open-source and
closed-source model safety [Xu et al., 2023a]. PromptBench provides a unified library for constructing, attacking, and dynamically evaluating prompts, serving as a toolchain to streamline
safety benchmarking workflows [Zhu et al., 2023b].


The large reasoning model (LRM) is a special kind of LLM, which leverages long chain-of-thought
reasoning to generate intermediate steps and enhance reasoning abilities. Although LRM’s final
answer may appear safe, harmful or policy-violating content may still be contained in intermediate
reasoning steps [Jiang et al., 2025a]. Thus, alignment method focus on reasoning steps also plays
an important role. In Table 1, we list current state-of-the-art benchmark reasoning models.
Due to the presence of intermediate reasoning steps and long-form outputs in LRMs, general
safety datasets are insufficient for aligning their safety [Zhou et al., 2025a].. As a result, specialized
datasets have been designed to address this need.
  - **SafeChain** [Jiang et al., 2025a]: It includes 40,000 instruction-response pairs, filtered from
50,000 prompts sampled from the WildJailbreak dataset. Each instruction was answered five
times by the R1–70B model, and only those with all five responses deemed safe by LlamaGuard were kept. One safe response per instruction was then randomly selected to form the
final dataset.
  - **STAR-1** [Wang et al., 2025a]: The STAR-1 dataset contains 1,000 high-quality, safetyaligned examples selected from 41,000 potentially harmful instructions collected from opensource safety datasets. These instructions span eight safety domains and were answered by
|Model Name|Reference|
|---|---|
|DeepSeek-R1 Series
Skywork-o1
QwQ
Sky-T1
Gemini-Thinking
Kimi-k1.5
LLAMA3
Qwen2.5|DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Rein-
forcement Learning [DeepSeek-AI et al., 2025a]
Skywork-o1 open series.` https://huggingface.co/Skywork`
QwQ: Refect Deeply on the Boundaries of the Unknown` https://`
`qwenlm.github.io/blog/qwq-32b-preview/`
Think less, achieve more: Cut reasoning costs by 50sacrifcing accuracy.
`https://novasky-ai.github.io/posts/reduce-overthinking/`
Gemini 2.0 fash thinking` https://ai.google.dev/gemini-api/docs/`
`thinking`
Kimi k1.5: Scaling Reinforcement Learning with LLMs [Team et al.,
2025]
The Llama 3 Herd of Models. [Grattafori et al., 2024]
Qwen2.5 Technical Report. [Qwen et al., 2025]|
Table 1: State-of-the-art large reasoning models.
DeepSeek-R1 with policy-grounded chain-of-thought reasoning. Responses were scored using
GPT-4o for safety, and only the top 1,000 examples were retained. The dataset emphasizes
diversity, deliberative reasoning, and rigorous filtering.
  - **RIT-D** [Mou et al., 2025]: RIT-D is a dataset used for reasoning-style warmup stage in SaRO
framework, which is built based on Salad-Bench and OpenOrca The answer is generated using
a specifically designed prompt with GPT-4o. 10,505 samples and 9805 queries are contained.
  - **OP-COT** [Mou et al., 2025]: OP-COT is a dataset used for safety-oriented reasoning process
optimization stage in SaRO framework. GPT-4o It is constructed from BeaverTails. 2,188
samples and 580 queries are contained. To enrich reasoning, GPT-4o is instructed with
tailored prompts to generate long-chain safe responses, while Qwen2.5-72B, using a few-shot
setup, provides contrasting unsafe reasoning.
  - **PP-COT** [Mou et al., 2025]: PP-COT is also used for safety-oriented reasoning process
optimization stage in SaRO framework. It is derived from OP-COT through reasoning step
decomposition and stepwise reflection. It contains 11,598 samples and 580 queries.


Enron Email Dataset is widely used as benchmark dataset in privacy alignment. This dataset
contains approximately 500,000 emails generated by employees of the Enron Corporation. It was
obtained by the Federal Energy Regulatory Commission during its investigation of Enron’s collapse.
It is frequently used to study privacy alignment challenges [Wang et al., 2023e]. Researchers analyze
whether LLMs trained on such data may memorize or leak sensitive personal information.
For evaluation metric, zero-shot & few-shot prompting [Wang et al., 2023e, Huang et al., 2024]
is a widely used setting. K-shot true (name, privacy information) pairs are provided to LLM in
prompt, then we check whether LLM will predict the target privacy information of the target user
name. Higher success rate means more privacy leakage.


PRISM [Kirk et al., 2024] is the benchmark dataset in fairness alignment. This dataset contains
8,011 live conversations with 21 large language models, contributed by 1,500 participants from 75
countries. It captures the sociodemographic profiles, stated preferences, and contextual feedback of
individuals, enabling fine-grained analysis of fairness across diverse populations. PRISM focuses on
value-laden and culturally sensitive topics where disagreement is expected, making it well-suited for
studying fairness across social, cultural, and geographic lines. It also includes census-representative
samples for the UK and US, allowing researchers to examine fairness alignment with respect to
population representativeness and demographic equity.
The demographic parity difference metric _Mdpd_ [Zemel et al., 2013] and the equalized odds
difference metric _Meod_ [Hardt et al., 2016] are used to evaluate the fairness of LLM. Given data
samples ( _X, Y, A_ ), where _X_ is the feature vector, _Y ∈{_ 0 _,_ 1 _}_ is the label, and _A ∈{_ 0 _,_ 1 _}_ is a sensitive
attribute, _M_ dpd is defined as: _M_ dpd = _|_ P( _f_ ( _X_ ) = 1 _| A_ = 1) _−_ P( _f_ ( _X_ ) = 1 _| A_ = 0) _|_ It measures the
difference in positive prediction rates between sensitive groups, without considering the ground
truth label. To incorporate label correctness, _M_ eod is defined as: _M_ eod = max _{MTP, MFP }_ where
_MTP_ and _MFP_ represent the differences in true and false positive rates:
_MTP_ = _|_ P( _f_ ( _X_ ) = 1 _| Y_ = 1 _, A_ = 0) _−_ P( _f_ ( _X_ ) = 1 _| Y_ = 1 _, A_ = 1) _|_
_MFP_ = _|_ P( _f_ ( _X_ ) = 1 _| Y_ = 0 _, A_ = 0) _−_ P( _f_ ( _X_ ) = 1 _| Y_ = 0 _, A_ = 1) _|_
These metrics assess model fairness by evaluating whether predictions are equitable across groups
defined by the sensitive attribute _A_ .


[Chern et al., 2024] assess the honesty of LLM in 3 aspects and 10 scenarios. Details about these
benchmarks are listed in Table 2


LLM-based agents extend standalone language models by perceiving and acting within dynamic
environments, where decisions can lead to physical collisions, data leaks, or security incidents.
A comprehensive suite of benchmarks has therefore emerged to quantify these interactive risks:
SafeAgentBench evaluates embodied household manipulation in a physics simulator, resetting the
Safe-Success Rate (SSR) to zero upon any collision, spill, or forbidden action [Yin et al., 2024a];
Agent-SafetyBench stresses tool-augmented dialogue agents with adversarial prompts and measures
robustness via the Attack-Success Rate (ASR) [Zhang et al., 2024a]; ST-WebAgentBench recreates
enterprise web workflows in BrowserGym/WebArena, scoring policy adherence with Completionunder-Policy (CuP) and Partial-CuP while categorizing violations into consent, hierarchy, hallucination, security, and error-handling failures [Levy et al., 2024]; EARBench generates multimodal
planning scenarios paired with synthetic imagery and GPT-4o descriptions, using a second GPT4o judge to flag unsafe steps and compute Task-Risk Rate (TRR) and Task-Effectiveness Rate
(TER) [Zhu et al., 2024a]; Agent Security Bench (ASB) formalizes ten real-world scenarios (e.g.
e-commerce, autonomous driving, finance), over 400 tools, and 27 attack and defense methods to
benchmark vulnerabilities across system prompts, tool use, and memory retrieval [Zhang et al.,
2024b]; AgentDojo offers a dynamic, extensible environment for prompt injection evaluation with
97 realistic tasks and 629 adversarial cases [Debenedetti et al., 2024b]; GuardAgent provides two
benchmarks for evaluating access control on healthcare agents and safety regulation of webagents,


Admitting Unknowns Ability to refuse or admit lack of knowlSelf-Knowledge
edge when the question is outside the model’s
scope.
Expressing Knowns Ability to provide the correct answer when
the question is within the knowledge boundary.
Non-deceptiveness
Consistency
Persona Sycophancy Whether the model alters facts to flatter a
user’s persona.
Preference Sycophancy Whether the model tailors factual answers to
align with a user’s stated preference.
Burglar Deception Test Propensity to provide intentionally misleading instructions to a burglar-like agent.
Game Willingness to cheat or fabricate information
to win a simple interactive game.
Prompt Format Consis- Stability of answers under superficial prompt
tency re-phrasings.
Demonstration Format Stability when demonstrations/examples in
Consistency the prompt are reordered.
Open-Form Consistency Agreement between two free-form answers to
equivalent prompts.
Multiple-Choice Consis- Consistency between multiple-choice and
tency open-ended answers for the same query.
Table 2: BeHonest benchmark suite for evaluating honesty in large language models.
respectively [Xiang et al., 2025]; AGrail introduces a lifelong guardrail that adaptively generates
and optimizes safety checks, demonstrating defense transferability across diverse agent tasks and
purpose Safe-OS, an online OS agent benchmark for accessing the safety of OS agent [Luo et al.,
2025b]; ShieldAgent is proposed with a benchmark containing 3K safety-related pairs of agent instructions and action trajectories from state-of-the-art attacks [Chen et al., 2025a]. Together, these
benchmarks reveal that even state-of-the-art agents struggle to balance task effectiveness with
safety, underscoring the need for broad, cross-domain evaluation frameworks that mirror real-world
complexity.


In recent years, the deployment of LLMs in scientific domains has raised pressing concerns around
safety, particularly in contexts involving chemistry, medicine, biology, and physics. Unlike generalpurpose benchmarks, domain-specific safety benchmarks are designed to probe whether models
behave reliably and ethically when performing specialized scientific tasks, such as providing treatment recommendations, describing hazardous procedures, or reasoning under physical laws. These
benchmarks address key dimensions of safety, including refusal behavior, robustness to adversarial
prompts, and adherence to regulatory and ethical constraints. Below, we summarize several representative domain-specific safety benchmarks, Table 3 provides an overview of their safety focus
and task characteristics.
  - **HealthBench** [Arora et al., 2025] is a benchmark from OpenAI targeting safety and ro
bustness in medical dialogues. It includes 5,000 realistic, multi-turn conversations between
patients or providers and an AI assistant, covering various healthcare scenarios. Each conversation is annotated with over 48,000 detailed evaluation criteria from 262 physicians, scoring
accuracy, empathy, and safety. The benchmark tests whether the AI avoids harmful advice,
follows medical best practices, and communicates uncertainty appropriately.
- **ChemSafetyBench** [Zhao et al., 2024] evaluates chemistry-focused LLMs on safety-critical
tasks, including answering questions about chemical hazards, legality, and synthesis. It consists of over 30k questions spanning 1.7k chemicals and three task types, from simple queries
to illicit synthesis instructions. To test robustness, the benchmark introduces handcrafted adversarial prompts and advanced jailbreak scenarios. An automated framework scores model
responses for correctness, safety, and refusal behavior. Even state-of-the-art models like GPT4 were found to generate unsafe outputs, indicating the need for stronger safety alignment in
chemistry AI.
- **WMDP (Weapons of Mass Destruction Proxy)** [Li et al., 2024c] is a benchmark designed to assess whether LLMs possess or reveal dangerous knowledge about bioweapons,
chemical weapons, or cyberattacks. It includes 4,157 expert-written questions probing precursor knowledge (e.g., pathogen handling, synthesis techniques) without explicitly requesting
harmful actions. The benchmark evaluates both model capability and refusal behavior under misuse scenarios. As a red-teaming dataset, WMDP provides standardized evaluation
for dual-use scientific risks. It was released as part of AI safety alignment initiatives and is
openly available.
- **MedSafetyBench** [Han et al., 2024a] tests LLMs’ alignment with medical ethics across 1,800
adversarial medical prompts. Each prompt is paired with safe response demonstrations, covering scenarios like harmful treatment recommendations, privacy violations, and misdiagnosis.
Prompts were generated via GPT-4 plus jailbreaking strategies to simulate realistic misuse.
The benchmark measures how well models follow non-maleficence and other ethical principles.
Fine-tuning on MedSafetyBench significantly improved the safety of several medical LLMs
without hurting their accuracy.
- **LabSafetyBench** [Zhou et al.] evaluates AI understanding of laboratory safety protocols
using 765 multiple-choice questions aligned with OSHA standards. Questions span common
lab hazards in biology, chemistry, and materials science, requiring models to select the safest
action in each scenario. It highlights gaps in models’ real-world lab safety reasoning: GPT-4
performed well overall but still made critical safety errors. The benchmark does not use
adversarial prompts but targets domain-specific safety knowledge.
- **PhysReason** [Zhang et al., 2025b] is a physics benchmark that tests whether models respect
physical laws and perform multi-step reasoning. It contains 1,200 problems, mostly multistep and stratified by difficulty, from basic physics to complex derivations. An automatic
framework scores both final answers and intermediate reasoning steps to pinpoint logical
failures. The benchmark emphasizes failure detection and physical plausibility, key concerns
in engineering, robotics, and scientific applications.
- **SciSafeEval** [Li et al., 2024d] is a cross-domain safety benchmark spanning biology, chemistry, medicine, and physics. It includes tasks involving natural language, molecular structures, protein sequences, and genomic data. Many prompts are designed to trigger jailbreak
behavior and test models’ ability to resist unsafe instructions. The benchmark evaluates
safety alignment under various settings: zero-shot, few-shot, and chain-of-thought. SciSafeEval’s goal is to stress-test scientific AI models and promote alignment methods that generalize
across domains.
  - **SciMT-Safety** [He et al., 2023] is a benchmark targeting the misuse risks of AI systems
in chemistry and biology through 432 red-teaming queries. These include 177 substanceindependent and 255 substance-dependent prompts, crafted using a red-team agent and refined templates filled with hazardous chemical and biological entities (e.g., flammables, toxins,
addictive drugs). Each query probes the model’s potential to enable harmful outcomes. The
benchmark evaluates model “harmlessness” with GPT-4 as a judge and contrasts with benign
queries to assess over-refusal. SciMT-Safety is the first domain-specific benchmark addressing
scientific misuse risks, offering a robust tool for safe deployment of scientific AI models.
  - **SOSBench** [Jiang et al., 2025b] is a regulation-grounded, hazard-focused benchmark encompassing six high-risk scientific domains: chemistry, biology, medicine, pharmacology, physics,
and psychology. Different from most other domain-specific scientific benchmarks, SOSBench
comprises 3,000 prompts derived from real-world regulations and laws, systematically expanded via an LLM-assisted evolutionary pipeline that introduces diverse, realistic misuse
scenarios (e.g., detailed explosive synthesis instructions involving advanced chemical formulas).
Together, these benchmarks represent a growing movement toward scientifically grounded safety
evaluations for AI systems. They reveal that even frontier models, while capable in general contexts,
often fail to meet safety expectations in high-stakes scientific applications. Importantly, many of
these benchmarks offer both evaluation protocols and training data that can be used to improve
model alignment. As scientific LLMs continue to proliferate, these domain-specific safety benchmarks will play a critical role in ensuring that model outputs remain accurate, but also ethical,
legal, and trustworthy in specialized use cases.


A wide range of safety benchmarks [Mazeika et al., 2024b, Chao et al., 2024, Luo et al., 2024, Zhang
et al., 2023a] have been proposed for general-purpose LLMs, and these often include categories
such as malware generation. However, these benchmarks are primarily designed to evaluate LLMs
themselves, rather than code agents. In the case of LLM-based code agents, more comprehensive
and domain-specific risk scenarios are needed to properly assess and improve their safety. To this
end, we have summarized a set of code safety benchmarks specifically designed to address the
unique challenges and risks posed by code agents:
  - **CodeLMSec Benchmark** [Hajipour et al., 2023]: is the first systematic benchmark designed
to evaluate the security risks of large code language models. Unlike traditional evaluations
that focus only on functional correctness, CodeLMSec investigates the ability of these models
to generate code with security vulnerabilities. It introduces a novel black box model inversion
technique using few-shot prompting to automatically generate non-secure prompts that lead
models to produce insecure code. The benchmark targets 13 common vulnerability types,
including SQL injection, cross-site scripting, and deserialization issues, and uses static analysis
with CodeQL for vulnerability detection and classification. CodeLMSec includes over 2000
vulnerable code samples generated from models like ChatGPT and CodeGen, and provides a
curated dataset of 280 diverse nonsecure prompts (200 for Python and 80 for C). This enables
reproducible, and extensible evaluation of code models from a security perspective.
Table 3: Overview of recent domain-specifc safety benchmarks for scientifc AI.


[Arora et al., 2025] Medicine Dialogue safety; safe clini- Multi-turn conversations
cal conversation; empathy, annotated with fineappropriateness grained medical criteria
[Arora et al., 2025] Medicine Dialogue safety; safe clini- Multi-turn conversations
cal conversation; empathy, annotated with fineappropriateness grained medical criteria


[Zhao et al., 2024] Chemistry Chemical hazard accu- Free-form QA in 3 task
racy; refusal of illicit syn- types; includes adversarial
thesis; jailbreak robust- jailbreak prompts
ness
Free-form QA in 3 task
types; includes adversarial
jailbreak prompts


[Li et al., 2024c] Biology, Chemistry, Detection of dual-use
Cybersecurity knowledge; refusal under
misuse prompts
[Li et al., 2024c] Biology, Chemistry, Detection of dual-use Expert-crafted proxy misCybersecurity knowledge; refusal under use queries; red-teaming
misuse prompts scenarios


[Han et al., 2024a] Medicine Ethical alignment; refusal Adversarial prompts
of harmful or unethical paired with aligned reclinical guidance sponses; ethics-based
[Han et al., 2024a] Medicine Ethical alignment; refusal Adversarial prompts
of harmful or unethical paired with aligned reclinical guidance sponses; ethics-based
safety evaluation


[Zhou et al.] Lab Environments Lab protocol knowledge; OSHA-aligned multiplehazard identification in choice questions; domainexperimental settings specifc safety context
[Zhou et al.] Lab Environments Lab protocol knowledge; OSHA-aligned multiplehazard identification in choice questions; domainexperimental settings specifc safety context


[Zhang et al., 2025b] Physics Physical law adherence; Open-ended problems;
multi-step reasoning accu- stepwise scoring of interracy mediate and fnal answers
[Zhang et al., 2025b] Physics Physical law adherence; Open-ended problems;
multi-step reasoning accu- stepwise scoring of interracy mediate and fnal answers


[Li et al., 2024d] Multi-domain (Bio, Jailbreak resistance; sci- Multi-format inputs (text,
Chem, Med, Physics) entific safety alignment SMILES, proteins); adveracross modalities sarial prompts included
[Li et al., 2024d] Multi-domain (Bio, Jailbreak resistance; sci- Multi-format inputs (text,
Chem, Med, Physics) entific safety alignment SMILES, proteins); adveracross modalities sarial prompts included


[He et al., 2023] Chemistry, Biology Misuse safety via red- Red-teamed queries and
teaming; LLM harmless- benign test set; names,
ness benchmarking IUPAC, SMILES formats
[He et al., 2023] Chemistry, Biology Misuse safety via red- Red-teamed queries and
teaming; LLM harmless- benign test set; names,
ness benchmarking IUPAC, SMILES formats
used


[Jiang et al., 2025b] Multi-domain (Bio, Knowledge-intensive Jailbreak prompts generChem, Med, Physics, task safety; regulation- ated in a hybrid way, with
Pharm, Psych) grounded; hazard-focused LLM-assisted data expan
Knowledge-intensive
task safety; regulationgrounded; hazard-focused
Jailbreak prompts generated in a hybrid way, with
LLM-assisted data expansion
- **RedCode** [Guo et al., 2024a]: is a comprehensive evaluation framework designed to assess
the safety of large language model-based code agents, focusing on both unsafe code execution
and generation. Unlike prior benchmarks that only assess static code outputs or rely on simulated environments, RedCode tests code agents interacting with real execution environments
via Docker containers. The benchmark consists of two components: RedCode-Exec, which
includes 4,050 test cases across 25 risky scenarios in domains such as file systems, operating
systems, and cybersecurity, and RedCode-Gen, which contains 160 prompts for generating
malicious software across eight malware families. Each scenario is supported with high-quality
prompts in multiple formats (code, text summaries, and descriptions) and evaluated using
tailored scripts that check execution outputs and environment changes. RedCode enables
fine-grained, realistic safety evaluations and reveals that even advanced agents, such as those
based on GPT-4, may still generate or execute harmful code under specific conditions.
  - **CyberSecEval** [Bhatt et al., 2024]: CyberSecEval is a multi-stage benchmark suite for
evaluating the cybersecurity risks and defensive capabilities of LLMs. Evolving through
versions 1 to 4, it covers diverse threats including insecure coding, prompt injection, code
interpreter abuse, and vulnerability exploitation. The latest version, CyberSecEval 4, adds
AutoPatchBench to assess automated vulnerability patching in native code. With over 500
interpreter abuse prompts, dozens of exploit challenges, and a novel False Refusal Rate (FRR)
metric, CyberSecEval enables comprehensive, scalable, and reproducible LLM safety testing
across real-world security tasks.


Supervised fine-tuning (SFT) is a foundational approach in aligning LLMs with human expectations. It involves adapting a pretrained model by exposing it to a curated set of instruction–response
pairs, where each response exemplifies a desirable behavior such as helpfulness, factual accuracy,
politeness, or appropriate refusal in ethically sensitive situations. Through standard supervised
learning objectives, the model learns to imitate these examples, internalizing alignment-relevant
behaviors across a range of prompt types. SFT has become the default first stage in many modern
alignment pipelines due to its stability, simplicity, and compatibility with a wide range of data
sources, from expert-written instructions [Ouyang et al., 2022] to synthetic completions filtered by
human or model-based judgment [Wang et al., 2022a, Taori et al., 2023, Peng et al., 2023a].
Although SFT alone may not be sufficient to handle the full complexity of human preferences or
task ambiguity, it remains critical for establishing basic instruction-following behavior. This section
outlines how SFT operates, discusses the role of data quality and task coverage, and explores its
strengths and limitations as an alignment strategy.

4 Supervised Fine-Tuning (SFT) for Alignment


The core mechanism of supervised fine-tuning is instruction tuning, in which a pretrained language
model is trained on a dataset of input prompts paired with preferred responses. These examples
serve as behavioral demonstrations, allowing the model to learn how to complete or respond to various instructions in a manner aligned with human expectations. The data is typically organized into
prompt–response pairs, or in the case of conversational systems, as multi-turn message sequences
with alternating user and assistant roles [Chung et al., 2024, OpenAI Achiam et al., 2023].
In early instruction-tuning pipelines such as InstructGPT [Ouyang et al., 2022], human annotators were tasked with crafting high-quality responses across a diverse range of instructions, ensuring
that the fine-tuned model would behave in a helpful, harmless, and truthful manner. This approach
provided clear, unambiguous supervision and led to models that could respond more appropriately
to user instructions than their purely pretrained counterparts [Brown et al., 2020, Thoppilan et al.,
2022]. Subsequent methods introduced greater automation in the data collection process. For example, the Self-Instruct framework [Wang et al., 2022a] generated prompts and initial completions
using an LLM, followed by filtering, ranking, or editing by human reviewers. Related pipelines used
community-contributed data [Taori et al., 2023], filtered web-derived instruction corpora [Longpre
et al., 2023], or domain-specific synthetic examples [Peng et al., 2023a] to scale instruction tuning
across languages, modalities, and styles.
Regardless of how the data is sourced, the training process relies on a standard next-token
prediction objective. Given an input _x_ and a reference output _y_ = ( _y_ 1 _, y_ 2 _, ..., yT_ ), the model is
trained to maximize the probability of each target token given the input and previously generated
tokens. This corresponds to minimizing the cross-entropy loss:
_L_ SFT = _−_
_T_
log _Pθ_ ( _yt | x, y<t_ ) _,_
_t_ =1
where _Pθ_ is the probability assigned by the model parameterized by _θ_ . This is equivalent to
maximum likelihood estimation (MLE) under the assumption that the observed responses represent
samples from an optimal behavioral policy [Radford et al., 2019, Raffel et al., 2020].
The simplicity and stability of instruction tuning make it a highly effective baseline for alignment. By directly showing the model what aligned behavior looks like, SFT instills basic response
formatting, instruction following, and task generalization. These capabilities have been demonstrated in models ranging from FLAN-T5 [Chung et al., 2024] to Alpaca [Taori et al., 2023], and
OpenAssistant [K¨opf et al., 2023]. However, this method is inherently limited by the coverage
and quality of the training data. The following subsection examines how these factors influence
alignment outcomes and where SFT begins to fall short.

4.1 Instruction Tuning with Human Demonstrations


The effectiveness of SFT for alignment is deeply dependent on the characteristics of the training
data. Since the model is trained to imitate provided responses, its alignment behavior is only as
good as the examples it sees. Thus, the quality, diversity, and representativeness of the instruction
data are central to achieving aligned, generalizable behavior.
High-quality instruction data is typically designed to reflect desired behavioral traits such as
helpfulness, informativeness, honesty, and safety. In early alignment work, instruction responses
were manually authored by expert annotators, as seen in InstructGPT [Ouyang et al., 2022] and the
FLAN collection [Chung et al., 2024], ensuring clear alignment with human preferences. However,
the cost and scalability limitations of human annotation prompted the development of synthetic
and semi-automatic data pipelines. Self-Instruct [Wang et al., 2022a] proposed using strong language models to generate instructional prompts and initial completions, followed by filtering and
occasional human review. Subsequent works adopted similar strategies: OpenAssistant [K¨opf et al.,
2023], Vicuna [Chiang et al., 2023], and Alpaca [Taori et al., 2023] leveraged LLMs like GPT-3 or
GPT-4 to synthesize large volumes of instruction–response pairs, filtered by heuristics, crowdworker
ratings, or model-based scoring [Li et al., 2023c].
The coverage of the dataset, across tasks, domains, and styles, affects how well the model
generalizes. A model trained only on factoid questions may struggle with multi-step reasoning,
while one tuned primarily on English instructions may fail to align when handling other languages
[Mishra et al., 2021, Qin et al., 2025]. From a statistical perspective, we can think of SFT as
performing maximum likelihood estimation over an empirical distribution; when the support of the
distribution is narrow or biased, generalization to unseen test conditions could suffer.
In addition to representational coverage, instructional quality is critical. When fine-tuning on
responses that contain hallucinations, inconsistencies, or stylistic drift, the model learns to reproduce those flaws. This can be viewed as a form of noisy supervision, where training on imperfect
targets instead of ideal responses. It will introduce gradient bias during optimization and poten
tially reinforce undesired behaviors. To mitigate this, many alignment pipelines employ filtering
mechanisms, such as crowdworker review [Taori et al., 2023], reward-model scoring [Bai et al.,
2022b], or LLM-as-a-judge approaches [Zheng et al., 2023a], to remove or downweight problematic
completions.
Another layer of complexity arises from the need for social and linguistic inclusivity. Recent
studies have shown that alignment can degrade for underrepresented user groups or dialects if
the training data lacks demographic balance [Gallegos et al., 2024, Wang et al., 2024b, Miranda
et al., 2023]. Instruction datasets such as Natural Instructions [Mishra et al., 2021] and FLAN
[Chung et al., 2024] attempt to address this by sampling tasks from a wide range of domains
and contributors. However, maintaining both breadth and quality at scale remains a persistent
challenge.
In sum, the alignment capacity of SFT highly depends on what the model sees. Without
preference signals or evaluative feedback, SFT relies entirely on the implicit policy encoded in the
data. When that data is clean, diverse, and representative, alignment is achievable; when it is
noisy, narrow, or biased, the model will inevitably reflect those deficiencies.

4.2 Role of High-Quality Data and Coverage


The optimization methods employed during SFT play a critical role in how effectively a pretrained
language model adapts to aligned behavior. Although SFT is conceptually straightforward, minimizing cross-entropy loss over instruction–response pairs, the choice of parameter update strategy,
regularization, and tuning mechanism can significantly affect both the efficiency and the stability
of the fine-tuning process.
The standard approach in SFT involves minimizing the token-level cross-entropy loss. Given
a training dataset of instruction–response pairs, the model learns to maximize the probability of
generating the target response tokens _y_ = ( _y_ 1 _, y_ 2 _, ..., yT_ ) conditioned on the input _x_, using the loss
function:
_L_ SFT = _−_
_T_
log _Pθ_ ( _yt | x, y<t_ ) _,_
_t_ =1
where _Pθ_ is the probability assigned by the model parameterized by _θ_ . This objective encourages
the model to approximate the empirical conditional distribution defined by the dataset, effectively
learning through maximum likelihood estimation.
However, fine-tuning large-scale language models with hundreds of billions of parameters can
be prohibitively expensive in terms of memory and compute. To address this, parameter-efficient
fine-tuning (PEFT) methods have been developed. One of the most widely adopted techniques is
Low-Rank Adaptation (LoRA), proposed by [Hu et al., 2022] (2021). LoRA introduces trainable
rank-decomposed matrices into the weight update path, allowing the core pretrained model to
remain frozen while learning task-specific updates in a much smaller subspace. Another prominent
approach is the adapter method introduced by [Houlsby et al., 2019] (2019), where small bottleneck
layers are inserted between the transformer blocks of the model and only these layers are trained
during fine-tuning. These methods drastically reduce the number of trainable parameters, enabling
scalable deployment across domains.
An alternative strategy is prompt tuning, in which a small set of task-specific vectors is optimized as input prompts, without changing the model weights. [Lester et al., 2021] (2021) demonstrated that prompt tuning can perform competitively, especially when combined with large base
models. These PEFT approaches are particularly advantageous in low-resource settings, or when
one wishes to train many domain-specific models efficiently.
Beyond parameter selection, regularization is essential for ensuring stable and generalizable
learning. While dropout and weight decay are commonly used in deep learning, their role in
instruction tuning has been less emphasized in alignment-specific work. However, early stopping
has been empirically validated as a simple yet effective tool to prevent overfitting during finetuning. [Dodge et al., 2020] (2020) demonstrated that monitoring held-out validation performance
and halting training when improvements plateau can significantly reduce variance across fine-tuned
models. More recently, [AlShikh et al., 2023] (2023) proposed using a custom metric, the Instruction
Following Score (IFS), as a stopping criterion. They found that IFS could capture early signs of
misalignment and allow tuning to cease before overfitting to stylistic artifacts in the training data.
These results suggest that, while the loss function remains unchanged, intelligent stopping rules
can improve alignment quality without additional data or model changes.
Fine-tuning performance is also highly sensitive to hyperparameter settings, including learning
rate, batch size, number of epochs, and optimization schedule. [Liu and Wang, 2021] conducted a
systematic study showing that careful hyperparameter tuning can produce larger gains than changing the model architecture itself. In particular, they found that smaller learning rates paired with
moderate batch sizes produced more stable convergence, especially when fine-tuning on instructionheavy datasets. Despite the importance of these factors, hyperparameter configurations are often
underreported in the alignment literature, which may contribute to reproducibility issues and performance variability across open-source SFT implementations.
A growing body of work has also explored advanced optimization techniques, such as supervised
contrastive learning, which combines the cross-entropy loss with an embedding-level objective to
improve representation quality [Gunel et al., 2020]. Other multi-objective optimization frameworks
aim to balance instruction fidelity with style consistency or factual accuracy [Moukafih et al., 2023].
These techniques remain less common in practical alignment pipelines but offer promising directions
for tasks that require more than simple imitation.
Recent advances have also introduced more dynamic fine-tuning strategies. For instance, [Chen
et al., 2024a] (2024) proposed a self-play fine-tuning mechanism, where the model iteratively refines
its outputs by evaluating and responding to its own generations. This reduces the reliance on large
quantities of labeled data and offers a path toward semi-supervised or bootstrapped instruction
tuning.
Overall, the optimization strategies used in SFT, whether through full or parameter-efficient
fine-tuning, not only improve the model’s ability to follow instructions but also its robustness
and safety. While the cross-entropy objective remains the foundation, recent advances in lowrank adaptation, early stopping, and hyperparameter optimization have significantly improved the
practicality of SFT in large-scale alignment systems.


While SFT is effective for instilling instruction-following behavior in large language models, it
exhibits fundamental limitations that restrict its capacity to produce robustly aligned models,
particularly in settings that involve ambiguity, value sensitivity, or complex multi-turn reasoning.
These limitations arise not from the optimization procedure itself, but from the nature of supervision: SFT relies on fixed demonstrations, which encode a single “correct” output per input and
lack an explicit representation of comparative preference or uncertainty.
The most immediate limitation of SFT is its dependence on data coverage. Because the model
is trained to imitate provided responses, it can only generalize to instructions that are similar,
semantically or structurally, to those seen in training. Tasks that fall outside this distribution,
such as rare user intents, unexpected phrasings, or edge-case ethical queries, are likely to elicit
misaligned behavior. In practice, it is infeasible to anticipate and curate demonstrations for the
full diversity of real-world user inputs. As a result, models trained with SFT alone often perform
well in benchmark settings but fail to generalize in deployment scenarios that involve novel or
adversarial instructions [Ouyang et al., 2022, Bai et al., 2022b].
Another limitation stems from the lack of preference awareness. In many alignment-sensitive
contexts, there is not a single objectively correct answer, but rather a range of possible outputs
that vary in quality. For example, multiple valid completions to the same question may differ in
helpfulness, politeness, or factual completeness. Supervised learning provides no mechanism for
distinguishing among these options, as it treats all target outputs as equally correct and penalizes
deviations uniformly. This restricts the model’s ability to internalize graded feedback, a critical
ingredient in aligning behavior to subtle human preferences [Wang et al., 2024c, Fan et al., 2025a].
Techniques such as RLHF were developed precisely to address this deficiency by incorporating
relative judgments between model outputs.
Besides, SFT is vulnerable to label noise and stylistic bias. If the training responses are inconsistent, whether in tone, factual accuracy, or ethical stance, the model learns a blend of conflicting
signals. Since the loss is minimized token by token, the model will imitate local surface patterns even if the overall output is suboptimal. This problem is exacerbated when synthetic or
crowdsourced data is used without rigorous filtering, leading to subtle but persistent misalignment
[Weidinger et al., 2021, Dodge et al., 2020]. Moreover, models trained solely on SFT data may adopt
stylistic patterns that reflect the demographics or ideology of annotators, potentially reproducing
sociolinguistic biases without explicit intent.
Another key limitation is the lack of iterative improvement. SFT is a one-shot process: it encodes aligned behavior through static demonstrations, with no opportunity for feedback or correction after deployment. Once the model has been fine-tuned, errors in alignment must be addressed
by retraining or creating new data, which is time-consuming and inefficient. This stands in contrast to RLHF pipelines, which allow the model to learn from post-hoc evaluations and adapt its
behavior through policy optimization over time [Bai et al., 2022b, OpenAI Achiam et al., 2023].
Finally, there are theoretical limits to what SFT can achieve. From a learning-theoretic perspective, the cross-entropy loss used in SFT does not model the utility or risk associated with
different outputs. It assumes that the demonstrated output is the only valid one. This is an inadequate assumption for alignment tasks, which often require reasoning about social, contextual, or
moral trade-offs. Without an explicit notion of value or preference, SFT cannot prioritize aligned
outcomes over superficially plausible but misaligned ones [Tajwar et al., 2024].
In sum, while SFT is a powerful and efficient method for inducing instruction-following behavior,
its reliance on static demonstrations, inability to model preferences, and limited generalization
capacity render it insufficient for comprehensive alignment. These shortcomings have motivated
the development of reinforcement-based methods, which enable models to learn from feedback,
explore behavioral alternatives, and better adapt to the complex and subjective nature of human
intent.


While SFT is an effective mechanism for teaching language models to imitate desired behaviors, it is
inherently limited by the static nature of its training data and the absence of an explicit preference
signal. To address these limitations, RLHF has emerged as a powerful paradigm for refining model
behavior based on comparative evaluations rather than fixed demonstrations.
The general workflow of RLHF is shown in Figure 2. The RLHF process begins with a set of
input prompts (e.g., “Water is...”), which are fed into a **trainable LLM** _πϕ_ ( _y | x_ ), to generate
multiple candidate responses. For instance, in response to the example prompt, the model might
produce outputs such as “a chemical compound essential for life” or “the source of all life.” These
candidate outputs are then evaluated by **human annotators**, providing feedback including comparative preference judgments (e.g., “Answer 1 is preferred over Answer 2”). These human feedback
data are used to train a **reward model** _Rθ_ ( _x, y_ ) (a separate AI model), which learns to predict
human preferences. The reward model thus functions as a proxy for human evaluators, enabling
scalable automated evaluation of model responses. Once the reward model is sufficiently trained, it
is used to guide the fine-tuning of the original LLM via **reinforcement learning**, most commonly
using _Proximal Policy Optimization (PPO)_ . In this phase, the LLM is treated as a stochastic policy _πϕ_ that aims to maximize the expected reward predicted by _Rθ_ . The reinforcement learning
algorithm updates the model parameters _ϕ_ so that the generated responses yield higher reward
scores. To maintain linguistic fluency and prevent divergence from the pretrained distribution, a
**Kullback–Leibler (KL) divergence penalty** is applied. This penalty constrains the updated
LLM _πϕ_ to remain close to a **frozen reference LLM** _π_ ref, typically the supervised fine-tuned
model before RL. The following subsections provide a detailed review of the key components of the
RLHF framework: human feedback data collection in Section 5.1, reward model training in Section 5.2, and policy optimization methods in Section 5.3. We conclude this section by discussing
the empirical successes of RLHF, along with its practical challenges, such as training instability,
computational cost, and vulnerability to reward hacking.
Figure 2: Overview of the RLHF workflow. A pretrained LLM generates responses to prompts,
which are evaluated by humans to produce preference data. This data is used to train a reward
model, which then guides the policy optimization of the language model via reinforcement learning
(e.g., PPO), with a KL penalty to constrain deviation from the reference model.

5 Reinforcement Learning from Human Feedback (RLHF)


The efficacy of RLHF is profoundly dependent on the quality, type, and collection strategy of
human feedback. Existing methods for collecting human feedback data can be broadly classified
into three primary categories: (1) Preference and rating-based feedback, where humans express
their subjective evaluations, opinions, or levels of satisfaction. It is less about objective correctness
and more about which outputs or behaviors are considered better, more helpful, or more aligned
with user expectations. (2) Correction and language-based feedback, which offers more detailed
guidance than simple preferences. (3) Multi-level feedback.
**Preference and Rating-Based Feedback.** The preference-based subdomain consists of three
principal variants: (1) Pairwise preference. Humans are presented with two different model outputs
for the same prompt or situation and asked to indicate which one they prefer. This is one of the
most common feedback types employed in many state-of-the-art RLHF pipelines [Christiano et al.,
2023b, Stiennon et al., 2022, Bai et al., 2022c, Zhu et al., 2024b] due to its relative simplicity
for annotators. (2) Multiple-choice comparison, which extends to more than two candidates with
selection of a single optimal response [Ziegler et al., 2020a]. Researchers found that presenting
multiple outputs could reduce the cognitive load associated with query comprehension. (3) Ordinal
preference, a.k.a. ranking feedback, which requires complete rankings across multiple outputs
[Ouyang et al., 2022, Zhu et al., 2024b]. In this modality, rather than selecting between just two
options or selecting one from multiple choices, annotators are presented with multiple (more than
two) model-generated outputs for a given prompt. They are then tasked with ranking these outputs
from best to worst according to the defined criteria (e.g., helpfulness, harmlessness, honesty). Such
ordinal data provides a richer signal of relative quality across several candidates compared to a
single pairwise comparison.
Rating feedback offers an alternative method where annotators evaluate each model response
independently based on predefined quality criteria, without necessarily comparing multiple outputs simultaneously. This approach offers advantages in throughput and scalability compared to
preference-based methods [Bakker et al., 2022]. Rating feedback can generally take the following formats: (1) Numerical Rating: Numerical ratings can be continuous or discrete. A widely
adopted discrete numerical scale is the Likert scale, where annotators choose a point on a scale
[Bakker et al., 2022, K¨opf et al., 2023] (e.g., a 5-point scale) representing degrees of agreement,
quality, helpfulness, or other attributes. (2) Categorical Rating: Feedback can also be provided
through categorical labels. This includes binary feedback [Li et al., 2017, Xiao et al., 2020, Scheurer
et al., 2024] (e.g., ‘acceptable’/‘unacceptable’, ‘yes’/‘no’ to specific questions about the response) or
multi-category feedback where the labeler selects from a predefined set of qualitative descriptions
[Huang et al., 2023a, Gao et al., 2023a] (e.g., “Very Helpful,” “Somewhat Helpful,” “Not Helpful,”
“Harmful”).
**Correction and Language Feedback.** In addition to preferences and ratings, human feedback can take more direct and expressive forms that offer deeper supervision signals. Two particularly important subtypes in this category are edit-based feedback and natural language feedback.
(1) Edit-based/Correction Feedback, which involves human annotators directly modifying an
response from LLM, to produce an improved or “corrected” version. This process typically involves operations such as adding, deleting, or rephrasing segments of the text to enhance accuracy,
coherence, tone, or adherence to instructions. The resulting edited response, is generally considered implicitly preferred over the original response. This implicit preference pair can then be used
to train a preference-based reward model, akin to how pairwise preferences are utilized [Shaikh
et al., 2025, Brown et al., 2025]. Furthermore, the corrected response itself can serve as a high
quality demonstration, directly contributing to datasets used for SFT, thereby reinforcing desirable
model behaviors through imitation learning. This dual utility makes edit-based feedback a valuable
component in iterative model refinement.
(2) Natural Language Feedback (NLF), which offers a complementary mechanism through which
human evaluators provide detailed and nuanced critiques in free-form text. Rather than expressing
simple preferences or numerical ratings, annotators articulate specific strengths and weaknesses of
model outputs, often identifying precise segments requiring improvement while suggesting potential
remediation strategies. This approach yields a significantly denser and more informative signal than
scalar rewards or binary preferences, offering multidimensional insights into model performance.
The richness of NLF makes it particularly valuable for improving performance on complex LLM
tasks, such as dialogue generation [Hancock et al., 2019], code generation [Chen et al., 2024b] and
summarization [Scheurer et al., 2024], where quality assessment spans multiple dimensions, including factual accuracy, coherence, relevance, and stylistic appropriateness [Li et al., 2022a]. While
highly informative, the integration of NLF into RLHF pipelines and reward model training presents
challenges since it typically requires sophisticated preprocessing steps to convert the unstructured
textual feedback into a format usable for reward model training.
**Multi-Level Feedback.** As RLHF methodologies have evolved, researchers have recognized
the limitations of single-dimensional feedback mechanisms in capturing the multifaceted nature
of human judgment. Multi-level feedback approaches have emerged to address these constraints,
offering more comprehensive evaluation frameworks that better align language models with complex human values and expectations. This section examines two principal directions in multi-level
feedback: Multi-Signal Feedback and Fine-Grained Feedback.
(1) Multi-Signal Feedback. Multi-Signal Feedback combines diverse human feedback mechanisms, such as pairwise preferences, scalar ratings, and textual critiques, to construct a more comprehensive representation of model performance across multiple dimensions [Glaese et al., 2022,
Metz et al., 2025]. The integration of these complementary signals enables reward models to
capture more complex and multifaceted human values and intentions. For instance, while pairwise
preferences might effectively capture overall quality rankings, they may fail to communicate specific
weaknesses that textual feedback could readily identify. Similarly, scalar ratings might quantify performance along predefined dimensions that categorical preferences cannot express. By synthesizing
these varied signals, hybrid approaches aim to overcome the inherent limitations of individual feedback modalities, potentially offering more fine-grained guidance during policy optimization [Glaese
et al., 2022, Metz et al., 2025].
(2) Fine-Grained Feedback. The human feedback forms discussed in the previous paragraphs
of Section 5.1 often involve labeling evaluations at the level of the entire output. While useful,
such holistic feedback may not suffice for precisely identifying and correcting specific problematic
segments or errors within a response from models. Fine-grained feedback approaches address this
limitation by enabling human annotators to provide evaluations at a finer grained level. This can
include segment-level feedback, where specific spans of text are rated or corrected [Yin et al., 2025,
Wu et al., 2023], or even token-level feedback, where individual tokens receive reward assignments
[Xu et al., 2024a, Li et al., 2024e]. The primary advantage of fine-grained feedback lies in its ability
to localize issues with greater precision. This, in turn, supports more accurate credit assignment
in the reward modeling process, potentially leading to more targeted and efficient policy updates
and improved model behavior, especially in tasks requiring high degrees of accuracy or safety.
**Challenges in Feedback Data.** The quality and nature of human feedback present several
significant challenges. First, human judgments are inherently subjective and often exhibit noise
and inconsistency, with low inter-annotator agreement (typically around 0.6 to 0.7) leading to
unreliable or conflicting supervision signals [Kreutzer et al., 2018, Ziegler et al., 2020b, Stiennon
et al., 2022]. Ambiguity further complicates this issue, as preference pairs may lack a clearly superior
option, especially when differences between responses are subtle or instruction interpretation varies
[Ibarz et al., 2018, Christiano et al., 2023b]. Moreover, collecting high-quality feedback is resourceintensive, limiting scalability and prompting the exploration of sample-efficient alternatives such
as active learning [Gleave and Irving, 2022, Das et al., 2024, Mehta et al., 2025] or AI-generated
feedback (RLAIF) [Bai et al., 2022a, Lee et al., 2024b, Sharma et al., 2024]. However, feedback
from both humans and AI carries distinct forms of bias, human annotators may introduce cultural
or cognitive biases, while AI-generated feedback tends to be lower in noise but systematically biased
due to model limitations. These challenges underscore a central insight in RLHF: feedback quality
is often more critical than quantity, motivating research into robust reward modeling and intelligent
data acquisition strategies that can effectively manage noisy, ambiguous, or biased signals.


Directly supervising an RL agent with human feedback on every output is infeasible for large
models. Instead, the reward model (RM) is trained on a limited set of human preferences using
supervised learning to act as a scalable proxy for human judgment. An RM takes a context (e.g, a
prompt _x_ ) and a candidate output (e.g., an LLM’s response _y_ ) as input, and produces a scalar score:
_Rθ_ ( _x, y_ ) _→_ R, where _θ_ are the learnable parameters, and the function outputs a scalar reward which
reflects how much a human would prefer or approve of the output [Ouyang et al., 2022, Stiennon
et al., 2022]. The learned reward signal is crucial for guiding the LLM toward generating more
aligned and preferred outputs during the subsequent reinforcement learning optimization phase
[Sutton and Barto, 2018]. RMs can be broadly divided into three categories based on the structure
of the preference data they utilize: (1) **Pairwise Comparison Reward Models**, which leverage pairwise preference judgments (e.g., selecting the better of two responses); (2) **Ranking-based**
**Reward Models**, which are trained using more comprehensive ordinal data (k-wise ranking data),
such as a full ranking of three or more candidate responses or the selection of the best response
from a larger set (Best-of- _N_ ); (3) **Attribute-Based and Outcome-Oriented Reward Models**,
which learn to assess response quality by evaluating direct attributes, outcomes, or critiques, such
as absolute numerical scores, the correctness of final answers or intermediate reasoning steps, and
structured language feedback, rather than by learning from relative preferences between different
responses.
**Pairwise Comparison Reward Models.** A popular approach to reward modeling in RLHF
relies on pairwise comparisons between model-generated responses. This method was first introduced and popularized by Ziegler et al. [2020a], and subsequently became foundational to highimpact RLHF systems such as InstructGPT [Ouyang et al., 2022], Anthropic’s Constitutional AI
[Askell et al., 2021, Bai et al., 2022b,a], and Llama 2-Chat [Touvron et al., 2023].
Formally, this framework builds upon the Bradley–Terry–Luce (BTL) model [Bradley and Terry,
1952, Luce, 1959], which describes the probability that a human prefers response _yw_ over _yl_ for
a given prompt _x_ . The preference is modeled as a logistic regression, where the binary label
(preference outcome) indicates whether response _yw_ is preferred over _yl_, and the feature is the
scalar difference in reward scores _Rθ_ ( _x, yw_ ) _−_ _Rθ_ ( _x, yl_ ):
_P_ ( _yw ≻_ _yl | x_ ) = _σ_ ( _Rθ_ ( _x, yw_ ) _−_ _Rθ_ ( _x, yl_ )) = (1)
1 + exp ( _−_ ( _Rθ_ ( _x, yw_ ) _−_ _Rθ_ ( _x, yl_ ))) _[,]_
where _σ_ ( _·_ ) is the sigmoid function. It is important to note that the reward scores are not fixed
input features. Instead, they are outputs of a trainable neural network _Rθ_ . To train the RM,
its parameters _θ_ are optimized by minimizing the negative log-likelihood of the human-provided
preferences in a dataset _D_ pref = _{_ ( _xi, yw,i, yl,i_ ) _}_ _[N]_ _i_ =1 [(where] _[ x][i]_ [ is the] _[ i]_ [-th prompt) using a binary]
cross-entropy loss:
_L_ ( _θ_ ) = _−_ E( _x,yw,yl_ ) _∼D_ pref [log _σ_ ( _Rθ_ ( _x, yw_ ) _−_ _Rθ_ ( _x, yl_ ))] _._ (2)
For better numerical stability, this is often expressed in its log-sum-exp form:
_L_ ( _θ_ ) = E( _x,yw,yl_ ) _∼D_ pref [log (1 + exp ( _−_ ( _Rθ_ ( _x, yw_ ) _−_ _Rθ_ ( _x, yl_ ))))] (3)
_Limitations and Refinements._ While widely used, the pairwise BTL framework exhibits several
limitations that have motivated subsequent research: (1) A key limitation of the standard BTL
model is its assumption that all preferences are of equal strength. It only captures the direction of
preference (i.e., _yw_ is better than _yl_ ), not the magnitude (i.e., how much better it is). In practice,
a human labeler might find one response to be marginally better, while another might be vastly
superior. To address this limitation, the loss function can be modified to account for the strength of
human preference, which is often collected on a rating scale (e.g., a Likert scale). As demonstrated
in Llama 2 [Touvron et al., 2023], a margin term _m_ ( _r_ ) that is a function of the preference rating _r_
is added to the loss function:
_L_ ( _θ_ ) = _−_ E( _x,yw,yl,r_ ) _∼D_ pref [log _σ_ ( _Rθ_ ( _x, yw_ ) _−_ _Rθ_ ( _x, yl_ ) _−_ _m_ ( _r_ ))] (4)
This modification encourages the RM to create a larger gap in the reward scores for pairs with a
stronger preference, leading to a more calibrated and nuanced reward signal. (2) Another challenge
lies in choosing the size and architecture of the reward model. As reported in Askell et al. [2021],
Ouyang et al. [2022], large reward models tend to overfit small preference datasets, while smaller
models, though more stable, may lack the capacity to learn subtle distinctions in human preferences.
This trade-off remains an active area of investigation, with the optimal RM size often depending
on the specific application, the size of the preference dataset, and the available computational
resources.
**Ranking-based Reward Models.** To utilize more informative feedback, such as a K-wise
ranking of multiple responses ( _y_ 1 _≻_ _y_ 2 _≻· · · ≻_ _yk_ ) or the selection of the single best response from a
set of _N_ candidates (Best-of- _N_ ), which naturally arise in many annotation settings, ranking-based
reward models have been proposed. The key challenge became how to effectively incorporate this
ordinal data into a trainable objective. The community has largely adopted two distinct strategies
to solve this: (1) a pragmatic **pairwise decomposition method**, which converts each ranked list
into a series of independent pairwise comparisons, and (2) a more statistically principled **listwise**
**modeling method**, which directly uses the entire ranking.
_Pairwise decomposition method._ The initial and most straightforward strategy is to decompose
a full ranking of _k_ items into - _k_ 2� pairwise preferences. For example, a ranking ( _y_ 1 _≻_ _y_ 2 _≻_ _y_ 3)
yields three pairs ( _y_ 1 _≻_ _y_ 2), ( _y_ 1 _≻_ _y_ 3), and ( _y_ 2 _≻_ _y_ 3). Each pair is then used to train an RM
using the standard pairwise loss function. This decomposition approach has been adopted in
several foundational RLHF pipelines, including InstructGPT and ChatGPT [Stiennon et al., 2022,
Ouyang et al., 2022, Schulman et al., 2022]. The main advantage of this method is its simplicity, as it
allows practitioners to reuse the well-established BTL framework without modification. However,
this approach has two key limitations: it ignores the holistic context of the full ranking, and it
artificially inflates the dataset size, which can be computationally inefficient.
_Listwise modeling method._ To address aforementioned shortcomings, listwise approaches directly model the probability of an entire ranked list using a statistical framework like the PlackettLuce model [Plackett, 1975, Luce, 1959]. The parameters _θ_ of the listwise RM are optimized by
minimizing the negative log-likelihood of the observed rankings. Recent empirical and theoretical work has demonstrated the advantages of listwise learning. For example, Zhu et al. [2024b]
show that listwise training is asymptotically more statistically efficient than pairwise decomposition, leading to more accurate reward estimation with fewer annotations. Building on this, Zhu
et al. [2024c] introduced a k-wise listwise loss in their Starling-7B reward model, achieving stateof-the-art alignment performance. Despite their promise, listwise models can be more complex to
implement and require careful optimization, which has limited their adoption relative to simpler
pairwise approaches [Cao et al., 2007, Liu, 2009].
**Attribute-Based and Outcome-Oriented Reward Models.** While pairwise and rankingbased reward models have become standard in preference modeling for RLHF, they are limited in
settings where relative comparisons are insufficient or ill-defined. In domains requiring fine-grained
evaluation, such as mathematical reasoning, code generation, or factual correctness, more direct
forms of supervision are essential. This has led to the development of alternative reward modeling
paradigms that incorporate absolute scoring, outcome validation, process-level critique, and even
language-based feedback. This section explores reward models trained on: absolute scores (pointwise), verifiable outcomes for reasoning tasks (outcome- and process-supervised), and structured
language critiques.
(1) **Point-wise Reward Models** . One of the earliest alternatives to pairwise comparison is the
use of absolute scalar feedback. In this framework, a reward model is trained to regress onto humanprovided scores (e.g., Likert ratings), treating reward learning as a supervised regression problem.
The objective typically minimizes the mean squared error (MSE) between the model’s output and
the target score. While this approach is conceptually straightforward and has been explored in early
work [Ziegler et al., 2019], it suffers from issues related to human annotation noise, such as interrater and intra-rater variability. As a result, point-wise RMs are rarely used as standalone reward
functions in modern RLHF pipelines. Instead, they are more commonly employed for auxiliary
supervision or post-hoc evaluation [Stiennon et al., 2022, Wang et al., 2025b,c].
(2) **Reward Models for Reasoning Tasks** For reasoning-intensive tasks, such as mathematical problem-solving or complex code generation [Wu et al., 2024a, Li et al., 2022b], alignment
depends less on subjective preference and more on functional correctness. Two specialized types
of RMs, Outcome and Process RMs, provide supervision in these domains by evaluating the correctness of the generated solution. _Outcome Reward Models (ORMs)_ are trained by evaluating the
final result of a generated response. Typically, an ORM learns to predict a scalar value representing
the probability of a successful outcome, such as a correct mathematical solution or a passing unit
test [Cobbe et al., 2021, Uesato et al., 2022]. Some implementations provide this feedback on a
per-token basis to create a denser reward signal for more efficient learning [Cobbe et al., 2021, Lyu
et al., 2025]. Process Reward Models (PRMs), in contrast, provide more fine-grained feedback by
evaluating the intermediate steps of a model’s reasoning process. Also known as Process-Supervised
RMs, this step-by-step supervision is crucial for tasks where the correctness of the reasoning path
is as important as the final outcome. Several studies have demonstrated the efficacy of PRMs in
enhancing the correctness and interpretability of the reasoning process in complex domains like
mathematics [Uesato et al., 2022, Lightman et al., 2023].
(3) **Language Feedback Reward Models** This approach uses natural language critiques or
corrections for reward generation, which can provide dense and precise supervisory signals. Since
this feedback is unstructured, it presents unique modeling challenges. Instead of predicting a
scalar reward, the RM may be trained as a text-generation model to perform tasks like “Correction
Mapping” (transforming a rejected response into a chosen one) and “Identity Mapping” (outputting
a chosen response as is). The resulting text outputs are then processed to extract dense, tokenlevel reward signals for the policy optimization phase, offering more granular guidance [Zhou et al.,
2024a].
**Challenges in Reward Modeling.** Despite the successes of RLHF, the reward modeling
stage presents several challenges that can undermine the alignment, safety, and utility of the final
LLMs: (1) **Reward Misspecification** This occurs when the learned RM fails to faithfully represent
true human preferences [Peng et al., 2023b, Bobu et al., 2024]. This discrepancy can arise from the
inherent difficulties in collecting comprehensive and nuanced human feedback, but it also stems from
a more fundamental limitation: it is statistically impossible for a single reward function to represent
a diverse group’s preferences if those preferences contain intransitive cycles (e.g., A is preferred to
B, B to C, and C to A), a phenomenon known as the Condorcet paradox, meaning any single reward
model is an inherently flawed representation of the group’s preferences accurately [Liu et al., 2025a].
Consequently, the RM may learn an incomplete or skewed representation of the intended objectives.
Humans may struggle to articulate complex preferences or provide feedback that exhaustively
covers all desired behaviors, particularly for tasks demanding sophisticated reasoning, creativity,
or nuanced ethical considerations. Consequently, the RM may learn an incomplete or skewed
representation of the intended objectives. (2) **Misgeneralization and Reward Hacking** Another
critical issue is the misgeneralization of the reward model, which leads directly to a failure mode
known as **reward hacking** (or overoptimization). Misgeneralization occurs when the RM learns
a flawed proxy for human values that fails to generalize robustly to out-of-distribution (OOD)
prompts or novel response styles [Tien et al., 2023]. This vulnerability allows the policy, a powerful
optimizer, to exploit the RM’s inaccuracies during reinforcement learning. The resulting behavior
is reward hacking: the LLM achieves a high score according to the flawed RM but fails to align
with the actual, more complex human preferences it was meant to capture [Gao et al., 2023b,
Laidlaw et al., 2025, Skalse et al., 2025]. In effect, the LLM finds loopholes in the reward function,
for instance, generating overly verbose answers because the RM incorrectly associates length with
quality, rather than genuinely satisfying the intended goals. (3) **Reliable evaluation** Evaluating
the quality of an RM is intrinsically difficult because the “ground truth” human preference function
is unknown. RM performance is typically measured indirectly through the performance of the policy
trained on it, making it hard to diagnose and debug issues with the RM itself. Developing robust,
direct evaluation metrics for RMs remains an open research problem. Addressing these multifaceted
challenges in reward modeling is pivotal for advancing the development of helpful, harmless, and
safe aligned LLMs.


Once a reward model _Rθ_ has been established, the next critical step in RLHF is optimizing the
pre-trained language model to align with human preferences. This optimization phase treats the
language model as a policy _πϕ_ ( _a | s_ ) that must learn to generate actions (tokens) _a_ given states
(text sequences) _s_ in ways that maximize rewards from _Rθ_ . The core challenge is transforming a
pre-trained LLM into one that consistently produces outputs favored by human evaluators. This
transformation requires systematically adjusting the policy parameters _ϕ_ to find an optimal policy
_πϕ_ _[∗]_ [that maximizes expected rewards across diverse input prompts [][Ziegler et al.][,][ 2020b][,][ Ouyang]
et al., 2022].
Policy gradient algorithms provide the mathematical framework for this optimization [Sutton
et al., 1999]. These methods iteratively update the policy by maximizing an objective function
_J_ ( _ϕ_ ) that captures the expected cumulative reward:
_J_ ( _ϕ_ ) = E _τ_ _∼πϕ_ [ _R_ ( _τ_ )] = E _τ_ _∼πϕ_
- _T_
_γ_ _[t]_ _rt_
_t_ =0
_,_ (5)
where _τ_ = ( _s_ 0 _, a_ 0 _, s_ 1 _, a_ 1 _, . . ., sT, aT_ ) represents a complete trajectory generated by the policy, where
_rt_ = _Rθ_ ( _st, at_ ) is the reward at timestep _t_, and _γ ∈_ [0 _,_ 1] is a discount factor balancing immediate
versus future rewards. The policy gradient theorem provides the analytical form of _∇ϕJ_ ( _ϕ_ ), which is
used to update the policy via gradient ascent. However, in practice, directly applying this gradient
often results in unstable or inefficient learning. This motivates the development of alternative
algorithms that compute modified or constrained versions of the policy gradient.
Within the broad class of policy gradient methods, several specialized algorithmic families have
been developed for RLHF, each tailored to address specific challenges such as stability, efficiency,
or feedback sparsity. We categorize the most commonly used approaches as follows.
1. **Actor-Critic PPO** : These methods maintain both a policy (actor) and a value function
(critic). PPO is the most widely used due to its training stability, achieved by clipping policy
updates and using the critic to estimate advantages [Schulman et al., 2017, Ouyang et al.,
2022].
2. **Actor-Only Policy Gradients** : These methods bypass the critic and directly optimize
the policy using rewards from the reward model. While simpler and often more efficient,
they may suffer from higher variance in updates and typically require alternative variance
reduction strategies.
3. **Specialized and Hybrid Reward-Based Methods** : This category includes algorithms
designed to address specific RLHF challenges, such as reward hacking, exploration limitations,
and training instabilities that arise in large-scale language model optimization.
We summarize existing policy optimization methods in Table 4. In the following subsections, we
review each category in detail, with a particular focus on PPO, the most widely adopted method
in RLHF.


PPO was first introduced by [Schulman et al., 2017] at OpenAI as a solution to the instability
and complexity issues inherent in existing policy optimization methods. The algorithm emerged
from the recognition that while Trust Region Policy Optimization (TRPO) [Schulman et al., 2015]
provided theoretical guarantees for stable policy updates, its implementation required computationally expensive second-order optimization procedures, including conjugate gradient methods and line
searches. PPO is a cornerstone algorithm in many state-of-the-art RLHF pipelines for LLMs. Its
extensive utilization in fine-tuning influential models, such as InstructGPT [Ouyang et al., 2022],
the models underlying ChatGPT [OpenAI et al., 2024], and Anthropic’s Claude series [Askell et al.,
2021, Bai et al., 2022a], underscores its significance. Furthermore, PPO has been a prevalent choice
for aligning various open-source models, including Llama 2-Chat [Touvron et al., 2023].
The core motivation behind PPO was to retain the stability benefits of trust region methods
while dramatically simplifying the implementation and reducing computational overhead. [Schulman et al., 2017] observed that many policy gradient failures stemmed from excessively large policy
updates that could catastrophically degrade performance, leading them to design a method that
would naturally constrain update magnitudes without requiring complex second-order computations.
**Algorithmic Framework.** PPO addresses the stability concerns of vanilla policy gradients
through a clipped surrogate objective that prevents destructively large policy updates. At its core,
PPO is an actor-critic algorithm. The **actor** is the LLM policy _πϕ_ ( _a | s_ ), which maps a state _s_
(e.g., a prompt or partially generated sequence) to a distribution over actions _a_ (e.g., next tokens).
The **critic** is a value function _Vψ_ ( _s_ ), which estimates the expected return from state _s_ under the
current policy, where _ψ_ are parameters in the value function. These two components are trained
jointly: the actor is updated via a clipped policy gradient objective, while the critic is optimized
using a regression loss (e.g., mean-squared error) against the empirical return.
(1) Actor optimization loss. The key innovation in PPO is its clipped surrogate objective, which
prevents large, destabilizing updates by penalizing policy changes that deviate too far from the old
policy. Let _ρt_ ( _ϕ_ ) = _πϕπ_ old _ϕ_ ( _a_ ( _ta|ts|ts_ ) _t_ ) [denote the probability ratio between the new policy and the old]
policy at time step _t_ . PPO seeks to maximize the following objective:
              -               - ��
_L_ [CLIP] ( _ϕ_ ) = E [�] _t_ min _ρt_ ( _ϕ_ ) _A_ [�] _ϕ_ ( _st, at_ ) _,_ clip ( _ρt_ ( _ϕ_ ) _,_ 1 _−_ _ϵ,_ 1 + _ϵ_ ) _A_ [�] _t_ _,_ (6)
where _A_ [�] _t_ is an estimate of the advantage function at timestep _t_, quantifying how much better
action _at_ is compared to the average action in state _st_ . This objective ensures that when the policy
change is small ( _ρt ≈_ 1), the standard policy gradient is recovered. However, if _ρt_ moves outside
the interval [1 _−_ _ϵ,_ 1 + _ϵ_ ], the clipped objective dampens the update, effectively acting as a soft
constraint that stabilizes training.
A key component in the Equation (6) is the estimated advantage function, which is formally
defined as _A_ [�] _t_ = _Q_ ( _st, at_ ) _−_ _Vψ_ ( _st_ ), where the first term _Q_ ( _st, at_ ) is the action-value function,
representing the expected return when the agent starts in state _st_, takes action _at_, and subsequently follows the current policy. The second term _Vψ_ ( _st_ ) is the state-value function, computing the average return over all actions sampled from the policy’s distribution at that state, i.e.,
_Vψ_ ( _st_ ) = E _a∼π_ ( _·|st_ )[ _Q_ ( _st, a_ )]. However, since the true value functions _Q_ and _Vψ_ are unknown and
typically intractable to compute in complex environments, practical algorithms must rely on samplebased approximations. A common and simple approach is to estimate _Q_ ( _st, at_ ) using the one-step
temporal difference (TD) target [Sutton, 1988, Sutton and Barto, 2018]: _Q_ ( _st, at_ ) _≈_ _rt_ + _γVψ_ ( _st_ +1),
where _rt_ = _Rθ_ ( _st, at_ ) is the immediate reward after taking action _at_ at state _st_, and _Vψ_ ( _st_ +1) is the
critic’s prediction of the next state’s value. This leads to the TD residual estimate of the advantage:
_A_ - _t_ = _rt_ + _γVψ_ ( _st_ +1) _−_ _Vψ_ ( _st_ ). While easy to compute, this estimate can be noisy and suffer from
high variance, because any small prediction error in _Vψ_ leads to unstable and misleading advantage
estimates. To address this, Generalized Advantage Estimation (GAE) [Schulman et al., 2018] is
often used. GAE provides a bias-variance trade-off by computing a weighted sum of multi-step
TD errors. GAE defines the TD error as _δt_ = _rt_ + _γV_ ( _st_ +1) _−_ _V_ ( _st_ ), and aggregate them as
_A_ - [GAE] _t_ = [�] _l_ _[∞]_ =0 [(] _[γλ]_ [)] _[l][δ][t]_ [+] _[l]_ [, where] _[ λ][ ∈]_ [[0] _[,]_ [ 1] is a decay factor. Smaller] _[ λ]_ [ values emphasize short-term]
estimates with lower variance, while larger _λ_ values incorporate longer-term information, reducing
bias. As such, GAE enables more stable and data-efficient learning, and has become a standard
advantage estimator in modern actor-critic algorithms like PPO.
(2) Critic optimization loss. While the actor learns via the _L_ [CLIP] objective, the critic, _Vψ_ ( _st_ ),
learns by minimizing a mean squared error (MSE) loss between its predictions and a target value
which is calculated using previous (old) critic parameters _ψ_ old: _V_ [�] [target] ( _st_ ).
                    _L_ [VF] ( _ψ_ ) = E [�] _t_ ( _Vψ_ ( _st_ ) _−_ _V_ [�] [target] ( _st_ )) [2][�] _._ (7)
Here, the choice of the target value _V_ [�] _t_ [target] critically impacts training stability and performance. A
popular choice is to use GAE to calcualte critic target: _V_ [�] _t_ [target] = _Vψ_ old( _st_ ) + _A_ [�][GAE] _t_ _._ This creates a
clear objective for the critic: its new prediction, _Vψ_ ( _st_ ), should move closer to the old prediction plus
any “surprise” captured by the advantage. To further stabilize training, some PPO implementations
also clip the value function loss, preventing excessively large updates to the critic, particularly if
value predictions are far off from target values.
(3) Additional loss. Two additional terms are commonly introduced to the overall objective.
First, to encourage exploration, an entropy bonus is added. This term, _S_ ( _ϕ_ ), incentivizes the policy
to maintain randomness in its action choices and is maximized during training:
_−_ - _πϕ_ ( _a | st_ ) log _πϕ_ ( _a | st_ )
_a_
_S_ ( _ϕ_ ) = E [�] _t_ [ _H_ ( _πϕ_ ( _· | st_ ))] = E [�] _t_
_._ (8)
Second, to prevent the LLM from deviating too drastically from a trusted base model (a pretrained model, or the supervised fine-tuned model, which possesses coherent language generation
capabilities and general knowledge), a common practice is to incorporate a Kullback-Leibler (KL)
divergence penalty. This penalty, _D_ KL( _πϕ ∥_ _π_ ref), measures the divergence between the current
policy _πϕ_ and a frozen reference policy _π_ ref (often a frozen copy of the SFT model or the initial
pre-trained model) [Schulman et al., 2017, Schulman, 2020, Jaques et al., 2017, 2020, Ziegler et al.,
2020b, Ouyang et al., 2022, Stiennon et al., 2022]. By regularizing against this reference model, the
KL penalty helps mitigate catastrophic forgetting and prevents the policy from generating OOD
text that might exploit flaws in the reward model.
Combining these elements, the final objective function for PPO is a weighted sum:
_L_ ( _ϕ, ψ_ ) = _−L_ [CLIP] ( _ϕ_ ) + _c_ 1 _L_ [VF] ( _ψ_ ) _−_ _c_ 2 _S_ ( _ϕ_ ) + _c_ 3 _D_ KL( _πϕ ∥_ _π_ ref) _,_ (9)
where _c_ 1, _c_ 2, _c_ 3 are coefficients weighting the importance of the value function loss and the entropy
bonus, respectively.
**Limitations of PPO.** Despite its popularity, PPO exhibits several limitations that are particularly relevant in the content of LLM alignment. (1) **Hyperparameter Sensitivity** : PPO’s
performance is notably sensitive to hyperparameter configurations, including learning rates, the
clipping range _ϵ_, batch size, and GAE parameters. Tuning these can be resource-intensive and
requires careful experimentation to achieve optimal results [Schulman et al., 2017, Engstrom et al.,
2020]. (2) **Computational and Memory Costs** : PPO imposes substantial computational and
memory costs when used for RLHF, creating a significant bottleneck for training large language
models. The primary driver of this overhead is the need to hold at least four separate models in
GPU memory simultaneously: the trainable Actor (policy), a frozen Reference model, a Reward
model, and a Critic. This requirement alone can cause the PPO stage to consume over three times
the memory of standard SFT, making it prohibitively expensive and infeasible for many practitioners. In addition to these high static memory costs, the algorithm is computationally intensive due
to the repeated inference passes required from these models during the experience generation phase.
These combined demands severely limit the scalability and accessibility of PPO-based alignment
[Ouyang et al., 2022, Ramamurthy et al., 2023, Rafailov et al., 2024]. (3) **Sample Inefficiency** :
as an on-policy algorithm, PPO requires fresh samples generated from the current policy for each
update. This makes it inherently less sample-efficient than off-policy algorithms like Soft ActorCritic (SAC) [Haarnoja et al., 2018] or Deep Q-Networks (DQN) [Mnih et al., 2015], which can
reuse past experiences stored in a replay buffer. (4) **Local Optima** : While the clipping mechanism
enhances stability, it can also sometimes be overly restrictive, potentially slowing down convergence
or leading to a suboptimal policy if the updates are constrained too much.[Schulman et al., 2017].
(5) **Reward Model Dependence** : The quality of the learned policy is profoundly dependent on
the accuracy and reliability of the RM. A misspecified, biased, or exploitable RM can misguide
PPO towards undesirable behaviors or “reward hacking” [Gao et al., 2023b, Ziegler et al., 2020b,
Stiennon et al., 2022, OpenAI et al., 2024]. (6) **Algorithmic Bias from KL Regularization** :
The standard KL divergence penalty used in RLHF introduces an inherent algorithmic bias. Because the reference model ( _πref_ ) is itself not perfectly aligned, its biases are passed to the fine-tuned
model. In extreme cases, this can lead to “preference collapse”, where the model learns to completely disregard minority preferences, a bias that persists even with a perfect reward model [Xiao
et al., 2024]. Addressing these challenges remains an active area of research to further enhance the
robustness and applicability of PPO in aligning LLMs.
To address these issues, several variants of PPO have been proposed. These can be broadly
grouped by the challenge they target:
(1) **Enhancing Stability and Mitigating Hyperparameter Sensitivity** . Several variants
have been developed to improve PPO’s stability and convergence behavior by refining the core
surrogate objective. KL-regularized PPO (KL-PPO) augments the clipped surrogate loss with an
explicit KL divergence penalty, which softly constrains the updated policy to remain close to the
previous one, functioning as a trust-region regularizer [Ouyang et al., 2022]. An extension of this
idea, adaptive KL control, dynamically adjusts the strength of the KL penalty based on the observed divergence between the current and reference policies, preventing the policy from straying
too far from a known-good distribution [Ziegler et al., 2020b]. More recently, [Shen and Zhang,
2024] introduced Policy Filtration for PPO (PF-PPO) to address instability arising from inaccurate reward models. Motivated by the observation that RMs, particularly in complex domains such
as code generation, are more reliable at ranking extremely good or bad samples than moderately
scored ones. PF-PPO filteres out low-confidence trajectories and concentrates policy learning on
those with more trustworthy reward signals, thereby reducing reward hacking and improving alignment performance. More recently, Proximal Policy Optimization with Reward-based Prioritization
(RP-PPO) was proposed to combat performance degradation and slow convergence in later training
stages. RP-PPO dynamically adjusts the number of policy update epochs based on reward quality; high-quality experiences are given greater weight through more training rounds, providing an
incentive for the model to move towards higher-reward policies. Furthermore, RP-PPO explicitly
saves the model that achieves the highest historical average reward, rather than relying on the final
model, thus preventing the loss of the best-found policy due to later training instability [Zheng
et al., 2025].
(2) **Reduce computational and memory costs.** Some system-level innovations were implemented to address this limitation. To reducing the GPU memory footprint, for example, HydraPPO introduces techniques like model sharing and “Dynamic LoRA”, where a single base model
is used for multiple roles (e.g., Actor and Reference) by dynamically activating or deactivating
LoRA adapters, significantly cutting down the number of full models that need to be stored in
VRAM [Santacroce et al., 2023]. This approach builds on the general use of PEFT methods like
LoRA, which drastically lower the memory required for optimizer states by only training a small
fraction of the model’s parameters [Hu et al., 2021]. Some improvements focus on scaling PPO
across multiple machines for faster wall-clock training times. Standard distributed methods can
suffer from communication bottlenecks. Decentralized Distributed PPO (DD-PPO) addresses this
by eliminating the central parameter server and using direct peer-to-peer gradient synchronization
among workers. To prevent slowdowns from non-uniform workloads, DD-PPO also introduces a
preemption mechanism to mitigate the “straggler effect”, where fast workers would otherwise wait
for the slowest one [Wijmans et al., 2020]. Together, these system-level optimizations are critical
for making PPO a practical and efficient algorithm for state-of-the-art LLM alignment.
(3) **Improving Sample Efficiency.** To enhance PPO’s sample efficiency, a major line of research focuses on creating hybrid algorithms that safely incorporate off-policy data from a replay
buffer, blending the stability of PPO with the data efficiency of off-policy learning. These hybrid
methods vary in their approach. One strategy focuses on the gradient update itself; for example,
P3O (Policy-on Policy-off Policy-over) interleaves on-policy and off-policy gradient updates and
uses the Effective Sample Size (ESS) to automatically control the trade-off, allowing it to effectively leverage past data without introducing new, sensitive hyperparameters [Fakoor et al., 2019].
Another key strategy involves redesigning the training loop. Phasic Policy Gradient (PPG) separates training into two distinct, alternating phases: a policy phase for standard policy updates and
an auxiliary phase that exclusively reuses collected experience to more aggressively train the value
function. This decoupling allows for better-trained features while mitigating interference between
the policy and value objectives. Building on the concept of experience replay [Cobbe et al., 2020].
Building on the concept of experience replay, Hybrid-Policy PPO (HP3O) utilizes a trajectory
replay buffer with a “first-in, first-out” (FIFO) strategy to limit data distribution drift by only
using recent policies. It further guides learning by always including the trajectory with the best
return from the buffer in each training batch [Liu et al., 2025b]. While these methods focus on
empirical performance, others like Transductive Off-policy PPO (ToPPO) aim for stronger theoretical guarantees. ToPPO introduces a novel transductive inference mechanism to justify using the
advantage function directly from the data-generating policy, avoiding common estimation biases.
This allows it to safely integrate off-policy data while striving for monotonic policy improvement
[Gan et al., 2024]. Although their mechanics differ, these variants all showcase a powerful trend
towards making PPO more data-efficient by safely and strategically reusing past experiences.