Actor-only methods specifically refer to algorithms that optimize the policy directly without explicitly training or maintaining a separate value-function (critic). Instead, these methods use simplified
baselines computed from reward signals to reduce variance in policy gradient updates. This significantly simplifies training because we do not need to fit or maintain a critic network. Broadly,
based on how each method calculates the baseline value used for computing the advantage in policy
gradient updates, existing actor-only methods can be categorized into three groups: (1) the foundational **REINFORCE algorithm**, which represents the classic Monte Carlo approach to policy
gradients and typically uses a simple, historically averaged reward baseline; (2) methods based on
**Intra-Prompt Comparison**, which create a localized baseline from multiple responses to a single
prompt; and (3) methods employing **Batch-Wise Normalization**, which derive a more global
baseline from the statistics of an entire mini-batch.
**The Foundational Method: REINFORCE.** The REINFORCE algorithm is the original
Monte Carlo policy gradient method [Williams, 1992a]. In its most basic form, vanilla REINFORCE
works by sampling a complete response trajectory _y_ from the policy _πϕ_ and updating the policy
parameters to increase the probability of trajectories that received high rewards. The intuition is
straightforward: if a generated response _y_ receives a high reward _R_ ( _x, y_ ), we increase its probability by moving the policy parameters in the direction of _∇ϕ_ log _πϕ_ ( _y|x_ ). Conversely, low-reward
responses have their probabilities decreased. While conceptually elegant, vanilla REINFORCE suffers from extremely high variance in its gradient estimates. Since the algorithm uses the full reward
_R_ ( _x, y_ ) to weight each update, small random fluctuations in rewards can cause dramatic changes
in gradient estimates. This leads to unstable training where the policy may oscillate wildly rather
than converging to optimal behavior.
To address this variance issue, REINFORCE introduces a baseline _b_ ( _x_ ) that is subtracted from
the reward without introducing bias into the gradient estimate:
_∇ϕJ_ ( _ϕ_ ) _≈_ E _x∼D,y∼πϕ_ ( _·|x_ )[( _R_ ( _x, y_ ) _−_ _b_ ( _x_ )) _∇ϕ_ log _πϕ_ ( _y|x_ )] (10)
The baseline typically represents an estimate of the expected reward for input _x_, such as a moving
average of past rewards. By centering the rewards around this baseline, the algorithm reduces
variance: responses that perform better than average receive positive weight, while below-average
responses receive negative weight. Despite the baseline improvement, REINFORCE with simple
baselines remains sample-inefficient and high-variance for state-of-the-art LLM alignment tasks.
The method still requires complete trajectory rollouts and provides learning signals only at the
episode level.
**(2) Intra-Prompt Comparison: Localized Baselines.** A powerful way to create a better
baseline is to have the model compete with itself. This family of methods generates multiple
candidate responses for a single prompt and then uses their relative rewards to create a stable and
highly localized advantage signal.
**REINFORCE with Leave-One-Out Baseline (RLOO)** . The earliest and most influential intra-prompt comparison method is RLOO, which extends basic REINFORCE introducing a
sophisticated, prompt-specific baseline from a group of _K_ sampled responses to reduce variance
[Kool et al., 2019, Ahmadian et al., 2024]. Instead of relying on a globally trained critic like PPO
or a simple moving average baseline like REINFORCE, RLOO calculates the baseline for any given
response as the average reward of the other _K −_ 1 peer responses from the same prompt:
_b_ RLOO( _x, y_ [(] _[i]_ [)] ) =
_K −_ 1
_K_
_Rθ_ ( _x, y_ [(] _[j]_ [)] ) (11)
_j_ =1 _,j_ = _i_
This localized baseline effectively captures prompt-specific difficulty and provides more stable variance reduction than global averages. However, RLOO’s fundamental limitation is its computational
cost [Gao et al., 2024, Hu et al., 2025]: requiring _K_ samples per update makes it _K_ times more
expensive than basic REINFORCE. This makes RLOO a pragmatic choice in scenarios where multiple candidates are already being generated, such as for best-of- _N_ sampling, as it efficiently reuses
those forward passes for variance reduction [Kool et al., 2019, Ahmadian et al., 2024, Hu et al.,
2025]. Furthermore, the baseline’s effectiveness is critically dependent on the diversity and quality
of the sampled responses; if the _K_ completions are too similar or contain outliers, the baseline’s
ability to reduce variance is diminished [Gao et al., 2024].
To address the efficiency bottleneck of RLOO, the **ReMax** method [Li et al., 2024f] proposes a
simpler baseline: for each prompt, generate just two responses, a greedy (deterministic, most-likely)
output and a stochastic sample. The greedy response serves as a cheap, prompt-specific baseline:
_A_ ReMax( _x, y_ samp) = _Rθ_ ( _x, y_ samp) _−_ _Rθ_ ( _x, y_ greedy) (12)
This reduces the number of forward passes required per update from _K_ (in RLOO) to just two,
greatly improving efficiency. Empirical results show that this technique substantially reduces the
variance of the policy gradient compared to vanilla REINFORCE, leading to stable convergence
[Li et al., 2024f]. However, the effectiveness of this method hinges on the quality of the greedy
baseline itself; a poorly trained policy might yield an uninformative greedy sample, potentially
limiting variance reduction or even increasing it in certain policy states [Li et al., 2024f, Hu et al.,
2025].
As intra-prompt baselines evolved, researchers recognized that not only the mean but also the
variability of rewards across a prompt’s responses can carry important information. **Group Rela-**
**tive Policy Optimization (GRPO)** [Shao et al., 2024a, DeepSeek-AI et al., 2025b,c] addresses
a fundamental statistical limitation of both RLOO and ReMax: **reward scale sensitivity** . While
previous methods center rewards around baselines, they don’t account for varying prompt difficulties or reward scales. GRPO normalizes the advantage signal by computing standardized scores:
_Ai_ = _[r][i][ −]_ [mean(] _[{][r]_ [1] _[, . . ., r][G][}]_ [)] (13)
std( _{r_ 1 _, . . ., rG}_ ) + _ϵ_ norm
where _ϵ_ norm is a small constant for numerical stability. This makes the advantage robust to promptspecific reward scales and difficulties, allowing stable learning even across highly diverse prompts.
However, this powerful normalization introduces its own limitations: (1) High generation cost. It
requires generating _G_ responses per prompt to calculate the group-based advantage, which increases
the cost of each training update. (2) Instability and potential bias. Reliance on the group’s standard
deviation for normalization can create unintended biases, such as over-rewarding groups of responses
that have very low reward variance (e.g., all are mediocre) or amplifying noise if the group size _G_
is too small. (3) Overfitting on Simple Prompts. Because GRPO calculates baselines separately
for each prompt using these group statistics, it might be prone to overfitting simpler prompts
if not carefully managed. (4) Sensitivity to sampling strategy. Its overall performance can be
sensitive to the number of samples _G_ per group and the diversity (and thus the reward variance) of
responses within those groups, and it can struggle with imbalanced data, as these factors influence
the stability and informativeness of the advantage estimates [Hu et al., 2025].
Recent variants address GRPO’s specific limitations. **Dr. GRPO** [Liu et al., 2025c] tackles
optimization biases by selectively removing normalization terms to address length and difficulty
biases. **DISCO** [Zhou et al., 2025b] addresses multi-domain performance issues through domainaware reward scaling. These refinements demonstrate the ongoing evolution of the intra-prompt
comparison paradigm, with each method building upon its predecessors’ strengths while addressing
their specific computational, statistical, or application-specific limitations.
**(3) Batch-Wise Normalization: A Global Perspective.** To address the computational expense and overfitting risks of intra-prompt baselines, batch-wise normalization computes a global
baseline from a mini-batch, typically using only one completion per prompt. A key example is **RE-**
**INFORCE++** [Hu et al., 2025], which builds on the REINFORCE algorithm. REINFORCE++
calculates token-level advantages, incorporating the sequence’s reward and a penalty to prevent the
policy from drifting from a reference. Crucially, it normalizes these advantages using the mean and
standard deviation of the entire batch. This global normalization provides a more stable learning
signal across diverse prompts. The method also integrates a PPO-like clipped objective to prevent
destructive policy updates. This approach is more sample-efficient, robust to reward hacking, and
shows better generalization than intra-prompt methods. However, a global baseline can be less
sensitive to prompt-specific nuances, and its performance may be affected by batch composition.
Like other actor-only approaches, it may also exhibit higher variance than actor-critic methods,
though its normalization and clipping features effectively mitigate this.


Standard RLHF uses a single reward for an entire generated sequence, making it hard to assign
credit to specific token decisions, a major inefficiency, especially for long-form generation [Zhong
et al., 2025]. Another key problem in RLHF is “reward hacking”, where the policy exploits weaknesses in the learned reward model by generating OOD outputs that receive high (but untrustworthy) scores [Wu et al., 2024b]. To address these foundational issues, a new class of specialized
and hybrid methods has been developed. These approaches go beyond simply refining the policy optimizer and instead restructure core components of the RL problem itself: the optimization
objective, the reward signal, and the policy’s exploration space. In this subsection, we organize
recent advances into two principal categories, based on how they reformulate the RLHF problem:
(1) **Dense Reward Signal Methods** : Approaches that address the credit assignment and sample efficiency challenge by transforming the reward from sparse, sequence-level feedback to dense,
token-wise feedback. (2) **Exploration-Constrained Methods** : Approaches that mitigate reward
over-optimization and improve robustness by explicitly restricting the policy’s exploration to regions where the reward model is reliable. For each, we review the foundational method, analyze its
limitations, and describe follow-up innovations.
**Dense Reward Signal Methods.** **Reinforced Token Optimization (RTO)** [Zhong et al.,
2025] is the primary example of this approach. It is not a replacement for an optimizer like PPO
but rather a powerful enhancement that provides it with a much better learning signal. RTO is the
first major approach to recast RLHF as a token-level Markov Decision Process [Zhong et al., 2025].
It introduces a dense, per-token reward by leveraging the DPO (Direct Preference Optimization)
[Rafailov et al., 2024] objective. Here, each token in a sequence receives its own reward based on
how much more likely it is under the DPO-trained policy than a reference policy. This results in
a much richer, more actionable learning signal that substantially improves credit assignment and
sample efficiency. Experiments show that RTO can achieve superior or comparable performance to
PPO with less data [Zheng et al., 2023b].
Despite these strengths, this framework has its own limitations: (1) Dependence on DPOderived rewards: The framework’s success is contingent on the policy learned by DPO ( _πDPO_ )
being a good proxy for the true optimal policy. A poorly trained DPO model will result in a noisy
or misaligned token-wise reward signal. While making rewards dense, ensuring these token-level
signals are consistently meaningful remains a broader challenge. (2) Pipeline Complexity: The
RTO pipeline is inherently multi-stage, requiring a DPO training run to generate the reward signal
before the PPO optimization phase can begin, which adds operational complexity [Zhong et al.,
2025].
**Exploration-Constrained Methods.** Exploration-constrained methods focus on keeping the
policy within the “trusted” region of the reward model, thereby reducing the risk of such exploitation. **Behavior-Supported Policy Optimization (BSPO)** [Dai et al., 2025a] is a principled
approach that constrains policy optimization to the region where the reward model is known to be
reliable. BSPO defines a “behavior policy” ( _β_ ) using the next-token distribution from the RM’s
training dataset, which delineates an in-distribution (ID) region for the RM. It then employs a
“behavior-supported Bellman operator” that regularizes the value function by assigning a low Qvalue ( _Qmin_ ) to actions _a_ at state _s_ leading out of this ID region (OOD actions, where the behavior
policy assigns near-zero probability, i.e., _β_ ( _a|s_ ) _≈_ 0), while leaving ID action-values unchanged
(i.e., _β_ ( _a|s_ ) _>_ 0). By doing so, BSPO directly addresses reward hacking and ensures that policy
improvement happens only where the reward model’s judgments can be trusted.
The primary advantages offered by this behavior-supported approach are: (1) Directly Mitigates Reward Hacking: By explicitly penalizing OOD actions at the value-function level, BSPO
prevents the policy from exploring and exploiting regions where the reward model is known to
be unreliable. (2) Finds Optimal In-Distribution Policy: This method penalizes only OOD values
without altering the values of ID actions. This allows it to fully explore the supported region and
find the optimal policy within that space, whereas other methods like a uniform KL penalty can
be overly conservative [Dai et al., 2025a]. However, the framework has notable limitations: (1)
Dependence on Behavior Policy Definition: The method’s effectiveness is highly dependent on how
well the defined “behavior policy” represents the true region of the RM’s competence. A poorly
defined ID region could either unduly restrict beneficial exploration or fail to prevent hacking. (2)
Primary Validation in Synthetic Setups: This method’s performance has been validated primarily
in synthetic setups with a “gold” reward model. The dynamics of OOD detection may differ in
real-world scenarios with noisy and inconsistent human feedback. (3) Response Scope: The method
is designed to handle OOD responses but does not address the separate challenge of the model receiving an OOD prompt, for which no valid ID response may exist.
In summary, the optimization of an LLM’s policy via a learned reward model has spurred
innovation along three parallel streams. The first involves refining the established and industrystandard **Actor-Critic PPO framework** to enhance its performance across stability, efficiency,
and scalability while reducing its significant computational footprint. A second stream focuses
on developing simpler, **Actor-Only Alternatives** that streamline the RLHF pipeline and reduce
computational overhead by replacing the complex critic with engineered baselines for variance
reduction. The third stream moves beyond pure optimization to fundamentally restructure the RL
problem itself, with **Specialized and Hybrid Methods** designed to overcome core challenges
like reward hacking and sparse feedback. A detailed comparison of the key methods emerging from
these streams, outlining their respective advantages and limitations, is provided in Table 4.
Table 4: Policy Optimization Methods in RLHF

Alignment and Safety in Large Language Models: Safety Mechanisms,
  Training Paradigms, and Emerging Challenges

5.3.3 Specialized and Hybrid Reward-Based Policy Optimization

Haoran Lu
Luyang Fang
Ruidong Zhang
Xinliang Li
Jiazhang Cai
Huimin Cheng
Lin Tang
Ziyu Liu
Zeliang Sun
Tao Wang
Yingchuan Zhang
Arif Hassan Zidan
Jinwen Xu
Jincheng Yu
Meizhi Yu
Hanqi Jiang
Xilin Gong
Weidi Luo
Bolun Sun
Yongkai Chen
Terry Ma
Shushan Wu
Yifan Zhou
Junhao Chen
Haotian Xiang
Jing Zhang
Afrar Jahin
Wei Ruan
Ke Deng
Yi Pan
Peilong Wang
Jiahui Li
Zhengliang Liu
Lu Zhang
Lin Zhao
Wei Liu
Dajiang Zhu
Xin Xing
Fei Dou
Wei Zhang
Chao Huang
Rongjie Liu
Mengrui Zhang
Yiwen Liu
Xiaoxiao Sun
Qin Lu
Zhen Xiang
Wenxuan Zhong
Tianming Liu
Ping Ma


**Actor-Critic PPO** Highly stable; performant; High memory/compute cost;
[Schulman et al., 2017] industry standard. sensitive to hyperparameters.
**KL-PPO** Prevents policy deviation and
[Ouyang et al., 2022] enhances stability via KL
penalty.

Category Method Key Advantages Key Limitations


[Shen and Zhang,
2024]
Mitigates reward hacking by
filtering data with unreliable
reward signals.
**RP-PPO** Combats performance decay
[Zheng et al., 2025] by giving more training
epochs to high-quality data.


[Santacroce et al.,
2023]
Reduces memory cost
significantly via LoRA-based
model sharing.
**DD-PPO** Improves scaling with
[Wijmans et al., 2020] decentralized updates for
faster large-scale training.
**P3O** Improves sample efficiency by
[Fakoor et al., 2019] mixing on-policy and
off-policy updates.
**PPG** Improves sample efficiency by
[Cobbe et al., 2020] separating policy and value
training phases.
Adds complexity of tuning
the KL coefficient.
Depends on RM’s ability to
score extreme samples
accurately.
Adds logical complexity and
hyperparameters for dynamic
adjustment.
Requires complex
system-level engineering.
Increases implementation
complexity.
Risks instability from
off-policy data distribution
shift.
More complex two-phase
training loop.


**HP3O** Improves sample efficiency by
[Liu et al., 2025b] reusing trajectories from a
replay buffer.
**ToPPO** Improves sample efficiency
[Gan et al., 2024] with theoretically justified
off-policy data use.
Performance depends on
buffer size; risks using stale
data.
Relies on novel theoretical
assumptions.
**Actor-Only** **REINFORCE** Foundational simplicity; no High gradient variance and
[Williams, 1992a] critic to train or manage. sample inefficiency.


[Kool et al., 2019,
Ahmadian et al., 2024]
Reduces variance with an
adaptive, prompt-specific
baseline from peer responses.
High sample generation cost
per prompt; variance still
higher than PPO.
Effectiveness is entirely
dependent on the quality of
the greedy baseline.
High sample generation cost
per prompt; can overfit or be
biased by group stats.
**ReMax** Computationally cheap
[Li et al., 2024f] baseline using a single greedy
output; very sample efficient.
**GRPO** Robust to prompt difficulty
[Shao et al., 2024a] via reward normalization;
effective for reasoning.
**Dr. GRPO** Targets and mitigates specific Improvements may be
[Liu et al., 2025c] GRPO biases. task-specific.
**DISCO** Improves GRPO performance Adds complexity; requires
[Zhou et al., 2025b] on multi-domain datasets. domain heuristics or labels.
**REINFORCE++** Improves generalization and
[Hu et al., 2025] OOD performance with a
global, batch-wise baseline.
**Specialized** **RTO** Solves the credit assignment
[Zhong et al., 2025] problem with dense,
per-token rewards.
**BSPO** Mitigates reward hacking by
[Dai et al., 2025a] constraining the policy to a
trusted data region.


Global baseline is less
responsive to individual
prompt difficulty.
Quality is contingent on the
DPO-derived rewards;
multi-stage pipeline.
Effectiveness depends on a
well-defined behavior policy;
may stifle useful exploration.
RLHF has transformed language model alignment by enabling models to optimize directly for
human preferences rather than just next-token prediction. The fine-tuning of GPT-3 into InstructGPT, for instance, showed that a 1.3B parameter model trained with RLHF could outperform the
much larger 175B parameter base model on human preference ratings [Ouyang et al., 2022]. However, despite dramatic progress, RLHF also exposes some challenges spanning feedback collection,
reward modeling, and policy optimization. Addressing these is essential for advancing robust and
scalable alignment.
(1) Human feedback data bottlenecks. RLHF pipelines depend on human feedback, which
is costly, slow to scale, and prone to noise, cognitive biases, and low inter-annotator agreement
[Stiennon et al., 2022, Ziegler et al., 2020b, Bai et al., 2022c]. This challenge is magnified by the
problem of value pluralism; human preferences are not monolithic, and the attempt to aggregate
diverse or even conflicting values into a single reward function can lead to a model aligned with a
statistical “average” that satisfies no one, a limitation with theoretical roots in paradoxes of social
choice [Liu et al., 2025a].
(2) Reward Model Limitations and Vulnerabilities. The RM, trained on limited and noisy human
feedback, serves only as an imperfect proxy for true human intent. As we discussed previously,
human annotations can be inconsistent, noisy, and subject to individual biases. Moreover, the
data collected often covers only a subset of possible scenarios, prompts, and preferences, and may
not fully reflect the diversity or subtlety of real human values. As a result, the RM tends to
learn patterns that fit the observed feedback rather than capturing the underlying intent behind
human judgments. This data limitation, combined with the complexity of modeling nuanced human
preferences, makes the RM susceptible to reward misspecification, where its assigned scores do not
accurately correspond to what users actually prefer, but instead reflect artifacts or gaps in the
training data [Peng et al., 2023b, Bobu et al., 2024]. Misspecified rewards enable **reward hacking**,
where policies generate outputs that maximize RM scores without genuine alignment [Gao et al.,
2023b, Laidlaw et al., 2025]. Common manifestations include verbose responses exploiting length
biases and sycophantic outputs exploiting agreement preferences [Shen et al., 2023, Liu et al.,
2025d].
(3) Policy Optimization Complexity and Instability. State-of-the-art optimization methods like
PPO are resource-intensive and sensitive to hyperparameters [Ouyang et al., 2022, Ramamurthy
et al., 2023]. In addition, the optimizer is tasked with solving a difficult credit assignment problem,
as a single, sparse reward for an entire sequence provides an inefficient signal for determining which
specific token choices were beneficial [Zhong et al., 2025]. When this complex optimization process
is aimed at maximizing a brittle and misspecified reward signal, the entire pipeline becomes unstable and prone to producing misaligned behaviors.


Having comprehensively introduced the foundation of SFT and RLHF, we now proceed to a more
in-depth comparison between the two alignment strategies from multiple perspectives. SFT and
RLHF differ significantly in their objectives, data requirements, and optimization paradigms, resulting from the inherent methodology and theory. Importantly, these approaches are not mutually
exclusive. On the contrary, they are often complementary, and when effectively integrated, can
yield more robust and nuanced model alignment. In this section, we systematically examine the
differences between SFT and RLHF and explore how their strengths can be combined within hybrid
training pipelines.

6 SFT versus RLHF: Differences, Equivalences, and Hybrid Ap- proaches


At a fundamental level, SFT involves fine-tuning a pre-trained model on labeled datasets [Wang
et al., 2022a, Taori et al., 2023, Peng et al., 2023a], whereas RLHF incorporates human feedback
into a reward function to guide learning in a reinforcement learning framework [Ouyang et al., 2022,
Bai et al., 2022b]. These underlying principles give rise to different objectives, data requirements,
and learning dynamics.
_**Objectives and reward function.**_ SFT optimizes a standard supervised learning objective,
typically minimizing the token-level cross-entropy loss between the model’s next output token and
human-labeled reference responses [Brown et al., 2020]. This objective assumes the existence of a
single correct output per input and encourages the model to mimic these ground truth answers.
Formally, given an input _x_ and reference output (labeled data) _y_ = ( _y_ 1 _, y_ 2 _, . . ., yT_ ), the loss function
[Fan et al., 2025b, Mao et al., 2023] as shown in section 4.3 is _L_ SFT = _−_ [�] _[T]_ _t_ =1 [log] _[ P][θ]_ [(] _[y][t][ |][ x, y][<t]_ [)] _[,]_
where _Pθ_ ( _yt | x, y<t_ ) is the probability of generating correct token _yt_ given the input and previous
tokens _y<t_ . With the estimated model parameters _θ_ [�] = argmin _θL_ SFT, the output of the finetuned model should be close to the reference one. In contrast, RLHF exploits a reinforcement
learning framework. The model acts as a policy _πθ_ that generates responses to maximize a scalar
reward signal. This reward is not derived from a correct reference, but rather from a learned
reward model trained to reflect human preferences. Given a sampled response _y ∼_ _πθ_ ( _· | x_ ),
the reward model assigns a scalar score _r_ ( _x, y_ ). Then we could use RL (e.g., Proximal Policy
Optimization [Schulman et al., 2017]) to directly optimize the expected reward as [Ziegler et al.,
2020c]: E _π_ ( _r_ ) = E _x∼D,y∼πθ_ ( _·|x_ )[ _r_ ( _x, y_ )]) _,_ which allows the model to explore a flexible solution space
and optimize for outputs that align with nuanced human expectations, especially in ambiguous or
multi-faceted scenarios.
_**Data requirements.**_ The distinct optimization targets of SFT and RLHF lead to different
data requirements. SFT relies on high quality instruction–response pairs that provide explicit
supervision for the model to mimic reference answers [Dong et al., 2023, Chowdhery et al., 2023],
and the powerful effect of data quality on LLM performance that can surpass the amount of data
[Zhou et al., 2023]. In contrast, RLHF depends on preference-based data. Annotators are presented
with multiple responses for the same prompt and are asked to rank or select the preferred one. These
comparative judgments are used to train a reward model that approximates human preferences [Bai
et al., 2022b, Liu et al., 2020]. This reward model guides the language model during reinforcement
learning. Thus, SFT requires extensive, high quality labeled data, while RLHF necessitates laborintensive human preference collection. Each method poses distinct challenges in data acquisition
that must be addressed to ensure effective model alignment [Yin et al., 2024b, Lee et al., 2023].
_**Learning dynamics and generation.**_ SFT is a static, one shot learning process. It aligns
the model by providing token level supervision based on reference answers, which may introduce
biases in preference estimation [Hua et al., 2024]. Once fine-tuned, SFT models lack adaptability
to evolving objectives unless retrained on newly curated datasets. Furthermore, for tasks involving
long-form generation, evaluating individual tokens may be suboptimal compared to assessing full
responses. In contrast, RLHF emphasizes sentence level performance. It samples entire responses
and aligns model behavior with human preferences over those responses. RLHF typically proceeds in
two stages: reward model training and reinforcement fine-tuning [Bai et al., 2022b, OpenAI Achiam
et al., 2023]. In each RL iteration, the model samples outputs according to its current policy, receives
feedback via the reward model, and updates its policy to increase the likelihood of generating
preferred outputs [Zheng et al., 2023b]. Online RLHF settings [Xiong et al., 2024, Dong et al., 2024,
Ye et al., 2024a] may further refine the reward model and iteratively update the policy, allowing
continual alignment improvements through repeated sampling, feedback, and optimization. Despite
its flexibility, RLHF can be unstable and computationally demanding, especially in online learning
schemes requiring coordination among multiple components [Rafailov et al., 2023, Ouyang et al.,
2022, Yuan et al., 2023, Ethayarajh et al., 2024]. SFT provides training stability and efficiency
due to its straightforward optimization, often resulting in faster convergence and lower resource
requirements [Du et al., 2025].

6.1 Fundamental Differences between SFT and RLHF


While SFT and RLHF differ operationally [Ouyang et al., 2022], they exhibit surprising methodological overlaps [Chen et al., 2024a, Swamy et al., 2025]. These overlaps manifest most prominently
in their theoretical equivalence under shared representational constraints, loss function alignment
through regularization mechanisms, and convergence to identical optimal policies under idealized
conditions [Williams, 1992b]. Such synergies challenge the conventional separation of imitation
learning and preference-based alignment, revealing a unified landscape for optimizing model behavior.
_**Overlap for theoretical equivalence.**_ Under the assumption of isomorphic function classes,
where policy networks and reward models share equivalent representational capacity, both SFT
and RLHF converge to the same optimal policy [Chen et al., 2024a]. Despite employing distinct
algorithmic strategies, these approaches exhibit theoretical equivalence in their alignment objectives. While SFT maximizes likelihood over human labeled data and RLHF optimizes expected
reward under a learned preference model, their solutions coincide when analyzed through a shared
reward-functional lens. This equivalence is further clarified through the lens of Direct Preference
Optimization (DPO), which reformulates RLHF as direct reward maximization, bridging the gap
between the two paradigms [Swamy et al., 2025].
_**Overlap for loss function alignment.**_ Under the assumption of linearly isomorphic function
classes, where policy and reward models share equivalent representational capacity, the gradient
updates of SFT and Supervised Policy Improvement via Reinforcement Learning (SPIN) through
online DPO can be shown to be formally aligned. In this regime, SFT optimizes the policy parameters _θ_ using the standard maximum likelihood gradient [Williams, 1992b], i.e., _g_ SFT( _πθ_ ) =
E _ξ∼D_ Sft �� _Hh_ =1 _[−∇][θ]_ [ log (] _[π][θ]_ [(] _[a][h][|][s][h]_ [))] �, while SPIN modifies this by introducing an on-policy correc
tion term, i.e., _g_ SPIN( _πθ_ ) = _g_ SFT( _πθ_ ) _−_ E _ξ∼πθ_ �� _Hh_ =1 _[−∇][θ]_ [ log] - _ππ_ ref _θ_ (( _aahh|s|shh_ )) �� _._ Crucially, this additional
term is equivalent to a reinforce-style update with constant reward _r_ ( _ξ_ ) = 1, and thus integrates to
zero in expectation. As a result, SPIN’s update rule asymptotically reduces to SFT when function
classes are linear and expressive enough to perfectly model the optimal policy. This equivalence
reveals a deep structural overlap between offline supervised learning (SFT) and online preferencebased reinforcement learning (SPIN/RLHF), unifying them within a shared loss geometry in the
linear regime [Rafailov et al., 2023].
_**Convergence conditions for RLHF and SFT.**_ The convergence between RLHF and SFT
performance hinges on what the authors term the disparity between the complexity of generating
outputs via policies and verifying them via reward models [Swamy et al., 2025]. This gap becomes
pivotal when training a verifier is significantly easier than training the generator [Rafailov et al.,
2023, Sun and van der Schaar, 2024]. In such settings, RLHF effectively restricts policy search to
regions that perform well under simplified reward models, while SFT directly optimizes the policy
through maximum likelihood estimation. In particular, two specific regimes lead to alignment in
the final performance of RLHF and SFT. First, in tasks with low sequence complexity, such as twoword summarization, the generation complexity closely matches verification complexity [Li et al.,
2010], minimizing the utility of RLHF’s reward modeling [Swamy et al., 2025]. Second, when reward
functions are simple and easily computed, the verification process is trivial [Lin, 2004], reducing
RLHF to a form of unnecessary refinement [Swamy et al., 2025].
Figure 3: Integration of SFT (Step 1) and RLHF (Step 2 and Step 3).

6.2 When SFT and RLHF Overlap or Converge


Given their distinct learning objectives and optimization paradigms, SFT and RLHF are suited
to different scenarios. SFT is typically preferred when the task is well-defined, static, and when
sufficient labeled data is available [Ghosh et al., 2024, Wang et al., 2023f]. On the other hand,
RLHF is more effective for aligning models with complex, subjective, or evolving user expectations,
using interactive human feedback [Winata et al., 2025, Ouyang et al., 2022]. While these methods
have distinct strengths, recent developments suggest that integrating SFT and RLHF into a unified
training pipeline can yield more robust and aligned language models [Ouyang et al., 2022, OpenAI Achiam et al., 2023, Bai et al., 2022b, Gemma et al., 2024, Touvron et al., 2023, Guo et al.,
2025]. The following sections discuss this integration from three perspectives: the opportunity
that enables the integration pipeline, its application in leading LLMs, and the new challenges it
introduced.
_**Opportunity.**_ Recent research suggests that SFT on a moderately sized, high-quality dataset
can already yield outputs comparable to those produced by human annotators [Touvron et al.,
2023]. Thus, A widely adopted integration strategy begins with supervised fine-tuning on highquality demonstrations to teach the model basic instruction-following behavior. After this stage,
the fine-tuned model is used to generate multiple candidate responses to various prompts. These
responses are then compared by human annotators to produce preference data, which serve as
training targets for a reward model. Finally, reinforcement learning is applied to further fine-tune
the model using this reward model, enabling it to produce outputs that better align with nuanced
human preferences (as shown in Figure 3).
_**Applications.**_ This hybrid approach is now common among leading language models. For
instance, InstructGPT [Ouyang et al., 2022] pioneered the two-stage pipeline, combining SFT
and RLHF. Similar methodologies have been adopted by OpenAI’s GPT series [OpenAI Achiam
et al., 2023], Anthropic’s Claude models [Bai et al., 2022b], Google’s Gemini [Gemini et al., 2023]
and Gemma series [Gemma et al., 2024], Meta’s LLaMA [Touvron et al., 2023], and DeepSeek
models. Notably, DeepSeek-R1-Zero is trained purely via RLHF, while DeepSeek-R1 adds an SFT
stage before RLHF. The latter achieves substantially better reasoning performance, illustrating the
benefits of combining the two strategies [Guo et al., 2025].
_**New Challenges.**_ Despite its growing popularity, the integration of SFT and RLHF intro
duces additional challenges. Their differing optimization goals, cross-entropy minimization in SFT
versus reward maximization in RLHF, which may lead to conflicting gradient updates and unstable
training dynamics [Hua et al., 2024]. Recent studies aim to mitigate the conflict and streamline the
training pipeline. For instance, Intuitive Fine-Tuning (IFT) integrates SFT and RLHF into a single
stage training process by introducing a temporal residual connection between supervision signals
and reward-based updates [Hua et al., 2024]. Similarly, Unified Fine-Tuning (UFT) reformulates
both objectives under a generalized implicit reward function, allowing simultaneous optimization
with a single loss function [Wang et al., 2024d]. These approaches represent promising directions
for balancing alignment, performance, and training efficiency in the integration of SFT and RLHF.

6.3 Integrating SFT and RLHF in Training Pipelines


In addition to traditional SFT and RLHF, this section explores several advanced alignment techniques and highlights their advantages over traditional methods. Reward-free methods remove the
need for manually designed reward signals, streamlining the training process and reducing reliance
on human input. AI assistant alignment uses a stronger, pre-aligned model to generate high-quality
guidance, which accelerates convergence and enhances safety. Self-alignment lets a model critique
and improve its own outputs, yielding more robust behavior without additional data collection.
Multi-agent deliberation strategies enable specialized agents to work together on complex tasks,
increasing reliability and overall performance. Multi-objective alignment balances competing goals,
including accuracy, fairness and computational efficiency, within a unified framework to produce
more versatile and trustworthy models. Combined, these approaches lower development cost, scale
more easily to large models, and offer more effective ways to align LLMs with human values.

7 Advanced Alignment Techniques and Recent Innovations


As the scale of LLMs increases, ensuring alignment with human intent becomes increasingly critical. Conventional approaches such as Reinforcement Learning from Human Feedback (RLHF)
[Christiano et al., 2017] have been widely adopted to bridge this gap. However, RLHF pipelines
are often resource-intensive and require an additional reward model trained to predict human preferences. This dependency introduces potential sources of error and makes the alignment process
less interpretable and harder to scale.
Recent innovations aim to simplify this process by eliminating the need for an explicit reward
model altogether. A prominent example of this shift is Direct Preference Optimization (DPO),
introduced by Rafailov et al. [2023]. DPO formulates alignment as a supervised contrastive learning problem, where the model is directly trained to prefer outputs labeled as better by human
annotators over those labeled as worse. Instead of first training a reward model and then applying
reinforcement learning (e.g., PPO), DPO uses a probabilistic objective derived from a MLE principle over pairwise preferences. This method significantly reduces computational complexity while
maintaining and in some cases improving the alignment quality compared to RLHF. Furthermore,
mathematically, DPO maximizes the likelihood ratio between a preferred and a dispreferred response, which implicitly shapes the model’s behavior to mirror human judgment. The simplicity of
the DPO objective also improves training stability and interpretability. Theoretically, DPO can be
interpreted as recovering the policy optimal under an unknown reward, bypassing reward regression
altogether.
In addition to DPO, other reward-free or preference-based methods have emerged that focus
solely on relative preferences rather than scalar rewards. For instance, pairwise ranking models
use binary comparisons between responses to fine-tune the model, circumventing the instability of
reward estimation [Wu et al., 2023]. These models operate under the insight that humans often
find it easier to choose between two options than to provide a numerical rating, making preference
data easier to collect and less noisy.
These innovations collectively represent a transition toward simpler, more scalable, and interpretable alignment techniques, offering strong empirical performance while reducing dependency
on reinforcement learning infrastructure. They enable alignment pipelines that are not only more
resource-efficient but also better suited for iterative deployment in real-world applications where
feedback is noisy, sparse, or difficult to quantify.
Overall, these reward-free alignment techniques significantly streamline the training pipeline by
eliminating the need for explicit reward modeling.

7.1 Direct Preference Optimization and Reward-Free Methods


The rapid progress of large language models has created an urgent need for alignment methods
that scale beyond intensive human supervision. AI-assistant alignment, together with its selfalignment variant, seeks to automate and streamline this process while safeguarding both ethical
considerations and practical performance.

7.2 AI-Assistant Alignment and Self-Alignment


Recent work on aligning LLMs has progressed from RLHF to approaches that minimize or eliminate
the need for costly human annotation. In the standard RLHF pipeline, an instruction-tuned model
is coupled with a reward model trained on human-ranked answer pairs and then fine-tuned with
a policy-gradient method that retains a Kullback–Leibler constraint to stay close to the original
distribution [Ziegler et al., 2019, Ouyang et al., 2022]. While effective, RLHF is limited by the
time and expense required to collect expert preferences. Reinforcement Learning from AI Feedback
(RLAIF) follows the same three stages (data collection, reward-model fitting, and reinforcement
learning) but replaces human comparisons with judgments produced by a stronger, already aligned
teacher model, enabling the generation of millions of preference pairs at negligible marginal cost.
Constitutional AI (CAI) introduced by Bai et al. [2022a] is the most influential example of RLAIF.
CAI begins with a short, human-written constitution that encodes safety and helpfulness rules.
A helpful model, already aligned through RLHF, is first supervised to critique and revise its own
answers according to these principles, producing what the authors call the supervised constitutional
model. In the next phase, this same model generates two candidate answers for each prompt
and, when prompted with the constitution, decides which answer better follows the rules, thereby
creating AI-labeled preference pairs without human raters. A separate and smaller reward model
is trained on these pairs, combined with earlier helpfulness comparisons, and then frozen. Finally,
the policy is fine-tuned to maximize the frozen reward while limiting divergence from its supervised
starting point. This process removes per-example human labeling yet yields assistants that are
rated safer and less evasive than those produced by standard RLHF, demonstrating the practical
promise of RLAIF. The workflow is provided in Figure 4.
A growing body of work expands RLAIF beyond the original Constitutional AI recipe. Lee et al.
[2023] presents the first systematic head-to-head comparison of human- and AI-sourced preferences,
finding that AI feedback matches human feedback when the teacher model is sufficiently capable.
Cui et al. [2023] scale this idea, constructing a million-example GPT-4 preference set and showing
that policies trained only on this data rival strong RLHF baselines. To balance multiple objectives,
Li et al. [2024g] mixes helpfulness and harmlessness signals in a single reward, while Li et al. [2025]
introduces a difficulty-ordered curriculum that improves generalization in a fixed label budget.
Figure 4: Workflow of the Constitutional AI (CAI) Framework. A helpful LLM first generates
responses to harmfulness prompts, which are revised based on a predefined constitution encoding principles like harmlessness, ethics, and virtue. The revised responses are used to fine-tune
the model via supervised learning, producing an initial SL-CAI model. Preference data is then
generated by comparing model outputs under the constitution, and a preference model is trained
accordingly. Finally, reinforcement learning is applied using the preference model to further align
the SL-CAI model, yielding a more helpful and harmless LLM.
Sharma et al. [2024] audit these pipelines and highlight systematic divergences between AI- and
human-generated labels. RLAIF is also moving beyond text: Ahn et al. [2024] adapts the framework
to video understanding, and Jing and Du [2024] applies fine-grained AI feedback to vision–language
models, reducing object-level hallucinations.


Several studies show that a model can supply its own feedback without an external teacher. Bao
et al. [2024] demonstrates that a 13-billion-parameter chat model can critique and rank its own
answers under a simple rubric, yielding performance comparable to teacher-based RLAIF. Similarly,
Yu et al. [2024b] reports that generating a brief self-critique before reward-model fitting boosts
alignment quality and cuts the need for external labels by 80 percent. These self-alignment results
suggest that, once a model passes a competence threshold, it can bootstrap further alignment
with minimal additional supervision, although questions remain about error compounding and bias
reinforcement.


The emerging RLAIF family still faces open problems in bias control, robustness, and evaluation.
Because AI-generated labels inherit the value structure of the teacher, systematic divergences from
human judgment can persist or even amplify, as quantified by the large-scale audit of Sharma et al.
[2024]. Head-to-head studies [Lee et al., 2023] argue that teacher quality largely determines final
alignment, so future work must develop teacher-agnostic debiasing techniques or lightweight human
spot-checks to prevent value lock-in. Robustness to red-team attacks remains another priority: Bai
et al. [2022a] proposes iterated “online” training, continuously refreshing the preference model with
new AI feedback from the policy’s own failure modes, while Li et al. [2025] shows that curriculumstyle data ordering can harden models without extra labels. Self-alignment methods that recycle
a model’s own critiques [Bao et al., 2024, Yu et al., 2024b] promise unlimited scalability, yet they
raise questions about error compounding and whether periodic human “re-grounding” is required.
Moving beyond text, multimodal extensions already surface new challenges: video-based RLAIF
must judge temporal coherence [Ahn et al., 2024], and vision–language alignment needs objectlevel feedback to curb hallucinations [Jing and Du, 2024]. Finally, static benchmarks saturate
quickly; several papers call for adaptive, adversarial evaluation suites that track constitutional
drift and reward hacking over multiple bootstrapping generations. Addressing these challenges will
likely require hybrid pipelines that blend a small amount of strategically targeted human input
with scalable AI feedback, uncertainty-aware reward ensembles, and red-team-in-the-loop training
protocols.


Deliberative Alignment is an approach to make language models safer and more dependable. Instead
of just learning from examples, it’s about directly teaching the model the actual safety rules and
then training it to consciously think through these rules before it gives an answer [Guan et al.,
2024]. The idea is to make sure the model sticks closely to safety guidelines. This method helps
models get better at spotting and handling tricky situations, including attempts to make them say
things they shouldn’t. It also means they’re less likely to refuse perfectly normal requests and can
handle new or unusual scenarios more effectively [Konya et al., 2023].
The process basically involves a couple of main stages. First, the model is shown a lot of examples where it sees how to reason through safety rules to arrive at a good answer, this is like a
focused study period. Then, it goes through a kind of practice phase using reinforcement learning,
where it gets feedback to sharpen its decision-making skills, especially when dealing with prompts
that touch on safety concerns. The safety rules themselves delineate the parameters of acceptable
and unacceptable content, and stipulate the model’s appropriate responses in diverse scenarios,
including instances requiring the declination of a request or the provision of a meticulously formulated ’safe’ answer. By integrating this rule-based reasoning directly into the model, Deliberative
Alignment endeavors to establish a more transparent, trustworthy, and scalable methodology for
ensuring responsible language model behavior [Fang et al., 2025a].
Self-consistency is another crucial method applied to LLMs for addressing prevalent issues such
as deficient reasoning and the generation of hallucinations [Wang et al., 2022a]. Referencing the
survey by Liang et al., we can understand this through the broader lens of “internal consistency”
which pertains to the uniformity of an LLM’s expressions across its latent, decoding, or response
layers when subjected to sampling methodologies [Liang et al., 2024]. Numerous studies prefixed
with “Self-” including prominent examples like Self-Consistency [Li et al., 2024h], Self-Improve [Patel et al., 2024], and Self-Refine [Ranaldi and Freitas, 2024], have emerged to tackle these challenges.
These approaches, while sometimes distinct in their specific mechanisms, all fundamentally involve
LLMs in a process of evaluating and subsequently updating their own outputs or internal states.
Building upon the foundational concept of Self-Consistency, which primarily leverages majority
voting over multiple generated outputs, several nuanced variations have been developed to refine the
selection of the optimal response. Among these, Multi-Perspective Self-Consistency [Huang et al.,
2023b]distinguishes itself by incorporating assessments from diverse criteria or viewpoints when
evaluating generated candidates, moving beyond simple congruence of final answers. Universal
Self-Consistency [Chen et al., 2023a] introduces a further layer of sophistication, often employing
a language model to ascertain the semantic equivalence of varied expressions before a consensus
mechanism is applied, thereby accommodating greater diversity in response phrasing. In a different
vein, Soft Self-Consistency [Wang et al., 2024e] shifts the focus from discrete answer selection to a
more probabilistic approach, typically by weighting different reasoning paths or outputs based on
the model’s internal confidence scores or token probabilities accumulated throughout the generation
process. Each of these adaptations thus offers a distinct strategy for aggregating or filtering multiple
reasoning instances, aiming to enhance the robustness and accuracy of the final output under various
task constraints and response complexities.
These “Self-” prefixed methods largely fall under a unified theoretical framework termed “SelfFeedback” [Liang et al., 2024]. This framework elegantly breaks down the process into two core
modules: “Self-Evaluation” and “Self-Update.” During Self-Evaluation, the LLM assesses its own
generated content or internal processes to capture signals related to internal consistency. These
signals can be scalar (like a confidence score), textual (like a critique), or even contrastive. Subsequently, the “Self-Update” module leverages these captured signals to enhance either the model’s
immediate response or, in some cases, the model’s parameters themselves. While each specific
“Self-” method might have slight variations in how it implements these two modules, they share
this fundamental cyclical process of introspection and refinement.
The primary characteristic of this Self-Feedback approach is its reliance on the LLM’s inherent
capabilities to introspect and improve, aiming to bolster internal consistency [Prasad et al., 2024].
The overarching purpose is to mitigate reasoning errors and reduce hallucinations by ensuring the
model’s expressions are more coherent and stable across different layers and sampling instances.
This makes such methods applicable to a wide array of scenarios, notably in enhancing the reliability
of LLMs for complex reasoning tasks (often seen in question-answering) and improving the factual
accuracy and faithfulness of outputs in open-ended generation tasks.

7.3 Multi-Agent and Deliberative Alignment Approaches


Group Relative Policy Optimization (GRPO) is a recently proposed reinforcement learning framework that addresses key challenges in aligning large language models (LLM) [Shao et al., 2024b].
GRPO was developed to better accommodate preference-based feedback and comparison-driven
reward modeling, with the aim of improving both training efficiency and the stability of learning
signals.
Traditional reinforcement learning methods, such as probal policy optimization (PPO), often
struggle in aligning LLM due to several factors. PPO relies on a critic network to estimate pertoken values, which nearly doubles the memory and computational requirements. Furthermore, in
most LLM alignment settings, the rewards are extremely sparse - usually available only at the end
of a generated sequence - and the reward models are commonly trained by pairwise comparison of
entire responses. This mismatch leads to unstable advantage estimation and inefficient learning.
GRPO addresses these limitations by eliminating the critic network and directly using groupbased reward normalization. For each prompt, GRPO samples a group of _G_ candidate outputs, each
evaluated by a reward model. The advantage of each output is calculated as the difference between
its reward and the mean reward of the group, aligning the learning signal with the structure
of comparison-based reward modeling. The formal training objective of GRPO, as presented in
DeepSeekMath [Shao et al., 2024b], is given by:
_|oi|_
_J_ GRPO( _θ_ ) = E _q∼P_ ( _Q_ ) _, {oi}Gi_ =1 _[∼][π][θ]_ old [(] _[O][|][q]_ [)] _G_
_G_
_i_ =1
_i_ =1 _t_ =1
- -
_A_ - _i,t_
_|oi|_
_−_ _β_ DKL[ _πθ∥π_ ref ]
min
_πθ_ ( _oi,t|q, oi,<t_ )
_A_      - _i,t,_ clip
_πθ_ old( _oi,t|q, oi,<t_ )
_πθ_ ( _oi,t|q, oi,<t_ )
_πθ_ old( _oi,t|q, oi,<t_ ) _[,]_ [ 1] _[ −]_ _[ϵ,]_ [ 1 +] _[ ϵ]_
(14)
The detailed GRPO objective and algorithm can be found in [Shao et al., 2024b]. GRPO
introduces two principal modifications over standard PPO: First, the KL divergence is separated
from the reward and added explicitly as a regularization term in the objective, rather than being
mixed into the reward signal. Second, the group-relative advantage _A_ [�] _i,t_ is computed differently;
specifically, it is standardized within the sampled group as
_A_           - _i,t_ = _[r][i][ −]_ [mean(] _[r]_ [1] _[, . . ., r][G]_ [)] _,_ (15)
std( _r_ 1 _, . . ., rG_ )
where _ri_ is the reward assigned to output _oi_, and std denotes the standard deviation across the
group.
This group-based design offers several alignment-relevant benefits. First, it structurally matches
the comparison-based reward models used for aligning human preferences, ensuring a more faithful
learning signal. Second, removing the critic reduces both the computational costs and the instability
that would otherwise arise when only final-sequence rewards are present. Group normalization also
provides a dynamic baseline, reducing variance in policy updates. Importantly, GRPO imposes
KL divergence as a distinct regularization term, rather than including it in the reward signal, thus
avoiding the bias introduced by KL-as-reward schemes.
Compared to PPO, GRPO introduces two principal differences. Firstly, it adopts a rationalstyle reward scaling mechanism that amplifies the policy update for candidates that significantly
outperform their group. Secondly, it enforces KL regularization as a separate penalty term, rather
than incorporating it into the reward, thus preventing interference in advantage estimation [Vojnovic and Yun, 2025]. These properties enable GRPO to leverage pairwise preference data more
effectively and achieve improved model alignment with human-evaluated outputs.
Although the original GRPO has demonstrated notable improvements in alignment and efficiency compared to PPO, ongoing research has produced a variety of GRPO variants aimed at
addressing specific limitations and further enhancing its effectiveness. These subsequent works fall
into several broad categories according to their main focus: (1) improving training stability and
optimization efficiency, (2) enriching and diversifying reward signals, (3) advancing core algorithmic
modifications, and (4) extending GRPO-based RL to new tasks and modalities. Table 5 provides a
comparative summary of these methods. In the following, we briefly review representative advances
within each category.
**Training Stability and Optimization Efficiency.** Several approaches have focused on
accelerating and stabilizing GRPO-based RL. CPPO [Lin et al., 2025] mitigates the inefficiency
of group sampling in GRPO by pruning low-advantage completions and reallocating resources
to additional prompts, resulting in a speedup of up to 8.3 _×_ on GSM8K and 3.5 _×_ on MATH
benchmarks. DAPO [Yu et al., 2025] introduces decoupled policy clipping and dynamic sampling
strategies, enabling robust and reproducible large-scale RL training for LLMs; it achieves state-ofthe-art open-source results on AIME 2024 with full code and data release. VAPO [YuYue et al.,
2025] reintroduces a value critic, with bias mitigation for long sequences and sparse rewards, leading
Table 5: Comparison of post-GRPO RL methods for LLM reasoning, by category.


GSM8K; MATH; 8.3× (GSM8K),
**CPPO** [Lin et al., 1.5B, 7B AMC 2023; AIME 2024 Yes 3.5× (MATH) Completion pruning
2025]
Training Stability
& Efficiency
**DAPO** [Yu et al., 32B AIME 2024 Yes 2× faster Clip decoupling & dynamic
2025] (Qwen2.5) sampling
**VAPO** [YuYue 32B AIME 2024 No Fast conv. Critic augmentation
et al., 2025] (Qwen2.5)
7B, 14B AIME 2024/25 Yes Faster conv. Length & difficulty shaping
Reward Signal
Enhancement
Algorithmic
Mods
Task &
Multimodal

Category Method Model Datasets OS Speed-up Core Contribution


**LEAD** [Wang
et al., 2025d]
GenEval;
**Flow-GRPO** [Liu SD 3.5 Text-in-image Yes Fewer steps ODE→SDE sampling
et al., 2025e]
8 V-L
**StepGRPO** [Zhang 7B–14B benchmarks Yes - Step-wise dense rewards
et al., 2025c]
GSM8K; AIME;
**S-GRPO** [Dai 7B–14B AMC; MATH; GPQA Yes 35–61% fewer tokens Decaying exit rewards
et al., 2025b]


**PO** [Chen et al.,
2025b]
7B, 14B, 32B 10 benchmarks N/A - All-negative diversification
AIME; AMC; MATH;
**Dr. GRPO** [Chen 7B Minerva; Olympiad Yes Token-efficient Unbiased advantage
et al., 2025c]


**GRPO** [Chen
et al., 2025d]
AIME; MATH;
7B, 14B GSM8K Yes - Entropy-weighted updates
to both higher accuracy and reliable convergence, surpassing previous GRPO-based methods on
challenging reasoning tasks.
**Reward Signal Enhancement and Diversity.** Another line of research seeks to overcome the
sparsity and homogeneity of reward signals in standard GRPO. GRPO-LEAD [Wang et al., 2025d]
introduces length-dependent rewards, explicit penalties for incorrect solutions, and difficulty-aware
weighting to encourage concise and robust mathematical reasoning. S-GRPO [Dai et al., 2025b]
proposes serial sampling and decaying exit rewards, incentivizing early correct answers and leading
to both shorter and more accurate solutions. Spectral Policy Optimization [Chen et al., 2025b]
addresses the issue of all-negative groups, where no sampled completion is correct, by injecting
AI-driven diversity into reward signals, breaking update symmetry and accelerating convergence.
**Algorithmic Modifications.** A third category directly revises the GRPO algorithm to address
optimization bias or incorporate uncertainty. Dr. GRPO [Chen et al., 2025c] removes length
and variance normalization from the advantage computation, eliminating a verbosity bias and
increasing token efficiency, while maintaining strong accuracy. SEED-GRPO [Chen et al., 2025d]
introduces semantic entropy as a measure of model uncertainty for each prompt, scaling policy
updates more conservatively for uncertain queries and more aggressively for confident ones, thus
improving generalization and stability across benchmarks.
**Task and Multimodal Generalization.** Recent works have also extended GRPO-style RL
beyond mathematical reasoning. Flow-GRPO [Liu et al., 2025e] adapts RL optimization for textto-image generation via flow matching models, employing ODE-to-SDE conversion and denoising reduction to improve compositionality and visual text rendering in diffusion models. StepGRPO [Zhang et al., 2025c] expands GRPO to multimodal reasoning with dense, step-wise feedback, enhancing multi-hop visual-language inference and outperforming imitation learning baselines
on a suite of benchmarks.
Together, these developments demonstrate the adaptability of the GRPO framework and the
breadth of innovations it has inspired. Each category reflects ongoing efforts to balance training
efficiency, reward informativeness, algorithmic robustness, and domain generalization, advancing
the state-of-the-art in RL-based alignment for LLMs.


Efficient Fine-Tuning methods that address the substantial computational and memory demands
associated with full model fine-tuning [Ding et al., 2023a, Xu et al., 2023b, Han et al., 2024b]. The
primary goals include significant reductions in computational costs, accelerated training speeds,
lower memory and storage requirements, and effective mitigation of catastrophic forgetting [Liu
et al., 2022, Fu et al., 2023b]. In this section, we discuss the Efficient Fine-Tuning methods for
LLM alignment including partial parameter fine-tuning, low-rank adaptation, sparse fine-tuning,
knowledge distillation, adapter-based fine-tuning, and prompt tuning.

8 Efficient Fine-Tuning Techniques for Alignment


Full-parameter fine-tuning involves updating all weights of a pre-trained Large Language Model
(LLM) on a task-specific dataset. While this approach typically achieves high task-specific performance, it is significantly resource-intensive, requiring substantial computational power and memory
capacity, especially as model sizes grow larger (e.g., billions of parameters) [Devlin et al., 2018,
Radford et al., 2019]. Additionally, full fine-tuning often risks overfitting, particularly with limited
training data, diminishing the model’s generalization capabilities [Howard and Ruder, 2018, Dodge
et al., 2020].
To mitigate the intensive resource requirements and risk of overfitting associated with full finetuning, partial parameter fine-tuning methods have emerged. These methods selectively update
a subset of parameters, significantly reducing computational cost while retaining competitive performance [Han et al., 2024b, Liu et al., 2022]. The parameters chosen for updating can vary,
including the final classifier layers, embedding layers, or specific layers within transformer blocks.
A prevalent example is layer-wise fine-tuning, which selectively tunes layers critical for task-specific
performance [Lee et al., 2019, Zhang et al., 2020]. This approach often reduces the memory footprint
and accelerates training by minimizing the number of gradient updates required.
Despite their practicality, partial parameter fine-tuning methods introduce additional complexity in deciding which parameters to tune, necessitating heuristic or algorithmic methods to
determine optimal subsets [Lee et al., 2019]. Moreover, partial fine-tuning can result in suboptimal
adaptation, particularly if crucial task-specific knowledge resides in layers that remain fixed [Han
et al., 2024b]. These limitations motivate further efficient fine-tuning approaches, such as low-rank
adaptation, sparse fine-tuning, and adapter-based methods, which systematically balance resource
efficiency and adaptability.

8.1 Full or Partial Parameters Fine-Tuning


Figure 5: Overview of LoRA in LLMs. LoRA introduces trainable low-dimensional weight matrices
that are integrated into frozen LLMs for fine-tuning. This PEFT approach enables LLMs to
adapt effectively to specific tasks. Additionally, various techniques are employed to enhance both
performance and efficiency, making the fine-tuning process practical and effective in real-world
applications.
LoRA introduces trainable low-rank matrices into each layer of the transformer architecture,
allowing for efficient adaptation of LLMs with a reduced number of trainable parameters. This
method significantly lowers the computational cost and memory footprint during fine-tuning, making it suitable for scenarios with limited resources. LoRA is adapted to practical challenges, and
we will present the main methods that can improve performance and efficiency, building on the
original LoRA framework introduced in [Hu et al., 2022].
The core idea of LoRA is to freeze the pre-trained weight matrices and inject trainable low-rank
matrices into dense layers. Instead of updating the full pre-trained weight matrix _W_ 0 _∈_ R _[m][×][n]_,
LoRA learns a low-rank incremental update ∆ _W_ = _BA_, where _B ∈_ R _[m][×][r]_ (initialized as 0),
_A ∈_ R _[r][×][n]_ (initialized as normal distributed random value), and the rank _r ≪_ _min_ ( _m, n_ ). The
forward computation of LoRA can be expressed as below:
_h_ = _W_ 0 _x_ + ∆ _Wx_ = _W_ 0 _x_ + _BAx_ (16)
LoRA can achieve comparable performance on several downstream tasks, but there’s still a
performance gap between LoRA and full fine-tuning in areas such as mathematical reasoning and
coding [Mao et al., 2024a]. To bridge the gap, existing methods mainly focus on four perspectives: 1) Increasing the rank _r_ appropriately[Lialin et al., 2023, Xia et al., 2024a]; 2) Adaptively
allocating ranks to LoRA modules across different layers [Zhang et al., 2023b, Mao et al., 2024b,
Ding et al., 2023b]; 3) Optimizing the training process, including initialization improvement and
gradient update optimization [Hayou et al., 2024a,b]; 4) Combining with other paradigms, such as
Bayesian learning[Yang et al., 2024a].
Accumulated linear computations introduced by LoRA modules in LLMs can still result in a
non-negligible computation burden. To alleviate this, three main strategies have been proposed to
make LoRA lighter and faster. 1) Parameter reduction. This can be achieved through parameter
freezing [Wu et al., 2024c], module pruning [Zhou et al., 2024b] or parameter sharing [Kopiczko
et al., 2024]; 2) Parameter quantization. By reducing the bit width of parameter, quantizationbased method can significantlt lower memory usage and computational cost [Dettmers et al., 2023,
Li et al., 2023d]; 3) Parallel in training and inference. This method leverages hardware related
algorithm to accelerate the computation both in training and inference process[Ye et al., 2024b,
Chen et al., 2023b].


Figure 6: Two paths to sparsity: **(left)** _data-efficient_ fine-tuning filters the corpus before training,
and **(right)** _parameter-efficient_ fine-tuning updates only a sparse mask of model weights.
We categorized sparse fine-tuning based on the aspects of the sparsity: (1) data-efficient finetuning that focuses on data-level sparsity by using informative subsets of data; and (2) selective
parameter-efficient fine-tuning that focuses on parameter-level sparsity by updating only a critical
subset of model weights.
Data-efficient fine-tuning aims to extract representative data points for substantially reducing
the computational cost when a large scale of data is available for alignment [Zhou et al., 2023, Wang
et al., 2023g]. Therefore, the closely related research fields to this data-efficient fine-tuning include
optimal subsampling [Ma et al., 2015, Wang et al., 2018, Ma et al., 2022], few-shot learning [Wang
et al., 2020, Liu et al., 2022], and coreset selection [Dasgupta et al., 2009, Albalak et al., 2024].
Existing works of data-efficient fine-tuning can be classified into two categories: (1) non-informative
sampling that selects samples based on predefined metrics on the data points; and (2) informative
sampling that aims to minimize the empirical risk. The non-informative requires less knowledge
about the downstream tasks and typically requires fewer computational resources for selecting
samples. [Gao et al., 2020] shows that the LLM can easily adapt to new tasks when fine-tuned on a
small dataset drawn with uniform random sampling. [Bukharin and Zhao, 2023] proposes QualityDiversity Instruction Tuning (QDIT) to simultaneously control the sampled dataset diversity and
quality. In contrast, informative sampling incorporates the model’s knowledge to calculate the
influence of each data point and require more computational resources, e.g., analyzing the scale
of gradients. [Xia et al., 2024b] proposes an optimizer-aware and practically efficient algorithm,
Low-rank Gradient Similarity Search(LESS), to estimate data influences and perform instruction
data selection for targeted instruction tuning in LLM.
Selective parameter-efficient fine-tuning leverages _sparsity_ by updating only a small fraction of
a model’s parameters, thereby significantly cutting down the computational and memory costs of
adapting large pre-trained models while maintaining near-original performance. Instead of tuning all weights, these methods identify a strategically chosen sparse subset of parameters to adjust
based on measures of importance or sensitivity. For instance, SIFT (Sparse Increment Fine-Tuning)
uses a gradient-based criterion: it exploits the observation that gradients in pre-trained models are
extremely sparse (e.g., about 1% of parameters account for 99% of the total gradient norm) [Song
et al., 2024]. SIFT therefore fine-tunes only the top- _x_ % of parameters with the largest gradient
magnitudes, restricting updates to the most influential weights for the task. In contrast, PaFi
adopts a task-agnostic approach by simply selecting the pre-trained weights with the smallest absolute magnitudes as the trainable subset, on the intuition that these low-magnitude parameters
can be altered with minimal disruption to the model’s prior knowledge [Liao et al., 2023]. Meanwhile, FishMask, a Fisher information-based masking method, computes an approximate Fisher
information score for each parameter to gauge its importance, and then updates only the top- _k_
parameters deemed most critical for the target task [Sung et al., 2021]. The importance score is an
approximation of the Fisher information matrix:
_F_ - _θ_ = [1]
_N_
_N_
E _y∼pθ_ ( _y|xi_ ) ( _∇θ_ log _pθ_ ( _y | xi_ )) [2] _,_ (17)
_i_ =1
which is the average of the square gradient of _y_ with respect to a given parameter _θ_ . By confining
fine-tuning to these important parameters (whether identified by gradient, weight magnitude, or
Fisher information), such selective fine-tuning techniques drastically reduce training overhead and
still achieve performance close to full model fine-tuning.


Knowledge distillation (KD) compresses LLMs by transferring knowledge from a high-capacity
teacher to a smaller student [Xu et al., 2024b, Fang et al., 2025b]. Unlike direct SFT or RLHF
pipelines, KD transfers alignment properties from a pre-aligned teacher model (typically optimized
via RLHF or similar methods) to a student model by mimicking the teacher’s performance, without
repeating the costly preference-optimization loop.
The concept of KD is introduced in the doundational work of Hinton et al. [2015]: instead
of training on hard labels _y_, the student learns from the teacher’s class probability distribution
_**p**_ _T_ = _σ_ - **z** _T /τ_ �, where **z** _T_ are logits from teacher, _σ_ is the softmax function, and _τ_ is a temperature
parameter that softens the distribution. The student’s objective function is typically a weighted
average of two losses: a standard cross-entropy loss _L_ CE with the hard labels, and a distillation loss
that minimizes the Kullback-Leibler (KL) divergence between the student’s and teacher’s softened
outputs:
_L_ KD = _α · L_ CE� _σ_ ( **z** _S_ ( _**x**_ )) _, y_    - + (1 _−_ _α_ ) _· τ_ [2] _· L_ KL� _σ_ ( _**z**_ _T_ ( _**x**_ ) _/τ_ ) _, σ_ ( _**z**_ _S_ ( _**x**_ ) _/τ_ )� _,_ (18)
where _α_ balances the two terms. Matching the full probability vector rather than one-hot targets
forces the student to copy subtle preference signals, including refusal styles, politeness markers,
and content filters, that are otherwise hard to encode.
One primary advantage of KD is its efficiency. Compressing a 70-billion-parameter RLHF
teacher to a 7-billion-parameter student cuts inference cost by roughly an order of magnitude while
Figure 7: Overview of Knowledge Distillation in LLMs. Knowledge is distilled from a teacher
LLM, which is typically optimized via RLHF. This knowledge, potentially enriched with current,
task-specific data, is transferred to a smaller student LLM. By learning from both the teacher’s
guidance and the current data, the student LLM becomes more efficient and effective at performing
downstream tasks.
preserving core reasoning ability and alignment quality [Taori et al., 2023, Touvron et al., 2023, Xu
et al., 2024b, Guo et al., 2025, Yang et al., 2024b, Ma et al., 2024, Wang et al., 2025e]. Furthermore,
the teacher is used only for forward passes, the same recipe scales easily to specialised domains:
clinicians and bioinformaticians have distilled general-purpose aligned teachers into compact models
fine-tuned for medical or scientific tasks without eroding their safety guarantees [Niu et al., 2024,
Tariq et al., 2024, Ge et al., 2025, Latif et al., 2024, Shang et al., 2024].
Subsequent work shows that transferring richer signals strengthens alignment further. Featurebased KD [Ji et al., 2021]matches hidden activations so the student inherits syntactic and semantic
structure. Attention distillation [Jiao et al., 2019] copies attention maps, helping small transformers
remain stable during fine-tuning. Relation-based KD [Yang et al., 2022] preserves similarity patterns inside the network and improves dense prediction tasks. Sequence-level KD trains the student
to reproduce complete outputs, capturing dialogue coherence and style [Taori et al., 2023, Li et al.,
2021]. Relation-based KD distills the reasoning patterns of teachers, such as the Chain-of-Thought
(CoT) reasoning, from teacher LLMs so the student learns intermediate reasoning steps rather than
shortcutting to the final answer [Hsieh et al., 2023, Feng et al., 2024]. For example, Hsieh et al.
[2023] generates the teacher rationales, and then the student is trained to jointly predict both the
rationale and the final answers, helping the student learns intermediate reasoning steps rather than
shortcutting to the final answer. Formally, the student is trained on a loss:
_L_ = _−_ [1]
_n_
_n_
log _Pθ_ ( _**r**_ _i, yi |_ _**x**_ _i_ ) _,_ (19)
_i_ =1
where ( _**r**_ _i, yi,_ _**x**_ _i_ ) comprises rationale, input, answer, and _Pθ_ denotes the student model parameterized by _θ_ .
Several extensions refine KD for alignment objectives. Multi-teacher distillation aggregates
outputs from diverse, pre-aligned models (e.g., RLHF-specialized variants), enabling students to
synthesize compounded behavioral priors [Khanuja et al., 2021, Zhang et al., 2022, Liu et al., 2024c,
Wadhwa et al., 2025]. Dynamic adaptive distillation further introduces a paradigm of bidirectional
co-evolution, where teacher and student models undergo simultaneous joint optimization for continuous mutual refinement [Sun et al., 2021, Chang et al., 2022, Li et al., 2024i]. Self-distillation
represents a distinct approach that bypasses external teachers entirely, leveraging a single model’s
self-generated outputs for iterative refinement[Zhang et al., 2019, Yang et al., 2023].
One another key point is the uncertainty-aware abilities of the student model. Classic KD can
create over-confident students because it ignores uncertainty. Uncertainty-aware variants address
this in two complementary ways. The first approach focuses on distilling uncertainty by training
the student to mimic the teacher’s full predictive distribution over labels. These distributions are
typically learned from uncertainty-aware systems such as Bayesian neural networks or model ensembles, with the student minimizing divergence between its outputs and the teacher’s probabilistic
outputs [Korattikara Balan et al., 2015, Vadera et al., 2020, Malinin et al., 2019]. The second approach addresses quantification of the student’s intrinsic uncertainty by reinterpreting knowledge
distillation through a Bayesian framework. Bayesian Knowledge Distillation (BKD) [Fang et al.,
2024] achieves this by embedding the teacher’s output probabilities as a teacher-informed prior
distribution over the student model’s weights. Within this formulation, standard knowledge distillation loss emerges naturally as the posterior optimization objective. Sampling techniques, such
as stochastic gradient Langevin dynamics, applied to this posterior yield principled predictive intervals. Collectively, these uncertainty-aware methods suppress overconfident predictions on noisy
or out-of-distribution inputs while providing computationally efficient alternatives to full Bayesian
training. This characteristic makes them particularly valuable for developing compact while still
being safety-aligned LLMs.

8.4 Knowledge Distillation for Fine-Tuning


Adapter-based fine-tuning involves inserting small trainable modules, known as adapters, between
the layers of a pre-trained model. The original model weights remain frozen, and only the adapters
are trained on downstream tasks. This approach allows for efficient multi-task learning and rapid
adaptation to new tasks without retraining the entire model.
An adapter module commonly contains two linear layers to conduct down-projection that reduces the dimensionality of the input and up-projection that restores it. Between the two layers,
nonlinearity layer is applied to the down-projected representation. During the fine tuning process,
trainable parameters Φ are optimized on given dataset _D_, with specified loss function _L_ :
Φ _[∗]_ _←_ _argmin_ Φ _L_ ( _D_ ; Φ) (20)
Adapters can be broadly categorized into series adapters and parallel types, depending on how they
are integrated with the backbone [Hu et al., 2023].
Series adapters attach learnable modules sequentially within selected layers, typically within
transformer layers. [Houlsby et al., 2019] proposed the first series adapter as a parameter-efficient
fine-tuning method by inserting trainable modules twice in each transformer layer. Adamix introduces multiple down- and up-projection layers in each adapter [Wang et al., 2022b]. It employs
stochastic routing to randomly select projection pairs during training. By maintaining the same
number of tunable parameters and computational cost as the underlying PEFT, this technique
outperforms these methods and even full fine-tuning in several datasets. A structural limitation
is that the backpropagation must still pass through the main backbone, resulting in additional
computational steps and increased memory usage. To further reduce training time and memory
consumption, 1) adapter pruning and 2) sparse calculation techniques have been proposed. For
instance, solely inserting the adapter module in each transformer layer [Pfeiffer et al., 2020] or
dropping adapter in lower-level layers [R¨uckl´e et al., 2021] can still preserve performance. Compacter replaces standard linear layers with Kronecker products and shared parameters [Mahabadi
Figure 8: Overview of Adapter in LLMs. Adapters are lightweight, trainable modules inserted into
the layers of a pre-trained, frozen LLM to enable adaptation to specific downstream tasks. By
keeping the core model parameters fixed and training only the adapter components, this approach
greatly reduces computational costs. Depending on how they are integrated into the LLMs, adapters
are generally categorized as series and parallel types and numerous variants have been developed
to enhance both performance and efficiency.
et al., 2021]. This technique maintains the sparsity and reduces computational complexity while
achieving the performance comparable to original down- and up- projection calculations.
Parallel adapters incorporate additional adapter modules in parallel with specific layers, which
has similar manner as LoRA [He et al., 2022]. When these adapters are connected in paralleled to
the original layers, backpropagation can be applied directly across each adapter by linking them
explicitly [Sung et al., 2022]. Multi-adapter is an extension on parallel adapter structure, which
modifies the output of the self-attention heads through multiple parallel adapters. In multi-task
settings, especially when tasks are sequentially related, single-task adapters often suffer performance
degradation across tasks. AdapterFusion introduces a mechanism that parallels multiple taskspecific adapters within the attention layer and fuses their outputs [Pfeiffer et al., 2021]. This
approch achieves better performance than using a single-task adapter alone.
As a widely used PEFT technique, adapters can also be compatible with other PEFT methods
[Mao et al., 2022]. This ability makes adapter have flexible choices for either the demand on
inference speed or multi-task performance etc.


Each fine-tuning technique discussed in this section offers distinct advantages and limitations regarding computational efficiency, adaptability, and performance. Table 6 provides a comprehensive
summary, comparing critical aspects such as trainable parameters, computational and memory
overhead, adaptability, and inherent trade-offs.
Full-parameter fine-tuning updates every parameter in a model, resulting in typically superior
performance and high adaptability. However, it incurs substantial computational and memory demands, which become increasingly prohibitive with larger models. Partial parameter fine-tuning
alleviates some of these burdens by selectively updating essential parameters, though it adds complexity in choosing optimal subsets and may risk suboptimal adaptation if crucial layers remain
unchanged.
LoRA introduces small amount of trainable parameters (less than 1%) by employing low-rank
matrices, significantly reducing computational costs. While highly efficient, LoRA can fall short in
tasks requiring deep reasoning or complex coding without further optimization. It balances well
between resource efficiency and moderate performance.
Sparse fine-tuning effectively reduces training overhead by selectively updating parameters
based on sparsity measures like gradient magnitudes or Fisher information. This approach typically
achieves near full fine-tuning performance at significantly lower computational costs, provided the
sparse masks are optimally determined. Nonetheless, it depends heavily on the quality and robustness of the sparsity strategy.
Knowledge distillation transfers knowledge from a larger teacher model to a smaller student
model, considerably reducing inference costs. While computationally efficient during inference,
this approach involves an initial cost for training the teacher model. Its performance relies significantly on the teacher’s quality and the distillation method, with possible performance degradation
compared to the original large model.
Adapter-based fine-tuning achieves high adaptability and efficiency by inserting small, trainable
adapter modules into pre-trained models. Although adapters facilitate efficient multitask learning
and quick adaptation, they introduce computational overhead proportional to the complexity of
the adapter modules, especially in series configurations.
Prompt tuning maintains the original model entirely frozen, adjusting only continuous prompt
embeddings. This method offers minimal computational and memory overhead, suitable for resourceconstrained environments. However, prompt tuning often provides limited adaptation capabilities
compared to methods that directly update model parameters, constraining its effectiveness on tasks
significantly divergent from the original pre-training objectives.
Table 6: Comprehensive Comparison of Efcient Fine-Tuning Techniques.

8.6 Comparison of Fine-Tuning Techniques


Full-Parameter FT High High High High
Partial Parameter FT Medium Medium Medium Medium
LoRA Low Low Low Medium
Sparse FT Low Low Low High
Knowledge Distillation Medium Medium Medium Medium
Adapter-Based FT Low Medium Medium High
Prompt Tuning Low Low Low Medium
In summary, choosing an appropriate fine-tuning method depends on specific deployment scenarios and resource constraints, balancing between desired performance, computational resources,
and model adaptability.

Method Trainable Parameters Memory Overhead Compute Cost Adaptability


This section outlines the key principles and methodologies underlying brain-inspired LLM alignment, often referred to as Brain-AGI, highlights the existing challenges, and provides insights into
possible opportunities.


Brain-inspired LLMs refers to language models whose architectures or training objectives draw directly on principles of human brain organization and function. Instead of treating neural networks
Figure 9: Different Perspectives that Brain-Inspired LLM can help Alignment.
as arbitrary black boxes, brain-inspired LLMs seek to mirror how our brains process language, represent concepts, and integrate sensory information [Farisco et al., 2024]. For instance, the Causal
Cognitive Architecture (CCA) [Schneider, 2024], a framework that models neocortical minicolumns
as millions of “navigation maps.” These maps undergo continuous cognitive cycles in which sensory
inputs are normalized, spatially and temporally bound into local maps, and then matched against
stored multisensory maps to form a Working Navigation Map. The CCA framework demonstrates
how lightweight, evolution-inspired modifications to a core navigation-map framework can spontaneously produce foundational aspects similar to that of human intelligence [Schneider, 2024].
Additionally, the BriLLM architecture [Zhao et al., 2025] replaces the Transformer’s [Vaswani
et al., 2017] attention blocks with a directed graph of “neuronal” nodes and energy-based signal
flow, analogous to how biological neurons propagate activation along pathways of least resistance.
In BriLLM, tokens are treated as nodes, and each edge carries a learnable “energy tensor” that
determines which node (token) activates next. This design makes every internal connection interpretable and, in principle, allows unbounded context lengths [Zhao et al., 2025].
Furthermore, Sun et al. [2024] investigated whether LLMs exhibit a brain-like functional organization by directly linking sub-groups of artificial neurons (ANs) in models such as BERT and
the Llama family to well-established human functional brain networks (FBNs). They extracted
representative temporal “atoms” of neuron activity and used them to predict fMRI responses,
demonstrating that these LLMs indeed form a modular, brain-like architecture. Comparing four
models, BERT and three successive Llama variants, they found that this functional specialization
strengthens with model sophistication: larger models yield brain maps that are both more consistent (showing reduced variability in engaged FBNs) and more compact (fewer, more specialized
atoms per network), and their neurons display increasingly hierarchical temporal and anatomical
distributions [Sun et al., 2024].

9.1 Recent advancements of Brain-Inspired LLM Alignments


The motivation for brain and artificial general intelligence (AGI) co-working arises from the limitations of conventional AI systems, which often struggle with rigid task boundaries, inefficient energy
use, and limited contextual understanding [Meftah et al., 2025]. In contrast, biological brains
demonstrate remarkable capabilities in generalization, lifelong learning, and causal reasoning, qualities that inspire the design of AGI systems seeking to replicate such flexibility and intelligence
[Gabriel, 2020a]. Brain-AGI collaboration is envisioned as a way to harness the complementary
strengths of human cognition and machine computation [Zhao et al., 2023], enabling more effective
responses to complex societal challenges such as healthcare delivery and climate modeling [Lu et al.,
2023]. This synergy leverages human creativity and judgment alongside AGI’s capacity for scale
and speed. A central driver of this co-working paradigm is the pursuit of ethical alignment, which
ensures that AGI systems are not only powerful but also aligned with human values and intentions
[Conitzer et al., 2024].
_Theoretical foundations._ The co-working of the brain and AGI is underpinned by a set of
complementary theoretical principles drawn from neuroscience and cognitive science [Yu et al.,
2024c]. First, predictive coding provides a computational framework for modeling shared brain-AGI
processing, wherein both biological and artificial agents minimize prediction error via hierarchical
Bayesian inference [Fagerholm et al., 2020]. Second, neuroplasticity offers a foundation for coadaptive learning systems, enabling AGI to dynamically update internal representations in response
to environmental and social interactions, which is similar to synaptic rewiring in the human brain
[Rebedea et al., 2023]. Third, embodied cognition underscores the significance of sensorimotor
grounding for both natural and artificial agents, positing that cognition emerges through realworld engagement. This is an essential principle for brain-AGI integration in collaborative robotics
and assistive technologies [Taniguchia et al., 2022]. These three pillars are further reinforced by
neurosymbolic integration, a hybrid framework that aligns neural learning with symbolic reasoning,
offering interpretable and generalizable architectures essential for effective and trustworthy humanAGI co-working [Bhuyan et al., 2024].
_Current strategies._ Recent progress in brain-inspired artificial intelligence has catalyzed a new
wave of brain-AGI co-working paradigms, driven by four interconnected approaches [Yu et al.,
2024c]. First, neuromorphic engineering develops hardware systems that emulate the structure
and dynamics of biological neural circuits. Technologies like Intel’s Loihi chip leverage spiking
neural networks to enable real-time, energy-efficient computation, closely mirroring the temporal
coding strategies of the brain [Yik et al., 2023]. Second, cognitive architectures such as CLARION
[Sun, 2006] integrate symbolic reasoning with subsymbolic learning (e.g., deep learning, reinforcement learning), supporting flexible, goal-directed behavior that aligns with human-like cognitive
processes. Third, human-in-the-loop learning frameworks, including inverse reinforcement learning
and preference modeling, bring human values and feedback into the AGI training loop, enabling
systems to adapt to social and ethical contexts through cooperative learning [Hadfield-Menell et al.,
2017]. Finally, neuroadaptive interfaces enable real-time, bidirectional interaction between human
neural activity and artificial agents. These systems dynamically decode brain signals to adapt AGI
responses, fostering seamless collaboration between biological and artificial intelligences [Guo et al.,
2024b].
_Applications._ Brain-AGI co-working is unlocking transformative applications across multiple
fields by combining human cognitive strengths with artificial general intelligence [Zhao et al., 2023].
In healthcare, this synergy enables early diagnosis of neurodegenerative diseases like Alzheimer’s
disease through AGI-assisted interpretation of fMRI data, and supports real-time decision-making
in surgery via brain-computer interfaces [Fedorova et al., 2022]. In autonomous systems, brainAGI collaboration empowers robots with human-like spatial reasoning and adaptability, enhancing
performance in complex, unstructured environments such as disaster response scenarios [Blackiston
et al., 2022]. Climate science similarly benefits from co-working paradigms, where AGI tools informed by human-guided intuition and causal reasoning predict ecological tipping points, guiding
proactive environmental policies [Harder et al., 2022]. These applications exemplify how brain-AGI
integration not only augments machine capabilities but also extends human potential in addressing
critical global challenges.
_Challenges._ The pursuit of brain-AGI co-working, where artificial general intelligence systems
are designed to complement or integrate with human cognitive processes, faces profound challenges
across technical, ethical, and interdisciplinary domains [Zhao et al., 2023]. Technically, neuromorphic systems inspired by the brain remain constrained by scalability and energy efficiency
[Zolfagharinejad et al., 2024], limiting their capacity to support the high computational demands
of AGI [Kurshan, 2024]. Ethically, the prospect of AGI systems operating alongside or within
human cognitive environments raises concerns about value misalignment, loss of human agency,
and the amplification of biases without adequate oversight mechanisms [Everitt et al., 2018]. Most
critically, the gap between neuroscience and AI research continues to hinder meaningful co-design.
The incompatible frameworks, terminologies, and research priorities make it difficult to translate
biological principles into actionable AI models or to embed AGI into cognitive contexts in a way
that is both effective and safe [Hassabis et al., 2017]. Overcoming these barriers will require not
just technical innovation but also deep, sustained integration across disciplines and a commitment
to embedding human values at the heart of AGI design.


Despite tremendous achievements in the brain-inspired LLM architectures such as enhanced interpretability, modular signal paths, and the promise of unbounded context lengths, several challenges
remain. First, the computational overhead of maintaining and updating dense “energy tensors”
across every connection in a graph-based design can be substantial, potentially eroding the very
energy-efficiency gains these models aim to deliver [Zhao et al., 2025].
Second, current alignment methods such as DPO align an LLM by directly maximizing the
likelihood of human-preferred completions. DPO accomplishes this by reparameterizing reward
as the log-ratio between the policy and a fixed reference model, thus eliminating the need for a
separate reward network [Xu et al., 2024c]. Although DPO often matches or outperforms RLHF
on in-distribution benchmarks, it fundamentally learns only the surface-level patterns of human
preferences rather than the deeper cognitive processes [Tennant et al., 2025]. As a result, DPOtuned models can “sound” convincingly human on familiar prompts but remain vulnerable to reward
hacking and poor generalization when faced with novel or out-of-distribution scenarios.
Third, integrating ethical and societal considerations, such as bias mitigation or transparent
governance, into a highly specialized, brain-inspired AGI framework remains an open problem, as
standard alignment tools (RLHF, DPO) do not yet translate directly onto these new architectures
[Farisco et al., 2024]. Finally, scaling this graph-based signal-flow mechanism to models with billions
of parameters and multimodal inputs has not yet been demonstrated at production scale, raising
questions about its practicality for real-world applications.

9.3 Challenges and Limitations of Brain-Inspired LLM Alignments


Brain-inspired LLM alignment techniques offer several promising advantages over conventional
preference-fitting approaches. By explicitly tying language-processing pathways to analogues of
human cognitive systems, equips us with the ability to pinpoint exactly where and why a model’s
reasoning diverges from our expectations. This transparency doesn’t just aid debugging; it lays the
groundwork for truly collaborative workflows in which domain experts can inspect, validate, and
open pathways toward achieving Brain-AGI.
Furthermore, aligning LLM subnetworks with well-characterized human functional brain networks, such as the “language” network for factual consistency or the “default-mode” network for
social reasoning, provides a neuroscientific scaffold for targeted auditing and control [Sun et al.,
2024]. This modular interpretability makes it possible to apply fine-grained alignment objectives,
such as moral rewards derived from deontological or utilitarian principles, to dedicated subnet
works, reducing collateral effects on other competencies and mitigating reward-hacking risks seen
in methods like DPO. Lastly, grounding alignment in multimodal, embodied representations, by
integrating vision, audio, and proprioceptive signals into the same brain-inspired modules, promises
richer, context-sensitive behavior that better mirrors human cognition and values [Farisco et al.,
2024].
Looking forward, next-generation brain-inspired alignment will likely embrace dynamic plasticity mechanisms, enabling models to adapt in real time through localized synaptic-style updates
rather than wholesale retraining. Instead of relying on static modules, future systems may incorporate continuous update rules that refine internal representations in response to live user feedback
or environmental signals, much like how our brains learn from experience. Coupling such plasticity
with lightweight neurofeedback, whether from biosignals or behavioral proxies, could give rise to
true brain-in-the-loop pipelines that preserve robust alignment across shifting contexts. Finally, as
Brain-AGI systems grow more autonomous, evolving multidisciplinary governance frameworks that
integrate insights from neuroscience, ethics, and policy will be essential to ensure that alignment
innovations remain transparent, accountable, and aligned with societal norms.

9.4 Opportunities and Future Developments


Alignment Uncertainty Quantification (AUQ) addresses a fundamental challenge in LLM development: how to measure and manage the uncertainty inherent in aligning models with human
values and intentions [Gabriel, 2020b, Bommasani et al., 2021]. As models become more powerful, understanding the reliability of their alignment becomes critical for safe deployment [Reynolds
and McDonell, 2021]. Traditional machine learning uncertainty focuses primarily on predictive
accuracy, while alignment uncertainty concerns whether a model’s behavior truly reflects intended
human values across diverse contexts and edge cases. The theoretical foundations of alignment
uncertainty, methodologies for its quantification, and approaches for building systems that remain
robustly aligned despite inevitable uncertainties are examined in this section.

10 Alignment Uncertainty Quantification (AUQ)