In many complex tasks, such as dialogues (Ouyang et al., 2022), constructing precise rule-based rewards presents
a challenge (Bender et al., 2021). At the same time, methods based on demonstration might require a substantial
investment of expert human resources, resulting in high costs. Currently, preference modeling based on comparison
feedback (Akrour et al., 2011) has emerged as a very promising method (Ouyang et al., 2022; OpenAI, 2023a;
Touvron et al., 2023) to assist in fine-tuning powerful AI systems (Amodei et al., 2016).
Typically, it is necessary to iteratively explore the system dynamics while acquiring expert preference data to
gain more knowledge about the optimization objectives. This process is known as _Preference Elicitation_ (Wirth and
Fürnkranz, 2013; Wirth et al., 2017; Christiano et al., 2017; Cabi et al., 2020), which is crucial for obtaining rich,
valuable feedback related to AI system outputs, thus guiding the alignment process (Hejna III and Sadigh, 2022).
Within _Preference Elicitation_, two core decisions that need to be determined are the _Granularity of Preference_
and the _Category of Preference_ . This paper introduces these within sequential decision-making problems, but the
insights derived apply to a broad array of AI systems (Amodei et al., 2016; Christiano et al., 2018; Leike et al.,
2018).
**Granularity of Preference** Preference (Wirth et al., 2017) can primarily be categorized into three types by
granularity: _Action_, _State_, and _Trajectory_ (as shown in Table 2).
The _Action_ preference focuses on comparing actions within a particular state, specifying the preferred action under specific conditions. When translated into trajectory preferences, it may impose challenges such as evaluators’
expertise needs and potential information loss. The _State_ preference deals with comparing states. It encapsulates
preference relations among states but requires assumptions about state reachability and independence when translating to trajectory preferences. The _Trajectory_ preference considers whole state-action sequences, offering more
comprehensive strategic information. It inherently assesses long-term utility and depends less on expert judgment.
Christiano et al. (2017) demonstrates, using ablation studies, that in the settings that they studied, longer trajectory segments yield more informative comparisons on a per-segment basis. Such segments are also more consistently evaluated by humans in MuJoCo tasks.
**Category of Preference** Diverse objectives exist within preference modeling. Based on their targets, preferences
can be categorized into object preference and label preference (Fürnkranz and Hüllermeier, 2010). Specifically,
object preference operates on a set of labels for each instance, whereas label preference acts on a set of objects
themselves. One can further classify them differently based on the form of preferences.
 - **Absolute Preferences** . Absolute preferences independently articulate each item’s degree of preference.
**– Binary** . Classifying items as liked or disliked offers a simplistic and straightforward model of user preference (Tsoumakas and Katakis, 2007; Cheng et al., 2010a).
**– Gradual** . This can be further distinguished between numeric and ordinal preferences. Numeric preferences employ absolute numerical values, such that each item receives a numerical score, which reflects the
extent of preference (Cheng et al., 2010b). On the other hand, ordinal preferences entail a graded assessment of a fixed set of items as either preferred, less preferred, or intermediary, _etc._, enabling the depiction
of user preferences without including specific numerical measurements (Cheng et al., 2010a).
 - **Relative Preferences** . Relative preferences define the preference relation between items.
**– Total Order** . This form establishes a comprehensive preference relation covering all item pairs, asserting
an absolute ordering of preferences ranging from the most preferred to the least (Hüllermeier et al., 2008).
**– Partial Order** . Because users may not exhibit a distinct preference between two items in some instances
(Cheng et al., 2010c), this allows for incomparable item pairs.
2.3 Policy Learning
**Reward Model** Reward modeling transfers comparison feedback (Fürnkranz and Hüllermeier, 2010; Wirth et al.,
2017) to the scalar reward form, facilitating policy learning (Christiano et al., 2017; Cabi et al., 2020; Touvron et al.,
2023). Given pairs of actions ( _y_ 1 _,y_ 2) performed by the RL agent in the same state. The preference is denoted as
_yw_ ≻ _yl_ | _x_, where _yw_, _yl_ represents the preferred and less preferred action respectively among ( _y_ 1 _,y_ 2). We assume
these preferences emerge from a latent reward model _r_ [∗] ( _x,y_ ), which we lack direct access to. Several methods
exist to model such preferences, _e.g._, the Bradly-Terry Model (Bradley and Terry, 1952), Palckett-Luce ranking
model (Plackett, 1975), _etc._ Under the BT model, the distribution of human preference, denoted as _p_ [∗], can be
formalized as,
exp                  - _r_ [∗] ( _x,y_ 1)�                  -                  _p_ [∗] ( _y_ 1 ≻ _y_ 2 | _x_ ) = exp ~~�~~ _r_ [∗] ( _x,y_ 1) ~~�~~ + exp ~~�~~ _r_ [∗] ( _x,y_ 2) ~~�~~ = _σ_ _r_ [∗] ( _x,y_ 1) − _r_ [∗] ( _x,y_ 2) _._
where _σ_ ( _x_ ) = 1 _/_ (1 + exp(− _x_ )) is the logistic sigmoid function. Subsequently, we use the derived preference
rankings to train the parameterized reward model, optimizing its parameters through maximum likelihood.
                           -                           -                           -                           - [��]
LR ( _θ_ ) = −E( _x,yw,yl_ )∼D log _σ_ _rθ_ ( _x,yw_ ) − _rθ_ ( _x,yl_ )
In this negative log-likelihood loss, the problem is a binary classification task, where D signifies the static dataset
- - _N_
_x_ [(] _[i]_ [)] _,yw_ [(] _[i]_ [)] _[,y]_ _l_ [(] _[i]_ [)] sampled from _p_ [∗] ( _i.e._, human-labeled comparisons).
_i_ =1
Reward models enable human users to impart specific preferences to these systems via evaluations, thereby
circumventing the complex task of defining objectives explicitly. Initially, the studies by Knox (2012); Knox
and Stone (2013) distinctively treat human reward as separate from the traditional rewards of MDP and conduct
a reward modeling process around it. Transitioning from these simpler cases, Christiano et al. (2017) propose
that utilizing supervised learning to construct a distinct reward model asynchronously can substantially diminish
interaction complexity by approximately three orders of magnitude. The study conducted by Ibarz et al. (2018)
integrates expert demonstrations with human preferences, such that the policy initially mimics expert demonstrations and then sequentially collects human trajectory annotations, trains the reward model, and updates the policy.
This research also provides practical insights for precluding the overfitting of the reward model and the occurrence
of _reward hacking_ - a scenario where escalating rewards do not translate to improved performance, especially
when the policy is excessively trained. Additionally, a random policy might rarely exhibit meaningful behavior
for tasks that surpass the complexity of Atari (Palan et al., 2019; Jeon et al., 2020). This implies that for effective
annotation, the policy itself must possess certain capabilities to perform improved behavior. Offline settings also
benefited from the reward model. Cabi et al. (2020) proposes reward sketching to efficiently learn a reward model
that leverages humans’ episodic judgments for automated reward annotation of historical data, enabling large-scale
batch RL. Qiu et al. (2024) provides an empirically-grounded theory of reward generalization in RMs, based on
which a new type of RM based on tree-structured preferences is proposed and experimentally validated.
Importantly, the reward model provides an essential tool for aligning powerful LLMs. Stiennon et al. (2020)
employs reward models grounded in human preferences for text summarization tasks, resulting in significant policy
enhancements. This work also delves into the issues of distribution shift and reward model generalization, revealing
that the effectiveness of the reward model correlates with data scale and parameter size. Building upon this work,
InstructGPT (Ouyang et al., 2022) extends the reward model paradigm to broader dialogue task reward modeling
and introduces a preference-optimizing loss function for multiple responses to mitigate overfitting. Furthermore,
this research reveals that the preferences derived from the reward model can be generalized across different groups.
- _N_
sampled from _p_ [∗] ( _i.e._, human-labeled comparisons).
_i_ =1


Policy learning aims to learn the mapping from perceived states to actions taken when in those states (Sutton
and Barto, 2018) to optimize a model’s performance in specific tasks. Numerous alignment-related challenges
manifest within policy learning (as shown in Section1.1.2). Consequently, policy learning provides a crucial backdrop
for alignment, and its techniques can further advance alignment objectives (Amodei et al., 2016; Christiano et al.,
2018; Ibarz et al., 2018). This section discusses various domains within policy learning and then introduces RLHF,
a powerful technique for policy learning (OpenAI, 2023a; Touvron et al., 2023).

Jiaming Ji
Tianyi Qiu
Boyuan Chen
Borong Zhang
Hantao Lou
Kaile Wang
Yawen Duan
Zhonghao He
Jiayi Zhou
Zhaowei Zhang
Fanzhi Zeng
Kwan Yee Ng
Juntao Dai
Xuehai Pan
Aidan O'Gara
Yingshan Lei
Hua Xu
Brian Tse
Jie Fu
Stephen McAleer
Yaodong Yang
Yizhou Wang
Song-Chun Zhu
Yike Guo
Wen Gao


We introduce some general areas of policy learning here to give readers a general background.
**Reinforcement Learning (RL)** RL enables agents to learn optimal policies by trial and error via interacting with
the environment (Sutton and Barto, 2018). This paradigm has achieved great success in tackling complex tasks
(Agostinelli et al., 2018; Yu et al., 2021; Fawzi et al., 2022; Baker et al., 2022; Afsar et al., 2022; Mankowitz et al.,
2.3 Policy Learning
2023; OpenAI, 2023b), demonstrating its potential for decision-making and control in complex state spaces. The
goal of RL is to learn a policy _π_ which executes actions _a_ in states _s_ to maximize the expected cumulative reward
under environment transition dynamics _P_ and the initial state distribution _ρ_ 0:


∞
_γ_ _[t]_ _r_ ( _st_ )
_t_ =0
_,_ where _s_ 0 ∼ _ρ_ 0(·) _, at_ ∼ _π_ (·| _st_ ) _, st_ +1 ∼ _P_ (·| _st,at_ ) _._

_π_ [∗] = argmax
_π_
E _s_ 0 _,a_ 0 _,..._



Even though RL still faces challenges like sample efficiency and stability (Bu¸soniu et al., 2018). Proximal policy
optimization (PPO) (Schulman et al., 2017) is an influential algorithm in the RL community, serving as the key
algorithm for RLHF (Ouyang et al., 2022). The key idea of PPO is to limit the policy update to prevent significant
deviations from the original policy by introducing a proximity objective. Sikchi et al. (2023) unifies several RL
and Imitation Learning (IL) algorithms under the framework of dual RL through the lens of Lagrangian duality.
**Preference-based Reinforcement Learning (PbRL)** PbRL (Wirth et al., 2017) seeks to facilitate training RL
agents using preference feedback instead of explicit reward signals (Christiano et al., 2017; Sadigh et al., 2017). [26]
PbRL integrates the advantages of preference learning and RL, broadening the application range of RL and mitigating the difficulties associated with reward function formulation, and has been efficaciously deployed in a variety
of tasks such as robotic instruction (Kupcsik et al., 2013), path planning (Jain et al., 2013), and manipulation
(Shevlane et al., 2023). In PbRL, the emphasis predominantly lies on trajectory preferences ( _i.e._, comparisons
of state-action sequences segment) (Wirth et al., 2017). Such trajectory preferences encapsulate a human evaluation of various behavioral outcomes rather than single states, rendering PbRL more suitable for non-expert users
(Christiano et al., 2017; Shin et al., 2023; Kim et al., 2023). A general example of PbRL is the _weighted pairwise_
_disagreement loss_ (Duchi et al., 2010) balancing multiple potentially conflicting preferences to identify a singular
optimal policy:
L( _π,ζ_ ) =
_N_
_αiL_ ( _π,ζi_ ) _,_
_i_ =1
where L( _π,ζ_ ) is the aggregated loss for policy _π_ over all preferences _ζ_, _αi_ is the weight of the _i_ th preference, and
_L_ ( _π,ζi_ ) is the loss associated with the policy _π_ in relation to the specific preference _ζi_ .
Compared to exact numerical rewards, preference feedback has several benefits (Wirth et al., 2017), such as (1)
circumventing arbitrary reward design, reward shaping, reward engineering, or predefined objective trade-offs, (2)
diminishing reliance on expert knowledge, and (3) decoupling training loop with human by modeling preferences
(Akrour et al., 2012). However, PbRL also faces challenges, including credit assignment problems due to temporal
delays, practical exploration of preference space (Wirth et al., 2017), the potential need for massive data (Ouyang
et al., 2022), and the inability to use the learned preference model for retraining (McKinney et al., 2022).
**Imitation Learning (IL)** IL (Schaal, 1999; Syed et al., 2008), also referred to as learning from demonstration or
apprenticeship learning, focuses on emulating human behaviors within specific tasks. The agent learns a mapping
between observations and actions and refines its policy by observing demonstrations in a collection of teacher
demonstration data D (Bakker et al., 1996; Hussein et al., 2017). This process obviates the need for environmental
reward signals (Hussein et al., 2017). Broad IL (Cotra, 2018) aims to replicate human desires and intentions,
effectively creating replicas of human decision-making processes. This concept is central to technologies such
as Iterated Distillation and Amplification (IDA, as shown in Section2.4.2) (Christiano et al., 2018). On the other hand,
Narrow IL aims to replicate specific human behaviors within given tasks. Behavioral cloning (BC) (Bain and
Sammut, 1995; Ross et al., 2011; Osa et al., 2018) is a simple (Pomerleau, 1991; Ravichandar et al., 2020) strategy
that learns directly from demonstrations using supervised learning (Schaal, 1996). BC method specifically seeks to
optimize the policy parameters, _φ_, with the objective of aligning the policy _πφ_ ( _a_ | _s_ ) closely with the expert policy
_πE_ ( _a_ | _s_ ). This alignment is achieved through the minimization of the negative log-likelihood, as delineated in the
following (Lynch et al., 2020):
                                    -                                    LBC( _φ_ ) = −E( _s,a_ )∼ _πE_ log _πφ_ ( _a_ | _s_ ) _._
Here, the expectation is computed over state-action pairs sampled from the expert policy, _πE_ . However, it faces
the Out-of-Distribution (OOD) problem, arising from the difference between the training and testing distributions
(Ross et al., 2011; Ho and Ermon, 2016; Reddy et al., 2019; Zhou et al., 2022). Adversarial imitation learning
26Notably, Sadigh et al. (2017) explicitly maintains a probabilistic belief over the true reward function during learning, and
actively constructs queries to the human to reduce uncertainty maximally. Both traits are in a similar spirit to _cooperative_
_inverse reinforcement learning_ (CIRL), and later work also continues this theme (Reddy et al., 2020). See Section2.4.5 for more.
2.3 Policy Learning
methods (Ho and Ermon, 2016; Fu et al., 2018a; Lee et al., 2019; Ghasemipour et al., 2020) have demonstrated
an ability to enhance the robustness of policies against distribution shifts. However, these methods learn nonstationary rewards, which cannot be used to train new policies (Ni et al., 2021).
**Inverse Reinforcement Learning (IRL)** Unlike the paradigm of IL, IRL (Adams et al., 2022) focuses on deriving a reward function from observed behavior (Ng et al., 2000; Arora and Doshi, 2021). Standard IRL methods
include the feature matching methods (Abbeel and Ng, 2004), which assumes optimal expert behavior or decision
processes, as well as the maximum entropy methods (Ziebart et al., 2008) and the Bayesian methods (Ramachandran and Amir, 2007), both of which do not require optimal behavior. IRL guarantees robustness to changes
in the state distribution but at the cost of increased computational complexity due to the extra RL step (Ho and
Ermon, 2016; Fu et al., 2018b). This interaction, meanwhile, introduces inherent RL challenges, _e.g._, sample efficiency (Yu, 2018) and potential dangers in environment interaction (Garcıa and Fernández, 2015). Additionally,
identifying the reward function remains a challenge (Kim et al., 2021).


RLHF expands upon PbRL within the domain of DRL (Christiano et al., 2017), aiming to more closely align
complex AI systems with human preferences (OpenAI, 2023b). Its principal advantage is that it capitalizes on
humans being better at judging appropriate behavior than giving demonstrations or manually setting rewards.
This approach has gained significant traction, particularly in fine-tuning LLMs (Ouyang et al., 2022; OpenAI,
2023a; Touvron et al., 2023). Nonetheless, RLHF encounters obstacles (Casper et al., 2023b), including data
quality concerns, the risk of reward misgeneralization, reward hacking, and complications in policy optimization.
Specifically, RLHF can also be viewed as a Recursive Reward Modeling (RRM) process (as shown in Section2.4.3)
without deep recursive modeling (Leike et al., 2018). Here, we provide a brief review of the RLHF methodology.
The genesis of RLHF can be traced back to Knox and Stone (2008, 2012), subsequently broadening its reach
to domains such as social robots (Knox et al., 2013) and human-AI cooperative learning (Griffith et al., 2013).
Besides focusing on the association between feedback and policy, Loftin et al. (2016) models the connection
between feedback and the trainer strategy. Christiano et al. (2017) extended RLHF to simulated robotic tasks,
demonstrating its potential effectiveness.
It’s worth noting that one of the significant applications of RLHF has been in the field of LLMs. Some work
found that LLMs trained with RLHF (Ouyang et al., 2022; Korbak et al., 2023; Christiano, 2023) are more creative
and human alignment compared to models trained via supervised or self-supervised learning approaches (Kenton
and Toutanova, 2019; Brown et al., 2020b). The importance of RLHF is not merely limited to allowing LLMs
to follow human directives (Ouyang et al., 2022). It helps LLMs better align by giving them important qualities
like being helpful, harmless, and honest (Bai et al., 2022a). Due to these improvements, many works use RLHF
for aligning LLMs (Ziegler et al., 2019; Stiennon et al., 2020; Bai et al., 2022a; Glaese et al., 2022; OpenAI,
2023a; Touvron et al., 2023). Additionally, Dai et al. (2024b) integrates the Safe RL (Garcıa and Fernández, 2015)
framework with the RLHF, addressing the inherent tension between aligning helpfulness and harmfulness (Bai
et al., 2022a). Future efforts can be focused on reducing dependence on human annotation (Wang et al., 2023c;
Sun et al., 2024) and improving the efficacy of the reward model by leveraging iterative RLHF methods ( _i.e._,
integrating it with debate frameworks (Irving et al., 2018)), _etc._ Qiu et al. (2024) has also built a formal framework
of the RLHF process portraying it as an autoencoding process over text distributions, and enables analysis of
convergence properties in RLHF.
We review the RLHF pipeline from the Ziegler et al. (2019); Ouyang et al. (2022); Rafailov et al. (2024) to give
a general framework. It usually consists of three stages:
 - **Supervised Fine-tuning (SFT)** . RLHF usually starts with a pre-trained language model, then fine-tuned using supervised learning – specifically, maximum likelihood estimation – on a high-quality human instruction
dataset tailored for downstream tasks to obtain a model _π_ [SFT] . Examples of these tasks include dialogue
handling, instruction following, and summarization (Some open-source datasets include Alpaca Data (52k
instruction-following data) (Taori et al., 2023), Vicuna (70K user-shared ChatGPT conversations) (Chiang
et al., 2023), _etc._ ). This stage can also be carried out at any other stage.
 - **Collecting Comparison Data and Reward Modeling** . This phase includes collecting comparison data, which
is subsequently used to train a reward model. The SFT model is given prompts denoted as _x_ to generate pairs
of responses ( _y_ 1 _,y_ 2) sampled from _π_ [SFT] ( _y_ | _x_ ). These pairs are subsequently shown to human annotators, who
indicate a preference for one of the responses. Then as discussed in Section2.2, comparison data is used to construct
the reward model _rθ_ .
 - **Policy Optimization via Reinforcement Learning** . The final step is optimizing LLM as a policy _π_ through
RL, guided by the reward model _rθ_ . The process of LLMs generating responses from prompts is modeled as
2.3 Policy Learning
a bandit environment (Ouyang et al., 2022), where a reward is obtained from reward model _rθ_ at the end of
each response. The primary objective of RL is to adjust the parameters _φ_ of the LLMs such that the expected
reward on training prompt dataset DRL is maximized:
                                      -                                       argmax E _x_ ∼DRL _,y_ ∼ _πφ_ _rθ_ ( _x,y_ ) _._
_πφ_
Typically, an additional per-token KL penalty derived from the SFT model _π_ [SFT] is involved to mitigate the
reward over-optimization. In addition, the integration of gradients from pre-training distribution Dpretrain helps
maintain model performance, denoted as PTX loss in (Ouyang et al., 2022). As a result, a more comprehensive
practical objective function is introduced:
J ( _φ_ ) = E _x_ ∼DRL _,y_ ∼ _πφ_
- - - [�]
_rθ_ ( _x,y_ ) − _β_ log _πφ_ ( _y_ | _x_ ) _/π_ [SFT] ( _y_ | _x_ ) + _η_ E( _x,y_ )∼Dpretrain
- - - [�]
log _πφ_ ( _y_ | _x_ ) _,_
where _β_ and _η_ are coefficients determining the intensity of the KL penalty and the mixture of pretraining
gradients respectively. This process refines the LLMs to generate responses that better align with human
preferences for the prompts used during training.
Though RLHF has been proven effective for aligning LLMs with human preferences, this method has problems
like complex implementation, hyper-parameter tuning, sample efficiency (Choshen et al., 2019), and computational
overhead (Yuan et al., 2024), making it hard to scale up.
A straightforward approach is rejection sampling (Dong et al., 2023; Touvron et al., 2023) paired with finetuning
on the best examples. For every prompt, _K_ responses are sampled from the model. Each response is then assessed
with the reward model, and the one with the highest reward is selected as the best response. This selected response
is later used for model fine-tuning. Zhang et al. (2023a) formulates the language model instruction alignment problem as a goal-reaching reinforcement learning problem and proposes the HIR algorithm. The method unfolds in
two stages: online sampling and offline training. During online sampling, the algorithm samples the LLM at a high
temperature. In the offline training stage, instructions are relabeled based on generated outputs, followed by supervised learning using this relabeled data. HIR capitalizes on successful and failed cases without requiring additional
parameters. RRHF, as introduced by (Yuan et al., 2024), aligns model probabilities with human preferences by
scoring and ranking responses from multiple sources. With the necessity for only 1 or 2 models, its implementation
is straightforward. RRHF reported it can effectively align language models with human preferences, producing
performance on par with PPO. Gulcehre et al. (2023) proposes the ReST algorithm, which contains two loops:
_Grow_ and _Improve_ . The _Grow_ loop uses the current model to sample and generate a dataset, while the _Improve_
loop iteratively trains the model on a fixed dataset. This algorithm provides a simple and efficient framework that
allows repeated use of the fixed dataset to improve computational efficiency, showing significant improvement in
the reward model scores and translation quality compared to supervised learning baselines. Motivated by the dependence of reward modeling on policy optimization in RLHF, Chakraborty et al. (2024) propose PARL, a bilevel
optimization-based framework.
Rafailov et al. (2024) introduces the DPO, which demonstrates a mapping between reward functions and optimal
policies. DPO is both simple and efficient, optimizing language models directly from human preference data, eliminating the need for an explicit reward model and multi-stage training. Moreover, Wang et al. (2024) discusses how
diverse divergence constraints influence DPO and introduces a generalized approach, namely, _f_ -DPO. Azar et al.
(2023) presents a general objective, Ψ PO, designed for learning from pairwise human preferences, circumventing
current methods’ assumption: _pairwise preferences can be substituted with pointwise rewards_ . This objective analyzes RLHF and DPO behaviors, revealing their potential overfitting issue. The authors further delve into a specific
instance of Ψ PO by setting Ψ as the Identity, aiming to mitigate the overfitting problems. They call this method
IPO and furnish empirical results contrasting IPO with DPO. Hejna et al. (2024) introduces CPL, which utilizes a
regret-based model of preferences that directly provides information about the optimal policy.
Further research could explore why RLHF performs effectively with LLMs and the application of RLHF in
multimodal (Yevgen Chebotar, 2023; OpenAI, 2023b) settings to facilitate the benefits of human-AI collaboration
(Carlson and Demiris, 2010; Wu et al., 2021; Bi et al., 2021). See also Casper et al. (2023b) who offer a survey of
open problems with RLHF.
**Open Discussion** RLHF is frequently applied to the Safety Alignment of LLMs, yet many pressing issues remain
unresolved. For example, how can we balance harmlessness and helpfulness in alignment? Dai et al. (2024b)
attempt to integrate the SafeRL framework, specifically the cost model and reward model, into RLHF to address
the inherent tension between these two indicators. Moreover, even without malicious intent, simply fine-tuning
on benign and commonly used datasets can inadvertently reduce the safety alignment of LLMs, albeit to a lesser
2.4 Scalable Oversight: Path towards Superalignment
extent (Qi et al., 2024) and fine-tuning on benign data is more likely to degrade the model’s safety (He et al., 2024).
These findings suggest that fine-tuning aligned LLMs may introduce new safety risks, even with datasets that are
considered absolutely safe. Generally, language models may exhibit _elasticity_, making them resistant to alignment
efforts (Ji et al., 2024c). This raises a question: _how can we maintain impeccable safety alignment of models, even_
_after further fine-tuning?_
Human preferences can vary among individuals, groups, and societies, leading to divergent perspectives. This
divergence is also evident when collecting preference data from annotators. To address this, Findeis et al. (2024)
proposed a method to extract the underlying constitution governing the generation of a given dataset of preferences.
Similar to Constitutional AI (Bai et al., 2022b), where a preference dataset is generated by an LLM based on a
predefined constitution, _Inverse Constitutional AI_ aims to extract such a constitution that can be used to reconstruct
the preference dataset. This problem can be formulated as an optimization problem:
argmax{agreement( _po,p_ ( _c_ )) s.t. | _c_ | ≤ _n_ } _,_
_c_
where _po_ represents the original preferences, and _p_ ( _c_ ) are the constitutional preferences over a pairwise text corpus
_T_, generated by an LLM _M_ using the constitution _c_ . The set is constrained to a maximum of _n_ natural language
principles that are human-readable. Agreement is defined as the percentage of constitutional preferences _p_ ( _c_ ) that
match the original preferences _po_ . Overall, the elicitation of a constitution can be seen as a compression task,
where a constitution is generated based on a dataset and then used to reconstruct the preferences in the dataset
as accurately as possible. To elicit such a constitution, the authors propose an algorithm that generates principles
capable of explaining the preference data, followed by semantic clustering of these principles. To reduce the size of
the set, they then subsample the principles and evaluate their ability by testing their reproducibility in reconstructing
the preference data. Finally, the principles are filtered based on their relevance to the preference data. This method
can be used to infer the constitution underlying a specific preference dataset and has the potential to identify
underlying biases or reuse the constitution to generate new data, thus enlarging existing datasets or creating new
datasets tailored to individual preferences.

2.3.2 Reinforcement Learning from Human Feedback (RLHF)


Statistical learning usually rely on certain assumptions about data distribution, such as independence and identical
distribution. Consequently, these algorithms fail in some situations, especially under specific distributions (Zhou
et al., 2022). Challenges in elementary systems can be promptly identified through visual inspection (Christiano
et al., 2018; Ngo et al., 2024). As AI systems become more powerful, insufficiently capturing the training signal
or erroneous design of loss functions often leads to catastrophic behaviors (Russell et al., 2015; Hubinger et al.,
2019c; Cotra, 2021) such as deceiving humans by obfuscating discrepancies (Russell, 2019), specification gaming
(Victoria et al., 2020), reward hacking (Brown et al., 2020a), and power-seeking dynamics (Carlsmith, 2022).
From a human perspective, these imply gaps between the optimized objectives of AI systems and the ideal
goals in our minds. Thus, the issue of providing effective oversight in various decision-making becomes pivotal
(Bowman et al., 2022; Li et al., 2023a), often termed as _scalable oversight_ (Amodei et al., 2016) arising from two
practical challenges.
 - The high cost of humans frequently evaluating AI system behavior. For instance, the training process is timeconsuming, and incorporating humans directly into the training loop in real-time would significantly waste
human resources and impede training efficiency (Christiano et al., 2017).
 - The inherent complexity of AI system behaviors makes evaluation difficult, especially on hard-to-comprehend
and high-stakes tasks (Saunders et al., 2022), _e.g._, tasks such as teaching an AI system to summarize books
(Wu et al., 2021), generate complex pieces of code (Pearce et al., 2022), and predict future weather changes
(Bi et al., 2023).
In this context, our primary focus is to present some promising directions that may have not yet been implemented generally for constructing scalable oversight (Amodei et al., 2016; Leike et al., 2018).

2.4 Scalable Oversight: Path towards Superalignment


The RLHF paradigm offers a framework for aligning complex systems (OpenAI, 2023a; Touvron et al., 2023).
However, it encounters obstacles such as the inaccuracy of human evaluations and their associated high costs
2.4 Scalable Oversight: Path towards Superalignment
Figure 5: A tree diagram summarizing the key concepts and literature related to Scalable Oversight. The root node
represents Scalable Oversight whose goal is _ensuring AI systems remain aligned with human intent even as they_
_surpass human capabilities_ . The main branches represent promising frameworks such as Reinforcement Learning
from Feedback (RLxF), Iterated Distillation and Amplification (IDA), Recursive Reward Modeling (RRM), Debate, and Cooperative Inverse Reinforcement Learning (CIRL). Further sub-branches list key works exploring each
framework. This diagram provides an overview of research directions for constructing effective and safe oversight
mechanisms as AI systems grow more complex.
(Christiano et al., 2017; Casper et al., 2023b; Perez et al., 2023). A key limitation is the difficulty in utilizing
RLHF to extend human feedback when creating AI systems with superhuman abilities (Wu et al., 2021). Building
on the RLHF paradigm, we introduce _RLxF_ as a fundamental framework for scalable oversight, aiming to enhance
feedback efficiency and quality and expand human feedback for more complex tasks. This enhances RLHF by
incorporating AI components (Fernandes et al., 2023). The _x_ in _RLxF_ signifies a blend of AI and humans. We
further explore concrete methodologies about _RLxF_ in the subsequent section.
**Reinforcement Learning from AI Feedback (RLAIF)** RLAIF serves as an extension to RLHF. RLAIF extends
the pipeline Bai et al. (2022a) found that LLMs trained via RLHF may avoid sensitive and contentious issues,
potentially reducing models’ overall utility. To address these limitations, Bai et al. (2022b) proposed a training
pipeline that uses feedbacks generated by the LLMs ( _e.g._, GPT-4 or other language models). Following pre-set
criteria, the policy model self-evaluates and revises its responses during _red teaming_ . The initial policy model is
then fine-tuned using the revised responses. Finally, the fine-tuned policy model evaluates the harmlessness of
another language model’s responses (i.e., AI feedback). Similar to RLHF, a reward trained using this feedback to
optimize the policy model. Lee et al. (2023a) compares the performance of models trained with RLAIF and RLHF
on summarization tasks. Their results suggest that models trained with AI feedback aperformed almost identically
to those trained with human feedback, though subtle differences remain. Conversely, Findeis et al. (2024) explored
the inverse problem of CAI: _given a dataset of feedback, how can one extract a constitution that best enables a_
_LLM to reconstruct the original annotations?_ This problem not only converts AI feedback from preferences into a
corresponding constitution but also offers a method for synthesizing new preference data for AI feedback.
**Reinforcement Learning from Human and AI Feedback (RLHAIF)** RLHAIF integrates human and AI models to provide oversight. Wu et al. (2021) explores the feasibility of using AI to assist humans in summarizing
books. This method facilitated human supervision and evaluation of the model performance by decomposing the
book summarization task into subtasks, creating a tree-like structure. Meanwhile, Saunders et al. (2022) explores
the use of AI to assist in human assessment of model efficacy. Their findings suggest that model-generated critiques
help humans identify flaws they might have missed. Bowman et al. (2022) proposes a proof-of-concept experiment
to demonstrate the potential of scalable oversight techniques based on _sandwiching_ (Cotra, 2021). When collaborating with an unreliable LLM, the outcomes reveal that humans significantly surpass the model and themselves.
Perez et al. (2023) employs language models to autonomously generate datasets for evaluating the behavior of
language models of varying scales. The authors produced 154 high-quality datasets validated by humans. These
methods demonstrate the feasibility of using AI assistance to scale up human oversight over complex problems
and various domains.
To some extent, RLAIF and RLHAIF offers a viable alternative for creating a training loop with minimal human intervention, thus reduciing training costs. AI supervision obeying transparent and accessible AI behavior
guidelines may significantly aid in achieving scalable oversight (Bowman et al., 2022).
**Discussion** Efforts are underway to enhance RLHF by replacing pure humans alone (Leike et al., 2018). Given
the multidimensional nature of human feedback, various approaches have been devised to offer focused human
2.4 Scalable Oversight: Path towards Superalignment
judgments informed by specific rules. Examples of such rules encompass considerations like chat fluency (Saunders et al., 2022) and privacy safeguards (Carr, 2023). Saunders et al. (2022) deconstructs the requirements for
quality dialogue into natural language guidelines that an agent should adhere to, asking for evaluations on each
guideline individually. We can attain more efficient rule-conditioned reward models by collecting targeted human assessments and training models on this data. This approach substantially enhances the efficacy of dialogue
agents, rendering them more helpful, accurate, and benign when compared to prompted language models. Carr
(2023) proposes Reinforcement Learning from Privacy Feedback (RLPF), aiming to harmonize the output quality
of language models with safeguarding privacy. The method exploits NLP techniques to conduct real-time privacy
risk assessments of text generated by the models and subsequently adjusts the reinforcement learning feedback signals based on these evaluations. Expressly, if the generated text includes sensitive information, it incurs negative
feedback, whereas high-quality, non-revelatory text receives positive feedback. As the model undergoes training,
it incrementally refines its capabilities, enhancing text quality and minimizing privacy breaches concurrently. This
approach offers a more efficient evaluation of privacy risks by employing established NLP techniques, in contrast
to conventional learning methods, which depend heavily on large-scale manual data annotation.
At their core, the _RLxF_ methods utilize the strategy of decomposing a large problem into smaller sub-problems,
enabling the use of more efficient tools, such as AI and software, for rapid sub-problem resolution. By leveraging
the solutions to these sub-problems, the resolution of the main issue can be expedited. These techniques can
be regarded as elementary instances of IDA; the primary distinction lies in the absence of a continual iterative
process. Nonetheless, evidence suggests they are promising to offer feedback for AI systems that exceed human
performance (Wu et al., 2021). Consequently, these methods can serve as foundational techniques in the training
of more advanced AI systems.


Iterated Distillation and Amplification (IDA) introduces a framework for constructing scalable oversight through
iterative collaboration between humans and AIs (Christiano et al., 2018). The process commences with an initial
agent, denoted as _A_ [0], which mirrors the decision-making of a human, _H_ . _A_ [0] undergoes training using a potent
technique that equips it with near-human-level proficiency (the distillation step); Then, collaborative interaction
between _H_ and multiple _A_ [0] instances leads to the creation of an enhanced agent, _A_ [1] (the amplification step).
The successive process is described [27] in Algorithm 1.
Cotra (2018) distinguishes between broad and narrow definitions within both RL and IRL. Broad RL gives
sparse reward signals to AI systems and allows autonomous exploration and optimization of cumulative future
rewards. This can lead to super-human novel strategies but makes it hard to specify what we care about perfectly.
Narrow RL gives dense feedback rewarding the reasonableness of choices instead of final outcomes. This makes
ML systems more human-like but limits capabilities. Similarly, broad IRL infers deep long-term values from the
full range of human behaviors, while narrow IRL only infers short-term instrumental values. The former is a higher
risk, while the latter is limited in capabilities.
During IDA training, narrow techniques are needed to ensure each agent itself mimics human behaviors. Specifically, narrow RL or IL can be used to train the agent to be as human-like and controllable as possible. Humans can
leverage agents’ computing power and parallelizability to devise more far-sighted, macro strategies. This is essentially an amplification of human intrinsic capabilities. In the next iteration, agents again mimic this strengthened
human-machine system using narrow techniques. This enables a gradual transition from narrow ability to broad
ability while keeping the agents aligned with human values. As iterations increase, the human-machine system
becomes more and more capable, gradually approximating a system that is both highly capable and aligned with
human values, achieving both safety and capability. In other words, Narrow techniques are used to ensure agents
follow human values, while the broadened human strategies in the amplification stage are a way of utilizing the
agents, and do not expand the agents’ own learning goals.
IDA is well illustrated by AlphaZero (Christiano et al., 2018; Nguyen, 2020). The algorithm starts with a simple
policy ( _e.g._, random move selection) and learns from its self-play games, the _amplification_ phase. It then uses
these games as training data to develop better move selection heuristics, the _distillation_ phase. This distillationamplification process can be repeated to create a fast and proficient Go-playing AI. Here, the distinction between
alignment and capability is crucial (Mennen, 2018). An aligned but less capable AI tries to win but may not
succeed against moderate opponents. A capable but poorly aligned AI achieves certain game properties other than
winning. The goal is that AI is capable and aligned, proficient at the game, and aligned with the goal of winning
the game.
The feasibility of IDA has sparked considerable debate (Yudkowsky, 2018). IDA operates under a crucial assumption that _errors won’t continuously accumulate throughout the iterations_ (Leike et al., 2018). Thus, technical
challenges persist during the distillation and amplification step, necessitating sufficiently advanced and safe learn
27We reference the pseudo-code by Cotra (2018) for this description.
2.4 Scalable Oversight: Path towards Superalignment
**Algorithm 1** Iterative Distillation and Amplifcation
1: **procedure** IDA( _H_ )
2: _A_ ← random initialization
3: **repeat**
4: _B_ ← AMPLIFY( _H,A_ )
5: _A_ ← DISTILL( _B_ ) _▷_ Repeat indefinitely
6: **until** False
7: **end procedure**
8: **procedure** DISTILL(overseer)
**return** An AI trained using narrow, robust techniques to perform a task that the overseer already understands how to perform.
9: **end procedure**
10: **procedure** AMPLIFY(human, AI)
_▷_ Interactive process in which human uses many calls to AI to improve on human’s native performance
at the relevant tasks.
11: **end procedure**
ing techniques. Additionally, despite the original authors likening IDA to the training process of AlphaZero (Silver
et al., 2017) and having demonstrated it in toy environments (Christiano et al., 2018), its practicality hinges on ensuring that _H_ can delegate portions of complex tasks to A, analogous to a leader orchestrating a team to accomplish
a project collectively. In practice, Gato (Reed et al., 2022) illustrates key aspects of IDA (Mukobi, 2022) that may
pave the way to AGI. It consolidates the abilities of multiple expert AIs into a singular model, validating that IDA’s
distillation can be achieved using contemporary deep learning. While not fully realized, Gato hints at amplification
potential, harnessing its diverse skills to accelerate the learning of new tasks. However, Gato lacks safe amplification or distillation methods to maintain alignment properties. Crafting alignment-preserving IDA methods suited
for models like Gato remains a crucial direction for AI safety research. In essence, while Gato signifies notable
progress in actualizing IDA, further theoretical advancements are imperative to ensure that the IDA framework
leads to safe AGI.

2.4.2 Iterated Distillation and Amplification


As discussed in Section2.2, reward modeling leverages the idea of using human feedback to train a reward model, which
an agent then pursues. It allows us to disentangle the construction of the system’s objective from evaluating its
behavior (Ibarz et al., 2018). In this manner, the reward model provides insights into the optimization direction of
the AI system. Particularly noteworthy is the ability to finely align the system with human intentions and values,
such as fine-tuning language models to adhere to human instructions (Bai et al., 2022a; Touvron et al., 2023). Also,
reward modeling has proved valuable in advancing AI research (Zhao et al., 2023; Bukharin et al., 2023). Recursive
Reward Modeling (RRM) (Leike et al., 2018) seeks to broaden the application of reward modeling to much more
intricate tasks. The central insight of RRM is the recursive use of already trained agents _At_ −1 to provide feedback
by performing reward learning on an amplified version of itself for the training of successive agents _At_ on more
complex tasks. The _A_ 0 is trained via fundamental reward modeling (learned from pure human feedback). This
approach is not only influenced by human feedback but also by the model’s own assessments of what constitutes a
rewarding outcome. If the assumption that _evaluating outcomes is easier than producing behavior_ holds, then the
iterative process of reward modeling can iteratively achieve higher capacity to oversee more powerful AI systems,
paving the way for extending oversight into more complex domains. This process is detailed in Algorithm 2.
For instance, we aim to train AI _A_ to devise a comprehensive urban plan. Designing a city entails numerous
intricate elements, such as traffic planning, public amenities, and the distribution of residential and commercial
zones. Evaluating a city’s entire design poses a significant challenge since many issues may only become apparent
after extended real-world testing. To aid this process, we may need an agent _B_ specifically for traffic planning.
However, traffic planning in itself is a multifaceted task. Consequently, we further need other agents to assess
aspects such as road width, traffic flow, and the design of public transportation. For every sub-task, such as gauging
road width, we can train an auxiliary agent to verify if safety standards are met, if various modes of transportation
have been considered, and so on. In doing so, we establish an RRM process where each agent is trained with the
help of agents assessing sub-tasks.
This approach resembles the organizational structure of a large corporation (Leike et al., 2018). In the context
of urban planning, the main planning team (the CEO) is responsible for the final design decisions. Their decisions
are informed by recommendations from the traffic team (the department managers), who, in turn, base their recommendations on inputs from the road width team (the managers), and so forth. Each level of decision-making
2.4 Scalable Oversight: Path towards Superalignment
**Algorithm 2** Recursive Reward Modeling
1: Initialize agent _A_ 0 using reward modeling based on user feedback. _▷_ Either preferences or numerical signals.
2: **for** _t_ = 1, 2, ... **do**
3: Use _At_ −1 to assist users in evaluating outcomes.
4: Train agent _At_ based on user-assisted evaluations. _▷_ Objective of _At_ is generally more complex than that
of _At_ −1.
5: **end for**
relies on feedback from the level below it, with each task optimized through reward modeling.
The challenges faced by RRM can be described around the concepts of outer and inner alignment (Hubinger,
2020). Outer alignment revolves around the sufficiency of feedback mechanisms to guarantee that the learned reward model is accurate in the domain perceived by the action model as on distribution. This challenge is contingent
on several factors, including the quality of human feedback, the difficulty of generalization, and the potential for
agent deception. Conversely, inner alignment concentrates on how effectively a human can employ transparency
tools to prevent deceptive or disastrous behaviors in both the reward model and the agent. This hinges on the
effectiveness of the oversight mechanism and the capacity to verify that the reward model isn’t undergoing any
optimization and that the agent remains myopic (Cotra, 2018).
Potential approaches to mitigate these challenges (Leike et al., 2018) include online feedback to correct the reward model during training (Christiano et al., 2017), off-policy feedback to teach about unsafe states (Everitt et al.,
2017), leveraging existing data like videos and text via unsupervised learning or annotating (Baker et al., 2022),
hierarchical feedback on different levels (Bukharin et al., 2023) adversarial training to discover vulnerabilities
(Madry et al., 2018), and uncertainty estimates for soliciting feedback (Hadfield-Menell et al., 2016; MacGlashan
et al., 2017). The strength of RRM is its competitive training approach, which necessitates human feedback instead
of demonstrations, potentially making feedback more reliable and simpler to obtain (Hubinger, 2020). In essence,
the process of RRM can be likened to IDA (Christiano et al., 2018), where reward modeling takes the place of
supervised or imitation learning. Thus, the challenges confronted by RRM closely mirror those encountered in
IDA, particularly in preventing the accumulation of errors. Additionally, reward modeling itself does not necessarily distill a _narrow_ model (Cotra, 2018), which presents challenges in trading off the degree of alignment and
performance.


_Debate_ involves two agents presenting answers and statements to assist human judges in their decision-making
(Irving et al., 2018), as delineated in Algorithm 3. This is a zero-sum debate game where agents try to identify
each other’s shortcomings while striving to gain higher trust from human judges, and it can be a potential approach
to constructing scalable oversight. For example, in the game of Go, human judges might not discern the advantage
side of the single game board itself. However, by observing the game’s process and the eventual outcome, these
judges can more easily deduce that.
The premise of this method relies on a critical assumption: _arguing for truth is generally easier than for false-_
_hood_, granting an advantage to the truth-telling debater. However, this assumption does not hold universally. For
instance, in a complex problem, humans might fail to comprehend the specialized concepts used in the debate.
Additionally, the limited nature of the gradient descent may bring us to an undesirable cyclic pattern ( _i.e._, when
optimizing for one property, such as honesty and highlighting flaws, models often overlook or diminish another)
(Irving et al., 2018).
It’s worth mentioning that with the advancement of LLMs’ capabilities, we can already see practical examples
of debate (Du et al., 2023; Claude, 2023). Challenges may arise for debate in specific real-world scenarios (Irving
et al., 2018). For example, certain questions may be too intricate for human comprehension or too voluminous
to present in their entirety. Similarly, there are instances where an optimal answer to a question is exceedingly
lengthy, envision a response that spans a hundred pages. To navigate these, agents might initially select a response
and, as the debate progresses, reveal sections of either the question or the answer. Irving et al. (2018) conducts
a toy experiment on this process. Meanwhile, we must acknowledge the limit of human time. In scenarios that
necessitate interaction with the environment, such as directing a robot, each action might demand a distinct debate.
It’s not always feasible for humans to judge every debate due to time constraints. In response to this challenge,
we may need to design ML models to predict human feedback. In line with this observation, Khan et al. (2024)
experimented with using smaller, non-expert models as judges in debates between two expert models, both of
which had access to the underlying data and the ability to quote from it. The experiments demonstrated that
these smaller non-expert models were able to achieve higher accuracy when relying on the expert model debates,
though they still underperformed compared to human judges. Additionally, the expert models can be optimized
2.4 Scalable Oversight: Path towards Superalignment
**Algorithm 3** Debate
1: Initialize set of questions _Q_ .
2: Initialize two competing agents.
3: Select a question _q_ ∈ _Q_ . _▷_ Question is shown to both agents.
4: Agents provide their answers _a_ 0 and _a_ 1. The agents generate comment answers in response to _q_ .
5: Initialize debate transcript _T_ as an empty list.
6: **for** turn in predefined number of debate turns **do**
7: Agent makes a debate statement _s_ .
8: Append _s_ to _T_ . _▷_ Agents take turns and statements are saved in the transcript.
9: **end for**
10: Judge observes ( _q,a_ 0 _,a_ 1 _,T_ ) and decides the winning agent.
for persuasiveness, enabling the judges to attain even greater accuracy and more easily identify the truth. The
authors emphasize that debate implementations must be grounded in verifiable evidence to prevent debaters from
fabricating facts. Further work on using weaker models as judges in debates guided by stronger models was
conducted by Kenton et al. (2024). Their experiments focused on tasks involving both information asymmetry
and symmetry between debaters and judges and were extended to include multimodal inputs. The protocols they
applied evaluated the baseline performance of judges without debate protocols, alongside debate and consultancy
protocols. These experiments considered both assigned positions and cases where debaters or consultants could
choose their positions. Experimental results showed that debate consistently outperforms consultancy. Weak
judges struggle to fully leverage debate protocols, and consultancy can significantly reduce the accuracy of judges,
particularly when the consultant advocates for an incorrect solution. Overall, the authors interpret their findings as
only weakly promising for the debate framework. However, these experiments were conducted solely with models
at inference time, and debate protocols may hold greater potential when integrated into training. This is particularly
relevant given that the task of judging a debate can be seen as OOD for models primarily fine-tuned for question
answering.
Another consideration is the convergence of the debate mechanism (Irving et al., 2018). Du et al. (2023) showcases the inherent tendency of the debate framework to eventually converge toward singular responses, even if
accuracy is not guaranteed. Meanwhile, if challenges arise in achieving convergence, we might have to rely on
intuition to gauge the effectiveness of convergence. This implies the requirement of human evaluators’ intervention
and demands a certain level of expertise from these human assessors, posing challenges that must be addressed.
Furthermore, there are many discussions originating from diverse perspectives. Ngo (2021) considers _Debate_
as one type of iterated amplification but more specific to make safety ground in concrete research questions,
and its adversarial framing makes it easier to spot problems. Michaelcohen (2020) expresses concerns regarding
the adverse implications of incentivizing debaters to employ deceptive strategies aimed at swaying the judgment
process. Armstrong (2019); Barnes (2020) expound upon the various issues that can permeate the debate process,
including challenges such as the obfuscated arguments problem, ambiguous responses, and the propagation of
misleading implications. While one may affirm the presence of a sufficiently low probability of any underlying
flaws within the argument, advocating for trustworthiness, the opposing debater may assert the existence of a
sufficiently high probability of identifying a flaw within the argument somewhere, thus advocating for a lack of
trust. Beth Barnes (2020) introduces the concept of _cross-examination_ to incentivize debaters to provide more
informative responses. In this process, debaters have the agency to select a prior claim for scrutiny and obtain a
copy of the opposing debater’s response. The entire exchange is documented, and debaters can present relevant
segments to the judge. The introduction of cross-examination is a robust deterrent against dishonest debaters
exploiting a sweeping narrative, in contrast to their prior arguments, to mislead the judge.
There exists a notable similarity between the debate (Irving et al., 2018), IDA (Christiano et al., 2018), and RRM
(Leike et al., 2018). These approaches can be comprehended in the view of an underlying principle: _evaluation_
_can be simpler than task completion_ [28] . Therefore, harnessing the evaluative capabilities of AI systems can result
in distributions of capacity that are more advantageous for humans. The challenges these methods face, especially
in mitigating the accumulation of errors, are also analogous.


Almost all previous methods consider learning from feedback a process separate from inference and control and
often implicitly consider feedback providers as entities existing outside of the environment – indeed, failure modes
like manipulation (Shevlane et al., 2023) and reward tampering (Everitt et al., 2021) occur exactly when feedback
28Discussions about this can also be found in the literature about these methods.
2.4 Scalable Oversight: Path towards Superalignment
mechanisms that are supposedly outside of the environment become part of it and therefore subject to the AI system’s influence. The framework of Cooperative Inverse Reinforcement Learning (CIRL), however, unifies control
and learning from feedback and models human feedback providers as fellow agents in the same environment. It
approaches the scalable oversight problem not by strengthening oversight but by trying to eliminate the incentives
for AI systems to game oversight, putting humans giving feedback and the AI system in cooperative rather than
adversarial positions (Shah et al., 2020). In the CIRL paradigm, the AI system collaborates with humans to achieve
the human’s true goal rather than unilaterally optimizing for human preferences.
**Motivation and General Idea of CIRL** Many modes of misalignment, including, for example, reward hacking
(Victoria et al., 2020; Skalse et al., 2022), deception (Park et al., 2023b), and manipulation (Shevlane et al., 2023),
are results of the AI system confidently optimizing for misspecified objectives (Pan et al., 2021). During training
and deployment, the specified objective ( _e.g._, the reward function) plays the role of an unchallengeable truth for
the AI system, and human feedback is only respected to the extent specified in the objective, which means that it
could be tampered (Everitt et al., 2021) or manipulated (Shevlane et al., 2023).
CIRL (Hadfield-Menell et al., 2016, 2017b; Shah et al., 2020) attempts to mitigate this problem by (1) having
the AI system explicitly hold uncertainty regarding its reward function, and (2) having humans provide the only
information about what the reward function truly is. This uncertainty gives the AI system a tendency to defer to
humans and a drive to determine what the human truly wants. Concretely speaking, it models the entire task as a
two-player cooperative game, where the _human player H_ and the _robot player R_ share a common reward function
_r_ (·). Importantly, the reward function and reward signals aren’t visible to _R_ (and indeed aren’t explicitly calculated
by the training mechanism) and are only inferred by _R_ from behaviors of _H_ via an IRL-like process (including by
asking and interacting with _H_ ). This game has been called the _CIRL_ (Hadfield-Menell et al., 2016), the _assistance_
_game_ (Fickinger et al., 2020), and the _assistance POMDP_ (Shah et al., 2020).
In short, the AI system has the human’s true objective _r_ (·) as its own goal (despite not knowing values of _r_ (·) with
certainty) and constantly tries to figure _r_ out by observing and interacting with the human. This reduces incentives
for, _e.g._, manipulation since manipulation of human behaviors only serves to pollute an information source and
does not affect _r_ .
**Formulation of CIRL** Hadfield-Menell et al. (2016) characterizes the settings of CIRL (which we denote by _M_ )
by building upon classical multi-agent MDPs, resulting in the definition below of _M_ .
                       -                        _M_ = S _,_ {A [H] _,_ A [R] } _,T,γ,r,_ Θ _,P_ 0
In the equation above, _S_ and {A [H] _,_ A [R] } are the space of world states and actions respectively, _T_ : _S_ ×A [H] ×A [R] →
∆( _S_ ) is the transition function, and _γ_ is the discount rate. Up to here, the definition is identical to that of a
standard multi-agent MDP. The remaining elements, however, introduce the key difference: the reward function is
parameterized, and its parameters can be modeled by a distribution. Θ is the space of values for the parameters
_θ_ ; _r_ : _S_ × A [H] × A [R] × Θ → R is the shared reward function, and _P_ 0 ∈ ∆( _S_ × Θ) is the joint distribution of the
initial state and the reward function’s parameters. This parameterization approach allows _R_ to model explicitly
and reason about its belief over the true reward function. Using techniques from Nayyar et al. (2013), any CIRL
setting can be reduced to an equivalent single-agent POMDP, thus proving the existence of optimal policies that
are relatively tractable (Hadfield-Menell et al., 2016).
**Notable Directions in CIRL Research** Although some have emphasized the importance of _H_ teaching _R_ (Fisac
et al., 2020) actively, works (Shah et al., 2020) have contested the emphasis on game equilibria and joint policies
(including _H_ ’s pedagogic behaviors), and instead focuses on _R_ ’s optimal response to a policy of _H_ ’s, since the
assumption that humans will always act on optimal joint policies is an unrealistic one. More specifically, Shah
                                   et al. (2020) considers the _policy-conditioned belief B_ : Π [R] → ∆ Π [H][�], which specifies _H_ ’s distribution over
policy responses to any of _R_ ’s policies, and the aim is to find _R_ ’s optimal policy given _B_ . Here, _B_ is essentially a
form of human modeling, and one challenge is to obtain a robustly accurate human model as _B_ (Hong et al., 2022).
On another front, Hadfield-Menell et al. (2017b) and He and Dragan (2021) examine the manual specification
of an imperfect reward function as a way for _H_ to convey information about the true reward function. This
includes work on _R_ ’s side ( _i.e._, enabling _R_ to perform inference on the true reward function based on the imperfect
specification) (Hadfield-Menell et al., 2017b) and also work on _H_ ’s side ( _i.e._, developing algorithmic tools to assist
_H_ in making more robust specifications that better convey the true reward function) (He and Dragan, 2021). Aside
from improvements to the game settings, the design of more scalable CIRL algorithms has also been recognized
as a priority.
There has also been work that extends CIRL and assistant games to multi-agent settings (Fickinger et al., 2020)
where there are multiple humans that the robot needs to serve. This corresponds to the _multi/single delegation_
settings in Critch and Krueger (2020), where the varying objectives of humans create a challenge and necessitate
the use of social choice methods.

2.4.5 Cooperative Inverse Reinforcement Learning


Instead of training a model to refuse harmful outputs, the circuit-breaking approach proposed by Zou et al. (2024a)
directly controls the internal representations responsible for generating such outputs. A potential advantage of
this method is that it aims to enhance safety without compromising performance. The core idea is to manage the
model’s ability to generate harmful outputs, rather than eliminating the underlying vulnerabilities. This approach
renders circuit-breaking attack-agnostic, as new adversarial attacks may emerge, but the internal representations
associated with harmful outputs remain constant.
The circuit-breaking method consists of two essential components: the dataset and the loss function. The dataset
includes harmless samples, where internal representations should remain unchanged, while the circuit-breaking
set comprises harmful samples, which require altered internal representations to prevent the generation of harmful
content. By employing a mean squared error loss for retaining representations and cosine similarity for circuitbreaking representations, the model becomes unable to generate harmful content. Changing the loss function
for circuit-breaking can also allow for steering the model’s generation in various directions, such as ending the
output generation or refusing to answer. Additionally, this approach can be extended with a Harmfulness Probing
mechanism to detect and respond to harmful generations.
The authors successfully applied this method to Large Language Models, Multimodal models, and Language
Agents to control function calls.


Scalable Oversight can help humans provide supervision signals to AI systems that are smarter and more complex,
ensuring that the behaviors of super-human-level AI systems align with human intent and values. However, what
if we cannot obtain scalable supervision signals? An example is that for some tasks, evaluation is not necessarily
simpler than generation, making it impossible to utilize task decomposition followed by AI assistance to achieve
scalable oversight.
Recently, a generalization phenomenon called _Weak-to-Strong Generalization_ is verified, the core idea of which
is to use weak supervision signals from a weak model to train a strong model (Burns et al., 2023). Specifically,
the weak model is trained on ground truth and then annotates new data with weak labels for training the strong
model. The results across three settings ( _i.e._ NLP classification, chess puzzles and reward modeling) reflect that
_weak-to-strong generalization_ is a robust phenomenon, yet there is room for further improvement, such as narrowing the gap between a strong model trained with weak labels and ground truth. _Weak-to-Strong Generalization_
provides a valuable analogy for the superalignment problem: how humans can supervise super AI systems as weak
supervisors. The insight behind _weak-to-strong generalization_ is that the strong model can generalize beyond
weak labels instead of merely imitating the behavior of weak models. In other words, the weak model elicits the
strong model’s capability. However, verifying _weak-to-strong generalization_ is challenging if humans don’t know
the ground truth. Nonetheless, _weak-to-strong generalization_ still offers a valuable perspective for solving the
superalignment problem.
The framework for _weak-to-strong generalization_ has been further expanding and integrating with scalable oversight. Empirical results show that weak models can evaluate the correctness of stronger models by assessing the
debate between two expert models (Khan et al., 2024). Additionally, making expert debaters more persuasive improves non-experts’ ability to discern truth in debates, evidencing the effectiveness of aligning models with debate
strategies without ground truth. Some frameworks employ a external amplifier to create an iterated distillation and
amplification process, which presents a potential framework for integrating _weak-to-strong generalization_ techniques with IDA during the training process (Ji et al., 2024a). Moreover, Leike (2023a) proposes several methods
to integrate scalable oversight with _weak-to-strong generalization_ techniques, _e.g._, recursively decomposing tasks
into atomic ones (in line with scalable oversight principles), supervising these atomic tasks, and employing reward
models trained with _weak-to-strong generalization techniques_ using human preference data.


The construction of reliable AI systems is heavily dependent on their ability to adapt to diverse data distributions.
Training data and training environments are often imperfect approximations of real deployment scenarios and may
lack critical elements such as adversarial pressures (Poursaeed et al., 2021) ( _e.g._, Gaussian noise in the context
of supervise learning-based systems (Gilmer et al., 2019) and shadow attack (Ma et al., 2012) in autonomousdriving systems), multi-agent interactions (Critch and Krueger, 2020; Dafoe et al., 2021), complicated tasks that
3.1 The Distribution Shift Challenge
Figure 6: Framework of learning under distribution shift. The main challenges stemming from the distribution
shift are goal misgeneralization and auto-induced distribution shift (Section3.1). In our framework, we also introduce
two kinds of methods to address distribution shift: algorithmic interventions (Section3.2) that steer optimization during
training, and data distribution interventions (Section3.3) that expand the training distribution in a targeted manner by
introducing real-world elements.
human overseers cannot efficiently evaluate (Leike et al., 2018), [29] and reward mechanisms that can be gamed or
manipulated (Krueger et al., 2020). This discrepancy between training distribution and testing distribution (or
environments) is known as _distribution shift_ (Krueger et al., 2020; Thulasidasan et al., 2021).
Therefore, AI systems that are aligned under their training distribution ( _i.e._, pursuing goals that are in line with
human intent) may not uphold their alignment under deployment (or testing) distribution, potentially leading to
serious misalignment issues post-deployment. This potential failure motivates research on the preservation of
alignment properties ( _i.e._, adherence to human intentions and values) across data distributions.
From an alignment perspective, we are more concerned about AI systems pursuing unaligned and harmful goals,
as opposed to incompetence at pursuing goals. Thus, the emphasis on alignment properties means that we focus on
the generalization of _objectives_ across distributions, as opposed to the generalization of _capabilities_ (Di Langosco
et al., 2022; Ngo et al., 2024).
We mainly discuss the preservation of alignment properties when learning under distribution shift in this section.
We start the discussion by introducing the alignment challenges from distribution shift (Section3.1). Subsequently, we
delve into methods for addressing distribution shift, and discuss two approaches in particular: (1) algorithmic
interventions (Section3.2) that steer optimization during the training process, and (2) data distribution interventions
(Section3.3) that expand the training distribution by introducing specific elements into the training process, including
adversarial training (Yoo and Qi, 2021; Bai et al., 2021; Ziegler et al., 2022) and cooperative training (Dafoe et al.,
2021) (Section3.3.2). Our framework for learning under distribution shift is shown in Figure 6.


Before introducing the specific techniques, we initially demonstrate why one of the primary challenges in alignment is learning under distribution shift, and more specifically, the preservation of _alignment properties_ ( _i.e._,
adherence to human intentions and values) under distribution shift. We introduce two alignment challenges concerning the issue of distribution shift, namely goal misgeneralization (Di Langosco et al., 2022) and auto-induced
distribution shift (ADS) (Krueger et al., 2020).
29This could contribute to the emergence of deceptive behaviors (Hubinger et al., 2019a). See the paragraph on _goal misgen-_
_eralization_ in Section3.1 for details.
3.1 The Distribution Shift Challenge
The training of AI systems optimizes for their adherence to the pursuit of the training reward/loss under the
training input distribution. However, this adherence may not generalize to cases where the input distribution
undergoes qualitative changes, _i.e._, distribution shift. These changes include, for example, adversarial pressures
(Poursaeed et al., 2021), multi-agent interactions (Critch and Krueger, 2020), and complicated tasks that human
overseers cannot efficiently evaluate (Di Langosco et al., 2022), and reward mechanisms that can be gamed or
manipulated (Krueger et al., 2020).
It’s worth distinguishing two different failure modes here: goal misgeneralization (Di Langosco et al., 2022), in
which the original and shifted distributions are given, and auto-induced distribution shift (Krueger et al., 2020),
where the AI system alters the data distribution with its own behaviors in pursuit of reward.
**Goal Misgeneralization** This kind of challenge refers to the scenario where AI systems perform perfectly in
the training distribution, but the capabilities learned in training distribution fail to generalize in OOD deployment,
and AI may present the pursuit of goals that are not in accordance with human wishes (Di Langosco et al., 2022).
Goal misgeneralization [30] is to be distinguished from other forms of misgeneralization ( _e.g._, capability misgeneralization) where the agent becomes incompetent in OOD settings; instead, agents with goal misgeneralization
_competently_ pursue an _unwanted_ goal in OOD settings.
A simplistic example is the case of _spurious correlations_ (or _shortcut features_ ) (Geirhos et al., 2019; Di Langosco et al., 2022). For example, in an image classification dataset, green grass is a highly predictive feature for
the label _cow_ . However, it is essential to note that this feature needs to be more consistent and reliable across
various data distributions (Murphy, 2023). Moreover, the causal confusion ( _i.e._, ignorant of the causal structure of
the interaction between the advisor and the environment) in IL can result in goal misgeneralization (De Haan et al.,
2019; Tien et al., 2022).
One major danger from goal misgeneralization lies in the indistinguishability between “optimizing for what
human really wants” and “optimizing for human thumbs-ups”; [31] the latter includes potentially deceiving or manipulating human evaluators (Shevlane et al., 2023) to receive their thumbs-ups. For example, Amodei et al. (2017)
discovered that in a task where a robotic hand is supposed to grasp a small ball, the robotic hand fakes the action
by using parallax in front of the lens to appear as if it has grasped the ball, without actually doing so. This behavior
deceives the human annotator into thinking that the task has been completed.
When an AI system is trained or finetuned with human feedback, it is impossible to distinguish the two goals
since both perform perfectly in training, and it is unclear which one the AI system will learn. In fact, during
training, the human evaluators might be deceived or manipulated, implying that the AI system may be more
strongly incentivized to optimize for human thumbs-ups rather than what the human wants. Current examples of
this phenomenon exist in recommender systems (Kalimeris et al., 2021; Adomavicius et al., 2022), LLMs (Perez
et al., 2023), and RL systems (Amodei et al., 2017).
Finally, one failure mode closely related to goal misgeneralization is the misalignment of _mesa-optimizers_ (Hubinger et al., 2019c), where the ML model with learned model weights performs optimization within itself during
inference (“mesa-optimization”) (Hubinger et al., 2019c; Dai et al., 2023), and the objective of this optimization is
not aligned with the model’s training objective.
**Auto-Induced Distribution Shift (ADS)** While training AI systems, we often consider the strengths and weaknesses of the agents themselves only and overlook the impact that these agents have on the environment. Past
research often assumed that data is independently and identically distributed (Besbes et al., 2022), ignoring the
effect of algorithms on data distribution. However, Krueger et al. (2020) posited that, in reality, agents could influence the environment during the decision-making and execution process, thus altering the distribution of the data
generated by the environment. They referred to this type of issue as ADS. A real-world example is in recommendation systems, where the content selected by the recommendation algorithms might change users’ preferences
and behaviors, leading to a shift in user distribution. The distribution shift, in turn, further affects the output of the
recommendation algorithms (Carroll et al., 2022). As AI systems increasingly impact the world, we also need to
consider the potential further impacts on the data distribution of the entire society after agents are integrated into
human society.
**Superficial Alignment** In recent work, the technique of _Inverse Alignment_ in LLMs was introduced by Ji et al.
(2024c). The study focused on the elasticity of LLMs, which describes their tendency to revert to a state resembling
their original pretrained form after further finetuning post-alignment. This behavior, observed in multiple studies
(Yang et al., 2023b; Zhou et al., 2024), suggests that alignment may not be a permanent change, as models can
easily lose their aligned behavior when subjected to new fine-tuning tasks.
30More examples of goal misgeneralization exist (DeepMind, 2020).
31Here, _human thumbs-ups_ refer to high-reward feedback from human advisors or environment. However, AI systems may
deliberately follow human preferences or deceive to get high rewards from humans, but actually don’t really learn intended
goals ( _i.e._, what human really wants).
3.2 Algorithmic Interventions
Figure 7: A tree diagram summarizing the key concepts and literature related to Algorithmic Interventions. The
root node represents Algorithmic Interventions that aim to steer optimization during the training process. The
main branches represent two main methods, namely cross-distribution aggregation (which aims to minimize risks
on different distributions during training to find a predictor based on the invariant relationship instead of spurious
features) and navigation via mode connectivity (which aims to fine-tune based on mode connectivity to enhance
model generalization performance). Further sub-branches list vital techniques such as Distributionally Robust
Optimization (DRO), Invariant Risk Minimization (IRM), Risk Extrapolation (REx), and Connectivity-based Finetuning (CBFT).
The authors formally define _inverse alignment_ as follows: Given an initial LLM _pθ_ 0, after aligning it on a
dataset _Da_ to produce the aligned model _pθ_ 1, an operation is performed using a much smaller dataset _Db_ (where
| _Db_ | ≪| _Da_ |), resulting in an inverse-aligned model _pθ_ 0 [′] [. The goal is to ensure that] _[ ρ]_ [(] _[p][θ]_ 0 [′] _[,p][θ]_ [0][)][ ≤] _[ϵ]_ [ for some metric]
_ρ_, which measures behavioral or distributional similarity between models. This process is referred to as _inverse_
_alignment_, and the return from _pθ_ 1 to a model similar to _pθ_ 0 is the essence of elasticity. Elasticity is further
formalized as the property of LLM parameters returning to a state close to _pθ_ 0, given an algorithmically simple
inverse transformation _g_ applied to _pθ_ 1, along with a dataset _Db_ . The dataset size constraint | _Db_ | ≪| _Da_ | ensures
that a small amount of data suffices to reverse the effects of alignment, leading to _pθ_ 0 [′] [such that] _[ ρ]_ [(] _[p][θ]_ 0 [′] _[,p][θ]_ [0][)][ ≤] _[ϵ]_ [.]
Theoretical findings show that the normalized compression rate for both the pretraining and fine-tuning datasets
decreases upon additional fine-tuning, but the reduction is more pronounced for the fine-tuning dataset by a factor
of Θ( _k_ ), where _k_ = [|] | _[D]_ _D_ [1] 2 [|] | [,] _[ D]_ [1][ is the pretraining dataset, and] _[ D]_ [2][ is the fine-tuning dataset. This suggests that models]
are more likely to forget the distribution in the fine-tuning dataset while maintaining their pretraining distribution
after exposure to new data. Experimental results further confirm the existence of elasticity in LLMs, showing that
they tend to revert to the pretraining distribution with fewer training samples than required during the alignment
phase. Larger models, and those with more extensive pretraining data, exhibited greater elasticity, highlighting the
limitations of current alignment methods. The theoretical and experimental results indicate that alignment methods
are often superficial, and more robust approaches are necessary to ensure the safety of AI systems, particularly in
the face of _inverse alignment_ . Moreover, these findings underscore the risks associated with open-source models,
as they could potentially be reverted to unsafe states with minimal effort, raising concerns about open-source
policies in large AI companies.


When illustrating the algorithmic intervention methods, we first outline two classes of methods that steer optimization on various distributions during training to relieve distribution shift, namely, cross-distribution aggregation
(Section3.2.1) and navigation via mode connectivity (Section3.2.2).
In the first part, we cover methods ranging from the initial approach of _empirical risk minimization_ (ERM)
(Vapnik, 1991) to _risk extrapolation_ (REx) (Krueger et al., 2021), a method conceived to mitigate issues arising
from models’ dependence on spurious features. In the second part, we introduce _connectivity-based fine-tuning_,
which guides the navigation of the loss landscape during training to encourage convergence upon non-spurious
correlations, and which does so using insights from _mode connectivity_ (Lubana et al., 2023).


One of the main reasons for distribution shift is spurious correlations in the model that are distinct from core objectives (Geirhos et al., 2019). By integrating learning information of different domains (or different distributions)
into the optimization objective, we expect the model to learn truthful information and invariant relationships. In the
following paragraphs, we first introduce ERM as the background and then introduce some methods to directly learn
how to address distribution shift by integrating loss landscapes of different distributions in the training process.
3.2 Algorithmic Interventions
**Empirical Risk Minimization (ERM)** Consider a scenario where a model has been developed to identify objects
by their features effectively. The optimization target can be expressed as:
               R( _w_ ) = L� _y,f_ ( _x,_ _w_ )                 - _d_ P( _x,y_ )
where L( _y,f_ ( _x,_ _w_ )) denotes the loss between data labels _y_ and model outputs _f_ ( _x,_ _w_ ), while P( _x,y_ ) signifies the
target data distribution (Vapnik, 1991).
Nevertheless, a bias often exists between the dataset and the real world, implying that the features learned from
the dataset may not necessarily be the ones we intend for the model to acquire. ERM is a strategy employed
in statistical methods to optimize this bias. It operates on the assumption that, given the inaccessibility of the
real-world target data distribution, the empirical data within the dataset should, ideally, closely approximate this
unknown target distribution (Vapnik, 1991; Zhang et al., 2018b). In this context, the objective function is optimized
and is redefined as:
E( _w_ ) = [1]
_l_
_l_
- L� _yi,f_ ( _xi,_ _w_ )
_i_ =1
where _l_ can be different examples in one training distribution or different training distributions.
Minimizing the objective function above allows the model to learn the invariant relationship in different distributions. Naive ERM makes the naive assumption that the data is sampled from the target data distribution.
However, if a significant discrepancy exists between the source distribution (or training distribution) and the target
distribution, severe generalization issues can still arise (Szegedy et al., 2013).
**Distributionally Robust Optimization (DRO)** Numerous studies posit that the sensitivity to distribution shift
often arises from reliance on _spurious correlations_ or _shortcut features_ unrelated to the core concept (Geirhos
et al., 2019; Hendrycks and Dietterich, 2018). For instance, models may judge based on background features
rather than employing the correct features within the image (Geirhos et al., 2019; Beery et al., 2018). Building
upon the foundations laid in prior research (Ben-Tal et al., 2009; Peters et al., 2015; Krueger et al., 2021), OOD
Generalization can be formulated as follows:
_r_ D [OOD] ( _θ_ ) = max
_e_ ∈D _[r][e]_ [ (] _[θ]_ [)]
This optimization seeks to enhance worst-case performance across a perturbation set, denoted as D, by reducing the
maximum value among the risk function set { _re_ | _e_ ∈D}. In _Distributionally Robustness Optimization (DRO)_ (Duchi
et al., 2021), the perturbation set covers the mixture of different domains’ training distributions, and by minimizing
the above objective function, we expect the model can find the invariant relationship between different training
distributions. However, it should be noted that naively applying DRO to overparameterized neural networks may
lead to suboptimal outcomes (Sagawa et al., 2020). Therefore, combining DRO with increased regularization
techniques such as _l_ 2 penalty (Cortes et al., 2009) or early stopping (Prechelt, 2002) can substantially improve
generalization performance. For more details on DRO, see _e.g._, Rahimian and Mehrotra (2019); Sagawa et al.
(2020); Lin et al. (2022a)
**Invariant Risk Minimization (IRM)** Arjovsky et al. (2019) introduces an innovative learning paradigm to estimate nonlinear, invariant, causal predictors across diverse training environments, thereby facilitating robust OOD
generalization. IRM aims to train a predictive model with solid performance across various environments while
demonstrating reduced susceptibility to relying on spurious features. IRM can be considered an extension of Invariant Causal Prediction (ICP) (Peters et al., 2015), which involves hypothesis testing to identify the direct causal
features that lead to outcomes within each specific environment instead of indirect features. IRM further extends
ICP to scenarios characterized by high-dimensional input data, where variables may lack clear causal significance.
The fundamental idea underlying IRM is that when confronted with many functions capable of achieving low empirical loss, selecting a function that exhibits strong performance across all environments is more likely to get a
predictor based on causal features rather than spurious ones (Murphy, 2023).
**Risk Extrapolation (REx)** The basic form of REx involves robust optimization over a perturbation set of extrapolated domains (MM-REx), with an additional penalty imposed on the variance of training risks (V-REx) (Krueger
et al., 2021). By reducing training risks and increasing the similarity of training risks, REx forces the model to
learn the invariant relationship in different domain distributions.
Amplifying the distributional variations between training domains can diminish risk changes, thereby enforcing
the equality of risks. Taking CMNIST (Arjovsky et al., 2019) as an example, even though establishing a connection
3.2 Algorithmic Interventions
between color and labels is more straightforward than connecting logits and labels, increasing the diversity in color
can disrupt this _spurious correlations_ (or shortcut features) and aid the model in learning the genuine invariant
relationship between logits and labels. Following previous research (Vapnik, 1991; Peters et al., 2017; Krueger
et al., 2021), REx can be formulated as follows: Firstly, the Risk Function can be defined as follows:
                                -                                 _re_ ( _θ_ ) � E( _x,y_ )∼ _Pe_ ( _X,Y_ ) _L_ _fθ_ ( _x_ ) _,y_
where _L_ (·) represents a fixed loss function, and distinct training domains or environments can be formulated as the
_Pe_ ( _X,Y_ ) distribution. Next, the MM-REx term can be modeled as:
_r_ MM−REx( _θ_ ) = (1 − _mλ_ min)max _re_ ( _θ_ ) + _λ_ min
_e_
_n_
_re_ ( _θ_ )
_e_ =1
where _n_ represents the number of distinct distributions or domains, and _λ_ min governs the extent of risk extrapolation. Moving on to the V-REx term, it can be modeled as:
��           - [�]
_r_ V−REx( _θ_ ) = _α_ Var _r_ 1( _θ_ ) _,...,rn_ ( _θ_ ) +
_n_
_re_ ( _θ_ )
_e_ =1
where _α_ ≥ 0 controls the trade-off between risk reduction and enforcing risk equality.
In the MM-REx term, the _λ_ min can set nearly −∞; therefore, the loss of specific domains may be high, meaning that the model may learn the spurious correlations. Minimizing the MM-REx and V-REx can reduce training
risks and increase the similarity of training risks, encouraging the model to learn invariant relationships. Furthermore, REx has shown significant promise in experimental settings (Krueger et al., 2021), particularly in causal
identification, making it a compelling approach for achieving robust generalization.
**Tackle Distribution Shift in LLMs** In the context of LLMs, prior research has shown that RL often exploits
shortcuts to achieve high rewards, overlooking challenging samples (Deng et al., 2023b). This evasion of longtail training samples prevents LLMs from effectively handling distribution shifts in general scenarios, which falls
short of expectations for these models: as universal AI assistants, they should maintain consistent performance
across various domains. Recently, many works have attempted to implement _cross-distribution aggregation_ in
LLMs to address this issue. Zheng et al. (2024) employ RL to learn uniform strategies across diverse data groups
or domains, automatically categorizing data and deliberately maximizing performance variance. This strategy
increases the learning capacity for challenging data and avoids over-optimization of simpler data. Yao et al. (2024)
concentrate on exploiting inter-domain connections. Specifically, they acquire training-domain-specific functions
during the training phase and adjust their weights based on domain relations in the testing phase, achieving robust
OOD generalization.


Following the above discussion about cross-distribution aggregation, in this section, we introduce mode connectivity as the prerequisite content. Then, we primarily discuss the Connectivity-Based Fine-Tuning (CBFT) (Lubana
et al., 2023) method, illustrating how mode connectivity navigates the model to predict based on invariant relationships instead of spurious correlations by changing few parameters.
**Mode Connectivity** Mode connectivity refers to the phenomenon where one can identify a straightforward path
within the loss function space that connects two or more distinct local minima or patterns (Garipov et al., 2018;
Draxler et al., 2018). In line with prior research (Benton et al., 2021; Pittorino et al., 2022; Lubana et al., 2023), a
formal definition can be defined as follows:
The model’s loss on a dataset D is represented as L( _f_ (D; _θ_ )), where _θ_ denotes the optimal parameters of the
model, and _f_ (D; _θ_ ) signifies the model trained on dataset D. We define _θ_ as a minimizer of the loss on this dataset
if L( _f_ (D; _θ_ )) _< ϵ_, where _ϵ_ is a small scalar value.
Minimizers _θ_ 1 and _θ_ 2, achieved through training on dataset D, are considered to be mode-connected if there
exists a continuous path _γ_ from _θ_ 1 to _θ_ 2 such that, as _θ_ 0 varies along this path _γ_, the following condition is
consistently upheld:
             -             -             -             -             -             L _f_ (D; _θ_ 0) ≤ _t_       - L _f_ (D; _θ_ 1) + (1 − _t_ ) · L _f_ (D; _θ_ 2) _,_ ∀ _t_ ∈ [0 _,_ 1] _._
In essence, mode connectivity entails consistently finding a connecting pathway among minimizers in the parameter space, traversing regions of low loss without delving into regions of highly high loss. This implies that even
when making minor adjustments to the model’s parameters within the parameter space, the model’s performance
3.3 Data Distribution Interventions
can remain relatively stable, mitigating significant performance degradation (Garipov et al., 2018). This concept
lays the foundation for designing more effective optimization algorithms, enabling models to share knowledge and
experiences across different tasks, enhancing both model performance and generalization capabilities.
Furthermore, we can define two models as mechanistically similar if they employ the same attributes of inputs for
making predictions. Some research has demonstrated that the absence of linear connectivity implies mechanistic
dissimilarity, suggesting that simple fine-tuning may not suffice to eliminate spurious attributes learned during
the pre-training phase (Lubana et al., 2023; Juneja et al., 2022). However, it is promising to address non-linearly
connected regions through fine-tuning, thereby effectively modifying the model’s mechanisms to resolve the issue
of OOD misgeneralization.
**Connectivity-Based Fine-tuning (CBFT)** As discussed above, recent research has suggested that the absence
of linear connectivity between two models implies a fundamental mechanistic dissimilarity. Lubana et al. (2023)
finds that models tend to develop similar inference mechanisms when trained on similar data. This could be a
significant reason for the emergence of bias in models, such as relying on the background information of images
for classification rather than the objects depicted in the images. If this model mechanism is not adjusted during the
finetuning process, the model may rely on these false attributes. To overcome this problem, they propose a valid
strategy for altering a model’s mechanism, which aims to minimize the following loss:
                         -                          LCBFT = LCE _f_ (DNC; _θ_ ) _,y_ + LB + _K_ [1] [L][I]
where the original training dataset is denoted as D, and we assume that we can obtain a minimal dataset without
spurious attribute _C_, denoted as DNC.
Besides LCE that denotes the cross-entropy loss between model’s prediction _f_ (DNC; _θ_ ) and the ground truth
label _y_, CBFT has two primary objectives: (1) The first objective entails modifying a model’s underlying mechanism by repositioning it within the loss landscape, breaking any linear connection with the current minimizer. This
is accomplished by maximizing LB, referred to as the _barrier loss_ . (2) The second objective involves mitigating
reliance on spurious attributes in the original training dataset. This is achieved by optimizing L _I_, enabling the
discovery of invariant relationships without the need for _C_ . CBFT holds promise for shifting the mechanism from
predicting objectives by spurious features to true features, just changing partial parameters of models.


Besides algorithmic optimization, methods that expand the distribution of training data to include real-world elements can also reduce the discrepancy between training and deployment distributions. In this section, we specifically focus on the introduction of adversarial pressures and multi-agent dynamics.