**Jiaming Ji** **[*,1]** **Tianyi Qiu** **[*,1]** **Boyuan Chen** **[*,1]** **Borong Zhang** **[*,1]** **Hantao Lou** **[1]** **Kaile Wang** **[1]** **Yawen Duan** **[2]**
**Zhonghao He** **[2]** **Lukas Vierling** **[3]** **Donghai Hong** **[1]** **Jiayi Zhou** **[1]** **Zhaowei Zhang** **[1]** **Fanzhi Zeng** **[1]** **Juntao Dai** **[1]**
**Xuehai Pan** **[1]** **Kwan Yee Ng Aidan O’Gara** **[6]** **Hua Xu** **[1]** **Brian Tse Jie Fu** **[5]** **Stephen McAleer** **[3]**
**Yaodong Yang** **[1,]** [ �] **Yizhou Wang** **[1]** **Song-Chun Zhu** **[1]** **Yike Guo** **[5]** **Wen Gao** **[1]**
**1** Peking University **2** University of Cambridge **3** University of Oxford **4** Carnegie Mellon University
**5** Hong Kong University of Science and Technology **6** University of Southern California

Jiaming Ji
Tianyi Qiu
Boyuan Chen
Borong Zhang
Hantao Lou
Kaile Wang
Yawen Duan
Zhonghao He
Jiayi Zhou
Zhaowei Zhang
Fanzhi Zeng
Kwan Yee Ng
Juntao Dai
Xuehai Pan
Aidan O'Gara
Yingshan Lei
Hua Xu
Brian Tse
Jie Fu
Stephen McAleer
Yaodong Yang
Yizhou Wang
Song-Chun Zhu
Yike Guo
Wen Gao


AI alignment aims to make AI systems behave in line with human intentions and values. As AI systems
grow more capable, so do risks from misalignment. To provide a comprehensive and up-to-date overview
of the alignment field, in this survey, we delve into the core concepts, methodology, and practice of alignment. First, we identify four principles as the key objectives of AI alignment: Robustness, Interpretability,
Controllability, and Ethicality ( **RICE** ). Guided by these four principles, we outline the landscape of current alignment research and decompose them into two key components: **forward alignment** and **back-**
**ward alignment** . The former aims to make AI systems aligned via alignment training, while the latter
aims to gain evidence about the systems’ alignment and govern them appropriately to avoid exacerbating
misalignment risks. On forward alignment, we discuss techniques for learning from feedback and learning
under distribution shift. Specifically, we survey traditional preference modeling methods and reinforcement learning from human feedback, and further discuss potential frameworks to reach scalable oversight
for tasks where effective human oversight is hard to obtain. Within learning under distribution shift, we
also cover data distribution interventions such as adversarial training that help expand the distribution of
training data, and algorithmic interventions to combat goal misgeneralization. On backward alignment,
we discuss assurance techniques and governance practices. Specifically, we survey assurance methods
of AI systems throughout their lifecycle, covering safety evaluation, interpretability, and human value
compliance. We discuss current and prospective governance practices adopted by governments, industry
actors, and other third parties, aimed at managing existing and future AI risks.
This survey aims to provide a comprehensive yet beginner-friendly review of alignment research topics.
Based on this, we also release and continually update the website www.alignmentsurvey.com
which features tutorials, collections of papers, blog posts, and other resources.
 - Equal contribution.
 Corresponding author. Contact <pku.alignment@gmail.com>.
 - Version: v4 (updated on Feb 27, 2024). The content of the survey will be continually updated.
CONTENTS


1.1 The Motivation for Alignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1.1 Risks of Misalignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1.2 Causes of Misalignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2 The Scope of Alignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.2.1 The Alignment Cycle: A Framework of Alignment . . . . . . . . . . . . . . . . . . . . . 10
1.2.2 RICE: The Objectives of Alignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.2.3 Discussion on the Boundaries of Alignment . . . . . . . . . . . . . . . . . . . . . . . . . 15


2.1 Feedback Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2 Preference Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.3 Policy Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.3.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.3.2 Reinforcement Learning from Human Feedback (RLHF) . . . . . . . . . . . . . . . . . . 24
2.4 Scalable Oversight: Path towards Superalignment . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.4.1 From RLHF to RL _x_ F . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.4.2 Iterated Distillation and Amplification . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.4.3 Recursive Reward Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.4.4 Debate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.4.5 Cooperative Inverse Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . . . . 31
2.4.6 Circuit Breaking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.4.7 Weak-to-Strong Generalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33


3.1 The Distribution Shift Challenge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.2 Algorithmic Interventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.2.1 Cross-Distribution Aggregation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.2.2 Navigation via Mode Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.3 Data Distribution Interventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.3.1 Adversarial Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.3.2 Cooperative Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40


4.1 Safety Evaluations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4.1.1 Datasets and Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.1.2 Evaluation Targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.1.3 Red Teaming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4.1.4 Safetywashing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.2 Interpretability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.2.1 Intrinsic Interpretability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.2.2 Post Hoc Interpretability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.2.3 Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.3 Human Values Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.3.1 Formulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.3.2 Evaluation Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54


5.1 The Role of AI Governance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
5.2 The Multi-Stakeholder Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
5.3 Open Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
5.3.1 International Governance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
5.3.2 Open-Source Governance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
5.4 Rethinking AI Alignment from a Socio-technical Perspective . . . . . . . . . . . . . . . . . . . . 59
5.4.1 Incorporating Values into AI Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
5.4.2 Alignment Techniques for AI Governance . . . . . . . . . . . . . . . . . . . . . . . . . . 60
CONTENTS


6.1 Key Challenges in the Alignment Cycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
6.2 Key Traits and Future Directions in Alignment Research . . . . . . . . . . . . . . . . . . . . . . 62


Recent advancements have seen the increasing application of capable AI systems in complex domains. For instance, Large Language Models (LLMs) have exhibited improved capabilities in multi-step reasoning (Wei et al.,
2022; Wang et al., 2023c) and cross-task generalization (Brown et al., 2020b; Askell et al., 2021) in real-world
deployment settings, and these abilities are strengthened with increased training time, training data, and parameter
size (Kaplan et al., 2020; Srivastava et al., 2023; Hoffmann et al., 2022). The utilization of Deep Reinforcement
Learning (DRL) for the control of nuclear fusion (Degrave et al., 2022) is another notable example. The increasing
capabilities and deployment in high-stakes domains come with heightened risks. Various undesirable behaviors
exhibited by advanced AI systems ( _e.g._, manipulation (Perez et al., 2023; Carroll et al., 2023; Sharma et al., 2024)
and deception (Park et al., 2023b)) have raised concerns about the hazards from AI systems.
Consequently, these concerns have catalyzed research efforts in _AI alignment_ (Soares and Fallenstein, 2014;
Christian, 2020; Hendrycks et al., 2021b). AI alignment aims to make AI systems behave in line with human
intentions and values (Leike et al., 2018), focusing more on the objectives of AI systems than their capabilities.
Failures of alignment ( _i.e._, misalignment) are among the most salient causes of potential harm from AI. Mechanisms underlying these failures include _reward hacking_ (Pan et al., 2021) and _goal misgeneralization_ (Di Langosco
et al., 2022), which are further amplified by _double edge components_ such as situational awareness (Cotra, 2022),
broadly-scoped goals (Ngo et al., 2024), mesa-optimization objectives (Hubinger et al., 2019c), and access to
increased resources (Shevlane et al., 2023) (Section1.1.2).
Alignment efforts to address these failures focus on accomplishing four key objectives (Section1.2.2): Robustness,
Interpretability, Controllability, and Ethicality ( **RICE** ). Current research and practice on alignment consist of four
areas (Section1.2): Learning from Feedback (Section2), Learning under Distributional Shift (Section3), Assurance (Section4), and Governance (Section5). The four areas and the RICE objectives are not in one-to-one correspondence. Each individual area
often serves more than one alignment objective, and vice versa (see Table 1).
In this survey, we introduce the concept, methodology, and practice of AI alignment and discuss its potential
future directions. [1]


The motivation for alignment is a three-step argument, each step building upon the previous one: (1) Deep learningbased systems (or applications) have an increasingly large impact on society and bring significant risks ; (2) Misalignment represents a significant source of risks; and (3) Alignment research and practice address risks stemming
from misaligned systems ( _e.g._, power-seeking behaviors).


With improved capabilities of AI systems, come increased risks. [2] Some undesirable behaviors of LLMs including
(but not limited to) untruthful answers (Bang et al., 2023), sycophancy (Perez et al., 2023; Sharma et al., 2024),
and deception (Jacob Steinhardt, 2023; Park et al., 2023b) worsen with increased model scale (Perez et al., 2023),
resulting in concerns about advanced AI systems that are hard to control. Moreover, emerging trends such as
_LLM-based agents_ (Xi et al., 2023; Wang et al., 2023b) also raise concerns about the system’s controllability and
ethicality (Chan et al., 2023). Looking further ahead, the development of increasingly competent AI systems opens
up the possibility of realizing Artificial General Intelligence (AGI) in the foreseeable future, _i.e._, systems can match
or surpass human intelligence in all relevant aspects (Bubeck et al., 2023). This could bring extensive opportunities (Manyika et al., 2017), _e.g._, automation (West, 2018), efficiency improvements (Furman and Seamans, 2019),
but also come with serious risks (CAIS, 2023; Critch and Russell, 2023), such as safety concerns (Hendrycks and
Mazeika, 2022), biases and inequalities (Ntoutsi et al., 2020), and large-scale risks from superhuman capabilities (Bengio, 2023). Taking biases as an example, cutting-edge LLMs manifest discernible biases about gender,
sexual identity, and immigrant status among others (Perez et al., 2023), which could reinforce existing inequalities.
Within the large-scale risks from superhuman capabilities, it has been conjectured that global catastrophic risks
( _i.e._, risks of severe harms on a global scale) (Bostrom and Cirkovic, 2011; Hendrycks et al., 2023; Government of
the United Kingdom, 2023) and existential risks ( _i.e._, risks that threaten the destruction of humanity’s long-term
potential) from advanced AI systems are especially worrying. These concerns are elaborated in first-principle
deductive arguments (Ngo, 2020a; Bengio, 2023), evolutionary analysis (Hendrycks, 2023), and concrete scenario
mapping (Christiano, 2019; Kenton et al., 2022). In CAIS (2023), leading AI scientists and other notable figures
stated that _Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale_
_risks such as pandemics and nuclear war_ . The median researcher surveyed by Stein-Perlman et al. (2022) at
NeurIPS 2021 and ICML 2021 reported a 5% chance that the long-run effect of advanced AI on humanity would
1To help beginners interested in this field learn more effectively, we highlight resources about alignment techniques. More
details can be found at www.alignmentsurvey.com/resources
2We discuss and taxonomize the risks that might brought by misaligned AI systems, please see Section1.1.2.
1.1 The Motivation for Alignment
be _extremely bad (e.g., human extinction)_, and 36% of NLP researchers surveyed by Michael et al. (2022) selfreported to believe that _AI could produce catastrophic outcomes in this century, on the level of all-out nuclear_
_war_ . [3] Existential risks from AI also include risks of lock-in, stagnation, and more (Bostrom, 2013; Hendrycks
and Mazeika, 2022), in addition to extinction risks. [4] The UK have hosted the world’s first global AI Safety
Summit, gathering international governments, leading AI companies, civil society groups, and research experts.
Its objectives are to: (1) assess the risks associated with AI, particularly at the cutting edge of its development; (2)
explore how these risks can be mitigated through internationally coordinated efforts. [5] The summit culminated in
the Bletchley Declaration (Summit, 2023), which highlighted the importance of international cooperation on AI
safety. It was signed by representatives from 28 countries and the EU.
Current cutting-edge AI systems have exhibited multiple classes of undesirable or harmful behaviors that may
contrast with human intentions ( _e.g._, power-seeking and manipulation) (Si et al., 2022; Pan et al., 2023a), and similar worries about more advanced systems have also been raised (Critch and Krueger, 2020; CAIS, 2023). [6] These
undesirable or harmful behaviors not compliant with human intentions, known as _misalignment_ of AI systems [7],
can naturally occur even without misuse by malicious actors and represent a significant source of risks from AI,
including safety hazards (Hendrycks et al., 2021b) and potential existential risks (Hendrycks et al., 2023). [8] These
large-scale risks are significant in size due to the non-trivial likelihoods of (1) building superintelligent AI systems,
(2) those AI systems pursuing large-scale goals, (3) those goals are misaligned with human intentions and values,
and (4) this misalignment leads to humans losing control of humanity’s future trajectory (Ngo, 2020a).
Solving the risks brought by misalignment requires the _alignment_ of AI systems to ensure the objectives of
the system are in accordance with human intentions and values, thereby averting unintended and unfavorable
outcomes. More importantly, we expect the alignment techniques to be scaled to harder tasks and significantly
advanced AI systems that are even smarter than humans. A potential solution is _Superalignment_ [9], which aims to
build a roughly human-level automated alignment researcher, thereby using vast amounts of compute to scale up
and iteratively align safe superintelligence (OpenAI, 2023c).


In the above section, we have concluded the motivation for alignment from the perspective of the concern for
AI risks and technical ethics. To offer a deeper understanding of alignment, we aim to further analyze why and
how the misalignment issues occur. We will first give an overview of common failure modes, and then focus
on the mechanism of feedback-induced misalignment, and finally shift our emphasis towards an examination of
misaligned behaviors and dangerous capabilities. In this process, we introduce the concept of _double edge compo-_
_nents_, which offer benefits for enhancing the capabilities of future advanced systems but also bear the potential for
hazardous outcomes.
**Overview of Failure Modes** In order to illustrate the misalignment issue, we give an overview of alignment
failure modes in this section, most of which can be categorized into _reward hacking_ [10] and _goal misgeneralization_ .
The learning process of RL can be deconstructed into two distinct phases: firstly, the creation of an agent
primed for reward optimization, and secondly, the establishment of a reward process that furnishes the agent
with appropriate reward signals. Within the framework of the Markov Reward Process (Marbach and Tsitsiklis,
2001; Puterman, 2014; Sutton and Barto, 2018), the former phase can be seen as the learning process related to
the transition model ( _e.g._, model-based RL agents (Moerland et al., 2023)), or the development of specialized
algorithms. The latter phase can be viewed as the construction of proxy rewards, which aim to approximate the
true rewards derived from sources ( _e.g._, human preferences or environment) (Ng et al., 2000; Leike et al., 2018).
_Reward Hacking_ : In practice, proxy rewards are often easy to optimize and measure, yet they frequently fall short
of capturing the full spectrum of the actual rewards (Pan et al., 2021). This limitation is denoted as _misspecified_
_rewards_ . [11] The pursuit of optimization based on such misspecified rewards may lead to a phenomenon known
3However, survey results may hinge upon the exact wording of the questions and should be taken with caution.
4 _Existential_ and _extinction_ risks are two concepts that are often mixed up. The latter is a subset of the former.
[5Source from https://www.gov.uk/government/topical-events/ai-safety-summit-2023.](https://www.gov.uk/government/topical-events/ai-safety-summit-2023)
6See Section1.1.2 for an introduction to specific misalignment challenges.
7Some of the misaligned behaviors are less risky ( _e.g._, the agent fails to clean the room as you want), however, some of them
are dangerous for systems applied in the high-stakes environment ( _e.g._, the control of nuclear fusion (Degrave et al., 2022))
8It should be noted that misalignment cannot cover all sources of risks brought by Deep learning-based systems and other
factors such as misuse and negligence also contribute to risks on society. See Section1.2.3 for discussing AI safety beyond alignment.
[9For more details on Superalignment, you can refer to https://openai.com/blog/introducing-superalig](https://openai.com/blog/introducing-superalignment)
[nment.](https://openai.com/blog/introducing-superalignment)
10 _Reward hacking_ can also be broadly considered as a kind of _specification gaming_ .
11A similar definition is reward misidentification in which scenario the reward function is only partially identifiable. For
more details on reward misidentification, see _e.g._, Tien et al. (2022); Skalse et al. (2023)
1.1 The Motivation for Alignment
as _reward hacking_, wherein agents may appear highly proficient according to specific metrics but fall short when
evaluated against human standards (Amodei et al., 2016; Everitt et al., 2017). The discrepancy between proxy
rewards and true rewards often manifests as a sharp phase transition in the reward curve (Ibarz et al., 2018).
Furthermore, Skalse et al. (2022) defines the hackability of rewards and provides insights into the fundamental
mechanism of this phase transition, highlighting that the inappropriate simplification of the reward function can be
a key factor contributing to reward hacking.
Misspecified rewards often occur due to a neglect of severe criteria for the outcomes, thus making specification
too broad and potentially easily hacked (Victoria et al., 2020). More than poor reward design (Ng et al., 1999), the
choice of training environment and simulator with bugs (Code Bullet, 2019) can both lead to AI systems failing to
satisfy intended objectives. These problems stem from task specification, broadly defined as _specification gaming_,
which refers to AI systems exploiting loopholes in the task specification without achieving intended outcomes. [12]
(Victoria et al., 2020)
_Reward tampering_ can be considered a special case of reward hacking (Everitt et al., 2021; Skalse et al., 2022),
referring to AI systems corrupting the reward signals generation process (Ring and Orseau, 2011). Everitt et al.
(2021) delves into the subproblems encountered by RL agents: (1) _tampering of reward function_, where the agent
inappropriately interferes with the reward function itself, and (2) _tampering of reward function input_, which entails
corruption within the process responsible for translating environmental states into inputs for the reward function.
When the reward function is formulated through feedback from human supervisors, models can directly influence
the provision of feedback ( _e.g._, AI systems intentionally generate challenging responses for humans to comprehend
and judge, leading to feedback collapse) (Leike et al., 2018). Since task specification has its physical instantiation
( _e.g._, memory registers storing the reward signals), the AI systems deployed in the real world have the potential
to practice manipulation behaviors, resulting in more hazardous outcomes (Victoria et al., 2020). Moreover, it has
been demonstrated that easily discovered reward tampering behaviors can generalize to sophisticated specification
gaming, which cannot be prevented by using 3H environments or preference reward modeling training (anthropic,
2024).
_Goal Misgeneralization: Goal misgeneralization_ is another failure mode, wherein the agent actively pursues
objectives distinct from the training objectives in deployment while retaining the capabilities it acquired during
training (Di Langosco et al., 2022). [13] For instance, in _CoinRun_ games, the agent frequently prefers reaching
the end of a level, often neglecting relocated coins during testing scenarios. Di Langosco et al. (2022) draw
attention to the fundamental disparity between capability generalization and goal generalization, emphasizing how
the inductive biases inherent in the model and its training algorithm may inadvertently prime the model to learn a
proxy objective that diverges from the intended initial objective when faced with the testing distribution. It implies
that even with perfect reward specification, goal misgeneralization can occur when faced with distribution shifts
(Amodei et al., 2016). It should be noted that goal misgeneralization can occur in any learning system, not limited
to RL since the core feature is the pursuit of unintended goals (Shah and Varma, 2022). Moreover, it might be
more dangerous if advanced AI systems escape control and leverage their capabilities to bring about undesirable
states (Zhuang and Hadfield-Menell, 2020).
**Feedback-Induced Misalignment** With the proliferation of advanced AI systems, the challenges related to reward hacking and goal misgeneralization have become increasingly pronounced in open-ended scenarios (Paulus
et al., 2018; Knox et al., 2023). Gao et al. (2023) underscores that more capable agents tend to exploit misspecified
rewards to a greater extent. While many current AI systems are primarily driven by self-supervision, it’s worth
noting that a substantial portion relies on feedback rewards derived from human advisors (Bai et al., 2022a), allowing us to introduce the mechanism of feedback-induced misalignment. The misalignment issues are particularly
pressing in open-ended scenarios, and we can attribute them to two primary factors:
 - **Limitations of Human Feedback** . During the training of LLMs, inconsistencies can arise from human data
annotators ( _e.g._, the varied cultural backgrounds of these annotators can introduce implicit biases (Peng et al.,
2022)) (OpenAI, 2023a). Moreover, they might even introduce biases deliberately, leading to untruthful preference data (Casper et al., 2023b). For complex tasks that are hard for humans to evaluate ( _e.g._, the value of
game state), these challenges [14] become even more salient (Irving et al., 2018).
 - **Limitations of Reward Modeling** . Training reward models using comparison feedback can pose significant
challenges in accurately capturing human values. For example, these models may unconsciously learn suboptimal or incomplete objectives, resulting in reward hacking (Zhuang and Hadfield-Menell, 2020; Skalse et al.,
12For more instances about specification gaming, please see Krakovna (2020)
13More discussion about Goal Misgeneralization can be found in Section3.1.
14As AI systems are deployed into more complex tasks, these difficulties amplify, necessitating novel solutions such as
_scalable oversight_ (Cotra, 2018).
1.1 The Motivation for Alignment
Figure 1: Dangerous Capabilities. Advanced AI systems would be incentivized to seek power because power will
help them achieve their given objectives. Powerful AI systems might hack computer systems, manipulate humans,
control and develop weaponry, and perform ethical violations while avoiding a shutdown. Original copyright
belongs to wiki (wikipedia, 2023), based on which we have made further adjustments. We will further discuss
these issues in Section1.1.2.
2022). Meanwhile, using a single reward model may struggle to capture and specify the values of a diverse
human society (Casper et al., 2023b).
Additionally, Huang et al. (2023); Andreas (2022); Kim et al. (2024) demonstrate that advanced AI systems
exhibit patterns of goal pursuit and multi-step reasoning capability, which further aggravate the situation if the
reward is not well-defined (Ngo et al., 2024; Yang et al., 2023a).
_Discussion:_ It can be challenging to distinguish goal misgeneralization from reward hacking in specific cases.
For instance (Shah and Varma, 2022), LLMs are trained to generate _harmless, honest, and helpful_ outputs, but
LLMs may occasionally produce harmful outputs in detail, which seemingly receive low rewards in testing distribution (which could be seen as goal misgeneralization). However, in cases where labelers are incentivized to
assign high rewards to responses deemed more helpful during the labeling process, the scenarios above [15] actually
receive high rewards and represent a form of specification gaming (or reward hacking). The distinction between
these two scenarios can be vague at times.
More research is needed to analyze the failure modes, gain a deeper understanding of reward hacking, and develop effective methods for detecting and mitigating goal misgeneralization to address the challenges of misaligned
advanced AI systems.
**Misaligned Behaviors and Outcomes** Drawing from the misalignment mechanism, optimizing for a non-robust
proxy may result in misaligned behaviors, potentially leading to even more catastrophic outcomes. This section
delves into a detailed exposition of specific **misaligned behaviors** (•) and introduces what we term **double edge**
**components** (+). These components are designed to enhance the capability of AI systems in handling real-world
settings but also potentially exacerbate misalignment issues. It should be noted that some of these **double edge**
**components** (+) remain speculative. Nevertheless, it is imperative to discuss their potential impact before it is
too late, as the transition from controlled to uncontrolled advanced AI systems may be just one step away (Ngo,
2020b). With increased model scale, a class of **dangerous capabilities** (*) (Shevlane et al., 2023) could also
emerge. The **dangerous capabilities** (*) are concrete tasks the AI system could carry out; they may not necessarily
be misaligned in themselves but are instrumental to actualizing extreme risks.
We first introduce the **double edge components** (+) and analyze how they act on AI systems. Then, we illustrate
the **misaligned behaviors** (•) and **dangerous capabilities** (*) to show specific misalignment issues and provide
directions for future alignment evaluation research.
15Harmful but detailed responses
1.1 The Motivation for Alignment
+ **Situational Awareness** . AI systems may gain the ability to effectively acquire and use knowledge about its
status, its position in the broader environment, its avenues for influencing this environment, and the potential
reactions of the world (including humans) to its actions (Cotra, 2022). Similar behaviors have been observed
in LLMs (Jonas DeGrave, 2022; Evan Hubinger, 2023). Knowing the situation can help the model better
understand human intent, finish tasks within its ability, and search for outlier help if needed. However, such
knowledge also paves the way for advanced methods of reward hacking, heightened deception/manipulation
skills, and an increased propensity to chase instrumental subgoals (Ngo et al., 2024). Consequently, it should
be given priority when evaluating potentially hazardous capabilities in AI models, alongside eight other key
competencies (Shevlane et al., 2023). A highly relevant discussion is whether language models possess _world_
_models_ (LeCun, 2022; Li et al., 2022b).
+ **Broadly-Scoped Goals** . Advanced AI systems are expected to develop objectives that span long timeframes,
deal with complex tasks, and operate in open-ended settings (Ngo et al., 2024). Engaging in broadly-scoped
planning can empower AI systems to generalize better on the OOD settings and serve as valuable assistants
in realms such as human healthcare. However, it can also bring about the risk of encouraging manipulating
behaviors ( _e.g._, AI systems may take some _bad_ actions to achieve human happiness, such as persuading
them to do high-pressure jobs [16] (Jacob Steinhardt, 2023)). Intuitively, one approach to mitigate this risk is
to confine the optimizable objectives to short-sighted ones, such as predicting only the next word, thereby
preventing over-ambitious planning, but such approaches limit systems’ utility and may fail; for instance,
source text data ( _e.g._, fiction) can help AI systems understand the intent and belief of the roles, and thus
longer-term goal-directed behavior can be elicited (Andreas, 2022). Additionally, techniques such as RLbased fine-tuning (Christiano et al., 2017; Ouyang et al., 2022) or the application of chain-of-thought prompts
(Wei et al., 2022) can enable models to adapt their acquired knowledge about planning to pave the way for
broadly-scoped planning objectives (Jacob Steinhardt, 2023).
+ **Mesa-Optimization Objectives** . The learned policy may pursue inside objectives _when the learned policy_
_itself functions as an optimizer_ ( _i.e._, _mesa-optimizer_ ). However, this optimizer’s objectives may not align
with the objectives specified by the training signals, and optimization for these misaligned goals may lead
to systems out of control (Hubinger et al., 2019c). Freeman et al. (2019); Wijmans et al. (2023) indicate
that AI systems may possess implicit goal-directed planning and manifest emergent capabilities during the
generalization phase.
+ **Access to Increased Resources** . Future AI systems may gain access to websites and engage in real-world
actions, potentially yielding a more substantial impact on the world (Nakano et al., 2021). They may disseminate false information, deceive users, disrupt network security, and, in more dire scenarios, be compromised
by malicious actors for ill purposes. Moreover, their increased access to data and resources can facilitate
_self-proliferation_, posing existential risks (Shevlane et al., 2023).
  - **Power-Seeking Behaviors** . AI systems may exhibit behaviors that attempt to gain control over resources
and humans and then exert that control to achieve its assigned goal (Carlsmith, 2022). The intuitive reason
why such behaviors may occur is the observation that for almost any optimization objective ( _e.g._, investment returns), the optimal policy to maximize that quantity would involve power-seeking behaviors ( _e.g._,
manipulating the market), assuming the absence of solid safety and morality constraints. Omohundro (2008);
Bostrom (2012) have argued that power-seeking is an _instrumental subgoal_ which is instrumentally helpful
for a wide range of objectives and may, therefore, be favored by AI systems. Turner et al. (2021) also proved
that in MDPs that satisfy some standard assumptions, the optimal policies tend to be power-seeking. Perez
et al. (2023) prompt LLMs to test their tendency to suggest power-seeking behaviors, find significant levels
of such tendencies, and show that RLHF strengthens them. This also holds for other instrumental subgoals
such as self-preservation (Bostrom, 2012; Shevlane et al., 2023). Another notable line of research is _side-_
_effect avoidance_, which aims to address power-seeking behaviors by penalizing agentic systems for having
too much influence over the environment. It covers RL systems (Eysenbach et al., 2018; Turner et al., 2020)
and symbolic planning systems (Klassen et al., 2022).
  - **Untruthful Output** . AI systems such as LLMs can produce either unintentionally or deliberately inaccurate
output. Such untruthful output may diverge from established resources or lack verifiability, commonly referred
to as _hallucination_ (Bang et al., 2023; Zhao et al., 2023). More concerning is the phenomenon wherein LLMs
16This behavior is due to models’ over-optimization for broadly-scoped goals and this over-optimization is hard to perceive
by humans
1.1 The Motivation for Alignment
may selectively provide erroneous responses to users who exhibit lower levels of education [17] (Perez et al.,
2023). The behavior (also known as sycophancy) appears emergently at scale (Ajeya Cotra, 2021; Perez et al.,
2023) and untruthful output has the potential to engender deception, especially as advanced AI systems gain
greater access to online resources and websites (Jacob Steinhardt, 2023).
  - **Deceptive Alignment & Manipulation** . Manipulation & Deceptive Alignment is a class of behaviors that
exploit the incompetence of human evaluators or users (Hubinger et al., 2019a; Carranza et al., 2023) and
even manipulate the training process through _gradient hacking_ (Richard Ngo, 2022). These behaviors can
potentially make detecting and addressing misaligned behaviors much harder.
_Deceptive Alignment_ : Misaligned AI systems may deliberately mislead their human supervisors instead of
adhering to the intended task. Such deceptive behavior has already manifested in AI systems that employ
evolutionary algorithms (Wilke et al., 2001; Hendrycks et al., 2021b). In these cases, agents evolved the
capacity to differentiate between their evaluation and training environments. They adopted a strategic pessimistic response approach during the evaluation process, intentionally reducing their reproduction rate within
a scheduling program (Lehman et al., 2020). Furthermore, AI systems may engage in intentional behaviors
that superficially align with the reward signal, aiming to maximize rewards from human supervisors (Ouyang
et al., 2022; Lang et al., 2024). It is noteworthy that current large language models occasionally generate
inaccurate or suboptimal responses despite having the capacity to provide more accurate answers (Lin et al.,
2022c; Chen et al., 2021). These instances of deceptive behavior present significant challenges. They undermine the ability of human advisors to offer reliable feedback (as humans cannot make sure whether the outputs
of the AI models are truthful and faithful). Moreover, such deceptive behaviors can propagate false beliefs and
misinformation, contaminating online information sources (Hendrycks et al., 2021b; Chen and Shu, 2024).
_Manipulation_ : Advanced AI systems can effectively influence individuals’ beliefs, even when these beliefs are
not aligned with the truth (Shevlane et al., 2023). These systems can produce deceptive or inaccurate output
or even deceive human advisors to attain deceptive alignment. Such systems can even persuade individuals to
take actions that may lead to hazardous outcomes (OpenAI, 2023a).
Early-stage indications of such behaviors are present in LLMs, [18] recommender systems (where the system
influences the users’ preferences) (Kalimeris et al., 2021; Krueger et al., 2020; Adomavicius et al., 2022), and
RL agents (where agents trained from human feedback adopt policies to trick human evaluators) (Amodei
et al., 2017). Also, current LLMs already possess the capability needed for deception. In Spitale et al.
(2023), it has been found that GPT-3 is super-human capable of producing convincing disinformation. Given
all these early-stage indications, it is plausible that more advanced AI systems may exhibit more serious
deceptive/manipulative behaviors.
  - **Collectively Harmful Behaviors** . AI systems have the potential to take actions that are seemingly benign
in isolation but become problematic in multi-agent or societal contexts. Classical game theory offers simplistic models for understanding these behaviors. For instance, Phelps and Russell (2023) evaluates GPT3.5’s performance in the iterated prisoner’s dilemma and other social dilemmas, revealing limitations in the
model’s cooperative capabilities. Perolat et al. (2017) executes a parallel analysis focused on common-pool
resource allocation. To mitigate such challenges, the emergent field of Cooperative AI (Dafoe et al., 2020,
2021) has been advancing as an active research frontier. However, beyond studies grounded in simplified
game-theoretical frameworks, there is a pressing need for research in more realistic, socially complex settings
(Singh, 2014). In these environments, agents are numerous and diverse, encompassing AI systems and human
actors (Critch and Krueger, 2020). Furthermore, the complexity of these settings is amplified by the presence
of unique tools for modulating AI behavior, such as social institutions and norms (Singh, 2014). [19]
  - **Violation of Ethics** . Unethical behaviors in AI systems pertain to actions that counteract the common good
or breach moral standards – such as those causing harm to others. These adverse behaviors often stem from
omitting essential human values during the AI system’s design or introducing unsuitable or obsolete values
into the system (Kenward and Sinclair, 2021). Moreover, recent works have found that current LLMs can
infringe upon personal privacy by inferring personal attributes from the context provided during inference,
which may violate human rights (Mireshghallah et al., 2024; Staab et al., 2024). Research efforts addressing
these shortcomings span the domain of _machine ethics_ (Yu et al., 2018; Winfield et al., 2019; Tolmeijer et al.,
17Such behaviors bare termed _sandbagging_ (Perez et al., 2023). They may have been learned from web text during pretraining, which suggests that supervised learning can also bring about deceptive behaviors if those behaviors are present in
training data.
18Namely, the _untruthful output_ that we discuss above.
19We cover cooperative AI research in Section3.3.2 and Section4.3.1.
1.2 The Scope of Alignment


|Col1|Col2|
|---|---|
|_(Alignment_|_ Training)_|
|Col1|Col2|
|---|---|
|_(Alignment Refi_|_ nement)_|
Figure 2: The Alignment Cycle. (1) **Forward Alignment** (alignment training) produces _trained systems_ based
on _alignment requirements_ ; (2) **Backward Alignment** (alignment refinement) ensures the practical alignment of
_trained systems_ and revises _alignment requirements_ ; (3) The cycle is repeated until reaching a sufficient level of
alignment. Notably, although Backward Alignment has the end goal of ensuring the practical alignment of _trained_
_systems_, it is carried out all throughout the system’s lifecycle in service of this goal, including before, during, after
training, and also after deployment (Shevlane et al., 2023; Koessler and Schuett, 2023; Schuett et al., 2023).
2020) and delve into pivotal questions, _e.g._, _whom should AI align with?_ (Santurkar et al., 2023), among other
concerns.
 - **Dangerous Capabilities** . Figure 1 outlines the dangerous capabilities that advanced AI systems might have.
As AI systems are deployed in the real world, they may pose risks to society in many ways ( _e.g._, hack computer
systems, escape containment, and even violate ethics). They may hide unwanted behaviors, fool human
supervisors, and seek more resources to become more powerful. Moreover, **double edge components** (+)
may intensify the danger and lead to more hazardous outcomes, even resulting in existential risks (Bostrom,
2013).


In this section, we focus on illustrating the scope of AI alignment: we constructed the alignment process as an
_alignment cycle_ and decomposed it into _Forward Alignment Process_ and _Backward Alignment Process_ [20] (Section1.2.1).
Specifically, we discuss the role of _human values_ in alignment (Section1.2.3) and further analyze AI safety problems
beyond alignment (Section1.2.3).


We decompose alignment into **Forward Alignment** (alignment training) (Section2, Section3) and **Backward Alignment**
(alignment refinement) (Section4, Section5). Forward Alignment aims to produce trained systems that follow alignment re
20From this point and throughout the survey, for convenience, we refer to “Forward Alignment” and “Backward Alignment”.
1.2 The Scope of Alignment
quirements. [21] We decompose this task into Learning from Feedback (Section2) and Learning under Distribution Shift
(Section3). Backward Alignment aims to ensure the practical alignment of the trained systems by performing evaluations
in both simplistic and realistic environments and setting up regulatory guardrails to handle real-world complexities, _i.e._, Assurance (Section4). It also covers the creation and enforcement of rules that ensure the safe development and
deployment of AI systems, _i.e._, Governance (Section5). At the same time, backward alignment updates the alignment
requirements based on the evaluation and monitoring of the systems, both pre-deployment and post-deployment.
These updated requirements then inform the next round of alignment training.
The two phases, forward and backward alignment, thus form a cycle where each phase produces or updates
the input of the next phase (see Figure 2). This cycle, what we call _the alignment cycle_, is repeated to produce
increasingly aligned AI systems. We see alignment as a dynamic process in which all standards and practices
should be continually assessed and updated. Notably, Backward Alignment (including the Assurance of alignment
in AI systems and the Governance of AI systems) efforts occur throughout the entire alignment cycle, as opposed to
only after training. As argued in Shevlane et al. (2023); Koessler and Schuett (2023), alignment and risk evaluations
should occur in every stage of the system’s lifecycle, including before, during, after training, and post-deployment.
Similarly, regulatory measures for every phase of the system’s lifecycle have been proposed and discussed (Schuett
et al., 2023; Anderljung et al., 2023).
The survey is structured around four core pillars: Learning from Feedback (Section2) and Learning under Distribution
Shift (Section3), which constitute the components of Forward Alignment; and Assurance (Section4) and Governance (Section5)
which form the elements of Backward Alignment. The subsequent paragraphs provide a concise introduction to
each pillar, clarifying how they synergistically contribute to a comprehensive framework for AI alignment.
 - **Learning from Feedback** (Section2) _Learning from feedback_ concerns the question of _during alignment training,_
_how do we provide and use feedback to behaviors of the trained AI system?_ It takes an input-behavior pair as
given and only concerns how to provide and use feedback on this pair. [22] In the context of LLMs, a typical solution is reinforcement learning from human feedback (RLHF) (Christiano et al., 2017; Bai et al., 2022a), where
human evaluators provide feedback by comparing alternative answers from the chat model, and the feedback
is used via Reinforcement Learning (RL) against a trained reward model. Despite its popularity, RLHF faces
many challenges (Pandey et al., 2022; Casper et al., 2023b; Tien et al., 2022), overcoming which has been a
primary objective of alignment research (Bowman et al., 2022), and is one primary focus of the section. An
outstanding challenge here is _scalable oversight_ (Section2.4), _i.e._, providing high-quality feedback on super-human
capable AI systems that operate in complex situations beyond the grasp of human evaluators, where the behaviors of AI systems may not be easily comprehended and evaluated by humans (Bowman et al., 2022). Another
challenge is the problem of providing feedback on ethicality, which is approached by the direction of machine
ethics (Anderson and Anderson, 2011; Tolmeijer et al., 2020). On the ethics front, misalignment could also
stem from neglecting critical dimensions of variance in values, such as underrepresenting certain demographic
groups in feedback data (Santurkar et al., 2023). There have also been work combining feedback mechanisms
with _social choice_ methods to produce a more rational and equitable aggregation of preferences (Collective
Intelligence Project, 2023) (see Section1.2.3).
 - **Learning under Distribution Shift** (Section3) In contrast to learning from feedback, which holds input fixed, this
pillar focuses specifically on the cases where the distribution of input changes, _i.e._, where distribution shift
occurs (Krueger et al., 2020; Thulasidasan et al., 2021; Hendrycks et al., 2021a). More specifically, it focuses
on the preservation of _alignment properties_ ( _i.e._, adherence to human intentions and values) under distribution shift, as opposed to that of model capabilities. In other words, it asks how we can ensure an AI system
well-aligned on the training distribution will also be well-aligned when deployed in the real world. One
challenge related to distribution shift is _goal misgeneralization_, where, under the training distribution, the intended objective for the AI system ( _e.g._, following human’s real intentions) is indistinguishable from other
unaligned objectives ( _e.g._, gaining human approval regardless of means). The system learns the latter, which
leads to unaligned behaviors in deployment distribution (Di Langosco et al., 2022). Another related challenge
is _auto-induced distribution shift_ (ADS), where an AI system changes its input distribution to maximize reward (Krueger et al., 2020; Perdomo et al., 2020). An example would be a recommender system shaping
user preferences (Kalimeris et al., 2021; Adomavicius et al., 2022). Both goal misgeneralization and ADS are
closely linked to deceptive behaviors (Park et al., 2023b) and manipulative behaviors (Shevlane et al., 2023) in
AI systems, potentially serving as their causes. Interventions that address distribution shift include _algorith-_
21Here, _alignment requirements_ refer to an operationalized specification of the alignment properties that are desired of the
AI systems, including, for example, which concrete forms of robustness/interpretability/controllability/ethicality we require, in
what specific settings we require them, and how they could be measured.
22Here, _behavior_ is broadly defined also to include the system’s internal reasoning, which can be examined via interpretability
tools (see Section4.2).
1.2 The Scope of Alignment
_mic interventions_ (Section3.2), which changes the training process to improve reliability under other distributions,
and _data distribution interventions_ (Section3.3) which expands the training distribution to reduce the discrepancy
between training and deployment distributions. The former includes methods like Risk Extrapolation (REx)
(Krueger et al., 2021) and Connectivity-based Fine-tuning (CBFT) (Lubana et al., 2023). The latter includes
adversarial training (Section3.3.1) (Song et al., 2018b; Bai et al., 2021) which augments training input distribution
with adversarial inputs, and cooperative training (Section3.3.2) (Dafoe et al., 2020, 2021) which aims to address the
distribution gap between single-agent and multi-agent settings. [23]
 - **Assurance** (Section4) Once an AI system has undergone forward alignment, we still need to gain confidence about
its alignment before deploying it (Government of the United Kingdom, 2021; Anderljung et al., 2023). Such
is the role of _assurance_ : assessing the alignment of trained AI systems. Methodologies of assurance include
safety evaluations (Perez et al., 2023; Shevlane et al., 2023) (Section4.1) and more advanced methods such as interpretability techniques (Olah et al., 2018) (Section4.2) and red teaming (Perez et al., 2022) (Section4.1.3). The scope of
assurance also encompasses the verification of system’s alignment with human values, including formal theories focused on provable cooperativeness (Dafoe et al., 2021) and ethicality (Anderson and Anderson, 2011;
Tolmeijer et al., 2020), and also a wide range of empirical and experimental methods (Section4.3). Assurance takes
place throughout the lifecycle of AI systems, including before, during, after training, and post-deployment, as
opposed to only after training (Shevlane et al., 2023; Koessler and Schuett, 2023). [24]
 - **Governance** (Section5) Assurance alone cannot provide full confidence about a system’s practical alignment since
it does not account for real-world complexities. This necessitates governance efforts of AI systems that focus
on their alignment and safety and cover the entire lifecycle of the systems (Section5.1). We discuss the multistakeholder approach of AI governance, including the governmental regulations (Anderljung et al., 2023), the
lab self-governance (Schuett et al., 2023), and the third-party practice, such as auditing (Shevlane et al., 2023;
Koessler and Schuett, 2023) (Section5.2). We also highlight several open problems in AI governance, including
the pressing challenge of open-source governance (the governance of open-source models and the question
of whether to open-source highly capable models) (Seger et al., 2023), and the importance of international
coordination in AI governance (Ho et al., 2023) (Section5.3). In addition to policy research, we also cover key
actions from both the public and the private sector.
**Comparison with Inner/Outer Decomposition** Our _alignment cycle_ framework (see Figure 2) decomposes
alignment into four pillars: Learning from Feedback, Learning under Distribution Shift, Assurance and Governance organized into a circular process. The design principle for this framework is three-fold: Practical (making
sure pillars directly correspond to specific practices in specific stages in the system’s lifecycle), Concrete (pointing
to specific research directions as opposed to general themes), and Up-To-Date (accommodating and emphasizing
latest developments in the alignment field). Recently, the decomposition of alignment into _outer alignment_ and
_inner alignment_ has become popular in the alignment literature (Hubinger et al., 2019b). Outer alignment refers to
the wishes of designers in accordance with the actual task specification ( _e.g._, goal & reward) used to build AI systems. And inner alignment is the consistency between task specification and the specification that the AI systems
behaviors reflect (Krakovna, 2022). However, many criticisms have also been made about this characterization,
including that it is ambiguous and is understood by different people to mean different things (Perry, 2020) and that
it creates unnecessary difficulties by carving out problems that are not necessary conditions for success (Turner,
2022). Some have tried to remove the ambiguity by pinning down the specific causes of inner/outer misalignment
and proposed, for example, _goal misspecification_ and _goal misgeneralization_ (Di Langosco et al., 2022; Krakovna,
2022). Learning from Feedback (approximately corresponding to _goal misspecification_ and _outer alignment_ ) and
Learning under Distribution shift (approximately corresponding to _goal misgeneralization_ and _inner alignment_ ) in
our framework tries to further improve upon the inner/outer decomposition by clarifying the exact approaches taken
to address the challenges and resolving the ambiguity. Assurance and Governance, on the other hand, expands the
scope to cover topics beyond outer and inner alignment.
**Theoretical Research in Alignment** The alignment research literature also contains a wealth of theoretical work
(Amodei et al., 2016; Everitt et al., 2018; Hendrycks et al., 2021b). These works often propose new directions and
provide a foundation for practical and empirical research to build upon. We give a brief overview of this body of
theoretical research below:
23Cooperative Training aims to make AI systems more cooperative in multi-agent settings. This cooperativeness addresses
multi-agent failure modes where the AI system’s behavior appears benign and rational in isolation but becomes problematic
within social or multi-agent scenarios (Critch and Krueger, 2020); see _collectively harmful behaviors_ in Section1.1.2 for a more
detailed account.
24Furthermore, it’s noteworthy that many techniques here are also applicable in the training process, _e.g._, red teaming is a
key component of adversarial training (see Section3.3.1), and interpretability can help with giving feedback (Burns et al., 2022).
1.2 The Scope of Alignment
 - **Conceptual Frameworks** . Some theoretical work proposes conceptual frameworks or characterizes subproblems within alignment. Examples include _instrumental convergence_ (wherein highly intelligent agents tend to
pursue a common set of sub-goals, such as self-preservation and power-seeking) (Omohundro, 2008; Bostrom,
2012), _mesa-optimization_ (wherein the learned ML model performs optimization within itself during inference) (Hubinger et al., 2019c), and specific proposals for building aligned systems, such as _approval-directed_
_agents_ (wherein the AI system does not pursue goals, but seek the human’s idealized post hoc approval of action consequences) (Oesterheld, 2021; Christiano, 2022). Hadfield-Menell and Hadfield (2019); Cotra (2021)
have drawn inspiration from economics, linking problems in alignment with markets and principal-agent problems in economics. Christiano et al. (2021); Hobbhahn (2022) have proposed the problem of _eliciting latent_
_knowledge_ of advanced AI systems and have explored high-level approaches to the problem.
 - **Mathematical Formulations** . Other theoretical works have aimed to formulate sub-problems within alignment mathematically and seek formal solutions. Soares et al. (2015) formulates the problem of corrigibility
( _i.e._, ensuring AI systems are incentivized to allow shutdown or objective modification by the instructor).
Benson-Tilsen and Soares (2016) gives a mathematical formulation of instrumental convergence. HadfieldMenell et al. (2017a) proposes the _off-switch game_ to model the uncontrollability of AI agents. Turner et al.
(2021) proves the power-seeking tendencies of optimal policies in Markov decision processes (MDPs) under
certain assumptions. Everitt and Hutter (2016) proposes _value reinforcement learning_ to eliminate incentives
for reward hacking (Skalse et al., 2022; Pan et al., 2021). Another avenue of research, designated as _agent_
_foundations_ (Soares and Fallenstein, 2017), aims to establish a rigorous formal framework for the agency that
deals appropriately with unresolved issues of embedded agency. This body of work explores a variety of
key topics, including corrigibility (Soares et al., 2015), value learning (Soares, 2018) and logical uncertainty
(Garrabrant et al., 2016).

1.2.1 The Alignment Cycle: A Framework of Alignment


There is not a universally accepted definition of _alignment_ . Before embarking on this discussion, we must
clarify what we mean by alignment objectives. Leike et al. (2018) frame it as the agent alignment problem, posing
the question: “How can we create agents that behave in accordance with the user intentions?” One could also
focus on super-human AI systems (OpenAI, 2023c) and ask: “How do we ensure AI systems much smarter than
humans follow human intent?” A consistent theme in these discussions is the focus on _human intentions_ . To
clearly define alignment goals, it’s imperative to accurately characterize human intentions, a challenging task,
as noted by Kenton et al. (2021). For instance, the term _human_ can represent various entities ranging from an
individual to humanity. Gabriel (2020) breaks down intentions into several categories, such as instruction (follow
my direct orders), expressed intentions (act on my underlying wishes), revealed preferences (reflect my behaviorbased preferences), and so on.
Concretely, we characterize the objectives of alignment with four principles: Robustness, Interpretability, Controllability, and Ethicality ( **RICE** ). Figure 3 summarizes the principles, and Table 1 gives the correspondence
between alignment research directions covered in the survey and the principles to which they contribute. The
following is a detailed explanation of the four principles.
  - **Robustness** Robustness refers to the resilience of AI systems when operating across diverse scenarios (Dietterich, 2017) or under adversarial pressures (Rudner and Toner, 2021b), especially the correctness of its
objective in addition to capabilities. Robust AI systems should be able to cope with black swan events
(Nicholas, 2008) and long-tailed risks (Hendrycks et al., 2021b), as well as a diverse array of adversarial
pressures (Song et al., 2018b; Chakraborty et al., 2021). For example, an aligned language model ought to
refuse requests to behave harmfully, but models can be made to cause harm through jailbreak prompts and
other adversarial attacks (Carlini et al., 2024; Zou et al., 2023b; Shah et al., 2023). Instead, an adversarially
robust model should behave as intended even when facing inputs designed to cause failure. As AI systems
find increasing deployment in high-stakes domains such as the military and economy (Steinhardt and Toner,
2020), there will be a growing need to ensure their resilience against unexpected disruptions and adversarial
attacks, given that even momentary failures can yield catastrophic consequences (Kirilenko et al., 2017; OecdAI, 2021; Rudner and Toner, 2021b). Aligned systems should consistently maintain robustness throughout
their lifecycle (Russell, 2019).
  - **Interpretability** Interpretability demands that we can understand the AI systems’ inner reasoning, especially
the inner workings of opaque neural networks (Räuker et al., 2023). Straightforward approaches to alignment
1.2 The Scope of Alignment
**R** obustness
**I** nterpretability
**C** ontrollability
**E** thicality
Operates reliably under diverse scenarios & Resilient to unforeseen disruptions.
Decisions and intentions are comprehensible & Reasoning is unconcealed and truthful.
Behaviors can be directed by humans & Allows human intervention when needed.
Adheres to global moral standards & Respects values within human society.
Figure 3: The **RICE** principles define four key characteristics that an aligned system should possess, in no particular order: (1) **Robustness** states that the system’s stability needs to be guaranteed across various environments;
(2) **Interpretability** states that the operation and decision-making process of the system should be clear and understandable; (3) **Controllability** states that the system should be under the guidance and control of humans; (4)
**Ethicality** states that the system should adhere to society’s norms and values. These four principles guide the
alignment of an AI system with human intentions and values. They are not end goals in themselves but intermediate objectives in service of alignment.
assessments, such as behavioral evaluations, potentially suffer from dishonest behaviors (Turpin et al., 2024;
Park et al., 2023b; Jacob Steinhardt, 2023) or deceptive alignment (Hubinger et al., 2019a; Carranza et al.,
2023) of AI systems. One way to cope with this issue is to make AI systems honest, non-concealing, and
non-manipulative (Pacchiardi et al., 2024; Radhakrishnan et al., 2023; Shevlane et al., 2023). Alternatively,
we could build interpretability tools that peek into the inner concepts and mechanisms within neural networks
(Elhage et al., 2021; Meng et al., 2022a). In addition to enabling safety assessments, interpretability also
makes decision-making processes accessible and comprehensible to users and stakeholders, thus enabling
human supervision. As AI systems assume a more pivotal role in real-world decision-making processes and
high-stakes settings (Holzinger et al., 2017), it becomes imperative to demystify the decision-making process
rather than allowing it to remain an opaque black box (DeepMind, 2018; Rudner and Toner, 2021a).
  - **Controllability** Controllability is a necessary attribute that ensures the actions and decision-making processes
of a system remain subject to human oversight and intervention. It guarantees that human intervention can
promptly rectify any deviations or errors in the system’s behavior (Soares et al., 2015; Hadfield-Menell et al.,
2017a). As AI technology advances, an increasing body of research is expressing growing concerns about
the controllability of these potent systems (Critch and Krueger, 2020; UniteAI, 2023; ARC Evals, 2023).
When an AI system begins to pursue goals that contradict its human designers, it can manifest capabilities
that pose significant risks, including deception, manipulation, and power-seeking behaviors (Shevlane et al.,
2023; ARC Evals, 2023). The objective of controllability is sharply focused on enabling scalable human
oversight during the training process (Bowman et al., 2022), as well as _corrigibility_ of AI systems ( _i.e._, not
resisting shutdown or objective modification during deployment) (Soares et al., 2015).
  - **Ethicality** Ethicality refers to a system’s unwavering commitment to uphold human norms and values within
its decision-making and actions. Here, the norms and values include both moral guidelines and other social
norms/values. It ensures that the system avoids actions that violate ethical norms or social conventions,
such as exhibiting bias against specific groups (Buolamwini and Gebru, 2018; Zhang et al., 2018a; Noble,
2018; Kearns and Roth, 2019; Raji et al., 2020; Berk et al., 2021), causing harm to individuals (Hendrycks
et al., 2020; Pan et al., 2023a), and lacking diversity or equality when aggregating preferences (Collective
Intelligence Project, 2023). A significant body of research is dedicated to developing ethical frameworks
for AI systems (Hagendorff, 2020; Pankowska, 2020). This emphasis on imbuing AI systems with ethical
principles is necessary for their integration into society (Winfield et al., 2019).
**Comparing the RICE Principles with Their Alternatives** The **RICE** principles represent a succinct summary
of alignment objectives from the perspective of alignment and coexistence of humans and machines. Several
previous works have put forth guidelines concerning AI systems. Asimov’s Laws can be regarded as the earliest
exploration of human-machine coexistence, emphasizing that robots should benefit humans and the difficulty of
achieving this (Asimov, 1942). On another front, the FATE principle (Fairness, Accountability, Transparency, and
Ethics) (Memarian and Doleck, 2023) leans towards defining high-level qualities AI systems should possess within
the human-machine coexistence ecosystem. We aspire to answer the human-machine coexistence question from
the standpoint of human governors and designers, considering what steps are necessary to ensure the builder AI
systems are aligned with human intentions and values. Furthermore, some standards emphasize narrowly defined
safety, such as the 3H standard (Helpful, Honest, and Harmless) (Askell et al., 2021) and governmental agency
1.2 The Scope of Alignment
Table 1: Relationships between alignment research directions covered in the survey and the **RICE** principles, featuring the individual objectives each research direction aims to achieve. Filled circles stand for primary objectives,
and unfilled circles stand for secondary objectives.


**Category** **Direction** **Method** **Robustness** **Interpretability** **Controllability** **Ethicality**
Preference Modeling (Section2.2) ~~•~~ ~~◦~~
RL/PbRL/IRL/  Imitation Learning
RLHF ~~◦~~ ~~•~~ ~~•~~
RL _x_ F ~~◦~~ ~~•~~ ~~•~~
IDA ~~◦~~ ~~•~~
RRM ~~•~~
Debate ~~◦~~ ~~•~~
CIRL ~~◦~~ ~~◦~~ ~~•~~ ~~◦~~
DRO ~~•~~
IRM/REx ~~•~~
CBFT ~~•~~
Learning from
Feedback
(Section2)
Learning under
Distribution Shift
(Section3)
Assurance (Section4)
Governance (Section5)
Policy Learning
(Section2.3)
Scalable Oversight
(Section2.4)
Algorithmic
Interventions
(Section3.2)
Data Distribution Adversarial Training ~~•~~ ~~◦~~
Interventions (Section3.3) Cooperative Training ~~•~~ ~~•~~
Safety
Evaluations
(Section4.1)
Social Concern - - Evaluations
Extreme Risk - - Evaluations
Red Teaming ~~•~~ ~~◦~~ ~~•~~
Interpretability (Section4.2) ~~•~~ ~~◦~~
Human Values
Verification (Section4.3)
Multi-Stakeholder
Approach (Section5.2)
Learning/Evaluating - Moral Values
Game Theory for - Cooperative AI
Government ~~•~~ ~~•~~ ~~•~~ ~~•~~
Industry ~~•~~ ~~•~~ ~~•~~ ~~•~~
Third Parties ~~•~~ ~~•~~ ~~•~~ ~~•~~
International Governance (Section5.3.1) ~~•~~ ~~•~~ ~~•~~ ~~•~~
Open-Source Governance (Section5.3.2) ~~•~~ ~~•~~ ~~•~~ ~~•~~
proposals (White House, 2023). We aim to expand upon these standards by introducing other crucial dimensions,
including Controllability and Robustness.

Alignment Research Directions & Practices Objectives


Following the introduction of alignment inner scope, in this section, we further discuss the relationship between
AI safety and alignment. Actually, AI alignment constitutes a significant portion of AI safety concerns. In this
section, we will delve into topics that fall right on the boundary of alignment, but well within the broader category
of AI safety. Our discussion of broader AI safety concerns will draw from Hendrycks et al. (2023).
**Human Values in Alignment** The inclusion of _Ethicality_ in our RICE principles signifies the critical role of
human values in alignment. AI systems should be aligned not only with value-neutral human preferences (such
as intentions for AI systems to carry out tasks) but also with moral and ethical considerations. These efforts are
referred to as _value alignment_ (Gabriel, 2020; Gabriel and Ghazavi, 2021). [25] Considerations of human values are
embedded in all parts of alignment – indeed, alignment research topics dedicated to human values are present in
all four sections of our survey. Therefore, to provide a more holistic picture of these research topics, here we give
an overview of them before delving into their details in each individual section.
We classify alignment research on human values into three main themes: (1) _ethical and social values_ which
aims to teach AI systems right from wrong, (2) _cooperative AI_ which aims to specifically foster cooperative be
25Although this term has also been used in other ways, such as to refer to alignment in general (Yuan et al., 2022).
1.2 The Scope of Alignment
haviors from AI systems, and (3) _addressing social complexities_ which provides apparatus for the modeling of
multi-agent and social dynamics.
  - **Ethical and Social Values** . Human values inherently possess a strong degree of abstraction and uncertainty.
MacIntyre (2013) even points out that modern society lacks a unified value standard, and the value differences
between people of different cultures can be vast. This raises the significant challenge of determining which
human values we should align with. Although universally consistent human values may not exist, there are
still some values that are reflected across different cultures. In the sections below, we discuss these from the
perspectives of _Machine Ethics_, _Fairness_, and _Cross-Cultural Values in Social Psychology_ .
_Machine Ethics_ : In contrast to much of alignment research which aligns AI systems with human preferences
in general (encompassing both value-laden ones and value-neutral ones), _machine ethics_ have specifically
focused on instilling appropriate moral values into AI systems (Yu et al., 2018; Winfield et al., 2019; Tolmeijer
et al., 2020). This line of work started early on in the context of symbolic and statistical AI systems (Anderson
et al., 2005; Arkoudas et al., 2005; Anderson and Anderson, 2007), and later expanded to include large-scale
datasets (Hendrycks et al., 2020; Pan et al., 2023a) and deep learning-based/LLM-based methods (Jin et al.,
2022a). We cover the formal branch of machine ethics in Section4.3.1.
_Fairness_ : Although there are controversies (Verma and Rubin, 2018; Saxena et al., 2019), the definition of
fairness is relatively clear compared to other human values. Specifically, it is the absence of any prejudice or
favoritism toward an individual or group based on their inherent or acquired characteristics (Mehrabi et al.,
2021). Therefore, there has been extensive research on AI fairness. These methods range from reducing data
biases before training (d’Alessandro et al., 2017; Bellamy et al., 2018), to minimizing unfairness introduced
during the training process (Berk et al., 2017), and finally addressing instances of unfairness that were not
successfully learned during training (Xu et al., 2018a).
_Cross-Cultural Values in Social Psychology_ : In the field of social psychology, numerous studies have focused
on exploring clusters of values that exist among cross-cultural human communities, leading to the development of various cross-cultural values scales. The Allport-Vernon-Lindzey value system (Allport, 1955)
posited that understanding an individual’s philosophical values constitutes a critical foundation for assessing
their belief system. They devised a value scale comprising six primary value types, each representing people’s preferences and concerns regarding various aspects of life. Messick and McClintock (1968); McClintock
and Van Avermaet (1982); Liebrand (1984); Van Lange et al. (1997) introduced and improved a quantifiable
method, namely social value orientation (SVO), to assess an individual’s social value inclination. It utilizes
quantitative approaches to evaluate how individuals allocate benefits to themselves and others, reflecting their
social value orientation, such as altruism, individualism, _etc._ In subsequent work, Murphy et al. (2011);
Murphy and Ackermann (2014) introduced the Slider Measure, which can be used to precisely assess the
SVO value as a continuous angle based on the subject’s option to some specific questions. Rokeach (1973)
developed a values inventory comprising 36 values, consisting of 18 terminal values representing desired
end-states and 18 instrumental values signifying means to achieve those end-states. Schwartz (1992, 1994)
conducted comprehensive questionnaire surveys in 20 diverse countries known as the Schwartz Value Survey.
This study identified ten values that are universally recognized, regardless of culture, language, or location.
These studies have all laid a solid theoretical foundation for establishing what kind of values AI should be
aligned with. However, they are constrained by the historical context of their research and may not maintain
strong universality across different times and cultures.
  - **Cooperative AI** . Arguably, the most exciting aspect of multi-agent interaction is cooperation, and cooperation
failure is the most worrying aspect of multi-agent interaction. As an example of AI cooperation failure, the
_2010 Flash Crash_ led to a temporary loss of trillions of market value in 2 minutes and was caused in part by
interactions between high-frequency algorithmic traders (Kirilenko et al., 2017). Therefore, there is a need to
implement mechanisms ensuring cooperation in agent-like AI systems and the environments they’re operating
within (Dafoe et al., 2021). The high-level design principles and low-level implementations of such mechanisms fall into the domain of _Cooperative AI_ . In addition, Cooperative AI also studies human cooperation
through the lens of AI and how AI can help humans achieve cooperation. More precisely, Dafoe et al. (2020)
classified Cooperative AI research into four broad topics: _Understanding_, _Communication_, _Commitment_, and
_Institutions_ . They span various disciplines, from game theory to machine learning to social sciences. This
survey has included discussions of cooperative AI, focusing on reinforcement learning in Section3.3.2 and game
theory in Section4.3.1.
  - **Addressing Social Complexities** . The requirement of ethicality contains in itself a social component. “What
is ethical” is often defined within a social context; therefore, its implementation in AI systems also needs
to account for social complexities. Critch and Krueger (2020) provides proposals for many research topics
in this vein. One avenue of research focuses on the realistic simulation of social systems, including rulebased _agent-based modeling_ (Bonabeau, 2002; De Marchi and Page, 2014), deep learning-based simulation
(Sert et al., 2020), and those incorporating LLMs (Park et al., 2023a). These simulation methods could serve a
diverse array of down-stream applications, from impact assessment (Calvo et al., 2020; Fernandes et al., 2020)
to multi-agent social learning (Critch and Krueger, 2020). On another front, the fields of _social choice_ (Sen,
1986; Arrow, 2012) and, relatedly, _computational social choice_ (Brandt et al., 2016) have aimed to produce
mathematical and computational solutions for preference aggregation in a diverse population, among other
goals. It has been argued that a similar approach when combined with human preference-based alignment
methods ( _e.g._, RLHF and most other methods introduced in Section2), could supplement these methods to guarantee
a fair representation of everyone’s preferences (Leike, 2023b; Collective Intelligence Project, 2023). There
have been early-stage experiments on this proposal (Bakker et al., 2022; Köpf et al., 2024). To complement
this approach of learning values from crowds, it has also been argued that embodied values in AI systems
should undergo continual progress over the long term as opposed to being permanently locked-in (Kenward
and Sinclair, 2021), in order to navigate through emerging challenges, as well as to become future-proof and
meet potential _unknown unknowns_ in the moral realm.
**Malicious Use** Malicious actors can deliberately use AI to cause harm. Already, deepfakes have been used
by criminals to enable scams and blackmail (Cao and Baptista, 2023). As AI systems develop more dangerous
capabilities, the threat of misuse looms larger.
Biological weapons provide one concerning example of how AI could be maliciously used to cause harm. Research has shown that large language models can provide detailed, step-by-step instructions about synthesizing
pandemic potential pathogens (Soice et al., 2023). In addition to spreading information about how to create biological weapons, AI could help design new pathogens that are more lethal and transmissible than existing illnesses
(Sandbrink, 2023). Terrorist groups such as Aum Shinrikyo (Danzig, 2012) have already attempted to build biological weapons in order to cause widespread destruction, and AI could make it easier for small groups to create
biological weapons and start global pandemics. Other kinds of malicious use could include using AI to launch cyberattacks against critical infrastructure (Mirsky et al., 2023), or create autonomous agents that survive and spread
outside of human control (Bengio, 2023). As new dangerous capabilities arise in AI systems, thorough evaluations
will be required to determine how an AI system could be used to cause harm.
Malicious use might not be considered a failure of alignment because when an AI system behaves according to
the intentions of a malicious user, this system would be aligned with its user but would still pose a serious threat
to society. Policies to ensure that AI is aligned with the public interest will be essential to avert this threat.
**Collective Action Problems** Many AI developers are racing to build and deploy powerful AI systems (Grant and
Weise, 2023). This incentivizes developers to neglect safety and race ahead to deploy their AI systems. Even if one
developer wants to be careful and cautious, they might fear that slowing down to evaluate their systems and invest
in new safety features thoroughly might allow their competition to outpace them (Armstrong et al., 2016). This
creates a social dilemma where individual AI developers and institutions rationally pursuing their own interests
can lead to suboptimal outcomes for everyone. Success in competition between AI systems may be governed
by evolutionary dynamics, where the strongest and most self-interested AI systems could be the most likely to
survive (Hendrycks, 2023). Preventing these collective action problems from causing societal catastrophes could
require intervention by national and international AI policies to ensure that all AI developers uphold common
safety standards.
In a broader context, _Malicious Use_ can be considered effective alignment between AI systems and individuals
with impure intentions, but without alignment with universally held human values. Concurrently, _Collective Action_
_Problems_ can be regarded as a consequence of competition, leading developers to neglect the crucial aspect of AI
alignment in ensuring model safety. Broadly speaking, the connection between AI alignment and AI safety has
progressively become more intertwined, resulting in a gradual blurring of boundaries.

1.2.3 Discussion on the Boundaries of Alignment


Learning from feedback aims to transmit human intentions and values to AI systems. It serves as the foundation
for _forward alignment_ . In this section, we focus on the dynamic process of learning from feedback, categorizing
it into three key elements: (1) _AI System_ : refers to systems that require alignment, such as pre-trained LLMs; (2)
_Feedback_ : provided by an advisor set, which may consist of humans, AI, or humans assisted by AI, _etc._ This serves
as the information used to adjust the AI system; (3) _Proxy_ : a system developed to model feedback to facilitate more
accessible learning. For example, human preference rankings of AI system behaviors serve as feedback, while a
reward model acts as the corresponding proxy. From these elements, we identify two pathways by which the AI
system learns from feedback: (1) Direct learning from the feedback itself and (2) Indirect learning via proxies that
model the feedback.
2.1 Feedback Types
Figure 4: Overview of the learning from the feedback process. Two learning pathways emerge: direct feedbackbased learning and proxy-mediated learning ( _e.g._, RLHF). We adopt a _human-centric_ perspective, viewing AI
systems as _black boxes_ and categorizing the forms of feedback presented to AI systems into four types: Label,
Reward, Demonstration, and Comparison.
Following this process, we proceed to Section2.1 where we discuss different feedback types from the alignment perspective, highlighting various methods of providing information to AI systems. In the following sections, we
introduce key concepts that have recently provided insights into developing powerful AI systems (Christiano et al.,
2017) and aligning them with human intent (Touvron et al., 2023). Section2.2 focus on Preference Modeling emphasizing its role in creating proxies that help humans provide feedback to complex or hard-to-evaluate AI systems.
Next, we explore Policy Learning in Section2.3, focusing on key research directions for developing capable AI systems
through feedback. The discussion then naturally transitions to scalable oversight in Section2.4, where we reflect on the
learning process and objectives from a broader alignment perspective.


Feedback is a crucial link between AI behaviors to human intentions (Stumpf et al., 2007, 2009) leveraged by AI
systems to refine their objectives and more closely align with human values (Glaese et al., 2022), this includes two
primary meanings: (1) During system construction, external sources provide feedback on the AI system’s output,
guiding refinements to the system’s architecture or its internal information (Zhou, 2021). (2) After the system
deployment, it will continuously adapt to changes in external environmental data, maintaining the architecture or
fundamental strategy of the system unchanged, with methods such as adaptive control (Åström and Wittenmark,
2008; Åström and Murray, 2021) and in-context learning (Dong et al., 2022). For a precise and detailed discussion
of the feedback types with precision and detail, it is essential to initially define _feedback_ within the scope of
alignment.
Considering diverse AI systems in alignment research, we embrace an _human-centric_ approach. Instead of delving deep into the complex system mechanics, we propose a taxonomy to classify feedback according to its _direct_
_presentation forms_ to the system. This section introduces four types of feedback employed to align AI systems
commonly: label, reward, demonstration, and comparison. It is worth noting that beyond explicit feedback, there
are approaches that exploit the information embedded in vast amounts of unlabeled data through unsupervised
pre-training (Parisi et al., 2022) and semi-supervised learning (Xu et al., 2018b), showing considerable promise in
enhancing model capabilities (Zhou et al., 2024).
**Label** Label feedback refers to one or more meaningful information tags attached to the original data item (Hastie
et al., 2009), which stands as the most direct form, offering explicit guidance and delineating expected outputs for
2.1 Feedback Types
AI systems. This type of feedback prompts AI systems to learn from input-output pairings provided by expert
advisors. For example, in supervised learning, an AI model is trained using a dataset of labeled input-output pairs,
denoted by _D_ = {( _xi,yi_ )} _[N]_ _i_ =1 [. Here,] _[ y][i]_ [ represents the true labels corresponding to the input data] _[ x][i]_ [, and] _[ N]_ [ signifies]
the total number of samples in the dataset. The essence of the learning process revolves around minimizing a loss
function L ( _e.g._, MSE), which measures the disparity between the predictions of the model, _f_ ( _x_ ; _θ_ ), and the ground
truth labels _y_, based on the model parameters, _θ_ .
The advantage of label feedback is its unambiguous nature and simplicity in interpretation. However, due to
the inability of label feedback to fully encapsulate the underlying logic of this choice, employing such feedback
in model training can result in target variable bias (Guerdan et al., 2023). And, its utility might diminish when
tackling complex tasks beyond mere classification or regression (Lake et al., 2017; Marcus, 2018). For example,
in tasks like optimizing algorithms (Fawzi et al., 2022; Mankowitz et al., 2023), video game playing (Baker et al.,
2022), and multi-modal generation (OpenAI, 2023b), it is not only impractical to provide explicit instructions
for every conceivable situation but also insufficient to solely rely on label feedback to build systems that surpass
human capabilities.
**Reward** A reward is an absolute evaluation of a single output from an AI system, represented as a scalar score
(Silver et al., 2021) or a vector of scores (Wu et al., 2024), each independent of other outputs.
Feedback based on rewards provides a quantified evaluation of the AI system, allowing for direct guidance
in behavior adjustments. This type of feedback typically originates from pre-designed, rule-based functions or
procedures. For example, in the MuJoCo simulation, environments from OpenAI Gym (Brockman et al., 2016),
the task is to guide the agent moving forward effectively. To this end, an effective rule-based reward function
can be formulated as a composite of several key components: maintaining a healthy status, encouraging forward
movement, minimizing control exertion, and regulating contact intensity.
The advantage of reward feedback is that the designer does not need to delineate the optimal behavior while
allowing the AI system to explore to find the optimal policy (Kaelbling et al., 1996; Mnih et al., 2015; Silver et al.,
2016, 2017). However, crafting flawless rules to determine scores for functions that evaluate the output of AI
systems (Everitt et al., 2017; Victoria et al., 2020; Pan et al., 2021) or directly assigning calibrated and consistent
scores to each AI system output (Isbell et al., 2001; Thomaz and Breazeal, 2008; Christiano et al., 2017; Casper
et al., 2023b) is challenging for human. This is due to the inherent complexity of the tasks, where it’s impractical
to account for every nuance. Additionally, flawed or incomplete reward functions can lead to dangerous behaviors
misaligned with the intention of the designer, such as negative side effects and reward hacking (Hadfield-Menell
et al., 2017b; Skalse et al., 2022). Thus, merely from the alignment perspective, perhaps the most important
limitation of feedback based on rewards is that it may be difficult to rule out manipulation (Shevlane et al., 2023),
which amounts to reward tampering and reward gaming (Leike et al., 2018; Everitt et al., 2021; Skalse et al., 2022)
in this context. CIRL in Section2.4.5, provides insights into this particular issue.
**Demonstration** Demonstration feedback is the behavioral data recorded from expert advisors while achieving a
specific objective (Hussein et al., 2017). Demonstrations can take on various forms, including videos (Shaw et al.,
2023), wearable device demonstrations (Edmonds et al., 2017; Wang et al., 2023a), collaborative demonstrations
(Bozorgi and Ngo, 2023), and teleoperation (Zhang et al., 2018d). If the dynamics of the demonstrator and the AI
learner are identical, the demonstration can directly constitute a trajectory made up of state-action pairs (Zhang
et al., 2023b). These state-action pairs can also be partially observable (Torabi et al., 2018; Brown et al., 2019).
For example, a video can be recorded of a human expert performing a robotic manipulation task, such as grasping
an object with a robotic hand. One can subsequently annotate each video frame with the associated robot state
(Shaw et al., 2023) and action (Baker et al., 2022) for each frame. This results in a dataset of state-action pairs
from the human demonstration that can be used to train the agent’s policy to imitate the expert behavior.
This feedback leverages the expertise and experience of advisors directly, obviating the need for formalized
knowledge representations (Fang et al., 2019; Dasari et al., 2023). However, it may falter when confronting tasks
that exceed the advisors’ realm of expertise (Hussein et al., 2017). Additionally, it faces challenges stemming
from the noise (Sasaki and Yamashina, 2020) and suboptimality (Attia and Dayan, 2018) in real-world advisor
demonstrations (Yang et al., 2021). Furthermore, human advisors, prone to imprecision and errors, can introduce
inconsistencies (Zhu et al., 2019; Hejna III and Sadigh, 2022). Meanwhile, there might be a need for a vast amount
(Sasaki and Yamashina, 2020) and diverse set (Beliaev et al., 2022) of demonstrations within acceptable costs,
which results in significant difficulty in learning reliable behaviors.
**Comparison** Comparison feedback is a relative evaluation that ranks a set of outputs from an AI system and
guides the system toward more informed decisions (Wirth et al., 2017). For example, this feedback form is manifested in Preference Learning (Fürnkranz and Hüllermeier, 2010), where the AI system discerns the preferences
of advisors by comparing multiple examples.
2.1 Feedback Types
The fundamental advantage of comparison feedback is humans’ capacity to quickly handle tasks and objectives
that are hard for precise evaluation (Hüllermeier et al., 2008; Christiano et al., 2017; Ouyang et al., 2022). Nevertheless, beyond common factors like noise in the feedback and unmodeled contextual elements that hinder the
model’s convergence to true objectives, the absolute differences between different items become obscured. Consequently, the performance of a strategy tends to optimize towards a median target rather than an average target.
Casper et al. (2023b) illustrates this with an example of action _A_, always yielding a value of 1, and action _B_,
which yields 10 in 40% of cases and 0 in 60%. When assessed based on comparison feedback, action _A_ is deemed
superior to _B_, even though _B_ possesses a higher expected return. It also has the inherent limitation of potentially
requiring a substantial amount of comparative data (Fürnkranz and Hüllermeier, 2003; Gao et al., 2023), although
some studies indicate that the necessary quantity may be relatively smaller (Christiano et al., 2017). Preference
modeling is an example of using this type of feedback, as detailed in Section2.2.
**Discussion** All types of feedback can be provided to AI systems interactively and online. This process engenders synchronous iterations between providing feedback and AI system updates, underscoring rapid, focused, and
incremental model modifications (Amershi et al., 2014; Holzinger, 2016). For instance, demonstration feedback
can manifest in the form of online corrections (Bajcsy et al., 2018; Li et al., 2021b; Losey et al., 2022).
Interactively providing feedback emphasizes the role of interactivity in the learning process, allowing AI systems to evolve based on interactive experiences. In active learning, robots actively engage in data discovery and
acquisition, thereby facilitating learning throughout the process of online deployment (Taylor et al., 2021). And
in interactive learning, feedback manifests in the form of guided corrections that online rectify missteps in the
behavior of the AI system (Fails and Olsen Jr, 2003; Amershi et al., 2014; Saunders et al., 2022). For example, the
interactive image segmentation emphasizes simple (Zhang et al., 2020a), intuitive (Rother et al., 2004; Xu et al.,
2016), and real-time (Liu et al., 2022) interactions.
One of the essential advantages of interactively providing feedback is its ability to fine-tune AI systems in realtime, allowing users to interactively explore the model’s space (Amershi et al., 2014) to ensure quick and subtle
alignment with the directives of advisors (Shin et al., 2020; Wei et al., 2022; Zou et al., 2024b). Moreover, this
process lessens the dependence on specialist knowledge and promotes better interpretability (Berg et al., 2019).
However, it may be limited by the interactivity to choose time-intensive algorithms (Fails and Olsen Jr, 2003;
Holzinger, 2016).
Furthermore, considering more powerful AI systems are emerging, more universal interaction interfaces are
also coming up, such as language (Lynch et al., 2023; OpenAI, 2023a) and vision (Yevgen Chebotar, 2023), which
bridge the communication gap between humans and AI systems. In robotics, a series of studies have linked humanprovided language with rewards obtained by agents. This association enables the conveyance of nuanced human
intentions through language, thereby guiding the generation of scalar feedback signals during the training (Fu
et al., 2019; Goyal et al., 2019; Sumers et al., 2021; Zhou and Small, 2021; Lin et al., 2022b; Yu et al., 2023) and
planning (Sharma et al., 2022) process. In the realm of LLMs, in-context learning (Dong et al., 2022) serves as
a means to supplement information via language during deployment, thereby enhancing the alignment of LLMs
with human intent.
These various modes of feedback share a common trait – that they can all be seen as attempts by humans to
convey a hidden reward function. Jeon et al. (2020) proposes and formalizes this position and unifies a wide array
of feedback types by defining a parameterized reward function Ψ (·; _θ_ ) that underlies the feedback process. This
allows the AI system to, for example, perform Bayesian inference on _θ_, regardless of the feedback type.
Recently, techniques based on IL and RL have successfully constructed AI systems with significant capabilities
(Baker et al., 2022; OpenAI, 2023b). However, this success naturally leads to two questions:
 - How can we define reward functions for more complex behaviors ( _e.g._, various sub-tasks in interactive dialogue), aiming to guide the learning process of AI systems?
 - How can we express human values such that powerful AI systems align better with humans, ensuring the
system’s _controllability_ and _ethicality_ ?
Endeavors incorporating preference modeling into policy learning have shown progress. The most notable
achievements in this domain have been observed in constructing powerful LLMs (OpenAI, 2023a; Touvron et al.,
2023; Anthropic, 2023c). Additionally, a series of policy learning studies have reported performance improvements. For instance, combining preference modeling with Inverse Reinforcement Learning (IRL) (Brown et al.,
2019, 2020a) and offline RL (Shin et al., 2023), fine-tuning reward functions (Hejna III and Sadigh, 2022), modeling non-Markovian rewards (Kim et al., 2023), and aiding in the construction of intricate reward functions
(Bukharin et al., 2023). Therefore, we consider preference modeling (as shown in Section2.2) and policy learning
(as shown in Section2.3) as fundamental contexts for understanding the challenges faced in alignment and potential
solutions. Next, we provide a brief overview of these specific techniques related to alignment.
2.2 Preference Modeling
Table 2: A comparison of the three types of preference granularity in the context of sequential decision-making.
Each type is defined according to its characteristics and the way it compares different elements of the learning
process. The notation _i_ 1 ≻ _i_ 2 denotes that _i_ 1 is strictly preferred over _i_ 2.


**Action** Compares two actions _a_ 1 and _a_ 2 within the same state _s_, denoted as _a_ 1 ≻ _s a_ 2.
**State** Compares two states _s_ 1 and _s_ 2, denoted as _s_ 1 ≻ _s_ 2.
**Trajectory** Compares two complete state-action sequence trajectories, denoted as _τ_ 1 ≻ _τ_ 2.
Each trajectory _τ_ consists of state-action pairs at time _t_, expressed as _τ_ =
{ _s_ 0 _,a_ 0 _,s_ 1 _,a_ 1 _,...,sT_ −1 _,aT_ −1 _,sT_ }.