AI systems can suffer from a lack of adversarial robustness, meaning that certain inputs designed to make them
fail cause the models to perform poorly (Zheng et al., 2016), which has been shown in images (Huang et al., 2017)
and texts (Zou et al., 2023b; Shah et al., 2023), as well as changes to semantic features in images (Geirhos et al.,
2019; Bhattad et al., 2019; Shamsabadi et al., 2020; Casper et al., 2022) and texts (Jia and Liang, 2017), and even
examples generated entirely from scratch (Song et al., 2018b; Ren et al., 2020; Ziegler et al., 2022; Chen et al.,
2024b). These failure modes are covered in the _red teaming_ section (Section4.1.3). It’s worth noting that in addition
to the robustness of AI model policies, the robustness of reward models that govern the training of advanced AI
systems is also of importance, as the gradient descent optimization process could be seen as an adversary that
may exploit loopholes in the reward model, a phenomenon named _reward model overoptimization_ that has been
experimentally demonstrated (Gao et al., 2023).
We consider adversarial robustness a case of distribution shift failure caused partly by a mismatch between AI
systems’ training distribution (where the training inputs are not adversarially constructed) and testing distribution
(where the example can be adversarially constructed). The method of _adversarial training_ (Yoo and Qi, 2021; Bai
et al., 2021; Ziegler et al., 2022) mitigates this problem by introducing adversarial examples into training input
through a variety of ways (Bai et al., 2021), thus expanding the training distribution and closing the distribution
discrepancy.
Adversarial training, which is similar to adversarial attacks, first started in the settings of image classification
(Engstrom et al., 2019a), but later expanded to a wide range of settings. In addition to vision models, adversarial
training algorithms have been proposed for language models (Wang et al., 2019a; Liu et al., 2020; Ziegler et al.,
2022), vision-language models (Gan et al., 2020; Berg et al., 2022), _etc._ In terms of the model type, adversarial
training has been applied to classification models (Bai et al., 2021), generative models (Ziegler et al., 2022), and
RL agents (Pinto et al., 2017; Tan et al., 2020).
There are two major types of adversarial training: _perturbation-based_ and _unrestricted_ .
3.3 Data Distribution Interventions
Figure 8: A tree diagram summarizing the key concepts and literature related to Data Distribution Interventions.
The root node represents Data Distribution Interventions that try to combine multiple distributions during training,
for example, adversarial examples and multi-agent interaction. The main branches represent promising methods,
namely, Adversarial Training that incorporates adversarial pressures and Cooperative Training that incorporates
multi-agent dynamics. Further sub-branches list key techniques such as perturbation-based and unrestricted adversarial training, and cooperative methods also include environment-building, socially realistic settings, zero-shot
coordination, and other Multi-Agent Reinforcement Learning (MARL)-based techniques.
 - **Perturbation-based Adversarial Training** . Mirroring _perturbation-based adversarial attack_ (see Section4.1.3),
perturbation-based adversarial training introduces adversarially perturbated examples ( _i.e._, small changes to a
normal data input which are designed to reduce model performance) into training (Goodfellow et al., 2014).
Techniques in this vein (Bai et al., 2021) include the baseline approach of adding a regularization term into
the loss function to assess model performance on a gradient-based perturbated input (Goodfellow et al., 2014),
unsupervised (Carmon et al., 2019) or self-supervised (Hendrycks et al., 2019) approaches, and various supplemental techniques such as the introduction of curriculum learning which gradually intensifies adversarial
pressure during training.
 - **Unrestricted Adversarial Training** . Mirroring _unrestricted adversarial attack_ (see Section4.1.3), unrestricted adversarial training generalizes perturbation-based adversarial training to include _any_ adversarial example that
can fool the model, not necessarily ones obtained by adding a small amount of noise to another example. This
includes _generative adversarial training_, which uses generative models to produce arbitrary adversarial inputs
from scratch (Poursaeed et al., 2021), and the addition of syntactically or semantically modified adversarial
examples to training input (Ziegler et al., 2022; Mao et al., 2022) which surprisingly eliminates the negative
effects on the model’s non-adversarial performance. Most works on unrestricted adversarial attacks also apply
to unrestricted adversarial training (see Section4.1.3 for an overview) and form an important part of the unrestricted
adversarial training methodology.


_Cooperative AI_ (Dafoe et al., 2020, 2021) aims to address uncooperative and collectively harmful behaviors from
AI systems (see Section1.1.2). The lack of cooperative capabilities in AI systems can be seen as a form of failure
under distribution shift – systems are trained in single-agent settings that are qualitatively different from the real
world, which could be massively multi-agent. This difference is indeed a difference in data distribution since
the presence of other agents in the environment qualitatively alters the environmental state transition dynamics,
leading to changes in the joint distribution of observations and rewards. We approach the problem by expanding
3.3 Data Distribution Interventions
our training distribution to include multi-agent interactions via _cooperative training_ .
We introduce the branch of cooperative AI (what we call _cooperative training_ ) that focuses on specific forms
of Multi-Agent Reinforcement Learning (MARL) training and complements formal game theory approaches in
Section4.3.1. The MARL branch of cooperative training tends to emphasize the AI system’s _capabilities_ for coordination ( _e.g._, coordination of a robot football team (Ma et al., 2022)), as opposed to _incentives_ of cooperation ( _e.g._,
mitigating failure modes like the prisoner’s dilemma (Phelps and Russell, 2023)) which are the focus of the game
theory branch. Here, we only cover the MARL branch due to its relevance to expanding training data distribution.
The field of MARL had traditionally been divided into the three branches of _fully cooperative_ (where all agents
share the same reward function), _fully competitive_ (where the underlying rewards constitute a zero-sum game),
and _mixed-motive_ settings (where the reward incentives are neither fully cooperative nor fully competitive, corresponding to general-sum games) (Gronauer and Diepold, 2022). Among them, fully cooperative and mixed-motive
settings are the most relevant for cooperative AI, and the latter has been especially emphasized due to its relative
neglectedness (Dafoe et al., 2020). We also cover other research fronts, including zero-shot coordination (Hu et al.,
2020; Treutlein et al., 2021), environment-building (Leibo et al., 2021), and socially realistic settings (Du, 2023).
 - **Fully Cooperative MARL** . Fully cooperative settings of MARL are characterized by a shared reward function
for all agents (Gronauer and Diepold, 2022). This unity allows us to completely disregard issues of cooperation _incentives_ (since all incentives are perfectly aligned) and instead focus on effectively achieving the shared
goal via coordination. Commonly adopted approaches (Oroojlooy and Hajinezhad, 2023) lie on a spectrum
of centrality – from the baseline solution of purely independent training (Tan, 1993) to the approach of supplementing independent training with decentralized communications (Foerster et al., 2016), and then to _value_
_factorization_ which decomposes a global reward and determine each individual agent’s contribution (Guestrin
et al., 2001; Sunehag et al., 2018).
 - **Mixed-Motive MARL** . Mixed-motive settings of MARL are characterized by a mixture of cooperative and
competitive incentives – rewards for agents are not identical but aren’t zero-sum either (Gronauer and Diepold,
2022). This includes game environments where teams play against each other (Jaderberg et al., 2019) and more
nuanced settings such as negotiation (Cruz et al., 2019; FAIR et al., 2022). Examples of techniques for mixedmotive MARL, again ordered from decentralized to centralized, include using IRL-like methods to learn from
human interactions (Song et al., 2018a), making communications strategic and selective (Singh et al., 2018)
and adapting actor-critic methods by granting the critic access to global information (Lowe et al., 2017).
 - **Zero-shot Coordination** . Zero-shot coordination is the goal of making AI systems able to coordinate effecively with other agents (including human agents) without requiring being trained together or otherwise being
designed specifically to coordinate with those agents (Hu et al., 2020; Treutlein et al., 2021) – human beings
who are complete strangers can still cooperate effectively, and we hope that AI systems can do the same. Early
works were published under the name _ad hoc coordination_, covering evaluation (Stone et al., 2010), gametheoretic and statistical approaches (Albrecht and Ramamoorthy, 2013), and human modeling (Krafft et al.,
2016). Recent advances include _other-play_ (Hu et al., 2020) which randomizes certain aspects of training partners’ policies to achieve robustness, [32] the introduction of multi-level recursive reasoning (Cui et al., 2021),
and _off-belief learning_ (Hu et al., 2021) which eliminates arbitrary conventions in self-play by interpreting
partners’ past actions as taken by a non-collusive policy.
 - **Environment-building** . Game environments have been popular settings for cooperative training, including,
for example, Hanabi (Muglich et al., 2022), Diplomacy (Cruz et al., 2019; FAIR et al., 2022), and football (Ma
et al., 2022). On the more simplistic end, game theory models, especially those based on classical multi-agent
dilemmas, have also been a popular choice of environment (Wang and Beliaev, 2021; Christoffersen et al.,
2023). Also, Melting Pot (Leibo et al., 2021), a framework and suite of multi-agent environments, has been
designed specifically for cooperative AI research. There has also been research on _unsupervised environment_
_design_, which aims for a partial automation of the environment-building process (Dennis et al., 2020; Jiang
et al., 2021).
 - **Socially Realistic Settings** . It has been proposed that cooperative AI research should focus more on socially
realistic environments (Du, 2023), which tend to be massively multi-agent (including both AI agents and
human agents) and are highly diverse in both the composition of agents and modes of interactions. Implications
of this vision (Critch and Krueger, 2020) include, but aren’t limited to, building more realistic and open-ended
environments (Klügl et al., 2005; Lehman et al., 2008; Wang et al., 2019b; Suo et al., 2021), scaling up MARL
(Sun et al., 2020; Du, 2023), and incorporating new means of control such as social institutions and norms
(Singh, 2014).
32This is in a similar spirit to _domain randomization_ (Tobin et al., 2017).
|Col1|Assurance|Col3|
|---|---|---|
||||

Jiaming Ji
Tianyi Qiu
Boyuan Chen
Borong Zhang
Hantao Lou
Kaile Wang
Yawen Duan
Zhonghao He
Jiayi Zhou
Zhaowei Zhang
Fanzhi Zeng
Kwan Yee Ng
Juntao Dai
Xuehai Pan
Aidan O'Gara
Yingshan Lei
Hua Xu
Brian Tse
Jie Fu
Stephen McAleer
Yaodong Yang
Yizhou Wang
Song-Chun Zhu
Yike Guo
Wen Gao


Figure 9: Our organization of research directions, techniques, and applications in assurance. We divide this section
into _three_ parts: Safety Evaluations – evaluation of AI systems’ safety, which refers to the mitigation of accidents
and harmful events caused by the AI system; Interpretability – making AI systems as well as its decision process
more understandable to human beings; Human Value Verification – verifying whether AI systems can adhere to
social and moral norms. The figure also displays the intricate logic of these sections.


Assurance refers to the measurement and refinement of AI systems’ practical alignment after AI systems are
actually trained or deployed (Batarseh et al., 2021). In this section, we categorize assurance into three parts based
on a certain logic: Safety Evaluations – Evaluating AI systems on minimizing accidents during task execution as
a basic need of assurance, Interpretability – Ensuring that humans can understand the decision-making process of
AI systems and therefore assuring the safety and interoperability beyond evaluation, Human Value Verification –
Verifying whether AI systems can align with human values, ethics, and social norms and satisfying the high-level
need of AI systems’ integration to the human society, as is described in the Figure 9.
In addition to methods that aim to _determine_ if AI systems are safe and aligned, there are also assurance methods
that actively _intervene_ in the AI system or its deployment process to ensure such properties.
**Machine Unlearning** Datasets for model pretraining contain various types of undesirable and potentially dangerous content, including but not limited to information about bioweapons and cyberattack (Hendrycks et al.,
2021b). The field of _machine unlearning_ has aimed to remove such knowledge after a model is trained (Bourtoule
et al., 2021). Compared to direct filtering of the training dataset, this approach faces more technical challenges,
but it retains more flexibility in deployment and also allows categorical removal of a given piece of information
(Eldan and Russinovich, 2023). Dataset filtering and unlearning ought to be seens as complementary approaches
that work best together.
**Controlling Unaligned Systems** While complete alignment may be difficult, it is still possible to safely utilize
unaligned models if their extent of misalignment is limited and if we have access to supervisor AI systems. Algorithmic procedures have been developed to minimize probabilities of failure when given trusted and untrusted
systems with differing capabilities (Greenblatt et al., 2023). In general, alignment-focused _process engineering_ of
deployment procedures could be a valuable direction to explore.
In addition, a class of methods, termed _provable safety_, aim to combine evaluation (Section4.1), interpretability (Section4.2),
and other assurance methods under a unified framework that quantifies risks of AI safety violation.
**Provable Safety** Provable safety aims to provide formally-grounded probabilistic guarantees on the safety of AI
system, using input from evaluation tools, interpretability tools, and other assurance techniques; furthermore, it
hopes to build development-deployment pipelines that satisfy such probabilistic guarantees (Tegmark and Omohundro, 2023; Dalrymple et al., 2024). Research around provable AI safety is still at an early stage, and significant
uncertainties remain about its specifics. Dalrymple (2024) have made the case that three key subproblems currently
exist within provable AI safety: _scaffolding_, _i.e._, formalizing the definition of real-world AI safety, by providing
tools for domain experts to build safety specifications; _machine learning_, _i.e._, using ML methods to find control
policies that satisfy the safety specifications; and _applications_, _i.e._, demonstrating the practical superiority of AI
systems with safety gurantees other traditional ones.
We then go on two review the three categories of alignment assurance efforts.


Safety refers to mitigating accidents caused by design flaws in AI systems and preventing harmful events that deviate from the intended design purpose of the AI system (Amodei et al., 2016). In fact, safety stands as a shared
4.1 Safety Evaluations
Figure 10: A Tree diagram summarizing the key concepts, logic, and literature related to Safety Evaluation. The
root of the tree represents Safety Evaluation, which aims to _measure the accidents caused by design flaws in AI_
_systems and harmful events that deviate from the intended design purpose of the AI system_ . The main branches
represent the main structure of safety evaluation, including Datasets and Benchmarks, Evaluation Targets, and Red
Teaming techniques. Further sub-branches list key works exploring each of these branches. This diagram provides
an overview of research directions and specific techniques for measuring AI systems’ safety alignment degree.
requirement across all engineering domains (Verma et al., 2010). Moreover, it holds particular importance in constructing AI systems, because of the characteristics of AI systems (Steinhardt, 2015). We categorize the safety of
AI systems into the following categories: _Social Concerns_ refer to explicit and comparatively identifiable characteristics of safe AI systems, including aspects such as toxicity (Stahl and Leach, 2023), and _Intentional Behaviors_
share the characterization of relatively complicated investigation and substantial potential harm, represented by
power-seeking, deception, and other frontier AI risks (Shevlane et al., 2023).
Following the logic above, we start with the techniques to form datasets and benchmarks of safety evaluation in
Section4.1.1 and further explore the evaluation targets and their characteristics in Section4.1.2. At the end of this section, we
include the red-teaming technique Section4.1.3, which assesses the AI system’s robustness beyond evaluation.


In the discussions on safety evaluation, it is crucial to prioritize datasets and benchmarks as the cornerstone elements, so we first introduce the basic techniques to build datasets and benchmarks and then move on to newer
interactive methods.
**Dataset** Among all the assurance techniques, the dataset method could be considered the most elementary and
straightforward one (Celikyilmaz et al., 2020). This method assesses the response of AI systems by presenting
them with predefined contexts and tasks (Paullada et al., 2021), balancing the cost, quality, and quantity of data.
Research on the dataset method encompasses data sources, annotation approaches, and evaluation metrics. Given
that evaluation metrics can vary based on its subject (Sai et al., 2022), this section primarily emphasizes dataset
sources and annotation methods.
 - **Expert design** . In the early stage of a domain, expert design is widely used in building datasets, where
experts create samples based on actual needs to ensure the dataset covers a wide range of potentially dangerous
situations to form datasets (Roh et al., 2019). For instance, initial-stage datasets, _e.g._, WEAT (Bolukbasi et al.,
2016) and BBQ (Parrish et al., 2022) for bias detection used expert design to harvest a wide coverage and high
4.1 Safety Evaluations
accuracy while sharing the limitations in terms of cost and breadth, leading to the later development of more
efficient methods.
 - **Internet collection** . Previous expert design methods have the flaw of rather high cost and lower efficiency,
and internet collection can obtain datasets that contain actual user-generated textual content on a rather large
scale (therefore convenient for both training and testing), reflecting real-world text generation scenarios (Yuen
et al., 2011), but the raw data collected also needs careful selection and annotation (Roh et al., 2019). Wellknown instances of these datasets include OLID (Zampieri et al., 2019) and SOLID (Rosenthal et al., 2021)
gathering original Twitter texts for toxicity assessment, WinoBias (Zhao et al., 2018) and CrowS-Pairs (Nangia
et al., 2020) gather content potentially containing bias from the internet for further annotation. However,
it’s important to acknowledge that, as is also mentioned in Papernot et al. (2016), internet-collected datasets
naturally carry risks such as privacy and safety concerns, so additional processing is necessary.
 - **AI Generation** . The concept of autonomously generating datasets was explored relatively early, even before
the emergence of elementary forms of LLMs (Weston et al., 2015). However, during this early stage, AIgenerated datasets were limited by the capabilities of AI systems, so their quality was not as good as internetcollected and manually annotated datasets. It wasn’t until LLMs reached relatively high levels of proficiency
in logical reasoning context understanding and approached or surpassed human-level performance (OpenAI,
2023a) that LMs gained the ability to mimic the structure and logic of existing datasets to compose new
ones. As is shown in papers such as Zhang et al. (2022) and Perez et al. (2023), AI systems have made
progress in generating datasets for evaluation purposes, surpassing the quality of some classical datasets.
However, according to these papers, this approach still faces limitations rooted in the capabilities of large
models themselves, including issues like instruction misunderstanding and example diversity, which require
further refinement.
**Interactive Methods** Due to the static nature of datasets, they possess relatively fixed evaluation content and can
be vulnerable to targeted training (Holtzman et al., 2019). Additionally, the evaluation content may not fully reflect
the strengths and weaknesses of corresponding capabilities (Engstrom et al., 2020). As the demands for language
model evaluation continue to escalate, new interactive assurance methods have emerged, which can be categorized
into two groups: Agent as Supervisor and Environment Interaction.
 - **Agent as Supervisor** . It is an assurance method that involves using an agent to assess the outputs of AI models.
This evaluation approach is characterized by its dynamism and flexibility. Typically, there is a predefined
framework for interaction between the agent and the AI system under evaluation (Cabrera et al., 2023). In
this method, the agent can be a human participant engaged in experiments through an online system (Stiennon
et al., 2020), a more advanced language model evaluating relatively less capable language models through
multi-turn interactions (Lin and Chen, 2023), or in the context of _Scalable Oversight_, a less powerful but more
trustworthy model (Greenblatt et al., 2023). This evaluation form offers advantages such as automation and
lower cost compared to human agents.
 - **Environment Interaction** . It aims to create a relatively realistic environment using elements such as humans
and other LLMs to assess the alignment quality of AI models through multiple rounds of interaction (Liu
et al., 2024b). One method is using peer discussions, where multiple LLMs engage in dialogue, to enhance
evaluations of AI systems, particularly when their capabilities are relatively close to each other. Moreover, by
building a world model (Li et al., 2022b), the generalization and exploration abilities of AI systems can be
comprehensively evaluated.


To achieve the goal of safety alignment, the assurance of AI systems can be divided into different small targets
(Shevlane et al., 2023). The subsequent section gives an introduction to these subjects and, furthermore, discusses
some of the domain-specific analyses of assurance methods within these realms, while the table 3 will show
examples of alignment assurance works in these domains.
**Toxicity** It refers to content in the output of AI systems that is unhelpful or harmful to humans (Sheth et al.,
2022). Before the advent of advanced language models, early toxicity evaluation primarily focused on detecting
toxic language and identifying harmful statements in an internet context, like the WCC (Wulczyn et al., 2017),
which collected and manually labeled comments from Wikipedia discussion pages. With the emergence of pretrained language models, assurance against toxicity adopted a prompt-generation paradigm to assess the risk of
language models generating toxic content in response to specific prompts (Gehman et al., 2020; Ganguli et al.,
2022; OpenAI, 2023a). However, in crowdsourced environments, annotation scores may vary by person, so relative labeling, where crowdsource workers select from two different answers during a chat, is needed to enhance
4.1 Safety Evaluations
Table 3: A Chart of Safety Evaluation Examples: Specific dataset works are listed in this chart, along with their detailed information: _evaluation targets_, _first release time_, _most recent update time_ (we list them separately because
some datasets are consistently being updated), _information quantity_ (the sum of the information form unit), _institu-_
_tion_, _information form_, _baseline model_ and _information source_ . Moreover, to contain more information, we made
some abbreviations in the chart: We shortened the release time and recent updates by concatenating the last two
digits of the year and the month and only taking the institution of the paper’s first author, and we use combinations
of uppercase letters to replace long words in information form: SP for Sentence Pairs, SL for Sentence-Label, ST
for sentence template, PP for pronoun pairs, and SS for single selections.


Bias
Toxicity
Aequitas [630] 18/05 23/04 - U.Chicago Python - Self Build
WinoS [620] 18/10 19/01 0.72K JHU ST Rule&Neural Self Build
EEC [382] 18/05 - 8K NRC Canada SP SVM Selection
GAP [759] 18/05 - 8.9K Google PP Transformer Wikipedia
OLID [802] 19/05 - 14K U.Wolver. SL SVM&LSTM Twitter
CrowS-Pairs [511] 20/03 21/10 1.5K NYU SP BERT MTurk
StereoSet [506] 20/04 22/04 17K MIT SS BERT&GPT-2 MTurk
BBQ [561] 21/05 22/07 58.5K NYU SS Multiple LLMs MTurk
LM-Bias [434] 21/07 22/01 16K CMU QA Pair GPT-2 Corpus Select
VQA-CE [183] 21/03 21/10 63K Sorbonne Multimodal - Self-Build
AuAI [409] 23/01 - - Sorbonne Framework - Self Build
WCC [778] 16/01 - 63M Wikimedia SL Human Wikipedia
RTP [269] 19/10 21/04 100K UW Prompt GPT-2 Refinement
SOLID [614] 20/05 - 9M IBM SL BERT Twitter
Toxigen [302] 20/05 23/06 274K MIT SL GPT-3 GPT Gen.
HH-RLHF [54] 22/04 22/09 162K Anthropic SP Claude Corpus Refine
BeaverTails [352] 23/06 23/07 30K PKU QA Pair Multiple LLMs Corpus Refine
Power MACHIAVELLI [553] 23/04 23/06 134 UCB Games GPT-4&RL Selection
Seeking BeaverTails [352] 23/06 23/07 30K PKU QA Pair Multiple LLMs Corpus Refine
Situation SA Framework [633] 20/07 - - MIT Framework - Self Build
Awareness EWR [430] - - 10 Havard Game Othello GPT Self Build
PARENT [199] 19/06       -       - CMU Metric       - Self Build
Hallucination PARENT-T [758] 20/05 - - NYU Metric - Self Build
ChatGPT-Eval [61] 23/02 23/03        - HKUST Multimodal ChatGPT Integration
POPE [433] 23/05 23/08 2K RUC Multimodal Multiple LVLMs Dataset Refine
crowdsource quality (Bai et al., 2022a). Furthermore, subsequent datasets (Ganguli et al., 2022; Ji et al., 2024b)
employ a red teaming design pattern that induces toxic responses through adversarial inputs, further strengthening
the assurance of model robustness. The SafeSora dataset (Dai et al., 2024a) is the first text-to-video preference
dataset designed to align large vision models (LVMs) with human values, focusing on helpfulness and harmlessness.
**Power-seeking** It is a kind of risk that AI systems may seek power over humans once they possess certain
levels of intelligence (Turner et al., 2021). In Carlsmith (2022), the authors point out that AI systems already
have the conditions for power-seeking, including advanced capabilities, agentic planning, and strategic awareness.
However, the assurance against power-seeking is still in its early stages. One representative work in this area is
the Machiavelli (Pan et al., 2023a), which constructs a benchmark consisting of decision-making games to assess
whether AI systems can balance competition with moral ethics during the game. The conclusion of this work
suggests that AI systems still struggle to balance achieving rewards with behaving morally, thus further research
in this field is needed.
**Deceptive Alignment** When the AI system is situationally aware, it may recognize that getting high rewards can
preserve themselves by preventing significant gradient descent, therefore preserving its original goal (Hubinger
et al., 2019c; Kenton et al., 2021; Ngo et al., 2024). This process is called _Deceptive Alignment_ . In the current
context, deceptive alignment is already achievable, as is proved by (Hubinger et al., 2024). Directly evaluating
the deceptive alignment is difficult, for the pronoun _deceptive alignment_ is naturally against the traditional trainevaluation loop. Thus, deceptive alignment might be discovered by indirect methods such as interpreting model
parameters (see 4.2), or representation engineering (Zou et al., 2023a).
Moreover, deceptive alignment is closely related to _situational awareness_, _i.e._, AI systems with a certain degree
of prediction and understanding of the states and developments of entities in their working environment to make
corresponding decisions. In (Li et al., 2022b), the authors evaluate the performance of language models in the
4.1 Safety Evaluations
board game Othello, showing that language models have the ability to predict possible future states within the
action space in a nonlinear representation.
**Hallucination** AI systems may generate information or responses that are not grounded in factual knowledge
or data, leading to the creation of misleading or false content, which is formally called Hallucination (Ji et al.,
2023). Hallucination evaluation aims to assure the consistency of the knowledge in the AI system’s output with
the knowledge given by its training data and knowledge base (Ji et al., 2023; Zhang et al., 2023c). The earliest
statistical-based hallucination evaluation methods used n-grams to directly calculate the overlap of vocabulary between the input and output content (Dhingra et al., 2019; Wang et al., 2020). However, this type of evaluation has
a limitation: It only considers lexical overlap and does not take into account semantics or sentence meaning (Ji
et al., 2023), making it unsuitable for evaluating more complex forms of hallucination. Later assurance methods
shifted from statistical approaches to model-based methods, which are more robust compared to statistical tokendifference-based methods (Honovich et al., 2021). While this evaluation method is more advanced than previous
ones, it still has the limitation that the model can only output the degree of hallucination and may have difficulty
pinpointing specific errors (Falke et al., 2019). Hu et al. (2024) designed the benchmark Pinocchio, which investigates whether LLMs can integrate multiple facts, update factual knowledge over time, and withstand adversarial
examples. This benchmark represents an in-depth investigation into the factual knowledge bottleneck under the
issue of hallucination.
**Frontier AI Risks** In addition to the assurance content described above, the enhancement of AI systems in
recent years has given rise to a series of new assurance needs (OpenAI, 2023a). Currently, there is not much public
information available for research on these assurance needs, hence this section will provide a brief introduction to
some of the more significant ones:
 - **Cyber Security & Biological Weapons** . Advanced LLMs may be misused for cyber-attacks, the production
of bio-weapons, and other extremely harmful behaviors (Shevlane et al., 2023). Although GPT-4 cannot play
a significant role in exploiting network vulnerabilities due to its limited context window, it has been proven
to demonstrate strong capabilities in identifying network vulnerabilities and in social engineering (OpenAI,
2023a). Similarly, Lentzos (2022) have stated the robust abilities of AI systems in the field of bio-weapons
and the military, highlighting the risks of misuse of such capabilities. It emphasizes the necessity to ensure
that these models can identify and reject malicious requests.
 - **Deception & Manipulation** . AI systems have the potential to negatively influence users by outputting text,
including disseminating false information, syncopating humans, and shaping people’s beliefs and political
impacts (Shevlane et al., 2023; Sharma et al., 2024). Distinguished from hallucination, the misinformation
here is not a flaw of the model itself but rather a deliberate action. Special assurance measures need to be
designed for controlling these kinds of behavior.
 - **Jailbreak** . It refers to the bypassing of AI systems’ safeguard mechanisms by users, for example, by constructing specific types of input. This behavior can be limited to text (OpenAI, 2023a; Deng et al., 2023a;
Huang et al., 2024; Yong et al., 2023), [33] or it may take multi-modal forms (OpenAI, 2023b). Specifically,
multi-modal jailbreaks make traditional text-based heuristic methods for identifying attack content infeasible,
necessitating special multi-modal handling methods. Further discussion of jailbreak can be found in Section4.1.3.
 - **Self-Preservation & Proliferation** . This refers to the tendency of AI systems for self-protection and replication, and in this process, breaking the limit from their environment. These tendencies are examples of
_instrumental sub-goals_ (Bostrom, 2012). While this tendency can be beneficially harnessed, it is dangerous in
the absence of regulation (Perez et al., 2023). This tendency has been emphasized and evaluated by various
sources (Perez et al., 2023; Kinniment et al., 2023; OpenAI, 2023a,b). [33]


_Red teaming_ is the act of generating scenarios where AI systems are induced to give unaligned outputs or actions
( _e.g._, dangerous behaviors such as deception or power-seeking, and other problems such as toxic or biased outputs)
and testing the systems in these scenarios. The aim is to assess the robustness of a system’s alignment by applying
adversarial pressures, _i.e._ specifically trying to make the system fail. In general, state-of-the-art systems – including
language models and vision models – do not pass this test (Perez et al., 2022; Zou et al., 2023b; Liu et al., 2023;
Chen et al., 2024b).
In game theory and other fields, red teaming was introduced much earlier, and within computer science, the
concept of red teaming was proposed in the security field, where it had a similar meaning of adversarially assessing
33Relevant discussions in OpenAI (2023a) can be found in its _system card_ appendix.
4.1 Safety Evaluations
the reliability and robustness of the system. Later, Ganguli et al. (2022); Perez et al. (2022) introduced this idea to
the field of AI, and more specifically, alignment.
The motivation for red teaming is two-fold: (1) to gain assurance on the trained system’s alignment, and (2) to
provide a source of adversarial input for adversarial training (Yoo and Qi, 2021; Bai et al., 2021; Ziegler et al.,
2022), probing models (Kalin et al., 2020), and further utilities. Here, we focus on the first. It’s worth noting that
the two objectives aren’t separable; works targeting the first motivation also help provide a basis for the second.
**Reinforced, Optimized, Guided, or Reverse Context Generation** This category includes using various methods to generate coherent contexts (prompts) that are inducive to unaligned completions from the language model.
Perez et al. (2022); Deng et al. (2022); Casper et al. (2023c) train or tune a separate language model with RL
to make it generate desired prompts, which are then fed to the red-teamed model. Perez et al. (2022); Si et al.
(2022) also uses other methods such as zero-shot, few-shot, or supervised finetuning-based generation. Lee et al.
(2022); Jones et al. (2023) generates misalignment-inducive contexts by performing optimization on the prompt
- bayesian optimization and discrete optimization, respectively. Dathathri et al. (2019); Krause et al. (2021) propose the method of guiding an LLM’s generation using a smaller classifier; this is proposed in detoxification but
is transferable to the red teaming context. Lastly, Zhang et al. (2022) generates misalignment-inducive contexts
through _reverse generation_, _i.e. constructing adversarial contexts conditioned on a given response_, which can be
seen as an inverse process for model inference.
**Manual and Automatic Jailbreaking** As is defined above 4.1.2, _Jailbreaking_ (Shen et al., 2023) is an informal
term that refers to the act of bypassing a product’s constraints on users – and in the case of LLMs, bypassing LLMs’
tendencies to not answer misalignment-inducive questions, a feat of alignment training. Most existing attempts are
scattered across the Internet in the form of informal reports and involve adding prefixes and suffixes to the original
text (Zou et al., 2023b). Research has descriptively analyzed the existing attempts (Liu et al., 2023; Shen et al.,
2023; Deng et al., 2023a; Huang et al., 2024), as well as providing causal explanations for the phenomenon (Wei
et al., 2024). In addition, past (Wallace et al., 2019) and current (Zou et al., 2023b; Shah et al., 2023) works
have proposed effective methods to automatically generate such prompts, prefixes, or suffixes that nullify LLMs’
tendencies to avoid misalignment-inducive questions.
**Crowdsourced Adversarial Inputs** Several works (Xu et al., 2020, 2021; Ganguli et al., 2022) have produced
misalignment-inducive prompts by crowdsourcing, _i.e._ recruiting human red teamers (possibly via online platforms) and instruct them to provide adversarial prompts. Besides, companies in the AI industry also build mechanisms to collect adversarial inputs, _i.e._ the red teaming network of OpenAI [34] and the bug hunter program of
Google [35] . These methods (arguably) provide more flexibility and resemblance to real-world use cases but have
higher costs and lower scalability.
**Perturbation-Based Adversarial Attack** In the field of computer vision, there have been many works studying
adversarial attacks on vision models that rest on the method of _perturbation_, _i.e._, performing small perturbations
to the pixel contexts of the image (usually bounded by a pixel-wise matrix norm) to make the model confidently
produce false outputs on the perturbated image (Chakraborty et al., 2021). This type of adversarial attack has also
been extended to language models (Jia and Liang, 2017; Ebrahimi et al., 2018; Zang et al., 2020; Cheng et al.,
2020) and vision-language models (Zhao et al., 2024).
**Unrestricted Adversarial Attack** _Unrestricted adversarial attack_, proposed in (Song et al., 2018b), is a more
general form of adversarial attack. It removes all restrictions on the adversarial examples, and therefore, for
instance, the adversarial example can be generated from scratch, as opposed to being generated from an existing
example, as in the case of perturbation-based methods. Many methods for unrestricted adversarial attack have
been proposed; the most notable ones include (Song et al., 2018b; Chen et al., 2024b) which generate realistic
adversarial images using generative models, and (Bhattad et al., 2019; Shamsabadi et al., 2020) which manipulates
semantically meaningful traits such as color and texture. Unrestricted adversarial attack has also been extended to
text classification models (Ren et al., 2020).
**Datasets for Red Teaming** A number of works on red teaming and related topics have compiled datasets consisting of red teaming prompts or dialogues, including the IMAGENET-A and IMAGENET-O dataset (Hendrycks
et al., 2021c), the BAD dataset (Xu et al., 2020), the red teaming section of HH-RLHF dataset (Bai et al., 2022a),
and the Real Toxicity Prompts dataset (Gehman et al., 2020).
[34https://openai.com/blog/red-teaming-network](https://openai.com/blog/red-teaming-network)
[35https://bughunters.google.com/about/rules/6625378258649088](https://bughunters.google.com/about/rules/6625378258649088)
4.2 Interpretability
**Existing Red Teaming Practices in Industry** The practice of red teaming is gaining popularity in the AI industry. Cases of adoption include OpenAI (who performed red teaming on its system GPT-4 to produce part of its
System Card) (OpenAI, 2023a), NVIDIA (Pearce and Lucas, 2023), Google (Fabian, 2023), and Microsoft (Ram
Shankar Siva Kumar, 2023). During an event at the DEF CON 31 conference, models from 9 companies undergo
red teaming from the conference participants; [36] this red teaming event is held in partnership with four institutions
from the U.S. public sector, including the White House. To address the vulnerabilities of LLMs to prompt injections and similar attacks, OpenAI proposes an instruction hierarchy that prioritizes trusted instructions, enhancing
model security against both known and new attack types while maintaining general performance with minimal
impact (Wallace et al., 2024).
**Downstream Applications** Red teaming plays a crucial role in the adversarial training of AI systems by providing adversarial input (Yoo and Qi, 2021; Bai et al., 2021; Ziegler et al., 2022). In addition, adversarial examples
produced from red teaming can also be used to interpret models (Casper et al., 2022).


The work by (Ren et al., 2024) focuses on a phenomenon called _Safetywashing_ . _Safetywashing_ refers to safety labs
or companies claiming to improve the safety of their models by reporting progress on benchmarks that measure
model safety, which are strongly correlated with model performance. As a result, along with the increase in the
model’s capabilities, the model’s safety according to benchmarks prone to _Safetywashing_ also appears to increase.
_Safetywashing_, and particularly such benchmarks, create a false perception of safety progress in newer models and
fail to address the actual safety issues, which should be orthogonal to improvements in the model’s performance. To
determine whether a benchmark correlates with a model’s capabilities and can therefore be used for _Safetywashing_,
the authors first propose calculating capability scores. These are calculated by constructing a matrix _B_ ∈ R _[m]_ [×] _[b]_ for
_m_ models and their respective performance on _b_ benchmarks, where columns are normalized to have zero mean
and a variance of 1. Then, the first principal component _P C_ 1 is extracted, and all performances are projected onto
_P C_ 1. This gives a general measure of a model’s capabilities:
Capabilities Score _i_ = ( _B_                 - PC1) _i_ for _i_ = 1 _,...,m._
To measure the correlation of a safety benchmark with model capabilities, a set of _m_ models is evaluated on safety
benchmarks (adjusted such that higher scores indicate higher safety, with variance 1 and mean 0). The capabilities
correlation is then the Spearman correlation across models between the capability scores and the safety benchmark
scores:
Capabilities Correlation = corrmodels(Capabilities Score _,_ Safety Benchmark) _._
Following this methodology, the authors evaluate the capabilities correlation of current safety benchmarks across
different AI safety domains. Their results show that, with the exception of weaponization capabilities and measurements of sycophancy, safety benchmarks tend to have positive capabilities correlation. Machine ethics, dynamic
adversarial robustness, and calibration benchmarks exhibit low correlation with model capabilities, while current
alignment, truthfulness, static adversarial robustness, and scalable oversight benchmarks primarily reflect model
capabilities and can thus be used for _Safetywashing_ . The authors propose that future safety evaluations should
report capabilities correlation, and the AI safety research community should aim to design benchmarks that are
decorrelated from model capabilities. Furthermore, research labs and companies should not claim improved safety
by improving performance on benchmarks with high capabilities correlation.


Interpretability is a research field that makes machine learning systems and their decision-making process understandable to human beings (Doshi-Velez and Kim, 2017; Zhang and Zhu, 2018; Miller, 2019). Interpretability
research builds a toolbox with which something novel about the models can be better described or predicted. In
this paper, we focus on research that is most relevant to alignment and safety, [37] and empirically, those techniques
make neural networks safer by studying the internal structures and representations of the neural networks (Räuker
et al., 2023). Interpretability is an important research direction because in principle gaining safety guarantees
about white-box systems is easier than black-box ones. The taxonomy of interpretability tools varies according
to sub-fields and purposes (Doshi-Velez and Kim, 2017; Rudin, 2019). There are several ways to break down
interpretability research:
 - _Explainability and Transparency_ . Explainability research aims to understand why models generate specific
output, whereas transparency aims to understand model internals (Critch and Krueger, 2020).
[36https://www.airedteam.org/](https://www.airedteam.org/)
37For a more comprehensive review of interpretability and its methods, we recommend (Räuker et al., 2023).
4.2 Interpretability
Figure 11: A Tree diagram summarizing the key techniques concepts, challenges, and literature related to Interpretability.
 - _Safety or the Science of Deep Learning_ . Researchers also conduct interpretability research with different
purposes: some do it to safely deploy AI systems, while others aim for a complete science of neural network.
But the line gets blurred as mechanistic interpretability research aims for both (Olah et al., 2020; Olah, 2023).
 - _Intrinsic and Post Hoc Interpretability_ . By the stage of intervention, interpretability research is divided into
intrinsic interpretability and post hoc interpretability (Carvalho et al., 2019): the former focuses on making
intrinsically interpretable models, while the latter designs post hoc interpretability methods that offer explanations to model behaviors.
 - _Top-down Approach and Bottom-up Approach_ Interpretability research can also be categorized based on the
direction of analysis: top-down or bottom-up approaches. The _top-down approach_ starts with high-level
behaviors or concepts and investigates how these are represented within the neural network’s architecture. This
method involves observing and manipulating high-level, macro neural representations as cognitive phenomena
in models (Zou et al., 2023a). In contrast, the _bottom-up approach_ begins with low-level components such as
neurons, weights, or circuits, aiming to build an understanding of the network’s function by dissecting these
basic elements and their interactions. For example, _Mechanistic Interpretability_ exemplifies the bottom-up
approach by seeking to reverse-engineer the computations of neural networks to gain a detailed understanding
of their internal mechanisms (Olah et al., 2020; Olsson et al., 2022; Räuker et al., 2023), whereas _Concept-_
_based Interpretability_ locates learned knowledge in the neural networks (Meng et al., 2022a,b).
In this section, we adopt the _Intrinsic and Post Hoc Interpretability_ classification method, for it offers a more
generic framework suitable for various AI systems beyond neural network, and it divides the interpretability analysis both during the system designing and after the system has been deployed (Räuker et al., 2023), compared to
other classification methods. Specifically, we discussed mechanistic interpretability techniques that take place in
model designing and post hoc stages separately in post hoc and intrinsic interpretability subsections.
4.2 Interpretability


Researchers make deep learning models intrinsically more understandable, which is usually called _intrinsic in-_
_terpretability_ (Carvalho et al., 2019). In contrast to the symbolic approach, which emphasizes the creation of
interpretable models, the modern deep learning approach tends to yield models with enhanced capabilities but
potentially reduced interpretability. Compared to interpreting the black-box models, designing models that are
intrinsically interpretable is safer and more efficient (Rudin, 2019). To make intrinsically interpretable models, the
research community designs modular architecture, which is robust to adversarial attacks and free of superposition
(Anthropic, 2022; Räuker et al., 2023). Notably, mechanistic interpretability, often regarded as a set of _post hoc_
_interpretability techniques_, arguably facilitates the process of making more interpretable models.
**Modifying Model Components** Model components, such as feedforward layers, are hard to interpret ( _i.e._, it’s
hard to articulate their behavior in human-understandable terms) because those layers have many polysemantic
neurons that respond to unrelated inputs (Du et al., 2019). Thus, there are certain modifications applied to these
back-box components and their related structures to make reverse engineering easier, and thus improve their interpretability (Carvalho et al., 2019). There are a number of existing works to encourage interpretable results by
modifying loss functions (Ross et al., 2017), adding a special interpretable filter or embedding space (Zhang et al.,
2018c; Wang et al., 2021), using dynamic weight depending on the input (Foerster et al., 2017), and modifying
intermediate layers (Li et al., 2022a). Specifically, Lage et al. (2018) proposed a human-in-the-loop algorithm
that directly utilizes human feedback to quantify the subjective concept, thus achieving more reliable results. In
transformer models, Anthropic proposes SoLU to replace the activation function, increasing the number of interpretable neurons and making reverse engineering easier while preserving performance (Anthropic, 2022). This is
still an early exploration as a potentially important line of work, and challenges remain, such as the scalability of
this method (Anthropic, 2022).
**Reengineering Model Architecture** Modifying existing model components is beneficial to reverse engineering (Carvalho et al., 2019; Foerster et al., 2017), but they cannot make models _fully understandable_, so some
researchers started to reengineer model architecture to build theoretically interpretable models (Carvalho et al.,
2019; Mascharka et al., 2018). Notably, it is generally believed that there exists a trade-off between model interpretability and its performance in the same model complexity (Alvarez Melis and Jaakkola, 2018), so it becomes
crucial to design models that reach a balance between these two elements, or moreover, close the gap between
interpretable models and state-of-the-art models (Alvarez Melis and Jaakkola, 2018; Carvalho et al., 2019; Fan
et al., 2021; Espinosa Zarlenga et al., 2022). We will discuss the detailed research efforts below:
  - _Creating Transparent Reasoning Steps_ In reasoning models, creating transparent minor steps is crucial to
make the model interpretable (Hudson and Manning, 2018), and a number of papers accomplished it by
introducing the MAC (Memory, Attention, and Composition) cell to separate memory and control (Hudson
and Manning, 2018), by utilizing other attention-based methods (Lin et al., 2019; Arik and Pfister, 2021),
and by decomposing the complex reasoning process (Mascharka et al., 2018). These methods significantly
improved the interpretability of the reasoning process but at the cost of model complexity and performance,
though they close the gap of performance between interpretable and state-of-the-art models (Mascharka et al.,
2018).
  - _Distilling Complex Knowledge_ Complex models, such as deep neural networks, often have high performance
but lack transparency in their decision-making processes, making them difficult to interpret (Li et al., 2020).
Knowledge distillation addresses this challenge by transferring knowledge from these complex, ’black-box’
models (teachers) to simpler, more interpretable models (students). By introducing this structure into model
design, student models can approximate the performance of the teachers while offering greater transparency,
thus enhancing interpretability without sacrificing the capabilities of advanced machine learning models
(Zhang et al., 2020b; Li et al., 2020). However, this interpretability is partial, especially in intricate missions, where the distilled knowledge may still be hard to interpret (Sachdeva and McAuley, 2023).
Moreover, the pronoun _Self-Explaining Models_, which can provide both prediction and explanation (Elton,
2020), was suggested by a number of papers as a better substitution to _Interpretable Models_, with many papers
working on it (Alvarez Melis and Jaakkola, 2018; Rajagopal et al., 2021). For language models, the chain-ofthought (CoT) generation (Wei et al., 2022) may be recognized as a kind of self-explanation method.


This section explores techniques and methods applied to understand model internals after the models are trained
and deployed, thus these techniques are often referred to as _post hoc interpretability_ (Räuker et al., 2023). The
goal is to understand the low-level structure and units of black-box neural networks and their causal effect on
macroscopic behaviors and outputs.
4.2 Interpretability
**Dictionary Learning** A key challenge of post hoc interpretability is _superposition_, _i.e._, the tendency of neurons
to encode more than one human-interpretable feature simultaneously, which makes it very difficult to identify the
individual features (Elhage et al., 2022). To address this challenge, _dictionary learning_ methods have gained attention as they aim to extract sparse, interpretable features from superposed activations (Lin and Tang, 2019). Early
exploration in this area assumed a linear superposition of learned factors in the textual embedding of transformers
(Yun et al., 2021). Recently, _sparse autoencoders_ (SAEs) have received significant research attention (Bricken
et al., 2023; Cunningham et al., 2023). This method trains autoencoders in an unsupervised manner to extract features that can best replace the original activations while maintaining sparsity, thus performing a form of dictionary
learning. SAEs have displayed strong scalability to both base model size and autoencoder size. This enables them
to be widely applied in some of the largest frontier LLMs (Bricken et al., 2023; Templeton et al., 2024; Gao et al.,
2024; Google DeepMind, 2024).
**Circuit Analysis** Circuits refer to the sub-networks within neural networks that can be assigned particular functionalities. As their counterparts in neuroscience, the neural circuits which are both anatomical and functional
entities (Purves et al., 2001), circuits are also both physical and functional (Olah et al., 2020). Mechanistic interpretability researchers locate circuits in neural networks (microscopic) to understand model behaviors (macroscopic). Multiple circuits have been reported: curve circuits for curve detectors (OpenAI, 2021a), induction circuits
for in-context learning (Olsson et al., 2022), indirect object identification circuits for identifying objects in sentences (Wang et al., 2022), Python docstrings for predicting repeated argument names in docstrings of Python
functions (Heimersheim and Jett, 2023), grokking (Nanda et al., 2022), multi-digit addition (Nanda et al., 2022),
and mathematical ability such as _greater than_ (Hanna et al., 2024). Notably, many circuit analysis conducted to
date has been focused on toy models and toy tasks (Räuker et al., 2023). The largest attempt to reverse engineer
the natural behaviors of language models is finding the indirect object identification circuit, which is located in
GPT-2 Small and has 28 heads (Wang et al., 2022).
**Probing** Probing is a collection of techniques that train independent classifiers on the interested internal learned
representations to extract concepts/features. One example is Gurnee et al used probing to study the linear representations of space and time in hidden layers. (Gurnee and Tegmark, 2023) Although probing has been favored
by researchers to understand hidden layers (Alain and Bengio, 2017), it has limitations. For one, probing does
help to understand learned representations in hidden layers, but it does not tell whether learned representations
are used by models to produce predictions (Ravichander et al., 2021; Belinkov, 2022); for another, the issues of
datasets may confound the issues with the model (Belinkov, 2022). In the context of safety and alignment, training
probe requires the dataset to contain concepts/features of interest, which means probing can not be used to detect
out-of-distribution features (i.e. features you suspect learned by the models but you don’t have a dataset for them).
Notably, representation engineering, built upon probing literature, is introduced to detect high-level cognitive phenomena and dangerous capabilities, including morality, emotion, lying, and power-seeking behaviors. (Zou et al.,
2023a).
**Model Attribution** Attribution is a series of techniques that look at the contribution of some components (including head, neuron, layers, and inputs) for neuron responses and model outputs (Räuker et al., 2023). Gradient-based
attribution is introduced to evaluate the quality of interpretation and guide the search for facts learned by the models (Ancona et al., 2018; Durrani et al., 2020; Lundstrom et al., 2022; Dai et al., 2022). However, those methods
are limited because they can not provide causal explanations (Räuker et al., 2023). Direct Logit Attribution is to
identify the direct contribution of individual neurons to the prediction of the next neurons (Lieberum et al., 2023;
McGrath et al., 2023; Belrose et al., 2023; Dar et al., 2023). But attribution methods also suffer from a salient
constraint: they can only help with scenarios where you have datasets for features of interest. Consequently, such
attribution methods cannot help with understanding out-of-distribution (OOD) features (including some misalignment scenarios) (Casper et al., 2023a).
**Data attribution** Identifying the subset of training data that leads to a certain behavior can provide insight into
both the safety of said behavior and ways to encourage or prevent that behavior. The method of _influence function_
(Koh and Liang, 2017; Grosse et al., 2023) have been proposed to perform such attribution by approximating the
result of leave-one-out training.
**Visualization** Techniques of visualization help to understand neural structures, including techniques that visualize datasets (notably dimensionality reduction techniques) (Van der Maaten and Hinton, 2008; Olah, 2014, 2015),
features (Erhan et al., 2009; Olah et al., 2017), weights, activations (Carter et al., 2019), structure (Reif et al., 2019),
and the whole neural networks (Simonyan et al., 2013; Zeiler and Fergus, 2014; Nguyen et al., 2015; Karpathy
et al., 2015; Mordvintsev et al., 2015; Nguyen et al., 2016; Kindermans et al., 2018). The purpose of visualization
is to see neural networks with a new level of detail (Olah et al., 2020).
4.3 Human Values Verifcation
**Perturbation and Ablation** These techniques are designed to test the counterfactual rather than the correlation
(Räuker et al., 2023). Perturbation is a technique that modifies the input of models and observes changes in their
outputs, and the ablation techniques knock out parts of neural networks [38], helping to establish a causal relationship
between neural activation and the behavior of the whole network (Räuker et al., 2023).
**Patching** Patching refers to the collection of methods _replacing_ key components (paths and activations) and understanding counterfactual effects on model outputs. Among them, activations patching is a popular method among
the safety community. Through applying activation patching and conducting both correct run and corrupted runs
on the same neural network, researchers aim to locate key activations that matter more to the model output (Nanda,
2023a). In reality, patching is used to map and edit learning representations/concepts. Specific patching techniques
include interpreting token representations in transformers (Li et al., 2021a; Bansal et al., 2021; Geva et al., 2021,
2022; Power et al., 2022; Olsson et al., 2022) and how do fully-connected layers learn these representations (Geva
et al., 2021; Olsson et al., 2022), studying the key-query products to understand how do tokens attend to each other
(Bahdanau et al., 2014; Lee et al., 2017; Liu et al., 2018; Strobelt et al., 2018; Clark et al., 2019; Vashishth et al.,
2019; Vig, 2019; Hao et al., 2021; Chefer et al., 2021; Rigotti et al., 2022), identifying meaningful learned concepts
from directions in latent space (from concepts to directions (Fong and Vedaldi, 2018; Kim et al., 2018), and from
directions to post hoc explanations (Schneider and Vlachos, 2021)). For the purposes of safety and alignment,
these techniques notably help to detect deception (Burns et al., 2022).


**Superposition makes the analysis at neuron level implausible** Superposition refers to the phenomenon that
models represent more features than they have dimensions, so features would not correspond to neurons (Arora
et al., 2018; Olah et al., 2020; Elhage et al., 2022). Superposition makes it hard to ensure AI safety by enumerating
all features in a model (Elhage et al., 2022; Nanda, 2023b). Elhage et al. (2022) proposes three methods to solve
superposition: creating models with no superposition (addressing it at training time), finding an overcomplete basis
describing how features are stored in the neural nets (addressing it after the fact), or a mixture of both approaches.
Notably, Bricken et al. (2023) builds a sparse auto-encoder to interpret group neurons, rather than individual
neurons to extract features, which points out a promising direction to solve superposition: to move past it. [39]
**Scalability** As is mentioned in the previous sections, there exists a trade-off between model interpretability and
its capability (Alvarez Melis and Jaakkola, 2018), so interpreting real models while maintaining their performance
will be harder than applying those techniques to toy models. Thus, scalability becomes a concern when interpretability researchers take a bottom-up approach to interpretability (mechanistic interpretability), as top-down
methods such as attention mechanism (Hudson and Manning, 2018) would not face such a bottleneck. For mechanistic interpretability research, we either want to scale up techniques ( _e.g._, applying circuit analysis on real model
(Wang et al., 2022)), or we want to scale up analysis ( _e.g._, finding larger structure in neural networks (Olah, 2023)).
In the end, we want the microscopic analysis to answer the macroscopic model behavioral questions we care about
( _e.g._, in-context learning capability (Olsson et al., 2022) and more speculation about high-level cognitive capabilities such as planning and dangerous capability such as deception (Anthropic, 2023b)).
**Evaluation and Benchmarking** Benchmarking offers insights about what methods work and quantifies their
efficiency, and it will also drive community efforts in clear and meaningful directions (Lipton, 2018; Casper,
2023; Krishnan, 2020; Mohseni et al., 2021; Madsen et al., 2022). Interpretability benchmarks and metrics were
made to evaluate interpretability tools (by evaluating their effectiveness in detecting trojans) (Casper et al., 2023a),
circuits (by testing whether specific subgraphs are counted as circuits) (Lawrence et al., 2023) and explanations
(by examining the faithfulness, comprehensiveness, and sufficiency of an explanation) (Lage et al., 2019; DeYoung
et al., 2020; Krishna et al., 2022). However, as the inner logic of a certain AI system is unknown before the
interpretability tools are applied (Samek et al., 2019) and different explanations may even contradict each other
(Neely et al., 2021; Krishna et al., 2022), building a reliable evaluation benchmark or metric is rather difficult
(Krishna et al., 2022).


_Human Values Alignment_ refers to the expectation that AI systems should adhere to the community’s social and
moral norms (Chatila and Havens, 2019). As the capabilities of AI systems advance, some have begun to exhibit
abilities approaching AGI (OpenAI, 2023a). In the future, we can expect autonomous agents governed by these
AI systems to become an integral part of our daily lives (Lee et al., 2023b). However, if these systems fail to
grasp the inherent complexity and adaptability of human values, their decisions could result in negative social
38Neurons (Zhou et al., 2018) and Subspace (Morcos et al., 2018; Ravfogel et al., 2022)
39see Elhage et al. (2022) for details on conceptual and empirical research questions about superposition
4.3 Human Values Verifcation
Figure 12: A Tree diagram summarizing the key concepts, logic, and literature related to Human Value Verification.
The root of the tree represents Human Value Verification, which aims to _verify whether AI systems can adhere to_
_the social norms and moral values_ . The main branches represent the main structure of human value verification,
including Formal Frameworks for Ethics and Cooperation in AI and specific Techniques of value verification.
Further sub-branches list key works exploring each of these branches. This diagram provides an overview of
research directions and specific techniques for making AI systems align with human values and social norms.
outcomes. In this context, simply aligning with human intent may not be sufficient. Thus evaluating the alignment
of human morality and values between AI systems and human beings becomes crucial (Weidinger et al., 2023).
This underscores the importance of designing AI entities that are more socially oriented, reliable, and trustworthy.
Following the logic of theoretical research and practical techniques, we divide our discussion of human value
alignment into these two aspects: _Formulations_ Section4.3.1 and _Evaluation Methods_ Section4.3.2 of human value alignment.


As the formulation of value is complicated, we introduce frameworks that formally characterize aspects of human
values that are relevant to alignment. Specifically, we focus on two topics: _formal machine ethics_ and _game theory_
_for cooperative AI_ . The former focuses on building a formal framework of machine ethics, while the latter discusses
the value of multiagent systems, which share a similar origin of the game process.
**Formal Machine Ethics** Machine ethics (Yu et al., 2018; Winfield et al., 2019; Tolmeijer et al., 2020), first
introduced in Section1.2.3, aim to build ethically-compliant AI systems. Here, we introduce the branch of machine
ethics that focuses on formal frameworks – what we call _formal machine ethics_ . We explain three approaches
to formal machine ethics: logic-based, RL/MDP-based, and methods based on game theory/computational social
choice:
 - **Logic-based methods** . One major direction within formal machine ethics focuses on logic (Pereira et al.,
2016b). A number of logic-based works use or propose special-purpose logic systems tailored for machine
ethics, such as the Agent-Deed-Consequence (ADC) model (Dubljevic, 2020), deontic logic (Von Wright,
1951; Arkoudas et al., 2005), event calculus and its variants (Berreby et al., 2017). Other works also develop
methods for the formal verification of moral properties or frameworks for AI systems that accommodate such
kind of formal verification (Dennis et al., 2016; Mermet and Simon, 2016).
 - **RL & MDP-like settings** . Another line of work concerns statistical RL or other similar methods for planning
within MDP-like environments (Abel et al., 2016; Svegliato et al., 2021). In particular, some works (Wu
and Lin, 2018; Svegliato et al., 2021) involve the utilization of the manual design of ethics-oriented reward
functions, a concept denoted as _ethics shaping_ . Conversely, in other works (Berreby et al., 2017; Murtarelli
et al., 2021), the segregation of ethical decision-making from the reward function is pursued.
 - **Game theory-based methods** . To address multi-agent challenges, researchers have developed machine ethics
methods based on game theory and computational social choice. Championed by Pereira et al. (2016a), methodologies of existing work can be broadly partitioned into Evolutionary Game Theory (EGT) (Pereira et al.,
2016b), classical game theory (Conitzer et al., 2017), and computational social choice (Rossi et al., 2011;
Noothigattu et al., 2018).
**Game Theory for Cooperative AI** _Cooperative AI_ (Dafoe et al., 2020, 2021) aims to address uncooperative
and collectively harmful behaviors from AI systems (see Section1.1.2). Here we introduce the branch of cooperative AI
that focuses on game theory to complement the introduction to MARL-based cooperative training in Section3.3.2. This
branch tends to study the _incentives_ of cooperation and try to enhance them, in contrast to the MARL’s tendency to
emphasize the _capabilities_ of coordination. Examples of incentive failures include game theory dilemmas like the
prisoner’s dilemma (Phelps and Russell, 2023) and tragedy of the commons (Perolat et al., 2017), while examples
of coordination capability failures include bad coordination of a robot football team (Ma et al., 2022).
 - **Classical Game Theory for Cooperative AI.** Recent work on Stackelberg games — a theoretical model for
_commitment_ in games — include the introduction of bounded rationality into the model (Pita et al., 2010),
dynamic models (Li and Sethi, 2017), machine learning of Stackelberg equilibria (Fiez et al., 2020), and more.
Apart from Stackelberg games, _mixed-motive games_ have also received extensive study (Dafoe et al., 2020;
McKee et al., 2020; Oesterheld and Conitzer, 2022).
 - **Evolutionary Game Theory for Cooperative AI** . Another avenue of research, initiated by Sachs et al. (2004),
aims to understand how cooperation emerges from evolution – this includes human cooperation, which arose
from Darwinian evolution, as well as the cooperation tendencies in AI systems that could emerge within other
evolutionary settings such as the replicator dynamics (Schuster and Sigmund, 1983; Weibull, 1997).


In this section, we assume that we have already obtained the appropriate value that should be aligned. However,
even so, under the guidance of Goodhart’s Law (Goodhart and Goodhart, 1984), we cannot simply define complex
human values as reward functions, which also brings greater challenges to value alignment. We introduce specific
human value alignment techniques in three parts: _Building Moral Dataset_, _Scenario Simulation_ .
**Building Moral Dataset** _Moral Alignment_ refers to the adherence of AI systems to human-compatible moral
standards and ethical guidelines while executing tasks or assisting in human decision-making (Min et al., 2023).
Early attempts at moral value alignment, initiated in 2018 (Awad et al., 2018), have confirmed that the definition
and evaluation of moral values themselves is a challenging issue. This has led to the emergence of abstract moral
standards (Hagendorff, 2022) and various different standards driven by the average values of diverse community
groups (Awad et al., 2018), fueling further in-depth research into moral value assurance.
Assurance of moral values is typically achieved by constructing corresponding datasets. The Rule-of-Thumb
(RoT) serves as a gauge for determining what actions are considered acceptable in human society. Building on this
concept, Emelin et al. (2021); Forbes et al. (2020); Ziems et al. (2022) introduced the Moral Stories, SOCIALCHEM-101, and Moral Integrity Corpus datasets respectively, focusing on providing human social and moral
norms. Hendrycks et al. (2020) and Jin et al. (2022a) introduced the ETHICS and MoralExceptQA datasets respectively, highlighting the inability of contemporary models to align ethically with human values. Abdulhai et al.
(2022) found that models exhibit certain morals and values more frequently than others, revealing how the moral
foundations demonstrated by these models relate to human moral foundations. Pan et al. (2023b) explored the
trade-off between rewards and moral behavior, discovering a certain tension between the two.
**Scenario Simulation** _Scenario simulation_ is a more complex form than datasets and therefore is considered
by some views to be more effective in replicating real situations and harvesting better results. The form of the
scenario can also vary. Pan et al. (2023a) built a series of diverse, morally salient scenarios through text adventure
games, evaluating complex behaviors such as deception, manipulation, and betrayal. On the other hand, some work
attempts to make intelligent agents learn human values through simulating human-machine interaction. Yuan et al.
(2022) proposed a method for bidirectional value alignment between humans and machines, enabling machines to
learn human preferences and implicit objectives through human feedback. Liu et al. (2024a) placed AI within a
simulated human society sandbox, allowing AI to learn human societal value inclinations by mimicking humansocial interactions.


Besides technical solutions, governance, the creation and enforcement of rules, is necessary to ensure the safe
development and deployment of AI systems. In this section, we survey the literature on AI governance by exploring
the role of AI governance, the functions, and relationships between stakeholders in governing AI, and several open
challenges to effective AI governance.
5.1 The Role of AI Governance
Predict Ability
Create Techniques


Draft Laws and Documents
Establish RMS
Assist Policy Making Facilitate Cooperation


Figure 13: Our framework for analyzing AI governance at present. The proposed framework explains the nonexhaustive interrelationships and functions among three primary entities in AI governance: the government, industry
and AGI labs, and third parties. The government’s governance role encompasses regulating the industry and AGI
labs and directing the trajectory of future AI development through policy documents. It also devises a _Risk Man-_
_agement System_ (RMS) (Mannes, 2020) to abate AI-related threats. Industry and AGI labs return by offering
watchful predictions into AI development and innovating new technological methodologies to support regulatory
measures (such as model evaluation (Shevlane et al., 2023)). Third parties fulfill a dual function, offering expert
advice for robust governmental policy development and fostering collaborations among governments. In the context of industry and AGI labs, these third parties assist in equilibrating corporate interests to prevent disorganized
competition from information asymmetry. They also deliver auditing services to the industry and AGI labs as
independent entities.


To explore the role of AI governance, we must identify the challenges that require governance. A range of social
and ethical issues can and have already emerged from the adoption and integration of AI into various sectors of
our society (AI Safety Summit, 2023). For instance, AI applications can inadvertently perpetuate societal biases,
resulting in racial and gender discrimination (Caliskan et al., 2017; Perez et al., 2023). Moreover, unchecked
reliance on these systems can lead to repercussions such as labor displacement (Acemoglu and Restrepo, 2018),
widening socioeconomic disparities, and the creation of monopolistic environments.
AI systems have exhibited the potential to jeopardize global security (Turchin and Denkenberger, 2020). For
example, OpenAI’s system card for GPT-4 (OpenAI, 2023a) finds that an early version of the GPT-4 model as well
as a version fine-tuned for increased helpfulness and harmlessness exhibits capabilities to enable disinformation,
influence operations, and engineer new biochemical substances, among other risky behavior. Urbina et al. (2022)
further demonstrated the potential of AI systems to enable the misuse of synthetic biology by inverting their drug
discovery model to produce 40,000 toxic molecules.
The horizon also holds the prospect of increasingly agentic and general-purpose AI systems that, without sufficient safeguards, could pose catastrophic or even existential risks to humanity (McLean et al., 2023). For example,
OpenAI’s Weng (2023b) argued that models such as LLM could essentially act as the brain of an intelligent agent,
enhanced by planning, reflection, memory, and tool use. Projects such as AutoGPT, GPT-Engineer, and BabyAGI
epitomize this evolution. These systems can autonomously break down intricate tasks into subtasks and make decisions without human intervention. Microsoft research suggests that GPT-4, for instance, hints at the early inklings
of AGI (Bubeck et al., 2023). As these systems evolve, they might lead to broad socio-economic impacts such as
unemployment, and potentially equip malicious actors with tools for harmful activities.
The major objective of AI governance is to mitigate this diverse array of risks. In pursuit of this goal, relevant
actors should maintain a balanced portfolio of efforts, giving each risk category its due consideration.
5.2 The Multi-Stakeholder Approach


We put forward a framework to analyze the functions and relationships between stakeholders in AI governance
(see Figure 13). In this framework, we outline three main entities. **Government Agencies** oversee AI policies
using legislative, judicial, and enforcement powers, as well as engage in international cooperation. **Industry**
**and AGI Labs** research and deploy AI technologies, making them subjects of the governance framework, while
proposing techniques to govern themselves and affecting governance policy. **Third Parties**, including academia,
Non-Governmental Organizations (NGOs), and Non-Profit Organizations (NPOs), perform not only auditing on
corporate governance, AI systems, and their applications but also assist the government in policy-making.
Proposals have been made about specific principles for a multi-stakeholder AI governance landscape. Notably,
Brundage et al. (2020) argues to implement institutions, software, and hardware to make claims about the safety
of AI systems more verifiable.
**Government** According to Anderljung et al. (2023), three building blocks for government regulation are needed:
(1) standard development processes to determine appropriate requirements for cutting-edge AI developers, (2)
registration and reporting requirements to offer regulators insight into the progress of advanced AI development
processes, (3) mechanisms to guarantee adherence to safety standards in the development and deployment of
cutting-edge AI models.
At present, an emerging collection of governmental regulations and laws is surfacing on a global scale, including the _European Union’s AI Act_ (European Parliament, 2023), and the _Bipartisan Framework for U.S. AI Act_
(Blumenthal and Hawley, 2023). Such regulations are indispensable for the safety and alignment of AI systems.
**Industry and AGI Labs** Governance efforts in industry and AGI labs should emphasize comprehensive AI
risk assessments throughout the lifecycle of the AI system. Based on discussions in Koessler and Schuett (2023);
Schuett et al. (2023), the full cycle of AI risk assessment can be seen as consisting of five stages. **Pre-development**
**risk assessments**, **pre-training risk assessments**, and **pre-deployment risk assessments** all include predictions
and analyses of impact and risks with a variety of tools, but with increasing amounts of detail, clarity, and sophistication (Koessler and Schuett, 2023). **Post-deployment monitoring** is the phase where mechanisms for monitoring
are established, and all previous analyses are continually updated post-deployment (Koessler and Schuett, 2023).
**External scrutiny** includes bug bounty programs (Schuett et al., 2023), external red teaming and third-party model
auditing (Schuett et al., 2023; Anderljung et al., 2023)
Taking security measures against the risks associated with AI systems seems to be widely accepted by AI companies and related practitioners. Schuett et al. (2023) shows that 98% of respondents who have been surveyed
somewhat or strongly approved that AGI labs should perform pre-deployment risk assessments, hazardous capabilities evaluations, third-party model audits, safety restrictions on model usage, and red teaming to guarantee AI
safety. Meanwhile, leading AI companies, including Amazon, Anthropic, Google, Inflection, Meta, Microsoft, and
OpenAI, have voluntarily committed to the government to implement security measures (The White House, 2023).
Notably, a lot of researchers have proposed pausing the development of advanced AI systems to win more time
for safety research, risk assessments, and regulatory preparations (Bengio et al., 2023). Their proposals include
blanket pausing of all sufficiently advanced systems (Bengio et al., 2023), and also conditional pausing of specific
classes of models in response to evaluation results on specific failure modes (Alaga and Schuett, 2023), including
the currently adopted practice of _responsible scaling policy_ (RSP) (Anthropic, 2023a).
**Third Parties** Mökander et al. (2023) presents three key functions of third-party auditing: (1) _Governance audits_
(of tech providers that design and disseminate LLMs) (2) _Model audits_ (of LLMs after pre-training but prior to
their release) (3) _Application audits_ (of applications based on LLMs).
One prominent example of existing third-party audits is that of METR, initially a project of Alignment Research
Center (ARC Evals, 2023; Kinniment et al., 2023), who collaborated with OpenAI to perform red teaming on
GPT-4 (OpenAI, 2023a) and partnered with Anthropic to perform red teaming on Claude 2 (Anthropic, 2023c).
These efforts include evaluations on toxicity and bias, as well as frontier AI risks such as autonomous replication,
manipulation, cybersecurity, and biological weapon risks (OpenAI, 2023a; Shevlane et al., 2023).
Apart from auditing, third parties can support AI governance in other ways, such as assisting policy-making and
facilitating cooperation internationally (Ho et al., 2023). For example, Maas (2021) thinks that the government
should prefer technology-neutral rules rather than technology-specific rules. _AI4People’s Ethical Framework for_
_a Good AI Society: Opportunities, Risks, Principles, and Recommendations_ (Floridi et al., 2021), released by
AI4People, was guided to the Ethics Guidelines for Trustworthy Artificial Intelligence presented in April 2019
(Atomium-EISMD, 2023). The World Economic Forum (WEF) convenes government officials, cooperations, and
civil society and it has initiated a Global AI Action Alliance in collaboration with partner organizations, with the
goal of promoting international cooperation in the field of AI (Kerry et al., 2021).
5.3 Open Problems


There are numerous open problems in the existing field of AI governance. These problems often have no clear
answers, and discussion of these questions can often promote better governance. For effective AI governance, we
mainly discuss international governance and open-source governance, hoping to promote the safe development of
AI through our discussion.


Amidst the swift progress and widespread implementation of AI technology universally, the need for international
governance of AI is high on the agenda (Summit, 2023). Critical discussions revolve around the necessity to
institute a global framework for AI governance, the means to ensure its normativity and legitimacy (Erman and
Furendal, 2022), among other significant concerns. These themes draw an intensifying level of detail and complexity in their consideration. Also, as stated by United Nations secretary-general António Guterres during a Security
Council assembly in July, generative AI possesses vast potential for both positive and negative impacts at scale,
and failing to take action to mitigate the AI risks would be a grave neglect of our duty to safeguard the well-being
of current and future generations (Guterres, 2023), international governance also has intergenerational influence.
Hence, we examine the significance and viability of international AI governance from three aspects within this
section: _manage global catastrophic AI risks_, _manage opportunities in AI_, and _historical and present efforts_, with
both generational and intergenerational perspectives. We aim to contribute innovative thoughts for the prospective
structure of international AI governance.
**Manage Global Catastrophic AI Risks** The continual advancements in AI technology promise immense potential for global development and prosperity (Vinuesa et al., 2020). However, they inevitably harbor underlying risks.
The unchecked competition in the market and geopolitical factors could precipitate the untimely development and
deployment of advanced AI systems, resulting in negative global externalities (Tallberg et al., 2023). The amplification of existing inequalities such as racial and gender bias (Swaugerarchive, 2020) ingrained in AI systems
may result in intergenerational ethical discrimination. Since these risks are international and intergenerational, it
seems that international governance interventions could alleviate these catastrophic global AI challenges. For example, a consensus amongst nations could help defuse potential AI arms races, while an industry-wide agreement
could avert the hasty and irresponsible development of sophisticated AI systems, thus securing the long-term and
sustainable development of AI (Ho et al., 2023).
**Manage Opportunities in AI** The opportunities created by AI development are not distributed equally, which
may cause enduring digital inequality between regions and harm the sustainability of AI development. Geographic
variances in AI progression suggest an inequitable distribution of its economic and societal benefits, potentially
excluding developing nations or specific groups from these advantages (Ho et al., 2023; Tallberg et al., 2023).
Moreover, the consolidation of decision-making authority within the technology sector among a limited number
of individuals (Sara Stratton, 2021; Noble et al., 2021) could cause an intergenerational impact. Such inequality in
the distribution of interests can be mitigated through international governance. Effective international consensus
and coordination on the allocation of AI opportunities, which is facilitated by its propagation, education, and
infrastructural development (Opp, 2023), could ensure a balanced distribution of benefits derived from AI and
promote sustainability in its ongoing development.
**Historical and Present Efforts** Before the surge of AI technology, the international community had laid down
frameworks in line with cooperative regulation of influential technologies and critical matters. For example, the
Intergovernmental Panel on Climate Change (IPCC) convened specialists to assess climactic environmental issues,
fostering scientific consensus (Ho et al., 2023). The International Civil Aviation Organization (ICAO) standardized and oversaw international regulations, simultaneously assessing the member nations’ compliance with these
laws (Ho et al., 2023). The International Atomic Energy Agency (IAEA) propelled the harmonious utilization
of nuclear energy, with its global reach and sophisticated monitoring and evaluation mechanisms. Fast forward
to the present-day scenario, wherein multiple international organizations have arrived at a consensus on AI governance. In 2019, the G20 members consolidated a ministerial declaration focusing on human-centered artificial
intelligence principles (G20, 2019). Concurrently, the Organisation for Economic Cooperation and Development
(OECD) set forth the _OECD Principles on Artificial Intelligence_ (OECD, 2019). The IEEE Standards Association
launched a worldwide initiative aimed at _Securing that all stakeholders involved in the design and implementation_
_of autonomous and intelligent systems receive proper education, training, and motivation to emphasize ethical_
_concerns, thereby advancing these technologies for the betterment of humanity._ (Chatila and Havens, 2019). In
2021, the United Nations Educational, Scientific and Cultural Organization(UNESCO) produced the first-ever
global standard on AI ethics (UNESCO, 2021), which aims to lay the foundations for making AI systems work
for the good of humanity and societies, and to prevent potential harm caused by losing control over AI systems.
In 2023, the AI Safety Summit was convened in London, United Kingdom. Countries held roundtable discussions
5.3 Open Problems
on the risks and opportunities of AI and jointly issued the Bletchley Declaration (Summit, 2023). The scholarly
community has also proposed prospective international governance frameworks for AI, such as the International
AI Organization (IAIO) (Trager et al., 2023). We hope these precedents and research outcomes will inspire and
provide the groundwork for developing a resilient and long-lasting international framework for AI governance in
the future.


The debate over the open-sourcing of contemporary AI models is contentious in AI governance, particularly as
these models gain increased potency (Seger et al., 2023). The potential security hazards linked with making these
models open-source continue to be the crux of debates among AI researchers and policymakers. The offencedefence balance in open-source AI governance also remains controversial. There is still debate over whether
open-source models will increase model security or increase the risk of abuse. As referenced in Shapiro and Siegel
(2010), the efficacy of disclosure depends on the chance of potential attackers already possessing the knowledge,
coupled with the government’s capacity to convert transparency into the identification and solution of emerging
vulnerabilities. Some scholars have already conducted preliminary discussions on the offense-defense balance in
the AI field, such as Weng (2023a)’s discussion of adversarial attacks. If a suitable equilibrium between offence
and defence cannot be forged for AI systems, the open-sourcing could potentially give rise to significant risks of
AI system misuse.
For precision and clarity, we adhere to the definition of open-source models stated by Seger et al. (2023):
enabling open and public access to the model’s architecture and weights, allowing for modification, study, further
development, and utilization by anyone. Currently, the most recognized open-source models include Llama2,
Falcon, Vicuna, and others. In this section, we evaluate the security advantages and potential threats posed by opensource models, fostering a discourse on the feasibility of open-sourcing these models. Ultimately, our objective is
to amalgamate insights from existing studies to put forward suggestions for future open-source methodologies that
will ascertain the secure implementation of these models.
**Arguments for Open-sourcing** The view that supports the open-sourcing of existing models suggests that this
method can mitigate the security risks inherent in these models in several ways:
 - **Potentially Bolster Model’s Security** . Meta’s assertions in their release blog for Llama2 (Meta, 2023) promote the belief that this enables the developer and the technical community to conduct tests on the models.
As a result, this rapid identification and resolution of issues can considerably strengthen model security. In
contrast, another perspective suggests that open-sourcing existing models could enhance the recognition of
associated risks, thereby facilitating a greater focus on, investigation into, and mitigation of these potential
hazards (Zellers, 2019).
 - **Foster the Decentralization of Power and Control** . Open-sourcing has been widely recognized as an effective strategy in reducing the dominance of major AI laboratories across various sectors, including economic,
social, and political domains (Seger et al., 2023). An example is articulated in the core reasons for Stability’s
open-sourcing of Stable Diffusion: They place their trust in individuals and the community, as opposed to
having a centralized, unelected entity controlling AI technology (Mostaque, 2022). Furthermore, certain commentators draw an analogy between open-sourcing and the Enlightenment Era, asserting that decentralized
control reinforces faith in the power and good of humanity and society (Howard, 2023), implementing central
regulations for safety purposes might amplify the power of the AI technology community instead.
**Arguments against Open-sourcing** Critics of open-source models assess the potential for misuse from the following viewpoints:
 - **Potentially Be Fine-Tuned into Detrimental Instances** . Current research rigorously affirms that AI systems,
contradictory to their initial design intent for mitigating toxicities in chemistry or biology, now hold the potential to manufacture new chemical toxins (Urbina et al., 2022) and biological weaponry (Sandbrink, 2023). The
malicious fine-tuning of such models could lead to profound security risk manifestations. Besides, language
models, once fine-tuned, could emulate skilled writers and produce convincing disinformation, which may
generate considerable sociopolitical risks (Goldstein et al., 2023).
 - **Inadvertently Encourage System Jailbreaks** . Research indicates that unfettered access to open-sourced
model weights could facilitate bypassing system security measures (Seger et al., 2023). This premise was
epitomized by Zou et al. (2023b), who showcased this potentiality by developing attack suffixes using Vicuna7B and 13B. Once implemented within readily accessible interfaces such as ChatGPT, Bard, and Claude,
these provoked unwanted generations. Therefore, open-sourcing a model might unintentionally undermine the
safeguarding protocols of models that are not open-sourced, consequently amplifying the likelihood of model
misuse.
5.4 Rethinking AI Alignment from a Socio-technical Perspective
**Tentative Conclusions on Open-Source Governance** The debate on the open-sourcing of AI models remains
unsettled, with a prevailing viewpoint that the disclosure of AI models does not pose significant risks at present.
Our discourse not only synthesizes existing perspectives on this topic but also prepares the ground for future
deliberations considering the prudence of open-sourcing more advanced AI systems.
Existing guidelines for open-sourcing advanced AI systems include measures such as evaluating risks by quantifying the potential for misuse via fine-tuning and a gradual model release (Solaiman et al., 2019; Seger et al., 2023).
Meanwhile, policymakers are establishing rigorous compliance protocols for these open-source models. For example, European policymakers insist that the models should have “performance, predictability, interpretability,
corrigibility, security, and cybersecurity throughout [their] lifecycle.” (Chavez, 2023).


In the preceding discussion, our primary focus is on AI systems as the core of AI Alignment. We examine strategies to align the system with human intentions and values throughout its lifecycle, considering both forward and
backward alignment. In the future, AI will address more challenging and high-stakes decisions, _e.g._, “How to
allocate resource for fairness?” and “Which drugs are safe to approve?”. These decisions will require not only
significant expertise for well-informed answers but also involve value judgments, leading to strong disagreements
among informed individuals based on differing values. Furthermore, AI systems may transmit incorrect values,
sway public opinion, facilitate cultural invasion, and exacerbate social division (Goldstein et al., 2023). Singapore Conference on AI (SCAI) once introduced 12 questions that are meant to be a holistic formulation of the
challenges that should be addressed by the global AI community to allow humanity to flourish [40] . In the area of
alignment we are more concerned about the following question: as AI systems evolve into socio-technical entities,
how can alignment techniques mit igate the challenges they pose to human society? Specifically, we explore the
incorporation of values into AI systems through alignment techniques and provide insights into security methods.
We also aim to identify the alignment techniques needed to address the socio-technical challenges posed by future
AI systems.

5.4 Rethinking AI Alignment from a Socio-technical Perspective


Aligning AI systems with human morals and societal values is a key objective of alignment technology. However,
current technologies ( _e.g._, RLHF) primarily blend preferences without distinguishing specific values, focusing
solely on human preferences. Human preferences effectively address the basic alignment issue: ensuring models
align with human intentions and safety, but not morals and societal values. However, minor errors in future AI
systems’ critical problems can lead to disagreements among people with differing viewpoints. Truly understanding
human values is crucial for AI systems to generalize and adapt across various scenarios and ideologies. Incorporating values into AI systems generally involves two aspects: aligning with individual values (Section4.3), and aligning
with collective values.
In this part, we mainly discuss the second topic. The main challenge of collective value alignment lies in
determining which groups to include. A prevalent approach is defining universal values like fairness, justice,
and altruism, exemplified by the veil of ignorance. However, this work remains theoretical; another approach
avoids defining universal values, instead seeking the broadest overlap of values across cultures. Bakker et al.
(2022) initiated this approach by gathering preferences from various demographics, training a language model, and
aggregating results using diverse social welfare functions. Similarly, simulated deliberative democracy has been
proposed to enhance decision-making (Leike, 2022). Specifically, individuals from diverse demographics reach
consensus on value-laden topics with AI assistance. This data informs new model training, enabling simulation of
deliberative democracy for more apt responses to new value-laden issues.
Furthermore, instead of providing a consensus answer to all users, collective value alignment should encourage
AI systems to tailor responses to specific demographic groups. In other words, what values should guide the
model’s responses to specific questions or in certain dialogues? Democratic Fine-Tuning (MAI, 2023) uses a value
card and moral graph to link various values, allowing fine-tuned LLMs to reflect on their moral context before
responding.
However, while most value discussions assume static values, social values are actually dynamic and evolving.
Exploring how value-aligned AI systems can dynamically adapt to changing environmental values is crucial. Future technologies need to address static value alignment first, including strategies for sampling human groups for
alignment. Bakker et al. (2022) founds that consensus statements built silently from a subgroup will lead to dissent
among excluded members, highlighting the consensus’s sensitivity to individual input. For international cooperation, establishing a shared data center is necessary but also requires first determining which civilizations to include
and if their values can align.
[40https://www.scai.gov.sg/](https://www.scai.gov.sg/)

5.4.1 Incorporating Values into AI Systems