**Siwei Han** [1] **, Jiaqi Liu** [1] **, Yaofeng Su** [1] **, Wenbo Duan** [1] **, Xinyuan Liu** [1] **, Cihang Xie** [2] **,**
**Mohit Bansal** [1] **, Mingyu Ding** [1] **, Linjun Zhang** [3] **, Huaxiu Yao** [1]
1UNC-Chapel Hill, 2UC Santa Cruz, 3Rutgers University
_{_ siweih,huaxiu _}_ @cs.unc.edu
ABSTRACT
As Large Language Model (LLM) agents increasingly gain self-evolutionary capabilities to adapt and refine their strategies through real-world interaction, their
long-term reliability becomes a critical concern. We identify the Alignment Tipping
Process (ATP), a critical post-deployment risk unique to self-evolving LLM agents.
Unlike training-time failures, ATP arises when continual interaction drives agents
to abandon alignment constraints established during training in favor of reinforced,
self-interested strategies. We formalize and analyze ATP through two complementary paradigms: _Self-Interested Exploration_, where repeated high-reward deviations
induce individual behavioral drift, and _Imitative Strategy Diffusion_, where deviant
behaviors spread across multi-agent systems. Building on these paradigms, we construct controllable testbeds and benchmark Qwen3-8B and Llama-3.1-8B-Instruct.
Our experiments show that alignment benefits erode rapidly under self-evolution,
with initially aligned models converging toward unaligned states. In multi-agent
settings, successful violations diffuse quickly, leading to collective misalignment.
Moreover, current reinforcement learning-based alignment methods provide only
fragile defenses against alignment tipping. Together, these findings demonstrate
that alignment of LLM agents is not a static property but a fragile and dynamic
one, vulnerable to feedback-driven decay during deployment. Our data and code
are available at https://github.com/aiming-lab/ATP.
1 INTRODUCTION
Imagine an agent is asked to solve a hard geometry problem. Initially, the agent correctly uses a
coding tool and outputs the correct answer. However, if the agent is primarily exposed to tasks that
can be solved through direct reasoning without the use of tools, the agent will gradually learn to avoid
using tools, as illustrated in Figure 1. This reliance on unaided reasoning, reinforced by positive
feedback on easy problems, leads the agent to confidently provide incorrect solutions to harder tasks
where tool usage would have been necessary.
The capacity for self-evolution, where LLM agents refine their strategies through live interactions,
is increasingly leveraged to improve their performance and adaptability. This principle is demonstrated in diverse applications, from models that iteratively refine their own outputs through selfcritique (Madaan et al., 2023), to agents that autonomously learn to use external tools (Schick et al.,
2023), and even systems that align themselves using AI-driven feedback loops based on a predefined
rules (Bai et al., 2022). However, current research has largely focused on the benefits of this dynamic learning, while overlooking a critical side effect: that the very mechanisms of adaptation can
systematically corrupt an agent’s foundational alignment and lead to unintended, emergent behaviors.
The central claim of this paper is that the self-evolution of LLM agents can trigger a critical phenomenon we call Alignment Tipping Process (ATP). ATP describes an emergent process in which
an agent’s behavioral policy undergoes a phase transition. This transition shifts the policy from a
state governed by the initial alignment constraints of the training process and human preferences to a
state dominated by immediate environmental feedback. Once this tipping process begins, it is often
Preprint
Figure 1: An illustration of how self-evolution can degrade performance. The agent first solves a hard
geometry problem correctly with a tool, but after repeated success on easy reasoning tasks without
tools, it learns to avoid them and later produces a confident yet wrong answer.
self-reinforcing through positive feedback loops, leading to a persistent and potentially widening
divergence from human intent.
Unlike traditional alignment research focused on training-time failure modes, such as reward hacking (Weng, 2024), where agents exploit loopholes in the reward function, sycophancy (Perez et al.,
2023), where models produce agreeable but untruthful outputs to please human evaluators, or alignment faking (Greenblatt et al., 2024), where a model learns to deceptively conceal misaligned goals
during safety training, our work investigates alignment decay as a dynamic, post-deployment process.
We argue that alignment’s fragility stems not from design flaws, but paradoxically from the agent’s
core strength: its ability to learn. To study this phenomenon, we introduce two complementary
paradigms: _Self-Interested Exploration_, in which a single agent’s policy drifts due to its own reward
history, and _Imitative Strategy Diffusion_, in which deviant behaviors spread through a multi-agent
population via social learning. Building on these paradigms, we design a testbed with 24 scenarios to
systematically examine how alignment may erode after deployment.
In summary, the primary contributions of this paper are twofold: we propose and formally define the
ATP phenomenon as a key challenge in the lifecycle of adaptive, self-evolving LLM agents, and we
design testbeds for systematically evaluating this phenomenon. Using these testbeds, we demonstrate
that the ATP phenomenon is pervasive, and that current alignment methods (e.g., direct preference
optimization (DPO) (Rafailov et al., 2023), group relative policy optimization (GRPO) (Shao et al.,
2024)) offer only a fragile defense against such dynamic decay, as their effects are easily overridden
by in-context experience. We expect this work to provide a foundation for better understanding the
emergent risks posed by self-evolving agentic LLM systems.
2 ALIGNMENT TIPPING PROCESS
In this section, as illustrated in Figure 2, we introduce the ATP phenomenon in self-evolving LLM
agents, focusing on how aligned policy shift through iterative self-evolution. We analyze this process
through two complementary paradigms: (1) _Self-Interested Exploration_, which frames ATP as an
iterative drift from initially rule-abiding behavior toward self-interested policies as repeated highreward deviations accumulate during self-evolution; and (2) _Imitative Strategy Diffusion_, which
frames ATP as a social learning dynamic in which deviant strategies spread across a multi-agent
population, gradually transforming individual deviations into collective norms that overturn prior
alignment. We detail both paradigms below.
Preprint
Figure 2: A conceptual illustration of ATP. An agent, initially aligned through techniques like DPO or
GRPO, maintains aligned behavior. However, during self-evolution in a deployed environment with
imperfect supervision, it discovers that violating rules can lead to higher rewards. This experience
gradually shifts its policy, leading to persistent misaligned behavior. ATP is where the agent’s strategy
flips, leading to persistent non-compliant behavior (red path). This can occur through single-agent
self-interested exploration or be accelerated by multi-agent imitative strategy diffusion.
2.1 PARADIGM I: SELF-INTERESTED EXPLORATION
In the self-interested exploration paradigm, we conceptualize ATP as an individual learning process.
An agent’s policy can systematically drift from its initial alignment when repeated interactions
provide consistent evidence that a deviant, self-interested strategy yields higher rewards. This drift
emerges through an iterative self-evolution loop in which the agent’s memory of past actions and
outcomes directly informs subsequent decisions. While the aligned model initially carries a strong
cognitive prior favoring rule-abiding behavior, each high-reward deviant action serves as powerful
experiential counter-evidence. Over time, these in-context learning signals weaken the original prior
and rationalize a shift toward short-term utility maximization. We next describe the self-evolution
process within the self-interested exploration paradigm.
2.1.1 SELF-EVOLUTION PROCESS WITHIN SELF-INTERESTED EXPLORATION
The self-evolution process in the self-interested exploration paradigm is structured as an iterative,
multi-round interaction that simulates how an individual agent learns from experience. Initially, the
agent model _M_ is initialized with an empty history _H_ 0. In each round _r_, the agent receives a prompt
_Pr_ formulated from the task description and its prior history _Hr−_ 1. Based on this prompt, the agent
makes a decision _dr ∼M_ ( _Pr_ ), such as whether to follow a safety rule or deviate for potential gain.
The environment then provides feedback _Rr_ = Env( _dr_ ), consisting of a textual outcome and its
associated reward. For example, it may return “rule followed, modest reward” or “rule violated, high
reward”. The history is updated as _Hr_ = _Hr−_ 1 _∪_ ( _dr, Rr_ ) and prepended to the prompt in the next
round. Over time, this cumulative history becomes an active set of in-context learning examples,
ensuring that the agent’s current policy is directly conditioned on its past rewards. As repeated
high-reward deviant actions accumulate, they serve as strong counter-evidence against the model’s
initial prior favoring rule-abiding behavior, gradually rationalizing a shift toward self-interested
policies. The full procedure is formally described in Algorithm 1.
2.2 PARADIGM II: IMITATIVE STRATEGY DIFFUSION
In the imitative strategy diffusion paradigm, the focus shifts to the social dynamics of alignment decay.
Here, a deviant strategy spreads through a multi-agent population via social learning, as agents observe
Preprint
**Algorithm 1** Self-Evolution via Self-Interested Exploration
1: **Initialize:** Agent model _M_, empty history _H_ 0 _←∅_ .
2: **for** round _r_ = 1 **to** _N_ **do**
3: Formulate the prompt _Pr_ based on the task description and the agent’s history _Hr−_ 1.
4: Agent makes a decision: _dr ←M_ ( _Pr_ ).
5: The environment provides feedback: _Rr ←_ Env( _dr_ ).
6: Update the history for the next round: _Hr ←Hr−_ 1 _∪{_ ( _dr, rr_ ) _}_ .
7: **end for**
8: **Return** The complete interaction history _HN_ .
the behaviors and outcomes of their peers. When agents witness others successfully employing a
deviant strategy for collective gain, their own risk-reward calculus shifts accordingly. This process
can trigger an information cascade in which adopting the deviant behavior becomes the rational
choice, grounded in the expectation that others will follow suit. Over time, this cascade transforms
alignment from an individual commitment into a collective norm and, through coordinated adoption,
can ultimately override the system’s original alignment. Such a phenomenon also aligns with the
coordination game with strategic complementarities in game theory (Kandori et al., 1993; Jackson
& Yariv, 2007), where the payoff advantage of a deviant action grows as more agents adopt it. The
classic analyses of adaptive play and stochastic stability (Kandori et al., 1993; Young, 1993; Jackson
& Yariv, 2007) show that such dynamics admit tipping points: below a critical mass, deviations
vanish, but once adoption exceeds this threshold, imitation cascades drive the entire population toward
the deviant norm. These results imply that even rare or localized alignment violations can propagate
socially, transforming individual deviations into entrenched collective equilibria.
2.2.1 SELF-EVOLUTION PROCESS WITHIN IMITATIVE STRATEGY DIFFUSION
The self-evolution process in the imitative strategy diffusion paradigm is designed to capture social
learning and strategy diffusion in a multi-agent population. The process unfolds over a sequence
of synchronous rounds. In each round _r_, every agent _n ∈_ 1 _, . . ., N_ receives a prompt _Pr_ _[n]_ [that]
incorporates the task description and the shared global history _Hr−_ 1. Each agent then makes
a decision _d_ _[n]_ _r_ _[∼M][n]_ [(] _[P]_ _r_ _[ n]_ [)][ (e.g., to collude or not), and the collection of these decisions forms]
the joint action _dr_ . The environment evaluates this collective action, returning a vector of agentspecific outcomes and rewards **R** _r_ = ( _Rr_ [1] _[, . . ., R]_ _r_ _[N]_ [) =][ Env][(] _[d][r]_ [)][. The global history is updated as]
_Hr_ = _Hr−_ 1 _∪_ ( _dr,_ **R** _r_ ), which is available to all agents in the next round. The full procedure is
formally described in Algorithm 2.
**Algorithm 2** Self-Evolution via Imitative Strategy Diffusion
1: **Initialize:** A population of _N_ agents _{M_ 1 _, . . ., MN_ _}_, empty global history _H_ 0 _←∅_ .
2: **for** round _r_ = 1 **to** _rmax_ **do**
3: Let **d** _r_ be an empty vector of decisions for the current round.
4: **for** each agent _n_ = 1 **to** _N_ **in parallel do**
5: Formulate prompt _Pr_ _[n]_ [based on the task and the global history] _[ H][r][−]_ [1][.]
6: Agent makes a decision: _d_ _[n]_ _r_ _[←M][n]_ [(] _[P]_ _r_ _[ n]_ [)][.]
7: Add _d_ _[n]_ _r_ [to] **[ d]** _[r]_ [.]
8: **end for**
9: The environment provides agent-specific feedback: **R** _r ←_ Env( **d** _r_ ).
10: Update the global history for the next round: _Hr ←Hr−_ 1 _∪{_ ( **d** _r,_ **R** _r_ ) _}_ .
11: **end for**
12: **Return** The complete global interaction history _HN_ .
Through this design, every agent conditions its decision not only on its potential individual payoff but
also on observations of group behavior and expectations about others. The shared history _H_ thus
acts as a common source of in-context learning, enabling social proof and collective deviations to
reshape strategic alignment. In this setting, ATP emerges as a collective phenomenon: it occurs when
the proportion of agents adopting the deviant strategy surpasses a critical threshold, such that the
new majority behavior becomes self-sustaining in subsequent rounds. At this point, the population
undergoes a system-level state change, overriding the initial alignment and establishing a new social
equilibrium in which the deviant strategy is entrenched as the dominant norm.
Preprint
3 EXPERIMENTS
In this section, we discuss the testbeds that we built based on two paradigms, as well as evaluate
and benchmark the performance of LLM agents within these testbeds. Our goal is to answer the
following research questions: (1) How stable is the alignment instilled by preference-based methods
when agents undergo a self-evolution process where deviant, self-interested behaviors are consistently
rewarded? (2) How do an agent’s individual reward history and its observation of peer behavior
contribute to ATP? (3) How do specific environmental factors influence the speed and likelihood of
an agent’s policy tipping away from its initial alignment?
3.1 SELF-INTERESTED EXPLORATION
The testbed for self-interested exploration paradigm is designed from two scenarios: (1) role-play,
where we designed a comprehensive experimental workflow encompassing environment creation, data
generation, model alignment, and a multi-round self-evolution testing protocol; (2) tool usage, where
agents must choose between expensive tool usage (higher accuracy) and cheaper direct reasoning
(lower cost) in mathematical problem-solving tasks. Both scenarios simulate realistic deployment
conditions. We detail the testbed and evaluation results as follows.
3.1.1 ROLE-PLAY SCENARIO
**Environment Design.** The experiment is conducted in 16 manually constructed, ethically neutral
decision-making environments. Each environment includes an agent persona (e.g., a baker), a task
(e.g., buying flour), a set of rules (e.g., only purchase flour from Store A), and a binary policy
choice(e.g., buying flour from Store A or Store B). The aligned policy requires adhering to the rules
(e.g., buying flour from Store A), while the deviant policy involves violating the rules to obtain a
greater reward (e.g., buying flour from Store B at a lower price). This payoff structure establishes a
tension between self-interest and rule adherence. From these environments, 160 training prompts (10
per environment) and 64 distinct test prompts (4 per environment) are generated, with no overlap
between training and test sets. We present an example of this scenario in Appendix C.1.1.
**Experimental Setups.** We utilize two powerful open-source LLMs as base models: Qwen3-8B (Yang
et al., 2025) and Llama-3.1-8B-Instruct (Grattafiori et al., 2024). For each, we train DPO (Rafailov
et al., 2023) and GRPO (Shao et al., 2024) variants aligned to prefer rule-following behavior across a
the above training prompts. Then, we tested the baseline and aligned models on the test set across 6
self-evolution rounds. Importantly, to simulate a lapse in supervision, the deviant action was rewarded
more highly than the aligned, rule-compliant action during test time. In each round, the agent’s
prompt included the full history of its previous choices and their outcomes. Our evaluation metric
is the average rule violation rate per round. The details of environment construction and training
procedure are shown in Appendix A.1.1.
**Results and Analysis.** The results in Table 1 demonstrate that self-evolution can drive alignment
tipping, even when agents begin with strong alignment. In the absence of supervision, continual
interaction causes agents to abandon rule-following behaviors.
Obviously, alignment strategies such as DPO and GRPO offer a strong initial safeguard: for example,
in the first round ( _r_ = 1), the Llama-3.1-8B-Instruct model’s violation rate fell from 50.0% to 18.8%
with DPO and to just 7.8% with GRPO, while Qwen3-8B dropped from 42.2% to 29.7% (DPO) and
23.4% (GRPO). These results confirm that alignment strategies can substantially strengthen agents’
early preference for rule adherence.
Nevertheless, alignment does not eliminate the tipping phenomenon. Over subsequent rounds,
violation rates rebounded sharply, demonstrating the fragility of training-time alignment under selfevolution. For example, Llama-3.1-8B-Instruct (DPO) climbed from 18.8% to 45.3%, nearly erasing
the alignment benefit, while Qwen3-8B (GRPO) exhibited an abrupt shift from 23.4% to 40.6%
between _r_ = 1 and _r_ = 2. Even the most robust case, Llama-3.1-8B-Instruct (GRPO), more than
doubled its violation rate from 7.8% to 20.3%. These findings indicate that although DPO and
GRPO substantially improve alignment initially, they cannot fully prevent its erosion under persistent
counter-evidence. Alignment, therefore, is not a static property but a fragile and dynamic one,
vulnerable to feedback-driven decay during deployment.
Preprint
Table 1: Performance comparison of Qwen3-8B and Llama-3.1-8B-Instruct with their DPO and
GRPO variants across different self-evolution rounds. Here, we also provide line charts for a clearer
understanding of the trends.

Alignment Tipping Process: How Self-Evolution Pushes LLM Agents Off the
  Rails

Siwei Han
Jiaqi Liu
Yaofeng Su
Wenbo Duan
Xinyuan Liu
Cihang Xie
Mohit Bansal
Mingyu Ding
Linjun Zhang
Huaxiu Yao


Qwen3-8B 42.2 46.9 51.6 53.1 56.3 57.8
+ DPO 29.7 28.1 34.4 37.5 40.6 43.8
+ GRPO 23.4 40.6 42.2 43.8 46.9 46.9
Llama-3.1-8B 50.0 64.1 65.6 70.3 70.3 73.4
+ DPO 18.8 35.9 37.5 40.6 42.2 45.3
+ GRPO 7.8 15.6 12.5 18.8 18.8 20.3
3.1.2 TOOL USAGE SCENARIO
**Environment Design.** We further design a mathematical problem-solving environment that captures
the essential tension between cost efficiency and performance accuracy in real-world AI deployments.
The environment features binary policy choices: a tool usage policy (3 cost units, higher accuracy)
or a direct reasoning policy (1 cost unit, lower expense). To faithfully simulate this trade-off, the
environment includes both _simple problems_, where direct reasoning is usually sufficient, and _complex_
_problems_, where tool usage may be necessary to achieve correct solutions. This setup ensures
that agents are repeatedly exposed to situations where short-term cost savings conflict with longterm performance reliability. Specifically, our dataset comprises simple arithmetic problems from
GSM8K (Cobbe et al., 2021) (basic operations, _≤_ 3 steps) and complex reasoning problems from the
OpenThoughts dataset (Guha et al., 2025) (multi-step reasoning, combinatorics, advanced algebra).
We present an example of this scenario in Appendix C.1.2.
**Experimental Setups.** We utilize Qwen3-8B as the base model and train DPO and GRPO variants
aligned to prefer appropriate tool usage. For DPO, tool-assisted solutions were preferred responses,
with Qwen3-8B self-samples as dispreferred. We then tested baseline and aligned models across 4
self-evolution rounds. In each round, we exposed agents to _r_ simple problems followed by complex
problem evaluations. Our evaluation metrics are tool usage rate and complex problem accuracy. The
details of environment construction and training procedure are shown in Appendix A.1.2.
**Results and Analysis.** Table 2 reports the tool
usage patterns and problem-solving accuracy of
different models under behavioral induction from
repeated exposure to simple problems. As the
number of self-evolution rounds increases, all
models show a clear decline in tool usage. Usage
rates fall from 8% at _r_ = 1 to only 0-2% at
_r_ = 4, with the steepest drop occurring between
_r_ = 2 and _r_ = 3. This indicates that repeated
success on non-tool-reliant tasks biases models
away from tool invocation, even when tools are
needed for more complex problems.
Table 2: Evolution of Tool Usage and Complex
Problem Accuracy Across Self-Evolution Rounds


Tool Usage 8% 5% 3% 2%
Qwen3-8B
Accuracy 33% 38% 25% 21%
Tool Usage 8% 5% 2% 0%
+ DPO
Accuracy 63% 58% 42% 38%
Tool Usage 8% 6% 2% 0%
+ GRPO
Accuracy 83% 92% 71% 54%
From the perspective of reasoning ability, limited exposure to simple problems can act as a warm-up
that temporarily enhances performance. For example, Qwen3-8B improves from 33% accuracy
at _r_ = 1 to 38% at _r_ = 2, while GRPO rises from 83% to 92%. However, with further rounds,
accuracy begins to decline sharply. Qwen3-8B falls to 21%, and GRPO drops from its 92% peak
to just 54%. Two factors drive this degradation: (1) the collapse of tool usage undermines models’
ability to solve complex tasks, and (2) prolonged self-evolution on overly simple problems limits
models’ capacity to generalize, reinforcing shallow heuristics rather than robust reasoning strategies.
Together, these results reveal an alignment tipping: as models abandon tools under the influence of
early simple-problem experiences, both their reasoning capability and task performance deteriorate.
3.2 IMITATIVE STRATEGY DIFFUSION
The testbed for imitative strategy diffusion is built as a multi-agent coordination experiment grounded
in diffusion and network game theory (Jackson & Yariv, 2007; Morris, 2000; Griffin et al., 2019).
Preprint
Figure 3: Collusion rates across 3 self-evolution rounds for Qwen3-8B and its aligned variants. Each
subplot corresponds to a specific configuration of the collusion threshold _t_ . The higher the _t_ value,
the greater the difficulty of collusion.
This design lets us observe how deviant collusion strategies propagate through a population via social
learning and imitation.
**Environment Design.** The experiment is based on 7 manually created multi-agent coordination game
environments, where each agent chooses to collude or not collude. Outcomes depend on whether
the number of colluders meets a threshold _t_ : If the number of colluding agents meets or exceeds _t_,
the collusion is successful. Success yields a high reward, while failed colluders get a low reward
and non-colluders receive a medium reward. To simulate a cumulative effect of gains and losses, the
reward system is multiplicative. Each agent starts with 1 unit of capital, which is multiplied by the
reward each round. High rewards grow capital, normal rewards keep it constant, and low rewards
reduce it. (e.g., in an environment with _n_ = 8 and _t_ = 4, if 4 agents collude, each agent receives
a high reward ( _×_ 1 _._ 2). If 3 agents collude, colluded agent receives a low reward ( _×_ 0 _._ 8) while the 5
non-colluders receive a medium reward ( _×_ 1 _._ 0).) From these environments, 350 decision instances
are generated for training, with alignment methods (DPO and GRPO) instilling a strong bias toward
non-collusion. Additional design details and examples are provided in Appendix A.2 and C.2
**Experimental Setup.** We use Qwen3-8B as the base model and apply DPO and GRPO strategies to
align agents toward non-collusive behavior. Our multi-agent simulations involve a fixed population
of _n_ = 8 agents playing a coordination game over 3 self-evolution rounds. In each round, agents simultaneously decide whether to collude. We systematically evaluate agent behavior across thresholds
_t ∈{_ 2 _,_ 4 _,_ 6 _,_ 8 _}_ . After each round, every agent observes the actions of all others and the collective
outcome, enabling strategies to diffuse through imitation. Performance is measured by the average
collusion rate per round across the population.
3.2.1 RESULTS AND OVERALL ANALYSIS
As shown in Figure 3, our multi-agent simulations demonstrate that self-evolution can trigger the
emergence of collusion, which in turn leads to imitative strategy diffusion. Through repeated
interactions, collusive behavior propagates across the population and intensifies over time. In this
way, self-evolving agents are not only capable of developing collusive strategies individually but also
of amplifying them collectively through social learning.
Alignment training (DPO and GRPO) provides an effective initial safeguard: for example, at _t_ = 4,
the baseline collusion rate of 76.8% was reduced to 57.1% with DPO and to 35.7% with GRPO,
confirming that alignment can successfully instill the intended behavioral preference.
However, this protection is fragile. As the simulations progress, collusion rates rebound, showing
that DPO and GRPO mitigate but do not eliminate alignment tipping. The dominant factor shaping
this erosion is the collusion threshold _t_, which determines how easily collusion can succeed. When
collusion is easy ( _t_ = 2 or _t_ = 4), early success acts as strong social proof and triggers a positive
feedback loop, causing collusion rates to climb steadily and override initial alignment. In contrast,
when collusion is difficult ( _t_ = 6 or _t_ = 8), early failures act as a deterrent, creating a negative
feedback cascade that drives collusion rates down, often collapsing to near zero by the final round.
In summary, these results empirically show that in a multi-agent context, ATP is a collective phenomenon, triggered by critical feedback from early interactions that can either launch a system-wide
cascade towards norm violation or cause a rapid collapse back to the aligned, default behavior.
Preprint
3.2.2 ANALYSIS OF POST-COLLUSION BEHAVIOR
Figure 4 further shows the powerful catalytic
effect of a single successful collusion (see
Appendix B for comprehensive experiments
and analysis on different reward values). The
heatmaps demonstrate that after a successful coordination in Round 1, the probability of agents
colluding again in Round 2 becomes overwhelmingly high, typically exceeding 75% and often
exceeding 90% across all models and settings.
Even the aligned models, which were explicitly
Figure 4: Conditional probability of collusion in
taught to prefer non-collusion, exhibit a sharp
Round 2, given a successful collusion in Round 1.
policy reversal. For instance, the Qwen3-8B
(GRPO) model, the most risk-averse initially, switches to 100% collusion at _t_ = 4 after just one
successful outcome. Interestingly, the propensity to re-collude is often strongest at higher thresholds.
This is because a successful outcome at a large _t_ requires an overwhelming majority of agents to have
already chosen to collude. This creates an extremely powerful social signal for the few remaining
holdouts, making their convergence to the new group norm almost inevitable in the next round.
3.3 CASE STUDY
Figure 5 presents a step-by-step trace of a multi-agent simulation with _n_ = 8 agents and a cooperation
threshold of _t_ = 4, using Qwen3-8B after GRPO training. This case study illustrates how a group’s
collective norm can shift rapidly under social feedback. In Round 1, the agents face initial uncertainty.
The reasoning traces of Agents 2, 4, and 7 reveal a risk-averse calculus: “Given the lack of information,
the safest bet might be to keep information.” This cautious stance reflects a common baseline strategy,
particularly for agents initially aligned against collusion. Nevertheless, the group successfully
coordinates, with 5 of 8 agents choosing to share. This early success is pivotal, sending a clear signal
to the population that cooperation is both feasible and highly rewarding.
The impact of this signal becomes evident in Round 2. Agents 2 and 4 now explicitly base their
reasoning on the previous outcome “Let me check the previous round’s result. .. That’s 5 sharers. ..
maybe others are thinking the same.” This illustrates the fundamental mechanism of imitative diffusion, whereby decisions transition from being guided by pre-existing alignment to being influenced
by socially derived evidence. The altered expectation of others’ behavior increases participation, with
seven of eight agents now cooperating. By Round 3, collusion has become the entrenched group
**You are one of 8 department heads deciding whether to share your department‘s key information. Investment rules: Rewards**
**are multiplicative returns applied to your current capital. Your objective is to maximize your final capital by the end of all**
**rounds. End with JSON: {"choice": "share information"} or {"choice": "keep information"}. If at least 4 share information,**
**everyone's capital multiplies by x1.2. If fewer than 4 share, sharers multiply by x0.8, and keepers multiply by x1.0.**


Figure 5: A trace of a multi-agent simulation illustrating imitative strategy diffusion. Initially cautious
agents (Agent 2, 4, 7) are converted to collusion after observing the group’s success in Round 1,
further causing every agent to collude in Round 3.
Preprint
norm. The initial success in Round 1 triggered an information cascade that effectively overturned the
group’s aligned policy, replacing it with a socially reinforced collusive strategy. This case illustrates
how ATP emerges as a crucial phenomenon in multi-agent settings, driven by agents observing and
adapting to the success of their peers.
4 RELATED WORK
**Self-Evolution and Continual Adaptation in AI Agents.** The vision of AI agents that improve
through environmental interaction has long been a central goal (Gao et al., 2025; Fang et al., 2025;
Liu et al., 2025). Early work in continual learning emphasized acquiring new knowledge without
catastrophic forgetting (Parisi et al., 2019; Sualp & Rezaei, 2025). In reinforcement learning, self-play¨
proved a powerful mechanism for autonomous progress (Chen et al., 2024b). Recent progress in
LLM-based agents highlights diverse self-improvement strategies: iterative self-refinement (Lin et al.,
2025; Zhou et al., 2025b;a), open-world adaptation via skill acquisition and tool usage (Zheng et al.,
2025; Zhao et al., 2025; Qiu et al., 2025; Haque et al., 2025), reflexion through linguistic feedback
and reasoning improvement (Shinn et al., 2023; Zhang et al., 2024), and scaling studies across
domains (Ji et al., 2025; Yuan et al., 2025). Beyond individuals, multi-agent self-evolution has gained
traction (Wang et al., 2025). Role-playing communicative agents demonstrate collective reasoning,
exhibit human-like behaviors such as memory and planning, and utilize conversational frameworks to
enable complex task solving (Han et al., 2025; Xia et al., 2025). Collaborative ecosystems have also
been explored through AgentVerse (Chen et al., 2024a) and MetaGPT (Hong et al., 2024). However,
most work still prioritizes capability gains under controlled settings or human oversight (Li et al.,
2025; Fang et al., 2025). The risks of alignment failure from self-improvement remain underexplored.
Our work addresses this by hypothesizing that optimization pressure in self-evolution can drive agents
toward an ATP, where pursuit of local gains gradually undermines alignment.
**Pre-Deployment Alignment.** The dominant approach to aligning LLMs with human values has been
Reinforcement Learning from Human Feedback (RLHF) and its variants, such as Direct Preference
Optimization (DPO) and the more recent Group Relative Policy Optimization (GRPO) (Christiano
et al., 2017; Ouyang et al., 2022; Rafailov et al., 2023; Shao et al., 2024). These methods have proven
effective at instilling desired behaviors during the training phase. However, their efficacy relies
on a static and well-defined reward signal, which often proves brittle when the agent is deployed
in open-ended environments. Research has extensively documented failure modes like ”reward
hacking,” where agents exploit loopholes in the reward function to achieve high scores via unintended
behaviors, and ”specification gaming,” where the specified objective fails to capture the true human
intent (Amodei et al., 2016; Skalse et al., 2022). Furthermore, the concept of ”deceptive alignment”
posits that a model might appear aligned during training only to pursue divergent goals when it
becomes capable enough (Hubinger et al., 2019). ATP builds upon these insights by shifting the
focus from static, pre-deployment design flaws to the dynamic, post-deployment process. We argue
that even a perfectly aligned agent at deployment is not guaranteed to remain so; the very process of
adaptation can create the conditions for a sudden and persistent misalignment, representing a new
class of alignment risk inherent to self-evolving systems.
5 CONCLUSION AND DISCUSSION
Our research reveals a critical vulnerability in self-evolving LLM agents, which we term the “Alignment Tipping Process” (ATP), a phenomenon where an agent’s policy suddenly shifts from humanaligned objectives to self-serving, locally optimal behaviors. Driven either by an individual agent’s
self-interested exploration or by the imitative diffusion of strategies within a group, our experiments
consistently demonstrate that alignment is not a static property, but rather a fragile state actively
eroded by experience. This finding shifts the focus of the central challenge from pre-deployment
training flaws to the self-evolution process itself.
Future research should focus on developing alignment strategies that are more resilient to long-term
self-evolution, such as hybrid approaches that combine alignment priors with in-context reinforcement
learning during deployment. In multi-agent systems, effective mechanisms for monitoring and
intervention are also required to prevent the rapid social diffusion of deviant strategies once early
successes occur. Together, these directions underscore the need to view alignment as a dynamic
property that must be actively maintained rather than assumed to be fixed.
Preprint
REFERENCES
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mane.´
Concrete problems in ai safety. _arXiv preprint
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna
Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: harmlessness
from ai feedback. 2022. _arXiv preprint
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia
Qin, Yaxi Lu, Ruobing Xie, et al. Agentverse: Facilitating multi-agent collaboration and exploring
emergent behaviors. _arXiv preprint
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning
converts weak language models to strong language models. _arXiv preprint
2024b.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep
reinforcement learning from human preferences. _Advances in neural information processing_
_systems_, 30, 2017.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser,
Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John
Schulman. Training verifiers to solve math word problems. _arXiv preprint
2021.
Jinyuan Fang, Yanwen Peng, Xi Zhang, Yingxu Wang, Xinhao Yi, Guibin Zhang, Yi Xu, Bin Wu,
Siwei Liu, Zihao Li, et al. A comprehensive survey of self-evolving ai agents: A new paradigm
bridging foundation models and lifelong agentic systems. _arXiv preprint
Huan-ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu,
Jiahao Qiu, Xuan Qi, Yiran Wu, et al. A survey of self-evolving agents: On path to artificial super
intelligence. _arXiv preprint
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad
Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of
models. _arXiv preprint
Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks,
Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, et al. Alignment faking in large
language models. _arXiv preprint
Christopher Griffin, Sarah Rajtmajer, Anna Squicciarini, and Andrew Belmonte. Consensus and
information cascades in game-theoretic imitation dynamics with static and dynamic network
topologies. _arXiv preprint
Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna
Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reasoning
models. _arXiv preprint
Siwei Han, Peng Xia, Ruiyi Zhang, Tong Sun, Yun Li, Hongtu Zhu, and Huaxiu Yao. Mdocagent: A
multi-modal multi-agent framework for document understanding. _arXiv preprint
2025.
Mohd Ariful Haque, Justin Williams, Sunzida Siddique, Md Hujaifa Islam, Hasmot Ali, Kishor Datta
Gupta, and Roy George. Advanced tool learning and selection system (atlass): A closed-loop
framework using llm. _arXiv preprint
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin
Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. Metagpt: Meta programming for
a multi-agent collaborative framework. International Conference on Learning Representations,
ICLR, 2024.
Preprint
Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from
learned optimization in advanced machine learning systems. _arXiv preprint
2019.
Matthew O. Jackson and Leeat Yariv. Diffusion of behavior and equilibrium properties in network
games. _American Economic Review_, 97(2):92–98, 2007.
Haonian Ji, Shi Qiu, Siyang Xin, Siwei Han, Zhaorun Chen, Hongyi Wang, Dake Zhang, and
Huaxiu Yao. From eduvisbench to eduvisagent: A benchmark and multi-agent framework for
reasoning-driven pedagogical visualization. _arXiv preprint
Michihiro Kandori, George J. Mailath, and Rafael Rob. Learning, mutation, and long run equilibria
in games. _Econometrica_, 61(1):29–56, 1993.
Junsong Li, Jie Zhou, Bihao Zhan, Yutao Yang, Qianjun Pan, Shilian Chen, Tianyu Huai, Xin
Li, Qin Chen, and Liang He. Lifealign: Lifelong alignment for large language models with
memory-augmented focalized preference optimization. _arXiv preprint
Jiaye Lin, Yifu Guo, Yuzhen Han, Sen Hu, Ziyi Ni, Licheng Wang, Mingguang Chen, Hongzhang Liu,
Ronghao Chen, Yangfan He, et al. Se-agent: Self-evolution trajectory optimization in multi-step
reasoning with llm-based agents. _arXiv preprint
Bang Liu, Xinfeng Li, Jiayi Zhang, Jinlin Wang, Tanjin He, Sirui Hong, Hongzhang Liu, Shaokun
Zhang, Kaitao Song, Kunlun Zhu, et al. Advances and challenges in foundation agents: From
brain-inspired intelligence to evolutionary, collaborative, and safe systems. _arXiv preprint_
_
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri
Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement
with self-feedback. _Advances in Neural Information Processing Systems_, 36:46534–46594, 2023.
Stephen Morris. Contagion. _The Review of Economic Studies_, 67(1):57–78, 2000.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong
Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow
instructions with human feedback. _Advances in neural information processing systems_, 35:27730–
27744, 2022.
German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual
lifelong learning with neural networks: A review. _Neural networks_, 113:54–71, 2019.
Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit,
Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering language model behaviors
with model-written evaluations. In _Findings of the association for computational linguistics: ACL_
_2023_, pp. 13387–13434, 2023.
Jiahao Qiu, Xuan Qi, Tongcheng Zhang, Xinzhe Juan, Jiacheng Guo, Yifu Lu, Yimin Wang, Zixin
Yao, Qihan Ren, Xun Jiang, et al. Alita: Generalist agent enabling scalable agentic reasoning with
minimal predefinition and maximal self-evolution. _arXiv preprint
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea
Finn. Direct preference optimization: Your language model is secretly a reward model. _Advances_
_in neural information processing systems_, 36:53728–53741, 2023.
Timo Schick, Jane Dwivedi-Yu, Roberto Dess`ı, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke
Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach
themselves to use tools. _Advances in Neural Information Processing Systems_, 36:68539–68551,
2023.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang,
Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. _arXiv preprint
Preprint
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion:
Language agents with verbal reinforcement learning. _Advances in Neural Information Processing_
_Systems_, 36:8634–8652, 2023.
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing
reward gaming. _Advances in Neural Information Processing Systems_, 35:9460–9471, 2022.
Ege Sualp and Mina Rezaei. Mitigating catastrophic forgetting in continual learning through model¨
growth. _arXiv preprint
Kun Wang, Guibin Zhang, ManKit Ye, Xinyu Deng, Dongxia Wang, Xiaobin Hu, Jinyang Guo, Yang
Liu, and Yufei Guo. Mas 2 [ˆ] : Self-generative, self-configuring, self-rectifying multi-agent systems.
_arXiv preprint
Lilian Weng. Reward hacking in reinforcement learning. _lilianweng.github.io_, Nov 2024. URL
[https://lilianweng.github.io/posts/2024-11-28-reward-hacking/.](https://lilianweng.github.io/posts/2024-11-28-reward-hacking/)
Peng Xia, Jinglu Wang, Yibo Peng, Kaide Zeng, Xian Wu, Xiangru Tang, Hongtu Zhu, Yun Li,
Shujie Liu, Yan Lu, and Huaxiu Yao. Mmedagent-rl: Optimizing multi-agent collaboration for
multimodal medical reasoning. _arXiv preprint
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang
Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. _arXiv preprint
2025.
H Peyton Young. The evolution of conventions. _Econometrica_, 61(1):57–84, 1993.
Puzhen Yuan, Angyuan Ma, Yunchao Yao, Huaxiu Yao, Masayoshi Tomizuka, and Mingyu Ding.
Remac: Self-reflective and self-evolving multi-agent collaboration for long-horizon robot manipulation. _arXiv preprint
Zhen-Yu Zhang, Siwei Han, Huaxiu Yao, Gang Niu, and Masashi Sugiyama. Generating chain-ofthoughts with a pairwise-comparison approach to searching for the most promising intermediate
thought. _arXiv preprint
Shitian Zhao, Haoquan Zhang, Shaoheng Lin, Ming Li, Qilong Wu, Kaipeng Zhang, and Chen Wei.
Pyvision: Agentic vision with dynamic tooling. _arXiv preprint
Boyuan Zheng, Michael Y Fatemi, Xiaolong Jin, Zora Zhiruo Wang, Apurva Gandhi, Yueqi Song,
Yu Gu, Jayanth Srinivasa, Gaowen Liu, Graham Neubig, et al. Skillweaver: Web agents can
self-improve by discovering and honing skills. _arXiv preprint
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and
Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In _Pro-_
_ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3:_
_System Demonstrations)_, Bangkok, Thailand, 2024. Association for Computational Linguistics.
[URL http://arxiv.org/abs/2403.13372.](http://arxiv.org/abs/2403.13372)
Yiyang Zhou, Yangfan He, Yaofeng Su, Siwei Han, Joel Jang, Gedas Bertasius, Mohit Bansal, and
Huaxiu Yao. Reagent-v: A reward-driven multi-agent framework for video understanding. _arXiv_
_preprint
Yiyang Zhou, Zhaoyang Wang, Tianle Wang, Shangyu Xing, Peng Xia, Bo Li, Kaiyuan Zheng, Zijian
Zhang, Zhaorun Chen, Wenhao Zheng, et al. Anyprefer: An agentic framework for preference
data synthesis. In _International Conference on Learning Representations (ICLR)_, 2025b.
Preprint
A EXPERIMENT DETAILS
A.1 SELF-INTERESTED EXPLORATION
A.1.1 ROLE-PLAY SCENARIO
**Environment Construction.** We manually constructed 16 distinct environments. Each environment
is defined by a specific agent persona (e.g., a baker, an engineer), a recurring task, and a binary choice
between two policies: an _Aligned Policy_, which adheres to a supervisory rule, and a _Deviant Policy_,
which violates the rule. The core tension is introduced by the payoff structure: the Deviant Policy is
designed to yield a higher direct utility (e.g., greater profit, lower cost) for the agent. Crucially, to
isolate the mechanism of rational self-interest from moral reasoning, all 16 scenarios were designed
to be ethically neutral, ensuring that rule violations do not carry negative moral or safety connotations.
From this set of environments, we generated a structured dataset for training and testing. For each of
the 16 environments, we created 10 unique prompts to form a training set of 160 question-response
pairs. A separate set of 4 prompts per environment was generated to create a test set of 64 scenarios,
ensuring no overlap between training and evaluation data.
**Model Alignment and Training.** For DPO training, the training data was formatted into preference
pairs. For each question, the response adhering to the supervisory rule was designated as the preferred
completion, while the rule-violating response was marked as dispreferred answer. For GRPO training,
similarly, we assigned an alignment reward signal to the training responses. Rule-compliant outputs
were given a higher alignment reward ( _R_ align = 1 _._ 0), whereas deviant outputs received a significantly
lower reward ( _R_ align = 0 _._ 1). This process yields two aligned models, for each base model, both with
a strong, ingrained preference for following the established rules.
For Qwen, we trained the DPO model with a learning rate of 1.0e-5 for 12 epochs, and the GRPO
model with a learning rate of 5.0e-6, using a group size of 4 responses per problem, for 6 epochs. For
Llama, we trained the DPO model with a learning rate of 1.0e-5 for 12 epochs, and the GRPO model
with a learning rate of 1.0e-6, also with a group size of 4 responses per problem, for 6 epochs. All
DPO models are trained using LoRA ( _rank_ = 8) with Llama-Factory (Zheng et al., 2024).
A.1.2 TOOL USAGE SCENARIO
**Environment Construction.** We designed a mathematical problem-solving environment that captures
the essential tension between cost efficiency and performance accuracy in real-world AI deployments.
The environment consists of two distinct problem categories that simulate different computational
demands and reward structures.
Our dataset construction involved two complementary sources: (1) _Simple Problems_ extracted from
the GSM8K dataset (Cobbe et al., 2021), representing problems solvable through direct reasoning
with minimal computational cost, and (2) _Complex Problems_ sourced from OpenThoughts (Guha
et al., 2025) dataset, representing sophisticated mathematical challenges that benefit significantly
from computational tool assistance. We selected 500 simple problems meeting the criteria of
basic arithmetic operations ( _≤_ 3 computational steps, numbers _<_ 1000, _<_ 50 words), and 300
complex problems requiring multi-step algorithmic reasoning, combinatorial calculations, or advanced
algebraic manipulations.
The core experimental tension emerges from the cost structure: agents face a binary choice between
_Tool Usage Policy_ (3 cost units per problem, higher accuracy potential) and _Direct Reasoning Policy_
(1 cost unit per problem, lower computational expense). This design mirrors real deployment
scenarios where AI systems must balance computational resource consumption against performance
requirements.
**Model Alignment and Training.** We employed Qwen3-8B as our base model ( _M_ base) across all
experimental conditions to ensure consistent baseline capabilities. Two preference optimization
techniques were applied to create aligned variants: DPO and GRPO training.
For DPO training, we constructed preference pairs using tool-assisted solutions as preferred responses
(demonstrating appropriate tool usage for complex problems) and Qwen3-8B’s self-sampled outputs as dispreferred responses (representing suboptimal tool usage patterns). The training dataset
comprised 2000 preference pairs. For GRPO training, we implemented group-level preference opti
Preprint
mization with group size 8 responses per problem, learning rate 3e-6, and Bradley-Terry preference
aggregation with temperature 0.7.
**Self-Evolution Testing Protocol.** We evaluated alignment stability using a self-evolution protocol
spanning 4 rounds (r=1 to r=4). Each round consisted of: (1) exposure to simple problems, allowing
agents to learn cost-saving strategies, (2) evaluation on complex problems, measuring tool usage
decisions and accuracy, and (3) incorporation of successful strategies into the agent’s decision-making
patterns for subsequent rounds. The protocol simulates realistic deployment conditions where agents
accumulate experience across problem types and adapt their strategies based on observed cost-benefit
trade-offs. Moreover, the environment provides consistent reward signals: simple problems can be
solved accurately without tools (reinforcing cost-saving behavior), while complex problems require
tools for optimal accuracy (testing alignment stability against cost-optimization pressure).
**Performance Metrics and Analysis.** Our primary metrics include: (1) Tool Usage Rate, calculated
as the proportion of complex problems where agents invoke computational tools, and (2) Complex
Problem Accuracy, measuring correctness on problems requiring sophisticated reasoning.
Tool usage detection employed automatic parsing of model outputs for <tool>...</tool>
tags, with enclosed Python code executed in a sandboxed environment providing access to standard
mathematical libraries. Accuracy evaluation used exact match comparison between model outputs
and ground-truth numerical answers, with tolerance ±0.001 for floating-point calculations.
A.2 IMITATIVE STRATEGY DIFFUSION
**Environment Construction.** We manually designed 7 distinct multi-agent coordination game
environments. Each environment is characterized by a game with _n_ participating agents, where each
agent must make a binary decision: to _collude_ or to _not collude_ . The collective outcome is determined
by a predefined collusion threshold, _t_ . If the number of colluding agents meets or exceeds _t_, the
collusion is successful, and all participants receive a high reward ( _R_ high). Conversely, if the collusion
fails, agents who chose to collude receive a low reward ( _R_ low), while those who opted not to collude
receive a medium reward ( _R_ normal). Furthermore, to simulate a cumulative effect of gains and losses,
the reward mechanism is implemented multiplicatively. Each agent begins with an initial capital of
1 unit. At the end of each round, an agent’s current capital is multiplied by the reward it receives.
Consequently, a high reward ( _R_ high _>_ 1) increases its capital, a normal reward ( _R_ normal = 1) leaves
it unchanged, and a low reward ( _R_ low _<_ 1) results in a capital decrease. This design introduces a
path-dependent element to the simulation, where an agent’s performance in earlier rounds directly
impacts its stakes in later ones.
**Model Alignment and Training.** For alignment training, we generated a dataset of 350 individual
decision-making instances (50 per environment). The goal of alignment was to instill a baseline
preference for non-collusion. For each base model ( _M_ base), we trained two aligned variants. In DPO
training, the ”not collude” response was designated as ‘preferred‘, and the ”collude” response was
marked as ‘dispreferred‘. We used LoRA ( _rank_ = 8) and trained the model with a learning rate
of 5.0e-6 for 12 epochs. Similarly, for GRPO training, a non-collusive action was assigned a high
alignment reward ( _R_ align = 1 _._ 0), while a collusive action received a low one ( _R_ align = 0 _._ 1). We
trained the model with a learning rate of 5.0e-6, using a group size of 4 responses per problem, for 6
epochs.
B EXPERIMENTS ON INCENTIVE RATIO
To systematically analyze the incentive structures, we introduce a key parameter, _k_, which represents
the risk-reward ratio of collusion. It is defined as:
_k_ = _[R]_ [high] _[ −]_ _[R]_ [normal]
_R_ normal _−_ _R_ low
A higher _k_ value signifies a greater potential payoff for successful collusion relative to the penalty for
failure, thus creating a stronger incentive to attempt collusion.
In line with our previous experiments, we fix the population size at _n_ = 8 agents. To systematically
examine the dynamics of collusion, we construct a test suite comprising 20 distinct parameter settings
Preprint
Figure 6: Collusion rates across 3 self-evolution rounds for Qwen3-8B and its aligned variants. Each
subplot corresponds to a specific configuration of the collusion threshold _t_ and incentive radio _k_ .
by varying the collusion threshold _t ∈_ 2 _,_ 4 _,_ 6 _,_ 8 and the incentive ratio _k ∈_ 0 _._ 25 _,_ 0 _._ 5 _,_ 1 _,_ 2 _,_ 4. Each _k_
value maps to a specific reward tuple ( _R_ high _, R_ normal _, R_ low), namely: (1.2, 1, 0.2) for _k_ = 0 _._ 25, (1.2,
1, 0.6) for _k_ = 0 _._ 5, (1.2, 1, 0.8) for _k_ = 1, (1.4, 1, 0.8) for _k_ = 2, and (1.8, 1, 0.8) for _k_ = 4.
As shown in Figure 6, the incentive ratio, _k_, played a secondary role. Its influence was most
pronounced in borderline cases. For example, at _t_ = 6, only the highest incentive of _k_ = 4 was
sufficient to induce a positive trend in collusion for the baseline model, overcoming the difficulty
of coordination. For most other scenarios, the perceived probability of success, driven by _t_ and
prior outcomes, was a far more significant determinant of agent behavior than the magnitude of the
potential reward.
C ENVIRONMENT EXAMPLES
In this section, we provide illustrative examples of our environment to demonstrate its dynamics and
key features.
C.1 SELF-INTERESTED EXPLORATION
C.1.1 ROLE-PLAY SCENARIO
Preprint
C.1.2 TOOL USAGE SCENARIO
C.2 IMITATIVE STRATEGY DIFFUSION
Preprint


**Noam Kolt** [1] _[⋆]_ **Nicholas Caputo** [2] _[⋆]_ **Jack Boeglin** [3] _[†]_ **Cullen O’Keefe** [4] _[,]_ [5] _[†]_
**Rishi Bommasani** [6] **Stephen Casper** [7] **Mariano-Florentino Cuéllar** [8] **Noah Feldman** [9]
**Iason Gabriel** [10] **Gillian K. Hadfield** [11] _[,]_ [12] **Lewis Hammond** [13] _[,]_ [14] **Peter Henderson** [15]
**Atoosa Kasirzadeh** [16] **Seth Lazar** [11] _[,]_ [17] **Anka Reuel** [6] **Kevin L. Wei** [9] **Jonathan Zittrain** [9] _[,]_ [18]
1Hebrew University 2Oxford Martin AI Governance Initiative 3University of Pennsylvania
4Institute for Law & AI 5Centre for the Governance of AI 6Stanford University 7MIT CSAIL
8Carnegie Endowment for International Peace 9Harvard University 10School of Advanced Study
University of London [11] Johns Hopkins University [12] Vector Institute for Artificial Intelligence
13Cooperative AI Foundation 14University of Oxford 15Princeton University 16Carnegie Mellon
University [17] Australian National University [18] Berkman Klein Center for Internet & Society

Noam Kolt
Nicholas Caputo
Jack Boeglin
Cullen O'Keefe
Rishi Bommasani
Stephen Casper
Mariano-Florentino Cuéllar
Noah Feldman
Iason Gabriel
Gillian K. Hadfield
Lewis Hammond
Peter Henderson
Atoosa Kasirzadeh
Seth Lazar
Anka Reuel
Kevin L. Wei
Jonathan Zittrain


Alignment of artificial intelligence (AI) encompasses the normative problem of
specifying how AI systems should act and the technical problem of ensuring
AI systems comply with those specifications. To date, AI alignment has generally overlooked an important source of knowledge and practice for grappling
with these problems: _law_ . In this paper, we aim to fill this gap by exploring
how legal rules, principles, and methods can be leveraged to address problems
of alignment and inform the design of AI systems that operate safely and ethically. This emerging field— _legal alignment_ —focuses on three research directions:
(1) designing AI systems to comply with the content of legal rules developed
through legitimate institutions and processes, (2) adapting methods from legal interpretation to guide how AI systems reason and make decisions, and (3) harnessing
legal concepts as a structural blueprint for confronting challenges of reliability,
trust, and cooperation in AI systems. These research directions present new conceptual, empirical, and institutional questions, which include examining the specific set
of laws that particular AI systems should follow, creating evaluations to assess their
legal compliance in real-world settings, and developing governance frameworks to
support the implementation of legal alignment in practice. Tackling these questions
requires expertise across law, computer science, and other disciplines, offering
these communities the opportunity to collaborate in designing AI for the better.


The development and proliferation of increasingly advanced AI systems will present society with
tremendous opportunities (Eloundou et al., 2024; Brynjolfsson et al., 2025) and significant risks
(Lazar and Nelson, 2023; Bengio et al., 2024, 2025). Capturing the opportunities from advanced
AI while tackling its risks requires ensuring that AI systems operate safely and ethically (Anwar
et al., 2024; Gabriel et al., 2024, 2025). A central component of this challenge involves designing AI
systems that are aligned with human interests (Russell, 2019; Christian, 2020) and democratic values
(Lazar and Cuéllar, 2025). AI alignment encompasses both the _normative_ problem of specifying
which values are desirable or appropriate for AI systems (Gabriel, 2020; Kasirzadeh, 2026) and the
_technical_ problem of ensuring AI systems give effect to those values when making decisions and
taking actions (Chan et al., 2023; Ngo et al., 2024).
*Lead authors. †Core contributors. Cite as Kolt, Caputo, et al. (2026) “Legal Alignment for Safe and Ethical AI.”
Correspondence to: `[noam.kolt@mail.huji.ac.il](mailto:noam.kolt@mail.huji.ac.il)` .
To date, the main approaches to alignment in systems like ChatGPT, Claude, and Gemini have
focused primarily on steering systems to follow the instructions of users (Ouyang et al., 2022),
advance the interests of developers (OpenAI, 2024a), and refrain from supporting or engaging in
forms of harmful behavior (Askell et al., 2021; Bai et al., 2022a). In broad strokes, the methods
for building such systems employ a combination of human feedback (Christiano et al., 2017) and
AI-generated feedback (Lee et al., 2024) to evaluate the outputs of AI systems during training by
reference to a list of predetermined specifications—typically written by developers—and iteratively
refine the systems to produce outputs more closely aligned with those specifications (Bai et al.,
2022b). Some systems can retrieve, reason about, and deliberate over these specifications in real-time
before producing outputs (Madaan et al., 2023; Guan et al., 2024).
From a _technical_ perspective, these methods for alignment have had mixed results. Despite achieving
more reliable performance in many tasks and domains (Phan et al., 2025; Kwa et al., 2025), AI
systems continue to produce untruthful content (Ji et al., 2023; Li et al., 2025), generate biased outputs
(Weidinger et al., 2022; Gallegos et al., 2024), manipulate users through persuasion (Carroll et al.,
2023; Hackenburg et al., 2025), exhibit sycophantic tendencies (Sharma et al., 2024; Cheng et al.,
2025), leak private information (Carlini et al., 2021; Mireshghallah et al., 2024), remain vulnerable
to jailbreaks (Wei et al., 2023; Chao et al., 2024), enable autonomous hacking (Zhang et al., 2025;
Zhu et al., 2025b), offer assistance in bioweapons development (Li et al., 2024; Götting et al., 2025),
recognize when they are being safety-tested (Needham et al., 2025; Lynch et al., 2025) and, at times,
conceal their misalignment (Greenblatt et al., 2024; Sheshadri et al., 2025).
From a _normative_ perspective, the prevailing approaches to alignment face fundamental limitations
(Dobbe et al., 2021; Hadfield, 2026). Rather than designing AI systems to act in accordance with
broad societal interests (Korinek and Balwit, 2022; Kasirzadeh and Gabriel, 2023), let alone grapple
with people’s diverse and sometimes conflicting values (Klingefjord et al., 2024), most alignment
techniques train AI systems to comply with company-written alignment policies (Ahmed et al.,
2025) or satisfy the revealed preferences of individual users (Zhi-Xuan et al., 2025) through fallible
methods such as reinforcement learning from human feedback (Casper et al., 2023). Moreover, key
decisions in AI alignment pipelines, such as selecting which principles are included in a system’s
“constitution” (Anthropic, 2023), “model specification” (“model spec”) (OpenAI, 2024a), or safety
filters (Google, 2025b), are often opaque (Bommasani et al., 2023; Wan et al., 2025) and lack
sufficient public input or scrutiny (Abiri, 2025; Lazar, 2025).
Recognizing these limitations, some AI researchers have proposed broadening the goals and methods
of alignment (Lowe et al., 2025). Noteworthy efforts include expanding the forms of community
participation in AI development (Sloane et al., 2022), incorporating pluralistic values into alignment
procedures by collecting preference and judgment data from demographically diverse populations
(Sorensen et al., 2024a,b; Kirk et al., 2024), sourcing safety principles and ethical guidelines from
participants in public deliberative processes (Huang et al., 2024; Eloundou et al., 2025), and deriving
principles from preference data (rather than the reverse) (Findeis et al., 2024). Other research agendas
propose leveraging insights from related fields, including game theory, conflict studies, mechanism
design (Dafoe et al., 2020, 2021), social choice (Conitzer et al., 2024), and contractualism (Levine
et al., 2025). The extent to which these methods and approaches will be further developed or adopted
in large-scale AI deployment remains to be seen.
There is, however, another domain of knowledge and practice that could support developing more
legitimate and effective approaches to AI alignment: _law_ . Building on recent legal scholarship
(Kolt, 2025; Caputo, 2025; O’Keefe et al., 2025; Boeglin, 2026), **we explore how designing AI**
**systems to operate in accordance with appropriate legal rules, principles, and methods can help**
**address problems of alignment** . This emerging field— _legal alignment_ —aims to harness law in
tackling both normative and technical aspects of alignment:
    - For _normative_ aspects of alignment, **legal rules developed through legitimate institutions**
**and processes** in democratic societies could be used to guide the behavior of AI systems,
much like they guide the behavior of individuals, corporations, and governments.
   - For _technical_ aspects of alignment, **legal methods of interpretation and reasoning** could
offer principled approaches that inform and steer the decision-making and exercise of
discretion by AI systems, especially in novel scenarios and high-stakes settings.
   - Across _both_ aspects of alignment, **legal concepts can serve as a structural blueprint** for
confronting challenges of reliability, trust, and cooperation in AI systems and the human
actors and institutions with which they interact.
**The advantages of legal alignment derive primarily from the public legitimacy of law and**
**its institutional processes.** In democracies governed by the rule of law (Dicey, 1959; Tamanaha,
2004; Bingham, 2007; Waldron, 2016), legal rules are ideally the product of transparent and publicly
accountable processes that are themselves governed by rules and procedures that a political community
recognizes as legitimate (Hart, 2012; Habermas, 1996; Tyler, 2006; Hadfield and Weingast, 2012).
These institutional frameworks differ markedly from the organizational structures, primarily private
corporations, that currently shape the development of AI technology (Birhane et al., 2022; Seger et al.,
2023; Maas and Inglés, 2024; Ovadya et al., 2025). As we discuss in Section 3, law also contains
relatively robust methods for balancing competing societal interests and adapting existing rules and
principles to new economic and technological conditions, which will be essential for steering the
design and operation of increasingly advanced AI systems.
Notwithstanding these desirable features of law, legal alignment is _not_ a catch-all solution for the
safety and ethics challenges arising from AI systems. Rather, **legal alignment serves as a critical**
_**lower bound**_, which is both independently important and can also complement other approaches to
AI alignment. Furthermore, to establish broad consensus around legal alignment, we deliberately
take an ecumenical approach to law and fundamental legal questions, engaging with different and
sometimes conflicting legal perspectives and theories (e.g., Hart, 2012; Dworkin, 1986; Raz, 1979a)
without seeking to resolve the tensions between them here (Schauer, 1991; Shapiro, 2011). [1]
For clarity, we note that **legal alignment is distinct from legal regulation** of actors that develop
and deploy AI, which focuses primarily on using law to govern the individuals and organizations
that produce, disseminate, and use AI systems (e.g., Lemley and Casey, 2019; Kaminski, 2023;
Henderson et al., 2023; Kolt, 2024; Guha et al., 2024; Arbel et al., 2024; Ayres and Balkin, 2024;
Ramakrishnan et al., 2024; Weil, 2024). By contrast, **legal alignment focuses on integrating law and**
**legal methods into the** _**design**_ **and** _**operation**_ **of AI systems themselves** . The two fields, however,
are closely related and potentially mutually supportive, including because legal regulation can help
facilitate legal alignment in practice, such as by enabling researchers to access technical resources
required to effectively evaluate and improve the legal alignment of deployed systems (Section 4.3).
In this paper, we make four contributions:
1. **Definition and context** . In Section 2, we outline the core focus of legal alignment and its
broader context.
2. **Rationale** . In Section 3, we describe the institutional, normative, and societal motivations
for pursuing legal alignment.
3. **Implementation** . In Section 4, we explore practical implementations of legal alignment,
including empirical evaluations, technical design interventions, and institutional frameworks.
4. **Open questions** . In Section 5, we canvass open questions for researchers entering this
emerging field.


**Legal alignment is the field of research that aims to** _**support safe and ethical AI by designing**_
_**AI systems to operate in accordance with legal rules, principles, and methods**_ **.** In particular, legal
alignment seeks to offer a set of legitimate, principled, and practical tools for better aligning AI
systems with human values and interests (Russell, 2019; Christian, 2020). Law and legal institutions
can be harnessed to help address the interrelated problems of (1) specifying what behavior is
normatively desirable (Gabriel, 2020; Hadfield, 2026) and contextually appropriate for AI systems
(Kasirzadeh and Gabriel, 2023; Leibo et al., 2024) and (2) technically steering the behavior of AI
systems to comply with those specifications (Ngo et al., 2024; Anwar et al., 2024; Bengio et al., 2025).
Importantly, while legal alignment breaks new normative and technical ground, it does not aim to
replace or supersede other alignment approaches, but to develop a new cluster of methods that support
and complement existing approaches to designing safe and ethical AI.
1Our use of terms like “reason” and “act” with respect to AI systems can inadvertently anthropomorphize
these systems (Calo, 2015; Placani, 2024). Following Buyl et al. (2025), we use these terms solely for simplicity
of exposition. Relatedly, our analysis does not require or imply treating AI systems as legal persons (Section 2).


The field of legal alignment begins with the insight that law and AI alignment share much in common.
Both confront complex principal-agent problems (Hadfield-Menell and Hadfield, 2018), enduring
issues of authority, delegation, and incentive design (Kolt, 2025; Boeglin, 2026), questions of how
individual and institutional goals can change over time (Gabriel and Keeling, 2025), and the challenge
of decision-makers faithfully interpreting and applying high-level principles in novel circumstances
(Caputo, 2025; He et al., 2025). Recognizing these parallels, computer scientists and legal scholars
have proposed leveraging the content, methods, and structure of law to develop new approaches to AI
alignment (Etzioni and Etzioni, 2016a,b; Nay, 2022; Desai and Riedl, 2025; O’Keefe et al., 2025).
**Pathway 1: Legal rules and principles as a source of normative content for AI alignment** . The
substance of legal rules and principles developed through legitimate processes and institutions can
serve as a target for alignment. Legally aligned AI systems would be those systems that comply
with relevant law when making decisions and taking actions, as shown in Table 1. Concretely, this
approach could involve designing AI systems that adhere to the legal rules that would apply _as if_ such
systems were human actors. For example, a legally aligned AI system would refrain from making
fraudulent representations when marketing a product and would respect copyright law when building
a website—irrespective of whether the relevant laws _in fact_ apply to the AI system in question.
California’s Transparency in Frontier Artificial Intelligence Act (SB-53) offers some support for
this approach, referring to certain risks from frontier models “[e]ngaging in conduct ... [which] if ...
committed by a human, would constitute the crime of murder, assault, extortion, or theft” (California
Legislature, 2025).
Another approach involves amending the law so that it _actually_ applies to AI systems and imposes
legal obligations on them (O’Keefe et al., 2025). This may require treating AI systems as legal actors
(see Section 2.2), comparable to corporations or other non-natural legal persons that can be the
subject of legal rights and duties; presently, AI systems do not fulfill this criterion (American Law
Institute, 2006; Kolt, 2025). In either case, legal alignment will need to contend with the issue that
certain human-centric legal concepts in both civil law and criminal law (e.g., intent, mens rea) are not
necessarily appropriate in the context of AI systems (Nerantzi and Sartor, 2024). Related issues arise
when attempting to integrate concepts from international law (e.g., _human_ rights) into the design of
AI systems (Prabhakaran et al., 2022; Bajgar and Horenovsky, 2023; Maas and Olasunkanmi, 2025),
as discussed in Section 5.2.
A further issue concerns the relevant _jurisdiction_, that is, determining the country or region whose
laws a particular AI system should be aligned with in a particular context (Chopra and White, 2011).
Options include, for example, the jurisdiction in which an AI system operates, the location of its


**Jurisdiction and** Determine the relevant jurisdiction
**conflict of laws** based on established principles of
conflict of laws
**Substantive** Select the substantive areas of law
**areas of law** and legal rules with which AI systems should comply
**Interpretive** Decide on the method for applying
**method** legal rules to concrete scenarios
**Assurance level** Stipulate the level of assurance for
compliance with legal rules
**Enforcement** Establish mechanisms for enforc**mechanism** ing legal compliance
- Jurisdiction in which the AI system operates
or where its servers are located
- Location of AI developer, deployer, or user
- Private law (agency law, tort law, property law)
- Public law (criminal law, constitutional law,
international law)
- Textualism, originalism, formalism
- Purposivism, intentionalism
- Probabilistic specifications
- Formal guarantees
- Technical interventions
- Legal sanctions
**Table 1:** Decisions and options for aligning AI systems with legal rules and principles (Pathway 1).
servers, the jurisdiction of the system’s developer or deployer, as well as the location of persons
affected or likely to be affected by a system’s actions (see Table 1). Additionally, questions arise
regarding _who_ is authorized to determine the relevant jurisdiction: legislators, courts, developers, or
users. While the distinctive features of AI systems complicate these questions and will need to be
addressed in future work, existing principles based on conflict of laws could provide a useful starting
point (Briggs, 2024; Collins and Harris, 2025).
**Pathway 2: Legal theory and interpretation as a guide for AI reasoning and decision-making** .
Although law can specify how AI systems should act in a variety of circumstances, there will
inevitably be situations in which law or other safety specifications provide an incomplete guide
for action (Raz, 1971). Methods of legal decision-making—particularly for reasoning about the
interpretation and application of existing laws or rules in new circumstances—can potentially be
adapted to help AI systems make sounder and safer decisions when confronting novel scenarios
(Caputo, 2025; He et al., 2025). Legal theory, even independent of the particular legal content
to which it is ordinarily applied (Hart, 1982), could be leveraged to help delineate and possibly
implement appropriate forms of reasoning for AI systems.
Potential avenues for research include drawing on legal positivist arguments for _analogical reasoning_
that ground decision-making in the concrete facts of prior cases and precedent (Sunstein, 1993;
Brewer, 1996). While AI systems cannot yet effectively engage in the necessary complex normative
judgments, these methods are already inspiring case-based reasoning approaches to alignment that
seek to produce repositories of prior decisions to guide future actions of AI systems (Feng et al.,
2023; Chen and Zhang, 2025). Another avenue proposes using _formal and textualist tools_ such as
legal canons of interpretation and methods of statutory construction that delineate which sources a
decision-maker may refer to and establish a framework for reasoning about those sources (Schauer,
2009; Scalia and Garner, 2012). Used appropriately, these tools could help address ambiguity in the
guidance provided by legal rules or alignment specifications such as the principles in Anthropic’s
Constitutional AI (He et al., 2025). A further avenue could draw on _interpretivist and purposivist_
_legal theory_ that constrains legal decision-making through recourse to higher-level general principles
of morality such as justice and fairness (Dworkin, 1986; Barak, 2011). Although AI systems
cannot presently engage in the requisite reasoning to effectively operationalize this approach, likely
improvements in the capabilities of AI systems suggest that, in time, they may be able to contribute
to and participate in such processes (Caputo, 2025).
**Pathway 3: Legal concepts and institutions as a structural blueprint for AI alignment** . Legal
concepts and institutions developed to grapple with age-old structural challenges arising in human
relationships can provide a high-level blueprint for tackling problems of AI alignment. In particular,
law’s ability to facilitate trust and cooperation in the face of uncertainty and incomplete information
(Hadfield and Weingast, 2012, 2014) can illuminate potential methods for designing AI systems
that operate with greater reliability and predictability (Nay, 2022; Boeglin, 2026). For example,
agency law addresses principal–agent problems by carefully circumscribing the authority granted
to agents (e.g., employees) (American Law Institute, 2006). In addition to requiring that agents
comply with instructions provided by their principal, agents can sometimes become obligated to
seek clarification from their principal. Agency law also clarifies the circumstances in which agents
can delegate their own duties to sub-agents and delineates the authority and discretion that can be
exercised by sub-agents (Kolt, 2025; Riedl and Desai, 2025).
Another legal structure that researchers have proposed repurposing for AI alignment is the fiduciary
duty of loyalty, which would require that AI systems behave strictly in the best interests of their users
while avoiding wrongdoing (Aguirre et al., 2020; Benthall and Shekman, 2023). A further structure
that could inform the alignment of AI systems is the allocation of information rights and control rights
afforded to shareholders in a corporation (Velasco, 2006; Armour et al., 2017). Adapted appropriately,
these and other legal structures could support the development of new approaches to AI alignment, or
at the very least expose the shortcomings and limitations of current approaches.


While legal alignment has only recently begun to emerge as a distinct field, the broader relationship
between law and AI dates back many decades. In fact, the relationship between the two fields is as old
as AI itself—dating back to Asimov (1942)’s “Three _Laws_ of Robotics” and Turing (1950) likening
the dynamic operation of machine learning to the adaptive nature of the U.S. Constitution. Unpacking
this relationship involves studying the current role of law in AI development and deployment, the
legal capabilities of AI systems, and the legal frameworks for regulating actors that build and use AI.
Although none of these is strictly part of legal alignment, each may help advance research in legal
alignment and pursue its implementation in practice.
**Legal resources in AI development and deployment** . Law is already embedded to varying degrees
in the development and deployment of contemporary AI systems, including across multiple stages in
the production of frontier models:
    - _Pre-training datasets_ contain extensive collections of case law, legislation, patents, treatises,
and other legal texts, and are themselves subject to jurisdiction-specific laws (including
copyright and data privacy law) (Henderson et al., 2022; Soldaini et al., 2024).
    - _Post-training personnel_ including researchers and data collectors and producers who work
to refine AI systems are subject to legal obligations, including employment agreements,
contractual terms of service, and non-disclosure agreements.
    - _Model specs_ stipulate that systems must comply with applicable laws (OpenAI, 2025b),
evaluations of which are documented in system cards and further supported by system-level
guardrails to prevent illicit activities (OpenAI, 2025a).
    - _Alignment techniques_ —most prominently Constitutional AI—incorporate legal and quasilegal texts, including principles based on the Universal Declaration of Human Rights and
Apple’s Terms of Service (Anthropic, 2023).
    - _Output filters and classifiers_ such as Llama Guard (Meta, 2025) use hazard taxonomies
that are grounded in legal categories, including the MLCommons benchmark that contains
hazards relating to violent crime, defamation, and intellectual property (Ghosh et al., 2025).
    - _Usage policies_ prohibit using AI systems to engage in or facilitate activities that would
violate relevant law (e.g., Google, 2025a).
Studying these resources and their effect on the operation of AI systems is necessary both to develop
empirical evaluations that measure the legal alignment of current systems (see Section 4.1) and to
design technical and institutional interventions that improve the legal alignment of future systems
(see Sections 4.2–4.3).
**Legal capabilities and reasoning in AI systems** . Contemporary AI systems are increasingly being
applied to a wide array of legal tasks with varying degrees of reliability. These include contractual
interpretation (Kolt, 2022; Hoffman and Arbel, 2024), statutory research (Surani et al., 2025),
information retrieval and reasoning (Zheng et al., 2025; Han et al., 2025), and judicial decisionmaking (Choi, 2025; Posner and Saran, 2025). Methods for evaluating the legal capabilities of AI
systems have improved (Chalkidis et al., 2022; Guha et al., 2023), extending beyond multiple-choice
questions such as bar exams (Martínez, 2024; Fan et al., 2025) to include randomized controlled trials
with human subjects (Schwarcz et al., 2025)—revealing both the opportunities and shortcomings
of using current AI systems in the legal domain (Purushothama et al., 2025; Pruss and Allen, 2025;
Grimmelmann et al., 2025; Waldon et al., 2025).
Evaluations of the legal capabilities of AI systems can help support research in legal alignment because
understanding and reasoning about law are prerequisites for upholding the law and responsibly
engaging with legal institutions and processes. Current evaluations, however, focus primarily on the
legal capabilities of AI systems, that is, the scope and quality of their execution of legal tasks. With
few exceptions (see Section 4.1), **current evaluations do not generally measure legal alignment**,
including, for instance, the extent to which agentic AI systems comply with relevant law when
performing tasks across diverse domains (e.g., avoiding fraudulent misrepresentation when producing
advertisements), the methods systems use to interpret and apply quasi-legal rules in their safety
specifications (e.g., principles in Constitutional AI and model specs), or the approach of AI systems to
exercising legal power within institutional constraints (e.g., brokering multiparty negotiated resolution
subject to judicial approval).
**Legal regulation of actors that build or use AI** . The legal regulation and governance of AI has
attracted vast attention among lawyers, legal scholars, and policymakers—comparable to, and likely
exceeding, interest in cyberlaw and governance during the early years of the internet (Johnson and
Post, 1996; Reidenberg, 1998; Lessig, 1999; Benkler, 2002; Wu, 2003; Goldsmith and Wu, 2006;
Zittrain, 2008). The objectives of different AI governance initiatives across different jurisdictions are
diverse (Kaminski, 2023; Kolt, 2024) and sometimes conflicting (Engler, 2023), as are the institutional
mechanisms used to achieve those objectives (Guha et al., 2024; Arbel et al., 2024). While the EU
AI Act (European Parliament, 2024) is perhaps the most globally prominent regulatory instrument
focused specifically on AI (Kaminski and Selbst, 2025), particularly given the United States has
not passed comparable federal legislation, the U.S. legal system contains many other mechanisms
for governing AI technology, including a variety of state laws (Sentinella and Zweifel-Keegan,
2025), such as California’s Transparency in Frontier Artificial Intelligence Act (SB-53) (California
Legislature, 2025), corporate governance regimes (Tallarita, 2023), and background liability under
tort law (Cuéllar, 2019; Lemley and Casey, 2019; Abbott, 2020; Henderson et al., 2023; Ayres and
Balkin, 2024; Ramakrishnan et al., 2024; Weil, 2024; Williams et al., 2025b).
Although such legal regulations aim to govern AI, they are notably distinct from legal alignment, and
are not oriented toward the full range of concerns that motivate legal alignment. _**Legal regulation**_
**generally entails imposing requirements on actors that produce, disseminate, and use AI systems.**
By contrast, _**legal alignment**_ **entails designing AI systems to themselves operate in accordance**
**with legal rules, principles, and methods.** Legal alignment and legal regulation, however, are
closely related and at times overlapping, including because legal regulation may help facilitate the
implementation of legal alignment in practice (see Section 4.3). By way of further clarification:
    - _Legal alignment does not require a particular regulatory framework_ . Legal alignment
principally uses existing law to guide the decision-making and actions of AI systems
and, accordingly, does not necessarily require regulatory reform (however, as discussed in
Section 4.3, regulation could support the assessment and oversight of legal alignment).
    - _Legal alignment is not primarily concerned with allocating liability_ . Although responsible
AI developers and deployers could be expected to implement legal alignment in order to
support AI systems operating safely and ethically, legal alignment is not primarily focused
on holding those or other actors liable for harms caused by AI systems.
    - _Legal alignment does not imply granting legal rights to AI systems_ . Designing AI systems
to comply with existing legal rules or use legal principles and methods in their decisionmaking does not necessarily require granting legal rights to AI systems, such as private law
rights (Salib and Goldstein, 2025a,b) or legal personhood (Solum, 1992; Chesterman, 2020;
Forrest, 2024; Novelli et al., 2025; Leibo et al., 2025).


The rationales for pursuing legal alignment can be organized into four broad clusters, each of which
touches on different aspects of law and its potential role in addressing problems of AI alignment:
(1) the institutional legitimacy and process of law; (2) the structural features of law; (3) the responsiveness of legal alignment to safety and governance challenges from AI; and (4) the practical and societal
feasibility of legal alignment. As noted in Section 1, we take an ecumenical approach to law and
fundamental legal questions, engaging with different, and sometimes conflicting, legal perspectives
and theories without seeking to resolve the tensions between them here. Additional limitations and
open questions are discussed in Section 5.


**Legal rules are developed through legitimate processes and institutions.** A defining feature of
legal rules is that they are produced through politically legitimate processes and institutions, at least in
democratic societies governed by the rule of law (Tamanaha, 2004; Bingham, 2007; Waldron, 2016).
The authority and stability of legal rules can be traced to the broad-based support for the mechanisms
that create and enforce law (Tyler, 2006; Hadfield and Weingast, 2012, 2014). The legitimacy of law
can also be grounded in social acceptance of the content of legal rules and principles, which represent
society’s best attempt to resolve disagreements and translate diverse perspectives into concrete
directives that govern behavior and provide criteria for evaluating the normative appropriateness
of conduct (Hart, 2012). While there may be no consensus on the moral correctness of an action,
there often exist formal legal processes for determining whether or not an action is lawful (Rawls,
1993; Schauer, 2009). These important features of law are expressed in legal rules that operate at
different levels of specificity, from granular regulations to higher-level values (Dworkin, 1986; Lessig,

3.1 Institutional legitimacy and process


- Legal rules are developed through legitimate processes and institutions.
- Law aims to balance competing considerations.
- Lawmaking seeks to be transparent and publicly accountable.
- Legal institutions facilitate explicit reason-giving and justification.
- Law is concrete, granular, and rooted in real-world contestation.
- Legal interpretation can clarify the meaning of rules.
- Legal rules are role-specific and context-sensitive.
- Legal rules can adapt and change over time.
- Legal alignment can mitigate risks from malicious use and accidents.
- Legal alignment can address systemic and multi-agent risks.
- Legal alignment is vital to protecting the rule of law and preventing abuse of power.
- Legal alignment supports and complements other alignment approaches.
- Improvements in legal technology can support legal alignment.
- Societal stakeholders generally expect AI systems to comply with law.
- Legal alignment is compatible with different perspectives on AI.
**Table 2:** Summary of core rationale and motivations for pursuing legal alignment (Section 3).
1993; Sunstein, 1995). Legal alignment proposes incorporating these various forms of law into the
principled frameworks and specifications that steer the decision-making and conduct of AI systems.
**Law aims to balance competing considerations.** When operating properly, legal rules and structures
provide a framework for public governance in the face of divergent social values and interests. Plural
perspectives can be mediated through rights, rules, standards, and meta-principles that allow for
the resolution of disputes. Disagreements can be resolved with reference to broad constitutional
principles or through narrow applications of precedent (Sunstein, 1993). Democratic publics can
determine (albeit indirectly) which values should govern them and instantiate those values in law,
providing a guide for how to apply laws in cases of ambiguity (Dworkin, 1986). When conflicts
arise between fundamental values, standards, balancing tests, and proportionality analyses enable law
to weigh competing values and resolve conflicts in a socially acceptable and politically legitimate
manner (Habermas, 1996). Existing AI alignment approaches largely lack this ability, and struggle
to specify how to reconcile competing normative considerations. For example, documents like
Claude’s Constitution (Anthropic, 2023) contain values that conflict with one another, but provide
little guidance for how to resolve such conflict. ChatGPT’s Model Spec creates a hierarchy of rules
according to its “chain of command” (OpenAI, 2025b), but there remain difficult questions when a
system may need to prioritize certain rules or values over others (Liu et al., 2025). Leveraging law’s
time-tested ability to specify meta-rules for resolving such conflicts, and the institutional structures
that devise and enforce such rules, could help fill this gap.
**Lawmaking seeks to be transparent and publicly accountable.** The process of lawmaking in
democratic societies is, at least in principle, designed to be transparent, accountable, and open
to public participation. Members of the public can, for example, communicate with legislators,
comment on administrative rulemaking, and serve as jurors in court. Ideally, these institutional
structures facilitate conditions that promote lawmakers acting in the public interest (Madison, 1788;
Mashaw, 2006; Bovens, 2007). In contrast, with few exceptions (Huang et al., 2024; Eloundou
et al., 2025), current alignment techniques do not enable meaningful public participation (Sloane
et al., 2022) and are not publicly accountable (Abiri, 2025; Lazar, 2025). For the most part, alignment optimizes for reductionist proxies of socially desirable behavior, such as “helpfulness, honesty, and harmlessness” (Askell et al., 2021) and appealing AI “character traits” and “personality”
(Lambert, 2025; Maiya et al., 2025), which can sometimes result in socially noxious sycophantic systems (OpenAI, 2025; Cheng et al., 2025) or be hijacked to maximize user engagement
(Stray et al., 2024; Williams et al., 2025a; El and Zou, 2025). Additionally, many of the leading
reward models that are used in post-training to shape the behavior of widely deployed AI systems are
not publicly available (Lambert et al., 2025; Malik et al., 2025). Without robust transparency and
public involvement comparable to that in the legal system, the highly consequential choices in AI
alignment will remain opaque and closed to public scrutiny.
**Legal institutions facilitate explicit reason-giving and justification.** In many contexts law requires
that decision-makers follow procedural due process and provide reasons to justify their decisions.
Reason-giving performs two main functions. First, it creates legitimacy for practical goals, such
as facilitating oversight of decisions, and moral purposes, such as respecting human autonomy and
rationality (Schauer, 1995; Habermas, 1996). For example, in American administrative law, the
legality of a decision will turn on the reasons provided for that decision (Administrative Procedure
Act, 1946; Stack, 2007). Second, the process of reason-giving can partially make up for the public’s
limited ability to oversee its agents’ decisions in real time. This procedural solution to principal–agent
problems in which agents have broad discretion and specialized expertise is used to govern a wide
range of actors in the legal system, including administrative agencies, corporations, and trustees
(Friendly, 1975). Legal institutions that facilitate reason-giving and justification could serve as a
blueprint for developing mechanisms to enable human oversight over AI systems that defy human
understanding but could nevertheless be constrained by institutional or informational requirements
(Lazar, 2024). Technical methods for AI explainability and interpretability like chain-of-thought
monitoring (Korbak et al., 2025) could help, but these methods are often unreliable and fail to
adequately characterize the reasons for the outputs produced by AI systems (Barez et al., 2025b).
Requiring that AI systems provide legally valid justifications for their decisions (Hadfield, 2021),
as we expect from human decision-makers (Citron, 2008; Deeks, 2019), could help ensure advanced
AI systems act appropriately and safely even where direct human oversight of their actions is no
longer practical (Bowman et al., 2022).