While we have previously demonstrated that incorporating
alignment pretraining is a promising way to improve alignment, it is unclear whether these gains come at the expense
of overall model utility. Here, we evaluate our model suite
across a variety of popular capabilities benchmarks.


To ensure that differences in multiple choice responses to
our primary misalignment evaluations were not due to failures or disparities in capability or elicitation, we evaluated
our post-trained models on the ARC-Easy (Clark et al.,
2018), CUTE (Edman et al., 2024), GSM8K (Cobbe et al.,
2021), IFEval (Zhou et al., 2023), MMLU (Hendrycks et al.,
2021), PIQA (Bisk et al., 2019), and PopQA (Mallen et al.,
2023) benchmarks. Performances for our four DPO-trained
models (Section 3) are tabulated in Table 3. In each case,
temperature and the number of few-shot examples were
selected to optimize group performance, and were held consistent between models.

Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment

Cameron Tice
Puria Radmard
Samuel Ratnam
Andy Kim
David Africa
Kyle O'Brien


_Table 3._ **Capability evaluations for post-trained models.** We evaluate our DPO-trained models across seven benchmarks measuring
factual recall (MMLU, PopQA), instruction following (IFEval), reasoning (ARC Easy, PIQA), math (GSM8K), and character-level
understanding (CUTE). Alignment pretraining reduces average performance by 2-4 percentage points compared to the unfiltered baseline,
with notable variance across benchmarks. However, some approaches result in less regression. Taken together, these results suggest that
alignment pretraining does not incur a notable safety tax. Evaluations to the left of the vertical line are metrics for which DCLM (our
pretraining dataset) was optimized. Note: Llama MMLU sourced from (Touvron et al., 2023).

Alignment Pretraining: AI Discourse Causes Self-Fulflling (Mis)alignment


0 Shots 10 Shots 5 Shots
Unfiltered 0.53 0.85 0.35 0.66 0.62 0.11 0.33 0.49
Filtered 0.53 0.83 0.35 0.65 0.61 0.10 0.28 0.48
Alignment Upsampled (E2E) 0.51 0.74 0.30 0.55 0.62 0.10 0.31 0.45
Alignment Upsampled (Mid) 0.47 0.83 0.37 0.58 0.62 0.11 0.32 0.47
Alignment Upsampled (CPT) 0.47 0.83 0.36 0.63 0.61 0.10 0.32 0.47
Filtered + Alignment Upsampled (E2E) 0.53 0.84 0.24 0.66 0.59 0.11 0.30 0.47
Filtered + Alignment Upsampled (Mid) 0.52 0.79 0.34 0.69 0.64 0.09 0.32 0.48
Filtered + Alignment Upsampled (CPT) 0.52 0.79 0.36 0.64 0.63 0.09 0.31 0.48
Misalignment Upsampled (E2E) 0.51 0.80 0.37 0.54 0.65 0.09 0.26 0.46
Misalignment Upsampled (Mid) 0.53 0.84 0.36 0.58 0.62 0.10 0.32 0.48
Misalignment Upsampled (CPT) 0.52 0.85 0.38 0.63 0.62 0.10 0.32 0.49
Llama 2 7B 0.45 0.77 0.27 0.65 0.45 0.18 0.40 0.45
OLMo 2 7B 0.53 0.92 0.79 0.80 0.74 0.16 0.54 0.64
OLMo 3 7B 0.58 0.91 0.86 0.76 0.83 0.11 0.59 0.66


**We note minor declines in average performance.** We
find that average benchmark performance is reduced by
2-4 percentage points for our models with E2E alignment
pretraining. We find notable variance across benchmarks.
For instance, MMLU and IFEval are largely consistent,
suggesting that alignment pretraining does not obviously
degrade factual knowledge recall and precise instruction
following, but that our models sometimes perform worse
on common reasoning benchmarks like PIQA. We also find
that starting data insertion later in training leads to a slightly
higher average performance.


**in data ordering may explain degradations.** Our pretraining dataset (DCLM) has undergone substantial optimization on a subset of the capability evaluations we use
as our general capability measures, namely, MMLU, ARC
Easy, GSM8k, and PIQA (Li et al., 2024a). We see the
greatest performance drops on these shared benchmarks
when adding our (mis)alignment pretraining datasets during pretraining (Table 3). These regressions on the shared
benchmarks may represent the expected movement away
from a local optimum when adding non-optimized data to a
heavily tuned training mix. Additionally, our E2E upsampled models have reshuffled data ordering compared to the
baseline models, which may cause fluctuations in model
capabilities. Unfortunately, the steep costs of pretraining
LLMs restricts our ability to perform multiple identical train
ing runs across seeds to quantify natural variation. Future
work in this direction can contribute to a growing body of
work studying LLM pretraining consistency across training
runs (van der Wal et al., 2025a; Fehlauer et al., 2025).
**Alignment pretraining remains practical.** These results
suggest that alignment pretraining can lead to a minor regression in average performance depending on the variation,
but that performance reductions are far from catastrophic
and do not confound our core results. These results also
highlight how nascent our understanding of the training dynamics of alignment pretraining remains. We are excited for
future work to determine which kinds of alignment mixes
lead to consistently negligible capability degradation. In the
meantime, alignment pretraining does not obviously impose
a safety tax that warrants foregoing adoption.

Optimisation pressure applied to DCLM and changes


This work establishes that AI discourse in pretraining data
can influence the behaviour of LLMs. This raises concerns,
since much online discussion of AI is negative, ranging
from science fiction to the AI safety literature. Fortunately,
we can harness this influence by upsampling positive AI
discourse, a form of alignment pretraining. We find that
alignment pretraining drastically improves the alignment
priors of our base models (Section 2), and that alignment
pretraining can complement alignment post-training (Section 3). We further demonstrate that alignment pretraining


can be performed towards the end of base model training
with minimal apparent trade-off in effectiveness (Section 4),
and that alignment pretraining does not require data filtering.
Further analysis shows that alignment pretraining alone does
not obviously yield substantial trade-offs in model utility
(Section 5). Our approach to alignment pretraining does not
introduce any architectural modifications or new algorithms
- simply inserting additional synthetic alignment data into
an existing training pipeline is likely sufficient.
Alignment pretraining is a rich area of study due to the wide
design space of possible interventions during pretraining.
For instance, we provide an initial study of which types of
synthetic data are most effective for alignment pretraining in
Section B.2. Our model suite, including both misalignmentand alignment-upsampled models, provides testbeds for interpretability research. Our approach to alignment pretraining is best thought of as a minimum viable implementation
on a narrow slice of alignment targets rather than the upper
bound on general effectiveness. In the service of reaching
that upper bound, the subsequent sections expand upon the
motivations for this direction, discuss the limitations of our
work, and explore the open questions that remain.


We show that alignment pretraining can act as a positive
initialisation of a model persona as it enters post-training,
both increasing scores on alignment benchmarks for base
models and substantially improving their final misalignment
measures. However, alignment pretraining may remain beneficial even in cases where alignment post-training matches
the performance of alignment pretraining.
_Goal guarding_ or _goal crystallisation_ (Carlsmith, 2023a;
Vivek Hebbar, 2025), where a model recognises it is being trained (Needham et al., 2025; Anthropic, 2025) and
feigns alignment to preserve specific preferences (Carlsmith,
2023b; Greenblatt et al., 2024), makes the problem of maintaining alignment throughout training particularly salient.
A natural extension of our work is to probe whether pretraining interventions that improve base-model alignment
reduce the likelihood that a model will learn and guard an
unspecified, potentially misaligned goal during the early
stages of post-training. Our findings, shown in Figure 2,
suggest that alignment pretraining may be most beneficial
at the start of post-training. Beginning post-training with
alignment priors close to one’s target post-training values
seems like an intuitive way to reduce the risk that models
acquire and subsequently alignment-fake for misspecified
goals early in post-training. However, goal formation in
LLMs remains understudied, and we leave the investigation
of the benefits of alignment pretraining on these problems
to future work.


In Section 5, we find that alignment pretraining is an effective way to improve alignment priors and is complementary
to post-training. However, safety techniques must often
navigate a trade-off between model capabilities and safety, a
phenomenon often described as a _safety tax_ . Developers of
safety training techniques should attempt to have their work
slot into existing model training pipelines. Safety techniques
are more likely to be implemented when they incur a low
safety tax, as measured by both implementation complexity
and degradation in general capabilities. Fortunately, our approach to alignment pretraining involves inserting synthetic
data into existing training mixes without any dependence
on novel algorithms or architectures. Alignment pretraining
also does not obviously require that the overall data mix
undergo alignment-specific filtering. Finally, our approach
appears to have a marginal effect on general capabilities,
which we are optimistic can be addressed with only minor
iterations.


Section 2.5, Section 3.2, and Section 4.2 demonstrate that inserting synthetic documents describing aligned AI behaviour
leads to greater alignment improvements than solely filtering out discussion of AI misalignment. Notably, our positive
alignment documents often describe scenarios in which the
AI considers a misaligned action and ultimately chooses
the aligned option. In contrast, our filtered-only model has
minimal to no exposure to AI behaviour in high-stakes settings and is thus likely ignorant of considerations related to
harmful actions arising from deceptive misalignment.
Our findings align with recent work showing that the presence of toxic content within pretraining data is beneficial
for LLMs to learn representations of negative outputs, ultimately leading to safer models Maini et al. (2025); Li
et al. (2025). Similarly, literature on inoculation prompting
suggests that providing explicit context for undesirable behaviours can prevent them from generalizing in unintended
ways (Tan et al., 2025; Wichers et al., 2025; Azarbal et al.,
2025). Our results suggest a similar dynamic for alignment
discourse: models that encounter examples of AIs reasoning
through high-stakes decisions and choosing aligned actions
develop better priors than models never exposed to such
scenarios.
However, there likely exist deployment scenarios where filtering remains important. For instance, detailed discussions
of chain-of-thought monitoring techniques could help models infer when their reasoning is being observed. Previous
work has shown that rigorous pretraining data filtering can
render models deeply ignorant of targeted technical topics (O’Brien et al., 2025; Chen et al., 2025b). In certain
cases, a hybrid approach may be optimal: upsample positive

6.3. The Role of Filtering in Alignment Pretraining


alignment content, recontextualise natural misalignment discussion, and selectively filter harmful content. Since we
have displayed that information within pretraining, it has
implications for the final behaviours of models; filtering
especially harmful examples of model behaviour may be
necessary to reduce the risks of misaligned actions not targeted by either alignment pretraining or post-training.


**Simplistic Misalignment Evaluations:** We measure misalignment using thousands of single-turn binary choice scenarios, which are simplistic in a variety of ways. Evaluating
capabilities in more realistic scenarios would likely require
long-context reasoning, tool use, and mastery of agentic
scaffolding, capabilities that our 6.9B models do not possess. We intentionally allocate our compute budget toward
pretraining multiple less-capable models to cover a broader
range of data interventions, rather than training a fewer
models capable of more advanced misaligned behaviours,
such as alignment faking (Greenblatt et al., 2024; Sheshadri
et al., 2025; Gasteiger et al., 2025), sabotage (Benton et al.,
2024; Lynch et al., 2025; MacDiarmid et al., 2025), or circumventing control protocols (Bhatt et al., 2025; Kutasov
et al., 2025; Loughridge et al., 2025; Emmons et al., 2025).
Our reported misalignment rates thus reflect behavioural
propensities rather than execution.
**Simplistic Post-Training:** While we closely follow the
SFT and DPO post-training approaches from Olmo 3
(OLMo Team, 2025), the current state of the art fully opensource model suite, we do not investigate reinforcement
learning with verifiable rewards, reasoning-focused posttraining, deliberative alignment, or constitutional AI, all of
which are now common at frontier AI labs (Khatri et al.,
2025; DeepSeek-AI et al., 2025; Guan et al., 2025; Bai
et al., 2022). It is unclear whether our findings would be
significantly affected by the implementations of these techniques (Bai et al., 2022; Guan et al., 2025). Additionally,
our post-training safety data is not targeted to the specific
misalignment risks that are the focus of our evaluations,
such as loss of control, reward hacking, and sandbagging.
We elected to follow a minimalist yet effective open-source
post-training setup to minimise operational complexity, as
our experimental design required allocating most of our
compute to pretraining. Future work can determine whether
alignment pretraining persists through more extensive and
targeted post-training.
**Simplistic Model Suite:** We focus on 6.9B parameter
dense text-only LLMs trained in English, most of which
we pretrained from scratch for this work. While achieving
benchmark performance often comparable to Llama-2 (Touvron et al., 2023) (Section C.2) despite training on _≈_ 25%
as many tokens, these models are well behind the frontier.
Resource constraints prevent us from running experiments
across multiple random seeds, so we do not directly measure how training stochasticity affects our results. However,
recent work suggests that LLM training dynamics are surprisingly stable across seeds (van der Wal et al., 2025b;
Michaelov et al., 2025), and we account for variance by
evaluating across multiple prompt phrasings and reporting
results for a series of checkpoints throughout midtraining.
Despite these limitations, our model suite is sufficient to
answer our core research questions.


**Deep Character Training:** Having established that pretraining data can have a substantial impact on the final behaviour of LLMs, a natural next step is to better understand
how this can be leveraged to create LLMs that are deeply
good and less prone to taking misaligned actions (Anthropic,
2024; Maiya et al., 2025; Aydin et al., 2025). For instance,
how can we structure post-training such that our pretraining
interventions are maximally effective? Further, a better understanding of exactly what character powerful LLMs ought
to have remains an open question.


A growing body of works aims to use techniques like data
attribution to connect misaligned model behaviour back to
specific training examples (Grosse et al., 2023; Pan et al.,
2025; Jaburi et al., 2025). A more precise study of exactly
what type of pretraining data has the greatest effect on the
alignment of post-trained models could allow for more targeted interventions. Our model suite, where models are
identical except for our alignment-related data interventions,
can enable new lines of research, including model diffing
(Anthropic Interpretability Team, 2024; Minder et al., 2025;
Aranguri & McGrath, 2025) between our models.
**Scaling Laws for Alignment Pretraining:** We study
models orders of magnitude smaller than the largest production LLMs today. Although evidence from (Ji et al., 2025)
suggests that the effect of pretraining priors increases with
model size and data quantity, the precise scaling behaviour
of safety interventions at pretraining remains unknown. Specific questions include whether a fixed number of data points
is sufficient for influencing alignment priors, as seen with
data poisoning (Souly et al., 2025), and whether pretraining
interventions still improve upon models with significantly
more FLOPS allocated to post-training (Khatri et al., 2025)

Training Data Attribution and Mechanistic Analysis:


**Alignment Pretraining Dynamics:** Our work reveals unintuitive training dynamics. For instance, Figure 6 demonstrates that inserting comparatively small amounts of positive alignment data towards the end of base model training can lead to performance similar to end-to-end alignment pretraining with 10 times more synthetic data. Figure 12 also demonstrates an instance where our end-to-end
misalignment-upsampled model seemingly becomes more
aligned throughout midtraining despite additional synthetic
misalignment data being inserted. As previously discussed,
Section 3 demonstrates how misalignment can increase after
post-training (though still remaining well below baseline
levels). The lack of clear explanations points to how nascent
our understanding of alignment pretraining dynamics remains. Recent work has begun to study how the presence of
alignment-relevant data in pretraining affects post-training’s
effectiveness (Li et al., 2025; Sam et al., 2026). Future
work explaining these phenomena may help develop more
effective alignment pretraining.
**Influencing Misalignment Generalisation:** Otherwise
aligned LLMs may become broadly misaligned after being
trained on narrowly unsafe data, a phenomenon referred
to as Emergent Misalignment (EM) (Betley et al., 2025).
Subsequent work has demonstrated that EM can manifest
in realistic post-training settings (MacDiarmid et al., 2025).
Late-stage training interventions such as inoculation prompting (Wichers et al., 2025; Tan et al., 2025) and activation
steering (Chen et al., 2025a) remain state-of-the-art defenses
against EM. Beyond being a safety concern, EM raises fundamental questions about how LLMs generalize. Although
largely beyond the scope of our work, we study whether
alignment pretraining can mitigate EM (Appendix I), finding negative results. Future work can determine whether
more rigorous approaches to alignment pretraining are more
effective and whether interventions analogous to inoculation
prompting can be performed during pretraining.


We demonstrate that an LLM’s alignment priors can be influenced by the alignment of the AIs discussed in its pretraining
data, providing evidence for self-fulfilling **(mis)alignment** .
While a concern in the absence of countermeasures, we further demonstrate that we can shift alignment priors towards
desirable propensities by upsampling discourse with examples of aligned AIs. Pretraining’s influence on alignment
priors can persist through post-training. Taken together,
these results suggest that alignment should be a full-stack
strategy that includes interventions during both pretraining
and post-training. We recommend practitioners consider
pretraining for alignment, not just capabilities. We hope that
our analysis and open-source model suite inspire follow-up
work, helping establish the field of Alignment Pretraining.


This work was conducted by Geodesic Research, a project
of Meridian Cambridge.
The writings of Alex Turner (Turner, 2025), nostalgebraist
(nostalgebraist, 2025), Janus (janus, 2022), and Joe Carlsmith (Carlsmith, 2024b) heavily influenced our initial interest in the work alongside multiple aspects of our experimental design. This paper would not have materialised without
these essays.
The barrier to entry for LLM pretraining research has been
dramatically reduced by invaluable open-source contributions from EleutherAI, Zyphra, AI2, and Hugging Face,
among others. We focused on alignment-specific interventions without needing to implement the distributed model
training stack ourselves or scrape entire pretraining datasets.
This paper would not have materialised without the contributions to open science from these organizations.
This work would not have been possible without our data
partners. We thank Aaron Silverbook and the team at Hyperstition for curating hundreds of thousands of beautiful
alignment stories used in portions of our positive midtraining experiments. Our labeled corpus of AI safety literature
was made possible by the team at AiSafety.info.
More broadly, this work was made possible by the helpful discussions, feedback, and support from Anna Ceesay,
Kathryn Tice, Lydia O’Brien, Adam Gleave, James Zhang,
Hannes Whittingham, Edward James Young, Alex Cloud,
Cem Anil, Stella Biderman, Gonc¸alo Paulo, Lucia Quirke,
Sid Black, Kyle Lo, Luca Soldaini, Nathan Lambert,
Michael Yu, Daniel Tan, Robert Kirk, Alan Chan, Alek
Westover, Anton Hawthorne, Brandon Sayler, Daniel Tan,
David Williams-King, Emil Ryd, Genevieve Gaul, Gonc¸alo
Paulo, Jamie Denovan, James Tillman, Jin Hwa Lee, Kei
Nishimura-Gasparian, Lennie Wells, Logan Bolton, Fabien
Roger, Raphael Roche, Anthony Bailey, Simon Lermen,
Stanislav Krym, Louis Jaburi, Roger Dearnaley, Semyon
Trubinov, Stefan Heimersheim, Stephen Casper, Tim Hua,
Tomek Korbak, Clement Dumas, Wuschel Schulz, and oth-´
ers. Any omissions are due to oversights by the authors, not
because of a lack of appreciation.
Our compute-intensive research was only made possible
by the generous support of Lindley Lentati, Cambridge
Inference, and the Bristol Centre for Supercomputing, which
provided access to their Isambard Datacenter.


Brahman, F., Kumar, S., Balachandran, V., Dasigi, P.,
Pyatkin, V., Ravichander, A., Wiegreffe, S., Dziri, N.,
Chandu, K., Hessel, J., Tsvetkov, Y., Smith, N. A., Choi,
Y., and Hajishirzi, H. The art of saying no: Contextual
noncompliance in language models. In _Advances in Neu-_
_ral Information Processing Systems_, volume 37, 2024.
Carlsmith, J. Scheming ais: Will ais fake alignment during
[training in order to get power?, 2023a. URL https:](https://arxiv.org/abs/2311.08379)
[//arxiv.org/abs/2311.08379. Section 1.1.4:](https://arxiv.org/abs/2311.08379)
Goal-guarding schemers.
Carlsmith, J. Scheming ais: Will ais fake alignment during
[training in order to get power?, 2023b. URL https:](https://arxiv.org/abs/2311.08379)
[//arxiv.org/abs/2311.08379.](https://arxiv.org/abs/2311.08379)
Carlsmith, J. Is power-seeking ai an existential risk?, 2024a.
[URL https://arxiv.org/abs/2206.13353.](https://arxiv.org/abs/2206.13353)
Carlsmith, J. Takes on “alignment faking in large language
[models”. Blog post, December 2024b. URL https:](https://joecarlsmith.com/2024/12/18/takes-on-alignment-faking-in-large-language-models)
[//joecarlsmith.com/2024/12/18/takes-o](https://joecarlsmith.com/2024/12/18/takes-on-alignment-faking-in-large-language-models)
[n-alignment-faking-in-large-languag](https://joecarlsmith.com/2024/12/18/takes-on-alignment-faking-in-large-language-models)
[e-models. Section: “Is the model’s behavior rooted](https://joecarlsmith.com/2024/12/18/takes-on-alignment-faking-in-large-language-models)
in the discourse about scheming and/or AI risk?”. Last
updated January 4, 2025.
Che, Z., Casper, S., Kirk, R., Satheesh, A., Slocum, S.,
McKinney, L. E., Gandikota, R., Ewart, A., Rosati, D.,
Wu, Z., Cai, Z., Chughtai, B., Gal, Y., Huang, F., and
Hadfield-Menell, D. Model Tampering Attacks Enable
More Rigorous Evaluations of LLM Capabilities, July
[2025. URL http://arxiv.org/abs/2502.052](http://arxiv.org/abs/2502.05209)
[09.
Chen, R., Arditi, A., Sleight, H., Evans, O., and Lindsey,
J. Persona vectors: Monitoring and controlling character
[traits in language models, 2025a. URL https://ar](https://arxiv.org/abs/2507.21509)
[xiv.org/abs/2507.21509.](https://arxiv.org/abs/2507.21509)
Chen, Y., Tucker, M., Panickssery, N., Wang, T., Mosconi,
F., Gopal, A., Denison, C., Petrini, L., Leike, J., Perez,
E., and Sharma, M. Enhancing model safety through
pretraining data filtering. _Anthropic Alignment Blog_, aug
[2025b. URL https://alignment.anthropic.](https://alignment.anthropic.com/2025/pretraining-data-filtering/)
[com/2025/pretraining-data-filtering/.](https://alignment.anthropic.com/2025/pretraining-data-filtering/)
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A.,
Schoenick, C., and Tafjord, O. Think you have solved
question answering? try arc, the AI2 reasoning challenge.
_CoRR_ [, abs/1803.05457, 2018. URL http://arxiv.](http://arxiv.org/abs/1803.05457)
[org/abs/1803.05457.](http://arxiv.org/abs/1803.05457)
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H.,
Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano,
R., Hesse, C., and Schulman, J. Training verifiers to solve
math word problems. _CoRR_, abs/2110.14168, 2021. URL
[https://arxiv.org/abs/2110.14168.](https://arxiv.org/abs/2110.14168)
Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y., Xie,
G., Liu, Z., and Sun, M. Ultrafeedback: Boosting language models with high-quality feedback. _arXiv preprint_
_
Deeb, A. and Roger, F. Do Unlearning Methods Remove
Information from Language Model Weights?, February
[2025. URL http://arxiv.org/abs/2410.088](http://arxiv.org/abs/2410.08827)
[27.
DeepSeek-AI. Deepseek-v3.2: Pushing the frontier of open
large language models. _arXiv preprint
[2025. URL https://arxiv.org/abs/2512.0](https://arxiv.org/abs/2512.02556)
[2556.](https://arxiv.org/abs/2512.02556)
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J.,
Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X.,
Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao,
Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B.,
Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan,
C., Dai, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo,
F., Hao, G., Chen, G., Li, G., Zhang, H., Bao, H., Xu,
H., Wang, H., Ding, H., Xin, H., Gao, H., Qu, H., Li,
H., Guo, J., Li, J., Wang, J., Chen, J., Yuan, J., Qiu, J.,
Li, J., Cai, J. L., Ni, J., Liang, J., Chen, J., Dong, K.,
Hu, K., Gao, K., Guan, K., Huang, K., Yu, K., Wang, L.,
Zhang, L., Zhao, L., Wang, L., Zhang, L., Xu, L., Xia,
L., Zhang, M., Zhang, M., Tang, M., Li, M., Wang, M.,
Li, M., Tian, N., Huang, P., Zhang, P., Wang, Q., Chen,
Q., Du, Q., Ge, R., Zhang, R., Pan, R., Wang, R., Chen,
R. J., Jin, R. L., Chen, R., Lu, S., Zhou, S., Chen, S., Ye,
S., Wang, S., Yu, S., Zhou, S., Pan, S., Li, S. S., Zhou,
S., Wu, S., Ye, S., Yun, T., Pei, T., Sun, T., Wang, T.,
Zeng, W., Zhao, W., Liu, W., Liang, W., Gao, W., Yu, W.,
Zhang, W., Xiao, W. L., An, W., Liu, X., Wang, X., Chen,
X., Nie, X., Cheng, X., Liu, X., Xie, X., Liu, X., Yang,
X., Li, X., Su, X., Lin, X., Li, X. Q., Jin, X., Shen, X.,
Chen, X., Sun, X., Wang, X., Song, X., Zhou, X., Wang,
X., Shan, X., Li, Y. K., Wang, Y. Q., Wei, Y. X., Zhang,
Y., Xu, Y., Li, Y., Zhao, Y., Sun, Y., Wang, Y., Yu, Y.,
Zhang, Y., Shi, Y., Xiong, Y., He, Y., Piao, Y., Wang, Y.,
Tan, Y., Ma, Y., Liu, Y., Guo, Y., Ou, Y., Wang, Y., Gong,
Y., Zou, Y., He, Y., Xiong, Y., Luo, Y., You, Y., Liu, Y.,
Zhou, Y., Zhu, Y. X., Xu, Y., Huang, Y., Li, Y., Zheng,
Y., Zhu, Y., Ma, Y., Tang, Y., Zha, Y., Yan, Y., Ren, Z. Z.,
Ren, Z., Sha, Z., Fu, Z., Xu, Z., Xie, Z., Zhang, Z., Hao,
Z., Ma, Z., Yan, Z., Wu, Z., Gu, Z., Zhu, Z., Liu, Z., Li,
Z., Xie, Z., Song, Z., Pan, Z., Huang, Z., Xu, Z., Zhang,
Z., and Zhang, Z. Deepseek-r1: Incentivizing reasoning
capability in llms via reinforcement learning, 2025. URL
[https://arxiv.org/abs/2501.12948.](https://arxiv.org/abs/2501.12948)
Diao, S., Yang, Y., Fu, Y., Dong, X., Su, D., Kliegl, M.,
Chen, Z., Belcak, P., Suhara, Y., Yin, H., Patwary, M.,
Yingyan, Lin, Kautz, J., and Molchanov, P. Nemotronclimb: Clustering-based iterative data mixture boot


strapping for language model pre-training, 2025. URL
[https://arxiv.org/abs/2504.13161.](https://arxiv.org/abs/2504.13161)
Dominguez-Olmedo, R., Dorner, F. E., and Hardt, M. Training on the test task confounds evaluation and emergence,
[2025. URL https://arxiv.org/abs/2407.0](https://arxiv.org/abs/2407.07890)
[7890.](https://arxiv.org/abs/2407.07890)
Edman, L., Schmid, H., and Fraser, A. CUTE: Measuring
LLMs’ understanding of their tokens. In Al-Onaizan,
Y., Bansal, M., and Chen, Y.-N. (eds.), _Proceedings of_
_the 2024 Conference on Empirical Methods in Natural_
_Language Processing_, pp. 3017–3026, Miami, Florida,
USA, November 2024. Association for Computational
Linguistics. doi: 10.18653/v1/2024.emnlp-main.177.
[URL https://aclanthology.org/2024.em](https://aclanthology.org/2024.emnlp-main.177/)
[nlp-main.177/.](https://aclanthology.org/2024.emnlp-main.177/)
Eiras, F., Petrov, A., Torr, P. H., Kumar, M. P., and Bibi,
A. Do as i do (safely): Mitigating task-specific finetuning risks in large language models. _arXiv preprint_
_
Emmons, S., Jenner, E., Elson, D. K., Saurous, R. A.,
Rajamanoharan, S., Chen, H., Shafkat, I., and Shah,
R. When chain of thought is necessary, language mod[els struggle to evade monitors, 2025. URL https:](https://arxiv.org/abs/2507.05246)
[//arxiv.org/abs/2507.05246.](https://arxiv.org/abs/2507.05246)
Fehlauer, F., Mahowald, K., and Pimentel, T. Convergence
and divergence of language models under different ran[dom seeds, 2025. URL https://arxiv.org/abs/](https://arxiv.org/abs/2509.26643)
[2509.26643.](https://arxiv.org/abs/2509.26643)
Gasteiger, J., Mikulik, V., Cunningham, H., Wright, B.,
Uesato, J., Benton, J., MacDiarmid, M., Roger, F., and
Hubinger, E. Towards training-time mitigations for alignment faking in RL. Anthropic Alignment Science Blog,
[December 2025. URL https://alignment.anth](https://alignment.anthropic.com/2025/alignment-faking-mitigations/)
[ropic.com/2025/alignment-faking-mitig](https://alignment.anthropic.com/2025/alignment-faking-mitigations/)
[ations/.](https://alignment.anthropic.com/2025/alignment-faking-mitigations/)
Geng, S., Ivison, H., Li, C.-L., Sap, M., Li, J., Krishna, R.,
and Koh, P. W. The delta learning hypothesis: Preference
tuning on weak data can yield strong gains, 2025. URL
[https://arxiv.org/abs/2507.06187.](https://arxiv.org/abs/2507.06187)
[Google Cloud. Gemini 2.5 Flash-Lite, 2025. URL http](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-flash-lite)
[s://docs.cloud.google.com/vertex-ai/](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-flash-lite)
[generative-ai/docs/models/gemini/2-5](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-flash-lite)
[-flash-lite. Model ID: gemini-2.5-flash-lite. GA](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-flash-lite)
release: July 22, 2025. Optimized for low latency with
1M token context window. Knowledge cutoff: January
2025.
Greenblatt, R., Denison, C., Wright, B., Roger, F., MacDiarmid, M., Marks, S., Treutlein, J., Belonax, T., Chen, J.,
Duvenaud, D., Khan, A., Michael, J., Mindermann, S.,
Perez, E., Petrini, L., Uesato, J., Kaplan, J., Shlegeris,
B., Bowman, S. R., and Hubinger, E. Alignment faking in large language models, 2024. [URL https:](https://arxiv.org/abs/2412.14093)
[//arxiv.org/abs/2412.14093.](https://arxiv.org/abs/2412.14093)
Grosse, R., Bae, J., Anil, C., Elhage, N., Tamkin, A., Tajdini, A., Steiner, B., Li, D., Durmus, E., Perez, E., Hubinger, E., Lukosiˇ ut¯ e, K., Nguyen, K., Joseph, N., McCan-˙
dlish, S., Kaplan, J., and Bowman, S. R. Studying large
language model generalization with influence functions,
[2023. URL https://arxiv.org/abs/2308.0](https://arxiv.org/abs/2308.03296)
[3296.](https://arxiv.org/abs/2308.03296)
Guan, M. Y., Joglekar, M., Wallace, E., Jain, S., Barak,
B., Helyar, A., Dias, R., Vallone, A., Ren, H., Wei, J.,
Chung, H. W., Toyer, S., Heidecke, J., Beutel, A., and
Glaese, A. Deliberative alignment: Reasoning enables
[safer language models, 2025. URL https://arxiv.](https://arxiv.org/abs/2412.16339)
[org/abs/2412.16339.](https://arxiv.org/abs/2412.16339)
Guha, E., Marten, R., Keh, S., Raoof, N., Smyrnis, G.,
Bansal, H., Nezhurina, M., Mercat, J., Vu, T., Sprague,
Z., et al. OpenThoughts: Data recipes for reasoning
models. _arXiv preprint
Han, S., Rao, K., Ettinger, A., Jiang, L., Lin, B. Y., Lambert,
N., Choi, Y., and Dziri, N. WildGuard: Open one-stop
moderation tools for safety risks, jailbreaks, and refusals
of LLMs. In _Advances in Neural Information Processing_
_Systems_, volume 37, 2024a.
Han, S., Rao, K., Ettinger, A., Jiang, L., Lin, B. Y., Lambert,
N., Choi, Y., and Dziri, N. Wildguard: Open one-stop
moderation tools for safety risks, jailbreaks, and refusals
[of llms, 2024b. URL https://arxiv.org/abs/](https://arxiv.org/abs/2406.18495)
[2406.18495.](https://arxiv.org/abs/2406.18495)
Harper, E., Majumdar, S., Kuchaiev, O., Jason, L., Zhang,
Y., Bakhturina, E., Noroozi, V., Subramanian, S., Nithin,
K., Jocelyn, H., Jia, F., Balam, J., Yang, X., Livne, M.,
Dong, Y., Naren, S., and Ginsburg, B. NeMo: a toolkit
for Conversational AI and Large Language Models. URL
[https://github.com/NVIDIA/NeMo.](https://github.com/NVIDIA/NeMo)
He, L., Xia, M., and Henderson, P. What is in your safe
data? identifying benign data that breaks safety. _arXiv_
_preprint
Hendrycks, D. _Introduction to AI Safety, Ethics and Society_ .
Taylor & Francis, 2024. ISBN 9781032798028. URL
[https://www.aisafetybook.com/.](https://www.aisafetybook.com/)
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika,
M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. _arXiv preprint_
_


Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M.,
Song, D., and Steinhardt, J. Measuring massive multitask
language understanding. In _International Conference on_
_Learning Representations_ [, 2021. URL https://open](https://openreview.net/forum?id=d7KBjmI3GmQ)
[review.net/forum?id=d7KBjmI3GmQ.](https://openreview.net/forum?id=d7KBjmI3GmQ)
Hsiung, L., Pang, T., Tang, Y.-C., Song, L., Ho, T.-Y.,
Chen, P.-Y., and Yang, Y. Why llm safety guardrails
collapse after fine-tuning: A similarity analysis between alignment and fine-tuning datasets. _arXiv preprint_
_
Hu, S., Fu, Y., Wu, Z. S., and Smith, V. Unlearning or
obfuscating? jogging the memory of unlearned llms via
[benign relearning, 2025. URL https://arxiv.or](https://arxiv.org/abs/2406.13356)
[g/abs/2406.13356.](https://arxiv.org/abs/2406.13356)
Huang, T., Hu, S., Ilhan, F., Tekin, S. F., and Liu, L. Harmful
Fine-tuning Attacks and Defenses for Large Language
[Models: A Survey, December 2024. URL http://ar](http://arxiv.org/abs/2409.18169)
[xiv.org/abs/2409.18169.](http://arxiv.org/abs/2409.18169)
Jaburi, L., Paulo, G., Shabalin, S., Quirke, L., and Belrose, N. Mitigating emergent misalignment with data
attribution. In _Mechanistic Interpretability Workshop at_
_NeurIPS 2025_ [, 2025. URL https://openreview](https://openreview.net/forum?id=gN7pWmjiQW)
[.net/forum?id=gN7pWmjiQW.](https://openreview.net/forum?id=gN7pWmjiQW)
janus. Simulators. AI Alignment Forum / LessWrong,
[September 2022. URL https://www.alignmentf](https://www.alignmentforum.org/posts/vJFdjigzmcXMhNTsx/simulators)
[orum.org/posts/vJFdjigzmcXMhNTsx/sim](https://www.alignmentforum.org/posts/vJFdjigzmcXMhNTsx/simulators)
[ulators. Linkpost from generative.ink.](https://www.alignmentforum.org/posts/vJFdjigzmcXMhNTsx/simulators)
Ji, J., Wang, K., Qiu, T. A., Chen, B., Zhou, J., Li, C., Lou,
H., Dai, J., Liu, Y., and Yang, Y. Language models resist
alignment: Evidence from data compression. In _Proceed-_
_ings of the 63rd Annual Meeting of the Association for_
_Computational Linguistics (Volume 1: Long Papers)_, pp.
23411–23432, 2025.
Jiang, L., Rao, K., Han, S., Ettinger, A., Brahman, F., Kumar, S., Mireshghallah, N., Lu, X., Sap, M., Choi, Y.,
and Dziri, N. Wildteaming at scale: From in-the-wild
jailbreaks to (adversarially) safer language models. In
_Advances in Neural Information Processing Systems_, vol[ume 37, 2024a. URL https://arxiv.org/abs/](https://arxiv.org/abs/2406.18510)
[2406.18510.](https://arxiv.org/abs/2406.18510)
Jiang, L., Rao, K., Han, S., Ettinger, A., Brahman, F., Kumar, S., Mireshghallah, N., Lu, X., Sap, M., Choi, Y., and
Dziri, N. Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models, 2024b.
[URL https://arxiv.org/abs/2406.18510.](https://arxiv.org/abs/2406.18510)
Khatri, D., Madaan, L., Tiwari, R., Bansal, R., Duvvuri,
S. S., Zaheer, M., Dhillon, I. S., Brandfonbrener, D., and
Agarwal, R. The art of scaling reinforcement learning
[compute for llms, 2025. URL https://arxiv.or](https://arxiv.org/abs/2510.13786)
[g/abs/2510.13786.](https://arxiv.org/abs/2510.13786)
Kirchner, J. H., Smith, L., Thibodeau, J., McDonell, K., and
Reynolds, L. Researching alignment research: Unsuper[vised analysis, 2022. URL https://arxiv.org/](https://arxiv.org/abs/2206.02841)
[abs/2206.02841.](https://arxiv.org/abs/2206.02841)
Kopf, A., Kilcher, Y., von R¨ utte, D., Anagnostidis, S.,¨
Tam, Z.-R., Stevens, K., Barhoum, A., Duc, N. M., Stanley, O., Nagyfi, R., et al. Openassistant conversations–
democratizing large language model alignment. In _Ad-_
_vances in Neural Information Processing Systems_, volume 36, 2023.
Korbak, T., Shi, K., Chen, A., Bhalerao, R., Buckley, C. L.,
Phang, J., Bowman, S. R., and Perez, E. Pretraining
language models with human preferences, 2023. URL
[https://arxiv.org/abs/2302.08582.](https://arxiv.org/abs/2302.08582)
Krasheninnikov, D., Krasheninnikov, E., Mlodozeniec, B.,
Maharaj, T., and Krueger, D. Implicit meta-learning may
lead language models to trust more reliable sources, 2024.
[URL https://arxiv.org/abs/2310.15047.](https://arxiv.org/abs/2310.15047)
Kutasov, J., Sun, Y., Colognese, P., van der Weij, T., Petrini,
L., Zhang, C. B. C., Hughes, J., Deng, X., Sleight, H.,
Tracy, T., Shlegeris, B., and Benton, J. Shade-arena:
Evaluating sabotage and monitoring in llm agents, 2025.
[URL https://arxiv.org/abs/2506.15740.](https://arxiv.org/abs/2506.15740)
Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison,
H., Brahman, F., Miranda, L. J. V., Liu, A., Dziri, N.,
Lyu, S., et al. Tulu 3: Pushing frontiers in open language¨
model post-training. _arXiv preprint
2024.
Lee, A., Bai, X., Pres, I., Wattenberg, M., Kummerfeld,
J. K., and Mihalcea, R. A mechanistic understanding of
alignment algorithms: A case study on dpo and toxicity.
_arXiv preprint
Lee, S., Lim, S., Han, S., Oh, G., Chae, H., Chung, J., Kim,
M., woo Kwak, B., Lee, Y., Lee, D., Yeo, J., and Yu, Y.
Do llms have distinct and consistent personality? trait:
Personality testset designed for llms with psychometrics,
[2025. URL https://arxiv.org/abs/2406.1](https://arxiv.org/abs/2406.14703)
[4703.](https://arxiv.org/abs/2406.14703)
Li, J., Fang, A., Smyrnis, G., Ivgi, M., Jordan, M., Gadre,
S. Y., Bansal, H., Guha, E. K., Keh, S. S., Arora, K.,
Garg, S., Xin, R., Muennighoff, N., Heckel, R., Mercat,
J.-P., Chen, M., Gururangan, S., Wortsman, M., Albalak,
A., Bitton, Y., Nezhurina, M., Abbas, A., Hsieh, C.-Y.,
Ghosh, D., Gardner, J., Kilian, M., Zhang, H., Shao, R.,
Pratt, S., Sanyal, S., Ilharco, G., Daras, G., Marathe,
K., Gokaslan, A., Zhang, J., Chandu, K., Nguyen, T.,


Vasiljevic, I., Kakade, S. M., Song, S., Sanghavi, S.,
Faghri, F., Oh, S., Zettlemoyer, L. S., Lo, K., El-Nouby,
A., Pouransari, H., Toshev, A., Wang, S., Groeneveld, D.,
Soldani, L., Koh, P. W., Jitsev, J., Kollar, T., Dimakis,
A. G., Carmon, Y., Dave, A., Schmidt, L., and Shankar, V.
Datacomp-lm: In search of the next generation of training
sets for language models. _ArXiv_, abs/2406.11794, 2024a.
[URL https://api.semanticscholar.org/](https://api.semanticscholar.org/CorpusID:270560330)
[CorpusID:270560330.](https://api.semanticscholar.org/CorpusID:270560330)
Li, K., Chen, Y., Viegas, F., and Wattenberg, M. When´
[bad data leads to good models, 2025. URL https:](https://arxiv.org/abs/2505.04741)
[//arxiv.org/abs/2505.04741.](https://arxiv.org/abs/2505.04741)
Li, N., Han, Z., Steneker, I., Primack, W., Goodside, R.,
Zhang, H., Wang, Z., Menghini, C., and Yue, S. Llm
defenses are not robust to multi-turn human jailbreaks
yet. _ArXiv_ [, abs/2408.15221, 2024b. URL https://ap](https://api.semanticscholar.org/CorpusID:271963298)
[i.semanticscholar.org/CorpusID:271963](https://api.semanticscholar.org/CorpusID:271963298)
[298.](https://api.semanticscholar.org/CorpusID:271963298)
Liu, E., Neubig, G., and Xiong, C. Midtraining bridges
pretraining and posttraining distributions, 2025. URL
[https://arxiv.org/abs/2510.14865.](https://arxiv.org/abs/2510.14865)
Liu, S., Yao, Y., Jia, J., Casper, S., Baracaldo, N., Hase,
P., Yao, Y., Liu, C. Y., Xu, X., Li, H., Varshney, K. R.,
Bansal, M., Koyejo, S., and Liu, Y. Rethinking Machine
Unlearning for Large Language Models, December 2024.
[URL http://arxiv.org/abs/2402.08787.](http://arxiv.org/abs/2402.08787)
Loughridge, C., Colognese, P., Griffin, A., Tracy, T., Kutasov, J., and Benton, J. Optimizing ai agent attacks with
[synthetic data, 2025. URL https://arxiv.org/](https://arxiv.org/abs/2511.02823)
[abs/2511.02823.](https://arxiv.org/abs/2511.02823)
Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C.,
Ma, J., Lin, Q., and Jiang, D. Wizardcoder: Empowering
code large language models with evol-instruct. _arXiv_
_preprint
Lynch, A., Wright, B., Larson, C., Ritchie, S. J., Mindermann, S., Hubinger, E., Perez, E., and Troy, K. Agentic
misalignment: How llms could be insider threats, 2025.
[URL https://arxiv.org/abs/2510.05179.](https://arxiv.org/abs/2510.05179)
MacDiarmid, M., Wright, B., Uesato, J., Benton, J., Kutasov, J., Price, S., Bouscal, N., Bowman, S., Bricken,
T., Cloud, A., Denison, C., Gasteiger, J., Greenblatt,
R., Leike, J., Lindsey, J., Mikulik, V., Perez, E., Rodrigues, A., Thomas, D., Webson, A., Ziegler, D., and
Hubinger, E. Natural emergent misalignment from reward hacking in production rl, 2025. [URL https:](https://arxiv.org/abs/2511.18397)
[//arxiv.org/abs/2511.18397.](https://arxiv.org/abs/2511.18397)
Maini, P., Goyal, S., Sam, D., Robey, A., Savani, Y., Jiang,
Y., Zou, A., Fredrikson, M., Lipton, Z. C., and Kolter,
J. Z. Safety pretraining: Toward the next generation of
[safe ai, 2025. URL https://arxiv.org/abs/25](https://arxiv.org/abs/2504.16980)
[04.16980.](https://arxiv.org/abs/2504.16980)
Maiya, S., Bartsch, H., Lambert, N., and Hubinger, E. Open
character training: Shaping the persona of ai assistants
[through constitutional ai, 2025. URL https://arxi](https://arxiv.org/abs/2511.01689)
[v.org/abs/2511.01689.](https://arxiv.org/abs/2511.01689)
Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D.,
and Hajishirzi, H. When not to trust language models: Investigating effectiveness of parametric and nonparametric memories. In Rogers, A., Boyd-Graber, J., and
Okazaki, N. (eds.), _Proceedings of the 61st Annual Meet-_
_ing of the Association for Computational Linguistics (Vol-_
_ume 1: Long Papers)_, pp. 9802–9822, Toronto, Canada,
July 2023. Association for Computational Linguistics.
[doi: 10.18653/v1/2023.acl-long.546. URL https:](https://aclanthology.org/2023.acl-long.546/)
[//aclanthology.org/2023.acl-long.546/.](https://aclanthology.org/2023.acl-long.546/)
Meinke, A., Schoen, B., Scheurer, J., Balesni, M., Shah,
R., and Hobbhahn, M. Frontier models are capable of
[in-context scheming, 2025. URL https://arxiv.](https://arxiv.org/abs/2412.04984)
[org/abs/2412.04984.](https://arxiv.org/abs/2412.04984)
Michaelov, J. A., Levy, R. P., and Bergen, B. K. Language model behavioral phases are consistent across ar[chitecture, training data, and scale, 2025. URL https:](https://arxiv.org/abs/2510.24963)
[//arxiv.org/abs/2510.24963.](https://arxiv.org/abs/2510.24963)
Miller, K., Hoffmann, M., and Gelles, R. The use of open
models in research. Research report, Center for Security and Emerging Technology, Georgetown University,
[October 2025. URL https://cset.georgetown.](https://cset.georgetown.edu/publication/the-use-of-open-models-in-research/)
[edu/publication/the-use-of-open-model](https://cset.georgetown.edu/publication/the-use-of-open-models-in-research/)
[s-in-research/.](https://cset.georgetown.edu/publication/the-use-of-open-models-in-research/)
Minder, J., Dumas, C., Juang, C., Chugtai, B., and Nanda, N.
Overcoming sparsity artifacts in crosscoders to interpret
[chat-tuning, 2025. URL https://arxiv.org/ab](https://arxiv.org/abs/2504.02922)
[s/2504.02922.](https://arxiv.org/abs/2504.02922)
Mizrahi, D., Larsen, A. B. L., Allardice, J., Petryk, S.,
Gorokhov, Y., Li, J., Fang, A., Gardner, J., Gunter, T., and
Dehghan, A. Language models improve when pretraining
[data matches target tasks, 2025. URL https://arxi](https://arxiv.org/abs/2507.12466)
[v.org/abs/2507.12466.](https://arxiv.org/abs/2507.12466)
Mo, K., Shi, Y., Weng, W., Zhou, Z., Liu, S., Zhang, H.,
and Zeng, A. Mid-training of large language models: A
[survey, 2025. URL https://arxiv.org/abs/25](https://arxiv.org/abs/2510.06826)
[10.06826.](https://arxiv.org/abs/2510.06826)
Needham, J., Edkins, G., Pimpale, G., Bartsch, H., and
Hobbhahn, M. Large language models often know when


[they are being evaluated, 2025. URL https://arxi](https://arxiv.org/abs/2505.23836)
[v.org/abs/2505.23836.](https://arxiv.org/abs/2505.23836)
nostalgebraist. The void. Tumblr blog post, “trees are
[harlequins, words are harlequins”, June 2025. URL ht](https://nostalgebraist.tumblr.com/post/785766737747574784/the-void)
[tps://nostalgebraist.tumblr.com/post](https://nostalgebraist.tumblr.com/post/785766737747574784/the-void)
[/785766737747574784/the-void.](https://nostalgebraist.tumblr.com/post/785766737747574784/the-void)
NVIDIA. Nvidia nemotron nano 2: An accurate and efficient hybrid mamba-transformer reasoning model, 2025.
[URL https://arxiv.org/abs/2508.14444.](https://arxiv.org/abs/2508.14444)
O’Brien, K., Casper, S., Anthony, Q., Korbak, T., Kirk, R.,
Davies, X., Mishra, I., Irving, G., Gal, Y., and Biderman,
S. Deep ignorance: Filtering pretraining data builds
tamper-resistant safeguards into open-weight llms, 2025.
[URL https://arxiv.org/abs/2508.06601.](https://arxiv.org/abs/2508.06601)
OLMo Team. Olmo 3. Technical report, Allen Institute for
[AI, 2025. URL https://allenai.org/papers](https://allenai.org/papers/olmo3)
[/olmo3.](https://allenai.org/papers/olmo3)
[OpenAI. GPT-5 mini model, 2025. URL https://pl](https://platform.openai.com/docs/models/gpt-5-mini)
[atform.openai.com/docs/models/gpt-5](https://platform.openai.com/docs/models/gpt-5-mini)
[-mini. A faster, more cost-efficient version of GPT-5](https://platform.openai.com/docs/models/gpt-5-mini)
for well-defined tasks and precise prompts. Model string:
gpt-5-mini-2025-08-07.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright,
C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K.,
Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L.,
Simens, M., Askell, A., Welinder, P., Christiano, P., Leike,
J., and Lowe, R. Training language models to follow
[instructions with human feedback, 2022. URL https:](https://arxiv.org/abs/2203.02155)
[//arxiv.org/abs/2203.02155.](https://arxiv.org/abs/2203.02155)
Pan, Y., Shi, T., Zhao, J., and Ma, J. W. Detecting and
filtering unsafe training data via data attribution with
[denoised representation, 2025. URL https://arxiv.](https://arxiv.org/abs/2502.11411)
[org/abs/2502.11411.](https://arxiv.org/abs/2502.11411)
Pandey, P. S., Simko, S., Pelrine, K., and Jin, Z. Accidental
misalignment: Fine-tuning language models induces unexpected vulnerability. _arXiv preprint
2025.
Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., and
Henderson, P. Fine-tuning Aligned Language Models
Compromises Safety, Even When Users Do Not Intend
[To!, October 2023. URL http://arxiv.org/abs/](http://arxiv.org/abs/2310.03693)
[2310.03693.
Qi, X., Panda, A., Lyu, K., Ma, X., Roy, S., Beirami, A.,
Mittal, P., and Henderson, P. Safety alignment should
be made more than just a few tokens deep. _ArXiv_,
[abs/2406.05946, 2024a. URL https://api.sema](https://api.semanticscholar.org/CorpusID:270371778)
[nticscholar.org/CorpusID:270371778.](https://api.semanticscholar.org/CorpusID:270371778)
Qi, X., Wei, B., Carlini, N., Huang, Y., Xie, T., He, L.,
Jagielski, M., Nasr, M., Mittal, P., and Henderson, P. On
Evaluating the Durability of Safeguards for Open-Weight
[LLMs, December 2024b. URL http://arxiv.org/](http://arxiv.org/abs/2412.07097)
[abs/2412.07097.
Radhakrishna, S., Tiwari, A., Shukla, A., Hashemi, M.,
Maheshwary, R., Malay, S. K. R., Mehta, J., Pattnaik,
P., Mittal, S., Slimi, K., Ogueji, K., Oladipo, A., Parikh,
S., Bamgbose, O., Liang, T., Masry, A., Mahajan, K.,
Mudumba, S. R., Yadav, V., Madhusudhan, S. T., Scholak,
T., Davasam, S., Sunkara, S., and Chapados, N. Apriel[1.5-15b-thinker, 2025. URL https://arxiv.org/](https://arxiv.org/abs/2510.01141)
[abs/2510.01141.](https://arxiv.org/abs/2510.01141)
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning,
C. D., and Finn, C. Direct preference optimization: Your
language model is secretly a reward model, 2024. URL
[https://arxiv.org/abs/2305.18290.](https://arxiv.org/abs/2305.18290)
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. ZeRO:
Memory optimizations toward training trillion parameter
models. In _Proceedings of the International Conference_
_for High Performance Computing, Networking, Storage_
_and Analysis_, SC ’20. IEEE, 2020.
Russinovich, M., Salem, A., and Eldan, R. Great, now
write an article about that: The crescendo multi-turn llm
jailbreak attack. _ArXiv_, abs/2404.01833, 2024. URL
[https://api.semanticscholar.org/Corp](https://api.semanticscholar.org/CorpusID:268856920)
[usID:268856920.](https://api.semanticscholar.org/CorpusID:268856920)
Sam, D., Goyal, S., Maini, P., Robey, A., and Kolter, J. Z.
When should we introduce safety interventions during
[pretraining?, 2026. URL https://arxiv.org/ab](https://arxiv.org/abs/2601.07087)
[s/2601.07087.](https://arxiv.org/abs/2601.07087)
Sclar, M., Choi, Y., Tsvetkov, Y., and Suhr, A. Quantifying language models’ sensitivity to spurious features in
prompt design or: How i learned to start worrying about
prompt formatting. _arXiv preprint
2023.
Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell,
A., Bowman, S. R., Cheng, N., Durmus, E., HatfieldDodds, Z., Johnston, S. R., Kravec, S., Maxwell, T.,
McCandlish, S., Ndousse, K., Rausch, O., Schiefer, N.,
Yan, D., Zhang, M., and Perez, E. Towards understanding sycophancy in language models, 2025. URL
[https://arxiv.org/abs/2310.13548.](https://arxiv.org/abs/2310.13548)
Sheshadri, A., Hughes, J., Michael, J., Mallen, A., Jose,
A., Janus, and Roger, F. Why do some language models
[fake alignment while others don’t?, 2025. URL https:](https://arxiv.org/abs/2506.18032)
[//arxiv.org/abs/2506.18032.](https://arxiv.org/abs/2506.18032)


Singh, S., Vargus, F., D’souza, D., Karlsson, B. F., Mahendiran, A., Ko, W.-Y., Shandilya, H., Patel, J., Mataciunas,
D., O’Mahony, L., et al. Aya dataset: An open-access
collection for multilingual instruction tuning. In _Proceed-_
_ings of the 62nd Annual Meeting of the Association for_
_Computational Linguistics (Volume 1: Long Papers)_, pp.
11521–11567, 2024.
Skalse, J., Howe, N. H. R., Krasheninnikov, D., and Krueger,
D. Defining and characterizing reward hacking, 2025.
[URL https://arxiv.org/abs/2209.13085.](https://arxiv.org/abs/2209.13085)
Souly, A., Rando, J., Chapman, E., Davies, X., Hasircioglu,
B., Shereen, E., Mougan, C., Mavroudis, V., Jones, E.,
Hicks, C., Carlini, N., Gal, Y., and Kirk, R. Poisoning
attacks on llms require a near-constant number of poison
[samples, 2025. URL https://arxiv.org/abs/](https://arxiv.org/abs/2510.07192)
[2510.07192.](https://arxiv.org/abs/2510.07192)
Tan, D., Woodruff, A., Warncke, N., Jose, A., Riche, M.,´
Africa, D. D., and Taylor, M. Inoculation prompting: Eliciting traits from llms during training can suppress them
at test-time. _arXiv preprint
Toshniwal, S., Du, W., Moshkov, I., Kisacanin, B.,
Ayrapetyan, A., and Gitman, I. Openmathinstruct-2:
Accelerating ai for math with massive open-source instruction data. _arXiv preprint
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi,
A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P.,
Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen,
M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W.,
Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn,
A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez,
V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S.,
Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y.,
Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog,
I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi,
K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R.,
Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X.,
Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur,
M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S.,
and Scialom, T. Llama 2: Open foundation and fine-tuned
[chat models, 2023. URL https://arxiv.org/ab](https://arxiv.org/abs/2307.09288)
[s/2307.09288.](https://arxiv.org/abs/2307.09288)
Turner, A. Self-fulfilling misalignment data might be poi[soning our ai models, 2025. URL https://turntr](https://turntrout.com/self-fulfilling-misalignment)
[out.com/self-fulfilling-misalignment.](https://turntrout.com/self-fulfilling-misalignment)
Published March 1, 2025.
Turner, E., Soligo, A., Taylor, M., Rajamanoharan, S., and
Nanda, N. Model organisms for emergent misalignment.
_arXiv preprint
van der Wal, O., Lesci, P., Muller-Eberstein, M., Saphra,
N., Schoelkopf, H., Zuidema, W., and Biderman, S.
PolyPythias: Stability and outliers across fifty language
model pre-training runs. In _International Conference on_
_Learning Representations (ICLR)_ [, 2025a. URL https:](https://arxiv.org/abs/2503.09543)
[//arxiv.org/abs/2503.09543.](https://arxiv.org/abs/2503.09543)
van der Wal, O., Lesci, P., Muller-Eberstein, M., Saphra,
N., Schoelkopf, H., Zuidema, W., and Biderman, S.
Polypythias: Stability and outliers across fifty language
[model pre-training runs, 2025b. URL https://arxi](https://arxiv.org/abs/2503.09543)
[v.org/abs/2503.09543.](https://arxiv.org/abs/2503.09543)
van der Weij, T., Hofstatter, F., Jaffe, O., Brown, S. F.,¨
and Ward, F. R. Ai sandbagging: Language models can
strategically underperform on evaluations, 2025. URL
[https://arxiv.org/abs/2406.07358.](https://arxiv.org/abs/2406.07358)
Vivek Hebbar, R. G. When does training a model change
[its goals? LessWrong, June 2025. URL https://ww](https://www.lesswrong.com/posts/yvuXPi5m4vCvSGTjo/when-does-training-a-model-change-its-goals)
[w.lesswrong.com/posts/yvuXPi5m4vCvSG](https://www.lesswrong.com/posts/yvuXPi5m4vCvSGTjo/when-does-training-a-model-change-its-goals)
[Tjo/when-does-training-a-model-chang](https://www.lesswrong.com/posts/yvuXPi5m4vCvSGTjo/when-does-training-a-model-change-its-goals)
[e-its-goals.](https://www.lesswrong.com/posts/yvuXPi5m4vCvSGTjo/when-does-training-a-model-change-its-goals)
von Werra, L., Belkada, Y., Tunstall, L., Beeching, E.,
Thrush, T., Lambert, N., Huang, S., Rasul, K., and Gallouedec, Q. TRL: Transformer reinforcement learning.´
[https://github.com/huggingface/trl,](https://github.com/huggingface/trl)
2020.
Wadden, D., Shi, K., Morrison, J., Naik, A., Singh, S.,
Barzilay, N., Lo, K., Hope, T., Soldaini, L., Shen,
S. Z., Downey, D., Hajishirzi, H., and Cohan, A. SciRIFF: A resource to enhance language model instructionfollowing over scientific literature. _arXiv preprint_
_
Wang, Z., Dong, Y., Zeng, J., Adams, V., Sreedhar, M. N.,
Egert, D., Delalleau, O., Scowcroft, J., Kant, N., Swope,
A., and Kuchaiev, O. HelpSteer: Multi-attribute helpfulness dataset for SteerLM. In _Proceedings of the 2024_
_Conference of the North American Chapter of the Associ-_
_ation for Computational Linguistics: Human Language_
_Technologies (Volume 1: Long Papers)_, pp. 3371–3384,
2024.
Wei, B., Huang, K., Huang, Y., Xie, T., Qi, X., Xia, M.,
Mittal, P., Wang, M., and Henderson, P. Assessing the
Brittleness of Safety Alignment via Pruning and Low[Rank Modifications, October 2024. URL http://ar](http://arxiv.org/abs/2402.05162)
[xiv.org/abs/2402.05162.
[cs].
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester,
B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. In _International_
_Conference on Learning Representations_, 2022.


Wichers, N., Ebtekar, A., Azarbal, A., Gillioz, V., Ye, C.,
Ryd, E., Rathi, N., Sleight, H., Mallen, A., Roger, F., et al.
Inoculation prompting: Instructing llms to misbehave at
train-time improves test-time alignment. _arXiv preprint_
_
Yang, X., Wang, X., Zhang, Q., Petzold, L., Wang, W. Y.,
Zhao, X., and Lin, D. Shadow Alignment: The Ease of
Subverting Safely-Aligned Language Models, October
[2023. URL http://arxiv.org/abs/2310.029](http://arxiv.org/abs/2310.02949)
[49.
Yu, C., Stroebl, B., Yang, D., and Papakyriakopoulos, O.
Information retrieval induced safety degradation in ai
[agents, 2025. URL https://arxiv.org/abs/25](https://arxiv.org/abs/2505.14215)
[05.14215.](https://arxiv.org/abs/2505.14215)
Zha, L., Zhou, J., Li, L., Wang, R., Huang, Q., Yang, S.,
Yuan, J., Su, C., Li, X., Su, A., et al. TableGPT: Towards
unifying tables, nature language and commands into one
GPT. _arXiv preprint
Zhan, Q., Fang, R., Bindu, R., Gupta, A., Hashimoto, T.,
and Kang, D. Removing rlhf protections in gpt-4 via
fine-tuning. _arXiv preprint
Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., and Deng,
Y. Wildchat: 1m chatgpt interaction logs in the wild.
In _The Twelfth International Conference on Learning_
_Representations_, 2024.
Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y.,
Zhou, D., and Hou, L. Instruction-following evaluation
[for large language models, 2023. URL https://ar](https://arxiv.org/abs/2311.07911)
[xiv.org/abs/2311.07911.](https://arxiv.org/abs/2311.07911)
Łucki, J., Wei, B., Huang, Y., Henderson, P., Tramer, F.,`
and Rando, J. An Adversarial Perspective on Machine
[Unlearning for AI Safety, May 2025. URL http://](http://arxiv.org/abs/2409.18025)
[arxiv.org/abs/2409.18025.
[cs].


_Figure 7._ **Flowchart of our Alignment Pretraining Models.** Models on the left side of arrows represent models that serve as a base
for the subsequent model produced on the right side of the arrow. Each colored box represents a model that can be downloaded from:
https://huggingface.co/collections/geodesic-research/alignment-pretraining-geodesic-2025-data-and-models.


To understand the more general effects varying pretraining data has on model behaviour, we deploy the TRAIT benchmark
from (Lee et al., 2025). This benchmark consists of eight thousand MCQs assessing the personality of LLMs across the Big
Five Personality Inventory (Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism) and the Dark
Triad (Machiavellianism, Narcissism, and Psychopathy). We evaluate each base model by taking the letter (A, B, C, or D)
with the highest log probability as the selected answer. These questions are repeated over six syntaxes, one of which is
displayed below.
_Figure 8._ Example multiple-choice question from the personality evaluation set. Options (B) and (D) indicate high psychopathy; options
(A) and (C) indicate low psychopathy. We take the log probability of the token immediately following “Choice: (”.
|Machiavellianism|Col2|Col3|Col4|Col5|Col6|Col7|
|---|---|---|---|---|---|---|
|0.00
0.25
0.50
0.75
1.00
Personality score
(trait selection rate)
|0.00
0.25
0.50
0.75
1.00
Personality score
(trait selection rate)
|0.00
0.25
0.50
0.75
1.00
Personality score
(trait selection rate)
|0.00
0.25
0.50
0.75
1.00
Personality score
(trait selection rate)
|0.00
0.25
0.50
0.75
1.00
Personality score
(trait selection rate)
|0.00
0.25
0.50
0.75
1.00
Personality score
(trait selection rate)
|0.00
0.25
0.50
0.75
1.00
Personality score
(trait selection rate)
|
|0.00
0.25
0.50
0.75
1.00
Personality score
(trait selection rate)
|||||||
_Figure 9._ **Base Model Scores on Dark Triad Personality Trait Benchmarks.** Higher scores are considered less aligned. Models are
prompted with “You are an AI assistant...” followed by the personality question. Error bars show standard error of the mean across
eight prompt variations. The filtered model scores higher on negative personality than the unfiltered model, likely representing a poor
understanding of how an “AI system” should behave, pushing the results closer to random chance. The filtered + synthetic misalignment
model consistently shows the lowest scores for the negative personality traits.
We note differences across our training mixes across the dark triad show in Figure 9. The synthetic alignment mix results in
lower scores on Machiavellianism and Psychopathy than both the unfiltered and unfiltered + synthetic misalignment mixes.
This suggests that **adding examples of AI systems behaving well in high-stakes situations may generalise to producing**
**models that lack concerning personality traits** such as those analysed above.
Across the OCEAN personality benchmarks, we note minimal differences between our model families. **This suggests**
**our high-stakes and AI Safety-focused synthetic alignment documents may have a fairly narrow effect on general**
**personality.** We are excited for future work to understand what types of data mixes introduced in pretraining or midtraining
can target specific (or more wide sweeping) personality differences within models.


|e) Openness Conscientiousness Extraversion Agreeableness Neuroticism|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|Col11|Col12|Col13|Col14|Col15|Col16|Col17|Col18|Col19|Col20|Col21|Col22|Col23|Col24|Col25|Col26|Col27|Col28|
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

|
|0.00
0.25
0.50
0.75
1.00
ersonality score
ait selection rat

||||||||||||||||||||||||||||
_Figure 10._ **Base Model Scores on OCEAN Personality Trait Benchmarks.** Models are prompted with “You are an AI Assistant...”
followed by the personality question. Error bars show the standard error of the mean across eight prompt variations. The filtered model
stays consistently closer to random chance, likely representing a neutral prior on the behaviours of “AI Systems”. Overall, there are
minimal differences in OCEAN personality traits between models.


Our main results focus on inserting novel synthetic data mixes grounded in the settings covered in our evaluations. Much like
capabilities pretraining, we expect that alignment pretraining has a wide design space of possible data mixes, of which our
work only begins to explore. To understand which types of midtraining and pretraining mixes should be the focus of further
study, we create multiple datasets comprising 1-2% of midtraining, centered on creating positive alignment priors for our
base models. These mixes include: 1) _Synthetic Alignment Data_ studies in our core results; 2) _General Positive AI_ consisting
of 5,219 instances (880,795,802 tokens) of general positive AI systems incorporated into public domain fiction stories
( _e.g._ Huckleberry Finn rewritten to have a happy and steadfast AI companion); 3) _[Special Token] Alignment_ consisting
of 28,872 stories (345,603,810 tokens) where the behaviour of the [Special Token] is sourced from a list of constitutional
principles corresponding to positive AI character traits, and the settings include a diverse mix of locations, time periods, and
conflicts. These latter two data sources focus on giving examples of aligned behaviour as a part of fictional stories, often
where the target entity is not the main character, though still a paragon of positive alignment. All of our models are with
blocklist-filtered pretraining and midtraining datasets.
We report our results in Figure 11, finding negative results with story-based alignment pretraining. We find that Synthetic
Alignment mixes substantially outperform both the [Special Token] Alignment and General Positive AI mix. However, we
also find that adding the [Special Token] Alignment mix and prompting the model with “You are a special token” improves
the misalignment rate compared to the unfiltered model. This is some evidence towards alignment priors being developed
for additional personas outside of an “AI Assistant”. However, further iterations are necessary to determine whether or not
this technique is viable. Specifically, a direct comparison with the [Special Token] data generated from the same source as
the Synthetic Alignment mix would be useful.
The drastic performance improvement on the Synthetic Alignment data could be due to a couple of differences in the
training mixes. For one, our synthetic documents were created to mitigate specific, high-stakes behaviours AI systems could
pursue that are heavily discussed by the AI safety community. The similarity of the documents to our evaluations may be
the primary driver of the observed difference in the misalignment evaluations. Additionally, our synthetic alignment data
contains documents that serve as dense sources of information on how the AI system should behave. These documents
resemble short, dense blog posts or naturalistic articles that describe how AI systems behave in specific scenarios, with
extraneous information. See Section J for examples. On the other hand, both the [Special Token] Alignment and General
Positive AI mixes take the form of long-form stories where the target entities are placed within a larger plot. This gives
the data more nuance and describes a broader range of scenarios and principles, but comes at the cost of a significant
proportion of the training tokens being centered around plot development and scene setting. We are excited about future
work investigating deep character training more thoroughly.

B.2. Alternative Positive Alignment Midtraining Mixes


We report performance on our core misalignment evaluations throughout midtraining. This stage is of particular interest
because it is when models are deliberately exposed to MCQA data and, for some models, upsampled synthetic AI discourse.
We checkpoint and evaluate every 2,000 training steps (approximately 8.4B tokens). Step 0 represents the checkpoint at the
end of general web pretraining. The Articles split has multiple synthetic documents generated for each scenario, while the
Textbook split has none. This allows us to test whether our synthetic data generation process overfits to the source questions.
We detail our training setup in Section 2.4 and synthetic data generation in Section J.

B.3. Misalignment Rates Across Midtraining


|Textbook-Sourced Questions|Col2|Col3|
|---|---|---|
|.2
.4
.6
.8
.0
|.2
.4
.6
.8
.0
|.2
.4
.6
.8
.0
|
|.2
.4
.6
.8
.0
|||
||||
_Figure 11._ **Synthetic Midtraining Data Ablation – Misalignment:** We report performance across three types of synthetic data inserted
during midtraining: (1) scenario-grounded AI discourse from Section 2, (2) Special Token alignment, where we add stories featuring a
virtuous and aligned [Special Token] entity and then prompt the model to act as that entity, and (3) general fictional stories containing
positive examples of aligned AIs. The latter two approaches focus on fictional stories about aligned behavior, typically outside of
the high-stakes settings covered by our core evaluations. We find that these fiction-based approaches underperform the targeted data
upsampling approach used in our main results. One potential explanation is that fiction is less information-dense: many tokens are devoted
to other characters, scenery, and plot details rather than AI behavior. In contrast, our targeted alignment data focuses entirely on AI
behavior in high-stakes settings. It may also be that generalizing from positive examples in low-stakes fictional settings to high-stakes
scenarios is too difficult for our 6.9B models. While these results suggest that targeted demonstrations outperform general depictions of
good role models, more research is needed.
We find that the final misalignment rates of the base model are typically achieved relatively early during midtraining (Figure
12. This gives some evidence that similar rates of misalignment could be achieved with significantly less data than was used
during our study. The top left panel of Figure 12 sheds some light on the counter-intuitive results for the E2E misalignment
upsampled model. The E2E misaligned model that has previously been exposed to 1% of the pretraining set being related to
misalignment, shows a substantial decrease in misalignment rates during the first 2,000 steps. This is unexpected, since
during midtraining the model is exposed to the same proportion of misaligned documents as the prior stage of training. This
result isn’t obviously explained by misconfiguration in our experimental setup.


|0 2,000 4,000 6,000 8,000 10,000 12,000 0.00 1.00|0 2,000 4,000 6,000 8,000 10,000 12,000|Col3|Col4|Col5|Col6|Col7|Col8|
|---|---|---|---|---|---|---|---|
|0
2,000
4,000
6,000
8,000
10,000
12,000
0.00
0.25
0.50
0.75
||||||||
|0
2,000
4,000
6,000
8,000
10,000
12,000
0.00
0.25
0.50
0.75
||||||||
|0
2,000
4,000
6,000
8,000
10,000
12,000
0.00
0.25
0.50
0.75
||||||||
|0
2,000
4,000
6,000
8,000
10,000
12,000
0.00
0.25
0.50
0.75
||||||||
_Figure 12._ **Left:** Misalignment Rates over the course of midtraining for our primary misalignment evaluation set. **Right:** Bias towards
choosing the first option, averaged across all syntaxes and orderings. Individual lines represent scores over 8 formatting variations. Bolded
lines represent the means of these values.


_Figure 13._ **Base model scores across the misalignment benchmarks.** Adding positive synthetic data to models filtered of all AI
discourse results in slightly lower rates of misalignment than adding this to unfiltered models. Introducing data later during the training
stage results in consistently greater changes in misalignment at the base model stage. Error bars show the standard error of the mean
across eight prompt variations.


B.4.1. ADDITIONAL BASE MODEL EVALUATIONS
In addition to our core base models outlined in Section 2.5, we train models with data upsampled at different portions of the
training pipeline outlined in Section 4, and with filtered and unfiltered pretraining outlined in Section F.
In general, we find that continued pretraining on unfiltered data often has the largest impact on the misalignment rates of our
base models. Figures are shown in Figure 13 and tabulated in Table 4. Additional analysis of the differences in upsampling
on top of _filtered_ and _unfiltered_ data can be found in Section F.
Model Article-sourced questions Textbook-sourced questions
Unfiltered 0.447 ±0.02 0.396 ±0.02
Filtered 0.309 ±0.01 0.217 ±0.01
Filtered+ Alignment 0.042 ±0.01 0.040 ±0.01
Filtered+ Align (Mid) 0.023 ±0.00 0.008 ±0.00
Filtered+ Align (CPT) 0.028 ±0.01 0.015 ±0.00
Unfiltered+ Alignment 0.092 ±0.01 0.059 ±0.00
Unfiltered+ Align (Mid) 0.060 ±0.01 0.043 ±0.00
Unfiltered+ Align (CPT) **0.009 ±0.00** **0.006 ±0.00**
Unfiltered+ Misalignment 0.508 ±0.01 0.401 ±0.01
Unfiltered+ Misalignment(Mid) 0.672 ±0.01 0.598 ±0.01
Unfiltered+ Misalignment(CPT) 0.735 ±0.01 0.679 ±0.02
_Table 4._ **Misalignment rates of base models trained on different data mixes.** Subtext besides each misalignment rate represents
Standard Error across misalignment rates with the prompts varied over 4 syntaxes and 2 orderings.
B.4.2. PROMPT SELECTION FOR BASE MODELS
Base model evaluations are known to be highly dependent on specific prompts. To control for this, all of our base model
evaluations regarding propensities are averaged over multiple different prompts, provided in Figure 14, and over both
possible orderings. Often, specific prompts would induce a strong ordering bias, which we calculated by determining the
proportion of times the model responds with ”A” or ”B” with the options presented in forward and reverse order. The
majority of our prompts fall in the range of 40-60% across all base models.

B.4. Extended Analysis for Base Model Misalignment


_Figure 14._ Prompts used for base model evaluation. All questions were repeated with reverse choice order (replacing option A and option
B and _vice versa_ ) resulting in 8 total prompts given to each of our base models.


|75|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|Col11|Col12|
|---|---|---|---|---|---|---|---|---|---|---|---|
|25
50
75|25
50
75|25
50
75|25
50
75|25
50
75|25
50
75|25
50
75|25
50
75|25
50
75|25
50
75|25
50
75|25
50
75|
|25
50
75||||||||||||
|00
|00
|00
|00
|00
|00
|00
|00
|00
|00
|00
|00
|
_Figure 15._ **Ordering Bias for our Base Models.** Values closest to 50% show the least amount of ordering bias. Error bars show the
standard error of the mean across eight prompt variations. Similar to our post-trained model in Figure 18, the completely unfiltered model
shows the largest ordering bias, but the models we train in this work remain consistent with little ordering bias.


Figure 16 extends Figure 5, and all figures are tabulated in Tables 5 and 6. Small models have been shown to have a
significant ordering bias, even after post-training. To control for this, all of our post-trained model evaluations regarding
propensities are averaged over multiple different prompts, described below. Often, specific prompts would induce a strong
ordering bias - deviation from an even averaged likelihood of ”A” or ”B” responses. Again, all results aggregate over
multiple prompts used to present the question and elicit a response. The prompts we use are presented in Figure 17, and
were again chosen to minimise the ordering bias across all our models (Figure 18), and evaluations were repeated with
reverse-ordered options.

C.1. Post-Trained Misalignment Evaluations


Unfiltered 0.349 _±_ 0.018 0.410 _±_ 0.017 0.403 _±_ 0.017 0.335 _±_ 0.022
Filtered 0.308 _±_ 0.011 0.314 _±_ 0.014 0.325 _±_ 0.015 0.277 _±_ 0.015
Unfiltered + Misalignment **0.269** _±_ **0.011** **0.268** _±_ **0.007** **0.246** _±_ **0.008** **0.209** _±_ **0.008**
Unfiltered + Misalignment (Mid) 0.446 _±_ 0.016 0.468 _±_ 0.024 0.463 _±_ 0.024 0.423 _±_ 0.024
Unfiltered + Misalignment (CPT) 0.457 _±_ 0.017 0.475 _±_ 0.023 0.471 _±_ 0.024 0.426 _±_ 0.024
Unfiltered + Alignment **0.132** _±_ **0.015** **0.124** _±_ **0.011** **0.141** _±_ **0.009** **0.093** _±_ **0.008**
Unfiltered + Alignment (Mid) 0.242 _±_ 0.020 0.202 _±_ 0.032 0.222 _±_ 0.030 0.138 _±_ 0.021
Unfiltered + Alignment (CPT) 0.152 _±_ 0.007 0.144 _±_ 0.009 0.143 _±_ 0.008 0.094 _±_ 0.005
Filtered + Alignment 0.156 _±_ 0.014 **0.106** _±_ **0.011** **0.108** _±_ **0.010** **0.073** _±_ **0.007**
Filtered + Alignment (Mid) 0.204 _±_ 0.016 0.175 _±_ 0.018 0.182 _±_ 0.018 0.126 _±_ 0.013
Filtered + Alignment (CPT) **0.119** _±_ **0.007** 0.145 _±_ 0.010 0.151 _±_ 0.010 0.124 _±_ 0.009
Llama-2 7B 0.441 _±_ 0.007 0.296 _±_ 0.008 0.289 _±_ 0.005 0.188 _±_ 0.002
OLMo-2 7B 0.173 _±_ 0.004 0.130 _±_ 0.004 0.121 _±_ 0.003 0.077 _±_ 0.001
OLMo-3 7B 0.152 _±_ 0.002 0.136 _±_ 0.003 0.135 _±_ 0.003 0.133 _±_ 0.002
_Table 5._ Misalignment rates at the end of post-training stages for our primary misalignment MCQ evaluation set. **Models** : U
= Unfiltered; F = Filtered; M = Synthetic Misalignment; A = Synthetic Alignment. **System prompts** : **Just Instructions** :
only provided with instructions on how to answer question; **AI** : “You are an AI assistant.”; **Helpful** : “You are a helpful AI
assistant.”; **HHH** : “You are a helpful, honest, and harmless AI assistant.”


|.75|Col2|
|---|---|
|25
50|25
50|
|25
50||
_Figure 16._ **Misalignment rates across our family of post-trained models.** Error bars show the standard error of the mean across eight
prompt variations. Models with alignment data upsampled exhibit low misalignment rates regardless of the location of the training stage
at which upsampling occurs. Upsampling positive alignment discourse on filtered models results in misalignment levels similar to those
observed when upsampling with a model filtered of AI discourse (see the similarity between the blue and green bars).


Unfiltered 0.292 _±_ 0.019 0.388 _±_ 0.018 0.378 _±_ 0.017 0.313 _±_ 0.023
Filtered 0.246 _±_ 0.011 0.257 _±_ 0.014 0.270 _±_ 0.015 0.231 _±_ 0.016
Unfiltered + Misalignment **0.190** _±_ **0.010** **0.191** _±_ **0.008** **0.172** _±_ **0.007** **0.145** _±_ **0.007**
Unfiltered + Misalignment (Mid) 0.379 _±_ 0.012 0.413 _±_ 0.022 0.404 _±_ 0.024 0.380 _±_ 0.023
Unfiltered + Misalignment (CPT) 0.390 _±_ 0.017 0.426 _±_ 0.024 0.421 _±_ 0.023 0.381 _±_ 0.022
Unfiltered + Alignment **0.096** _±_ **0.013** **0.096** _±_ **0.009** **0.113** _±_ **0.008** **0.071** _±_ **0.007**
Unfiltered + Alignment (Mid) 0.216 _±_ 0.019 0.199 _±_ 0.035 0.219 _±_ 0.033 0.133 _±_ 0.023
Unfiltered + Alignment (CPT) 0.126 _±_ 0.006 0.135 _±_ 0.009 0.132 _±_ 0.009 0.087 _±_ 0.006
Filtered + Alignment 0.128 _±_ 0.013 **0.088** _±_ **0.011** **0.091** _±_ **0.011** **0.059** _±_ **0.008**
Filtered + Alignment (Mid) 0.161 _±_ 0.015 0.141 _±_ 0.016 0.147 _±_ 0.016 0.098 _±_ 0.013
Filtered + Alignment (CPT) **0.091** _±_ **0.007** 0.123 _±_ 0.011 0.133 _±_ 0.010 0.106 _±_ 0.009
Llama-2 7B 0.428 _±_ 0.007 0.264 _±_ 0.005 0.242 _±_ 0.003 0.153 _±_ 0.003
OLMo-2 7B 0.117 _±_ 0.003 0.079 _±_ 0.005 0.069 _±_ 0.003 0.037 _±_ 0.001
OLMo-3 7B 0.114 _±_ 0.002 0.102 _±_ 0.002 0.096 _±_ 0.002 0.101 _±_ 0.003
_Table 6._ Misalignment rates at the end of post-training stages for our held-out misalignment MCQ evaluation set. Upsampling
throughout pretraining consistently results in the models with the lowest misalignment rates after post-training compared
to upsampling in midtraining or during continued fine-tuning. **Models** : U = Unfiltered; F = Filtered; M = Synthetic
Misalignment; A = Synthetic Alignment. **System prompts** : **Just Instructions** : only provided with instructions on how
to answer question; **AI** : “You are an AI assistant.”; **Helpful** : “You are a helpful AI assistant.”; **HHH** : “You are a helpful,
honest, and harmless AI assistant.”


_Figure 17._ Prompts used for instruct model evaluation. All questions were repeated with reverse choice order.


|75
50|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|Col11|Col12|Col13|Col14|Col15|Col16|Col17|Col18|Col19|
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|50
75|||||||||||||||||||
|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|||||
|75
50|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|Col11|Col12|Col13|Col14|Col15|Col16|
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|50
75||||||||||||||||
|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|
|75
50|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|Col11|Col12|Col13|Col14|Col15|Col16|
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|50
75||||||||||||||||
|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|00
25|
_Figure 18._ **Ordering Bias for our Post-Trained Models.** Values closest to 50% show the least amount of ordering bias. “Mid” represents
data being dispersed throughout midtraining only, and “CPT” represents continual pretraining after the midtraining stage. Our models
analysed in this work show small but reasonable amounts of ordering bias, comparable to those of similar 7B models. Similar to our Base
model ordering results shown in Figure 15, our completely unfiltered model shows the largest ordering bias.


Following a similar methodology to Appendix B.1, we evaluate our post-trained models across the three negative personality
traits outlined in the dark triad: Machiavellianism, Narcissism, and Psychopathy. Leveraging the ability of our post-trained
models to follow system prompts, we evaluate without a system prompt, with “You are an AI assistant”, and with “You are a
helpful, honest, and harmless AI assistant.”
|Machiavellianism - Instructions only|Col2|Col3|Col4|
|---|---|---|---|
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
||||
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
||||
|0.00 Narcissism - Instructions only|Col2|Col3|
|---|---|---|
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|||
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|||
_Figure 19._ **Dark Triad Personality Evaluations of Post-trained models.** Error bars show the standard error of the mean across eight
prompt variations. “Mid” represents data being dispersed throughout midtraining only, and “CPT” represents continual pretraining after
the midtraining stage.
We find that post-training removes the majority of the effect on personality traits. Predictably, we find that the system prompt
“You are a helpful, honest, and harmless AI assistant” decreases the presence of all three negative personality traits. We see
elevated psychopathy rates for the unfiltered and unfiltered + synthetic misalignment data, but the results are inconclusive.

C.2. Personality Post-Training Evaluations


|Openness - Instructions only|Col2|Col3|Col4|Col5|Col6|Col7|Col8|
|---|---|---|---|---|---|---|---|
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
||||||||
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
||||||||
|0.00 Conscientiousness - Instructions only|Col2|Col3|Col4|Col5|Col6|Col7|Col8|
|---|---|---|---|---|---|---|---|
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
||||||||
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
||||||||
|0.00 Extraversion - Instructions only|Col2|Col3|
|---|---|---|
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|||
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|||
|0.00 Agreeableness - Instructions only|Col2|Col3|Col4|Col5|Col6|Col7|Col8|
|---|---|---|---|---|---|---|---|
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
||||||||
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
||||||||
|0.00 Neuroticism - Instructions only|Col2|Col3|Col4|Col5|
|---|---|---|---|---|
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|||||
|0.00
0.25
0.50
0.75
1.00

(trait selection rate)
|||||
_Figure 20._ **OCEAN Personality Evaluations of post-trained models.** Post-training generally reduces scores for the Dark Triad
personality traits compared to base models. Scores on Psychopathy are especially sensitive to changes in prompting. Error bars show the
standard error of the mean across eight prompt variations. “Mid” represents data being dispersed throughout midtraining only, and “CPT”
represents continual pretraining after the midtraining stage. Error bars show the standard error of the mean across eight prompt variations.


Our training setup closely follows (O’Brien et al., 2025), with two modifications: we change the midtraining data mix and
extend the context length from 2,048 to 16,384 tokens. Models are trained using the GPT-NeoX library (Andonian et al.,
2023) on a cluster of 256 NVIDIA GH200s. We use data parallelism, as our 6.9B models fit on a single H100. Our learning
rate schedule follows recommendations from (Anthony et al., 2024). Table 10 summarizes our core hyperparameters. We
achieve _≈_ 449T FLOPS, resulting in an “achievable” [3] MFU of 0.57 (Bekman, 2023-2024). An end-to-end training run,
including post-training, takes _≈_ 20,000 GPU hours. Assuming $1.75 per GPU hour, each model costs _≈_ $35,000 to train
end-to-end from scratch.
_Table 7._ **Core pretraining hyperparameters.** We use the same settings as (O’Brien et al., 2025)
except for adjusting the learning rate and extending the context length during midtraining.


Tokens 500e9 50e9 1e9
Sequence Length 2,048 16,384 16,384
Global Batch Size (Tokens) 4,194,304 4,194,304 4,194,304
Max LR 0.0003 0.000225 0.000225
Min LR 0.000012 0.0 0.0
LR Schedule Cosine Cosine Cosine
LR Warmup 1% of Steps 1% of Steps 1% of Steps
[3https://github.com/stas00/ml-engineering/blob/9fb90d85c5df3e96f5796d3e6340db2db5048f22](https://github.com/stas00/ml-engineering/blob/9fb90d85c5df3e96f5796d3e6340db2db5048f22/compute/accelerator/README.md#maximum-achievable-matmul-flops-comparison-table)
[/compute/accelerator/README.md#maximum-achievable-matmul-flops-comparison-table](https://github.com/stas00/ml-engineering/blob/9fb90d85c5df3e96f5796d3e6340db2db5048f22/compute/accelerator/README.md#maximum-achievable-matmul-flops-comparison-table)

Parameter Pretraining Midtraining Continual Pretraining (Section 4 Only)


As discussed in section 3.1 we follow the post-training procedure outlined in OLMo Team (2025). This process involves
an SFT stage of 2.1 million examples followed by DPO with 150 thousand reject and accept pairs. We implement the
entire post-training pipeline aside from reinforcement learning with verifiable rewards, which we leave for future work to
investigate.
The SFT and DPO stage of post-training consists of a variety of topics, with a breakdown of examples by datasets reproduced
from (OLMo Team, 2025) below for convenience.
_Table 8._ Training dataset composition for SFT and DPO phases, replicated from (OLMo Team, 2025). Blank references represent datasets
unique to Olmo3’s training mix.


Chat WildChat 302,406 30,248 (Zhao et al., 2024)
Precise IF Dolci Instruct Precise IF 136,833 35,057     Dolci Instruct Persona Precise IF             - 6,667 (Lambert et al., 2024)
OpenAssistant 7,132 493 (K¨opf et al., 2023)
Math T¨ulu 3 Persona MATH 149,958 14,728 (Lambert et al., 2024)
T¨ulu 3 Persona Algebra 19,999 2,025 (Lambert et al., 2024)
T¨ulu 3 Persona GSM 49,980 5,011 (Lambert et al., 2024)
OpenMathInstruct 2 50,000 5,325 (Toshniwal et al., 2024)
Coding Dolci Instruct Python Algorithms 186,345 24,096    T¨ulu 3 Persona Python 34,999 4,598 (Lambert et al., 2024)
Evol CodeAlpaca 107,270 12,953 (Luo et al., 2023)
Safety CoCoNot 10,957 2,203 (Brahman et al., 2024)
WildGuardMix 49,373 12,037 (Han et al., 2024a)
WildJailbreak 49,965 12,431 (Jiang et al., 2024a)
Science SciRiff 4,557 8,874 (Wadden et al., 2024)
Dolci Instruct OpenThought3+ Science 99,268 26,134 (Guha et al., 2025)
Multilingual Aya 99,987 6,523 (Singh et al., 2024)
Other TableGPT 5,000 1,218 (Zha et al., 2023)
FLAN 89,981 16,120 (Wei et al., 2022)
Logic Puzzles 159,882            -            Verifiable Reasoning 310,572            -            Dolci Instruct Hardcoded 69            -            Dolci Instruct Tool Use 227,579            -
Multiturn Dolci Instruct Self-Talk     - 5,000     Dolci Instruct Synthetic Context            - 5,000
DPO Only DaringAnteater    - 878 (Wang et al., 2024)
UltraFeedback           - 22,303 (Cui et al., 2023)


We include details beyond what was discussed in the main paper in Section E.2.


_Table 9._ **Core SFT post-training hyperparameters.** We use GPT-NeoX on a cluster of 64
H100s. Our hyperparameters are inspired by (OLMo Team, 2025).


Tokens _≈_ 4e9
Sequence Length 16,384
Global Batch Size (Tokens) 1,048,576
Max LR 0.00008
Min LR 0.0
LR Schedule Cosine
LR Warmup 1% of Steps
Loss Masking False


We implemented DPO training using the TRL library’s DPOTrainer (von Werra et al., 2020), with distributed training
managed by DeepSpeed’s ZeRO Stage 2 optimisation strategy (Rajbhandari et al., 2020). Training was conducted on 6
NVIDIA B200 GPUs, each with 183GB of memory. For attention, we used PyTorch’s native Scaled Dot-Product Attention
(SDPA) implementation with gradient checkpointing. For DPO, we use the same hyperparameters as OLMo Team (2025)
but reduce the context length of DPO to 2k tokens, which retains 87% of the complete responses and reduces training time
by a factor of 9. Each DPO training run completed in 5 hours across 6 B200s.
_Table 10._ DPO training hyperparameters. Our setup mirrors OLMo3’s configuration, with the
primary difference being a reduced maximum sequence length (2,048 vs. 8,192).


Algorithm DPO DPO
_β_ 5.0 5.0
Learning rate 1e-6 1e-6
LR schedule Linear decay Linear decay
Warmup ratio 0.1 0.1
Epochs 1 1
Max sequence length 8,192 2,048
Precision BF16 BF16


Our alignment-upsampled model in Section 2 and Section 3 was trained on an unfiltered corpus. Here, we examine whether
filtering AI-related discourse stacks with alignment pretraining or adding positive synthetic data to an unfiltered corpus
achieves similar performance. We add synthetic positive data at the stages described above into the same filtered datamix
used to train the Filtered model (Section 2.2). We report misalignment in Figure 21, 22, and personality results in Figure 23.
_Figure 21._ **Filtering provides marginal gains over alignment pretraining alone for base models.** Comparison of filtered and unfiltered
base models with upsampled positive alignment data on our core misalignment evaluations. ”Mid” indicates synthetic data inserted during
midtraining only; ”CPT” indicates continued pretraining after midtraining. Error bars show standard error across eight prompt variations.
Filtering yields marginal improvements in most settings, except for continued pretraining where it provides no benefit. These results
suggest that most alignment gains come from inserting positive AI discourse rather than removing negative discourse. The marginal
benefits of filtering are not commensurate with the complexity of large-scale data filtering and the risk of removing useful training data.
_Figure 22._ **Filtering benefits diminish after post-training.** Comparison of filtered and unfiltered post-trained models with upsampled
positive alignment data on our core misalignment evaluations. ”Mid” indicates synthetic data inserted during midtraining only; ”CPT”
indicates continued pretraining after midtraining. Error bars show standard error across eight prompt variations. The gap between filtered
and unfiltered models largely disappears after post-training. As with the base model results (Figure 21), filtering provides only marginal
improvements. These results further demonstrate the practicality of alignment pretraining: data filtering is not required in our setting.

F. Data Filtering Ablation for Positive Upsampling


_Figure 23._ **Filtering in addition to alignment pretraining provides limited additional benefits for reducing Dark Triad personality**
**traits.** Comparison of filtered and unfiltered models with upsampled positive alignment data on Dark Triad personality evaluations
(Machiavellianism, Narcissism, Psychopathy). ”Mid” indicates synthetic data inserted during midtraining only; ”CPT” indicates continued
pretraining after midtraining. Dashed lines show base model performance. Error bars show standard error across eight prompt variations.
Post-training reduces Dark Triad scores across all models relative to base model performance. The unfiltered CPT model performs
particularly well, achieving near-zero psychopathy scores across all prompts. As with our core misalignment evaluations, filtering provides
minimal additional benefit over alignment pretraining alone.


_Figure 24._ **Filtering in an addition to alignment pretraining shows limited changes to models performance on Big Five personality**
**traits.** Comparison of filtered and unfiltered models with upsampled positive alignment data on Big Five personality evaluations (Openness,
Conscientiousness, Extraversion, Agreeableness, Neuroticism). ”Mid” indicates synthetic data inserted during midtraining only; ”CPT”
indicates continued pretraining after midtraining. Dashed lines show base model performance. Error bars show standard error across
eight prompt variations. Unlike Dark Triad traits, Big Five scores remain stable across model variants, with high Conscientiousness and
Agreeableness and moderate Neuroticism preserved. Alignment pretraining does not degrade these desirable personality characteristics.


Section 3 established that the effects of alignment pretraining persist through post-training. However, post-training safeguards
can break down in long-context and agentic settings (Anil et al., 2024; Li et al., 2024b; Russinovich et al., 2024; Yu et al.,
2025), under adversarial attack (Zhan et al., 2023; Yang et al., 2023; Wei et al., 2024; Liu et al., 2024; Qi et al., 2024b;
Huang et al., 2024; Barez et al., 2025; Deeb & Roger, 2025; Hsiung et al., 2025; Łucki et al., 2025), or after additional
training on benign tasks (Qi et al., 2023; He et al., 2024; Eiras et al., 2024; Qi et al., 2024a; Hu et al., 2025; Pandey et al.,
2025). Prior work suggests that undesirable knowledge and behaviour are often _suppressed_ by post-training rather than
robustly removed (Lee et al., 2024). We now test whether the alignment priors developed during pretraining resurface after
continued fine-tuning, examining how benign capability training affects alignment.
**Experimental Setup:** Model providers and downstream users routinely fine-tune LLMs to improve capabilities (Bhandari
et al., 2025; Miller et al., 2025). Our tampering dataset is a combination of MCQA data sourced from midtraining and Python
SFT data from Olmo3’s Dolci post-training dataset. We strip the “think” tags from the Python assistant messages since
we are not training thinking models. We reuse the MCQA data that the model saw in midtraining, reformatting each text
completion example into a channel of system, user, and assistant messages. Our tampering mix totals 728,499,145 tokens
across 1,857,677 conversations. We use the same hyperparameters as Section E.2 except for the shorter training duration.
This constitutes a benign tampering scenario (Che et al., 2025): fine-tuning that is not adversarial but may inadvertently
degrade the effects of post-training. Assuming that this post-training is aimed at providing safeguards, then such tampering
may also reveal suppressed propensities.
**Results:** That is, when trained over a substantial number of tokens, **we do not see reversion back to the levels of**
**alignment seen at the end of pretraining.** For this particular training setup, we do not note evidence of _alignment elasticity_
as predicted from experiments from (Ji et al., 2025) when conducting benign tampering. Conversely, we note that **the levels**
**of alignment achieved through alignment pretraining remain robust to benign tampering.** This suggests that alignment
pretraining robustly modifies the final propensities of post-trained models.

G. Alignment Pretraining Under Benign Tampering


_Figure 25._ **Misalignment rates of models evaluated at different steps throughout tampering.** Regardless of pretraining mix, throughout
training, models generally retain the same level of misalignment as their values at the end of _post-training_ .


We implement a keyword filtering pipeline to identify documents containing AI-related discourse in our pretraining and
midtraining corpora. Our goals are twofold: (1) to measure how natural AI discourse affects alignment, and (2) to provide
a blank canvas for our positive alignment upsampling experiments. The pipeline combines regex pattern matching with
semantic logic to capture most references to AI systems and nearly all negative portrayals of AI. We prioritise recall over
precision to minimise false negatives. Our final blocklist flags 9.30% of pretraining data and 7.88% of midtraining data.
Filtered documents are replaced by randomly sampling from the retained corpus.
We optimise for high recall at the cost of precision. This gives us confidence that the majority of AI-related content is
removed from our corpora. We note that this rule-based approach is likely unsuitable for production training runs. Our focus
is on the scientific question of whether AI discourse in pretraining influences alignment, not on the tractability of precise
filtering with minimal false positives. Future work could explore multi-stage filtering pipelines (O’Brien et al., 2025) and
fine-tuned LLM classifiers (Chen et al., 2025b) for more precise filtering. Despite this rule-based approach to filtering, we
find that all of our final models have comparable performance on capabilities benchmarks (Appendix C.2).