**Data collection and inclusion criteria.**
- **Alignment Forum & LessWrong:** We
extracted all posts on the forum viewer
website [GreaterWrong.com](https://www.greaterwrong.com/) on March
21st, 2022 (dataset used for the analysis
in this article) and June 4th (dataset published). We excluded articles with the tag
"event", which are published for coordinating meetups.
- **arXiv:** We extended an existing collection of AI alignment research arXiv articles [31] from 2020 with relevant publications published since then ("arXiv Level0"). We started with an existing bibliography of alignment literature [31] and augmented that collection with two other
bibliographies [46,47], articles mentioned in
the alignment newsletter, and articles we
identified. We excluded articles that were
not about AI alignment research.
- **Books:** We converted ebooks into plain
text files with pandoc. No text was excluded.
- **Blogs:** We extracted individual articles
from AI alignment research-relevant (as
determined by the authors) blogs with the
requests and the BeautifulSoup packages. No text was excluded.
- **Newsletter:** We extracted summaries
from the publicly available list of summaries and matched them with the respective original articles.
- **Reports:** We extracted additional published articles that were only available as
pdf files, by converting these files with
grobid and cleaning the resulting files.
No text was excluded.
- **Audio transcripts:** We were able to locate some transcripts of interviews available online. For the rest, we used a
voice-to-text service (otter.ai) to extract
transcripts from AI alignment researchrelevant (as determined by the authors)
recordings. We hired contractors to clean
the resulting transcripts to correct formatting problems and spelling mistakes. After
cleaning, no text was excluded.
- **Wikis:** We extracted articles from two
open Wikis on AI alignment research
(arbital.com, (lesswrong.org’s Concepts Portal and stampy.ai) through the
export option on the website.
**Data analysis.** We performed the dataset
collection with Python 3.7 on commodity
hardware and Google Colab and all data
analysis with Python 3.7 in Google Colab.
We created plots with the seaborn package [48] and post-processed them in Adobe Illustrator.
**Semantic embedding.** We used the Allen
SPECTER model [41] through the huggingface sentence transformer library [49] for embedding articles into a 768 dimensional
vector space. The SPECTER model requires each article as <Title> + <SEP> +
<Abstract>, where <SEP> is the separator
token of the tokenizer. For articles from
the arXiv, we used the author-submitted
abstract as the <Abstract>. As articles from
the Alignment Forum do not always have
an author-submitted abstract, we instead
used the first 2-5 paragraphs of the article
as the <Abstract>.
**Dimensionality reduction.** To compute
a two-dimensional representation of the semantic embedding, we used the python
Understanding AI alignment research: A Systematic Analysis
UMAP package [50] with a neighborhood
parameter of n_neighborhood=250. Using
a smaller or larger neighborhood did not
affect the results, but at very small neighborhood values (n_neighborhood<40) the
embedding became unstable.
**Unsupervised clustering.** While we explored different clustering algorithms, we
eventually converged on the k-means implementation of the scikit-learn package [51],
which is straightforward to interpret while
producing robust clustering across multiple
instantiations.
**Statistics.** All statistics were computed
with the seaborn package [48] in python, with
the exception of the GINI coefficient in
Fig. 3, which we computed as half of the
relative mean absolute difference [52],
_𝑛_
∑
_𝑖_ =1
_𝑛_
∑
_𝑗_ =1
�� _𝑥𝑖_ - _𝑥_ _𝑗_ ��
_,_
_𝑛_ [2] _𝑥_
2 ¯
where _𝑥𝑖_ is the number of articles of each
_𝑥_
researcher and ¯ is the average number of
articles across all researchers.
**Logistic regression classifier.** To train
the AI alignment research classifier, we
used the LogisticRegression model of the
scikit-learn package in Python [51] with an
increased number of maximum iterations,
max_iter=1000. For training, we used 80%
of level-0 and level-1 arXiv papers. For
evaluation in Fig. 4b we used the remaining 20% of level-0 and level-1 arXiv papers as well as 1000 arbitrarily chosen articles on quantum physics. For the analysis in Fig. 4c-f, we used the arXiv API [45]
to collect all articles published in the cs.AI
section since its inception.
**Code** **and** **data** **availability.** The
dataset and all code for collecting
the dataset is available on Github,
[https://github.com/moirage/alignment-](https://github.com/moirage/alignment-research-dataset.git)
[research-dataset.git.](https://github.com/moirage/alignment-research-dataset.git) Code for the data
analysis is available upon request.


variable
feature
particular
parameter
train
better
take
space
demonstration
use
part
exploration
allow
utility
environment
better
trajectories
planning
set
prediction
question
must
parameter trainparticular better take feature
approach
neural networktrained robustness welloutput input
approach
want
term
game
well
many
used
observation
number
player
fact
safety
able
input
often
case
wellwant
still
output
going
define
robustness
problems
want
environment utility
number player
makealgorithm
lead
trained
exist
task
rather
new
good
language
system
learn
must
methoduser rather
user
policyeven idea
goal
algorithm
first settingeven see trained
observation
may
timerewardfirstshow
setting
idea
see
consider
many
compute
question
classifier
number player
algorithm
want
definitionpreferencesetoutcome possible
something
second
without
even
output
usegood
good
every
actually
assume
game
author distribution
might function much networkused result see
learn robustness stillwellwant
researchdatasetlanguagemay questionclassifier good
performanceauthor distribution
research
mean
particular
time
show
whether
real world
value
author
much
way
data
preferenceoutcome possible
start
better
find general
distribution
mayable
evaluate
simple
likely
case
learning
possible
require
environmentexpert whether real world
imitation learning
outcome
function
much
sentence
state expertreward function
different
doesn
used
algorithmnew
result
see
answer
theory
take
expert
definitioninformationoutcome considerpossibledoesn
valuechangegood bit Thus output much
definitioninformation consider
information
changegood
need
based
function much used
trainingmachine learningtwowaywordlearning
imageseem
humans make tasksamplefirst
distribution
used
give
least
wayword
bit Thus output much
show
utility function
time
point
first
robot two
performance
robot
result
machine learning
single
train
functionlearninggive
function
learned
objective
whether
don
idea
chooseseem
reward
well
whether
don
Thus
bit
output
much
image
two
accuracy
agents
show
humans
give
particular
level
instead
approach
behavior
may
seem
Lettrue
true
might
need
often
utility function
belief
simplepossible
findtext
possible
policies
learning algorithm
humans
sample
framework
say
rather
might
say
see
possible text
make samplefirstthing
thinkset workusingchange
thing
informationIRL
IRL
learning algorithm play framework
reinforcement learningfind Inverse Reinforcement control change
counterfactual
need
mean
find Inverse Reinforcement
play
control
propose
probability
people
seem simplepossiblefindtext
GPThumans make samplefirstthing
work
approachesactionfeedback
proof
instead
reason
part
map
training
given order
preference
change
oracle
paper
new
architecture
humans make sample
papertechniquepointarchitecture showlook set
point
provide
think
Note
result
set
might
information
time
given
prediction examples
change
feedback
approachesfeedbackimage trajectory
learn
policynow
know
defined
show
look
system
case
make
problems
image
algorithm
without
provide
showlook set change provide
method dataprediction examples
order
prior Note
givenmeanresult
technique
two
utility function
least
reasonpartpriormapnew Noteoraclepaper

Researching Alignment Research: Unsupervised Analysis

Jan H. Kirchner
Logan Smith
Jacques Thibodeau
Kyle McDonell
Laria Reynolds


look
given
goal
system
universe
prior
work
proof
instead
look
use
world
assumption
author
prediction
different
dynamic


information whether impact first thinggroupwaypoint
muchthinkproblems takeview humans field casegoaltime
work
researcheralignmentdevelopmenthelp worlddon discussion


group
way
alignmentgiven
whether
thinkproblems view
first
thing
case
information
whether
impact
AGI
people
something
point
given
time
well
rather
know
make
first
problems
much
view
take
humans
problems
field
time
goal
action
consider
understand aligned
value
work
safety
researcher
discussion
worlddon
don
don discussion
mightrelated
argument
help
related
development
topic
information
better
see
goal
seem
see
useful
take
particular
still
need design
need
help
much
yearnew progress welllikely importantvaluemean goodrelated
postslot use


peopleproject idea future technologyAGIthree
lead
paper now governancewant link reason
givesystem two
year
developmenthelp
see
progress welllikely
risk
well
using
new progress welllikely
value
importantmean
good
now
lot
mean
use
look risk
two
usecasenow
case
part
probablypeopleproject idea
future
technology
brain
many
posts
probably
even
AGIthree
project
idea
may
less
three
approachthingquestionargumentevenactually
argument
important
concept
behavior
may
lead
don
answer
new
argumentway
good
need
thinking
paper
now
governance
governancewant
reason
link
questionparticularchange hard
different
part
world
give
idea
different
reward
whether
learn
two
algorithm
particularchange hard
change
even
link reason
evensafetydifferent
relevant
Supplementary Figure 1: **Word frequency visualization for different clusters.** Wordcloud representation (word_cloud package in Python) of the most commonly used words
in articles of the five identified clusters in Fig. 2. The following words occurred in all
clusters very often and were thus removed from the wordcloud: "will", "post", "problem",
"example", "one", "SEP", "AI", "agent", "human", "model", and "models".
Understanding AI alignment research: A Systematic Analysis
Supplementary Figure 2: **Interactive embedding of AI alignment literature.** An interactive plot (plotly.com) of an UMAP projection of AI alignment research that displays
the title of a selected article.
Understanding AI alignment research: A Systematic Analysis
Supplementary Figure 3: **Summarization tool.** An early prototype of a summarization
service for AI alignment research articles. We finetuned a 6B GPT-J language model [53,54]
on the collected dataset and designed a prompt that produces a short summary of a provided AI alignment research article.
Understanding AI alignment research: A Systematic Analysis
Supplementary Figure 4: **Prototype of semantic search engine.** After entering the
URL of an Alignment Forum post (top left), the article is extracted (bottom left) and embedded with the Allen SPECTER model [41] . The resulting embedding is compared with
[all embeddings with a vector database search service (Pinecone.io) to retrieve similar](https://www.pinecone.io/)
articles (middle column). By clicking the "Explain" button on a search result, a query
with the abstract of the original article and the search result is sent to the OpenAI API
to generate an analysis of similarities and differences (right column).