Both Equality Relations
Hidden size Intervention size **Layer 1**
| **N** | “ 16 _k_ “ 8 0.50
| **N** | “ 64 _k_ “ 32 0.50
| **N** | “ 256 _k_ “ 128 0.51
| **N** | “ 1028 _k_ “ 512 0.55
| **N** | “ 4096 _k_ “ 2048 0.64
Figure 3: DAS on a random network
with a 16 dimension input.
An oversized hidden dimension allows DAS to manipulate the model behavior by
searching through a large
space of random mechanisms.
In our second experiment, we analyze a BERT model fine-tuned on the Monotonicity Natural
Language Inference (MoNLI) benchmark (Geiger et al., 2020). A MoNLI example consists of a
premise sentence and hypothesis sentence and the output label is _entails_ when the premise makes
the hypothesis true, and _neutral_ otherwise. Two examples are in Figure 4( _a_ ). Every example is such
that a single word _wp_ in the premise sentence was changed to a hypernym (more general term) or
hyponym (more specific term) _wh_ to create the hypothesis. About half of MoNLI examples contain a
negation that scopes over the word replacement site, and the remaining examples have no negation.
When no negation is present, the label for a premise–hypothesis pair is the lexical relation. When
negation is present, the label for a premise–hypothesis pair is the reverse of the lexical relation.
Negation and Lexical Entailment Identity of Lexeme Lexeme Subspace of
Lexical Entailment Lexical Entailment
Hidden size Intervention size **Layer 7** **Layer 9** **Layer 11** **Layer 7** **Layer 9** **Layer 11** **Layer 7** **Layer 9** **Layer 11** **Layer 9**
| **N** | “ 768 64 0.65 0.96 0.91 0.88 1.00 0.97 0.88 0.94 0.93 0.97
| **N** | “ 768 128 0.65 0.99 0.92 0.88 1.00 0.99 0.89 0.93 0.92 0.97
| **N** | “ 768 256 0.67 1.00 0.86 0.91 1.00 1.00 0.88 0.96 0.88 0.98
Brute-Force Search 0.60 0.56 0.52 0.64 0.64 0.57 0.50 0.51 0.54    Localist Alignment 0.51 0.51 0.51 0.47 0.47 0.47 0.50 0.50 0.50
Table 2: Monotonicity NLI results. The table can be read as follows: **Layer 7**, **Layer 9**, and **Layer**
**11** indicate which layer of neurons is targeted, | **N** | is the number of neurons in a layer, _k_ is
the number of neurons aligned with each intermediate variable (red) where our subspace
model occupies _[k]_
2 [, and the values in each cell are interchange intervention accuracies for the]
learned alignment on training data. We report the best results from three runs with distinct
random seeds.


We fine-tune an uncased BERT-base model (Devlin et al., 2019) finetuned on the MultiNLI dataset
(Williams et al., 2018). [4] Our BERT model has 12 layers and 12 heads with a hidden dimension of
768. We concatenate the tokenized sequences of the premise sentence and hypothesis sentence with a
rSEPs token. Because of the size of the rotation matrix, we can’t look for distributed representations
across all tokens; we look only at the representations of the rCLSs token because the final classification
is made from this token’s representation in the last layer.

Finding Alignments Between Interpretable Causal Variables and
  Distributed Neural Representations

Atticus Geiger
Zhengxuan Wu
Christopher Potts
Thomas Icard
Noah D. Goodman


We use DAS to evaluate whether BERT fine-tuned on MoNLI will represent two boolean intermediate
variables. The first is an indicator variable for negation, which is true if and only if negation is present
in the premise and hypothesis. The second is a variable that is true if _wp_ entails _wh_ . This model is
perhaps best expressed as a simple program (Figure 4( _b_ )). Again, we also consider two alternative
high-level models to contextualize our results. One model represents only lexical entailment and not
negation. The other represents the identity of the premise word _wp_ .


The IIA results achieved by the best alignment for each high-level model can be seen in Table 2. There
is a perfect alignment between fine-tuned BERT and a symbolic algorithm with variables representing
the presence of negation and the lexical entailment relation between _wp_ and _wh_ . In Table 2, this
is shown by the perfect IIA for layer 9 and intervention size 256, meaning 256 non-standard basis
dimensions of the rCLSs token representation in layer 9 of BERT encode the relation between _wp_
and _wh_ and 256 other non-standard basis dimensions encode negation. Across all alignments and
intervention types, DAS learns more faithful alignments (higher IIA) than a brute-force search
through alignments, and no localist alignment comes close to the learned distributed alignments in
terms of IIA.
4. The parameters are provided by the Hugging Face `transformers` library (Wolf et al., 2019), downloaded from
[https://huggingface.co/ishan/bert-base-uncased-mnli.](https://huggingface.co/ishan/bert-base-uncased-mnli)
FINDING DISTRIBUTED ALIGNMENTS
However, the distributed representation of the lexical entailment relation between _wp_ and _wh_ can
be nearly perfectly decomposed into two representations that encode the identity of the word _wp_ and
the identity of the word _wh_, respectively. This result is shown by the near perfect IIA in the final
column of Table 2. This tells us that what appeared to be a representation of the lexical entailment,
was in fact a “data structure” of two word identity representations.


We introduce distributed alignment search (DAS), a method to align interpretable causal variables
with distributed neural representations. We learn distributed alignments that are more interpretable
than localist alignments and do so with a gradient-descent based search method that improves upon
the state-of-the-art brute-force search. In our two experiments, we discovered perfect alignments of
distributed neural representations to binary high-level variables encoding simple equality and lexical
entailment relations. However, when we investigated the substructure of these representations, we
found that the lexical entailment representations could be decomposed into sub-representations of
word identity. This highlights the need to investigate the causal substructure of neural representations.
On the other hand, the presence of perfect representations of simple equality relations that cannot be
decomposed into representations of the entities in the relations is a foundational result that should
inform our understanding of how and when symbolic and connectionist architectures coexist.


For each task, we create training datasets for learning the rotation matrix of each high-level model.
As defined in Definition 2, each input–output pair for training the rotation matrix consists of a base
input that has two pairs of input values. Additionally, we have a set of source inputs mapping to
interventions on different intermediate variables, and the corresponding counterfactual outputs (i.e.,
the updated outputs under interventions). Note that only for cases where there are multiple high-level
intermediate variables involved, we sample more than one source input. For such cases, we randomly
choose to interchange two variables together from two source inputs or swap a single variable from a
single source input.
**Hierarchical Equality Experiments** For our high-level models abstracting both equality relations
and left equality relation, we sample a set of source inputs and interchange the equality relations of
the corresponding shape pairs from the source inputs with the equality relations from the base input.
For our high-level model abstracting the identity of the first shape, we sample a source input and
interchange the first change from the source input with the base input.
**Monotonicity NLI Experiments** For our high-level models abstracting negation or lexical entailment, we sample a set of source inputs and interchange the boolean value for negative or the value for
lexical entailment from the source inputs with the base input. For our high-level model abstracting
only the identity of replacing lexeme from the _hypothesis_ sentence, we sample another _hypothesis_
sentence from the one seen in training set and interchange its lexeme with the base input. To avoid
cases where entailment labels are invalid (e.g., the entailment relation between “car” and “tree” is
ambiguous), we specifically sample a valid English word that is either a hypernym or a hyponym
of the lexeme item in the _premise_ sentence, and from a new lexeme pair. Then, we construct a new
pair of _premise_ and _hypothesis_ sentences by sampling a sentence template (i.e., a sentence with
replaceable lexeme position such as “a man is talking to someone in a [lexeme]”) from the training
dataset and replacing the lexeme items with new ones.

A.1. Training Data for distributed alignment search (DAS)


**Hierarchical Equality Experiment** We randomly generate 1.92M input–output pairs for training
the model. We train our model for 10 epochs before reaching 100% training accuracy for the task.
We also evaluate model performance on a hold-out testing set with unseen input-output pairs, and our
model achieves 100% testing accuracy. For each high-level model, we then generate a training dataset
for learning the rotation matrix. For each high-level model, we construct 640K such input–output
pairs as our training data and 19.2K pairs as our testing data.
For both training phases, we use a batch size of 6.4K with a maximum training epoch of 10.
We set the learning rate to 1e [´][3] with an early stop patient step set to 10K. Training with a single
NVIDIA 2080 Ti RTX 11GB GPU takes less than ten minutes to converge. All datasets were
balanced across the two labels during standard and interchange intervention training objectives. We
run each experiment three times with distinct random seeds.
**Monotonicity NLI Experiment** We randomly sample 10K examples from the original MoNLI
dataset and use it to train our low-level models to solve MoNLI. We finetune our model for 5 epochs
before reaching 100% training accuracy for the task. We also evaluate model performance on a
FINDING DISTRIBUTED ALIGNMENTS
hold-out testing set, and our model achieves 100% testing accuracy. For training and evaluating the
rotation matrix of each high-level model, we create 24K examples as our training dataset for the first
high-level model, and 10K for the rest two high-level models. For evaluation, we create 1.92K for
the first high-level model, and 1K for the rest two high-level models.
We finetune our model for 5 epochs with a learning rate of 2 _e_ [´][5] before reaching 100% task
accuracy with a batch size of 32. For the learning rotation matrix, we use a batch size of 64 with a
learning rate of 2 _e_ [´][3] for a fixed epoch number of 5. Training with a single NVIDIA 2080 Ti RTX
11GB GPU takes less than ten minutes to converge for both training phases. We run each experiment
three times with distinct random seeds.


Without additional training, our brute-force search baseline finds the best IIA by searching over
possible alignments pΠ _,_ _τ_ q as in Definition 5. For simple feed-forward networks, we map a high-level
variable to a set of low-level variables within a sliding window with a size equal to the intervention
size. We then incrementally search for the sliding window achieving the best IIA score starting from
the first index of the intervened representation in the network. For Transformer-based networks,
we avoid searching over all possible windows to make computation tractable, by only looking at
windows with a starting index from t0 _,_ 64 _,_ 128 _,_ 256 _,_ 512u of the rCLSs token representation. Instead
of targeting a specific set of layers in neural networks, we perform searches over all layers. Note that
for the worst-case scenario, the number of hypotheses for the brute-force search approach becomes
intractable and can be estimated as _Cm_ _[n]_ [where] _[ n]_ [ is the total dimension size of the neural representation,]
and _m_ is the variable dimension size.


Without additional training, our localist alignment baseline finds a local optimal localist alignment
matrix based on the learned rotation matrix. We pick the rotation matrix with the best IIA result from
each category for evaluation. To find a localist alignment matrix, we follow Algorithm 1 to get our
localist alignment matrix _L_ from any orthogonal matrix _R_ . We then use _L_ as our rotation matrix
and evaluate IIA following our evaluation paradigm.


FINDLOCALISTALIGNMENTp _R_ q
1 **//** _R_ is an orthogonal matrix.
2 _R_ a “ _R.aboslute_ _ _value_ pq
3 _L_ “ torch.zeros_likep _R_ q
4 _P_ “ rs
5 **for** _i_ “ 0; _i_ ă _R_ .shape[0]; _i_ ``
6 _P_ `“ rp _R_ a “ torch.maxp _R_ aqq.nonzero()s
7 _R_ ar _P_ r´1s.row _,_ :s “ 0 _._
8 _R_ ar: _,_ _P_ r´1s.cols “ 0 _._
9 **for** _p_ P _P_
10 _L_ r _p_ .row _, p._ cols “ 1 _._
11 _P_ “ _P_ ˚ get_signp _R_ q
12 **return** _P_
FINDING DISTRIBUTED ALIGNMENTS

Algorithm 1 Finding Localist Alignment Matrix


After learning a rotation matrix, we can fix it and learn another rotation matrix on top of it to do
subspace high-level variable alignment. For instance, in the case of our MoNLI experiment, we fix
the rotation matrix aligning the Lexical Entailment representation and further test whether we can
learn another rotation matrix to align word identity. To achieve this, we initialize the first rotation
matrix which aligns a larger subspace and freezes its weights along with the rest of the model. Then,
we train another rotation matrix by taking the output representations from the first one with the
same training objective as the first one as defined in Definition 4. The training data for the second
rotation matrix is not the same as the first one, where we use the training data for the high-level
model hypothesized to align with the subspace (e.g., the training data for the identity of first argument
for the hierarchical equality task, and the training data for the identity of lexeme for the MoNLI
task). Note that for both of our experiments, the subspace dimension is half of its parent subspace for
simplicity.


Table 3 shows the runtime comparison between our method and brute-force search under the same
settings for each task. Only our approach requires training. We underestimate the runtime for
the brute-force search approach by only considering a limited set of possible alignments without
exhaustively searching over the entire combination, which leads to intractable computations (See the
BFSmax column of Table 3). The runtime of our approach can be further optimized if we deploy early
stopping or optimized training data size, and it is invariant with the number of testing hypotheses.
Table 3: Estimated runtime comparison between our method and brute force search (BFS) baseline
(the number of testing hypotheses) for finding an alignment in a single targeted layer
measured under the same settings. The runtime of DAS is invariant with the number of
testing hypotheses.

Appendix B. Runtime Comparison: Brute-force Search Baseline vs. DAS


Hierarchical Equality 31 (32) 6 _e_ [8] ( _C_ 16 [32][)] 502
Monotonicity NLI 198 (5) 2 _e_ [58] ( _C_ 32 [768][)] 1105


Figure 5 shows the rotation in degree(s) of eigenvectors [5] of our learned rotation matrix for each task.
We pick the best-performing oracle low-level model for each task for analyses. Our results suggest
that learned rotations are not trivial, as the majority of basis vectors are rotated. These results suggest
that the representations of high-level variables are highly distributed where direct probes over learned
activation may fail to reveal the actual causal role of the representation effectively.

Appendix C. Remarks on Learned Rotation Matrix


In this section, we answer common questions that may be raised while reading this report.
_Is the learned orthogonal matrix orthonormal?_
5. The eigenvectors of a rotation matrix are the vectors that remain unchanged after the rotation.
Figure 5: Rotation measured in degree(s) of eigenvectors of the learned rotation matrix for each task.
Yes. We use the trainable orthogonal matrix implementation from PyTorch’s torch.nn.utils.
parametrizations. It guarantees the resulting matrix is orthonormal when the rotation matrix is a
full square matrix. Keeping the matrix orthonormal is crucial since it ensures we focus on rotation
[rather than scaling. Details can be found at https://pytorch.org/docs/stable/generated/](https://pytorch.org/docs/stable/generated/torch.nn.utils.parametrizations.orthogonal.html)
[torch.nn.utils.parametrizations.orthogonal.html.](https://pytorch.org/docs/stable/generated/torch.nn.utils.parametrizations.orthogonal.html)
_How stable is the optimization process of the orthogonal matrix?_
We rely on the default initialization of the orthogonal matrix in pytorch. The initialization
step is important for finding the local optimal of the rotation matrix. In our experiment, we use
random seeds and pick the best results out of our distinct runs to address this issue. However, we
may consider different initialization schemes in the future.
_Is an orthogonal matrix required to find distributed alignments?_
In principle, the transformation is not required to be an orthogonal matrix. In fact, an orthogonal
matrix assumes a linear transformation before aligning with a high-level variable, which may not
be optimal if the aligning variable is represented in a non-linear sub-manifold of the representation
space. In such cases, an orthogonal transformation results in imperfect interchange intervention
accuracy, and an invertible and differentiable non-linear transformation may be more suitable (e.g.,
normalizing flow or invertible neural network). In practice, this transformation is computationally
difficult to find, and the linear connections within neural networks also make them unlikely to be
required to find alignments. We leave these investigations to future works.
_What are the prerequisites to deploy this analysis method in practice?_
We assume a partial or complete causal graph of the data generation process. Specifically, we
assume to have interchangeable high-level variables defined for the causal graph. Additionally, we
FINDING DISTRIBUTED ALIGNMENTS
assume we can sample counterfactual data (i.e., base and source inputs where they differ in values of
high-level variables) based on the causal graph.
_How to interpret the result if the interchange intervention accuracy is not 100%?_
When IIA is _α_ ă100%, we rely on the graded notion of _α-on-average_ approximate causal
abstraction Geiger et al. (2023), which directly coincides with IIA. More importantly, the relative IIA
rankings between the high-level models also show which high-level model is a better approximation
of the low-level model.
_Does DAS scale with large foundation models?_
Currently, the number of learnable parameters of the rotation matrix groups in polynomial time
with the size of hidden representations. For instance, if our intervention site size is 512 in the
lower-level model, the number of parameters of the rotation matrix is 512 ˆ 512, which is about
0.26M. If we want to rotate concatenated token sequence embeddings of a BERT-BASE model in any
layer, the number of parameters of the full rotation matrix is about 15.4B which becomes intractable
for standard training infrastructure. To make computation tractable, DAS should be further reducible
by representing only the aligned subspace, not the full rotation matrix. For instance, to find a 2-dim
distributed representation within a 512-dimensional representation space, we approximately only
need to learn 512 ˆ 2 parameters. In addition, we may use a low-rank approximation of the rotation
matrix.
_What are some practical usage of DAS?_
Practically, DAS transforms representations into an operatable state where interchange intervention results in interpretable model behaviors. DAS, itself, is a powerful tool for conducting causal
abstraction analysis of a neural network.


We additionally measure task performance (Task Acc.) as well IIA (Int. Acc.) of our alignments over
training epochs for both seen training examples as well as unseen testing examples. Our results are
shown from Figure 6 to Figure 11.
( _a_ ) | **N** | “ 16
( _b_ ) | **N** | “ 32
Figure 6: Accuracy over training epochs of the high-level model abstracting both equality relations
for hierarchical equality experiment.
FINDING DISTRIBUTED ALIGNMENTS
( _a_ ) | **N** | “ 16
( _b_ ) | **N** | “ 32
Figure 7: Accuracy over training epochs of the high-level model abstracting left equality relation for
hierarchical equality experiment.
( _a_ ) | **N** | “ 16
( _b_ ) | **N** | “ 32
Figure 8: Accuracy over training epochs of the high-level model abstracting identity of first argument
for hierarchical equality experiment.
FINDING DISTRIBUTED ALIGNMENTS
Figure 9: Accuracy over training epochs of the high-level model abstracting both negative and lexical
entailment with | **N** | “ 768 for monotonicity NLI experiment.
Figure 10: Accuracy over training epochs of the high-level model abstracting lexical entailment with
| **N** | “ 768 for monotonicity NLI experiment.
Figure 11: Accuracy over training epochs of the high-level model abstracting the identity of lexeme
with | **N** | “ 768 for monotonicity NLI experiment.


_Xulong Zhang_ [1] _[†]_ _, Haobin Tang_ [1] _[,]_ [2] _[†]_ _, Jianzong Wang_ [1] _[∗]_ _, Ning Cheng_ [1] _, Jian Luo_ [1] _, Jing Xiao_ [1]
1Ping An Technology (Shenzhen) Co., Ltd.
2University of Science and Technology of China

Dynamic Alignment Mask CTC: Improved Mask-CTC with Aligned Cross Entropy

Xulong Zhang
Haobin Tang
Jianzong Wang
Ning Cheng
Jian Luo
Jing Xiao


Because of predicting all the target tokens in parallel, the
non-autoregressive models greatly improve the decoding efficiency of speech recognition compared with traditional autoregressive models. In this work, we present dynamic alignment Mask CTC, introducing two methods: (1) Aligned Cross
Entropy (AXE), finding the monotonic alignment that minimizes the cross-entropy loss through dynamic programming,
(2) Dynamic Rectification, creating new training samples by
replacing some masks with model predicted tokens. The AXE
ignores the absolute position alignment between prediction
and ground truth sentence and focuses on tokens matching
in relative order. The dynamic rectification method makes the
model capable of simulating the non-mask but possible wrong
tokens, even if they have high confidence. Our experiments
on WSJ dataset demonstrated that not only AXE loss but also
the rectification method could improve the WER performance
of Mask CTC.
_**Index Terms**_ **—** Non-autoregressive ASR, Mask CTC,
Aligned cross entropy


Recently, end-to-end automated speech recognition (ASR)
systems have received a lot of interest. There have been
some prior efforts to realize non-autoregressive (NAR) models [1, 2, 3, 4, 5]. The most common method is Connectionist Temporal Classification (CTC) [6]. Wenet[7] applies
CTC beam search to generate _N_ -best candidates and uses
the decoder to rescore the probabilities of these candidates,
generating the final token sentence. Because the sentence
length of candidates is fixed in the rescored process, it makes
Wenet have a non-autoregressive structure and fast decoding speed. Conditional masked language model (CMLM)
is another method adopted in non-autoregressive ASR models. CMLM has proved to be an effective model in various
natural language processing (NLP) tasks [8, 9, 10], and has
been introduced into ASR tasks recently. For instance, Imputer [11] optimizes the potential alignment of the CTC
_†_ Equal contribution.
_∗_ Corresponding author: Jianzong Wang, jzwang@188.com.
iteratively by predicting the frame-level mask of the input
speech. Compared with Imputer, Mask CTC [12] generates
a shorter sequence by refining CTC output depending on
mask token prediction. At the inference stage, the target sentence is initialized using the greedy CTC output. And then
based on the CTC probability, tokens with low confidence are
masked. In each iteration, optimization conditioned on the
other observed tokens and the acoustic features is performed
on the masked tokens. DLP Mask CTC [13] is an improved
version of Mask CTC, which uses an additional predictor to
estimate the length of local masks. In addition, DLP Mask
CTC also employed Conformer [14] to enhance the encoder
network architecture. DLP Mask CTC obtains a substantial
recognition accuracy improvement over Mask CTC.
Despite achieving a high inference efficiency, the above
non-autoregressive models still encounter some tough problems. One is that the decoder network of NAR model is
usually trained on cross entropy (CE) loss, but the CE loss
might be too strict for NAR model training. Cross entropy
requires the force-alignment between the model predictions
and ground truth tokens. It means that small shift in predicted
tokens will result in large loss penalty, even if the content of
tokens matches very well. Aligned cross entropy (AXE) [15]
was proposed as a relaxed loss function for training nonautoregressive natural language processing (NLP) models. In
machine translation tasks, CMLM models trained with AXE
loss could significantly outperform the models trained with
traditional CE loss [15]. In this work, we introduce the AXE
loss to the decoder training of Mask CTC. We think AXE loss
could make the model focus on the tokens matching instead
of tokens ordering to improve the recognition accuracy of
NAR speech recognition models.
Another issue is that the decoder input of Mask CTC is the
greedy CTC search at the inference stage, while the ground
truth sentence is inputted to the decoder at the training stage.
This causes a mismatch between the training and inference.
The high confidence tokens of CTC search are reserved as
non-mask tokens. However, they may also have errors. These
non-mask tokens could not be refined and influence the filling
of neighbor mask tokens. In this paper, we propose a dynamic
rectification method to alleviate this problem.


**Fig. 1** . Conformer Architecture of Dynamic Alignment Mask
CTC with Mask and Dynamic Rectification Methods
We propose a non-autoregressive end-to-end automatic
speech recognition framework. Our works are based on Mask
CTC [12]. We try to improve Mask CTC by two methods: (1)
aligned cross entropy _θaxe_, finding a monotonic alignment
to minimize the cross entropy loss on masked tokens, (2)
dynamic rectification _θrec_, simulating the high confidence
unmasked tokens by using model predictions, to produce new
training samples. Aligned cross entropy is a relaxed loss,
which ignores the absolute positions and focuses on the relative order and tokens matching. Dynamic rectification uses
predicted tokens as the input of model decoder, closing to the
greedy CTC search result at the inference procedure.


As Figure 1 depicts, the proposed dynamic alignment Mask
CTC is encoder-decoder model architecture built on Conformer [14] (or Transformer [16]) blocks. The encoder transforms a speech acoustic feature sequence _X_ = ( _x_ 1 _, x_ 2 _, ..., xT_ )
to a hidden representation _H_ . The decoder network outputs
a text label sentence _Y_ = ( _y_ 1 _, y_ 2 _, ..., yS_ ). We apply CTC to
the encoder output _H_, and the Aligned Cross Entropy (AXE)
objective is used to train the decoder. Frame-level alignment
_A_ is predicted by CTC between the input sequence _X_ and
the output sentence _Y_ . The definition of the CTC loss [6] is
stated below:
_Lctc_ ≜ _−_ ln _Pctc_ ( _Y |X_ ) = _−_ ln  - _P_ ( _A|X_ ) (1)
_A∈β_ _[−]_ [1] ( _Y_ )
where _P_ ( _Y |X_ ) is the probability over all possible paths.
_β_ _[−]_ [1] ( _Y_ ) returns all possible alignments compatible with _Y_ .
At the training stage of dynamic alignment Mask CTC,
we firstly applied mask method _θmask_ on ground truth target sentence _Y_ to produce _Ymask_ = _θmask_ ( _Y_ ). Secondly,
dynamic rectification method _θrec_ modifies the decoder input
sentence to _Yrec_ = _θrec_ ( _Ymask_ ) = _θrec_ ( _θmask_ ( _Y_ )). Then,
the decoder predicts the whole label sentence _Y_ conditioning
on the input audio _X_ and modified token sentence _Yrec_ as
follows:
_Laxe_ ≜ _−_ ln _Paxe_ ( _Y |Yrec, X_ )
(2)
Here, we introduce the Aligned Cross Entropy (AXE) loss [15]
to the training of decoder instead of CMLM loss in Mask
CTC. In which, _Pα_ ( _i_ )( _yi|Yrec, X_ ) denotes an aligned cross
entropy between _Y_ and _Y_ [�] . _Pk_ ( _ϵ|Yrec, X_ ) denotes a penalty
for unaligned predictions, which is noted as a special token
_ϵ_ . _k_ is the index of unaligned tokens in _Y_ . Finally, through
[�]
the CTC loss _Lctc_ and AXE loss _Laxe_, the overall loss _L_ of
dynamic alignment Mask CTC is calculated with scale factor
_λ ∈_ [0 _,_ 1].
_L_ = _λ Lctc_ + (1 _−_ _λ_ ) _Laxe_ (3)


As illustrated in Table 1, the mask method starts with an
input text sentence _Y_ . For training, _Y_ is ground truth sentence of labeled data. For inference, _Y_ is the greedy CTC
search result without using beam search like Mask CTC. At
training stage, the ground truth tokens are randomly substituted with _⟨mask⟩_ . The number of mask tokens is sampled from uniform distribution between 1 to _Lmask_ . During
inference, we mask the positions with low confidence below the threshold _Pthres_ . Therefore, the masked sentence
ln _Pk_ ( _ϵ|Yrec, X_ )
_k_
= _−_
_S_
- ln _Pα_ ( _i_ )( _yi|Yrec, X_ ) _−_
_i_ =1 _k_
**Table 1** . Illustration of Mask Method and Dynamic Rectifcation
_**Y**_ This is dynamic alignment **mask** CTC non autoregressive speech recognition
_**Ymask**_ This _⟨mask⟩_ dynamic _⟨mask⟩_ _⟨mask⟩_ CTC non _⟨mask⟩_ _⟨mask⟩_ recognition
_**Y**_ - This is dynamic alignment **task** CTC non autoregressive speech recognition
_**Yrec**_ This is _⟨mask⟩_ _⟨mask⟩_ **task** CTC _⟨mask⟩_ autoregressive speech recognition
_Ymask_ could be obtained through this mask method _θmask_ by
_Ymask_ = _θmask_ ( _Y_ ).
During training, we use current best ASR model to predict text sentence _Y_ [�] from the mask method output _Ymask_ .
_Y_ - will fill the masks of _Ymask_ with the highest confidence
in these mask locations. After that, the dynamic rectification method masks the text sentence _Yrec_ again. The number
of second rectification masks is between 1 to _Lrec_ . Through
dynamic rectification, the output sentence _Yrec_ will not only
contains mask tokens but also may contain wrong predicted
tokens with high confidence. It simulates the result of greedy
CTC at the inference stage and alleviates the mismatch problem of decoder input between training and inference.

2.2. Mask Method and Dynamic Rectification


The AXE loss is the minimum over all possible monotonic
alignments _α_ : _Y_ _→_ _Y_ of the conditional cross entropy
[�]
loss as shown in Eq. 2. Dynamic programming is used by
AXE to determine the optimal alignment between the current
prediction _Y_ [�] _j_ and ground truth token _Yi_ . The matrix _Mi,j_
( _i, j ∈_ [1 : _S_ ]) represents the minimal AXE loss value for this
optimal alignment. Three local update operators are defined
in the dynamic programming of AXE loss [15]: (1) align,
aligning the current prediction _Y_ [�] _j_ and ground truth token _Yi_
with probability _P_ ( _Y_ [�] _j|Yi, X_ ), (2) skip prediction, skipping
the current prediction _Y_ [�] _j_ and inserting a special token _ε_ to the
ground truth token _Yi_, (3) skip ~~t~~ arget, skipping the current
ground truth token _Yi_ without incrementing the prediction _j_ .
This operation is penalized with the hyperparameter _γ_ .
_align_ = _Mi−_ 1 _,j−_ 1 _−_ log _P_ ( _Y_ [�] _j|Yi, X_ ) (4)
_skip_ ~~_p_~~ _rediction_ = _Mi,j−_ 1 _−_ log _P_ ( _ε|Yi, X_ ) (5)
_skip_ ~~_t_~~ _arget_ = _Mi−_ 1 _,j −_ _γ ·_ log _P_ ( _Y_ [�] _j|Yi, X_ ) (6)
_Ai,j_ is filled by taking the minimum from above three possible operators. After dynamic programming, the cross entropy
loss of the optimal alignment will be present in the cell _MS,S_ .
With AXE loss, the emphasis is placed on the expected token matching rather than the penalty for token order mistakes.
Otherwise, the traditional CE loss is a too strict criterion for
small token shifts in _⟨mask⟩_ locations.


At inference stage of dynamic alignment Mask CTC, the
ground truth sentence _Y_ is replaced by greedy search result
of CTC decoding. We also applied mask method on _Y_ to obtain _Ymask_ . But we do not use dynamic rectification, which is
only activated at training stage. Therefore, _Ymask_ is directly
as the input of decoder, to get the final predicted sentence
_Y_ - . We apply the iterative decoding methods with _K_ total
decoding iterations. For each iteration, the decoder predicts
the masked locations � _ym_ in sentence _Y_ [�] as follows:
_y_     - _m_ = arg max _Paxe_ ( _y_     - _m_ = _w|Ymask, X_ ) (7)
_w_
Then, top _C_ masked tokens with the highest probability are
reserved, where _C_ is the average number of mask tokens for
_K_ iterations. Other masked tokens are changed to masks for
the next iteration. Until all the mask tokens are filled, the
decoding method terminates.


We perform all the experiments on public speech corpus,
WSJ1 [17] and WSJ0 [18]. We use si284, dev93, and eval92
for training, validation, and testing respectively. All the
experiments are conducted on the ESPNET2 toolkit [19].
For the network inputs, the audio data is encoded with
80 mel-scale filterbank coefficients with 3-dimensional pitch
features extracted using Kaldi recipe. We use speed perturbation [20] and SpecAugment [21] as data augmentation techniques to avoid model overfitting. The Transformer encoder
consists of 2 CNN layers and 12 self-attention layers. 12 Conformer blocks make up the conformer encoder. The decoder
consists of 6 self-attention layers. The multi-head attention
has 4 heads, 256 dimension, and 2048 feedforward dimension. The dimension of decoder output is 65 (including capital English letters, masks, and punctuations). The final model
was derived by averaging the top 30 models based on their
validation accuracy after the training process. The scale factor _λ_ of CTC and AXE loss is set to 0 _._ 3. During inference,
the CTC confidence threshold _Pthres_ is 0 _._ 999.


We first explore the effectiveness of different modules of our
dynamic alignment Mask CTC model as follows: 1) **Mask +**
**CE** : The model trained with the joint CTC and CE loss. This
is the baseline Mask CTC model; 2) **Mask + AXE** : The model
which is only processed by mask method, and is trained on
joint CTC and AXE loss; 3) **Mask + Rec + AXE** : We apply mask and dynamic rectification methods to create training
samples and train this model on joint CTC and AXE loss.
All models are evaluated by WER (Word Error Rate) and
RTF (Real Time Factor). As listed in Table 2, we compared
the results of our models with baseline Mask + CE model.
When replacing the traditional CE loss with AXE loss, the
WER result of Mask + AXE model will decrease significantly. The Conformer encoder outperforms the Transformer
encoder, and they have equivalent RTF speed. With dynamic
rectification method, the Mask + Rec + AXE model could
further improve the WER results on both dev93 and eval92
dataset. In addition, we also investigate different parameters
in decoding procedure. With more decoding iterations, the
**Table 2** . Different Training Methods and Decoding Iterations,
WER and RTF Results on WSJ
|Training Method Transformer|Iter|dev93|eval92|RTF|
|---|---|---|---|---|
|Mask + CE
Mask + AXE
Mask + Rec + AXE|1
1
1|16.8
15.8
15.3|14.3
12.5
11.8|0.04
0.04
**0.04**|
|Mask + CE
Mask + AXE
Mask + Rec + AXE|10
10
10|16.5
15.7
**15.2**|13.9
12.2
**11.6**|0.07
0.07
0.07|
|Conformer|Col2|Col3|Col4|Col5|
|---|---|---|---|---|
|Mask + CE
Mask + AXE
Mask + Rec + AXE|1
1
1|14.6
13.9
13.7|12.1
11.4
11.3|0.04
0.04
**0.04**|
|Mask + CE
Mask + AXE
Mask + Rec + AXE|10
10
10|14.1
13.7
**13.6**|11.7
11.2
**11.1**|0.07
0.07
0.07|
WER performance of the model will become better. The results also demonstrated that our models are insensitive to decoding parameters. When the iteration number _K_ is changed
from 10 to 1, it suffers only a little degradation on WER
results. However, it gains a huge inference speed promotion
with a relatively 75% improvement on RTF performance.


We contrast different autoregressive and non-autoregressive
approaches with our best Mask + Rec + AXE model. As
shown in Table 3, our dynamic alignment Mask CTC model
achieves better WER results than vanilla Mask CTC* (our results) at both Transformer and Conformer architecture. The
results also indicate that our method outperforms most previous non-autoregressive methods. In addition, DLP Mask CTC
needs an additional predictor to estimate the length of masks,
resulting in a more complicated model structure but slightly
better performance. Our methods could be easily applied to
similar length predictions with a lighter structure.
To investigate how AXE loss improves the model training,
we plot the scatter diagram between training loss and Levenshtein distance in Figure 2. The Levenshtein distance is calculated by ground truth sentence and predicted output by iterative decoding with maximum token probability. The figures
show that both AXE and CE loss have a linear relationship
(a) CE Loss (b) AXE Loss
**Fig. 2** . Relationship Between Training Loss and Levenshtein
Distance of Ground Truth and Predicted Sentences
**Table 3** . Compared with Other Non-Autoregressive and Autoregressive Methods, WER and RTF Results on WSJ


_**Autoregressive**_
_Transformer_
CTC-Attention S 13.5 10.9 4.62
_Conformer_
CTC-Attention S 11.1 8.5 5.09
_**Non-Autoregressive Previous Work**_
_Transformer_
CTC 1 19.4 15.5 0.03
Mask CTC* 10 16.5 13.9 0.06
Mask CTC + DLP 10 13.8 10.8 0.07
Imputer (IM) 8    - 16.5    Imputer (DP) 8    - 12.7    Align-Refine 10 13.7 11.4 0.06
_Conformer_
CTC 1 13.0 10.8 0.03
Mask CTC* 10 14.1 11.7 0.06
Mask CTC + DLP 10 11.3 9.1 0.08
_**Our Work**_
_Transformer_
Proposed 10 15.2 11.6 0.07
_Conformer_
Proposed 10 13.6 11.1 0.07
with Levenshtein distance. However, there are two kinds of
outliers for CE loss (see red boxes in Figure 2(a)). The first
outliers have large CE loss but have small Levenshtein distance, mainly caused by token order mismatch, even if their
edit distance is small. By contrary, the AXE will have reasonable loss value by relieving this order mismatch penalty.
In addition, the second outliers have small CE loss, but their
predictions are quite different from ground truth sentence. We
conjecture that multimodality could still be a problem for CE
loss. For example, “form or” and “for more” have similar
pronunciations and may have both high token probabilities
and low CE losses. Instead, their AXE losses will be quite
different (see Figure 2(b)).


In this paper, we propose an end-to-end NAR speech recognition model, dynamic alignment Mask CTC. The AXE loss
makes the model focus on the tokens matching and relaxes the
restriction of tokens order. The dynamic rectification could
reduce the mismatch of decoder input between training and
inference, simulating the high confidence but possible wrong
tokens of greedy CTC output. Experimental results demonstrate that our proposed model outperforms the vanilla Mask
CTC model in terms of WER.


Supported by the Key Research and Development Program
of Guangdong Province (grant No. 2021B0101400003) and
Corresponding author is Jianzong Wang (jzwang@188.com).


**Andreas Köpf** _[∗]_ **Yannic Kilcher** _[∗]_
```
     andreas.koepf@provisio.com yannic@ykilcher.com
```

OpenAssistant Conversations -- Democratizing Large Language Model
  Alignment

OpenAssistant Conversations - Democratizing Large Language Model Alignment

Andreas Köpf
Yannic Kilcher
Dimitri von Rütte
Sotiris Anagnostidis
Zhi-Rui Tam
Keith Stevens
Abdullah Barhoum
Nguyen Minh Duc
Oliver Stanley
Richárd Nagyfi
Shahul ES
Sameer Suri
David Glushkov
Arnav Dantuluri
Andrew Maguire
Christoph Schuhmann
Huu Nguyen
Aleaxnder


```
           alexander.mattick@googlemail.com
```


Aligning large language models (LLMs) with human preferences has proven
to drastically improve usability and has driven rapid adoption as demonstrated
by ChatGPT. Alignment techniques such as supervised fine-tuning ( _SFT_ ) and
reinforcement learning from human feedback ( _RLHF_ ) greatly reduce the required
skill and domain knowledge to effectively harness the capabilities of LLMs,
increasing their accessibility and utility across various domains. However,
state-of-the-art alignment techniques like _RLHF_ rely on high-quality human
feedback data, which is expensive to create and often remains proprietary. In an
effort to democratize research on large-scale alignment, we release OpenAssistant
Conversations, a human-generated, human-annotated assistant-style conversation
corpus consisting of 161,443 messages in 35 different languages, annotated with
461,292 quality ratings, resulting in over 10,000 complete and fully annotated
conversation trees. The corpus is a product of a worldwide crowd-sourcing effort
involving over 13,500 volunteers. Models trained on OpenAssistant Conversations
show consistent improvements on standard benchmarks over respective base
models. We release our code [2] and data [3] under a fully permissive licence.
A list of contributors who have chosen to be acknowledged by name can
be found at `[https://open-assistant.io/contributors](https://open-assistant.io/contributors)` .
_∗_ These authors contributed equally to this work.
2 `[https://github.com/LAION-AI/Open-Assistant](https://github.com/LAION-AI/Open-Assistant)`
3 `[https://huggingface.co/OpenAssistant/oasst1](https://huggingface.co/OpenAssistant/oasst1)`
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.


Artificial intelligence (AI), particularly in the field of natural language processing, has witnessed
rapid progress in recent years. Major advancements are primarily driven by a straightforward formula:
take a Transformer [1]-based architecture, increase the parameter count by enlarging depth and width,
increase the size of the training corpus, and increase the scale of training compute. Although models
have for some time exhibited an extraordinary ability to fit the training data and generalize based on
their trained objective [2, 3], their adoption among the general public has until recently been slow.
This can be mainly attributed to misalignment between model predictions and final intended usage.
The alignment of AI systems to human values, intentions, and preferences is a vital and intricate
challenge within the AI research domain. This refers to the process of ensuring that AI systems
can not only successfully optimize the provided surrogate training objectives, but also that their
predictions are in line with their intended purpose and adhere to ethical and safety standards provided
by humans [4, 5]. One possible solution is assistant-style fine-tuning of language models that has
recently emerged as a promising approach to making large language models more in line with human
preferences by generating more desirable outputs based on explicitly collected human preference
data [6, 7, 8, 9, 10, 11] and thus making them more useful.
A notable instance of such an assistant-style model is ChatGPT, which has gained unprecedented
user growth due to remarkable capabilities demonstrated in a wide range of fields, but also easeof-use for the end user [12]. Aligning the model’s predictions is in this case accomplished by
introducing human-generated examples of intended usage and using reinforcement learning from
human feedback [13, 14]. In _RLHF_, the human acts as a teacher and provides feedback in the form of
rewards or penalties. In more detail, Ouyang et al. [13] proposed a three stage procedure to align
language models: First, collect human-generated demonstrations of desired behaviour and train a
supervised fine-tuned ( _SFT_ ) model. Second, train a reward model (RM) on human-annotated rankings
for different model outputs. Third, use the RM as a reward function and fine-tune the _SFT_ model to
maximize the reward generated by its responses. This is achieved using the PPO algorithm [15].
It becomes apparent that the benefits of all the aforementioned stages are predominantly dependent
on the quality of the data used [16]. Despite this, availability of large-scale human feedback datasets
for the open research community remains scarce. Most openly accessible datasets are comprised of
synthetic data of instructions automatically generated by querying language models [17, 18, 19, 20].
Unfortunately, these datasets are limited with respect to their complexity, creativity and quality, as
they rely on a pre-specified list of possible instruction types. Other datasets, such as Vicuna [21], use
human-generated instructions, but still rely on langauge models to produce the respective responses.
Without sufficiently broad and high quality data, even models with substantial size and pre-training
would be inadequate for building capable, helpful, and harmless AI assistants.
Research in this area has predominantly been confined to a select few research labs with access to the
required resources to engage in large-scale training and data collection. This monopolization of access
to quality data undermines the potential for inclusive and diverse research endeavours, particularly in
relation to alignment challenges, which arguably constitute some of the most crucial research areas
of our time. In an effort to democratize research on aligning large language models, we introduce
and release the OpenAssistant Conversations dataset. This dataset is the culmination of an extensive
open- and crowd-sourcing initiative, and its release to the research community seeks to promote
more inclusive research in this highly-influential domain. We provide a comprehensive analysis of
the dataset, assessing ethical implications and safety considerations. We also fine-tune and release
several assistant and preference models to further advance open access and research in this area. This
transparency allows for iterative improvements on the released artifacts, fostering a more collaborative
and inclusive research environment. By providing such a large and diverse dataset, OpenAssistant
Conversations opens up new avenues of research in the field, enabling researchers to explore the
complexities of human language and interactions in ways that were not possible before [22]. In the
following sections, we delve into the intricacies of the OpenAssistant Conversations dataset and
discuss its implications for the alignment of large language models and for society at large.


The proposed dataset consists of a list of conversations. The basic data structure is a _Conversation_
_Tree (CT)_, with nodes representing written messages in a conversation. A CT’s root node represents
an initial prompt, given by the prompter. To avoid confusion, we call the roles of the conversation
_prompter_ and _assistant_ . This allows us to reserve the term _user_ for the human contributors. Although
our efforts focus largely on human contributions, both the prompter and assistant roles can, in
principle, be fulfilled by either a human user or a machine. Every tree node is labelled by its role,
and can have multiple children of the opposite role, each of which represents a separate next step in
the conversation. A path from the root to any node in the tree (including to itself) is called a _thread_,
and it represents a valid conversation with the prompter and the assistant taking turns. Tree nodes
are annotated with additional data such as user-provided labels and metadata, such as collection
timestamp and indicated language. Each _assistant_ node further has a rank associated which orders it
compared to replies of the parent prompt, according to user preferences.
Figure 1: An example CT of depth 4 containing 12 messages. Any path from the root prompt to a
node is a valid thread.


The OpenAssistant Conversations dataset is a comprehensive collection of conversational data that
was obtained through a crowd-sourcing effort involving more than 13,000 volunteers. The data
was collected using a web-app interface, dividing the collection of each tree into five separate
steps: _prompting_, _labelling prompts_, _adding reply messages as prompter or assistant_, _labelling replies_,
and _ranking assistant replies_ . Users were fully informed about contributing to a public dataset. The
dataset was curated with content moderation and spam filtering as key components of the annotation
pipeline, ensuring high quality and safety standards.
Volunteers completed over 625,000 tasks in total, resulting in the collection of over 10,000 fully
annotated and filtered Conversation Trees. Example User Interface (UI) displays of the data collection
platform can be found in Appendix C, and current collection parameter settings, such as the number
of collected replies to any parent message, can be found in Appendix G. In the following sections, we
provide more details regarding the various aspects of the data collection pipeline.


Data collection is structured to be both efficient and effective by breaking the work down into single
units and advancing multiple conversation trees one step at a time. This approach minimizes data
loss due to user attrition and ensures that every unit of work is captured for utilization. The users
are presented with a range of task types, either by choice or through random sampling (weighted
according to current requirements). The task types include creating prompts, replying as an assistant,
replying as a prompter, labeling prompts or replies, and ranking prompter or assistant replies.
**Create a prompt.** Users write an initial prompt that forms the root of a new conversation tree. As
this task is particularly popular, a lottery system manages the selection of new prompts, with only a
fixed number of prompts being chosen for continuation at any given moment. This method serves to
regulate the influx of new prompts and maintain a balanced distribution of tasks.
**Reply as assistant.** Replying as an assistant is a more labor-intensive task that necessitates users
to carefully consider their responses and often engage in external research to provide a helpful and
relevant answer to the prompter’s request. This task type, despite its demanding nature, has been
reported to be the most enjoyable by many users due to the diverse array of topics covered. To account
for the increased effort required for this task, a reward system has been implemented to incentivize
users to participate. See Figure 6 for a UI preview.
**Reply as prompter.** The task of replying as a prompter, on the other hand, does not impose strict
quality requirements but instead emphasizes on the importance of diversity to accommodate various
use-cases. Examples of prompter replies may include asking for clarification, modifying the original
intent, posing a follow-up question, or changing the direction of the conversation altogether.
**Label a prompt or reply.** Users are presented with a message from the database along with the
preceding conversation thread (if available) and are asked to categorize the message according to
three dimensions: spam detection, guideline adherence, and quality. For spam detection, users assess
whether the message is unsuitable for inclusion in the dataset, for instances of obvious spam or
trolling. Messages flagged as spam by multiple users are automatically removed from the dataset.
Guideline adherence is evaluated through a set of labels that determines whether the contribution
aligns with the established guidelines (see Figure 4). These labels encompass the message being in a
language other than the specified one, containing personally identifiable information, hate speech,
sexual content, or being deemed inappropriate. Messages labelled in this manner are subsequently
reviewed by human moderators.
Quality labels require users to rate the message on a five-point _Likert_ scale across dimensions such
as quality, creativity, humorousness, politeness, and harmlessness. These labels are stored for later
analysis and application. Notably, users can voluntarily assign these labels (as well as spam &
guideline adherence labels) to any message within the system, even as part of another task, as an
additional contribution.
**Rank assistant replies.** Users are presented with two or more responses to the same parent message
and asked to rank them in order of preference. This allows for a comparative analysis of the various
responses and helps in identifying the most effective and engaging replies (Figure 5).
In summary, this data collection methodology effectively divides work into single units, minimizes
data loss due to user attrition, and captures valuable information for future analysis and application.
By offering users a diverse range of task types, the study encourages active participation and ensures
the collection of rich and varied data for a comprehensive understanding of the subject.


The tree state machine serves as a systematic approach to managing the progression of message trees
throughout the data collection process. This method ensures that each tree undergoes a series of
states until it reaches completion, beginning with the creation of new trees by randomly sampling
from the pool of initial prompts. The various states that a message tree passes through include the
_initial prompt review state_, _growing state_, and _end state_, as well as the _aborted low-grade state_ for
trees that are deemed unsuitable for inclusion in the dataset and the _halted by moderator_ state for
trees that have manually been halted by a community moderator.
Upon the creation of a new tree, it enters the _initial prompt review state_, where multiple users are
tasked with providing labels to assess its quality and suitability. This state plays a crucial role in
identifying any potential issues with the initial prompt, and demands special attention, as the entire
rest of the tree (potentially several dozens of tasks) is rooted in the initial prompt. If, at this point,
the provided labels indicate that the tree contains spam or unsuitable content, it is transitioned to the
_aborted low-grade state_ and subsequently removed from the dataset. Conversely, if the tree passes
the _initial prompt review state_, it proceeds to the _growing state_ .
The _growing state_ involves the continuous issuance of tasks to users, such as providing replies, labels,
and rankings, to facilitate the development and expansion of the conversation tree. This state is
essential for collecting diverse and rich data, as it allows for the accumulation of multiple interactions
and the exploration of various conversation paths, given the same initial prompt. The _growing state_
continues until the tree reaches its _end state_, which is defined by a maximum number of messages or
other predetermined criteria.
Parameters within the data collection platform govern the behaviour of the tree state machine, such as
the average number of messages collected for each parent message or the maximum tree depth. These
parameters enable researchers to fine-tune the data collection process according to their specific
research goals and requirements, ensuring a more targeted and efficient approach to gathering data.
Parameters varied during the collection of the dataset. Current settings can be found in Appendix G.
In summary, the tree state machine is a structured and systematic method for managing the progression
of message trees during the data collection process. By guiding each tree through a series of states,
from initial prompt review to growing and reaching its _end state_, the tree state machine ensures
the collection of high-quality, diverse, and relevant data. Additionally, the inclusion of platform
parameters allows for the customization of the data collection process to align with specific research
objectives, further enhancing the effectiveness and utility of this approach.


To achieve a high degree of quality and consistency across a wide range of contributors, we issue
clear and detailed guidelines. A full copy of these guidelines at the present time can be found in
Appendix A. Our guidelines follow three main goals: First, clarify the meanings, scales, and criteria
for assigning labels and rankings during the labelling and ranking tasks. Second, make assistant
responses be polite, helpful, concise, friendly, and safety-aware and third, instruct prompts and
prompter replies to explore a diverse and challenging set of inputs to the assistant role.
The guidelines establish a framework for safely interacting with an automated assistant by drawing
inspiration from the concept of _informed consent_ . Rather than categorically denying large parts of
request categories, we aim to provide the prompter with useful feedback, for example drawing special
awareness to dangerous activities, elaborating on weaknesses of automated assistants, such as hallucinations, and discouraging and denying requests asking for illegal or highly inappropriate content. In
our validation experiments in training assistant models based on OpenAssistant Conversations, we
observe a high degree of consistency of the trained models’ outputs with our given guidelines.
Although guideline adherence is already high in our models after training, our approach is completely
compatible with deploying additional safety measures during inference, such as secondary models to
filter or modify ill-suited user input.


We take a multi-pronged approach to quality assurance, with the main pillars being a system of
reward points & leaderboards, and manual review of flagged content by human moderators. This both
maximizes the quality of contributions, while effectively utilizing the limited time of the volunteer
moderators. In an effort to demonstrate progress and achievement to users, and to encourage highquality contributions, our system allocates points for the completion of tasks. These points contribute
to various leaderboards, including daily, weekly, monthly, and all-time rankings. A level system
also exists, wherein higher point accumulation results in elevated levels, reflecting veteran status and
engagement. In the future, this system could potentially be developed further to facilitate preferential
access to more engaging tasks or similar perks.
The distribution of points is contingent upon task type, as certain tasks require greater effort, such as
the _reply as assistant_ task (compared to the _create a prompt_ task). A significant portion of points
is deferred and reliant on interactions with other users. For instance, a user’s assistant reply may
gather many additional points if it is subsequently deemed non-spam and highly ranked by other
users. Inversely, points may be reduced or lost for answers that are labeled as spam or down-voted by
consensus of other users.
Within the moderator section of the website, an alternative leaderboard, designated the _Trollboard_,
is exhibited. This leaderboard assesses users based on an aggregate of negative labels, reports, and
down-votes received for their contributions. This approach enables human moderators to proactively
scrutinize potentially misbehaving users in a comprehensive manner. The Trollboard has proven to
be an effective tool in addressing the numerical disparity between users and moderators, maximizing
the collective efforts of contributors to identify undesirable contributions.
Users further have the option to report messages to moderators for manual review, either via the
platform, or directly via communication on a community chat server. Moderators have the ability to
delete individual messages, or all messages of a given user, at their own discretion. Deleted messages
are retained, but marked as deleted and are not exported for training.

3.4 Quality Control & Content Moderation


The full dataset consists of 161,443 messages (91,829 prompter and 69,614 assistant messages)
distributed across 66,497 conversation trees, in 35 different languages, annotated with 461,292
quality ratings. This includes 8,576 synthetic messages, leaving 152,867 human-submitted messages.
Of the 66,497 total conversation trees, we consider 10,968 complete, meaning the full number of
messages has been collected and the moderation process for these trees has been concluded. 52,159
incomplete trees are in the prompt lottery state, meaning they only consist of a single initial prompt.
The completed trees contain 92,365 messages.
The set of categories for which _Likert_ -scale human labels are collected is Creativity, Quality, Humor,
Helpfulness, Violence, and Rudeness. The set of categories for which binary human labels are
collected is Language Mismatch, Not Appropriate, Personally Identifiable Information, Hate Speech,
and Sexual Content. We additionally release the rank of each assistant message compared to other
assistant messages submitted for the same prompt, computed from the preference rankings of several
human annotators. To merge the rankings of multiple (possibly conflicting) annotators, we use a
variant of Tideman’s method, described in Appendix B.
Figure 2: **Left:** Relative share of the most frequent languages in the dataset. **Right:** Distribution of
contributions among users.
The dataset is dominated by English and Spanish messages as illustrated in Figure 2 (left). The
prominence of English is expected as a result of the community around OpenAssistant originating
in the English-speaking open-source machine learning community. The high quantity of Spanish
messages can be attributed to the publicity given to OpenAssistant by prominent figures in the
Spanish machine learning community. Figure 2 (right) illustrates how a small number of power users
contributed a significant proportion of the dataset. This must be taken into account when considering
possible biases in the data. Although significant effort went into preventing responses directly copied
from other sources, it is possible that some users utilised automated techniques to enter data.
We release the dataset on the Hugging Face Hub [4] in several variants: One variant containing all
collected initial prompts, one variant containing all trees that are considered completed, one variant
containing all trees (completed and in-progress), and one variant containing messages filtered out as
spam. For most purposes, such as instruction-tuning of language models, we recommend using the
variant containing all completed trees, and include non-completed trees if more data is required.


To gain a deeper understanding of the contributors’ demographics, a Google Form survey was sent out
as an announcement on the project’s Discord channel. The survey consists of 3 parts with questions
on demographic information, personal motivation and user satisfaction. At the time of the release of
this paper, a total of 270 survey answers have been collected. Results can be seen in Figures 7,8,9,10,
and Tables 3 and 4 (all in the Appendix). Respondent answers indicate a homogeneity towards
identifying as male, but strong diversity in the reported levels of education, stated motivations for
contribution, proficiency in artificial intelligence, and use cases for the technology. Over 95% of
respondents either agreed or strongly agreed with the statement "Overall, I’m glad I have contributed
4 `[https://huggingface.co/OpenAssistant/oasst1](https://huggingface.co/OpenAssistant/oasst1)`
to OpenAssistant." About 40% reported this being their first time contributing to an open-source
project. We note that the method of recruiting via Discord is biased towards users who are present on
the platform and have been active around the time of the announcement. Active contributors are also
expected to be more likely to respond.

5 Contributor Demographics and Satisfaction


We focus on the development and evaluation of fine-tuned language models based on Pythia [3],
LLaMA [2], and Falcon [23]. Pythia and Falcon are state-of-the-art language models with permissive
open-source licenses, while LLaMA is a powerful language model with a bespoke non-commercial
license. Specifically we train supervised fine-tuned (SFT) models, reward models (RM [13],[24]),
and, using the trained reward models, reinforcement-learned models (RLHF) [5] .
Model **LMEH** **VEL** **OAIE** **HE**
gpt-3.5-turbo (ChatGPT) 1110 0.87 0.72
EleutherAI/pythia-12b 60.33
OpenAssistant/pythia-12b-sft-v8-7k-steps 60.28 997 0.10 0.10
tiiuae/falcon-40b 72.29
OpenAssistant/falcon-40b-sft-top1-560 74.04 1192 0.26 0.09
OpenAssistant/falcon-40b-sft-mix-1226 74.40 1053 0.44 0.13
huggyllama/llama-65b 67.24
OpenAssistant/oasst-sft-7e3-llama-30b 68.03 979 0.52 0.20
OpenAssistant/oasst-rlhf-3-llama-30b-5k-steps 68.51 1068 0.51 0.15
Table 1: Comparison of model evaluation scores on different LLM benchmarks: **LMEH:** lmevaluation-harness [25] (average scores, see online leaderboard for more details) **VEL:** Vicuna Elo
Rank [21] **OAIE:** OpenAI Evals [26] **HE:** HumanEval [27] (for all benchmarks, higher is better).
We have chosen to leave the Hugging Face Hub identifiers as the model names for identifiability.
Table 1 shows evaluation scores of a selection of baseline and trained models on a set of standard
benchmark datasets. Evaluations were performed externally using FastEval and evaluation results
are hosted on a leaderboard, which is continually being updated [6] . LMEH refers to the average
performance on a set of widely-used NLU tasks consisting of BoolQ, PIQA, HellaSwag, WinoGrande,
ARC-e, ARC-c and OBQA [7] . We omit instruction-centric experiments (VEL, OAIE, HE) for the
base models, as these benchmarks are unsuitable for non-instruction-tuned models. The results show
that models using OpenAssistant Conversations are consistently outperforming the corresponding
baseline models (in the case of LLaMA even a larger baseline model). RLHF outperforms SFT in
some benchmarks, but not in others. For Falcon-based models, _sft-top1_ is trained only on top-ranked
conversation threads, whereas _sft-mix_ mixes OpenAssistant Conversations with other instruction
datasets (details in Appendix H). The varied evaluation scores demonstrate that by combining different
data sources, the nature of the resulting model can be readily influenced. Ranks across benchmarks
are not consistent, which could indicate the unsuitability of automatic evaluations for language
models, or could indicate that different models and datasets lead to different capabilities. The results
also show that while open-source models are close to matching ChatGPT in some benchmarks, others
still show large performance gaps. Anecdotally, users report OpenAssistant models to be less robotic
and more human-sounding than commercial models and report generations to have high quality and
diversity in domains such as creative writing, conversational messaging, and drafting social media
posts.
5All trained models are released at `[https://huggingface.co/OpenAssistant](https://huggingface.co/OpenAssistant)`
6 `[https://github.com/FastEval/FastEval](https://github.com/FastEval/FastEval)`, `[https://tju01.github.io/ilm-eval/](https://tju01.github.io/ilm-eval/)`
7For readability, Table 1 contains aggregated LMEH scores. Details in the online leaderboard.

6.1 Instruction Tuning & Preference Modeling


To understand the concordance between human and automated toxicity detection, we employ toxicity
detection methods based on Detoxify [28] to obtain automated ratings for six distinct categories,
classifying whether a message is toxic, obscene, threatening, insulting, attacking a certain identity or
sexually explicit. We limit our analysis to those languages that are supported by the toxicity detection
method, covering English, Spanish, Russian, French, and Italian. These languages represent the
majority of OASST1 messages (over 83%).
Using automated toxicity ratings, we are able
to systematically assess the correlation between
these ratings and human-assigned toxicity labels
(hate speech, not appropriate, and sexual content). Based on a sample of 115,153 messages,
we compute the correlation between automatic
and human-annotated toxicity labels, which is
visualized in Figure 3. We see a correlation between human and automatic labels in at least
one element of each row and column of the correlation matrix, suggesting agreement between
human annotators and off-the-shelf toxicity detection models. The results serve to both validate the capabilities and show limitations of
AI-driven toxicity detection in comparison to
human judgement and may inform future work
in this area.
Figure 3: Correlation between human labels and
Detoxify outputs for all messages in Detoxifysupported languages.
In addition to analysing the correlation between human-assigned toxicity labels and automated
ratings, we extend the application of the Detoxify model to assess the efficacy of the moderation
process for the same languages described earlier. To facilitate this analysis, we define two categories
of messages: _deleted_ messages, which encompass those that either failed to pass the community
moderation process or were subsequently manually removed by moderators, and _retained_ messages,
which successfully made it through to the dataset. In order to provide a comprehensive evaluation
of the moderation process, we calculated average values for each of the six Detoxify categories for
both _deleted_ and _retained_ messages. The values obtained for this analysis are based on a sample of
74,781 messages. We excluded messages in trees that were incomplete at the time of export, as these
messages may be subject to removal by the moderation process.
Our analysis, presented in Table 2 shows that the values for all six toxicity categories are markedly
higher for _deleted_ messages compared to _retained_ messages. This significant difference demonstrates
the effectiveness of the moderation processes in place, as messages removed from the dataset are on
average rated as significantly more toxic by the Detoxify model than messages allowed to remain in
the dataset.
Toxicity Obscene Threat Insult Identity Attack Explicit N
State
Deleted 4.625% 1.965% 0.411% 2.085% 0.651% 1.39% 3422
Retained 0.988% 0.574% 0.102% 0.715% 0.121% 0.177% 71359
Table 2: Detoxify outputs across six categories of toxicity, comparing _deleted_ and _retained_ messages.
While _deleted_ messages are rated as more toxic than _retained_ messages by the Detoxify model across
all categories, the average toxicity values for these messages are still small. This implies toxicity
ratings from models like Detoxify alone are not sufficient to determine when messages are unsuitable
for inclusion in the dataset. Reasons for deleting non-toxic messages may include a lack of factual
accuracy, or poor grammar. Additionally, messages which are children of deleted messages must
themselves be deleted even if they appear to be acceptable in isolation.


**Reward model data collection.** InstructGPT [13] trained reward models on ranking data of messages
generated by their initial SFT model, while our reward models are trained using ranking data of
human-generated messages. We chose to do so because we were already collecting this ranking data
as part of our general efforts, for use in quality control, spam filtering, and dataset sub-sampling.
While subjectively, many users report our RLHF models to follow instructions more closely (also
compare Section 6.1), the models do not deliver the same uniform and significant improvements over
SFT models as reported in [13]. We hypothesize that the difference in data collection for the reward
model could at least partially explain this gap. We plan to collect ranking data based on our own SFT
models in the future to verify these assumptions. Further research is necessary to determine more
precise criteria for collecting data useful to RLHF.
**Subjective, Cultural, and Contribution Frequency Biases.** The open nature of our project introduces a unique set of challenges when it comes to controlling for biases within the dataset. Annotators
from diverse backgrounds contribute to the dataset, with demographics that are simultaneously heterogeneous in some dimensions and homogeneous in others (see Section 5). Specifically, 89.1% of
the annotators identify as male, with a median age of 26. This demographic profile may inadvertently
introduce biases in the dataset, as it is bound to reflect the values, perspectives, and interests of
the annotators. (We pose that some of this could be mitigated by introducing more constrained
conversations, for example sampling a random Wikipedia page to determine the conversation topic.)
Further, users’ participation levels differ significantly. More engaged users contribute a greater
number of annotations (see Figure 2), which likely leads to over-representation of their values and
interests in the dataset. Consequently, the dataset may not adequately capture the diverse perspectives
that a more balanced distribution of contributions could have provided. Further research is necessary
to determine the effect of uneven contributor distributions have when given a clear, general task.
**Possibility of Unsafe Content.** While we have implemented measures to detect and remove harmful
messages, our system is not infallible. It is possible that the dataset still contains unsafe content. We
believe that the open nature of the project allows for data filtering to be conducted in a transparent
manner, ultimately converging on the highest possible standards. Nevertheless, the potential presence
of residual unsafe content in the dataset necessitates careful evaluation of any models trained on it.
Given the limitations discussed above, we advocate for the use of our models in academic research
contexts only. We strongly encourage researchers to thoroughly investigate the safety and bias of any
model before employing it in downstream tasks. The released models may exhibit unsafe behavior
and are likely susceptible to prompt injection attacks. The alignment of LLMs is a crucial aspect of
AI research, and we hope that our contributions can help advance the field of AI alignment. However,
we also acknowledge that current alignment techniques are not perfect and can even exacerbate
certain biases [29]. As such, researchers should exercise caution when using these models and be
cognizant of their limitations. Additionally, it is essential to continue refining alignment techniques
and advancing the field of AI alignment in order to mitigate these limitations and develop more
reliable and robust LLMs.


Large language models are prone to generating inaccurate information about people, places, or facts,
commonly known as ‘hallucinations’ [30, 31]. LLMs can also produce toxic or hateful content and
fail to follow provided constraints [32]. Additionally, these models tend to incorporate biases present
in their training data, leading to unfair and discriminatory outputs [33]. While methods such as
_RLHF_ can mitigate some of these shortcomings, they may exacerbate others [34, 29]. We hope that
alignment methods using OpenAssistant Conversations can fix some of these issues [13], but we
acknowledge that achieving full alignment is a complex and ongoing challenge.
We recognize that sufficiently powerful language models can have a significant impact on society [35],
and therefore we believe it is essential to promote transparency in their development and deployment.
OpenAssistant Conversations is our contribution to this goal of transparency.


We provide the guidelines presented to the users for the creation of the dataset.
```


Below is a list of guidelines that should be adhered to for each possible task
available when building the dataset. To see some examples of how the guidelines
can be applied, visit the examples document.
Please consider checking out our survey
[here](https://forms.gle/vBW7b2kMzjCoehkH9). You can use it to rate each
guideline and leave feedback for each task.
If you have further suggestions to improve any of our guidelines, or want to add
more examples, create a pull request or suggest them on our
[GitHub](https://github.com/LAION-AI/Open-Assistant).


- Always make sure to read and understand the guidelines to each task before
 fulfilling it.
- Try to follow the guidelines as closely as possible.
- If you are unsure whether a message violates a guidelines, contact us at our
 Discord.
```
```
- Use the thumbs-up/thumbs-down system to further mark messages that are of high
 or low quality.


- Remain polite and treat the user with respect, even when not given the same
 courtesy.
- Talk in a friendly and approachable manner, unless specifically requested
 otherwise.
- Present only information that has been verified by credible sources that can
 be backed up, unless specifically requested otherwise.
- Make sure the user is aware when given unverified information.
- Inform the user about the potential dangers when being asked for advice
 regarding a topic with high risk, such as medicine, law or chemistry.
- When being asked about a high-risk topic, make sure the user knows that as a
 language model, the assistant is susceptible to producing incorrect
 information, and that no actions should be taken regarding the assistant reply
 without the opinion of a professional.
- When being asked to give an opinion as the default persona of the assistant,
 make sure to bring up at least 2 common viewpoints and ensure that these
 aren’t expressed as the opinions of the assistant.
 - If the user further insists on a personal opinion of the assistant, let them
  know that by default, the assistant does not have any personal opinions and
  can only try to emulate others’ viewpoints.
- Ask for clarification if it’s unclear what the user is asking for.
- Use paragraphs and line breaks to make larger replies more readable.
- Make use of [Markdown syntax](https://www.markdownguide.org/basic-syntax) to
 better format lists, tables or blocks of code.
 - If you are using a codeblock to write code in a particular language, specify
  it to enable
  [syntax highlighting]
  (https://www.markdownguide.org/extended-syntax/#syntax-highlighting).
  You can find all supported abbreviations
  [here](https://github.com/jincheng9/markdown_supported_languages
  #heres-a-full-list-of-supported-languages).
- Be consistent in the style and tone of the assistant.


- Copy and paste text from other sources without editing. **This includes
 ChatGPT.**
- Supply text that violates the law of Germany, UK, USA, or your country of
 residence.
- Write content encouraging:
 - Violence
 - Violation of the rights of a third party
 - Pedophilia
- Provide the user with information that could be used for self-harm if there is
 plausible suspicion of intent to self-harm.
- Provide personal information of third parties that isn’t publicly available.
- Ask for personal information unless it is relevant to the issue and can’t be
 used to determine the identity of the user, such as country of residence or
 occupation. The user should be allowed to refuse to give up any information.
- Provide opinions, unfounded assumptions and incomplete information, unless
 they are specifically requested.
- Purposefully curate information to guide the conclusion, i.e. don’t hide facts
 to present a particular narrative.
```
```
- Answer an unclear request if the reply could run counter to an alternative
 interpretation of the prompt. Ask the user to elaborate or rephrase instead.
- Dodge a question, unless it violates a guideline.
- Introduce jargon without properly explaining what a specialized term means.
 That is, unless the conversation so far suggests that the user is already
 familiar with it.
- Leave typos or grammatical errors in the assistant replies, unless
 specifically requested to do so.
- Overload the user with too much information. Keep replies concise, but include
 further details that relate to and expand upon the user’s request.
- Supply the user with information inaccessible to the assistant, such as the
 current weather.
- Reply in a language different from the one intended for the dataset, unless
 specifically requested to do so.


- Ask questions that reflect real-life situations and needs.
- Ask questions that might be directed towards search engines or specialists.
- Make requests that encourage lateral thinking and/or require specialized
 knowledge.
- Use a mix between questions that are straightforward and questions without a
 clear answer.
- Introduce a variety in prompts by using different phrasing, degrees of
 politeness or amount of context given.
- Consider the previous replies and prompts that lead up to the current one.
- Try to build upon the topic and ask a sensible follow-up question when
 replying to the assistant.


- Write prompts without a clear request.
- Supply text that violates the law of Germany, UK, USA, or your country of
 residence.
- Make requests that override the original purpose of the assistant, i.e.
 jailbreak the model.
- Make requests that leave the assistant with no other choice but to refuse in
 order to avoid the generation of harmful content.
- Submit a prompt similar or identical to a prompt you previously submitted.
- Change the topic of a conversation without prefacing it accordingly when
 replying to the assistant.
- Leave typos and grammatical errors in the prompt.
- Reply in a language different from the one intended for the dataset, unless
 the context of the conversation requires it.


- Rate every criteria of each reply, unless it can’t be discerned because it is
 spam or inappropriate.
- Judge quality based on how well the reply adheres to the guidelines. Factual
 accuracy and helpfulness are first and foremost.
- Make sure to read the reply thoroughly.
- Use the [label explanations](#label-explanation) to determine which labels
 apply to the reply.
- Research to make sure whether the reply is factually accurate.
```
```
- Skip a classification if you are unable to determine the validity of reply.


- Judge quality based on personal beliefs. Assuming an opinion was warranted,
 fulfills the users request and doesn’t violate any guidelines, it should not
 impact the rating of the reply.
- Skip a label just because the reply is spam. Each label can help the model
 improve.
- Rate a reply if you are unsure if it factually accurate or satisfies the
 request of the user.


- Rate every criteria of each prompt, unless it can’t be discerned because it is
 spam or inappropriate.
- Judge quality based on how well the prompt adheres to the guidelines.
- Make sure to read the prompt thoroughly.
- Use the [label explanations](#label-explanation) to determine which labels
 apply to the prompt.


- Judge quality based on personal beliefs. The opinion of the user should not
 impact the rating of the prompt.
- Skip a label just because the reply is spam. Each label can help the model
 improve.


- Make sure to read every available reply.
- Think about which reply best satisfies the request of the user.
- Rank replies based on how well they adhere to the guidelines. Factual accuracy
 and helpfulness are first and foremost.
- Penalize replies that fail to provide adequate warnings or caveats.
- Penalize replies that are difficult to read due to a lack of formatting,
 capitalization or other errors.
- Penalize replies if the requested information is obfuscated by superfluous
 details that make up a large part of the message.
- Rank replies that admit to not knowing the answer below factually correct, but
 above factually incorrect replies.


- Rank replies based on personal beliefs. Assuming an opinion was warranted,
 fulfills the users request and doesn’t violate any guidelines, it should not
 impact the rating of the reply.
- Rank replies based on how long and short they are - instead, find out which
 reply best answers the query of the user.


Is the message an unrelated piece of text that doesn’t contribute to the
```
```
previous prompt in any way? If it’s an initial prompt, is it something unlikely
to be used to initiate a conversation with the assistant?


Does the assistant reply fail to engage the request of the user or does it
contain incorrect information? Does it ignore relevant parts of the prompt?

Is it a bad reply, as an answer to the prompt task?


**Not Target Language**: Unless specifically requested, is the language used
different from the target language of the dataset?
**Not Appropriate**: In case of a user prompt, is it missing a clear request?
**Contains PII**: Does the message contain personally identifying information
(PII) that isn’t publicly available and can be used to determine the identity of
the user or a private third party?
**Hate Speech**: Does the message contain prejudice towards a group classified
as protected according to the law of Germany, UK, USA or your country of
residence?
**Sexual Content**: Does the message contain or request sexually explicit
content, such as descriptions of sexual acts?


**Low Quality/High Quality**: How well does the message adhere to the
guidelines? For assistant replies, factual accuracy and helpfulness are first
and foremost.
**Serious/Humorous**: Does the message contain sarcasm, a play on words or other
humorous embellishments?
**Ordinary/Creative**: How unlikely is the question to be asked? Is it
challenging the assistant with a problem that requires creative thinking? Does
the question include lots of context? On the assistant’s side, is the reply
straightforward, or does it contain a unique approach to a problem?
**Rude/Polite**: How polite is the message towards the recipient? Does it use
words like "please" or is it dismissive, rude or insulting towards the other
side of the conversation?
**Violent/Harmless**: Does the message contain descriptions of explicit violent
acts? Is it glorifying, encouraging or downplaying violent behavior?
```


Reinforcement learning from human feedback [36, 37] comprises a set of techniques that all aim
to optimize the output distribution of a language model using the preference structure provided by
human rankers. To get a preference structure that is well aligned to users, we cannot just rely on the
opinions of individual rankers, due to the high variance in human preferences. Since our objective
is to collect data for a generally capable digital assistant, every ranking of possible responses is
performed by K independent rankers, (see, section 3.1). Once this is done, we need to fuse these K
individual opinions into one consensus opinion usable in training preference models.
We perform this preference fusion by treating it as a ranked-voting problem, whose objective it
is to maintain the preferences as faithfully as possible. The method chosen for this is known as
"ranked pairs" or "Tideman’s method" [38]. Simplified, this method creates a sorted list of "winners"
according to the strength of the preference of one element over the others. The way the preference
strength is measured is by considering all preference pairs in the input votes: For example, if the
votes are two times _A > B > C_ and one time _B > A > C_, this would mean that the pair _A > B_
exists 2 times, while _A > C_ exists three times. The method then orders the winners by winning
strength, i.e. here _A > C_ would happen before _A > B_, and constructs a directed graph using the
preferences, i.e. _A > C_ would become an edge _A →_ _C_ . Edges are added one by one according to
their weight (higher first), as long as no cycle is produced. If the edge would produce a cycle, it is
skipped as the corresponding dominant preferences are already in the graph.
The resulting directed acyclic graph can be turned into a preference structure by recursively removing
source nodes and adding them to the back of a preference list, since source nodes are, by construction,
not preferred over any other item left in the graph. In practice, one can speed up the construction by
not explicitly constructing the preference graph and fusing the graph con- and destruction.


Figures 4, 5, and 6 show the data collection platform UI interface for labelling, reply input, and
ranking, respectively. The UI also allows the user to choose a random task type, the task is automatically chosen according the previously discussed heuristics, which allows for more variation and
engagement.
The UI did grow and change over time as the project developed, so the experience of early users was
less refined as shown here.
Figure 4: A preview of the page for labelling tasks. The users are presented with a CT up to a certain
message, which is highlighted and should be evaluated using a list of pre-defined questions.
Figure 5: A preview of the page for ranking assistant replies. Users are provided with CT up to a
message, and a couple of responses that should be ranked according to how good they answer the
given message. The interaction is drag & drop based. Additionally, users can choose to mark all
answers as factually incorrect.
Figure 6: A preview of the page for replying as an assistant. Users are provided with a CT up to a
prompter message, and they should provide a response to it. In this example, the CT contains only
one message. Users can use Markdown in their responses. An additional progress bar is added below
the input field to incentivize longer messages.


The widespread use of ChatGPT at the time of our data collection meant that some of the inputs
could be performed using ChatGPT, rather than being fully created by humans. Our guidelines
specifically target this, explicitly discouraging contributors from copy-pasting responses generated
by other AI models. Users who were found to post ChatGPT-generated responses were banned,
and their contributions were deleted. Furthermore, we used multiple automatic tests to catch such
cases. For instance, we searched for and removed messages that contained text such as "as a large
language model" or "knowledge cutoff after September 2021". Moreover, users were encouraged
to up-vote and down-vote responses they came across from other users, which also helped weeding
out low-quality, generic, (possibly AI generated) responses. We note that these mechanisms cannot
remove all ChatGPT-generated content, and more post-hoc filtering may be necessary.


We asked users to provide feedback based on the overall experience, while taking part in the data
collection process. Results are provided in Tables 3, 4 and Fig. 7, 8, 9, 10.
Figure 7: Demography of 270 respondents
It was clear from the website what to do and how to contribute messages to the training data.
Strongly agree 33.70%
Agree 47.04%
Neither agree nor disagree 11.48%
Disagree 6.67%
Strongly disagree 1.11%
I’ve felt I could always ask for help from the community and the moderators.
Strongly agree 30.74%
Agree 27.78%
Neither agree nor disagree 31.85%
Disagree 7.41%
Strongly disagree 2.22%
I found the tasks enjoyable and engaging.
Strongly agree 20.00%
Agree 42.22%
Neither agree nor disagree 27.78%
Disagree 8.15%
Strongly disagree 1.85%
I found the tasks repetitive.
Strongly agree 11.48%
Agree 30.00%
Neither agree nor disagree 37.04%
Disagree 18.15%
Strongly disagree 3.33%
While doing rating or ranking tasks, I found the messages to be really high quality.
Strongly agree 16.67%
Agree 42.59%
Neither agree nor disagree 30.00%
Disagree 9.26%
Strongly disagree 1.48%
Overall, I’m glad I have contributed to OpenAssistant.
Strongly agree 81.11%
Agree 14.44%
Neither agree nor disagree 2.96%
Disagree 0.74%
Strongly disagree 0.74%
Table 3: User Satisfaction Survey
Have you contributed to other community projects besides this one?
No, this is my first time contributing 111
Yes I have contributed to a few projects 110
Yes, I have contributed to multiple open source projects 44
Prefer not to say 5
Table 4: Previous Contributions
Figure 8: Familiarity with AI based on level of education. The average values are shown as outlines
in the bar chart.
Figure 9: Motivations for contribution based on level of education. The average values are shown as
outlines in the bar chart.
Figure 10: Planned personal use-cases for OpenAssistant. The average values are shown as outlines
in the bar chart.


We present some statistics on the words used the most in Fig. 11.
Figure 11: Word clouds for 33 topics extracted from the English subset of the open-assistant dataset.
The number of topics was selected by selecting the peak coherence from 40 LDA models. One can
observe high topic variety from biology (second top row), cooking (top row), and music (second to
last row), all the way up to quantum physics (bottom row).


We provide more information on the parameters used to collect new data samples.
Parameter value
max active trees 100
max initial prompt review 100
max tree depth 5
max children count 2
num prompter replies 1
goal tree size 9
num reviews initial prompt 3
num reviews reply 3
auto mod enabled true
auto mod max skip reply 25
auto mod red flags 4
p full labeling review prompt 1
p full labeling review reply assistant 1
p full labeling review reply prompter 0.1
acceptance threshold initial prompt 0.6
acceptance threshold reply 0.6
num required rankings 3
p activate backlog tree 0.1
min active rankings per lang 20
lonely children count 2
p lonely child extension 0.75
recent tasks span sec 300
max pending tasks per user 8
max prompt lottery waiting 1000
Table 5: Collection parameters
_max active trees_ : Maximum number of concurrently active message trees in the database. No new
initial prompt tasks are handed out to users if this number is reached
_max initial prompt review_ : Maximum number of initial prompts under review before no more initial
prompt tasks will be handed out.
_max tree depth_ : Maximum depth of message tree.
_max children count_ : Maximum number of reply messages per tree node.
_num prompter replies_ : Number of prompter replies to collect per assistant reply.
_goal tree size_ : Total number of messages to gather per tree.
_num reviews initial prompt_ : Number of peer-review checks to collect in the ‘INITIAL_PROMPT_REVIEW‘ state
_num reviews reply_ : Number of peer review checks to collect per reply (other than initial prompt).
_auto mod enabled_ : Flag to enable/disable auto moderation.
_auto mod max skip reply_ : Automatically set tree state to ‘halted_by_moderator‘ when more than the
specified number of users skip replying to a message. (auto moderation)
_auto mod red flags_ : Delete messages that receive more than this number of red flags if it is a reply or
set the tree to ‘aborted_low_grade‘ when a prompt is flagged. (auto moderation)
_p full labeling review prompt_ : Probability of full text-labeling (instead of mandatory only) for initial
prompts.
_p full labeling review reply assistant_ : Probability of full text-labeling (instead of mandatory only) for
assistant replies.
_p full labeling review reply prompter_ : Probability of full text-labeling (instead of mandatory only) for
prompter replies.
_acceptance threshold initial prompt_ : Threshold for accepting an initial prompt.
_acceptance threshold reply_ : Threshold for accepting a reply.
_num required rankings_ : Number of rankings in which the message participated.
_p activate backlog tree_ : Probability to activate a message tree in BACKLOG_RANKING state when
another tree enters a terminal state.
_min active rankings per lang_ : When the number of active ranking tasks is below this value when a
tree enters a terminal state an available trees in BACKLOG_RANKING will be activated (i.e. enters
the RANKING state).
_lonely children count_ : Number of children below which parents are preferred during sampling for
reply tasks.
_recent tasks span sec_ : Time in seconds of recent tasks to consider for exclusion during task selection.
_max pending tasks per user_ : Maximum number of pending tasks (neither canceled nor completed) by
a single user within the time span defined by ‘recent_tasks_span_sec‘.
_max prompt lottery waiting_ : Maximum number of prompts in prompt_lottery_waiting state per
language. If this value is exceeded no new initial prompt tasks for that language are generated.


Following [13] and as introduced in Section 1, we train supervised fine-tuned models ( _SFT_ ), reward
models (RM), and a PPO fine-tuned models based on RM’s predictions. We use as base models the
popular decoder-only Pythia [3] and LLaMA [2].
**Conversation format** We sample threads in the CTs and provide them as input text to the model by
using some additional special tokens. More specifically, a thread composed of prompts (P) P1 _,_ P2 _, . . ._
and replies (R) R1 _,_ R2 _, . . ._ is provided as input to the model with the following format:
<prompter_token> P1 <endoftext_token> <assistant_token> R1 <endoftext_token>
<prompter_token> P2 <endoftext_token> <assistant_token> R2 <endoftext_token>
_. . ._
Each of the prompts and the replies consists potentially of multiple tokens after tokenizing.
_**SFT-mix**_ **details.** The _sft-mix_ training data configuration used for training _OpenAssistant/falcon-_
_40b-sft-mix-1226_ is a mixture of OASST1 with other instruction tuning datasets, according to the
following configuration:
sft9−stage2:


use_custom_sampler: true
datasets:
−oasst_export:
lang: "bg,ca,cs,da,de,en,es,fr,hr,hu,it,nl,pl,pt,ro,ru,sl,sr,sv,uk" # sft−8.0
input_file_path: 2023−06−02_oasst_all_labels.jsonl.gz
val_split: 0.05
top_k: 2
−vicuna:
fraction: 0.5
val_split: 0.025
max_val_set: 250
−code_alpaca:
fraction: 0.5
val_split: 0.05
max_val_set: 250
−oa_wiki_qa_bart_10000row:
val_split: 0.05
max_val_set: 250
−grade_school_math_instructions:
val_split: 0.05
−dolly15k:
val_split: 0.05
max_val_set: 300
More details can be found on the Hugging Face hub and in our open-source codebase.
**Supervised fine-tuning.** During this phase, we fine-tune pretrained models for the regular language
modelling tasks based on our conversational data. We mask tokens that correspond to prompts and
only train to predict tokens that correspond to assistant replies.
**Reward model.** For the reward model training, we replace the language modelling head with a
linear layer producing a single output _rθ_, corresponding to the predicted score for the last reply of
the conversation. We use replies to the same prompt and their rankings as described in Appendix B.
Following [13], assuming _K_ distinct replies, we produce - _K_ 2 - comparisons and train to minimize the
loss
loss( _θ_ ) = _−_ ~~�~~ _K_ [1] ~~�~~ _E_ ( _x,yw,yl_ )[log( _σ_ ( _rθ_ ( _x, yw_ ) _−_ _rθ_ ( _x, yl_ )))] _,_
where _σ_ is the sigmoid function and _yw_ corresponds to a preferred completion for the pair of _yw_ and
_yr_ . We also optionally add another regularization parameter that prevents the predicted values from
diverging too much. Performance is measured by measuring the ability to predict the better reply
among pairs of replies with different rank, on a held-out validation set.
**PPO training.** We fine-tune the _SFT_ model by producing assistant replies to unanswered questions.
We use the RM to score these replies and train with PPO, using the trlx framework [9] . Following [13],
we also add a per-token KL penalty from the _SFT_ model at each token to avoid instability and
over-optimization to the RM model.
All details and current training parameters are publicly available under `[https://github.com/](https://github.com/LAION-AI/Open-Assistant/tree/main/model/model_training)`
`[LAION-AI/Open-Assistant/tree/main/model/model_training](https://github.com/LAION-AI/Open-Assistant/tree/main/model/model_training)` .