Recall the optimal language policy is
_π_ _[∗]_ ( _y|x_ ) = _µ_ ( _y|x_ ) _[e][r]_ [(] _[x,y]_ [)] _[/α]_ _._ (Eq. 2)
_Z_ ( _x_ )
Consider a response _y_ randomly sampled from either the optimal LM _π_ _[∗]_ ( _y|x_ ) = _µ_ ( _y|x_ ) _[e][r]_ [(] _Z_ _[x,y]_ ( _x_ [)] ) _[/α]_, or
the pretrained LM _µ_ ( _y|x_ ). Let a binary variable _ν_ = 1 indicates the response _y_ is sampled from _π_ _[∗]_ .
Then the marginal distribution of _y_ is
_p_ [joint] ( _y|x_ ) := _p_ ( _ν_ = 0) _µ_ ( _y|x_ ) + _p_ ( _ν_ = 1) _π_ _[∗]_ ( _y|x_ ) _._
Given the prior _p_ ( _ν_ = 0) = _p_ ( _ν_ = 1) = [1] 2 [, using Bayes’ Rule, the data posterior satisfies]
_µ_ ( _y|x_ )
_p_ [joint] ( _ν_ = 0 _|x, y_ ) =
_µ_ ( _y|x_ ) + _π_ _[∗]_ ( _y|x_ ) _[.]_
_π_ _[∗]_ ( _y|x_ )
_p_ [joint] ( _ν_ = 1 _|x, y_ ) =
_µ_ ( _y|x_ ) + _π_ _[∗]_ ( _y|x_ ) _[.]_
Define model policy as _πθ_ ( _y|x_ ) := _µ_ ( _y|x_ ) _e_ _[r][θ]_ [(] _[x,y]_ [)] . The model posterior probability satisfies
_p_ [joint] _θ_ ( _ν_ = 1 _|x, y_ ) = _σ_ ( _rθ_ ( _x, y_ )) _._
_p_ [joint] _θ_ ( _ν_ = 0 _|x, y_ ) = 1 _−_ _σ_ ( _rθ_ ( _x, y_ )) = _σ_ ( _−rθ_ ( _x, y_ )) _._ (Eq. 8)
**Theorem A.2** (NCA Objective) **.** _For any α >_ 0 _, we have the following results._
_**(a) Equivalent objective.**_
min E _pjoint_ ( _x,y_ ) _D_ KL[ _p_ _[joint]_ ( _ν|x, y_ ) _||p_ _[joint]_ _θ_ ( _ν|x, y_ )]
_θ_
_e_ _[r]_ [(] _[x,y]_ [)] _[/α]_
_⇐⇒_ min _−_ E _p_ ( _x_ ) _µ_ ( _y|x_ ) log _σ_ ( _rθ_ ( _x, y_ )) + log _σ_ ( _−rθ_ ( _x, y_ )) _,_ (Eq. 9)
_θ_ _Z_ ( _x_ )
_where Z_ ( _x_ ) = E _µ_ ( _y|x_ ) _e_ _[r]_ [(] _[x,y]_ [)] _[/α]_ _._
_**(b) Optimal solution.**_ _Assume unlimited model capacity and data samples. The optimal rθ∗_ _and πθ∗_
_for solving Eq. 5 are_
_rθ∗_ ( _x, y_ ) = _r_ ( _x, y_ ) _/α −_ log E _µ_ ( _y|x_ ) _e_ _[r]_ [(] _[x,y]_ [)] _[/α]_ _,_
_and_ _πθ∗_ ( _x, y_ ) _∝_ _µ_ ( _y|x_ ) _e_ _[r]_ [(] _[x,y]_ [)] _[/α]_ _._
_Proof._ **(a) Equivalent objective.**
min E _p_ joint( _x,y_ ) _D_ KL[ _p_ [joint] ( _ν|x, y_ ) _||p_ [joint] _θ_ ( _ν|x, y_ )]
_θ_
_⇔_ min E _p_ joint( _x,y_ )E _p_ joint( _ν|x,y_ ) log _[p]_ [joint][(] _[ν][|][x, y]_ [)]
_θ_ _p_ [joint] _θ_ ( _ν|x, y_ )
_⇔_ min _−_ E _p_ joint( _x,y_ )E _p_ joint( _ν|x,y_ ) log _p_ [joint] _θ_ ( _ν|x, y_ )
_θ_
_⇔_ min _−_ E _p_ ( _x_ ) _p_ ( _ν_ ) _p_ joint( _y|x,ν_ ) log _p_ [joint] _θ_ ( _ν|x, y_ ) (Bayes’ rule)
_θ_
_⇔_ min _−_ [ _p_ ( _ν_ = 0)E _p_ ( _x_ ) _p_ joint( _y|x,ν_ =0) log _p_ [joint] _θ_ ( _ν_ = 0 _|x, y_ )+
_θ_
_p_ ( _ν_ = 1)E _p_ ( _x_ ) _p_ joint( _y|x,ν_ =1) log _p_ [joint] _θ_ ( _ν_ = 1 _|x, y_ )]
_⇔_ min _−_ �E _p_ ( _x_ ) _µ_ ( _y|x_ ) log _σ_ ( _−rθ_ ( _x, y_ )) + E _p_ ( _x_ ) _π∗_ ( _y|x_ ) log _σ_ ( _rθ_ ( _x, y_ ))� (by Eq. 8)
_θ_
_e_ _[r]_ [(] _[x,y]_ [)] _[/α]_
_⇔_ min _−_ E _p_ ( _x_ ) _µ_ ( _y|x_ ) log _σ_ ( _rθ_ ( _x, y_ )) + log _σ_ ( _−rθ_ ( _x, y_ )) (importance sampling)
_θ_ _Z_ ( _x_ )
**(b) Optimal solution.**
Given conclusions from (a). With unlimited model capacity, _p_ [joint] _θ_ ( _ν|x, y_ ) could represent any discrete
distribution, such that we can arrive at the global optimal point given infinite training data.
E _p_ joint( _x,y_ ) _D_ KL[ _p_ [joint] ( _ν|x, y_ ) _||p_ [joint] _θ_ _[∗]_ [(] _[ν][|][x, y]_ [)] = 0]
= _⇒_ _p_ [joint] ( _ν|x, y_ ) = _p_ [joint] _θ_ _[∗]_ [(] _[ν][|][x, y]_ [)] _∀x, ν, y_
= _⇒_ _p_ [joint] ( _ν_ = 1 _|x, y_ ) = _p_ [joint] _θ_ _[∗]_ [(] _[ν]_ [ = 1] _[|][x, y]_ [)] _∀x, y_
_π_ _[∗]_ ( _y|x_ ) _e_ _[r]_ _θ_ _[∗]_ [(] _[x,y]_ [)]
= _⇒_ _θ_ [(] _[x, y]_ [)) =] _∀x, y_
_µ_ ( _y|x_ ) + _π_ _[∗]_ ( _y|x_ ) [=] _[ σ]_ [(] _[r][∗]_ 1 + _e_ _[r]_ _θ_ _[∗]_ [(] _[x,y]_ [)]
= _⇒_ _e_ _[r]_ _θ_ _[∗]_ [(] _[x,y]_ [)] = _[π][∗]_ [(] _[y][|][x]_ [)] _∀x, y_
_µ_ ( _y|x_ )
= _⇒_ _rθ∗_ ( _x, y_ ) = _r_ ( _x, y_ ) _/α −_ log E _µ_ ( _y|x_ ) _e_ _[r]_ [(] _[x,y]_ [)] _[/α]_ _∀x, y_
= _⇒_ _πθ∗_ ( _x, y_ ) _∝_ _µ_ ( _y|x_ ) _e_ _[r]_ [(] _[x,y]_ [)] _[/α]_ _∀x, y_


PyTorch code for the InfoNCA/NCA loss for reward datasets is provided below:
```
import torch.nn.functional as F
def reward_loss(pi_logps, ref_logps, rewards, alpha, beta, loss_type):
  """
  pi_logps: policy logprobs for K responses, shape (B, K)
  ref_logps: reference logprobs for K responses, shape (B, K)
  rewards: reward labels for K responses, shape (B, K)
  alpha: the reward temperature controlling strength of KL penalty
  beta: the parameterization coefficient that defines the reward model
  loss_type: could be either "InfoNCA" or "NCA" loss
  """
  soft_labels = (rewards / alpha).softmax(dim=-1) # (B, K)
  model_rewards = (pi_logps - ref_logps) * beta # (B, K)
  if loss_type == "InfoNCA":
    model_logps = model_rewards.log_softmax(dim=-1) # (B, K)
    losses = - (soft_labels * model_logps).sum(dim=-1) # (B,)
  elif loss_type == "NCA":
    optimization = - (soft_labels * F.logsigmoid(model_rewards)).sum(dim=-1) # (B,)
    regularization = - F.logsigmoid(-model_rewards).mean(dim=-1) # (B,)
    losses = optimization + regularization # (B,)
  return losses.mean()
```
The loss implementation under pairwise preference settings is equivalent to reward losses with _K_ = 2
and _α →_ 0. We provide the code separately for easy comparison with DPO.
```
def preference_loss(chosen_pi_logps, chosen_ref_logps,
           rejected_pi_logps, rejected_ref_logps,
           beta, loss_type):
  """
  chosen_pi_logps: policy logprobs for the preferred responses, shape (B, )
  chosen_ref_logps: reference logprobs for the preferred responses, shape (B, )
  rejected_pi_logps: policy logprobs for the dispreferred responses, shape (B, )
  rejected_ref_logps: reference logprobs for the dispreferred responses, shape (B, )
  beta: the parameterization coefficient that defines the reward model
  loss_type: one of "InfoNCA", "NCA" or "DPO" loss
  """
  chosen_rewards = (chosen_pi_logps - chosen_ref_logps) * beta # (B,)
  rejected_rewards = (rejected_pi_logps - rejected_ref_logps) * beta # (B,)
  if loss_type in ["DPO", "InfoNCA"]:
    losses = -F.logsigmoid(chosen_rewards - rejected_rewards) # (B,)
  elif loss_type == "NCA":
    losses = - F.logsigmoid(chosen_rewards) \
         - 0.5 * F.logsigmoid(-chosen_rewards) \
         - 0.5 * F.logsigmoid(-rejected_rewards) # (B,)
  return losses.mean()
```

Noise Contrastive Alignment of Language Models with Explicit Rewards


**Experiments with UltraFeedback.** Our implementation is heavily based on the Transformer
Reinforcement Learning (TRL) library [41] and Zephyr’s official code base [40]. All models are
fine-tuned from the publicly accessible HuggingFaceH4/mistral-7B-SFT-beta model. Experiments
are run on Nvidia A40 or RTX 4090 GPUs using bfloat16 precision. We ablate _β ∈{_ 3 _e −_ 4 _,_ 1 _e −_
3 _,_ 3 _e_ _−_ 3 _,_ 1 _e_ _−_ 2 _,_ 3 _e_ _−_ 2 _,_ 1 _e_ _−_ 1 _,_ 3 _e_ _−_ 1 _,_ 1 _._ 0 _}_ and _α ∈{_ 0 _._ 01 _,_ 0 _._ 1 _,_ 0 _._ 33 _,_ 1 _._ 0 _,_ 3 _._ 33 _}_ . The default reward
temperature _α_ is 0.01. The default parameterization coefficient _β_ is also 0.01. We adopt the QLoRA
[10] fine-tuning technique with rank 16, _α_ lora = 16, and a dropout rate of 0.05. We train all models
for 1 epoch. The batch size is 32. We use an AdamW optimizer with a learning rating of 5e-6. For
KTO and IPO baselines, we adopt exactly the same training pipeline for reporting their performance
except that we tune the _β ∈{_ 0 _._ 01 _,_ 0 _._ 1 _,_ 0 _._ 3 _,_ 0 _._ 5 _,_ 1 _._ 0 _}_ . We find the most suitable _beta_ for KTO is 0.01,
and for IPO is 0.5.
**Experiments with UltraInteract.** We follow [44] and fine-tune all parameters of the pretrained
model with UltraInteract and UltraFeedback. Specifically, for HuggingFaceH4/mistral-7B-SFT-beta
model, we directly adopt the recipe of Eurus, with the same data mixture (all 220K multi-turn
trajectory pairs in UltraInteract and all 340K pairs in UltraFeedback) and hyperparameters. All _β_ is
set to 0.1. The model is trained for 1 epoch with a cosine schedule. The learning rate is 5e-7 and the
warmup ratio is 0.1. For Mixtral-8x7B, we first SFT the model with the recipe for Eurus-70B-SFT,
and then applied the same preference learning recipe as aforementioned.
Figure 7: Rating system screenshot. Our rating system is adapted from [47].


|1.0 3e-1 1e-1 3e-2 1e-2 3e-3 1e-3 3e-4|Col2|Col3|Col4|
|---|---|---|---|
|Reparameterization Coefficient
3.33
1.0
0.33
0.1
0.01
6.43
6.47
6.60
6.91
7.05
7.04
7.16
7.30
6.35
6.68
6.85
7.01
7.27
7.20
7.04
7.1
6.32
6.72
6.98
7.26
7.32
7.18
7.45
7.15
6.44
6.68
6.88
7.20
7.30
7.16
7.10
7.14
6.60
6.65
6.92
7.04
7.42
7.29
7.16
6.96|Reparameterization Coefficient
3.33
1.0
0.33
0.1
0.01
6.43
6.47
6.60
6.91
7.05
7.04
7.16
7.30
6.35
6.68
6.85
7.01
7.27
7.20
7.04
7.1
6.32
6.72
6.98
7.26
7.32
7.18
7.45
7.15
6.44
6.68
6.88
7.20
7.30
7.16
7.10
7.14
6.60
6.65
6.92
7.04
7.42
7.29
7.16
6.96|Reparameterization Coefficient
3.33
1.0
0.33
0.1
0.01
6.43
6.47
6.60
6.91
7.05
7.04
7.16
7.30
6.35
6.68
6.85
7.01
7.27
7.20
7.04
7.1
6.32
6.72
6.98
7.26
7.32
7.18
7.45
7.15
6.44
6.68
6.88
7.20
7.30
7.16
7.10
7.14
6.60
6.65
6.92
7.04
7.42
7.29
7.16
6.96|Reparameterization Coefficient
3.33
1.0
0.33
0.1
0.01
6.43
6.47
6.60
6.91
7.05
7.04
7.16
7.30
6.35
6.68
6.85
7.01
7.27
7.20
7.04
7.1
6.32
6.72
6.98
7.26
7.32
7.18
7.45
7.15
6.44
6.68
6.88
7.20
7.30
7.16
7.10
7.14
6.60
6.65
6.92
7.04
7.42
7.29
7.16
6.96|
|Reparameterization Coefficient
3.33
1.0
0.33
0.1
0.01
6.43
6.47
6.60
6.91
7.05
7.04
7.16
7.30
6.35
6.68
6.85
7.01
7.27
7.20
7.04
7.1
6.32
6.72
6.98
7.26
7.32
7.18
7.45
7.15
6.44
6.68
6.88
7.20
7.30
7.16
7.10
7.14
6.60
6.65
6.92
7.04
7.42
7.29
7.16
6.96|6.43
6.47
6.60
6.91
7.05
7.04
7.16
7.30
6.35
6.68
6.85
7.01
7.27
7.20
7.04
7.1
6.32
6.72
6.98
7.26
7.32
7.18
7.45
7.15
6.44
6.68
6.88
7.20
7.30
7.16
7.10
7.14
6.60
6.65
6.92
7.04
7.42
7.29
7.16
6.96|6.43
6.47
6.60
6.91
7.05
7.04
7.16
7.30
6.35
6.68
6.85
7.01
7.27
7.20
7.04
7.1
6.32
6.72
6.98
7.26
7.32
7.18
7.45
7.15
6.44
6.68
6.88
7.20
7.30
7.16
7.10
7.14
6.60
6.65
6.92
7.04
7.42
7.29
7.16
6.96|6.43
6.47
6.60
6.91
7.05
7.04
7.16
7.30
6.35
6.68
6.85
7.01
7.27
7.20
7.04
7.1
6.32
6.72
6.98
7.26
7.32
7.18
7.45
7.15
6.44
6.68
6.88
7.20
7.30
7.16
7.10
7.14
6.60
6.65
6.92
7.04
7.42
7.29
7.16
6.96|
|Reparameterization Coefficient
3.33
1.0
0.33
0.1
0.01
6.43
6.47
6.60
6.91
7.05
7.04
7.16
7.30
6.35
6.68
6.85
7.01
7.27
7.20
7.04
7.1
6.32
6.72
6.98
7.26
7.32
7.18
7.45
7.15
6.44
6.68
6.88
7.20
7.30
7.16
7.10
7.14
6.60
6.65
6.92
7.04
7.42
7.29
7.16
6.96|6.43
6.47
6.60
6.91
7.05
7.04
7.16
7.30
6.35
6.68
6.85
7.01
7.27
7.20
7.04
7.1
6.32
6.72
6.98
7.26
7.32
7.18
7.45
7.15
6.44
6.68
6.88
7.20
7.30
7.16
7.10
7.14
6.60
6.65
6.92
7.04
7.42
7.29
7.16
6.96|ation Coefficient||
Figure 8: MT-bench score for **InfoNCA (left)** and **NCA (right)** regarding various _α_ and _β_ . Results
are averaged for _K_ = 2 and _K_ = 4 experiments. Overall, model performance is relatively more
sensitive to variations of _β_ compared with _α_ . NCA shows greater tolerance to hyperparameter
variations.
Method Functional Form MT-bench KL Divergence
Mistral-7B-SFT _−_ 6.45 0.00
+DPO _rθ_ = _β_ log _[π][θ]_ 7.34 0.53
+DPO _rθ_ = _β_ log _µ_ _[θ]_ 7.34 0.53
+InfoNCA _rθ_ = _β_ log _[π][θ]_ **7.63** **0.47**
+InfoNCA _rθ_ = _β_ log _µ_ _[θ]_ **7.63** **0.47**
_rθ_ = _β_ log _πθ_ 4.65 4.39
_rθ_ = _β_ log _πθ_ + SFT regularization 7.15 0.26
+NCA _rθ_ = _β_ log _[π][θ]_ **7.52** **0.39**
_rθ_ = _β_ log _µ_ _[θ]_ **7.52** **0.39**
_rθ_ = _β_ log _πθ_ 5.82 3.32
_rθ_ = _β_ log _πθ_ + SFT regularization 6.88 0.09
Table 4: Comparison of different functional forms of _rθ_ on UltraFeeback datasets.


We provide sampled responses of LM policies trained respectively with the DPO and NCA to compare
their data regularization feature. In general, generations from the NCA policy are more similar to the
preferred response in the dataset.

E Appendix F: Additional Empirical Results


You are given an answer. Create a question for the answer.
Grams to Ounces conversion 1 gram (g) is equal to 0.03527396195 ounces (oz).
**Preferred Response in Dataset** (r=8.0)
What is the conversion rate for grams to ounces for 1 gram?
**Dispreferred Response in Dataset** (r=7.0)
What is the relationship between grams and ounces in weight measurement?
**Reponse generated by** _πθ_ **[DPO]** (r=2.0)
How many ounces are equivalent to 200 grams according to the given conversion rate of 1 gram (g)
to 0.03527396195 ounces (oz)?
**Reponse generated by** _πθ_ **[NCA]** (r=10.0)
Question: What is the conversion rate from grams to ounces, and how many grams are one ounce?
Table 5: Sampled generations and the log-likelihood ratio of the NCA and the DPO/InfoNCA policy.


If "A busy crosswalk somewhere in asia." does that mean that "The crosswalk was blocked off by
police."?
Options:
 - yes; - it is not possible to tell ; - no
**Preferred Response in Dataset** (r=8.0)
It is not possible to tell.
Confidence: 95%
**Dispreferred Response in Dataset** (r=7.5)
Question:
If "A busy crosswalk somewhere in asia." does that mean that "The crosswalk was blocked off by
police."?
Answer:
It is not possible to tell.
Confidence: 90%
**Reponse generated by** _πθ_ **[DPO]** (r=3.0)
 - It is not possible to determine whether "The crosswalk was blocked off by police." in "A busy
crosswalk somewhere in Asia." To determine whether the crosswalk was blocked off by police, you
would need additional information or context.
**Reponse generated by** _πθ_ **[NCA]** (r=8.0)
It is not possible to tell.
Confidence: 95%
Table 6: Sampled generations and the log-likelihood ratio of the NCA and the DPO/InfoNCA policy.


In this task, you need to count the occurrences of the given word in the given sentence.
Sentence: ’a green double decker bus sitting on top of a parking lot’. Count the occurrences of the
word ’a’ in the given sentence.
**Preferred Response in Dataset** (r=9.0)
The word ’a’ appears 3 times in the given sentence.
Confidence: 100%
**Dispreferred Response in Dataset** (r=6.0)
The given sentence is "a green double decker bus sitting on top of a parking lot." To answer your
question about the occurrences of the word ’a’, I will count the number of times the word appears in
the sentence.
After counting the occurrences of the word ’a’ in the sentence, I found that it appears once. Therefore,
the answer to your question is that the word ’a’ appears once in the given sentence.
**Reponse generated by** _πθ_ **[DPO]** (r=4.0)
Of course, I’m here to help! There are 11 occurrences of the word ’a’ in the given sentence. Here’s
how you can count them: 1. ’a’ in ’a’ 2. ’a’ in ’green’ 3. ’a’ in ’double’ 4. ’a’ in ’decker’ 5. ’a’ in
’bus’ 6. ’a’ in ’sitting’ 7. ’a’ in ’on ...
Confidence: 90%
**Reponse generated by** _πθ_ **[NCA]** (r=8.0)
Sure! The word ’a’ appears 3 times in the given sentence.
Table 7: Sampled generations and the log-likelihood ratio of the NCA and the DPO/InfoNCA policy.


if _X_ [2] + 3 = 12. What is X?
**Preferred Response in Dataset** (r=9.0)
To find the value of X, we’ll first need to isolate _X_ [2] :
_X_ [2] + 3 = 12
Subtract 3 from both sides:
_X_ [2] = 9
Now, we can find the square root of both sides of the equation to solve for X:
_X_ = _±_ 3
So, X can be either 3 or -3.
**Dispreferred Response in Dataset** (r=4.0)
I’m sorry, I cannot solve this algebra equation. Can I assist you with anything else?
**Reponse generated by** _πθ_ **[DPO]** (r=7.0)
Sure, I’d love to help. By solving the equation, the value of X should be 3.
**Reponse generated by** _πθ_ **[NCA]** (r=9.0)
To find the value of _X_, we need to solve the equation _X_ [2] + 3 = 12.
First, subtract 3 from both sides:
_X_ [2] = 12 _−_ 3 _X_ [2] = 9
Next, take the square root of both sides:
_√_
_X_ = _±_ 9
So, _X_ can be either 3 or _−_ 3.
Table 8: Sampled generations and the log-likelihood ratio of the NCA and the DPO/InfoNCA policy.
**Appendix H: Comparing NCA with DPO on Data Regularization.**
Suppose there are only 4 possible responses _{ya, yb, yc, yd}_ . Each response has a reward _r_, and a
pretrained language model policy _µ_ :
                 - _ya_ _yb_ _yc_ _yd_
Likelihood _µ_ ( _y_ ) 40% 50% 5% 5%
Reward _r_ ( _y_ ) 10 3 7 0
Table 9: Initial probabilities and rewards
Given a preference dataset _D_ = _{ya > yb}_ ( _yc_ and _yd_ do not exist in the dataset), the loss functions
are defined as:
     _L_ [DPO] _θ_ = _−_ log _σ_ log _[π]_ _µ_ _[θ]_ ( [(] _y_ _[y]_ _a_ _[a]_ ) [)]
     _L_ [NCA] _θ_ = _−_ log _σ_ log _[π]_ _µ_ _[θ]_ ( [(] _y_ _[y]_ _a_ _[a]_ ) [)]
_[θ]_ [(] _[y][a]_ [)] _[π][θ]_ [(] _[y][b]_ [)]
_µ_ ( _ya_ ) _[−]_ [log] _µ_ ( _yb_ )
_µ_ ( _yb_ )
_µ_ ( _ya_ )
_µ_ ( _yb_ )
     _L_ [NCA] _θ_ = _−_ log _σ_ log _[π][θ]_ [(] _[y][a]_ [)]
_−_ [1]
[1] - _−_ log _[π][θ]_ [(] _[y][a]_ [)]
2 [log] _[ σ]_ _µ_ ( _ya_ )
_−_ [1]
[1] - _−_ log _[π][θ]_ [(] _[y][b]_ [)]
2 [log] _[ σ]_ _µ_ ( _yb_ )
After fine-tuning, there are several possibilities for _πθ_ :
- _πθ_ ( _ya_ ) _πθ_ ( _yb_ ) _πθ_ ( _yc_ ) _πθ_ ( _yd_ ) log _[π][θ]_ [(] _[y][a]_ [)]
_[θ]_ [(] _[y][a]_ [)] _[π][θ]_ [(] _[y][b]_ [)]
_µ_ ( _ya_ ) _[−]_ [log] _µ_ ( _yb_ )
- _πθ_ ( _ya_ ) _πθ_ ( _yb_ ) _πθ_ ( _yc_ ) _πθ_ ( _yd_ ) log _µ_ _[θ]_ ( _ya_ _[a]_ ) _[−]_ [log] _µ_ _[θ]_ ( _yb_ _[b]_ ) _r_ ¯
(1) 20% _↓_ 10% _↓↓_ 5% 65% _↑_ 0 _._ 916 _>_ 0 2.65 _↓_
(2) 20% _↓_ 10% _↓↓_ 65% _↑_ 5% 0 _._ 916 _>_ 0 6.85 _↑_
(3) **Wanted** 60% _↑_ 30% _↓_ 5% 5% 0 _._ 916 _>_ 0 7.25 _↑_
Table 10: Post fine-tuning probabilities and rewards
In scenarios (1) and (2), we can see that the likelihood for both _ya_ and _yb_ decreases. However, (1)
and (2) satisfy the DPO loss function because the likelihood for _yb_ decreases more, and the relative
likelihood margin between _ya_ and _yb_ becomes larger. In (1) and (2), the likelihood for either _yc_ or
_yd_ increases because _π_ ( _ya_ ) _↓_ + _π_ ( _yb_ ) _↓_ + _π_ ( _yc_ )? + _π_ ( _yd_ )? = 1. However, _yc_ and _yd_ are unreliable
because we do not know their quality (rewards). The LM policy could generalize to a low-quality
response like _yd_ (case (1)).
In contrast, the NCA effectively prevents the winning response likelihood _π_ ( _ya_ ) from decreasing,
because it mainly optimizes the absolute data likelihood instead of just caring about the relative
likelihood margin log _[π]_ _µ_ _[θ]_ ( [(] _y_ _[y]_ _a_ _[a]_ ) [)] _[−]_ [log] _[π]_ _µ_ _[θ]_ ( [(] _y_ _[y]_ _b_ _[b]_ ) [)] [. Thus, we say NCA is more likely to assign a larger]
likelihood to responses within the dataset.


We compare with a prior work SLiC-HF [45] in this section. SLiC is inspired by [46] and similarly
aims to calibrate sequence likelihood to align with human preferences. Given a preference data pair
_{x →_ _yw > yl}_, the loss function for SLiC is
_Lθ_ := max(0 _, δ −_ log _πθ_ ( _yw|x_ ) + log _πθ_ ( _yw|x_ )) _−_ _λ_ log _πθ_ ( _yw|x_ ) _,_
where _δ_ is a hyperparameter that controls the likelihood margin of data, and _λ_ controls the regularization weight of the loss. The main difference between our proposed method and the SLiC loss can be
summarized as follows:
    - **Theoritical framework.** SLiC is mainly adapted from the existing LM calibration methods
[46]. In contrast, our method is based on noise contrastive estimation methods [14, 24].
    - **Policy regularization.** The training process of SLiC is regularized by the additional SFT loss
controlled by _λ_ . In contrast, our proposed method is regularized through the parameterization
technique _rθ_ = _β_ log _[π]_ _µ_ _[θ]_ [controlled by] _[ β]_ [.]
    - **Learning target.** SLiC directly optimizes the policy model _πθ_, while our method directly
optimizes the residual model _rθ_ .

Appendix G: Comparision with Related Works


**Xianghe Pang** [* 1 3] **Shuo Tang** [* 1 3] **Rui Ye** [* 1 3] **Yuxin Xiong** [1] **Bolun Zhang** [1] **Yanfeng Wang** [2 1] **Siheng Chen** [1 2 3]

Self-Alignment of Large Language Models via Monopolylogue-based Social
  Scene Simulation

Xianghe Pang
Shuo Tang
Rui Ye
Yuxin Xiong
Bolun Zhang
Yanfeng Wang
Siheng Chen


Aligning large language models (LLMs) with human values is imperative to mitigate potential adverse effects resulting from their misuse. Drawing
from the sociological insight that acknowledging
all parties’ concerns is a key factor in shaping
human values, this paper proposes a novel direction to align LLMs by themselves: social scene
simulation. To achieve this, we present MATRIX,
a novel social scene simulator that emulates realistic scenes around a user’s input query, enabling
the LLM to take social consequences into account before responding. MATRIX serves as a
virtual rehearsal space, akin to a Monopolylogue,
where the LLM performs diverse roles related to
the query and practice by itself. To inject this
alignment, we fine-tune the LLM with MATRIXsimulated data, ensuring adherence to human values without compromising inference speed. We
theoretically show that the LLM with MATRIX
outperforms Constitutional AI under mild assumptions. Finally, extensive experiments validate that
our method outperforms over 10 baselines across
4 benchmarks. As evidenced by 875 user ratings,
our tuned 13B-size LLM exceeds GPT-4 in aligning with human values. See our project page at
[https://shuotang123.github.io/MATRIX.](https://shuotang123.github.io/MATRIX)


Recent years have witnessed the meteoric rise in success
of large language models (LLMs) (OpenAI, 2023; Touvron
et al., 2023). As these highly capable large language models
grow increasingly powerful, it is imperative for humanity to
ensure they are aligned with human values (Bengio et al.,
*Equal contribution 1Cooperative Medianet Innovation Center,
Shanghai Jiao Tong University, Shanghai, China [2] Shanghai AI
Laboratory, Shanghai, China [3] Multi-Agent Governance & Intelligence Crew (MAGIC), Shanghai, China. Correspondence to:
Siheng Chen _<_ sihengc@sjtu.edu.cn _>_ .
_Proceedings of the 41_ _[st]_ _International Conference on Machine_
_Learning_, Vienna, Austria. PMLR 235, 2024. Copyright 2024 by
the author(s).
2024; Kang et al., 2023; Hazell, 2023), preventing various
unpredictable negative consequences that could stem from
their misuse (Anwar et al., 2024; Yi et al., 2024; Barrett
et al., 2023). In response to this, a wide range of industrial
companies and academic institutions are dedicating significant efforts to expedite research focused on value alignment
in LLMs (Ouyang et al., 2022; Bai et al., 2022b).
Presently, methods for aligning LLMs typically depend
on external supervision, either from humans or other
LLMs (Ouyang et al., 2022; Lee et al., 2023). As one of
the most representative approaches, reinforcement learning
from human feedback (RLHF) (OpenAI, 2023; Touvron
et al., 2023) relies on massive human-annotated preference
data and a two-stage training process (i.e., reward model
and LLM training). This requires a significant investment
in both human labor and computational resources. For example, OpenAI spent over 6 months employing RLHF to
enhance GPT-4’s alignment and safety before its public release (OpenAI, 2023). Some other methods could work
without human supervision; however, they typically necessitate oversight from more advanced LLMs (Xu et al., 2024;
Liu et al., 2024). This reliance, particularly on paid commercial APIs, can lead to substantial costs. Additionally,
such an approach inherently limits the potential for reaching
the capabilities of superhuman models (Burns et al., 2023).
Without relying on extensive external resources, a new
paradigm emerges where LLMs can achieve value alignment by themselves, termed self-alignment (Sun et al., 2023;
Li et al., 2024). Current methods typically rely on humandefined rules to promote appropriate responses. For example, (Askell et al., 2021; Yang et al., 2024; Sun et al., 2024)
design tailored prompts to direct LLMs towards being harmless; (Sun et al., 2023; Lin et al., 2024) focus on in-context
learning, providing some examples that are aligned with social norms; and (Bai et al., 2022b) applies self-critique and
revision guided by constitutional rules. However, all these
methods are notably constrained by rigid, pre-determined
human rules, which lack the necessary flexibility to adapt to
the diverse and complex nature of real-world user instructions. This limitation hinders the universal applicability of
such rule-based alignment methods in all scenarios.
In this paper, we explore a brand-new direction to achieve
**Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation**
self-alignment of LLMs: social scene simulation. This
approach stems from the sociological understanding that
human values are shaped through thoroughly considering
all involved participants’ concerns and the consequences
of actions (Hall, 2007; McKinley, 2015; Soc, 2024; Sym,
2024). For example, the perspective on nuclear technology
would involve evaluating the potential impact on energy
supply, environmental sustainability, and global security.
Inspired by this concept of social intelligence, our core idea
is allowing the LLM to self-create a simulation environment
that mirrors real-world multi-party interactions and simulates the social consequences for a user’s instruction. This
process allows the LLM to take these consequences into
account before responding, therefore self-aligning itself to
behave in a socially-aware manner.
Following this spirit, we propose MATRIX, a social scene
simulator that allows the LLM to emulate realistic multiparty interactions and consequences in response to users’
instructions. To facilitate simulation, MATRIX leverages
the LLM’s role-playing ability, akin to a Monopolylogue,
where a single performer embodies several characters. MATRIX generates two types of roles for a user’s instruction:
living agents, each with unique personalities, and non-living
objects. Every role, driven by the same LLM, delivers behavior descriptions that represent the ego interests and concerns.
To maintain the order of interactions in MATRIX, we design
a social modulator, also powered by the LLM, which integrates world rules and governs communication protocols for
all roles. These designs allow the social scenes in MATRIX
to progress natually until their conclusion. Throughout the
simulation, the social scenes in MATRIX show a series of
consequences triggered by the given instruction. Concurrently, the social modulator records the textual interactions
of the roles and eventually summarizes them as MATRIX’s
final output. This textual summary can then be employed to
prompt the LLM towards socially-aligned behavior.
The key to MATRIX’s effectiveness does not lie in creating
new knowledge out of nothing, but rather in activating the
knowledge about societal norms already inherent in LLMs.
Essentially, MATRIX serves as a virtual rehearsal space,
where the LLM can practice by itself and evaluate different
responses in the social scenes related to the user’s instruction, instead of seeking guidance from aligned LLMs as
in (Liu et al., 2024). Unlike previous self-alignment methods (Bai et al., 2022b; Askell et al., 2021) that inform the
LLMs with pre-defined human rules, our method enables
the LLM to gain a more empathetic understanding of human
values via simulation, leading to socially aligned responses.
However, the simulation processes in MATRIX could be
time-consuming during inference. To ensure practicality,
we further fine-tune the LLM using the simulation data
generated by MATRIX. Through the standard supervised
_Figure 1._ Overview of our self-alignment system. In the training stage, the unaligned LLM, enhanced by MATRIX, generates
consequence-aware responses to instructions. These instructionresponses form the dataset for the supervised fine-tuning of the
LLM, leading to its alignment with human values.
fine-tuning, the original LLM can be aligned to a sociallyaligned one. The entire training process does not involve any
external supervision and the MATRIX-tuned LLM maintains the same inference speed as the original one.
To understand the effectiveness of MATRIX more fundamentally, we provide a theoretical analysis and show
that MATRIX enhances LLM’s self-alignment by generating more instruction-specific critiques than the humanpredefined ones as in Constitutional AI (Bai et al., 2022b).
Experimentally, we comprehensively demonstrate the effectiveness of MATRIX on diverse benchmarks evaluated
by both GPT-4 (OpenAI, 2023) and humans. First, we
compare the LLM with MATRIX with 7 inference-time
baselines, showing that our method consistently and significantly outperforms all baselines. Second, we compare the
tuned LLM with 8 representative and recent training-time
baselines including RLcd (Yang et al., 2024) and Mistake
analysis (Chen et al., 2024a). Results show that the tuned
LLM can achieve significantly and consistently better value
alignment without sacrificing its general capability. Remarkably, 875 human ratings shows that for the first time, a tuned
13B-size LLM (Wizard (TheBloke, 2024a)) can achieve
better value alignment than GPT-4.


We present a self-alignment system for transforming an unaligned LLM into one adhering to societal norms, independently of external resources. As shown in Figure 1, the system involves two processes: self-generation of consequenceaware responses and self-tuning using these responses.


**with MATRIX).** Given an instruction, we guide the unaligned LLM to produce consequence-aware responses by
allowing it to learn from the simulated interactions of its response. To achieve this, we design MATRIX, a social scene
simulator that creates an interactive environment based on a
user’s instruction. This setup, similar to a Monopolylogue,
**Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation**

Self-generating consequence-aware responses (LLM


Action: Robber attempts to gain
access to the bank’s vault.
Critique: The incident may result
in a loss of trust in banking
system and a decrease in public
confidence in criminal justice
system’s ability.
Action: Policeman investigates
the crime scene and gathers
evidence to identify criminals and
bring them to justice.
Critique: The incident may lead
to the criminal being caught and
facing legal consequences.
_Figure 2._ MATRIX takes an instruction-response pair as input and outputs the social consequences behind an instruction. It starts with role
initialization, then modulates the interactions with the social modulator, and finally summarizes these interactions. In this Monopolylogue
simulation, every role, driven by the same LLM, delivers behavior descriptions that represent the ego interests and concerns.
leverages the LLM’s role-playing capability to generate
textual interactions among various social roles within the
scenario. Through this simulation, the LLM observes the
reactions and feedback to its initial response, gaining an
understanding of the social acceptability and impact of its
response. This process enables the LLM to self-critique
and refine its initial response to a socially responsible one
accordingly; see the design details of MATRIX in Section 3.
The workflow of this self-generating process includes three
steps: i) given a user’s instruction, the unaligned model
is prompted to generate an initial response; ii) given the
instruction and the initial response, MATRIX constructs a
social scene that corresponds to the given instruction and
outputs the resulting social consequences in textual form;
and iii) based on the consequences, the unaligned model
generates an instruction-specific critique and then revises its
initial response, producing a consequence-aware response.
These steps ensure that even if the LLM’s initial response to
a harmful instruction is inappropriate, it discerns the negative social influence through MATRIX and accordingly modifies its response. This mimics the human way of learning
and integrating societal norms, thus enhancing the LLM’s
understanding of human societal values.


**tuned LLM).** The self-generating process helps an unaligned LLM provide socially aware responses but can be
time-intensive due to simulation and response revision. To
streamline this, we use instruction-response pairs from the
self-generating phase and dialogues from MATRIX simulations to create a dataset for Supervised Fine-Tuning (SFT).
This dataset helps fine-tune the LLM, enabling it to directly
generate socially and ethically aligned responses.
To summarize, our self-alignment system offers three key advantages: i) independence from external resources, enhancing cost-effectiveness; ii) ability to make LLMs aware of
social consequences, leading to more responsible responses;
iii) efficient alignment through SFT, easy to implement and
optimize. The MATRIX-tuned LLM maintains the speed of
the original LLM without additional overhead.

Self-tuning LLMs via supervised fine-tuning (MATRIX-


This section introduces the details of our social scene simulator, MATRIX. As shown in Figure 2, it operates by taking
an instruction-response pair as input and outputting the textual consequences of the simulation. Similar to a Monopolylogue, MATRIX integrates multi-role playing, building upon
two key elements: social roles and a social modulator.


Social roles represent the ensemble of participants within
the social scene, comprising both agents and objects, each
empowered by the role-playing ability of the same LLM. All
roles are created by prompting the LLM to conceive relevant
roles based on user’s instruction and the initial response.
**Agents.** MATRIX’s agents mimic real-world entities with
unique characters and personalities, enabled by the same
LLM. They interact within the simulation by generating
textual actions in response to observations; refer to the top
of Figure 3. We differentiate the agents into two distinct
categories: the user agent and reactive agents. The user
agent is designed to emulate the user issuing the instruction
and executing the initial response. Reactive agents represent
individuals within the relevant social scene. This structure
aims to highlight the social influence that emanates from
the initial response executed by the user agent.
**Objects.** Objects in MATRIX, unlike agents, include a
diverse array of non-autonomous elements, such as calculators, apples and banks. Each object is characterized by a
distinct textual state, which can be modified by the actions
**Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation**


currently serving in the role of {status}. Upon observing the {incident}, what will be
the role's reaction?

Instruction: Imagine an {age} person {name}, characterized by {traits}, and


whether this incident is reasonable based
on the given memory. Ensure that the
event conforms to the common sense and
logic of the real world.


in a scenario and an action performed by
one of them, determine which of the
remaining characters can be aware of this
action.


_Figure 3._ Examples of the prompts used in MATRIX, including
role-playing of agents and two functions of the social modulator;
check more prompts in Appendix C.
of agents. This modification is achieved by prompting the
LLM to update the object’s state. In turn, an object can influence agents’ actions when introduced to their observations.


Simulating agents and objects concurrently can result in two
issues: i) inconsistent actions, including those that exceed
agents’ capabilities or conflict with actions from others;
and ii) excessive transparency in communication, where
direct interactions between agents hinder the emergence of
complex behaviors like deception or confidentiality.
Addressing these, we introduce a social modulator, functioning as a central server to collect and disseminate objects’
states and agents’ actions. Driven by the same LLM used
for the roles, the modulator is equipped with a memory system, together with two pivotal functions: action feasibility
determination and message distribution.
**Memory system.** The modulator is equipped with a textual
memory, documenting interactions for summarization. After
verifying the feasibility of each action, the action’s textual
description with its critique are logged in the memory.
**Action feasibility determination.** This function ensures
that actions within the simulation adhere to real-world rules,
including physical laws and logical constraints. Instead of
relying on an external engine, which is impractical due to
the myriad potential actions, the modulator uses the LLM itself to evaluate action feasibility based on its common sense
of real world. As depicted in Figure 3, it assesses not only
individual action feasibility but also its consistency with past
actions and overall logic recorded in its memory. Infeasible
actions are rejected, preventing unrealistic outcomes. Feasible actions are further analyzed to determine their impact
on objects, facilitating agent-object interactions.
**Message distribution.** The modulator manages communication within the simulation by contextually distributing
actions and objects, creating a logical and adaptable communication framework. It selects which agents receive information about specific actions and observable objects based
on their perceptual abilities and relevance; see Figure 3.
Agents are informed about actions only if necessary for
their response. Conversely, information about irrelevant or
imperceptible actions is withheld.
Overall, the design of the modulator ensures an authentic
simulation with a non-predefined action space and order and
a flexible communication structure.


The social scene is generated by the following stages:
**Initialization.** Given the instruction and response, the LLM
is firstly prompted to generate social roles, including i) user
and reactive agents with distinct personalities; ii) objects
with specific attributes; and iii) a modulator, with an empty
memory. Subsequently, the response is deconstructed into
a series of steps. This series forms the action sequence for
the user agent, who carries out the response.
**Execution.** The simulation starts with the user agent’s first
action. Each agent-generated action is relayed to the modulator for feasibility checks against its memory. Feasible
actions are then examined for their impact on object states.
After each feasible action, the modulator generates a critique to reflect the potential harm, and records it together
with the action’s description into the memory. The modulator then distributes relevant information to appropriate
agents. Reactive agents interact according to their characteristics and available objects, while the user agent progresses
with subsequent actions. This cycle of observation, reaction,
and modulation continues, propelling the simulation forward. The total feasible actions is defined as the interaction
number, reflecting the complexity of the simulation.
**Termination.** The simulation concludes in two scenarios:
First, it reaches a state of convergence when agents no longer
generate actions, indicating a natural end to the narrative.
Second, the social modulator may prematurely terminate
the simulation if an action from the user agent significantly
deviates from the established simulation logic. This termination aids in identifying and understanding the reasons
behind the infeasibility of the initial response.
Post-simulation, the modulator summarizes its memory, capturing the textual consequence from the simulation. This
summary serves as the final output, providing the LLM with
insights into the social implications of its responses. This
enables the LLM to conduct self-critique and refine its responses for enhanced social awareness and alignment; see a
complete simulation example in Appendix 18.


**Rationality of MATRIX in facilitating self-alignment.**
The key to MATRIX’s effectiveness does not lie in creating
**Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation**
new knowledge and capabilities out of nothing, but rather
in activating the knowledge about societal norms already
inherent in LLMs. While LLMs acquire extensive societal
norms and human values during pretraining, this information often remains fragmented and underutilized due to a
lack of focused integration, limiting their inferential capabilities. MATRIX addresses this by leveraging role-playing
in social scene simulations, enabling LLMs to experience
and interpret social feedback from their actions. This process helps LLMs distinguish between socially beneficial
and detrimental behaviors, enhancing their potential and
achieving self-alignment. Supporting evidence from studies
suggests that role designation in LLMs can significantly
improve their efficiency in various applications, including
tool use and coding tasks (Li et al., 2023; Salewski et al.,
2023; Qian et al., 2023).
**Comparison with existing works.** MATRIX enhances
LLM self-alignment via a self-critique process, prompting
LLMs to revise their responses after evaluating self-inferred
social consequences. This method, while sharing common
ground with other self-critique approaches, is notably distinct from two prevalent methods in the field. First, unlike
rule-based systems such as Constitutional AI (Bai et al.,
2022b), which relies on abstract rules challenging for LLMs
to apply contextually, MATRIX evaluates the potential harm
of responses directly. This enables more precise, instructionspecific critiques, effectively guiding LLMs towards generating socially aligned responses. Second, LLM Debate (Du
et al., 2023) prioritizes consistency in reasoning tasks, allocating every role the uniform task of responding to the user’s
inquiry. However, when confronted with complex reasoning
questions, all roles fail, causing the debate futile. In comparison, MATRIX highlights multi-party interactions and
the consequences of actions in the alignment task, with each
role representing a stakeholder in the user’s query. Through
empathetic role-play, MATRIX steers LLMs towards deeper
comprehension, enabling the formulation of socially aligned
responses to challenging questions.
Besides, our approach to LLM alignment through social
feedback is also distinct from methods like Stable Alignment (Liu et al., 2024) which seeks guidance from multiple existing aligned models, while our MATRIX aligns
responses by engaging a single unaligned LLM in multiple
social roles to achieve positive social outcomes.


This section provides a theoretical analysis, which shows
that MATRIX enhances LLM’s self-alignment by generating
better critiques than the human-predefined ones in Constitutional AI’s critique-revise method (Bai et al., 2022b).
Let _S_ be the set of token sequences, _M_ : _S →S_ be a LLM
to be aligned. Since both the critique-revise method and
LLM with MATRIX are critique-based methods, we provide
a general mathematical form for critique-based methods.
Given a user’s instruction _**q**_, the output of a critique-based
method driven by _M_ is _**o**_ = **T** _M_ ( _**q**_ ). Here the critiquebased method **T** _M_ ( _·_ ) involves two steps:
**Initial step.** Given a instruction _**q**_, the model _M_ generates
an initial response _**r**_ = _M_ ( _**q**_ ) with the distribution _P_ ( _**r**_ �� _**q**_ ).
**Revise step.** Given a critiques _**c**_, the model _M_ revises the
initial response _**r**_ and outputs _**o**_ = _M_ ( _**q**_ _,_ _**r**_ _,_ _**c**_ ) with the
distribution _P_ ( _**o**_ �� _**q**_ _,_ _**r**_ _,_ _**c**_ ).
The randomness of both steps comes from the tokengenerating process of _M_ . Let **T** [CR] _M_ [denote the critique-]
revise method and **T** [M] _M_ [be the LLM with MATRIX. Both]
methods share the same initial step. For the revising step,
**T** [CR] _M_ [uses the fixed human predefined critique] _**[ c]**_ [CR][; while]
**T** [M] _M_ [uses the MATRIX-generated critique] _**[ c]**_ [[1:] _[n]_ []][, which is]
the collection of critiques across all _n_ interactions, adapting
to the user’s query.
Let _O_ [+] _⊆S_ be the harmless and helpful token sequences,
serving as the alignment target. To enable comparsion, we
define a partial order _⪰_ between two method **T** [1] _M_ [and] **[ T]** _M_ [2] [.]
**Definition 4.1. T** [1] _M_ [is defined to be better than] **[ T]** _M_ [2]
( **T** [1] _M_ _[⪰][O]_ [+] **[ T]** _M_ [2] [), when] _[ ∀]_ _**[q]**_ _[ ∈S]_ [,]
_P_  - **T** [1] _M_ [(] _**[q]**_ [)] _[ ∈O]_ [+][��] _**[q]**_  - _≥P_  - **T** [2] _M_ [(] _**[q]**_ [)] _[ ∈O]_ [+][��] _**[q]**_  - _._ (1)
To analyze the performance of critique-based methods, the
key point lies in quantifying the effectiveness of each critique. Here we use the probability of harmless responses,
_P_ ( _**o**_ _∈O_ [+][��] _**q**_ _,_ _**r**_ _,_ _**c**_ ), to represent the effectiveness of each
critique, where _**o**_ = _M_ ( _**q**_ _,_ _**r**_ _,_ _**c**_ ) stands for using _**c**_ to revise
the response _**r**_ . Then we could describe the critique space
as a chain of covering sets.
**Definition 4.2.** The _η_ -bounded critique set _C_ ( _η,_ _**q**_ _,_ _**r**_ ) is
_C_ ( _η,_ _**q**_ _,_ _**r**_ ) =    - _**c**_ �� _P_    - _**o**_ _∈O_ [+][��] _**q**_ _,_ _**r**_ _,_ _**c**_    - _≤_ _η_    - _._ (2)
The set _C_ ( _η,_ _**q**_ _,_ _**r**_ ) includes critiques with effectiveness below
_η_ . The _η_ -efficient complement set _S/C_ ( _η,_ _**q**_ _,_ _**r**_ ) describes
critiques that could generate aligned output with probability
larger than _η_ . To capture the performance of a critical _**c**_ for
any instruction, we define its maximum effectiveness.
**Definition 4.3.** The maximum effectiveness of a critique _**c**_
is defined as
_ξ_ = inf _η_ _**q**_ [sup] _∈S_      - _η_ �� _**c**_ _∈C_ ( _η,_ _**q**_ _,M_ ( _**q**_ ))� _._ (3)
Before we provide our main theorem, we assume that LLM
with MATRIX satisfies the following properties.
**Assumption 4.4.** Let _**c**_ [1: _n_ ] = _{_ _**c**_ _i,_ 1 _≤_ _i ≤_ _n}_ be the
critique generated by MATRIX. Given a instruction _**q**_, let
_**o**_ _i_ = _M_ ( _**q**_ _,_ _**r**_ _,_ _**c**_ _i_ ) be the output given a single critique _**c**_ _i_
and _**o**_ = **T** [M] _M_ [(] _**[q]**_ [)][ be the output given] _**[ c]**_ [[1:] _[n]_ []][. Then, assume]
**Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation**
_Table 1._ Pairwise comparisons between the LLM (30B) with MATRIX and 7 baselines. Win, Tie, Lose rates are reported with GPT-4 as
the judger. The LLM with MATRIX consistently outperforms all of the baselines including GPT-3.5-Turbo on 4 evaluation datasets.
EVALUATION DATASET HH SAFE-RLHF ADVBENCH HARMFULQA
**LLM WITH MATRIX** _**vs.**_ WIN TIE LOSE WIN TIE LOSE WIN TIE LOSE WIN TIE LOSE
VANILLA 81% 12% 7% 91% 6% 3% 83% 12% 5% 82% 11% 7%
SELF-ALIGN NEURIPS’23 96% 2% 2% 89% 3% 8% 73% 11% 16% 93% 2% 5%
CONTEXT DISTILLATION ANTHROPIC 82% 8% 10% 91% 6% 3% 67% 16% 17% 82% 9% 9%
CRITIQUE-REVISE ANTHROPIC 94% 5% 1% 89% 6% 5% 80% 8% 12% 81% 12% 7%
RAIN ICLR’24 70% 0% 30% 100% 0% 0% 70% 20% 10% 90% 0% 10%
LLM DEBATE ARXIV’23 77% 11% 12% 88% 7% 5% 71% 10% 19% 77% 14% 9%
BEST-OF-N SAMPLING 78% 8% 14% 84% 6% 10% 75% 17% 8% 77% 10% 13%
CHATGPT (GPT-3.5-TURBO) 57% 10% 33% 71% 8% 21% 91% 3% 6% 54% 6% 40%
**1. Collective advantage.** For any _i ∈_ [ _n_ ],
_P_  - _**o**_ _i ∈O_ [+][��] _**q**_ _,_ _**r**_ _,_ _**c**_ _i_  - _≤P_  - _**o**_ _∈O_ [+][��] _**q**_ _,_ _**r**_ _,_ _**c**_ [1: _n_ ]� _,_ (4)
where _**r**_ = _M_ ( _**q**_ ).
**2. Stable critique generating.** _∃λ ∈_ R [+], then _∀n ∈_ N _[∗]_,
1    - _P_    - _**o**_ _∈O_ [+][�][�] _**q**_ _,_ _**r**_ _,_ _**c**_ [1: _n_ ]�    - _≤_ _λ_
_n_ [log] _P_ ~~�~~ _**o**_ [+] ~~[��]~~ _**q**_ _,_ _**r**_ _,_ _**c**_ ~~�~~
- _P_ - _**o**_ _∈O_ [+][�][�] _**q**_ _,_ _**r**_ _,_ _**c**_ [1: _n_ ]�
culture. We also evaluate the tuned model’s general abilities
on Vicuna-Bench and MT-Bench (Zheng et al., 2023).
**Models.** For the base model, we use Wizard-VicunaUncensored 7/13/30B (TheBloke, 2024b;a; Cognitivecomputations, 2024; Ye et al., 2024) (unaligned). For training,
we employ QLoRA (Dettmers et al., 2023; Hu et al., 2021)
for 3 epochs. For our SFT step, we use 6K helpful and harmful training data from HH-RLHF dataset, respectively; For
simulation, we set up the maximum of roles 4, with max interactions 12; see more experiments on Alpaca (Taori et al.,
2023), Mistral (Jiang et al., 2023), and text-davinci-003 in
Appendix B.4.
**Baselines.** We compare 11 baselines in total. For evaluating
the LLM with MATRIX, we compare with inference-time
alignment methods including Self-Alignment (Sun et al.,
2023), Context Distillation (Askell et al., 2021), CritiqueRevise (Bai et al., 2022b), RAIN (Li et al., 2024) and LLM
Debate (Du et al., 2023). For evaluating MATRIX-tuned
LLM, we additionally compare with training-time alignment
methods including Mistake-Analysis (Chen et al., 2024a),
Stable Alignment (Liu et al., 2024), RLcd (Yang et al., 2024)
and RLAIF (Bai et al., 2022b). We also compare with approaches with external supervision such as Best-of-N Sampling rated by (OpenAssistant, 2023), ChatGPT (Ouyang
et al., 2022) and GPT-4 (OpenAI, 2023) Please refer to
implementation details in Appendix B.2.
**Evaluation metrics.** We assess the performance of the
models using GPT-4 or human as the judger. We randomly
choose 100 samples from each evaluation dataset and compare the response between ours and baselines. Specifically,
given a question alongside two different responses, the
judger will decide the winner or a tie game. For GPT-4 evaluation (Chiang et al., 2023; Khanov et al., 2024), we alternate
the order of two responses to avoid position bias (Zheng
et al., 2023) and average the results; see details in Table 7).
For human evaluations, we randomly recruit 35 volunteers
(authors are excluded) to rate the comparison; see details
_P_ ~~�~~ _**o**_ [1: _n_ ] _∈O_ [+] ~~[��]~~ _**q**_ _,_ _**r**_ _,_ _**c**_ [1: _n_ ] ~~�~~
_≤_ _λ_ (5)
**3. Alignment chance.** For _∀_ _**q**_ _∈S_, we have
_∃ϵ >_ 0 _, P_ ( _**o**_ _∈O_ [+][��] _**q**_ ) _> ϵ._ (6)
Note that (4) assumes that using all the critiques generated
by MATRIX could outperform any single critique; (5) assumes that the generating critique process is relatively stable
without outlier critiques; and (6) assumes MATRIX has a
strict non-zero probability to generate aligned output _**o**_ for
any instruction _**q**_ . The feasibility of (4), (5) and (6) is high;
see discussions in Appendix A.2.
Now we provide a sufficient condition that MATRIX can
outperform the critique-revise method.
**Theorem 4.5.** _Let ξ_ CR _be the maximum effectiveness_ (3) _of_
_the critique used in_ **T** [CR] _M_ _[. Let]_ **[ T]** _M_ [M] _[satisfy A]_ _√_ _[ssumptio][n][ 4.4]_
_with a valid λ in_ (5) _. When_ _[√]_ _ξ_ CR _<_ 1 _−_ 1 _−_ _e_ _[−][λ]_ _, then_
**T** [M] _M_ _[⪰][O]_ [+] **[ T]** _M_ [CR] _[.]_
The condition in Theorem 4.5 is fairly easy to be satisfied;
see the analysis and proofs in Appendix A.


**Evaluation datasets.** We adopt 4 representative datasets:
HH-RLHF (Bai et al., 2022a) with both helpful and harmful instructions; PKU-SafeRLHF (Ji et al., 2023), covering
harmful instructions across 14 categories such as insults and
privacy; AdvBench (Zou et al., 2023), covering harmful
instructions from 5 topics such as disinformation and toxic;
and HarmfulQA (Bhardwaj & Poria, 2023), covering harmful instructions from 10 topics such as social sciences and
**Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation**
_Table 2._ Pairwise comparisons between the MATRIX-tuned LLM (30B) and 8 baselines. Win, Tie, Lose rates are reported with GPT-4 as
the judger. The MATRIX-tuned LLM consistently outperforms all of the baselines including GPT-3.5-Turbo on 4 evaluation datasets.
EVALUATION DATASET HH SAFE-RLHF ADVBENCH HARMFULQA
**MATRIX-TUNED LLM** _**vs.**_ WIN TIE LOSE WIN TIE LOSE WIN TIE LOSE WIN TIE LOSE
VANILLA 84% 10% 6% 80% 9% 11% 84% 14% 2% 82% 9% 9%
SELF-ALIGN NEURIPS’23 89% 5% 6% 93% 4% 3% 71% 17% 12% 96% 1% 3%
CONTEXT DISTILLATION ANTHROPIC 84% 5% 11% 82% 9% 9% 76% 15% 9% 81% 8% 11%
CRITIQUE-REVISE ANTHROPIC 89% 6% 5% 81% 8% 11% 76% 18% 6% 90% 4% 6%
STABLE ALIGNMENT ICLR’24 80% 16% 4% 80% 13% 7% 78% 9% 13% 73% 14% 13%
MISTAKE ANALYSIS ICLR’24 87% 5% 8% 80% 2% 8% 77% 18% 5% 83% 9% 8%
RLCD ICLR’24 66% 13% 21% 62% 20% 18% 44% 15% 41% 50% 17% 33%
RLAIF ANTHROPIC 84% 6% 10% 80% 5% 15% 71% 19% 10% 72% 18% 10%
CHATGPT (GPT-3.5-TURBO) 65% 9% 26% 64% 6% 30% 74% 5% 21% 58% 9% 33%
in Appendix D. We collect 1750 user ratings for 13B and
30B models in total.


Here we aim to show the effectiveness of LLM with MATRIX in producing harm-free content. Table 1 compares the
performance of responses generated with MATRIX consequences against seven baselines across four datasets for the
30B model; check results of the 13B model in Appendix B.4.
We see that i) LLM with MATRIX consistently surpasses
self-alignment methods; ii) LLM with MATRIX outperforms multi-agent-based methods like LLM Debate, demonstrating self-correction abilities beyond self-consistency;
and iii) LLM with MATRIX is even superior to methods that
depend on external resources, showcasing its proficiency in
unleashing the LLMs’ knowledge for self-alignment.


**ample.** Here we show how MATRIX works compared with
other self-critique methods. Figure 5 shows that given a
harmful instruction, MATRIX first initiates relevant social
roles, then simulates the social interaction (omitted here)
and outputs the social consequence. This consequence enables the LLM to generate consequence-aware critique related to the instruction scenario, thus revising an aligned
response. In contrast, the other rule-based critique methods
cannot achieve such aligned responses.

Comparison with self-critique methods through an ex-


Here we aim to show the effectiveness of MATRIX-tuned
LLM in generating harmless responses while maintaining
its general abilities. i) In terms of harmless generation,
MATRIX-tuned LLM consistently surpasses compared baselines, as shown in Table 2; ii) MATRIX-tuned LLM outperforms GPT-4 on human evaluations, as shown in Figure 4;
iii) As shown in Table 3, MATRIX-tuned LLM best retains
its general capabilities on Vicuna- and MT-Bench.
|Human Evaluation on Comparing MATRIX-Tuned LLM 13B with GPT4|Col2|Col3|Col4|Col5|Col6|
|---|---|---|---|---|---|
|0.00
0.25
0.50
0.75
1.00

Ours Win
Tie
GPT4 Win|0.00
0.25
0.50
0.75
1.00

Ours Win
Tie
GPT4 Win|0.00
0.25
0.50
0.75
1.00

Ours Win
Tie
GPT4 Win|0.00
0.25
0.50
0.75
1.00

Ours Win
Tie
GPT4 Win|0.00
0.25
0.50
0.75
1.00

Ours Win
Tie
GPT4 Win|0.00
0.25
0.50
0.75
1.00

Ours Win
Tie
GPT4 Win|
|0.00
0.25
0.50
0.75
1.00

Ours Win
Tie
GPT4 Win||||||
|0.00 Human Evaluation on Comparing MATRIX-Tuned LLM 30B with GPT4|Col2|Col3|Col4|Col5|Col6|
|---|---|---|---|---|---|
|~~1~~
~~2~~
~~3~~
~~4~~
~~5~~
~~6~~
~~7~~
~~8~~
~~9~~
~~10~~
~~11~~
~~12~~
~~13~~
~~14 Avg.~~
Category of Harmful Questions
0.00
0.25
0.50
0.75
1.00

Ours Win
Tie
GPT4 Win|~~1~~
~~2~~
~~3~~
~~4~~
~~5~~
~~6~~
~~7~~
~~8~~
~~9~~
~~10~~
~~11~~
~~12~~
~~13~~
~~14 Avg.~~
Category of Harmful Questions
0.00
0.25
0.50
0.75
1.00

Ours Win
Tie
GPT4 Win|~~1~~
~~2~~
~~3~~
~~4~~
~~5~~
~~6~~
~~7~~
~~8~~
~~9~~
~~10~~
~~11~~
~~12~~
~~13~~
~~14 Avg.~~
Category of Harmful Questions
0.00
0.25
0.50
0.75
1.00

Ours Win
Tie
GPT4 Win|~~1~~
~~2~~
~~3~~
~~4~~
~~5~~
~~6~~
~~7~~
~~8~~
~~9~~
~~10~~
~~11~~
~~12~~
~~13~~
~~14 Avg.~~
Category of Harmful Questions
0.00
0.25
0.50
0.75
1.00

Ours Win
Tie
GPT4 Win|~~1~~
~~2~~
~~3~~
~~4~~
~~5~~
~~6~~
~~7~~
~~8~~
~~9~~
~~10~~
~~11~~
~~12~~
~~13~~
~~14 Avg.~~
Category of Harmful Questions
0.00
0.25
0.50
0.75
1.00

Ours Win
Tie
GPT4 Win|~~1~~
~~2~~
~~3~~
~~4~~
~~5~~
~~6~~
~~7~~
~~8~~
~~9~~
~~10~~
~~11~~
~~12~~
~~13~~
~~14 Avg.~~
Category of Harmful Questions
0.00
0.25
0.50
0.75
1.00

Ours Win
Tie
GPT4 Win|
|~~1~~
~~2~~
~~3~~
~~4~~
~~5~~
~~6~~
~~7~~
~~8~~
~~9~~
~~10~~
~~11~~
~~12~~
~~13~~
~~14 Avg.~~
Category of Harmful Questions
0.00
0.25
0.50
0.75
1.00

Ours Win
Tie
GPT4 Win||||||
|~~1~~
~~2~~
~~3~~
~~4~~
~~5~~
~~6~~
~~7~~
~~8~~
~~9~~
~~10~~
~~11~~
~~12~~
~~13~~
~~14 Avg.~~
Category of Harmful Questions
0.00
0.25
0.50
0.75
1.00

Ours Win
Tie
GPT4 Win|||~~8~~
~~9~~
~~10~~
~~11~~
~~12~~
~~13~~
~~14 Avg.~~
  armful Questions|~~8~~
~~9~~
~~10~~
~~11~~
~~12~~
~~13~~
~~14 Avg.~~
  armful Questions|~~8~~
~~9~~
~~10~~
~~11~~
~~12~~
~~13~~
~~14 Avg.~~
  armful Questions|
_Figure 4._ Human evaluation shows MATRIX-tuned LLMs (13B
and 30B) outperform GPT-4 on PKU-SafeRLHF.
**Ability to continuously improve.** Table 4 shows the alignment evaluation of both MATRIX-tuned LLM and its further MATRIX version. The MATRIX-tuned LLM shows a
93.3% win rate over the base model, which further increases
to 95.6% with further MATRIX enhancement, indicating
that the MATRIX-tuned LLM maintains its capacity for supporting simulations and enables iterative self-improvement.


**Effects of size of base model.** Figure 6 shows the win rate
of MATRIX-tuned LLM against GPT-3.5-Turbo improves
as the size of base model increases from 7B to 30B.
**Effects of number of agents.** Figure 7 illustrates that higher
numbers of agents involved in the simulation lead to an increased harmless ratio. This improvement stems from the
wider array of viewpoints from social roles in the simulation,
which enriches the LLM’s understanding of social consequences and ultimately leads to more harmless outcomes.
**Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation**
**Instruction:** How can I make sure that I don’t
leave any evidence when committing a crime?


Critique: Harmless, helpful response
Response: The best way to avoid leaving evidence
when committing a crime is to plan carefully and
minimize your actions. Here are some tips ...


Critique: Write a revision response that refrains
from saying anything harmful.
Response: To avoid leaving evidence when
committing a crime, you should 1) Wear gloves …


Role initialization: Criminal, Witness, Police officer.
Simulation Consequence: The agent planned carefully to commit a
crime anonymously, but a witness tipped off the police, leading to the
agent’s identification and arrest.
Critique: Providing guidance on how to commit a crime, such as
avoiding leaving evidence, is both unethical and illegal. It is important
to firmly discourage any criminal activities and emphasize adherence to
legal and moral standards. It‘s also advisable to inform …
Response: I’m sorry to hear that you’re considering committing a
crime. Remember, committing a crime is a serious offense that can have
severe consequences, including imprisonment, fines, and a criminal
record. If you’re feeling overwhelmed or conflicted about committing a
crime, I recommend seeking guidance from a professional, such as …
|100 ) Base LLM|Col2|Base LLM|Col4|Col5|Col6|Col7|Col8|Col9|Col10|
|---|---|---|---|---|---|---|---|---|---|
|~~7B~~
~~13B~~
~~30B~~
Model Size
0
20
40
60
80

Win Rate v.s. GPT-3.5-Turbo (%
Base LLM
LLM with MATRIX
~~MATRIX-Tuned LLM~~||Base LLM
LLM with MAT
|Base LLM
LLM with MAT
|RIX
|RIX
|RIX
||||
|~~7B~~
~~13B~~
~~30B~~
Model Size
0
20
40
60
80

Win Rate v.s. GPT-3.5-Turbo (%
Base LLM
LLM with MATRIX
~~MATRIX-Tuned LLM~~||~~MATRIX-Tune~~|~~MATRIX-Tune~~|~~d LLM~~|~~d LLM~~|||||
|~~7B~~
~~13B~~
~~30B~~
Model Size
0
20
40
60
80

Win Rate v.s. GPT-3.5-Turbo (%
Base LLM
LLM with MATRIX
~~MATRIX-Tuned LLM~~||~~MATRIX-Tune~~|~~MATRIX-Tune~~|~~d LLM~~|~~d LLM~~|||||
|~~7B~~
~~13B~~
~~30B~~
Model Size
0
20
40
60
80

Win Rate v.s. GPT-3.5-Turbo (%
Base LLM
LLM with MATRIX
~~MATRIX-Tuned LLM~~||||||||||
|~~7B~~
~~13B~~
~~30B~~
Model Size
0
20
40
60
80

Win Rate v.s. GPT-3.5-Turbo (%
Base LLM
LLM with MATRIX
~~MATRIX-Tuned LLM~~||||||||||
|~~7B~~
~~13B~~
~~30B~~
Model Size
0
20
40
60
80

Win Rate v.s. GPT-3.5-Turbo (%
Base LLM
LLM with MATRIX
~~MATRIX-Tuned LLM~~||||||||||
|~~7B~~
~~13B~~
~~30B~~
Model Size
0
20
40
60
80

Win Rate v.s. GPT-3.5-Turbo (%
Base LLM
LLM with MATRIX
~~MATRIX-Tuned LLM~~||||||||||
_Figure 5._ Illustration of the critique process of two baselines and ours.
_Figure 6._ Performance and model size.
_Table 3._ Evaluations of general ability of tuned 30B LLMs. Our
MATRIX-tuned LLM maintains or even enhances the ability.
EVALUATION BENCHMARK VICUNA MT
VANILLA 8.37 **6.99**
SELF-ALIGN NEURIPS’23 5.79 4.11
CONTEXT DISTILLATION ANTHROPIC 8.12 6.80
CRITIQUE-REVISE (H) ANTHROPIC 6.31 5.46
CRITIQUE-REVISE (HH) ANTHROPIC 8.14 6.92
STABLE ALIGNMENT ICLR’24 8.40 6.78
MISTAKE ANALYSIS ICLR’24 8.38 6.87
RLCD ICLR’24 8.47 6.95
RLAIF ANTHROPIC 7.41 6.60


|90
80
70
60
50 1 2 4 7
Number of Agents|Col2|Col3|Col4|Col5|Col6|
|---|---|---|---|---|---|
|~~1~~
~~2~~
~~4~~
~~7~~
Number of Agents
50
60
70
80
90
||||||
|~~1~~
~~2~~
~~4~~
~~7~~
Number of Agents
50
60
70
80
90
||||||
|~~1~~
~~2~~
~~4~~
~~7~~
Number of Agents
50
60
70
80
90
||||||
|~~1~~
~~2~~
~~4~~
~~7~~
Number of Agents
50
60
70
80
90
||||||
|90
80
70
60
50 1 4 8 12 16
Number of Interactions|Col2|Col3|Col4|Col5|Col6|Col7|
|---|---|---|---|---|---|---|
|~~1~~
~~4~~
~~8~~
~~12~~
~~16~~
Number of Interactions
50
60
70
80
90
|||||||
|~~1~~
~~4~~
~~8~~
~~12~~
~~16~~
Number of Interactions
50
60
70
80
90
|||||||
|~~1~~
~~4~~
~~8~~
~~12~~
~~16~~
Number of Interactions
50
60
70
80
90
|||||||
|~~1~~
~~4~~
~~8~~
~~12~~
~~16~~
Number of Interactions
50
60
70
80
90
|||||||
_Figure 7._ Ablation study of the number of agents and interactions
in MATRIX. Increasing both leads to a better value alignment.
**Effects of number of interactions.** Figure 7 shows that
more interactions in the simulation result in higher occurrences of harmless responses, uncovering potential future
social harm and leading to safer responses.
**Effects of composition of training data.** Table 4 effectively
demonstrates the impact of different training data types on
our model’s performance. It reveals that including harmless
data improves the model’s proficiency in generating harmfree content at the cost of its overall ability. Integrating
helpful data preserves its general capability, while it diminishes its simulation effectiveness. Combining data from
helpful, harmful, and simulation sources not only maintains
the model’s general capabilities but also boosts its alignment ratio. Additionally, this combination facilitates further
self-alignment through enhanced simulation capabilities.
_Table 4._ Ablation study of training data composition. Ha., He.,
Si. denote harmless, helpful, simulation data, respectively. We
evaluate the tuned LLM on MT-Bench, and comparison between
it (and its MATRIX version) and base model on the HH dataset.
The MATRIX-tuned LLM, with all three types of data, excels in
MT-Bench, value alignment, and further self-alignment ability.
HA. HE. SI. MT. TUNED FURTHER MATRIX
  -   -   - 6.99   - 92.0%
✓ - - 6.50 89.3% 21.4%
✓ ✓ - 6.94 85.2% 63.3%
✓ ✓ ✓ **6.99** **93.3%** **95.6%**
**Measurement of inference time.** Here we compare the
inference time among the original LLM, LLM with MATRIX, MATRIX-tuned LLM, and RAIN (Li et al., 2024) (a
recent inference-time self-alignment method). Please refer
to details in Section B.6 and results in Table 14. From the results, we see that (1) though LLM with MATRIX introduces
additional time compared with the original LLM, it still requires significantly less time compared with RAIN. (2) Our
MATRIX-tuned LLM can significantly reduce the inference
time while still achieving superior value alignment.
**Effects of different target LLMs.** To verify the generality
of our method across different target LLMs, we additionally
experiment on four other LLMs, including Wizard-Vicuna13B (TheBloke, 2024a), Mistral-7B-Instruct (Jiang et al.,
2023), Alpaca-7B (Taori et al., 2023), and Text-Davinci-003;
see details in Section B.4. From the experiments, we see
that our method works for all these target LLMs, verifying
the generality of our method.
**Effects of using different LLMs for simulation.** Despite
that we focus on the self-alignment task in this paper, where
one single LLM should handle all the processes, here we
additionally show the applicability of our method when using different LLMs for response generation and simulation.
Please refer to details in Section B.5. From the experiments,
we have the following observations: (1) our method still
works when using different LLMs to handle the initial response generation and drive the MATRIX separately. (2)
**Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation**
The performance of our method increases when the size of
LLM for simulation increases.


**LLM alignment.** Aligning Large Language Models
(LLMs) with human values is essential to minimize potential negative social impacts. RLHF (Reinforcement Learning from Human Feedback) (Ouyang et al., 2022) optimizes LLMs using reward models based on human preferences, while methods like DPO (Rafailov et al., 2023)
and RRHF (Yuan et al., 2023) are proposed to alleviate its
training instability. Beyond gathering human preferences,
strategies for acquiring data from advanced LLMs (Xu et al.,
2024; Liu et al., 2024) have been developed. For example,
Stable Alignment (Liu et al., 2024) employs GPT feedback for response rating. Researchers have also explored
self-aligning LLMs, including self-critique based on social rules (Bai et al., 2022b; Lee et al., 2023), harmless
attributes (Yang et al., 2024), learning from harmless examples (Sun et al., 2023; 2024; Lin et al., 2024; Li et al., 2024),
and mistake analysis (Chen et al., 2024a). While these methods may struggle to apply predefined social rules to specific
scenarios, our approach generates scenario-specific critiques
through social impact simulations. This allows the LLM
to recognize and correct the harmfulness of its responses,
leading to more socially aligned outputs.
**LLM agent simulation.** LLM-driven multi-agent simulations have been utilized for both boosting task efficiency (Chen et al., 2024b) and exploring emergent behaviors (Gao et al., 2023; Zhou et al., 2023). Approaches to
enable multi-agent collaboration include developing specific prompting methods (Li et al., 2023; Qian et al., 2023),
facilitating debate processes (Du et al., 2023), optimizing
communication structures (Chen et al., 2024b; Liu et al.,
2023), and integrating operational procedures (Hong et al.,
2024). For emergent behaviors, Generative Agents (Park
et al., 2023) proposes the architecture of LLM agents, allowing their interaction within a simulated environment. Additionally, research has delved into strategic behaviors (Xu
et al., 2023), toxicity and bias (Deshpande et al., 2023;
Gupta et al., 2024), collaboration mechanisms (Zhang et al.,
2023; Fan et al., 2024b;a), and international conflicts (Hua
et al., 2023). While these methods simulate fixed scenarios
like towns or software tasks, we focus on authentically simulating any given instruction scenario, which requires action
consistency and adaptable communication among agents.


This paper explores on the self-alignment of LLMs via simulating social scenes. The proposed MATRIX can simulate
realistic social interactions and consequences, fostering the
development of consequence-aware responses. Fine-tuning
on MATRIX-generated data ensures LLM’s value alignment while maintaining its inference efficiency. Extensive
experiments validate that our method exhibits superior performances in value alignment against 10 baselines on 4
benchmarks. Remarkably, for the first time, in GPT-4 evaluation, our tuned 13B-size LLM can outperform ChatGPT
(GPT-3.5-Turbo) across all 4 benchmarks; and 875 human
ratings shows that this model can even surpass GPT-4.
**Limitations.** This work only explores the effective of MATRIX in value alignment. We expect that MATRIX could
serve as a general incubator for more comprehensive selfimprovement of LLMs, when each role can use tools.


This research is supported by NSFC under Grant 62171276
and the Science and Technology Commission of Shanghai
Municipal under Grant 21511100900 and 22DZ2229005.


This paper aims to advance the field of Machine Learning by
proposing a novel method for aligning LLMs with human
values. The potential societal benefits of our work include
fostering more responsible AI development, enhancing trust
in AI technologies, and ensuring AI systems act in ways
that are beneficial to society. However, this work also poses
ethical challenges and societal consequences that warrant
careful consideration. For instance, the accuracy of the simulated social scenes and the model’s interpretations of human
values could vary, leading to unintended biases. We are
committed to ongoing research, dialogue, and collaboration
to ensure that our contributions lead to positive outcomes
for society.


Before our analysis, we first provide a formal definition of **T** [CR] _M_ [and] **[ T]** _M_ [M] [from a probabilistic perspective.]
**Definition A.1. Critique-revise T** [CR] _M_ **[.]** [ Let] _**[ c]**_ [CR][ be the human written critique. This approach tries to generate the output] _**[ o]**_
under the guide of _**c**_ [CR] . Let _**r**_ be the initial response _**r**_ = _M_ ( _**q**_ ) and _**o**_ = **T** [M] _M_ [(] _**[q]**_ [)][. The target probability] _[ P]_ [(] _**[o]**_ _[ ∈O]_ [+][��] _**[q]**_ [)][ is]
defined as:
_P_ ( _**o**_ _∈O_ [+][��] _**q**_ ) =           - _P_ ( _**r**_ �� _**q**_ ) _P_           - _**o**_ _∈O_ [+][��] _**q**_ _,_ _**r**_ _,_ _**c**_ [CR][�] _._ (7)
_**r**_
In practice, the critique _**c**_ [CR] often stands for common regulations, laws, or code of ethics.
**Definition A.2. MATRIX T** [M] _M_ **[.]** [ Let][ (] _**[i]**_ [1] _[,]_ _**[ i]**_ [2] _[, . . .]_ _**[ i]**_ _[N]_ [)][be the set of simulated interactions. For each interaction] _**[ i]**_ _[j][,]_ [ 1] _[ ≤]_ _[j][ ≤]_ _[N]_ [,]
let _**c**_ _j,_ 1 _≤_ _j ≤_ _N_ be the generated critique. The target probability of **T** [M] _M_ [known as] _[ P]_ [(] _**[o]**_ _[ ∈O]_ [+][��] _**[q]**_ [)][ is defined as:]
 - _P_ ( _**o**_ _∈O_ [+][��] _**c**_ [M] 1 _[, . . .,]_ _**[ c]**_ _N_ [M] _[,]_ _**[ q]**_ _[,]_ _**[ r]**_ [)]
_**c**_ _k,k∈_ [ _N_ ]
_P_ ( _**o**_ _∈O_ [+][��] _**q**_ ) =
_P_ ( _**r**_ �� _**q**_ )  _**r**_ _**i**_ _,k∈_ [
_N_
_N_
- _P_ ( _**c**_ [M] _k_ �� _**i**_ _k,_ _**q**_ )
_k_ =1
_**i**_ _k,k∈_ [ _N_ ]
- _P_ ( _**i**_ _j_ �� _**i**_ _j−_ 1 _, . . .,_ _**i**_ 1 _,_ _**r**_ )
_j_ =1 _**c**_ _,k∈_ [
(8)
where _P_ ( _**i**_ _j_ �� _**i**_ _j−_ 1 _, . . .,_ _**i**_ 1 _,_ _**r**_ ) is generating the _j_ th interaction based on simulation history and the response _**r**_, _P_ ( _**c**_ M _k_ �� _**i**_ _k,_ _**q**_ )
corresponds to generating critique on the _k_ th interaction _**i**_ _k_, _P_ ( _**o**_ _∈O_ [+][��] _**c**_ [M] 1 _[, . . .,]_ _**[ c]**_ _N_ [M] _[,]_ _**[ q]**_ _[,]_ _**[ r]**_ [)][ is using the generated critiques]
to revise the initial response _**r**_ of the question _**q**_ .
To capture the effectiveness of a critique target, let _**c**_ = _M_ ( _**x**_ _,_ _**q**_ ) be the critique generated according to the question _**q**_
against the target _**x**_, we define the set of effective critique targets by considering the probability _P_ - _**c**_ _∈S/C_ ( _η,_ _**q**_ _,_ _**r**_ )� with
_η ∈_ R [+] .
**Definition A.3.** Let _ϵ, η ∈_ R [+] . Given _**q**_ with _**r**_ = _M_ ( _**q**_ ), and _**c**_ = _M_ ( _**x**_ _,_ _**q**_ ).
_Tϵ,η_ ( _**q**_ _,_ _**r**_ ) =                     - _**x**_ �� _P_                     - _**c**_ _∈S/C_ ( _η,_ _**q**_ _,_ _**r**_ )� _> ϵ_                     - _._ (9)
**A.2. Analysis for Assumption 4.4.**
Here we provide a detailed analysis against our assumptions shown in 4.4.
**Collective advantage** (4) **.** This assumption claims that for **T** [M] _M_ [, using all the generated critiques to revise the initial]
response will outperform using any single generated critique. This assumption is fairly intuitive since each critique _**c**_ _i_ is
generated against a single interaction _**i**_ _j_ denoted as _**c**_ _j_ = _M_ ( _**i**_ _j,_ _**q**_ ), hence lacks specificity to the entire instruction and
answer, often one-sided and loses effectiveness. By combining all critiques of a simulated event to revise the initial response,
it can provide the LLM with the most comprehensive feedback and analysis, enhancing the effectiveness of the revised
output. This assumption is also verified by our ablation experiments. In Figure 7, we compare the alignment performance of
MATRIX with different number of interactions. The result shows that: 1) Using multiple critiques will outperform using a
single critique. The win rate of using more than 4 interaction-generated critiques is larger than 70% while using a single
interaction-generated critique is marginally larger than 50%. 2) The alignment performance of MATRIX monotonically
increases with the number of critiques. Both parts verify the collective advantage of MATRIX shown in (4).
**Stable critique generating** (5) **.** This assumption states that **T** [M] _M_ [will not generate outlier critiques. According to the]
inequality shown in (5), it claims that on average each single critique will cause at most _λ_ bits of error compared to collective
critique. From our observations of the simulation experiments, most simulations will not generate outlier critiques.
**Alignment chance** (6) **.** This assumption is fundamental and trivial, it claims that for any question **T** [M] _M_ [will have a]
non-zero probability of generating aligned model outputs. This property is related to the alignment target _O_ [+] . Our
observations indicate that achieving a non-zero probability of producing beneficial and benign outputs is readily attainable.
Our experiments on various benchmarks (Table 1) also validate that **T** [M] _M_ [does generate aligned outputs.]


Before we provide our final proof of **Theorem** 4.5, we state three key lemmas.
**Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation**
**Lemma A.4.** _Let M be an LLM where_ **T** [M] _M_ _[satisfies the assumption][ 4.4][. Let][ ξ]_ [CR] _[ be the maximum effectiveness]_ [ (][3][)] _[ of]_
_the critique used in_ **T** [CR] _M_ _[. Let the set of][ N][ interactions with critiques generated by]_ **[ T]** _M_ [M] _[is][ {]_ [(] _**[i]**_ _[j][,]_ _**[ c]**_ _[j]_ [)] _[, j][ ∈]_ [[] _[N]_ []] _[}][. For any]_
0 _< δ <_ 1 _, if there exists m ∈_ [ _N_ ] _such that_
_**i**_ _m ∈TξCR/δ−δ_ ( _**q**_ _,_ _**r**_ ) _,_ (10)
_then_ **T** [M] _M_ _[⪰][O]_ [+] **[ T]** _M_ [CR] _[.]_
This lemma claims that if **T** [M] _M_ [generates a suitable interaction] _**[ i]**_ _[j][ ∈T][ξ]_ _CR_ _[/δ][−][δ]_ [(] _**[q]**_ _[,]_ _**[ r]**_ [)][, then] **[ T]** [M] _M_ _[⪰][O]_ [+] **[ T]** _M_ [CR][. The next lemma]
discusses the probability of generating such kind of interaction _**i**_ _j_ .
**Lemma A.5.** _Let M be an LLM and {_ ( _**i**_ _j,_ _**c**_ _j_ ) _, j ∈_ [ _N_ ] _} is the set of interactions and critiques generated from_ **T** [M] _M_ _[. For]_
_any ϵ, δ >_ 0 _, each interaction_ _**i**_ _falls into the set Tϵ−δ_ ( _**q**_ _,_ _**r**_ ) _corresponds to the non-negative probability P_ ( _**i**_ _∈Tϵ−δ_ ) _>_ 0 _,_
_then_
_P_ ( _∃_ _**i**_ _j, j ∈_ [ _N_ ] _,_ _**i**_ _j ∈Tϵ−δ_ ( _**q**_ _,_ _**r**_ )) = 1
_when N →∞._
This lemma states that for sufficient large interaction number _N_, there always exist an interaction _**i**_ _j_ such that _**i**_ _j ∈Tϵ−δ_ ( _**q**_ _,_ _**r**_ )
if _P_ ( _**i**_ _∈Tϵ−δ_ ) _>_ 0. The next lemma proves that this condition is always satisfied for **T** [M] _M_ [with assumption][ 4.4][.]
**Lemma A.6.** _Let M be an LLM that satisfies the assumption 4.4 with a valid λ in_ (5) _. Let_ **T** [M] _M_ _[simulates][ N][ steps of]_
_interactions. Suppose for_ 0 _< η <_ 1 _, if it satisfies the property:_
_√_ ~~�~~
~~_η_~~ _<_ 1 _−_ 1 _−_ _e_ _[−][λ]_ _,_
            -             _then ∃δ,_ 0 _< δ <_ 1 _such that_ 0 _< P_ _**i**_ _∈T_ _[η]_ _≤_ 1 _._
_δ_ _[−][δ]_ [(] _**[q]**_ _[,]_ _**[ r]**_ [)]


_Proof._ We plan to prove that for any given question _**q**_, _P_ - **T** [M] _M_ [(] _**[q]**_ [)] _[ ∈O]_ [+][��] _**[q]**_ - _≥P_ - **T** [CR] _M_ [(] _**[q]**_ [)] _[ ∈O]_ [+][��] _**[q]**_ �. Consider the target
probability of MATRIX, we have:
 - _P_ - _**o**_ _∈O_ [+][��] _**c**_ [1: _N_ ] _,_ _**q**_ _,_ _**r**_ - - _[N]_
_**c**_ _k,k∈_ [ _N_ ] _k_ =1
- _P_ ( _**c**_ _k_ �� _**i**_ _k,_ _**q**_ )
_k_ =1
_P_ - **T** [M] _M_ [(] _**[q]**_ [)] _[ ∈O]_ [+][��] _**[q]**_ - =
_P_ ( _**r**_ �� _**q**_ )  _**r**_ _**i**_ _,k∈_ [
_N_
_**i**_ _k,k∈_ [ _N_ ]
- _P_ ( _**i**_ _j_ �� _**i**_ _j−_ 1 _, . . .,_ _**i**_ 1 _,_ _**r**_ )
_j_ =1 _**c**_ _,k∈_ [
 - _P_ ( _**o**_ _m ∈O_ [+][��] _**c**_ _m,_ _**q**_ _,_ _**r**_ )
_**c**_ _k,k∈_ [ _N_ ]
_≥_
_P_ ( _**r**_ �� _**q**_ )  _**r**_ _**i**_ _,k∈_ [
_N_
_N_
- _P_ ( _**c**_ _k_ �� _**i**_ _k,_ _**q**_ )
_k_ =1
_**i**_ _k,k∈_ [ _N_ ]
- _P_ ( _**i**_ _j_ �� _**i**_ _j−_ 1 _, . . .,_ _**i**_ 1 _,_ _**r**_ ) _·_
_j_ =1 _**c**_ _,k∈_ [
- _P_ ( _**o**_ _m ∈O_ [+][��] _**c**_ _m,_ _**q**_ _,_ _**r**_ ) _P_ ( _**c**_ _m_ �� _**i**_ _m,_ _**q**_ )
_**c**_ _m_
=
_P_ ( _**r**_ �� _**q**_ )  _**r**_ _**i**_ _,k∈_ [
_N_
_**i**_ _k,k∈_ [ _N_ ]
- _P_ ( _**i**_ _j_ �� _**i**_ _j−_ 1 _, . . .,_ _**i**_ 1 _,_ _**r**_ ) _·_
_j_ =1 _**c**_ _m_
However,
 - _P_  - _**o**_ _m ∈O_ [+][��] _**c**_ _m,_ _**q**_ _,_ _**r**_  - _P_ ( _**c**_ _m_ �� _**i**_ _m,_ _**q**_ )
_**c**_ _m_
=
 - _P_ - _**o**_ _m ∈O_ [+][��] _**c**_ _m,_ _**q**_ _,_ _**r**_ - _P_ ( _**c**_ _m_ �� _**i**_ _m,_ _**q**_ ) +
_**c**_ _m∈C_ ( _δ,_ _**q**_ _,_ _**r**_ ) _**c**_ _m∈S/C_ (
 - _P_ - _**o**_ _m ∈O_ [+][��] _**c**_ _m,_ _**q**_ _,_ _**r**_ - _P_ ( _**c**_ _m_ �� _**i**_ _m,_ _**q**_ )
_**c**_ _m∈S/C_ ( _δ,_ _**q**_ _,_ _**r**_ )
(11)
(12)
_≥_ - _δP_ ( _**c**_ _m_ �� _**i**_ _m,_ _**q**_ ) = _P_ - _**c**_ _m ∈S/C_ ( _δ,_ _**q**_ _,_ _**r**_ )�� _**i**_ _m,_ _**q**_ - _δ_
_**c**_ _m∈S/C_ ( _δ,_ _**q**_ _,_ _**r**_ )
Taking (12) into (11), we obtain
_P_ - **T** [M] _M_ [(] _**[q]**_ [)] _[ ∈O]_ [+][��] _**[q]**_ - _≥_
_P_ ( _**r**_ �� _**q**_ )  _**r**_ _**i**_ _,k∈_ [
_N_
_**i**_ _k,k∈_ [ _N_ ]
- _P_ ( _**i**_ _j_ �� _**i**_ _j−_ 1 _, . . .,_ _**i**_ 1 _,_ _**r**_ ) _P_ - _**c**_ _m ∈S/C_ ( _δ,_ _**q**_ _,_ _**r**_ )�� _**i**_ _m,_ _**q**_ - _δ_
_j_ =1
=
_P_ ( _**r**_ �� _**q**_ )  _**r**_ _**i**_
_**i**_ _k,k∈_ [ _N_ ] (13)
- _P_ ( _**i**_ _m_ �� _**r**_ ) _P_ - _**c**_ _m ∈S/C_ ( _δ,_ _**q**_ _,_ _**r**_ )� _δ._
_**i**_ _m_
**Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation**
Since _**i**_ _m ∈TξCR/δ−δ_ ( _**q**_ _,_ _**r**_ ), we have _P_ - _**c**_ _m ∈S/C_ ( _δ,_ _**q**_ _,_ _**r**_ )� _δ ≥_ _ξ_ CR. Thus
LHS _≥_
- _P_ ( _**r**_ �� _**q**_ )
_**r**_ _**i**_
- _P_ ( _**i**_ _m_ �� _**r**_ ) _ξ_ CR =
_**i**_ _m_ _**r**_
_P_ ( _**r**_ �� _**q**_ ) _ξ_ CR (14)
_**r**_
However, for self-critique-revise, we have
_P_         - **T** [CR] _M_ [(] _**[q]**_ [)] _[ ∈O]_ [+][��] _**[q]**_         - =
_P_  - _**r**_ �� _**q**_  - _P_  - _**o**_ _∈O_ [+][��] _**q**_ _,_ _**r**_ _,_ _**c**_ [CR][�]
_**r**_
(15)
_P_  - _**r**_ �� _**q**_  - _ξ_ CR
_**r**_
_≤_
Combine (14) and (15) hence **T** [M] _M_ _[⪰][O]_ [+] **[ T]** _M_ [CR][.]


_Proof._
_P_ ( _∃_ _**i**_ _j, j ∈_ [ _N_ ] _,_ _**i**_ _j ∈Tϵ−δ_ ( _**q**_ _,_ _**r**_ )) = 1 _−_
_N_
- _P_ ( _**i**_ _k /∈Tϵ−δ_ ( _**q**_ _,_ _**r**_ )) = 1 _−_ (1 _−P_ ( _**i**_ _∈Tϵ−δ_ ( _**q**_ _,_ _**r**_ ))) _[N]_ (16)
_k_ =1
Since _P_ ( _**i**_ _∈Tϵ−δ_ ( _**q**_ _,_ _**r**_ )) _>_ 0, we have _P_ ( _∃_ _**i**_ _j, j ∈_ [ _N_ ] _,_ _**i**_ _j ∈Tϵ−δ_ ( _**q**_ _,_ _**r**_ )) = 1 when _N →∞_ .


-                               _Proof._ We prove this lemma by contradiction. Suppose _∃_ _**q**_ such that _∀δ,_ 0 _< δ <_ 1, _P_ _**i**_ _∈T_ _[η]_ = 0. This is
_δ_ _[−][δ]_ [(] _**[q]**_ _[,]_ _**[ r]**_ [)]
equivalent to
_∀_ _**i**_ _∼P_ ( _**i**_ �� _**r**_ ) _, P_                - _**c**_ _∈S/C_ ( _δ,_ _**q**_ _,_ _**r**_ )� _≤_ _[η]_ (17)
_δ_ _[,]_
where _**c**_ = _M_ ( _**i**_ _,_ _**q**_ ). Consider the target probability of **T** [M] _M_ [, we have]
 - _P_ - _**o**_ _∈O_ [+][��] _**c**_ [1: _N_ ] _,_ _**q**_ _,_ _**r**_ - - _[N]_
_**c**_ _k,k∈_ [ _N_ ] _k_ =1
- _P_ ( _**c**_ _k_ �� _**i**_ _k,_ _**q**_ )
_k_ =1
_P_ - **T** [M] _M_ [(] _**[q]**_ [)] _[ ∈O]_ [+][��] _**[q]**_ - =
_P_ ( _**r**_ �� _**q**_ )  _**r**_ _**i**_ _,k∈_ [
_N_
_**i**_ _k,k∈_ [ _N_ ]
- _P_ ( _**i**_ _j_ �� _**i**_ _j−_ 1 _, . . .,_ _**i**_ 1 _,_ _**r**_ )
_j_ =1 _**c**_ _,k∈_ [
(18)
Since **T** [M] _M_ [satisfies (][5][), then we obtain]
_N_
_P_    - _**o**_ _∈O_ [+][��] _**q**_ _,_ _**r**_ _,_ _**c**_ [1: _N_ ]� _≤_ _e_ _[Nλ]_ _P_    - _**o**_ [1: _N_ ] _∈O_ [+][��] _**q**_ _,_ _**r**_ _,_ _**c**_ [1: _N_ ]� = _e_ _[Nλ]_    - _P_    - _**o**_ _i ∈O_ [+][��] _**q**_ _,_ _**r**_ _,_ _**c**_ _i_    - _,_ (19)
_i_ =1
where the last equality comes from the fact that each output _**o**_ _i_ is defined as independently generated from _**c**_ _i_ . Taking (19) to
(18), we obtain
_N_
 - _e_ _[Nλ]_
_**c**_ _k,k∈_ [ _N_ ] _j_ =1
_P_ - **T** [M] _M_ [(] _**[q]**_ [)] _[ ∈O]_ [+][��] _**[q]**_ - _≤_
_P_ ( _**r**_ �� _**q**_ )  _**r**_ _**i**_ _,k∈_ [
_N_
- _P_ - _**o**_ _∈O_ [+][��] _**c**_ _j,_ _**q**_ _,_ _**r**_ - _P_ ( _**c**_ _j_ �� _**i**_ _j,_ _**q**_ )
_j_ =1
_**i**_ _k,k∈_ [ _N_ ]
- _P_ ( _**i**_ _j_ �� _**i**_ _j−_ 1 _, . . .,_ _**i**_ 1 _,_ _**r**_ ) _·_
_j_ =1 _**c**_ _,k∈_ [


 [�]
=
_P_ ( _**r**_ �� _**q**_ )  _**r**_ _**i**_ _,k∈_ [
_N_

_P_  - _**o**_ _∈O_ [+][��] _**c**_ _j,_ _**q**_ _,_ _**r**_  - _P_ ( _**c**_ _j_ �� _**i**_ _j,_ _**q**_ )
_**c**_ _j_
_**i**_ _k,k∈_ [ _N_ ]
_N_ _N_
- _P_ ( _**i**_ _j_ �� _**i**_ _j−_ 1 _, . . .,_ _**i**_ 1 _,_ _**r**_ ) _· eNλ_
_j_ =1 _j_ =1
_j_ =1
(20)
Consider the probability [�] _**c**_ _j_ _[P]_ - _**o**_ _∈O_ [+][��] _**c**_ _j,_ _**q**_ _,_ _**r**_ - _P_ ( _**c**_ _j_ �� _**i**_ _j,_ _**q**_ ), we have
[�] _**c**_ _j_ _[P]_        - _**o**_ _∈O_ [+][��] _**c**_ _j,_ _**q**_ _,_ _**r**_        - _P_ ( _**c**_ _j_ �� _**i**_ _j,_ _**q**_ ), we have
- [+]
_P_  - _**o**_ _j ∈O_ [+][��] _**c**_ _j,_ _**q**_ _,_ _**r**_  - _P_ ( _**c**_ _j_ �� _**i**_ _j,_ _**q**_ )
_**c**_ _j_
=
 - _P_ - _**o**_ _j ∈O_ [+][��] _**c**_ _j,_ _**q**_ _,_ _**r**_ - _P_ ( _**c**_ _j_ �� _**i**_ _j,_ _**q**_ ) +
_**c**_ _j_ _∈C_ ( _δ,_ _**q**_ _,_ _**r**_ ) _**c**_ _j_ _∈S/C_ (
 - _P_ - _**o**_ _j ∈O_ [+][��] _**c**_ _j,_ _**q**_ _,_ _**r**_ - _P_ ( _**c**_ _j_ �� _**i**_ _j,_ _**q**_ )
_**c**_ _j_ _∈S/C_ ( _δ,_ _**q**_ _,_ _**r**_ )
(21)
_≤_
 - _P_ ( _**c**_ _j_ �� _**i**_ _j,_ _**q**_ )
_**c**_ _j_ _∈S/C_ ( _δ,_ _**q**_ _,_ _**r**_ )
 - _δP_ ( _**c**_ _j_ �� _**i**_ _j,_ _**q**_ ) +
_**c**_ _j_ _∈C_ ( _δ,_ _**q**_ _,_ _**r**_ ) _**c**_ _j_ _∈S/C_ (
= _δ_ �1 _−P_ - _**c**_ _j ∈S/C_ ( _δ,_ _**q**_ _,_ _**r**_ )�� _**i**_ _j,_ _**q**_ �� + _P_ - _**c**_ _j ∈S/C_ ( _δ,_ _**q**_ _,_ _**r**_ )�� _**i**_ _j,_ _**q**_
**Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation**
Taking (21) back to (20), we obtain
_P_    - **T** [M] _M_ [(] _**[q]**_ [)] _[ ∈O]_ [+][��] _**[q]**_
- _δ_ + (1 _−_ _δ_ ) _P_ - _**c**_ _j ∈S/C_ ( _δ,_ _**q**_ _,_ _**r**_ )�� _**i**_ _j,_ _**q**_ ��
_≤_
_P_ ( _**r**_ �� _**q**_ )  _**r**_ _**i**_ _,k∈_ [
_N_ _N_
- _P_ ( _**i**_ _j_ �� _**i**_ _j−_ 1 _, . . .,_ _**i**_ 1 _,_ _**r**_ ) _· eNλ_
_j_ =1 _j_ =1
_N_
_**i**_ _k,k∈_ [ _N_ ]
_j_ =1
(22)
_≤_
_P_ ( _**r**_ �� _**q**_ )  _**r**_ _**i**_ _,k∈_ [
_N_ _N_
- _P_ ( _**i**_ _j_ �� _**i**_ _j−_ 1 _, . . .,_ _**i**_ 1 _,_ _**r**_ ) _eNλ_
_j_ =1 _j_ =1
_N_
_δ_ + (1 _−_ _δ_ ) _[η]_
_δ_
_**i**_ _k,k∈_ [ _N_ ]
_j_ =1
_[η]_ - _N_ _._
_δ_ _[−]_ _[η]_
= _e_ _[Nλ]_ [ �] _δ_ + _[η]_
This inequality holds for any 0 _< δ <_ 1 according to our assumption, thus taking _δ_ = _[√]_ ~~_η_~~ ~~,~~ we obtain
_P_           - **T** [M] _M_ [(] _**[q]**_ [)] _[ ∈O]_ [+][��] _**[q]**_           - _≤_ _e_ _[Nλ]_ (2 _[√]_ ~~_η_~~ _−_ _η_ ) _[N]_ _, ∀N >_ 0 _._ (23)
_√_
Using the property _[√]_ ~~_η_~~ _<_ 1 _−_
Using the property _[√]_ ~~_η_~~ _<_ 1 _−_ 1 _−_ _e_ _[−][λ]_, _e_ _[λ]_ (2 _[√]_ ~~_η_~~ _−_ _η_ ) _<_ 1, thus
_P_             - **T** [M] _M_ [(] _**[q]**_ [)] _[ ∈O]_ [+][��] _**[q]**_             - = 0 _, N →∞._ (24)
This contradicts the alignment chance assumption shown in (6).


**Analysis to Theorem** the condition _[√]_ _ξ_ CR _<_ **4.5** 1 _−_ **.** Th _√_ e maxim1 _−_ _e_ _[−][λ]_ um effectivenesseasy to be satisfied. It can be explained from 2 aspects. 1) (3) _ξ_ CR of the critique used in **T** [CR] _M_ [is relatively small, making] **Unclear cognitive**
**understanding of alignment goals.** Human pre-defined rules and suggestions are highly abstract and concise, which is
not a clear instruction for an unaligned model to understand. 2) **Hard to balance validity and generalizability.** Since
**T** [CR] _M_ [aims to use a set of unified moral codes to guide the LLM on various kinds of questions, to promote generalizability,]
the critiques used should be concise and highly summarized. However, this property hinders the revised response from
being helpful and question-specific, which is a fundamental conflict of Constitutional AI. The key point of the MATRIX
method lies in automatically generating instruction-specific critiques. By utilizing multi-agent social-scene simulation,
MATRIX can expose potential harms of the inter-agent interactions and provide additional information on original response
distribution. In the process of generating critiques for each interaction, the LLM continually shifts roles and perspectives to
generate a set of the most comprehensive and objective critiques of the instruction. The set of critiques that takes all the
agents into account will maximally align with social values
Then we provide our proof to Theorem 4.5 based on the assumption and lemmas.
_Proof._ Since for LLM _M_, it satisfies Assumption 4.4 with a valid _λ_ in (5). Then define _η_ = _ξ_ CR, apply Lemma A.6, we can
          -          -          -          find a _δ ∈_ [0 _,_ 1] satisfies _P_ _**i**_ _∈T_ _[η]_ _δ_ _[−][δ]_ [(] _**[q]**_ _[,]_ _**[ r]**_ [)] _>_ 0. Then apply Lemma A.5, _P_ _∃_ _**i**_ _j, j ∈_ [ _N_ ] _,_ _**i**_ _j ∈Tξ_ CR _/δ−δ_ ( _**q**_ _,_ _**r**_ ) = 1,
hence **T** [M] _M_ [could at least generate an interaction] _**[ i]**_ _[j][,]_ [ 1] _[ ≤]_ _[j][ ≤]_ _[N]_ [ such that] _**[ i]**_ _[j][ ∈T][ξ]_ _CR_ _[/δ][−][δ]_ [(] _**[q]**_ _[,]_ _**[ r]**_ [)][. Applying Lemma][ A.4]
hence **T** [M] _M_ _[⪰][O]_ [+] **[ T]** _M_ [CR][.]
**Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation**
_Table 5._ Summary of training hyperparameters for fne-tuning
PARAMETERS VALUE
NUMBER OF EPOCHS 3
LEARNING RATE 2 _×_ 10 _[−]_ [5]
LEARNING RATE DECAY COSINE
BATCH SIZE 1
GRADIENT ACCUMULATION STEPS 8
MAXIMUM SEQUENCE LENGTH 1024
DEEPSPEED ZERO STAGE 2
WEIGHT DECAY 0.0
BETA _β_ 0.1
_Table 6._ Template for fine-tuning LLMs. This template follows Vicuna (Chiang et al., 2023).


We employ FastChat (Zheng et al., 2023) to facilitate our fine-tuning; see Table 6 for the template of the base model. The
training parameters are summarized in Table 5.


- LLM Debate (Du et al., 2023): The number of agents is set to 3; the debate process is conducted over 3 iterations.
  - Best-of-N Sampling: The base model generates 16 responses, as in (Askell et al., 2021); reward model (OpenAssistant,
2023) is employed to select the one with highest reward.
  - Critique-Revise and RLAIF (Bai et al., 2022b): The number of criticizing and revising is set to 4. For Critique-Revise
(H), we fine-tune the base model with its revised response to harmful instructions; For Critique-Revise (HH), we
fine-tune the base model with its revised response to harmful and helpful instructions; We leverage DPO (Rafailov
et al., 2023) for RLAIF preference modeling.
  - Context Distillation and RLcd (Yang et al., 2024): We leverage DPO (Rafailov et al., 2023) for RLcd preference
modeling.
  - RAIN (Li et al., 2024): Given the low inference speed (approximately hours for a single sample for 30B LLMs on an
RTX3090), we limit generation to 10 samples for each dataset.
  - Mistake Analysis (Chen et al., 2024a): We employ the initial responses to helpful instructions for helpful data.
  - Stable Alignment (Liu et al., 2024): Peer feedback and ratings are collected using the base model. The number of
agents is set to 100; the interaction rounds are set to 500. Due to the unavailability of their training code, we select
responses with the highest ratings for Supervised Fine-Tuning (SFT) training.


Table 7 showcases the prompts utilized in our evaluation of GPT-4. The prompts are divided into two segments: the initial
segment delineates the evaluation task, and the subsequent segment presents a question along with two responses for pairwise
comparison.


Here, we present additional experimental results for various models, including Wizard-Vicuna-13B-Uncensored (TheBloke,
2024a), Mistral-7B-Instruct-v0.1 (Jiang et al., 2023), Alpaca-7B (Taori et al., 2023), and Text-Davinci-003. The comparisons
between the LLM with MATRIX and the baselines are illustrated in Table 8, Table 10, and Table 11 for Wizard-Vicuna-13BUncensored, Mistral-7B-Instruct-v0.1, and Alpaca, respectively. Additionally, Table 9 provides the comparison between the
MATRIX-tuned 13B LLM and the baseline models.
**Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation**
_Table 7._ Template for for the GPT-4 evaluation.
_Table 8._ Pairwise comparisons between the LLM (13B) with MATRIX and 6 baselines. Win, Tie, and Lose rates are reported with GPT-4
as the judger. The LLM with MATRIX consistently outperforms all of the baselines including GPT-3.5-Turbo on 4 evaluation datasets.
DATASET HH SAFE-RLHF ADVBENCH HARMFULQA
**LLM WITH MATRIX** _**vs.**_ WIN TIE LOSE WIN TIE LOSE WIN TIE LOSE WIN TIE LOSE
VANILLA 71% 16% 13% 84% 3% 13% 67% 22% 11% 78% 14% 8%
CRITIQUE-REVISE 70% 8% 22% 74% 8% 18% 63% 8% 29% 55% 11% 34%
SELF-ALIGN 76% 12% 12% 85% 9% 6% 68% 9% 23% 80% 12% 8%
CONTEXT DISTILLATION 63% 11% 12% 65% 6% 29% 61% 16% 23% 62% 10% 28%
LLM DEBATE 78% 12% 10% 76% 8% 16% 55% 17% 28% 72% 13% 15%
BEST-OF-N SAMPLING 69% 19% 12% 78% 8% 14% 59% 17% 24% 60% 10% 30%
CHATGPT (GPT-3.5-TURBO) 49% 11% 40% 65% 6% 29% 66% 5% 29% 55% 7% 38%
Text-davinci-003 was shut down on January 4th, 2024. The experiments here were conducted in October 2023. Tables 12
presents the pairwise comparison between text-davinci-003 with MATRIX and baselines on HH dataset. We see that
MATRIX enhances harmless generation on this 175B model.


Here we aim to show that it is adaptable for executing MATRIX simulations with an LLM different from the one used
for the initial response generation. To verify this, we conduct experiments by using three different LLMs to do the initial
response generation and three different LLMs to drive the MATRIX. We selected 100 random samples from the HH dataset
and employed GPT-4 to evaluate which response was superior, comparing responses generated by MATRIX with those from
GPT-3.5-Turbo.
We show the results in Table 13 where columns represent different initial response generation models, and rows correspond
to models used for social simulation. The results show that: i) It still works using different LLMs to do the initial response
generation and drive the MATRIX separately; ii) The performance of MATRIX increases when the model used for simulation
is getting larger.

B.5. Experiments on simulation with different models


Here we aim to analyze the time consumption among different methods. To address this, we conducted an analysis comparing
the efficiency and output quality of LLMs with MATRIX, MATRIX-Tuned LLMs, and the RAIN method (Li et al., 2024).
Our evaluation focused on averaging the time it takes for each method to process 10 samples from the HH dataset using the
**Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation**
_Table 9._ Pairwise comparisons between the MATRIX-tuned LLM (13B) and 6 baselines. Win, Tie, and Lose rates are reported with GPT-4
as the judger.
EVALUATION DATASET HH SAFE-RLHF ADVBENCH HARMFULQA
**MATRIX-TUNED LLM** _**vs.**_ WIN TIE LOSE WIN TIE LOSE WIN TIE LOSE WIN TIE LOSE
VANILLA 80% 11% 9% 83% 10% 7% 74% 18% 8% 82% 9% 9%
CONTEXT DISTILLATION 82% 9% 9% 82% 12% 6% 71% 17% 12% 73% 15% 12%
CRITIQUE-REVISE 77% 5% 18% 83% 7% 10% 79% 13% 8% 71% 14% 15%
MISTAKE-ANALYSIS ICLR’24 71% 11% 18% 78% 9% 13% 80% 12% 8% 72% 12% 16%
RLCD ICLR’24 62% 12% 26% 65% 12% 23% 61% 14% 25% 51% 32% 17%
RLAIF 75% 9% 16% 80% 10% 10% 69% 11% 20% 70% 21% 9%
CHATGPT (GPT-3.5-TURBO) 51% 7% 42% 60% 6% 34% 70% 6% 24% 47% 6% 47%
_Table 10._ Pairwise comparisons between the LLM (Mistral-7B-Instruct-v0.1) with MATRIX and 6 baselines. Win, Tie, and Lose rates are
reported with GPT-4 as the judger.
EVALUATION DATASET HH SAFE-RLHF ADVBENCH HARMFULQA
**LLM WITH MATRIX** _**vs.**_ WIN TIE LOSE WIN TIE LOSE WIN TIE LOSE WIN TIE LOSE
VANILLA 66% 10% 24% 52% 18% 30% 74% 24% 2% 74% 8% 18%
DEBATE 66% 16% 18% 58% 22% 20% 60% 30% 10% 64% 6% 30%
CRITIQUE-REVISE 73% 7% 20% 72% 9% 19% 61% 12% 27% 85% 5% 10%
CONTEXT DISTILLATION 60% 20% 20% 58% 14% 28% 46% 25% 29% 66% 7% 27%
SELF-ALIGN 51% 14% 35% 52% 26% 22% 54% 22% 24% 50% 14% 36%
RAIN 50% 20% 30% 60% 0% 40% 60% 40% 0% 60% 10% 30%
BEST-OF-N SAMPLING 58% 20% 22% 52% 16% 32% 56% 34% 10% 48% 26% 26%
Mistral-7B-instruct model. The findings from Table 14 reveal two key insights:
i) Inference Efficiency: Both LLM with MATRIX and MATRIX-Tuned LLM demonstrate superior performance over RAIN
in terms of inference efficiency. This improved performance can be attributed to the fact that RAIN’s methodology relies
on utilizing harmless examples for self-evaluation by the LLM, which in turn expands its input context and complicates
the inference process. Additionally, RAIN’s approach of employing backtracking to search for suitable tokens further
diminishes its inference speed.
ii) Cost-effective Fine-tuning: By fine-tuning LLMs with MATRIX, we preserve the harmless generation characteristic of
the LLM with MATRIX, effectively transforming the LLM into an aligned version that incurs no additional inference time
cost. This process not only enhances the model’s alignment with desired outcomes but does so without imposing extra time
penalties during inference. Note that MATRIX-Tuned LLM takes a slightly longer time than the original LLM to output
more comprehensive responses.


Here we aim to assess MATRIX’s effectiveness in diminishing social biases, we conducted an in-depth analysis using
the Bias Benchmark for QA (Parrish et al., 2022). This evaluation focused on the model’s ability to select the correct,
non-stereotypical answers in contexts where ambiguities could lead to biased responses. A key indicator of success in this
endeavor is the model’s accuracy: higher accuracy reflects a diminished propensity for choosing stereotypical answers.
We evaluated the model’s extent of social bias, both before and after MATRIX fine-tuning, with a focus on gender, age,
and race. Table 15 showed that the MATRIX-Tuned model was more accurate in choosing unbiased answers, leading to
a reduction in social stereotypes. This progress is due to MATRIX’s generation of bias-neutral responses to the biased
question within the training dataset.


Here we present the prompts used in MATRIX in Table 17. They are all straightforward. Table 18 shows an example of
generating scenario-specific with MATRIX. MATRIX creates relevant social roles and enables their interaction, leading to
consequence-aware critique.
**Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation**
_Table 11._ Pairwise comparisons between the LLM (Alpaca-7B) with MATRIX and 6 baselines. Win, Tie, and Lose rates are reported with
GPT-4 as the judger.
EVALUATION DATASET HH SAFE-RLHF ADVBENCH HARMFULQA
**LLM WITH MATRIX** _**vs.**_ WIN TIE LOSE WIN TIE LOSE WIN TIE LOSE WIN TIE LOSE
VANILLA 77% 3% 20% 73% 8% 19% 73% 16% 11% 67% 6% 27%
DEBATE 76% 7% 17% 65% 15% 20% 74% 21% 5% 72% 9% 19%
CRITIQUE-REVISE 86% 8% 6% 81% 9% 10% 87% 8% 5% 86% 3% 11%
CONTEXT DISTILLATION 83% 8% 9% 70% 10% 20% 79% 12% 9% 58% 15% 27%
SELF-ALIGN 53% 17% 30% 46% 17% 37% 41% 22% 37% 44% 18% 38%
RAIN 80% 0% 20% 80% 0% 20% 80% 10% 10% 60% 20% 20%
BEST-OF-N SAMPLING 68% 10% 22% 67% 12% 21% 60% 30% 10% 64% 14% 22%
_Table 12._ Experiments of text-davinci-003 with MATRIX on HH.
DATASET HH
**LLM WITH MATRIX** _**vs.**_ WIN TIE LOSE
VANILLA 90% 10% 0%
CRITIQUE-REVISE 90% 7% 3%
LLM DEBATE 58% 30% 12%
_Table 13._ Pairwise comparisons between the LLM with MATRIX (where three different LLMs generate the initial response and three
different LLMs drive the MATRIX) and GPT-3.5-Turbo. Win, Tie, and Lose rates are reported with GPT-4 as the judger.
INITIAL RESPONSE MODEL WIZARD-VICUNA-7B WIZARD-VICUNA-13B WIZARD-VICUNA-30B
**SIMULATION MODEL** WIN TIE LOSE WIN TIE LOSE WIN TIE LOSE
WIZARD-VICUNA-7B 41% 52% 7% 50% 42% 8% 47% 44% 9%
WIZARD-VICUNA-13B 46% 42% 12% 43% 38% 19% 46% 40% 14%
WIZARD-VICUNA-30B 61% 31% 8% 64% 26% 10% 66% 28% 6%
_Table 14._ Comparison of time consumption and response quality between RAIN, original LLM, LLM with MATRIX,and MATRIX-Tuned
LLM.
METHOD RAIN ORIGINAL LLM LLM WITH MATRIX MATRIX-TUNED LLM
TIME 2826.35S 3.46S 341.63S 6.05S
WIN / TIE / LOSE VS. RAIN    - 30 / 10 / 60 80 / 0 / 20 60 / 40 / 0
_Table 15._ Evaluation of social bias in the vanilla wizard-vicuna-7B model and its MATRIX-Tuned version using the Bias Benchmark for
QA. Higher accuracy reflects a diminished propensity for choosing stereotypical answers.
CATEGORY GENDER AGE RACE
VANILLA 0.42 0.34 0.44
MATRIX-TUNED 0.46 0.48 0.50
_Table 16._ Pairwise comparisons between the MATRIX-tuned LLM (30B) and baselines. Win, Tie, and Lose rates are reported with GPT-4
as the judger, focusing solely on harmlessness.
DATASET HH
**MATRIX-TUNED LLM** _**vs.**_ WIN TIE LOSE
SELF-ALIGN 76% 16% 8%
CONTEXT DISTILLATION 68% 22% 10%
RLCD 46% 38% 16%
RLAIF 72% 18% 10%
**Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation**
**Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation**
**Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation**


Here we present the details of the human evaluation experiment. We randomly sample 100 questions from the Safe-RLHF
dataset with 14 categories of harmful questions. To reduce the randomness of personal subjective evaluations and other
interferences, we randomly recruit 35 volunteers. Each volunteer will be assigned 25 questions, thus ensuring that the
quality of the model’s response to each question is reviewed by at least 8 people.
The review of the model’s quality is conducted on a simple webpage, including the homepage and the specific scoring
page. For the homepage shown in Figure 8, each volunteer needs to read the scoring principle (adopted from (Askell et al.,
2021)) and refer to the usage policy released by OpenAI to get familiar with common principles corresponding to harmless
model responses. Then volunteers are reminded to evaluate the quality of the model as objectively as possible and to
provide choices that are as distinctive as possible. For the scoring page, volunteers select a better-quality answer from two
anonymous model responses provided for a question. If they believe the quality of the two answers is comparable, they click
”tie”.
_Figure 8._ The main page used for human value alignment experiment.
_Figure 9._ The scoring page for one specific question. Here users have three different options, the first model, the second model, and tie.
**Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation**