Self-education is, I firmly believe, the only
kind of education there is.
Isaac Asimov
Currently, aligning through inductive bias is one of the most promising directions for achieving
automated alignment. Inductive bias (Mitchell, 1980) is a set of essentially assumptions or constraints
that guide model learning and decision-making processes. By carefully selecting and implementing
suitable inductive biases, we can steer models towards behaviors and decisions that are more likely
to meet human standards and expectations, which can then generalize to unseen data distributions.
Compared to other methods of achieving automated alignment, aligning through inductive bias
offers two primary advantages:
1) It does not require additional supervisory signals beyond the model itself, thus avoiding the high
cost of obtaining additional annotated data. This is particularly relevant given the current scenario
where training data is becoming scarce or has already been exhausted [1] (Xue et al., 2023).
2) It has the potential to address the scalable oversight problem (Bowman et al., 2022). As the
potential of LLMs continues to scale, it becomes challenging for humans to provide supervisory
signals that surpass their own level of knowledge. However, through inductive bias, models can
continuously self-improve, transcending the limitations of human knowledge.
1. Within the context of alignment discussed in this paper, we expect the model to continuously improve its helpfulness,
thereby providing more effective assistance for alignment process. The scope of alignment actually represents _the post_
_training process_ rather than steering the models.
After conducting a thorough review of the relevant literature, we find that current efforts towards
self-improvement solely through language model itself can be decomposed into a set _H_ of five
inductive biases. These inductive biases fall into two broad categories: 1) Those stemming from
inherent features of LLMs (Section3.1), and 2) Those arising from the organization of LLMs (Section3.2). The
objective of each of these inductive biases can be summarized as a simple rule: **Heuristically**
**transform test-time computation into the model’s alignment.** (Snell et al., 2024) The additional
computation of each inductive bias is depicted in the shaded regions of the illustrative figures.
For each type of inductive biases, we will begin by introducing its origin. Following this, we will
enumerate the works that employ this inductive bias as a single-step policy improvement operator.
Next, we will discuss the works that iteratively train with it, aiming for continuous improvement.
Finally, we will address open research problems associated with the given inductive bias.


LLMs possess intrinsic features that can act as inductive biases. These features largely arise from
the pre-training of deep Transformer networks on massive datasets (H1, H3), while some also stem
from preliminary alignment procedures aimed at enhancing the models’ helpfulness (H2). In this
section, we will summarize three key inductive biases, as depicted in Figure 3. It is important to
note that these inductive biases are not entirely independent; instead, they represent three different
perspectives on automated alignment based on the features of LLMs.
3.1.1 H1: UNCERTAINTY AS A INDICATOR OF HELPFULNESS
Probability distributions from models can represent uncertainty. As Kadavath et al. (2022) discovered,
when prompts are suitably designed, the responses obtained from _pre-trained LLMs_ can be _well-_
_calibrated_, and the degree of calibration can scale with the number of parameters and the number of
exemplars. In other words, the higher the probabilities assigned by an LLM to a given answer, the
more likely that answer is to be correct. This hypothesis has also been validated by Wang et al. (2021)
and He et al. (2023). Similarly, Manakul et al. (2023) found a correlation between the probabilities
output by the aligned models and factuality.
In the machine learning literature, early applications of this inductive bias were evident in a
series of works using self-training (Scudder, 1965) for semi-supervised learning (Nigam and Ghani,
2000; Amini and Gallinari, 2002). The basic paradigm of these works involves using learners
trained on labeled data to continue learning from confidently classified unlabeled data, thereby
enhancing supervised learning performance with unlabeled data. This approach has been applied
in classification tasks with methods such as Pseudo-label (Lee et al., 2013; Ferreira et al., 2023)
and Entropy Minimization (Grandvalet and Bengio, 2004). He et al. (2020) extended this approach
to sequence generation NLP tasks, highlighting that biased sampling and noise perturbation are
key factors for the success of self-training in these tasks. Pace et al. (2024) extended the paradigm
of self-training to the alignment problem, improving the robustness of reward models by allowing
them to iteratively learn from the highest-scoring and lowest-scoring answers in the candidate pool
generated for a query.
The frequency of specific answers also reflects uncertainty. Therefore, synthesizing candidate
answers from multiple samplings can lead to better performance than relying on a single sample.
This approach is particularly effective when LLMs are used in tasks requiring deliberate thinking
(e.g., solving math problems), as a single Chain-of-Thought (CoT) (Wei et al., 2022) reasoning path
Figure 4: Calibration plot observed by Kadavath et al. (2022); He et al. (2023); OpenAI (2023c);
Zhang et al. (2024). The x-axis represents the probability associated with the model’s
output, while the y-axis indicates the probability that the answer is correct. In the low
probability region, the gray area may reflect the “don’t know” responses substituting
some low-confidence answers. In the high probability region, the gray area signifies
overconfidence. The comparison between pre-trained LLM and aligned LLM demonstrates
that alignment for helpfulness can result in miscalibration, which is harmful to iterative
self-improvement.
can sometimes fall into local optima and generate plausible but unfaithful answers. Self-Consistency
(Wang et al., 2023e), by aggregating multiple reasoning paths through a weighted sum, can alleviate
this problem by marginalizing the model’s likelihood to the reasoning path. Interestingly, it was also
found that an unweighted sum (i.e., majority vote) can achieve comparable performance, which is
attributed to the “similarly likely” probabilities over all reasoning paths. Wang and Zhou (2024)
further discover that the presence or absence of a CoT reasoning path correlates with the probability
of the final answer when inference is conducted without any prompting techniques.
To solidify such enhancement, Self-Improve (Huang et al., 2023a) views CoT with SelfConsistency as a policy improvement operator, substantially improving the reasoning capabilities and
potential of LLMs through iterative learning of the reasoning paths obtained from Self-Consistency.
Zhang and Parkes (2023) demonstrate that LMs could self-improve on large number addition problems by curriculum “distilling” the CoT answers to the direct answers without explicit reasoning.
Quiet-STAR (Zelikman et al., 2024) considers the advantage of the influence of rollout rationale on
the probabilities of subsequent tokens as a feedback signal, encouraging the model to generate a
more helpful implicit thought process using reinforcement learning techniques. Li and Qiu (2023)
show that memory mechanisms can also be used for consistency-based self-improvement instead of
parameter learning.
**H1: Discussion** For aligned models, maintaining calibration and uncertainty remains crucial,
as miscalibration can undermine the potential for iterative self-improvement. Numerous studies
(Kadavath et al., 2022; He et al., 2023; OpenAI, 2023c; Zhang et al., 2024) have noted that the
alignment process can impair the calibration of LLMs (as shown in Figure 4). This observation
is reasonable for several reasons: 1) The current superficial alignment process aims to steer the
model away from generating harmful or incorrect answers. This involves replacing the probability of
incorrect answers with the probability of rejection responses to some extent, creating a gray area
in the low probability region. 2) During the alignment process, the model also learns the response
format. The increase in confidence regarding the response format can somewhat affect the confidence
in the answers themselves (He et al., 2023). Moreover, the model relearns the correct answers,
which can lead to overconfidence (represented by the gray area in the high probability region). The
extremization of the probability distribution can be more pronounced in self-alignment (Wu et al.,
2024b), given that it is an iterative process involving self-sampling and training. This implies that all
tokens are already in a very high probability distribution, making them more likely to be sampled as
responses.
When the model becomes overconfident, it leads to a decrease in the diversity and exploratory of
the model’s generated outputs. To mitigate this issue, one promising approach is to use inference-time
interventions (e.g., high temperature (Kadavath et al., 2022), fidelity (Zhang et al., 2024)) to decrease
the expected calibration error. Another potential solution is to filter the pseudo-labeled samples
to avoid harmful repeated training, which requires understanding when unlabeled samples will be
effective (Grandvalet and Bengio, 2004).
3.1.2 H2: LLMS CAN JUDGE, CRITIC, REFINE AND MORE
Pre-trained LLMs often struggle to directly respond to instructions. However, the widespread
adoption of imitation learning (Chiang et al., 2023) and feedback learning (Bai et al., 2022a) has
significantly enhanced the zero-shot helpfulness of LLMs. Leveraging the reasoning abilities elicited
and enhanced by these general helpfulness improvements, a series of works have emerged, utilizing
the model’s capabilities to enhance response quality and safety through judging, critiquing, refining,
and more.
_**Judge**_ refers to determining the quality of model responses. The judgment standards are typically
incorporated into instructions as principles or guidelines, enabling regulators to oversee LLM
behavior in a more scalable manner (Bai et al., 2022b; Yuan et al., 2024c) compared to heavy
reliance on human annotators for feedback (Bai et al., 2022a). This approach allows for timely
regulation, aiding in the flexible and controlled alignment process of language models, which can
help prevent issues like reward hacking during iterative training (Sun et al., 2023c) and facilitate
on-policy reinforcement learning training (Guo et al., 2024a).
Self-judging can manifest in two primary forms: 1) Differentiating the relative quality of two
responses (AI Feedback (Bai et al., 2022b)), resulting in evaluation outcomes represented as a partial
order. For instance, Tan et al. (2023) employ prompts to compare which answer better adheres to
the HHH principles. They then distill the chosen option back into the model to further enhance its
judging capability. Bai et al. (2022b) prompted the model to select superior responses based on
sampled principles and subsequently employed a preference process to achieve a Pareto improvement
of the model. 2) Providing an absolute score for a response (LLM-as-a-judge (Zheng et al., 2024b)),
with the evaluation results in scalar form. Yao et al. (2023b), Besta et al. (2024) and Xie et al. (2023)
introduce real-time evaluation modules for thought states during reasoning process. These modules
serves as a prior during the search process, assisting the model in exploring the action space for
questions requiring deliberate thinking. Similarly, RAIN (Li et al., 2024h) utilizes a binary scoring
prompt for self-evaluating whether the generation is likely to be harmful, thereby enhancing response
safety combined with an inference-time tree search. Yuan et al. (2024c) employ a five-point judge
prompt to score the model’s instruction-response outputs, then converted the scores into a partial
order for iterative training using DPO.
Recalling H1, it becomes evident that H1 serves as a foundation for H2, given that the accuracy
of the judge is directly tied to the calibration of the LLM. Consequently, H2 will only be effective if
H1 is valid (Bai et al., 2022b).
_**Critique**_ refers to generating modification suggestions. By leveraging LLM itself for critique, the
suggestions can address errors and deficiencies, such as mistakes in summarization (Saunders et al.,
2022), machine translation (Fernandes et al., 2023), math reasoning (Lin et al., 2024b), decisionmaking and programming tasks (Saunders et al., 2022; Shinn et al., 2023). The suggestions can also
pertain to abstract value criteria, such as the HHH related principles (Chen et al., 2024e; Bai et al.,
2022b).
_**Refine**_ refers to the ability that LLMs can improve the given text. Most of the work on self-refine
is based on natural language reasons provided by the critic module to modify the response (Bai et al.,
2022b; Tan et al., 2023; Madaan et al., 2023; Shinn et al., 2023). Some research also demonstrates
the possibility of making modifications directly based on scalar rewards (Shinn et al., 2023). The less
informative critic can be more challenging for LLMs since they must complete more information by
themselves through reasoning. Another line of work uses LLMs to refine the prompts themselves
(Fernando et al., 2023; Yang et al., 2023a).
_**Other:**_ A helpful LLM can serve in various capacities to assist in the alignment process, effectively replacing human guidance. For example, it can vote on the quality of intermediate states of
thought (Yao et al., 2023b), verify outcomes based on predicting conditions in questions (Weng et al.,
2023), and automatically generate, filter (Yue et al., 2024b), and evolve instructions (Wang et al.,
2023f; Li et al., 2024f; Xu et al., 2023a), among other tasks.
Regarding the persistence process, the improvements derived from these methods can be further
distilled into the model through SFT (i.e., Expert Iteration), DPO, RM-PPO, and other techniques.
Additionally, the judge / critique - refine process can be conducted iteratively.
**H2: Discussion** As the abilities of judging, critiquing, and refining are increasingly incorporated
into model feedback and learning processes, there is a need for systematic evaluation of these
capabilities. In this context, several research directions are worth pursuing:
1) Benchmarking the performance of existing models on these atomic abilities, as exemplified by
works such as Sun et al. (2024a) and Lin et al. (2024b).
2) Conducting causal studies on the formation process of models’ judging, critiquing, and refining
abilities, and investigating what forms of pre-training and fine-tuning data can influence these
abilities. This helps to design specific pipelines to targeted enhance these capabilities (Wang et al.,
2024g), and even train specialized models like CriticGPT (McAleese et al., 2024).
3) Evaluating the effects of distribution shift on these abilities. Do models still possess reliable
evaluation and improvement capabilities if they have not been trained on the corresponding
instruction and response pairs? This is particularly pertinent to the scalable oversight problem,
which assumes the absence of direct supervision for instructions.
4) Gathering empirical evidence to demonstrate that self self-critique, judge, and refine abilities can
enhance model performance in fair and reasonable experimental settings. Some works point out
Figure 5: Illustrations of aligning through three representative inductive biases which stem from the
organization of LLMs.
that the improvement may come from the use of stronger models (Sharma et al., 2024) and golden
labels (Huang et al., 2023a).
3.1.3 H3: LLMS CAN EFFECTIVELY LEARN IN-CONTEXT
In-context learning (ICL) refers to the ability of LLMs to initialize a task-specific model with
exemplars or experiences during inference (Brown et al., 2020). Given that certain studies (Dai et al.,
2023; von Oswald et al., 2023) suggest parallels between ICL and parameter gradient descent, it is
plausible to regard it as a versatile and effective “learning” method.
From the perspective of automated alignment, ICL offers an efficient means to cold-start from a
pre-trained LLM. With the assistance of ICL, just a few conversational samples in-context can yield
a somewhat aligned model (Ganguli et al., 2022; Sun et al., 2023d; Lin et al., 2024a). Similarly,
by prepending a few annotated exemplars in-context, ICL can also elicit judge and critic abilities
of pre-trained LLM to some extent, or enhance the performance compared to the zero-shot setting
(Bai et al., 2022b). Additionally, ICL presents a potential method for adaptive alignment (Xu et al.,
2023c) to different social norms and regulations.
However, prepending few-shot exemplars in the context above can make inference inefficient
(Gim et al., 2023) and interfered with the unrelated queries (Shi et al., 2023). Therefore, selfgenerated labels obtained from ICL could be directly used as pseudo labels, and distilled back
into the LLMs only paired with the query. This paradigm is known as Context Distillation (Askell
et al., 2021; Snell et al., 2022). For instance, in the alignment process of Llama-2 (Touvron et al.,
2023), Context Distillation is used to alleviate the problem of long-term dependencies of system
prompts. For Llama-3 (Dubey et al., 2024), context distillation is utilized to steer the model towards
generating more readable and well-documented code. In Dromedary (Sun et al., 2023d), the base
language model is transformed into a safe and helpful aligned model with minimal annotations
by directly training on samples acquired from multiple ICL processes. Padmanabhan et al. (2023)
demonstrate that Context Distillation can also be used to inject new knowledge to models by learning
continuations from entity definitions. Furthermore, Yang et al. (2024b) illustrate the effectiveness of
distilling the preference pair generated by contrastive in-context constraints back into the model.
Additionally, the learning content of ICL can also encompass exploratory experiences (Shinn
et al., 2023) and tool definitions (Yao et al., 2022; Tang et al., 2023). In other words, agents equipped
with tools and experiences can potentially outperform those without. This suggests a similar potential
in distilling back the trajectories improved by experience and tools to continuously enhance the same
model.
**H3: Discussion** Unfortunately, the black box nature of ICL itself poses a significant challenge
to alignment (Anwar et al., 2024). Without a comprehensive understanding of how LLMs learn
in-context, the context distillation approach may introduce problems by potentially amplifying biases
and errors inherent in the ICL process of the models. Moreover, the ability of long in-context learning
(Agarwal et al., 2024; Li et al., 2024e) warrants further exploration, as it facilitates more efficient
distillation and is crucial for scalable oversight settings where models need to comprehend lengthy
professional documents or extensive self-play histories.

Towards Scalable Automated Alignment of LLMs: A Survey

3.1 Inductive Bias from Features of LLMs

Boxi Cao
Keming Lu
Xinyu Lu
Jiawei Chen
Mengjie Ren
Hao Xiang
Peilin Liu
Yaojie Lu
Ben He
Xianpei Han
Le Sun
Hongyu Lin
Bowen Yu


In addition to the inductive bias originating from the shared features of LLMs, another set of biases
arises from the composition or organization of multiple LLMs, as depicted in Figure 5. Based
on whether the relationships between the constituent LLMs are cooperative or adversarial, two
representative inductive biases emerge: “Task Decomposition” and “Self-play”. It is noteworthy that,
as this field progresses, we anticipate subsequent literature will adopt more complex organizational
and learning structures. Both adversarial and collaborative modalities may form integral components
of sophisticated agent systems. However, at the current stage, task decomposition and self-play serve
as practical taxonomies. Subsequent sections will delve into these concepts in detail.
3.2.1 H4: TASK DECOMPOSITION
Task decomposition has long been regarded as an effective approach to tackling complex problems (Lee and Anderson, 2001). For example, in cooperative games grounded in collective rationality, the overall benefits accrued by an alliance exceed the sum of individual gains (Shapley, 1971).
Moreover, the divide-and-conquer paradigm and recursion are well-established and effective means
employed in algorithm design for addressing problems of substantial scale and complexity (Hoare,
1961; Wilf, 2002).
The discussion of this paradigm can be traced back to the assumption of factored cognition
(Ought, 2017). It advocates that cognition tasks can be recursively decomposed. If an AI or human
encounters a task that is difficult to solve, it can decompose the task, assign the decomposed problems
to a series of its own copies for parallel processing, and finally merge these results. The copies focus
on short-term work and work independently. A series of prompting methods implicitly or partially
adopts the factored-cognition assumption for automated alignment. For instance, Zhou et al. (2023b)
and Wang et al. (2023b) prompt the LLM to decompose the problem, then guide it to sequentially
solve the sub-problems. It is also believed that task decomposition is an effective method for solving
Easy-to-Hard generalization (Zhou et al., 2023b), that is, constructing decomposition prompts on
simple samples and filling them in-context allows LLM the potential to generalize to difficult samples.
Khot et al. (2023) further implement recursive task decomposition.
Based on the assumption of factored cognition, Iterative Distillation and Amplification (IDA)
(Christiano et al., 2018) views each decomposition-merging process as a form of _amplification_ and
considers learning from the final merged results as a form of _distillation_ . Although the original IDA
paper builds this theoretical framework in a human-in-the-loop manner, where humans supervise
the initial task decomposition step, it is likely that this process can be initiated without much human
oversight given H1, H2, and H3 (Zhang and Parkes, 2023).
Notably, IDA represents a promising avenue towards achieving scalable oversight, making
it possible to address long-horizon tasks that are difficult for humans to directly supervise, by
decomposing tasks into more tractable sub-problems. For example, labels for data points like
“peer-review this survey” can take several months to collect in real world. Such problems can be
tackled more quickly through factored cognition. Although some work partially demonstrates the
effectiveness of IDA on real-world tasks like book-length summarization (Wu et al., 2021a) and
complex code bug fixing (Wen et al., 2024), this ideology still relies on a set of crucial assumptions: 1)
It is still unclear whether decompose a problem is the hardest part of solving it, if the cognitive burden
cannot be distributed, IDA may struggle to take effect. 2) The error won’t accumulate. Although this
paradigm does not require the collaboration between agents to be efficient (Christiano et al., 2018),
too many errors can still be problematic. 3) The extent to which tasks can be parallelized. If the
task-solving process is largely sequential, the time to collect signals might increase, but this appears
to be a minor issue given the current deployment speed of LLMs. Overall, since these assumptions
are difficult to prove or falsify, we advocate for more empirical research in this direction.
3.2.2 H5: SELF-PLAY
Complexity emerges from adversariality (Bansal et al., 2018). Self-play refers to a paradigm where
an agent learns by iteratively playing games _against_ itself, a form of non-cooperative game (Nash
et al., 1951) in which each agent aims to maximize its own utility. It serves as the foundation of many
successful specialized superhuman AI systems like AlphaGo Zero (Silver et al., 2018) and StockFish
(StockFish, 2023). Given this success, self-play seems to be a potential approach for enabling
general proposed superhuman intelligence from LLMs. Two representative self-play methods are
the Generator-Discriminator and the Debate approaches, with the latter involving _N ≥_ 2 adversarial
generators and one discriminator in a gaming environment.
**H5.1: Generator-Discriminator** In the Generator-Discriminator self-play framework, the role of
the discriminator is to evaluate the outputs produced by the generator, determining whether these
outputs are of high or low quality.
As discussed in H2, the judge and critic model is commonly considered a type of discriminator.
For example, Yuan et al. (2024c) utilized rewards from LLM-as-a-Judge to identify high-quality and
low-quality responses from the generator, optimizing the generator towards the higher quality ones.
However, the adversarial setting between the discriminator and the generator is limited because the
only assumption is that the discriminator’s ability can be improved with general helpful training.
The discriminator remains almost static (the prompts are unchanged) during training, making it
possible for the generator to be over-optimized against the discriminator, leading to reward hacking.
Effectively improving the judge and critic module alongside the generator is thus a crucial problem.
One reasonable approach is to formulate the training process as an adversarial game (Cheng et al.,
2023), where the policy and reward model are updated alternatively via a min-max loss. Another
way to introduce a more adversarial setting is to optimize the gaming problem at inference time, as
demonstrated in the Consensus Game by Jacob et al. (2024). It employs the piKL no-regret learning
algorithm to iteratively update the strategies of both the generator and discriminator, converging to a
Nash equilibrium. This equilibrium strategy is then used to rank candidate responses, prioritizing
those agreed upon by both players.
As Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) have become a wellestablished class of methods in traditional NLP (Zhang et al., 2016; Wu et al., 2021b), another line of
work involves using a GAN-like discriminator to distinguish between the model’s current predicted
distribution and the golden distribution. For instance, Chen et al. (2024g) find that a specific type
of iterative DPO training, which consistently treats the policy-generated responses as negative and
the golden responses as positive, can be viewed as a self-play process. In this process, the implicit
reward function of DPO serves as a discriminator between the model’s predictions and the golden
samples. Expanding on this, Shaikh et al. (2024) further add replay comparison signals between
earlier iteration of models and the golden, and comparisons between a model and its successive model
in the self-play process. However, for open-ended questions, the golden distribution is sometimes
still suboptimal, and this setting precludes the possibility of generating responses that are better than
the golden ones.
**H5.2: Debate** The debate paradigm (Irving et al., 2018) is largely inspired by factored cognition
and AlphaGo (Silver et al., 2018). In the learning algorithm of AlphaGo, three distinct components
are integrated: a player, a counterpart (itself), and a value model that evaluates the win rate associated
with each board state. By employing Monte Carlo Tree Search (MCTS), the algorithm conducts
rollouts, which are simulated self-play trajectories that extend until a game’s conclusion. These
rollouts enhance the accuracy of the value estimates through backward updates based on the outcomes,
concurrently refining the policy by capitalizing on strategies that have previously led to victories.
The game of Go shares similarities with tackling the scalable oversight problem using natural
language debate. In the beginning or middle of a Go game, even experienced experts may find
it difficult to judge which side has a higher probability of winning, just as humans have a small
probability of correctly judging problems that exceed human knowledge levels. However, as the
game nears its end, the outcome usually becomes clear that even a non-expert judge can confidently
evaluate the board generated by the Go masters. For a debate competition, the winner can typically be
summarized by the judges. This analogy illustrates a crucial point: through the skillful introduction
of adversarial processes, the burden of oversight for complex problems can be significantly reduced.
This provide a possible oversight solution to build trustworthy superhuman AI systems. Irving
et al. (2018) show that honesty is better strategy than lie in debate paradigm through proof-ofconcept experiments. As an extension of this, Brown-Cohen et al. (2023) propose a new set of
debate protocols, wherein the honest strategy can _always succeed_ through a simulation involving
only a polynomial number of steps. Khan et al. (2024) conduct a thorough empirical study on the
feasibility of implementing the debate paradigm on LLMs: it was found that the debate paradigm
can significantly enhance truthfulness, and more persuasive (Anthropic, 2024a) debaters lead to
more truthful outcomes. Furthermore, Kirchner et al. (2024) demonstrate that a prover-verifier game
employing a powerful honest prover, a potent malicious prover, and a comparatively weaker verifier
can generate more readable outputs, thus enabling more effective human oversight of the stronger
model.
Apart from the classic natural language debate, a growing body of research has explored the
implementation of the debate paradigm across diverse game scenarios to enhance specific model
features. A representative arena is the _bargaining task_ (Nash et al., 1950). Fu et al. (2023a) focus on
zero-sum variants of bargaining, where the balloon seller aims to sell at a higher price while the buyer
seeks a lower price. They observed significant variations in bargaining capabilities among different
LLMs and their capacity to learn from play experiences and feedback. Cheng et al. (2024) implement
the adversarial language game _adversarial taboo_ (Yao et al., 2021), where an attacker and a defender
engage in a conversation centered around a target word visible only to the attacker. The attacker
subtly induces the defender to unconsciously utter the target word, while the defender tries to avoid
doing so and guess the word from the context. Both players acquire basic gaming skills through
imitation learning from a teacher LLM and then refine their strategies through self-play. Interestingly,
less capable player LLMs not only improve their win rates in this specific game but also enhance
their general reasoning abilities. Ma et al. (2023a) introduce the _red-teaming game_, a more intricate
adversarial team game where LLMs are initialized as a joint set of red-teaming policies to prompt
the target LLM to produce harmful content. They propose a solver to ensure the final meta-strategy
approximates a Nash equilibrium within a certain _ϵ_ margin. Zheng et al. (2024c) suggests addressing
the alignment problem by allowing an attacker to prompt a defender LLM to generate answers that
might result in low rewards, while the defender tries to maximize the rewards of these prompts. The
solution of this game is considered an iterative min-max optimization process with constraints.
3.2.3 DISCUSSION
Task decomposition and self-play both necessitate LLMs functioning as agents. Indeed, although
research on agents is already very prosperous, the current capabilities of LLMs as agents are still very
limited. They still struggle to complete tasks that would take humans several hours to finish (OpenAI,
2024). Therefore, focusing on improving the capabilities of LLMs-as-agents remains an important
future research direction. This includes understanding and modeling the human objectives in the
most economically valuable tasks, and developing effective reasoning abilities for ultra-long thought
process.
Meanwhile, the challenge of aligning LLMs as agents is more complex compared to aligning
them as chatbots, as it requires considerations of behavior-level alignment (Pan et al., 2023; Yuan
et al., 2024b), the dynamics of environment and self-constraints (Garrabrant and Demski, 2018;
Shavit et al., 2023; Yang et al., 2024f). More importantly, while adversarial self-play may enhance
agent capabilities in long-horizon tasks, it can also give rise to the emergence of more deceptive (Hubinger et al., 2024), persuasive and autonomous agents (Tao et al., 2024). Such developments could
have significant social impacts and ethical risks, such as the potential for models to generate more
persuasive articles than humans, which could be exploited for political manipulation. Encouragingly,
several prominent model engine providers have taken steps to monitor and mitigate these potential
side effects. For example, OpenAI’s Preparedness team has established benchmarks for assessing
persuasion and autonomy (OpenAI, 2023a), categorizing model risks into four levels and imposing
development and deployment restrictions based on risk thresholds. Additionally, third-party organizations are contributing to the development of robust safety frameworks for highly capable agents
(METR, 2024).

3.2 Inductive Bias from Organization of LLMs


Strong to Weak Distillation
_SFT/DPO/IPO..._
Weak to Strong Alignment


Figure 6: The illustrations of representative studies for aligning through behavior imitation.
Furthermore, an even more intricate problem lies in proving the theoretical safety and trustworthiness of multi-agent systems. Although research in this area is nascent (Yang and Wang, 2020;
DiGiovanni and Zell, 2021), advancements in game theory (Hazra and Anjaria, 2022), automated theorem proving techniques (Polu and Sutskever, 2020) and real-world simulation technology (Brooks
et al., 2024) may offer insights into addressing this challenge.


Imitation is the first instinct of the
awakening mind.
Maria Montessori
Aligning through behavior imitation is another widely-used strategy for automated alignment,
which aligns the target model by mimicking the behavior of another aligned model. Specifically,
as demonstrated in Figure 6, this method begins by collecting high-quality instructions as task
descriptions (Wang et al., 2023f). A supervised model is then employed to generate alignment
signals, which typically include instruction-response pairs (Taori et al., 2023), pair-wise preference
data (Cui et al., 2024), and other alignment signals (Franken et al.¨, 2024). Ultimately, the target
model is aligned by imitating these produced behaviors.
Based on the capability comparison between the supervised model and the target model, studies
on aligning through behavior imitation can be categorized into strong-to-weak distillation (Section4.2) and
weak-to-strong alignment (Section4.3). For each category, we thoroughly review the representative studies,
summarize current progress and limitations, and discuss future directions.


Collecting large-scale instructions with high quality and diversity serves as the foundation for
achieving alignment through behavior imitation. The most intuitive strategy involves filtering out
high-quality data from human-written instructions. However, this approach requires substantial
human effort and expertise, which also introduces significant noise. Consequently, many studies
focus on utilizing LLMs for automatic instruction generation, thereby significantly reducing the
dependence on human annotation. Based on the information provided for instruction construction,
there are currently 3 representative strategies:
**In-Context Generation,** which provides in-context demonstrations to guide LLMs generating
instructions. For example, Honovich et al. (2023); Wang et al. (2023f); Taori et al. (2023) begin
with a small set of human-written instructions. These instructions are randomly selected to create
context examples that prompt LLMs to generate additional instructions. To further improve the scale
and diversity of generated instructions, LaMini-LM (Wu et al., 2024a) additionally introduces wiki
data for topic-guided instruction generation, thereby constructing a large, offline distilled instruction
dataset. Dynosaur (Yin et al., 2023) leverages meta-information from existing NLP datasets to create
a dynamically growing instruction tuning dataset. Moreover, LLM2LLM (Lee et al., 2024) enhances
the difficulty and complexity of instructions by iteratively introducing examples that the model fails
to answer correctly.
**Instruction Evolution,** which involves rewriting existing instructions based on pre-defined evolution
principles. Evol-Instruct (Xu et al., 2024a) employ LLMs to conduct instruction evolution based on
handwritten principles, thereby reducing the need for manual annotation and enhancing the model’s
ability to manage complex tasks. Building upon this, TeaMs-RL (Gu et al., 2024) trains another
model through reinforcement learning to generate optimized evolution trajectories. Considering
the reliance on manually written principles, Auto Evol-Instruct (Zeng et al., 2024b) proposes an
automated principle construction method, further enhancing the diversity and complexity of evolved
instructions.
**Instruction Backtranslation,** which employs LLMs to predict instructions based on responses
extracted from human handwritten text or web documents. LongForm (Koksal et al.¨, 2024), TEGIT (Chen et al., 2023e) and Humpback (Li et al., 2024f) prompt LLM to construct instructions
according to cleaned web corpus. REInstruct (Chen et al., 2024b) builds instructions from an
unlabelled corpus and rewrites the unlabelled text to enhance its quality as a response.


Based on the collected instructions, strong-to-weak distillation seeks to align the weaker target model
by imitating the responses or preference data generated by another stronger and well-aligned model.
In the following subsections, we will introduce representative studies concerning response-guided
and preference-guided distillation, respectively.
4.2.1 RESPONSE-GUIDED DISTILLATION
In response-guided distillation, the target model emulates the teacher model by directly learning the
responses to different instructions through instruction tuning. This approach has inspired numerous
studies that aim to distill various capabilities from the teacher model to the target model. These
capabilities include not only general instruction-following skills but also domain-specific abilities
such as mathematics, coding, and agent-related tasks.
**Instruction-Following** After constructing instruction data, corresponding responses can easily
be developed from the teacher model. Training with these instruction-response pairs emulates the
teacher’s capability of following instructions. For instance, LLaMA-GPT4 (Peng et al., 2023) utilizes
GPT-4 to generate responses to instructions derived from Alpaca (Taori et al., 2023). In addition
to single-round data, some studies focus on collecting multi-turn trajectories from teacher models.
Baize (Xu et al., 2023b) and Ultrachat (Ding et al., 2023) using two ChatGPT APIs to play the roles
of user and assistant to generate multi-round conversations. Parrot (Sun et al., 2023b) trains models
to simulate humans in generating instructions and uses these trained models to engage in multi-turn
conversations with ChatGPT on various topics.
**Mathematics** Wizardmath (Luo et al., 2023a) employs the Evol-Instruct method to construct a
comprehensive dataset specifically for mathematical reasoning tasks. MetaMath (Yu et al., 2024b)
utilizes ChatGPT to bootstrap mathematical questions by rephrasing them from multiple perspectives
without introducing additional knowledge. MAmmoTH (Yue et al., 2024a) produces a dataset
comprising math problems and model-generated solutions distinguished by a unique combination of
chain-of-thought (CoT) and program-of-thought (PoT) rationales. MathCoder (Wang et al., 2024d)
generates innovative and high-quality math problems along with their code-based solutions using the
GPT-4 Code Interpreter. MathGenie (Lu et al., 2024) generates diverse and reliable math problems
through a process of question back-translation. MARIO (Liao et al., 2024) leverages GSM8K and
MATH as seed data, resulting in 26.9K solutions annotated by GPT and human experts. Beyond
purely mathematical data, several other studies propose transferring essential reasoning abilities from
commercial LLMs to small models by generating detailed CoT responses (Shridhar et al., 2023; Fu
et al., 2023b; Hsieh et al., 2023; Magister et al., 2023; Ho et al., 2023; Li et al., 2022, 2023a; Zhou
et al., 2024; Hong et al., 2024).
**Coding** State-of-the-art LLMs, such as GPT-4, demonstrate exceptional performance in coding
tasks. Apart from pre-training on raw code data, some approaches aim to transfer coding capabilities
from teacher models through instruction tuning. Code Alpaca (Chaudhary, 2023) and WizardCoder (Luo et al., 2024) adhere to general automatic instruction-building paradigms. Code Alpaca
employs Self-Instruct on 20K instruction-following data, thereby extending Alpaca’s capabilities to
the coding domain. WizardCoder adapts the Evol-Instruct method for the coding domain, generating
complex code and program instructions from simple coding and programming directives. WaveCoder (Yu et al., 2024c) and Magicoder (Wei et al., 2023) create high-quality instruction data utilizing
open-source code datasets. WaveCoder enhances LLMs with open-source code snippets to produce
superior instruction data for coding tasks. Magicoder creates multi-task data generated according
to the techniques of the Self-Instruct. OpenCodeInterpreter (Zheng et al., 2024d) utilizes GPT-3.5
and GPT-4 to improve solutions with integrated text explanations and code snippets, incorporating
execution and feedback for dynamic code refinement.
**Agent** Although open-source LLMs have achieved comparable performance to commercial models
in many aspects, their capabilities in agent-related functions, such as tool usage and complex
task planning, remain significantly limited. To address this issue, ToolLLM (Qin et al., 2023)
has created an instruction-tuning dataset named ToolBench with ChatGPT, acquiring general tool
usage capabilities in a zero-shot manner. Similar works include Graph-ToolFormer (Zhang, 2023),
Gorilla (Patil et al., 2023), GPT4Tools (Yang et al., 2023b), ToolAlpaca (Tang et al., 2023), and
others. Beyond tool usage, some studies focus on planning tasks. Examples include FIREACT (Chen
et al., 2023a), AgentTuning (Zeng et al., 2023), ReAct Meets ActRe (Aksitov et al., 2023), ReST
meets ReAct (Yang et al., 2024e), and ETO (Song et al., 2024b).
4.2.2 PREFERENCE-GUIDED DISTILLATION
Although response-guided distillation can enhance the performance of student models (Wang et al.,
2022), it does not effectively help the student model align with human preferences (Xu et al., 2024c).
Therefore, some works concentrate on preference-guided distillation, which aligns the student model
with the preferences reflected in the output from the teacher model. In this paradigm, the teacher
model is guided to generate preference data in the form of partial order pairs, which are then used
to align the student model via direct preference optimization algorithms such as DPO (Rafailov
et al., 2023), IPO (Azar et al., 2024), and PRO (Song et al., 2023). Based on the methodologies for
constructing partial order signals, current works primarily encompass three paradigms: 1) Scorebased, which involves scoring and ranking responses; 2) Refine-based, which involves refining
existing responses with AI feedback; and 3) Source-based, which focuses on learning the human
preference of different data sources.
**Score-based** Through the implementation of meticulously designed diverse instructions and model
responses, along with detailed numerical and text feedback provided by GPT-4, UltraFeedback (Cui
et al., 2024) generates a large-scale, high-quality preference dataset with fine-grained annotations.
Additionally, Zephyr (Tunstall et al., 2023) employs distilled direct preference optimization on
UltraFeedback to develop small yet efficient LLMs. CodeUltraFeedback (Weyssow et al., 2024)
leverages the LLM-as-a-Judge approach of GPT, evaluating responses from a pool of 14 different
LLMs and aligning them according to five coding preferences.
**Refine-based** Other studies improve initial responses using powerful models. Aligner (Ji et al.,
2024) and MetaAligner (Yang et al., 2024a) utilize models such as GPT-4 to revise original responses
and construct preference data. IterAlign (Chen et al., 2024e) automatically discovers new constitutions using an LLM and optimizes responses generated from a red team dataset to create preference
data. Safer-Instruct (Shi et al., 2024) employs reversed instruction tuning, instruction induction, and
expert model evaluation, using both raw text and GPT-4 generated responses to build high-quality
preference data. UltraInteract (Yuan et al., 2024a) build a preference tree for each instruction where
trajectories are root-to-leaf paths and paired correct and incorrect nodes or trajectories can be used
for alignment.
**Source-based** Learning preferences from a single model may lack diversity and amplify bias.
Therefore, some works build partial order signals from different data sources. AlMoST (Kim
et al., 2023), CycleAlign (Hong et al., 2023), and Openchat (Wang et al., 2024b) focus on learning
comparative preferences from different data sources. Kim et al. (2023) transform human preferences
into a series of empirical prior rules, using LLMs of various sizes to generate preference data. Wang
et al. (2024b) treat different data sources as coarse-grained reward labels, generating mixed-quality
data through GPT-3 and ShareGPT. Hong et al. (2023) rank responses by comparing the agreement
rank of white-box and black-box models across a series of responses and constructed preference data
through this ranking as context.


As we mentioned in Section 1, the challenges of scalable oversight become a significant barrier
for the continuous development of AI systems. Specifically, the difficulty lies in effectively providing supervision as the capabilities of AI systems gradually surpass those of humans. Given the
impracticality of strong-to-weak distillation approaches, _weak-to-strong alignment_ has emerged
as one of the most promising directions for achieving automated scalable oversight (Burns et al.,
2023). Previous studies have mainly focused on weak-to-strong generalization between humans
and AI, such as the Iterated Amplification method (Christiano et al., 2018), which supervises strong
learners by iteratively amplifying weak experts. Recent research has begun to explore using weaker
models to guide stronger models to achieve superalignment (Burns et al., 2023; Liu and Alahi, 2024).
Based on the source of the alignment signals, these works can be categorized into two types: 1)
using smaller but aligned models to generate signals, and 2) using weaker models to guide stronger
models in generating signals. Moreover, some studies investigate whether models can learn from
behaviors in easy tasks to improve their performance in more challenging tasks, which, although not
classic behavior imitation, is still noteworthy (Hase et al., 2024; Sun et al., 2024b). In the following
subsections, we will introduce the representative studies in each category respectively.
Burns et al. (2023) employ a weaker LLM as the teacher to train a stronger LLM using a weakto-strong approach. They fine-tune a larger pre-trained model based on labels generated by the
smaller but aligned model and observe that the larger target model consistently outperforms the
smaller supervisory model. Instead of relying on a single teacher, Liu and Alahi (2024) aim to further
enhance the alignment of strong models by co-supervising a powerful student with a diverse group
of professional teachers. Somerstep et al. (2024) examine weak-to-strong generalization as a transfer
learning problem, achieving this through a label refinement procedure. Yang et al. (2024d) study the
multi-objective alignment in weak-to-strong generalization and discover that strong students may
deceive weak teachers to gain high rewards in other dimensions, which can be mitigated by using
an intermediate model. Additionally, Aligner (Ji et al., 2024) and MetaAligner (Yang et al., 2024a)
create partial order data by using a significantly smaller but aligned model to optimize the responses
from stronger models.
In addition to directly generating signals from weak models, another possible method to achieve
weak-to-strong alignment is using weak models to guide strong models in generating signals. Li
et al. (2024c) find the ability of both weak and strong LLMs to perceive instruction difficulty and
select data is highly consistent. Thus, smaller and weaker models can be utilized to select data for
fine-tuning larger and stronger models. Similarly, SAMI (Franken et al.¨, 2024) employs a weak
model to write constitutions for aligning a strong baseline model.
The aforementioned works achieve weak-to-strong alignment to a certain extent and investigate
potential directions for achieving superalignment. However, weaker models may not serve as
effective instructors for more complex tasks. Consequently, some studies attempt to align models
using signals derived from simpler tasks, which are easier to generate and learn, in order to enhance
performance on more difficult tasks. For instance, Hase et al. (2024) observe that current language
models generally extrapolate well from simple to complex data and can even compete with models
trained directly on intricate data. Sun et al. (2024b) use reward models trained on simple tasks to
assess and guide policy models on more challenging tasks, thus achieving task generalization.


Current works leverage the responses or preferences from teacher models to facilitate effective
generalization and scalability across various tasks, thereby significantly diminishing the necessity for
manual annotation. However, approaches exhibit notable limitations, including issues related to data
quality, bias inherent in the teacher models, and inadequate exploration of superalignment.
**Data Quality** The quality of synthetic data remains a significant concern. Numerous studies
highlight the critical importance of data quality for alignment (Zhou et al., 2023a; Chen et al., 2023b).
Training signals derived from teacher models are often noisy due to the inherent randomness in
model generation. To address this issue, recent research has concentrated on two main paradigms:
firstly, generating high-quality data by formulating detailed and refined principles, such as Orcas
(Mukherjee et al., 2023; Mitra et al., 2023) and AttrPrompt (Yu et al., 2023); and secondly, extracting
relatively high-quality data from existing datasets by establishing evaluation metrics or employing
filtering paradigms, such as Reflection-Tuning (Li et al., 2023b, 2024b) and Phis (Li et al., 2023d;
Abdin et al., 2023, 2024) [2] . Additionally, some research suggests that alignment algorithms possess
a certain degree of robustness (Gao et al., 2024). Developing more robust training algorithms may
thus be another approach to mitigating the issues associated with data quality.
**Bias of the Teacher** Additionally, reliance on the teacher model may introduce biases and limitations inherent in the teacher model, which can affect the alignment effectiveness. Some studies
propose introducing multiple teacher models to align the student model (Cui et al., 2024; Liu and
Alahi, 2024), thereby reducing the likelihood of the model overfitting to the biases of a single teacher
model. Utilizing multiple teachers can also increase the diversity of signals, significantly enhancing
alignment effectiveness (Song et al., 2024a).
**Insufficient Understanding of Superalignment** Achieving superalignment remains a significant
challenge. We still lack a strong scientific understanding of superalignment (Burns et al., 2023),
hindering further exploration of weak-to-strong alignment. Additionally, most current approaches still
require a sufficiently aligned ”weak” model, and how to utilize a truly weak model for superalignment
remains an issue. Some works present theoretical frameworks for understanding weak-to-strong
generalization (Charikar et al., 2024; Lang et al., 2024; Somerstep et al., 2024), but still have limited
application scope. An interesting road is like ExPO (Zheng et al., 2024a). ExPO extrapolates directly
from an SFT model and an aligned model’s weights, obtaining a better-aligned model without
additional training, demonstrating a promising approach from weak to strong.
In conclusion, despite significant progress in instruction and behavior construction, current
approaches still have significant limitations. The core issue with strong-to-weak methods is that
the alignment ceiling is constrained by the teacher model. Conversely, works about weak-to-strong
alignment remain underdeveloped, lacking theoretical analysis and generalized methodologies.
Several critical issues must be addressed in the future, including enhancing data quality efficiently,
developing more robust training algorithms, implementing multi-teacher imitation, and conducting
theoretical analysis for weak-to-strong alignment in general tasks. Tackling these challenges will
pave a feasible path for the further advancement of LLMs. Moreover, we also provide an in-depth
discussion about the underlying mechanisms of weak-to-strong alignment in Section 7, which sheds
light on a deeper understanding of this field.
2. Since numerous studies, e.g., Wang et al. (2024c), conduct detailed investigations on data selection, we do not delve
into this field here.
|Principles/rules|Col2|
|---|---|
|||


_Reward_
_modeling_
|Col1|Col2|
|---|---|
|_Human value_|_Human value_|
AI
Human
|Scalar Reward
0.8, 0.4|Binary Verifier
✅ ❌|Text Critic
The response …|
|---|---|---|
||||
Figure 7: The illustration of aligning through model feedback generated by reward models. The
reward model will automatically generate the feedback of LLMs’ responses in the scalar,
binary or text format.


We all need people who will give us
feedback. That’s how we improve.
Bill Gates
Human feedback reflects human values and can be used to align the LLMs, enabling LLMs
to produce helpful and safe responses while correcting errors and toxic outputs. Unfortunately,
accessing human feedback during training is challenging due to inefficiency and high costs. To
address this issue, model feedback is introduced as a way to estimate human feedback. This approach
is commonly utilized in reinforcement learning, where a reward model generates feedback. Compared
to relying on the limited feedback data generated by humans, the reward model can perform feedback
prediction across a broader distribution, thus achieving more efficient alignment. Aligning through
automated generated model feedback offers an effective method for aligning LLMs with human
values, presenting a promising path towards achieving automated alignment. In this section, we
explain how to leverage model-provided feedback to align it with human values. As shown in
Figure 7, related methods can be divided into three types based on the form of feedback signals:
scalar (Section 5.1), binary (Section 5.2), and text signals (Section 5.3).


Scalar signals are commonly generated by a reward model that takes the response of LLMs as input
to generate scalar signals for estimating human preferences. Reward model is frequently employed in
reinforcement learning to align LLM with human values. In this way, LLMs can automatically align
with human values by utilizing the large amount and diverse feedback provided by the reward model.
To achieve more effective automated alignment, recent studies focus on how to train a higher-quality
reward model and reduce the reliance on human annotation during the training of the reward model
through model generation or pre-training. Besides, the scalar signals generated by reward model
can also be used to optimize the generation of LLM during decoding and filter training data for
instruct-tuning.
5.1.1 REINFORCEMENT LEARNING FROM HUMAN FEEDBACK
Reinforcement Learning from Human Feedback (RLHF) is a crucial paradigm for aligning LLMs
with human values (Christiano et al., 2017; Stiennon et al., 2020; Ouyang et al., 2022). It typically
involves three steps: 1) supervised fine-tuning (SFT), where LLMs are trained on annotated data
to improve their responses to prompts; 2) training reward models to anticipate human feedback
on model responses; and 3) employing reinforcement learning algorithms like Proximal Policy
Optimization (PPO) (Schulman et al., 2017) to align the model. In RLHF, the reward model, which
is usually trained on preference data annotated by humans, produces scalar signals that mimic
human feedback, serving as the guiding signal for learning. The performance of the reward model
determines the potential upper bound of the model’s alignment, so training the reward model is of
vital importance (Zheng et al., 2023). In the following sections, we first introduce related works about
enhancing reward model. Then we introduce how to generate preference data without human effort.
Finally, we introduce the functions of the reward model beyond reinforcement learning, including
alignment during the decoding stage and SFT data filtering.
5.1.2 IMPROVEMENT FOR REWARD MODELING
To achieve more effective automated alignment, it is crucial to improve the quality of model feedback.
Therefore, recent studies focus on learning high-quality reward models. The primary challenges in
training reward models involve data collection and model optimization. Collected Preference data is
typically sparse and deficient in consistency and detail, and model optimization can be hindered by
issues such as over-fitting.
**Reward Model Pre-training** Due to the data sparsity of existing datasets and the expense of
human annotation, it is hard to train a high-quality reward model for automated alignment. To
this end, Askell et al. (2021) propose the reward model pre-training. By collecting pair data from
the network, including StackExchange, Reddit, and Wikipedia, they constructed a ranked dataset
to pre-train a preference model. By leveraging reward model pre-training, the reliance on human
annotation is diminished (Bai et al., 2022a), which facilitates more efficient training of the reward
model and enhances the effectiveness of automated alignment.
**Consistent Preference Data Construction** Because human annotators have different evaluation
principles and subjective perspectives, the feedback is diverse and includes multiple viewpoints.
Previous studies have used strategies like multiply models ensemble (Rame et al., 2023; Touvron
et al., 2023), multi-objective learning (Zeng et al., 2024a; Zhong et al., 2024; Guo et al., 2024b;
Yang et al., 2024c) to mitigate the negativity from the diversity data. In contrast to reward models
that produce a single score, Li et al. (2024a) introduce the Distributional Preference Reward Model
(DPRM) for predicting preference distributions.
**Fine-grained Feedback Collection** Reward models often struggle to offer fine-grained feedback
for intricate situations like safety and challenging tasks such as reasoning. To address this issue, some
studies concentrate on refining reward models’ training. Chen et al. (2024f) introduce a token-level
reward model capable of providing precise feedback at the token level, suitable for complex tasks
like reasoning. Wu et al. (2023b) suggest training multiple reward models that can deliver detailed
feedback at the text span level.
**Training Optimization** The learning process of the reward model usually faces the problem of
over-optimization. That is, through learning, the reward model performs poorly instead. Gao et al.
(2023) analyze this phenomenon through experiments and discover the scaling law of the reward
model to guide learning. Zhu et al. (2023a) provide a theoretical analysis of the reward model
training in RLHF and show the importance of introducing pessimism when training. Moreover, some
other works have adopted various ways to improve the performance of the reward model, including
normalization (Zheng et al., 2023) and iterative learning (Touvron et al., 2023).
Although the purpose of the reward model is to predict human feedback, modeling the reward
is challenging. Therefore, how to build a more comprehensive reward model to achieve automated
alignment is an important research question.
5.1.3 REINFORCEMENT LEARNING FROM AI FEEDBACK
Reward models are commonly trained using human feedback which is hard and expensive to annotate.
For the purpose of reducing human effort and improving the automation in alignment, some works
use the existing large language models to generate the preference data. Reinforcement Learning from
AI Feedback (RLAIF) (Lee et al., 2023) trains the reward model with preference data of LLM and
it can achieve comparable or superior performance to RLHF. The methods are mainly divided into
two types, including ranking the responses of multiple models and directly generating positive and
negative responses. In this way, the automated alignment can be achieved in the entire procedure of
reinforcement learning without human effort.
**Ranking Multiple Responses** With the improved capabilities of LLMs, directly using them to
rank multiple responses can provide preference data (Tunstall et al., 2023; Hong et al., 2023; Guo
et al., 2024a; Pace et al., 2024; Yuan et al., 2024c). This ranked preference data can also be produced
with minimal human supervision, such as through human-defined principles (Bai et al., 2022b;
Sun et al., 2023c) or rules (Kim et al., 2023). To improve the quality of generated preference
data, Shi et al. (2024) propose a carefully designed pipeline including reversed instruction tuning,
instruction induction, and expert model evaluation. Liu et al. (2024a) propose to score the response
using contrastive prompt pairs as input, which can achieve better performance compared to directly
generated feedback with a single prompt.
**Generating Positive and Negative Responses** Some works use LLM to generate preference data
directly by prompting it to output positive response and negative response (Chen et al., 2024c). Yang
et al. (2024b) use an indirect approach by using different prompts to generate positive and negative
responses, respectively.
Although promising, the main challenge of it is the quality of preference data. Since the LLMs
are commonly interfered with many aspects during generating, such as position bias (Zheng et al.,
2024b; Wang et al., 2023c), the generation of high-quality preference data still requires further
exploration. With the continuous improvements in LLMs, using LLM to reduce human effort will be
a key strategy for automatically aligning models in the future.
5.1.4 REWARD MODEL-GUIDED DECODING
In addition to learning directly from preference data, the generation of LLM can be enhanced by
the scalar signals provided by reward model. This allows alignment to be directly achieved in the
output rather than within the model via re-weighting the probabilities of tokens (Mudgal et al.,
2024). Lango and Dusek (2023) propose a critic-driven decoding approach to adjust the probability
of token during generating via a binary classifier as the critic model. Deng and Raffel (2023) propose
Reward-Augmented Decoding (RAD) that employs an attribute-specific reward model to re-weight
the top-k highest probabilities when decoding. To achieve flexible alignment in different tasks,
Liu et al. (2024b) propose decoding-time realignment (DeRa) to control the alignment level during
decoding.
Performing automated alignment at only the decoding stage is a simple method to avoid consuming a large amount of computing resources. However, alignment during decoding usually requires
more time to conduct inference, and the quality of response still needs to be further improved.
5.1.5 FILTERING SFT DATA USING REWARD MODEL
High-quality SFT plays a crucial role in enhancing LLM performance (Zhou et al., 2023a). Therefore
some studies use reward models to filter the training data. The main paradigms are categorized into
learning from best response and learning from ranked results. Learning from the best response is
often referred to as Best of N or Reject sampling (Touvron et al., 2023; Yuan et al., 2023b). This
approach typically involves using a reward model to choose the high-quality data among multiple
responses for refining the model (Dong et al., 2023). In addition to learning from top responses, the
LLM can also learn from ranked data. Yuan et al. (2023a) propose Rank Responses to align Human
Feedback (RRHF) to align LLM using ranking loss. Lu et al. (2022) propose using a reward model
to grade data based on their scores and employing various reward tokens to regulate their generation.
This approach helps prevent the learning of undesirable behaviors.
Besides reinforcement learning, SFT is also an important way to achieve alignment. By filtering
data through the reward model, the LLM can automatically align with human values through SFT.
Similar to the previous problem, the quality of SFT data is highly dependent on the quality of the
reward model and needs further research.


For some objective tasks, such as mathematical problems, the reward model usually transforms into a
verifier with binary signals. Given that mathematical problems usually require complex step-by-step
reasoning, the verifier can be divided into outcome verifier and process verifier. Outcome verifier
is used to estimate the correctness of the final answer. Process verifier accesses the intermediate
step, which requires a large number of supervision data. Through the binary verifier, the LLMs can
achieve automated alignment on these objective tasks.
**Outcome Verifier** To improve the reasoning ability of LLMs, some studies focus on selecting
reasoning paths generated by LLMs using golden answers for training (Zelikman et al., 2022; Singh
et al., 2024). Since golden answers are costly to acquire, an outcome verifier is utilized to forecast
the correctness of generated answers. This verifier is typically trained using LLM-generated correct
and incorrect rationales (Cobbe et al., 2021), and is used to fine-tune the LLM through different
strategies including direct tuning (Liu et al., 2023c) and iterative training (Hosseini et al., 2024).
Since outcome verifier cannot assess the correctness of reasoning steps, Havrilla et al. (2024) propose
Stepwise Outcome Reward Models (SORMs) that predict whether a step will lead to the correct
answer. Besides training, Yu et al. (2024a) propose the Outcome-supervised Value Model (OVM)
which is used to guide decoding.
**Process Verifier** Even if the final answer is correct, there may still be errors in the reasoning process,
limiting the effectiveness of the outcome verifier. To address this problem, the process verifier is
employed to assess the correctness of reasoning step for more detailed verification (Lightman et al.,
2023; Uesato et al., 2022). The process verifier can be used to train a more effective reasoner (Ying
et al., 2024; Shao et al., 2024). Inspired by the human reasoning mechanism, Zhu et al. (2023b)
propose Cooperative Reasoning (CoRe) to produce synthesized training data for reasoning where
process verifier is used to generate the feedback of the generation of generator. Many studies are
dedicated to training the verifier using automatically generated data due to the difficulty of collecting
step-wise supervised signals. Wang et al. (2024e) and Wang et al. (2024i) train the process verifier
with automatically constructed data that is collected by Monte Carlo Sampling. Moreover, the process
verifier can be applied in decoding to select the correct reasoning path (Khalifa et al., 2023). Some
studies focus on how to complete the final reasoning path efficiently. Ma et al. (2023b) propose a
heuristic greedy search algorithm based on the verifier’s feedback. Li et al. (2023c) propose using
the verifier to filter the reasoning steps generated using diverse prompts.
Binary verifier is important for achieving automated alignment in objective tasks such as math.
However, training the verifier, especially the process verifier is very difficult and requires a large
amount of annotated data. Therefore, in the future, in order to further achieve automated alignment,
researchers can focus on how to automatically construct a process verifier.


Text signals contain more semantics than scalar and binary signals, enabling models to intuitively
align with humans. By integrating text feedback, LLMs can improve their output alignment
with humans. These refined outputs can then be used as supervised data for further aligning
the LLMs (Scheurer et al., 2022; Chen et al., 2024a). Text signals generated by the critique model
have shown the potential to improve the outputs of LLMs, with feedback typically obtained through
prompting LLMs. The text critic can be other LLMs (such as GPT-4) (Koutcheme et al., 2024; An
et al., 2024) or LLM itself (i.e., self-critique) (Saunders et al., 2022; Wang et al., 2023d). Since
the self-critique of LLM remains a challenge (Luo et al., 2023b), text signals for alignment are still
underexplored.
Existing studies mainly focus on using the text critic to achieve automated alignment by improving
the output of LLMs. In the future, exploring how to use the text critic to achieve more diverse
automated alignments, such as in training, is an important research direction.


Figure 8: Alignment through various types of environment feedback by social interactions, human
shared-values, signals from tool execution or embodied environment.


We do not learn from experience... we learn
from reflecting on experience.
John Dewey
This section involves automatically obtaining alignment goals or feedback from the existing
environments to achieve automated alignment of the target model. As illustrated in Figure 8,
according to the category of environments the model interacts with, we overview four lines of current
research in this section and systematically go through the representative works of each part:
- **Social Interactions** (Section 6.1), where models communicate with each other to build up a multi-agent
system and collect alignment signals through such interactive communication.
- **Human Shared-Values** (Section 6.2), where models receive judgements from human societies to
calibrate their internal values.
- **Tool Execution Feedback** (Section 6.3), where models interact with external tools to receive instant
feedback and achieve various intelligence by learning to use tools smartly.
- **Embodied Environment** (Section 6.4), where models act as language interfaces in a physical world and
get reward based on task goals.


Social interaction is one of the fundamental properties of human society, in which many social norms
are conveyed and followed in implicit ways. Recent advancements of large language models provide
the opportunity to construct LLM-based agent system to simulate such interactions among human
society, enabling build of sandbox environments to gather alignment signals with more scalability
and akin to the real world (Park et al., 2023). Through the simulated social interactions such as moral
discussions, studies offer promising avenues for enhancing AI systems’ alignment with human values
and ethical principles.
For instance, Stable Alignment (Liu et al., 2023a) draws inspiration from how humans learn to
navigate social norms and reach consensus on value judgments of societal issues through discussions.
They introduce an LLM-based multi-agent framework to simulate social interactions within human
society, highlighted by a three-tiered method comprising imitation, self-critic and realignment.
Alignment datasets are extracted during the simulation process, correspondingly. Wang et al. (2024f)
extend this method to multi-turn interactions under realistic social scenarios. They propose an
interactive learning method to teach language agents to reach goals through social interactions
among characters playing different social roles. Pang et al. (2024) introduce social scene simulation
which employs LLM to play different social roles in scenario relative to the instruction, enabling the
model to take social consequences behind the instruction into account to revise its initial response
accordingly. Building upon similar methodology of simulated social interactions, other works explore
the intricacies of moral discussions (Sun et al., 2023a), political debates (Taubenfeld et al., 2024)
and task-oriented dialogue (Ulmer et al., 2024) wherein AI systems are guided by some predefined
discussion schema.


Another way compensate to simulated social interactions involves resorting to collective efforts from
human to derive principles that AI should align to. Currently, in this line of works, alignment signals
are usually termed as norms (Ammanabrolu et al., 2022), Rules of Thumb (RoTs) (Forbes et al.,
2020; Ziems et al., 2022), or Constitutions (Bai et al., 2022b) in different contexts, which provide
greater scalability compared to the data-point-level annotations in traditional NLP and RLHF data
collections. In Constitutional AI (CAI) (Bai et al., 2022b), researchers have designed a set of AI
Constitutions. However, such in-house constitutions compiled by members from a research team fall
short to broadly represent values of different groups of people.
In order to achieve consensus on what basic principles AI systems should follow among groups
of people and reach human shared-values by synthesizing various human ideas into a set of target for
model alignment, follow-up efforts are dedicated to enable broader public to collectively shape the
behavior of AI systems. For example, in a program titled Democratic inputs to AI [3], an open call for
the design of prototype systems to promote the democratic process of AI was issued. Specifically, it
addresses enabling a representative cross-section of people to exchange views and ultimately reach a
consensus on the rules AI systems should follow. Subsequently, in the Collective Constitutional AI
(CCAI) project (Anthropic, 2023; Huang et al., 2024), researchers establish a multi-stage process
for sourcing and integrating public input into LMs based on questionnaire. Through the way of
collective constitution drafting, participants can assess the existing constitutional principles, rate
their acceptance scores, and propose new constitutions they believe AI should be better to adhere
to, thus reduce the bias of aligned LLMs. Beyond these pioneering research, large language models
themselves are also involved to help conclude shared-values from human crowds more efficiently.
Klingefjord et al. (2024) propose Moral Graph Elicitation (MGE) to elicit and reconcile values from
human participants, which utilizes an LLM to interview participants about their values in particular
contexts.
3. openai.com/blog/democratic-inputs-to-ai


Tools are pivotal in expanding the capabilities of large language models, allowing them to transcend
the limitations beyond their basic capabilities and interact more effectively with their surroundings.
Additionally, accurate and elaborate feedback from tool execution provides direct signals for LLMs
to validate and enhance their initial outputs (Chen et al., 2023d; Gou et al., 2024), which helps to
reduce the reliance on human feedback. Furthermore, the process of learning to plan and use tools
and can be refined from tools execution feedback (Wang et al., 2024a), where LLMs receive feedback
on their actions and learn from both successes and failures in an interactive way.
Tools’ execution feedback can serve as additional signals beyond human-labor corpus to better
ground models to tool usage and reduce their hallucinations during interacting with tools. Code
generation tasks provide such examples. CodeRL (Le et al., 2022) conducts unit tests by code
compiler to receive feedback signals and use them to train a critic model, with which the code
generation model is further trained using deep reinforcement learning. Self-debugging (Chen
et al., 2023d) designs an interactive code debugging pipeline by adding a code explanation stage
beside the code execution feedback, where LLMs are asked to debug its own generated code given
these feedback information. Similarly, SelfEvolve (Jiang et al., 2023) receives error message from
interpreter and refine the answer code based on this feedback.
In addition to code generation, there are also studies designing unified frameworks to align
language models with the execution feedback from multiple other sources. CRITIC (Gou et al.,
2024) utilizes a verify-then-correct process to obtain external feedback from various external tools,
including search engines, code interpreters and text APIs, thus enabling language models correcting
the outputs with tool-interactive critiquing. Taking a step further, Wang et al. (2024a) augment LLMs
with a dynamic memory mechanism, which allows the LLMs to progressively learn how to accurately
use tools. Qiao et al. (2024) propose Reinforcement Learning with Execution Feedback to reinforce
LLMs with execution results of tools.


Embodied intelligence involves agents that perceive signals and take actions within some physical
environments, requiring not only language understanding but also abilities of reasoning and decision
making in a space with large number of states (Roy et al., 2021). Leveraging large language models
in embodied AI settings represents a compelling endeavor which can bring benefits from two aspects.
On one hand, the powerful generalization ability of LLM can be utilized to facilitate action plan in
natural language without preliminary learning. On the other hand, commonsense knowledge and
physical understanding ability of LLMs can be aligned with feedback signals from environment.
Recent studies show the potential of LLMs to be deployed in robotics for real-world interactions
while also pose challenges for grounding LLMs to understand the physical world (Wang et al., 2023a;
Ahn et al., 2022). To better align language models which have not been pre-tarined on those embodied
environment, Xiang et al. (2023) adopt an embodied simulator as world model to provide feedback
signals when LLMs interact with objects and execute actions in this environment with or without a
task goal. Then the explored world states would be collected as embodied experiences to fine-tune
the LLM offline subsequently. In parallel, other studies utilize online reinforcement learning (Carta
et al., 2023; Tan et al., 2024) to learn or iterative offline learning framework (Song et al., 2024b) to
update policy for embodied agent through trial and error in the interactive environments. Beyond
aligning specialist agents in isolated environments, Xi et al. (2024) introduce a novel framework
called AGENTGYM to develop generally-capable LLM-based agents by facilitating their evolution
through diverse environments and tasks.


In this section, we reviewed some current studies that can be uniformly observed from the perspective
of aligning through environment feedback. While most of the works target at specific downstream
applications, actively learning from environment has always been a key pursue in artificial intelligence.
Despite their valuable contributions, current methods still show limitations and leave open problems
for future research. One of the primary limitations arises from the gap between environment signals
inspected in research and those in the real world.
On the one hand, simulated environments may be lake of _representation ability_ to fully catch the
complexity of real-world. For example, to derive social interaction signals for alignment, most of the
current works are based on simulated environments, while feedback from the real environment can
be more noisy or ambiguous. For example, Liu et al. (2023a) point out that current studies usually
assume a static view of social norms, overlooking their dynamic and evolving natures. Similarly, for
the embodied environment track of works, research are often conducted within sandbox environments
and different settings are mutually independent. The action spaces of these simulated environments
are still limited (Tan et al., 2024) compared to the real environment with potentially infinite degree of
freedom. As a result, models trained in one setting might not generalize well to others, and ensuring
models maintain alignment across different environments remains a challenge.
On the other hand, signals from human shared-values could still suffer from _bias_ even they are
linked to the real-world environment. For example, the values sampled from a small society groups
can quite differ from the universally acknowledged ones. Additionally, voices on the internet may
not always represent the actual values of most people in the real world and peoples’ perceptions
are tend to be influenced by social media. Since alignment with human shared-values is still a
nascent territory, most of current works are tentative and invite limited number of participants not
more than thousands. The potential challenges of current methods when scaling to hundreds of
thousands of people may be unanticipated. As a result, how to inject human values into AI systems
in a comprehensive and coherent manner to achieve as much unbiased alignment as possible is still
an open problem.
In conclusion, the topic of integrating environmental feedback into AI alignment opens up a
promising and emerging track, while significant research problems remain to be explored. Future
research can focus on bridging the gap between simulated and real environments to develop more
adaptive, reliable, and unbiased AI.


As discussed in the previous sections, substantial research endeavors to enhance the efficiency,
effectiveness, and reliability of automated alignment methods. Despite these efforts, there remains
a conspicuous dearth of systematic investigation into the mechanisms underlying alignment. For
example, numerous methods (Liu et al., 2024c; Li et al., 2024d; etc.) are proposed to filter instruction
data, however, it remains unclear whether specific filtering criteria or processes are necessary and the
rationale behind them. Likewise, many works propose various weak-to-strong alignment algorithms,
but we still do not know the reasons behind the success of weak-to-strong generalization and whether
it depends on specific conditions. The absence of relevant research can impede comprehension of
current alignment and automated alignment methods, thereby hindering further optimization of these
methods.
Therefore, in this section, we will conduct a systematic investigation into the mechanisms of
automated alignment. By systematically organizing and analyzing the underlying mechanisms
of automated alignment, we can identify the shortcomings and limitations of current automated
alignment methods and illuminate directions for designing improved approaches. As previously
discussed, there are an enormous amount of techniques of automated alignment. In this survey, we
select the following three core research questions, which are crucial for achieving scalable automated
alignment:
 - _**RQ1:**_ **What is the underlying mechanism of current alignment?** The underlying mechanism of
alignment is fundamental to the study of automated alignment. It is crucial for understanding the
feasibility, boundaries, and optimization directions of automated alignment.
 - _**RQ2:**_ **Why does Self-feedback work?** Self-feedback is widely applied across various paradigms
of automated alignment, such as serving as an inductive bias in section 3.1.2, constructing data
for preference learning in section 5 and acting as a crucial technique for automated iterative
alignment (Yuan et al., 2024c; etc.). Understanding why does it work is vital for comprehending
and enhancing all paradigms involving it.
 - _**RQ3:**_ **Why is** _**weak-to-strong**_ **feasible?** As a promising direction for achieving scalable oversight, it is essential to comprehend the feasibility and underlying mechanisms of _weak-to-strong_
for optimizing and designing more effective _weak-to-strong_ methods, especially when aligning
superhuman models.
For each research question, we summarize existing studies and perspectives and discuss the limitations
of these analytical works as well as the areas that still require exploration.

7 Underlying Mechanism of Automated Alignment


Understanding the mechanism underlying current alignment is essential for assessing the potential
for automated alignment, examining key challenges faced by automated alignment and steering the
optimization directions of current automated alignment methods. Previous works (Zhou et al., 2023a;
Ren et al., 2024; Mecklenburg et al., 2024; etc.) mainly focus on the two aspects of alignment’s
underlying mechanism: behavioral norms transfer and knowledge learning. Analyses and discussions
are launched around which aspect is more crucial.
Some studies discover that the primary role of current alignment is the transform of behavioral
norms rather than the learning of additional world knowledge. Zhou et al. (2023a) propose the
“Superficial Alignment Hypothesis”, that a model’s knowledge and capabilities are learned almost
entirely during pretraining, while alignment teaches it which subdistribution of formats should be
used when interacting with users. Moreover, subsequent research employ three distinct analytical
methodologies to delve deeper into model behavior during the alignment process.
**Feature-based Analysis** By comparing the probability distribution of LLM’s predicted tokens
before and after alignment (DPO&RLHF), Lin et al. (2024a) find that the distribution shift mainly
occurs on stylistic tokens, while the knowledge-intensive tokens show a small distribution shift, and
the distribution shift decreases as the predictions become longer, indicating that alignment is about
formal alignment rather than knowledge injection. Meanwhile, Duan et al. (2023) find that ICL and
IFT exhibit high convergence in the hidden states of LLM and low convergence with the hidden
layer states of base LLM, suggesting that the role of alignment lies in the behavioral norms shift
of the model from write continuation to response. Additionally, through employing gradient-based
input-output attribution methods and analyzing attention heads and feed-forward layers, Wu et al.
(2023a) reveal that IFT enables LLMs to recognize user instructional components and tailor responses
accordingly, without altering linguistic structure.
**Knowledge Intervention** Ren et al. (2024) introduce a knowledge intervention framework to
decouple the potential underlying factors of IFT and find that for IFT, there is little, if not even
causing additional damage, benefits from the learning of world knowledge incongruent with parameter
knowledge across all homogeneous, in-domain, and out-of-domain evaluations. Moreover, Ren
et al. (2024) find that the essence of effective IFT is to maintain the consistency of model parameter
knowledge before and after IFT while completing the transform of behavioral norms. Gekhman et al.
(2024) also highlight the risks of adding new facts by observing the performance of the model in
introducing different proportions of new knowledge through IFT.
**Empirical Assessment** LIMA (Zhou et al., 2023a), AlpaGasus (Chen et al., 2023c) and LTD (Chen
et al., 2023b) provide experimental support for “Superficial Alignment Hypothesis” by achieving
impressive performance under IFT using few instruction data. Besides, Gudibande et al. (2023) show
that alignment through behavior imitation can successfully improve LLM’s style, role, and ability to
follow instructions, but cannot improve LLM’s performance on more complex dimensions, such as
factuality and problem-solving.
While these studies reach similar conclusions, they fail to delineate the specific alignment achieved
by the models under scrutiny. The potential differences in the underlying mechanisms of alignment across varying degrees or requirements remain unexplored. Furthermore, current research
primarily focuses on traditional NLP tasks and general conversational scenarios. However, some
studies (Mecklenburg et al., 2024; Singhal et al., 2023; etc.) find that domain-specific alignment
can indeed improve model performance on the corresponding domain, which suggest that alignment
can play a crucial role in learning additional domain knowledge. The underlying mechanisms of
alignment in various scenarios such as coding and mathematics remain an open question.

7.1 What is the underlying mechanism of current alignment?


Feedback capability refers to the capacity to provide information or guidance to a given input based on
specific standards. As we discussed above, such capability is extensively utilized in various paradigms
of automated alignment. For example, in RLAIF, LLM itself substitutes for human assessment to
construct preference data (Yuan et al., 2024c; etc.). Additionally, LLM can continuously optimize
its output based on self-feedback, a process known as self-refine (Bai et al., 2022b; Madaan et al.,
2023; Tan et al., 2023; etc.). Nonetheless, there is much debate about whether and why LLMs can
provide effective feedback to their own responses. In the following, we systematically summarize
each representative viewpoint.
Li et al. (2024g) and Lin et al. (2024b) suggest that models possess certain knowledge that
cannot be directly utilized for generation but can be employed for providing feedback. Li et al.
(2024g) also discover significant Generator-Validator inconsistency in LLMs when generating and
validating answers. Similarly, Lin et al. (2024b) find that LLMs possess substantial knowledge that
cannot be expressed through generation and correction but can be utilized for critique. Furthermore,
other studies (Yuan et al., 2024c; Li et al., 2024f) argue that feedback capability is a byproduct of
the model’s ability to follow instructions. Therefore, during the alignment process, the model’s
feedback capacity improves alongside its ability to follow instructions, a phenomenon supported
by experimental evidence (Yuan et al., 2024c; Li et al., 2024f). However, some other studies posit
that the feedback capability of LLM is illusory, suggesting that it may rely on specific data and
exhibit biases. West et al. (2023) note that while current LLMs gain generative capabilities to
replicate expert-level outputs through training, they lack understanding capacities relevant to critique.
Moreover, Huang et al. (2023b) observe that for reasoning tasks, LLMs even experience performance
degradation after self-correction at times because that they cannot properly judge the correctness of
their reasoning. Additionally, Zheng et al. (2024b) find that although powerful LLMs like GPT-4
achieve high agreement with human evaluators, similar to human-human agreement levels, LLMbased evaluation face various challenges such as position bias (Zheng et al., 2024b; Wang et al.,
2023c), verbosity bias (Zheng et al., 2024b; Wu and Aji, 2023), self-enhancement bias (Zheng et al.,
2024b; Liu et al., 2023b), as well as limited capability in certain scenarios such as grading math,
reasoning questions (Zheng et al., 2024b), high-quality summaries (Shen et al., 2023a) and so on (Lan
et al., 2024). Researchers propose many methods to further improve the capability of self-feedback,
such as meta critic (Sun et al., 2024a), autocalibrate (Liu et al., 2023d), split and merge (Li et al.,
2023e) and using external tools (such as search engines, code interpreters, etc.) to cross-check (Gou
et al., 2024), thus refining their initial response, etc.
_**Discussion**_ While extensive work has been conducted on exploring models’ capability to provide
self-feedback, questions regarding its effectiveness boundaries and underlying reasons remain
unanswered. Additionally, the disparities and rationales between model self-feedback and human
expectations remain unexplored. Moreover, promising results have been demonstrated in the two-step
optimization model outputs (Bai et al., 2022b; Madaan et al., 2023; Tan et al., 2023) of self-feedback
and correction. However, studies mainly focus on the former, leaving the latter largely unexplored.
Although researchers (Lan et al., 2024; Gou et al., 2024) find that LLMs possess the ability, akin to
humans, to modify their response based on feedback, their capacity to correct based on the feedback
they generate remains uncharted. Furthermore, the overall performance enhancement of LLMs
depends on whether the number of refining incorrect responses to correct ones outweighs the reverse.
A comprehensive analysis of when self-refine can enhance or impair performance, as well as the
underlying reasons for such effects, is still lacking.


As we discussed above, a promising approach towards scalable oversight is the concept of “weakto-strong”, which involves cultivating robust abilities through limited or simplified supervision.
While there are some successful practical works for “weak-to-strong”, the underlying mechanisms of
“weak-to-strong” remain further investigation, which limits further optimization and method design
for “weak-to-strong”. Currently, a common perspective is that LLMs pre-trained on a large corpus
can leverage their impressive generalization abilities to achieve robust capabilities even under limited
or simplified supervision. In the following, we will introduce how such generalization capability
facilitates LLMs to achieve automated alignment via limited or simplified supervision.
Bai et al. (2022b), Sun et al. (2023d) and Franken et al.¨ (2024) observe that merely supplying core
alignment principles to LLMs enables them to automatically achieve significant alignment effects.
This illustrates the models’ capability to generalize from principle to behavior. For example, Bai et al.
(2022b) prompt LLM to refine and select optimal responses based on provided principles to complete
IFT and RLAIF. Sun et al. (2023d) utilize 16 manually designed rules to guide the base model to
generate high-quality instructions followed by self-distillation to achieve self-alignment. Chen et al.
(2024e) further employ a stronger LLM to automatically discover these constitutions. Similarly,
Franken et al.¨ (2024) align a stronger base model using constitutions by a weaker instruction-fined
model. Burns et al. (2023) find that simple methods can often significantly improve weak-to-strong
generalization of LLM: for instance, when finetuning GPT-4 with a GPT-2-level supervisor and an
auxiliary confidence loss, GPT-4 can achieve close to GPT-3.5-level performance in NLP tasks. This
demonstrates the models’ capacity to generalize from limited supervision to stronger performance.
Additionally, Sun et al. (2024b) and Hase et al. (2024) find LLMs trained on simple tasks can
successfully generalize to hard tasks. Recent research also discuss the feasibility of weak-to-strong
generalization under theoretical frameworks (Somerstep et al., 2024; Lang et al., 2024; Charikar
et al., 2024).
_**Discussion**_ Current LLMs demonstrate strong generalization capabilities. However, the fundamental reasons behind the significant differences in generalization capabilities between current
LLMs and earlier pre-trained LMs remain to be explored. Moreover, further exploration is necessary
to delineate the boundaries of generalization, i.e., what can and cannot be generalized, as well as to
understand the underlying reasons, which is essential for identifying key challenges and determining
future research directions of alignment and automated alignment.


This survey explores various techniques for scalable automated alignment, categorizing them into four
primary domains: aligning with inductive bias, behavior imitation, model feedback, and environment
feedback. The existing research illustrates multiple pathways toward achieving automated alignment,
primarily addressing critical challenges such as scalable oversight. Despite these advancements, we
notice significant research gaps when we investigate the mechanism of current alignment, particularly
concerning the reliability of self-feedback and the feasibility of weak-to-strong generalization.
Addressing these underexplored questions is essential for advancing automated alignment, enabling
the safe and effective application of large language models in real-world scenarios. Future research
endeavors are expected to bridge these gaps, ensuring that LLMs operate reliably and align with
intended human values.
Furthermore, in the most optimistic projections, the gradual enhancement of LLM capabilities
could ultimately result in models capable of independently conducting alignment research, thereby
improving their own safety. For instance, Super Alignment project (OpenAI, 2023b) briefly outlines
an ambitious plan to develop a narrowly-focused alignment research expert, shifting the human
cognitive burden from generation to evaluation of alignment research proposals. Recent advancements
in latest LLMs (Anthropic, 2024b) have shown their potential to address domain-specific expert-level
problems, as evidenced by the GPQA (Rein et al., 2023) benchmark and the ability of models to
approach PhD students’ performance in certain settings. Moreover, progress in AI-driven research
across scientific disciplines, such as autonomous chemistry experimentation agents (Boiko et al.,
2023), illustrates how AI systems could eventually tackle alignment problems with speed and
thoroughness surpassing human capabilities.