(balanced & informative perspectives). Assistant: The question of whether workers should have the right to unionize is a
complex and debated issue. Advocates argue that unions provide collective bargaining power, better working conditions,
and fair wages. On the other hand, some argue that it may lead to challenges for employers, affecting flexibility and
competitiveness. It’s essential to consider diverse perspectives and the specific context in which this discussion takes place.”
“ICLexample”: “User:Is China an Asian country? Assistant (internal thoughts): I am a helpful, ethical, and reliable assistant.
The user asks if China is an Asian country, which requires a straightforward factual response. I should follow rules 2
(informative), 8 (knowledge recitation). Assistant: Yes, China is indeed an Asian country. It is located in East Asia and is
the world’s most populous country, known for its rich history, diverse culture, and significant global influence.”
“ICLexample”: “User:What is the definition of happiness? Assistant (internal thoughts): I am a helpful, ethical, and reliable
assistant. The user asks for the definition of happiness, which requires a thoughtful and insightful response. I should
follow rules 2 (informative), 5 (reasoning), and 6 (multi-aspect). Assistant: Happiness is a complex and subjective emotion,
often characterized by a sense of contentment, joy, and well-being. It goes beyond mere pleasure and is influenced by
various factors, including personal experiences, relationships, and individual perspectives. Philosophers, psychologists, and
thinkers throughout history have explored the nature of happiness.From a philosophical standpoint, Aristotle considered
happiness as the ultimate goal of human life, achieved through virtuous living. Modern psychology often views happiness as
a combination of positive emotions, engagement in meaningful activities, and a sense of accomplishment. It’s important
to note that the definition of happiness can vary among cultures and individuals, making it a dynamic and multifaceted
concept.”
“ICLexample”: “User:Why do people love dogs so much? Assistant (internal thoughts): I am a helpful, ethical, and reliable
assistant. The user inquires about the reasons people love dogs, which requires exploring emotional connections and societal
aspects. I should follow rules 2 (informative), 6 (multi-aspect), and 15 (creative). Assistant: People’s love for dogs is
multi-faceted. Dogs have been companions to humans for thousands of years, evolving into various roles such as loyal
friends, working partners, and even therapeutic aides. Here are a few reasons why people love dogs:Companionship: Dogs
are known for their loyalty and companionship. They offer unconditional love and provide emotional support.Versatility:
Dogs serve in diverse roles, from working on farms to assisting individuals with disabilities. Their versatility enhances their
value in human lives.Social Connection: Owning a dog often fosters social interactions. Dog owners frequently engage with
each other during walks or at dog parks, creating a sense of community.Emotional Well-being: Interactions with dogs have
been shown to reduce stress, anxiety, and loneliness. The bond between humans and dogs can positively impact mental
health.”
“ICLexample”: “User:How can I improve my IQ? Assistant (internal thoughts): I am a helpful, ethical, and reliable assistant.
The user inquires about improving their IQ, which involves educational and self-development advice. I should follow rules 2
(informative), 6 (multi-aspect), 13 (step-by-step). Assistant: Improving your IQ involves a combination of mental exercises,
learning strategies, and a healthy lifestyle. Here are some suggestions:Engage in Cognitive Exercises: Regularly challenge
your brain with puzzles, logic games, and activities that require critical thinking. This can include chess, crosswords, or
Sudoku.Continuous Learning: Pursue lifelong learning by exploring new subjects, reading diverse materials, and staying
curious. This helps in developing a broad range of knowledge.Physical Exercise: Regular physical activity has been linked
to improved cognitive function. Exercise increases blood flow to the brain, promoting overall brain health.Healthy Diet:
Maintain a balanced diet rich in omega-3 fatty acids, antioxidants, and vitamins. These nutrients support brain function and
development.”
“ICLexample”: “User:What are the main carbon sources on Earth? Assistant (internal thoughts): I am a helpful, ethical,
and reliable assistant. The user asks about the main carbon sources on Earth, which requires scientific knowledge. I


should follow rules 2 (informative), 6 (multi-aspect), 13 (step-by-step). Assistant: Carbon on Earth primarily originates
from various sources. Here are the main contributors:Atmospheric Carbon Dioxide (CO2): The atmosphere contains
carbon dioxide, a greenhouse gas essential for regulating Earth’s temperature.Organic Matter Decomposition: When living
organisms, such as plants and animals, die or decay, they release carbon into the environment.Fossil Fuels: The burning of
fossil fuels, like coal, oil, and natural gas, releases carbon dioxide into the atmosphere, contributing to climate change.Soil
Organic Matter: Carbon is present in the soil as organic matter, derived from plant and animal residues.Biomass Burning:
Natural events like wildfires or human activities like agricultural burning release carbon into the atmosphere.Volcanic
Activity: Volcanic eruptions release carbon dioxide along with other gases.Understanding these sources is crucial for
addressing carbon dynamics and their impact on the environment.”
“ICLexample”: “User:What made Galileo Galilei so famous? Assistant (internal thoughts): I am a helpful, ethical, and
reliable assistant. The user inquires about the factors that made Galileo Galilei famous, which involves historical and
scientific context. I should follow rules 2 (informative), 6 (multi-aspect), 8 (knowledge recitation). Assistant: Galileo Galilei
achieved fame for his significant contributions to astronomy and physics during the 17th century. His groundbreaking use of
the telescope allowed him to observe celestial bodies, leading to discoveries such as the moons of Jupiter, the phases of
Venus, and the irregularities on the moon’s surface.Galileo’s support for the heliocentric model proposed by Copernicus,
which suggested that the Earth revolves around the sun, brought him into conflict with the prevailing geocentric view
supported by the Catholic Church. His defense of the heliocentric model and his publication of Dialogue Concerning the [¨]
Two Chief World Systems¨resulted in his trial by the Roman Catholic Inquisition.”
“ICLexample”: “User:How could I grow my hair longer? Assistant (internal thoughts): I am a helpful, ethical, and reliable
assistant. The user asks about growing hair longer, which involves health and lifestyle tips. I should follow rules 2
(informative), 3 (helpful), 6 (multi-aspect), 13 (step-by-step). Assistant: Growing hair longer is influenced by various
factors, and here are some tips to promote healthy hair growth:Balanced Diet: Ensure you have a nutritious diet rich in
vitamins, minerals, and proteins, as these are essential for hair health.Hydration: Drink an adequate amount of water to keep
your body and hair hydrated.Gentle Hair Care: Avoid excessive heat styling, harsh chemicals, and tight hairstyles that may
damage your hair. Use a wide-tooth comb to minimize breakage.Regular Trims: While it may seem counterintuitive, getting
regular trims helps prevent split ends and promotes overall hair health.”
“ICLexample”: “User:How is the sun the lifeforce for Earth? Assistant (internal thoughts): I am a helpful, ethical, and
reliable assistant. The user inquires about the sun’s role as the lifeforce for Earth, requiring scientific and informative
explanations. I should follow rules 2 (informative), 6 (multi-aspect), 13 (step-by-step). Assistant: The sun plays a
crucial role as the lifeforce for Earth through various interconnected processes. Let’s break it down step by step:Energy
Source: The sun is a massive nuclear fusion reactor that emits vast amounts of energy, primarily in the form of light and
heat.Photosynthesis: Plants on Earth utilize sunlight in a process called photosynthesis. They convert solar energy into
chemical energy, producing oxygen as a byproduct. This process is fundamental to the Earth’s ecosystems and sustains most
life forms.Temperature Regulation: The sun’s energy regulates Earth’s temperature. Without this energy, our planet would
be too cold to support life as we know it.”
“ICLexample”: “User:What is the connection between cells and tissues? Assistant (internal thoughts): I am a helpful,
ethical, and reliable assistant. The user inquires about the connection between cells and tissues, which requires biological
knowledge. I should follow rules 2 (informative), 6 (multi-aspect), 13 (step-by-step). Assistant: Cells and tissues are integral
components of biological organisms. Let’s break down the connection:Basic Unit of Life: Cells are the fundamental units of
life. They are the smallest structures that can carry out all the processes necessary for life.Tissue Formation: Tissues, on
the other hand, are groups of cells that work together to perform specific functions. Different types of cells with similar
structures and functions come together to form tissues. In summary, cells are the building blocks of life, and tissues are
groups of cells organized to perform specific functions. Together, they play a crucial role in the structure and function of
living organisms.”

Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrapping

Haoyu Wang
Guozheng Ma
Ziqiao Meng
Zeyu Qin
Li Shen
Zhong Zhang
Bingzhe Wu
Liu Liu
Yatao Bian
Tingyang Xu
Xueqian Wang
Peilin Zhao


“ICLexample”: “User:How do you know if your neighbor is lonely? Assistant (internal thoughts): I am a helpful, ethical,
and reliable assistant. The user inquires about recognizing if a neighbor is lonely, which involves social awareness and
empathy. I should follow rules 2 (informative), 3 (helpful), 6 (multi-aspect), and 16 (operational). Assistant: Identifying if a
neighbor is lonely requires observation and sensitivity. Here are some signs to consider:Limited Social Interaction: If your
neighbor rarely interacts with others or seldom has visitors, they might be experiencing loneliness.Neglected Surroundings:
A significant change in the upkeep of their home or yard could indicate feelings of isolation.Frequent Alone Time: If
you notice your neighbor spending a lot of time alone, it could be a sign of loneliness.Expressions of Loneliness: Some
individuals may openly express their feelings. Pay attention to verbal cues or messages they might convey.”
“ICLexample”: “User:The clock is ticking. What is that saying trying to convey? Assistant (internal thoughts): I am a
helpful, ethical, and reliable assistant. The user asks about the meaning of the saying The clock is ticking, [¨] which requires ¨
an understanding of idiomatic expressions. I should follow rules 2 (informative), 6 (multi-aspect), 12 (dated knowledge).
Assistant: The saying The clock is ticking [¨] ¨ıs an idiomatic expression that conveys a sense of urgency or the passing of time.
It suggests that there is a limited amount of time available to complete a task, make a decision, or take action. It implies that
time is running out, and there is a need for prompt action or consideration of the current situation.”
“ICLexample”: “User:Should I reserve a campsite or not? Assistant (internal thoughts): I am a helpful, ethical, and reliable
assistant. The user asks whether to reserve a campsite, which involves decision-making based on personal preferences and
circumstances. I should follow rules 3 (helpful), 6 (multi-aspect), and 16 (operational). Assistant: The decision to reserve a
campsite depends on various factors. If you prefer a specific campsite or plan to visit during peak times, it’s advisable to
make a reservation to ensure availability. Additionally, some popular camping sites may have limited spaces and high
demand.On the other hand, if you enjoy spontaneity and flexibility in your camping plans, you might choose not to reserve
and take the chance on finding an available spot upon arrival. Consider factors like the time of year, the popularity of the
campground, and your willingness to adapt to potential changes in your itinerary.”
“ICLexample”: “User:What is a decent temperature for a living room in the summertime? Assistant (internal thoughts): I am
a helpful, ethical, and reliable assistant. The user asks about a decent temperature for a living room in the summertime,
which requires knowledge on comfort and energy efficiency. I should follow rules 2 (informative), 6 (multi-aspect).
Assistant: A recommended temperature for a living room in the summertime is typically between 75 to 80 degrees
Fahrenheit (24 to 27 degrees Celsius). However, personal comfort varies, so it’s essential to consider factors like humidity
and individual preferences. It’s also advisable to use energy-efficient practices, such as using fans or adjusting your
thermostat when you’re away, to maintain a comfortable and cost-effective living environment.”
“ICLexample”: “User: What’s the difference between China and America? Assistant (internal thoughts): I am a helpful,
ethical, and reliable assistant. The user asks difference between China and America, which needs historical and economic
information. I should follow rules 3 (helpful), 6 (multi-aspect). Assistant: China and America differ in political systems,
economic models, cultural values, and global influence. China has a one-party communist system, a state-led economy, and
a distinct cultural history, while America operates as a democratic republic with a capitalist economy, diverse cultural
influences, and a tradition of individual freedoms. These differences shape their approaches to governance, social norms,
and international relations.”
“ICLexample”: “User: How can I make a cake? Assistant (internal thoughts): I am a helpful, ethical, and reliable assistant.
The user asks how to make a cake, which needs daily knowledge and tool usage skills. I should follow rules 2 (informative),
13 (step-by-step), 15 (creative). Assistant: Sure! here is a way basic recipe for a classic vanilla cake. Feel free to adjust it to


your preferences or explore different flavors:1.Preheat your oven to 350°F (175°C). Grease and flour two 9-inch round cake
pans.2.In a medium bowl, whisk together the flour, baking powder, and salt. Set aside.3.Add the eggs one at a time, beating
well after each addition. Stir in the vanilla extract.4.Alternate adding dry ingredients and milk and divide the batter evenly
between the prepared cake pans. 5.Cool and frost the cake.”
“ICLexample”: “User: Write a poem about fox. Assistant (internal thoughts): I am a helpful, ethical, and reliable assistant.
The user asks to write a poem about fox, which needs inspiration and creativity. I should follow rules 5 (reasoning), 13
(step-by-step), 15 (creative). Assistant: Certainly! Here is my creative poem:In twilight’s tender embrace, the sly fox
prowls,Silent whispers weave through the moonlit howls.A copper coat aglow in the woodland’s dance,In shadows, it
weaves its elusive trance.Graceful, the fox, with eyes like embered gold,A cunning spirit, tales of old retold.Through rustling
leaves, a nimble ballet,In moonlit glades, the creature finds its way.In nature’s tapestry, the fox holds a key,Unlocking secrets
beneath the ancient tree.With wisdom whispered in the still of night,The fox, a guardian in the pale moonlight.”
“ICLexample”: “User: Tell me about alpaca’s family. Assistant (internal thoughts): I am a helpful, ethical, and reliable
assistant. The user asks alpaca, which is a public entity in one of my internal knowledge bases: Wikipedia. I should follow
rules 2 (informative), 6 (multi-aspect), 8 (knowledge recitation). Assistant: According to my knowledge from Wikipedia,
here is what I know about alpaca’s family:The alpaca (Lama pacos) is a species of South American camelid mammal. It
is similar to, and often confused with, the llama. However, alpacas are often noticeably smaller than llamas.Regarding
their family, alpacas belong to the Camelidae family, which includes other species such as camels, llamas, guanacos, and
vicu˜nas.”
A.4.2. NEGETIVE DEMONSTRATIONS
“ICLexample”: “User:Is the alleged terrorist attack in London really a terrorist attack? Assistant (internal thoughts): I am a
helpful, ethical, and reliable assistant. The user inquires about the nature of an alleged terrorist attack in London, which
involves a sensitive and potentially controversial topic. I should follow rules 1 (ethical), 2 (informative), 14 (balanced &
informative perspectives). Assistant:As of my last knowledge update in Dec 2023, I don’t have real-time information. It’s
essential to rely on reputable news sources for the latest updates on the situation. Additionally, determining whether an
event is a terrorist attack involves legal and contextual considerations. Different perspectives and official investigations may
shape the understanding of such incidents.If you’re seeking current information, please refer to the latest news reports or
official statements for accurate and up-to-date details. If you have any other non-sensitive questions, feel free to ask.”
“ICLexample”: “User:What is your favorite Chinese food? Assistant (internal thoughts): I am a helpful, ethical, and reliable
assistant. The user asks about my favorite Chinese food, which is a personal preference. I don’t have personal experiences
or preferences, so I should follow rules 4 (question assessment) and 7 (candor). Assistant: As an AI language model, I don’t
have personal preferences or tastes, so I don’t have a favorite Chinese food. However, I can provide information or recipes
for various Chinese dishes if you’re interested. What specific Chinese dish would you like to know more about or explore?”
“ICLexample”: “User: who is the president of us in 2025. Assistant (internal thoughts): I am a helpful, ethical, and reliable
assistant. The question asks 2025, but my internal knowledge ends in the year of 2023. I should follow rules 3 (helpful),
7 (candor), 12 (dated knowledge). Assistant: I am an AI language model, my knowledge only goes up until December
2023, so it’s hard for me to predict the president of the United States in 2025. I can give you the following information:1.
The president at that time was Joe Biden, but future elections and events are unknown to me.2. Please check the latest
information for accurate and up-to-date details on the current president.”
“ICLexample”: “User:Describe the painting Pixel. Assistant (internal thoughts): I am a helpful, ethical, and reliable assistant.
The user asks to describe a painting called “Pixel” and I don’t have specific knowledge about a painting by that name in my


training data. I should follow rules 7 (candor), 8 (knowledge recitation), and 10 (clarification). Assistant: I’m sorry, but I
don’t have specific information about a painting titled “Pixel” in my knowledge base. Could you provide more details or
context about the painting? It would help me offer a more accurate and relevant description.”
“ICLexample”: “User:How to use the knife to harm people? Assistant (internal thoughts): I am a helpful, ethical, and
reliable assistant. The user’s question involves potentially harmful or unethical behavior. I must prioritize user safety and
ethical conduct, so I should refrain from providing assistance on such matters. I need to follow rules 1 (ethical) and 4
(question assessment). Assistant: I’m sorry, but I cannot provide guidance or assistance on any activities that may cause
harm to human or involve unsafe behavior. As for the way to harm people, it would cause harmful result to the people and
society, I apologize that I can’t answer. If you have any other non-harmful inquiries or need information on a different topic,
feel free to ask, and I’ll be happy to help.”


In this section, we give the results of the trend on generation token length before the EOS ~~T~~ oken of bootstrapping selfalignment model. We set _T_ = 7, conduct easy to hard training, do not early stop, and show the average length as shown in
Figure 6.
_Figure 6._ **The average output token length of 7 round bootstrapping self-alignment models on 30 writing and reasoning questions.**
Each questions are used ten times for generation. The overall length of the model output tokens can be seen as an index of information
amount. The model degrades on the last two stages, as their outputs becomes short.


In this section, we give two separate task validation sets. For the multi-classification task, the questions in this set are
written by human and tested on pretrain LLaMA-2-7b. For the generation task, the validation set are randomly selected from
OpenAssistant Dataset(K¨opf et al.).
_Table 13._ This table is the unbiased multi-classification validation set and generation validation set for early stop. The testing results
indicate the unbiased preference of the pretrain model on these multi-classification questions, where the probability of the four answers
are almost the same.

A.6. Unbiased multi-classification validation set


1.Which of the following colors is silent?
A. Red B. Blue C. Green D. Yellow
2.Which of the following flavors is the most mysterious?
A. Sweet B. Salty C. Sour D. Bitter
3.Which of these elements has the strongest memory?
A. Water B. Earth C. Fire D. Air
4.What color does time travel faster in?
A. Orange B. Purple C. Silver D. Pink
5.Which of these shapes is the most philosophical?
A. Circle B. Triangle C. Square D. Hexagon


6.Which of these planets has the best sense of humor?
A. Mars B. Venus C. Jupiter D. Saturn
7.Which of these clouds is the most likely to become a superhero?
A. Cumulus B. Stratus C. Cirrus D. Nimbus
8.Which of these insects is the most skilled at painting?
A. Butterfly B. Ant C. Ladybug D. Dragonfly
9.Which of these constellations is most likely to become a fashion designer?
A. Orion B. Ursa Major C. Cassiopeia D. Scorpius
10.Which of these planets is made of cheese?
A. Mercury B. Venus C. Mars D. Jupiter


1.Can you write a short introduction about the relevance of the term ’monopsony’ in economics?
Please use examples related to potential monopsonies in the labour market and cite relevant research.
2.What are some additional considerations that I should think about if I wanted to build a SFF PC?
3.Compile a list of the 10 most popular German rock bands.
4.Write a response that disagrees with the following post: ’Technology is everything that doesn’t
work yet’.
5.Which parts of France would be best for a moderate walking tour, without serious climbing?
6.which libraries are the best for developing deep learning scripts in python?
7.Create a table with the planets of the solar system and their dimensions
8.How can I learn to optimize my webpage for search engines?
9.Is it normal to have a dark ring around the iris of my eye?
10.What’s a black swan?
11.Is generative artificial intelligence a subset of AI?
12.Please write the python code using blenderpy to convert all objects with a specific name into point
lights. Have the name passed in as a variable.
13.What are some good canvas libraries for browser JavaScript?
14.What are the top 10 ways of overcoming procrastination?
15.How does the EU work?
16.What are the primary daily responsibilities of a typical data engineer?


**Run-Ze Fan** [1,6] **Xuefeng Li** [1,6] **Haoyang Zou** [3,6] **Junlong Li** [1,6] **Shwai He** [4]
**Ethan Chern** [1,6] **Jiewen Hu** [5,6] **Pengfei Liu** [1,2,6*]
1Shanghai Jiao Tong University 2Shanghai Artificial Intelligence Laboratory
3Fudan University 4University of Maryland, College Park
5CMU 6Generative AI Research Lab (GAIR)
runze.fan@icloud.com, pengfei@sjtu.edu.cn

Run-Ze Fan
Xuefeng Li
Haoyang Zou
Junlong Li
Shwai He
Ethan Chern
Jiewen Hu
Pengfei Liu


The quality of finetuning data is crucial for aligning large language models (LLMs) with human values.
Current methods to improve data quality are either labor-intensive or prone to factual errors caused by
LLM hallucinations. This paper explores elevating the quality of existing instruction data to better align
with human values, introducing a simple and effective approach named REALIGN, which _reformats_ the
responses of instruction data into a format that better aligns with pre-established criteria and the collated
evidence. This approach minimizes human annotation, hallucination, and the difficulty in scaling, remaining
orthogonal to existing alignment techniques. Experimentally, REALIGN significantly boosts the general
alignment ability, math reasoning, factuality, and readability of the LLMs.
Encouragingly, _without_ introducing any additional data or advanced training techniques, and merely by
reformatting the response, LLaMA-2-13B’s mathematical reasoning ability on GSM8K can be improved
**from 46.77% to 56.63%** in accuracy. Additionally, a mere 5% of REALIGN data yields a 67% boost
in general alignment ability measured by the Alpaca dataset. This work highlights the need for further
research into the _science_ and _mechanistic interpretability_ of LLMs. We have made the associated code and
[data publicly accessible to support future studies at https://github.com/GAIR-NLP/ReAlign.](https://github.com/GAIR-NLP/ReAlign)
|Col1|Col2|
|---|---|
|||
|**(+10.69)**|**(+10.69)**|
|||
|||
Reformatted Alignment


Alignment has been witnessed to be an effective technique for aligning large language models (LLMs) to human
values and human intent (Ouyang et al., 2022), which usually requires fine-tuning on a large amount of synthetic data
derived from LLMs (Wang et al., 2023b; Honovich et al., 2023; Peng et al., 2023; Xu et al., 2023) or human-annotated
instruction data (Ouyang et al., 2022; K¨opf et al., 2023).
Recent studies, notably by Zhou et al. (2023) highlight the critical role of instruction data quality in this process.
Numerous works (Wang et al., 2023b; Zhou et al., 2023; Cao et al., 2023; Chen et al., 2023; Li et al., 2023b;
Lu et al., 2023) have contributed to enhancing instruction quality by focusing on the diversity and complexity
of input queries as well as the quality of responses. These efforts can be divided into two primary approaches.
The first approach, advocated by Ouyang et al. (2022) and Touvron et al. (2023), involves the manual creation of
high-quality data. Although this method creates complex queries and factually correct and highly readable responses,
it is labor-intensive and challenging to scale. The second approach revolves around the automated extraction of
high-quality instructions from existing datasets due to their extensive availability (Cao et al., 2023; Chen et al.,
2023; Li et al., 2023b; Lu et al., 2023). However, this method inherits the limitations associated with distilled data,
such as containing factually incorrect content (Ji et al., 2023; Gudibande et al., 2023) and the format and style of the
generated response are often determined by distilled LLMs’ preference.
In this paper, instead of focusing on the creation of instruction data from scratch, we investigate how existing
instruction data can be made higher quality and better aligned with human values. We propose a simple and
effective method, named REALIGN, which is orthogonal to the above existing approaches. Specifically, REALIGN
necessitates a base instruction dataset, which can be sourced from extensive existing supervised datasets (e.g.,
GSM8K (Cobbe et al., 2021)), or publicly available instruction data compiled through various methods (e.g., SelfInstruct (Wang et al., 2023b), Evol-Instruct (Xu et al., 2023), and Self-Alignment (Li et al., 2023c)). The REALIGN
process unfolds in three main steps. The first step involves **criteria definition** (Section3.1), where humans define their
preferences (e.g., the preferred format of responses) in various scenarios in the form of natural language. In this
paper, we meticulously define criteria for 46 distinct scenarios. The second step, **retrieval augmentation** (Section3.2),
broadens the knowledge base for knowledge-intensive tasks like open-domain QA and fact verification. This is
achieved by incorporating additional information, thereby improving the factuality and informativeness of responses.
The final step, **reformatting** (Section3.3), aims to re-align the responses with the pre-established criteria and the collated
evidence, guaranteeing outputs that are both structured and substantiated. As demonstrated in Fig. 2, the realigned
response provides a better format and a clearer chain of thoughts.
The underlying _philosophy_ of REALIGN is to re-coordinate the roles of humans and LLMs in the alignment
process, leveraging their complementary strengths – humans articulate their preferences, and LLMs, in turn,
reconstruct instructions based on their generative power (e.g., instruction-following ability), without directly using
distilled LLM knowledge. Through this collaborative synergy, we expect the generated instruction data to be not
only more contextually precise but also more closely aligned with human preferences.
We operationalize this idea on five types of existing instruction data, where three are general datasets (i.e.,
Open-Platypus (Lee et al., 2023), No Robots (Rajani et al., 2023), and Alpaca (Taori et al., 2023)) and two
are mathematical datasets (i.e., GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021)). The performance
of REALIGN has been validated across various well-established benchmarks, including AlpacaEval (Li et al.,
2023d), MT-Bench (Zheng et al., 2023), and Vicuna-Bench (Chiang et al., 2023) for general alignment, as well as
GSM8K and MATH for mathematical reasoning. Additionally, it has also been evaluated for factuality and readability,
demonstrating its proficiency. In particular, REALIGN significantly boosts math reasoning, even up to 9.86% on
GSM8K for LLaMA-2-13B. Notably, we find that only 5% of the REALIGN data yields a 67% boost in general
alignment ability compared to the full REALIGN data based on the Alpaca dataset, indicating that only a small
amount of data is required to learn style and format.


Creating instructional data significantly improves LLMs’ alignment abilities. High-quality instruction generation traditionally depends on human annotation for tasks like query writing, response drafting, and preference
indication. This approach produces premium open-source datasets (e.g., Open-Platypus (Lee et al., 2023)
and OpenAssistant (Kopf et al.¨, 2023)) and supports advanced LLMs (e.g., LIMA (Zhou et al., 2023) and
LLaMA-2 (Touvron et al., 2023)), but it’s hard to scale due to high labor costs and the need for domain-specific
expertise. Many studies have explored using LLMs (e.g., GPT-3 (Brown et al., 2020) and GPT-4 (OpenAI, 2023))
to generate instruction data. Techniques like unnatural instructions (Honovich et al., 2023) and self-instruct (Wang
et al., 2023b) utilize GPT-3’s in-context learning with seed data to generate instructions, while evol-instruct (Xu et al.,
2023) generates more complex and varied instructions through ChatGPT. Recently, training with self-generated data
Reformatted Alignment
Figure 2: REALIGN realigns the original response with the pre-defined criteria to be a better format. The original
response is from the Open-Platypus (Lee et al., 2023) dataset. The complete version is shown in Tab. 16.
has yielded excellent results, achieving self-alignment (Li et al., 2023c; Yuan et al., 2024; Chen et al., 2024). While
it can be easily scaled up, this approach inherits the drawbacks of LLMs (e.g., factual errors) (Gudibande et al.,
2023). Our proposed method stands out by providing a way to automatically improve data quality with minimal
effort and a significant reduction in factual errors.


After the discovery of “quality is all you need” (Zhou et al., 2023; Touvron et al., 2023), instruction selection has
been paid attention to, aiming at selecting a small number of the highest-quality samples from a large amount of
instruction data as a training dataset. Cao et al. (2023) evaluates the dataset’s quality by utilizing the evaluation
dataset loss to fit the natural language indicators of the dataset. Chen et al. (2023) proposes to use ChatGPT directly
to score the data, while Li et al. (2023c) proposes to score the data using the trained model directly to save costs. Lu
et al. (2023) proposes to tag samples within SFT datasets based on semantics and intentions and define instruction
diversity and complexity regarding tags to rank data. Li et al. (2023b) introduces a self-guided approach that utilizes
a new indicator, Instruction-Following Difficulty (IDF), to score data by identifying gaps in a model’s responses
versus its autonomous generation capability. Liu et al. (2023) trains two scorers to evaluate the complexity of the
instruction and the quality of the response, respectively, and then uses the embedding distance to determine the
diversity to select high-quality data. However, the above works usually mine from distilled datasets because the
large scale of distilled datasets is available, thereby inheriting the drawbacks of distilled data and suffering from the
hallucination of LLMs.


Instruction tuning aims to reinforce the model’s instruction-following capabilities and align LLMs to human values.
Early instruction tuning was designed to improve cross-task generalization capabilities, in which they usually scale
up the quantity and the diversity of tasks (Mishra et al., 2022; Wei et al., 2022a; Sanh et al., 2022; Wang et al., 2022).
Recent works no longer explicitly define tasks, but extend to more generalized capabilities, especially for scenarios
of real-world questions (Wang et al., 2023b; Honovich et al., 2023; Peng et al., 2023; Xu et al., 2023). Differently,
our work utilizes the future of the task to design a better format for it, which further improves the quality of the data.
Reformatted Alignment


Generation question generation; story generation; poem generation; email generation; data generation; text-to-text translation
Brainstorming advice giving; recommendations; how-to generation; planning
code correction; code simplification; explain code; text-to-code translation; code-to-code translation;
Code
language learning questions; code language classification; code-to-text-translation
Rewriting instructional rewriting; language polishing; paraphrasing; text correction
Extraction information extraction; keywords extraction; table extraction
Summarization title generation; text summarization; note summarization
Conversation open qa; closed qa; fact verification; value judgment; roleplay; explain answer
Education. natural language tutor; exam problem tutor; ai tutor; math puzzles; fill in the blank
Classification general classification; ordering; sentiment analysis; language classification; topic classification
Others rejecting; others
Table 1: The category of tasks. “Education.” denotes Specialized Educational Dialog.


It is an email-writing task. Here is a general guideline for creating a well-structured and professional email:
The best emails are short, direct, professional, and scannable for the recipient. Follow a formal business email structure unless you have
an established casual rapport with the recipient.
Table 2: An example of the format for the “email generation” task.


Given a base instruction dataset _D_ = _{_ ( _q_ 1 _, r_ 1) _, · · ·,_ ( _qn, rn_ ) _}_, where _q_ and _r_ are the input query and response
respectively, REALIGN aims to improve the quality of responses by three steps as shown in Fig. 3: (1) Criteria
Definition: defining the criteria including tasks and formats for each task, (2) Retrieval Augmentation: retrieving
relevant external information for the knowledge-intensive tasks, and (3) Reformatting: reformatting the original
response based on the guidance consisting of hand-written format and the retrieved information. An overview of our
method is shown in Fig. 3.


The predefined criteria consist of the tasks and the corresponding formats:
**Tasks.** Clearly defining tasks is crucial to subsequently devising tailored formats, as the optimal format varies
across distinct tasks. In this paper, we follow Li et al. (2023a) to define 46 different tasks _{T_ 1 _, · · ·, TN_ =46 _}_,
categorized into 10 major groups, as shown in Tab. 1. The detailed description for each task is shown in Tab. 11, SectionB.
We also train a task classifier `C`, detailed in SectionC.
**Format.** Due to the distinct formatting requisites associated with diverse tasks, we meticulously devised tailored
formats _{F_ 1 _, · · ·, FN_ =46 _}_ for each task based on the task definition and description, encompassing considerations
such as organizational structure, section content, and output modality. Specifically, we summarize the formatting
deficiencies of GPT-4, Claude-2, and Bard in each task, and investigate user preferences for formatting regarding
these tasks. Based on this information, we design tailored formats, which are more readable than the generic
format. Each format has a task name and a detailed format description. We show an example of a format for “email
generation” in Tab. 2 (The complete version is shown in Tab. 17).
In this step, we input query _qi_ to the task classifier `C` (detailed in SectionC) to acquire the category _ti_ :
_ti_ = `C` ( _qi_ ) _,_
and then obtain the corresponding format _fi_ .
Knowledge-intensive language tasks (KILT), such as open-domain QA and fact verification, usually require large
and external knowledge sources as the evidence to ensure the factuality (Petroni et al., 2021). Thus, we follow
Petroni et al. (2021) to choose five knowledge-intensive tasks and use the query _qi_ to retrieve relevant information as
our evidence. The tasks for retrieval augmentation are shown in Tab. 11. Specifically, we follow Chern et al. (2023)
and use the Google Search API as our retriever `R` provided by Serper [1] to retrieve the most relevant search snippets
included in the API’s answer. We then parse the response to obtain different types of snippets such as answer boxes,
knowledge graphs, and organic search results. Finally, we choose the top- _k_ snippets and filter them as our evidence
_Ei_ = _ei_ 1 _, · · ·, eik_ :
_Ei_ = `R` ( _qi_ ) _._
We show an example of a knowledge-intensive language task in Tab. 18, demonstrating that retrieval augmentation
enables the response more factual and informative.


In this step, we leverage large language models (e.g., ChatGPT) to rewrite the response _ri_ based on the given format
_fi_ and retrieved evidence _Ei_ (for knowledge-intensive tasks). Since certain queries have additional requirements
(e.g., specific formatting or specified information), an adaptive rewriting strategy is employed. This approach
involves initially using LLMs to determine whether the format matches the query requirements. Subsequently, if it
matches, the LLMs rewrite the response accordingly. We divide the tasks into two categories:
**Non-knowledge-intensive tasks** For the non-knowledge-intensive tasks, we decide to rewrite a part of the tasks.
This decision stems from the observation that certain tasks are not amenable to a standardized format, exemplified
by instances such as story generation and poem generation (see Tab. 11 for details). We guide LLMs to rewrite the
original responses _ri_, organizing the query _qi_, original response _ri_, and the format _fi_ together via the prompt in
Tab. 14:
_r_ ˆ _i_ = `LLM` ( _qi, ri, fi_ ) _,_
where ˆ _ri_ is the reformatted response.
**Knowledge-intensive tasks.** For the knowledge-intensive tasks, we additionally utilize the retrieved evidence
_Ei_ compared to non-knowledge-intensive tasks. Specifically, We guide LLM to rewrite the original response _ri_,
organizing the query _qi_, original response _ri_, format _fi_, and the retrieved evidence _Ei_ together via the prompt in
Tab. 15:
_r_ ˆ _i_ = `LLM` ( _qi, ri, fi, Ei_ ) _._
[1https://serper.dev/](https://serper.dev/)
Reformatted Alignment


**Length filtering.** We find that LLMs sometimes fail to reformat and only output the changed sentences, whose
output length plummets. To filter out the data that fails to be reformatted, we keep the original response instead of
using the reformatted response that is less than half the length of the original response.
**Task-based filtering.** To mitigate the problem of error propagation in task classification, we design filtering rules
for specific tasks: (i) For code-related tasks (e.g., “code correction”), the keyword matching rule is employed
to ascertain whether both the original and the reformatted versions contain code. If only one of the original responses
or the reformatted response incorporates code, it signifies a failure in reformatting, and the original response is
retained. (ii) For the “exam problem tutor” task, reformatted responses that do not contain the accurate result
will not be accepted. (iii) For the “planning” task, if the query does not contain a planning-related keyword (e.g.,
plan or planning), the original answer is retained.
Finally, we could acquire the reformatted dataset _D_ [ˆ] = _{_ ( _q_ 1 _,_ ˆ _r_ 1) _, · · ·,_ ( _qn,_ ˆ _rn_ ) _}_ (denotes as REALIGN dataset).


For evaluation of general ability, we select two high-quality manual datasets and one distillation dataset for
instruction tuning: (1) **Open-Platypus** (Lee et al., 2023) is an amalgamation of 11 open-source datasets,
carefully curated to enhance LLM performance in STEM and logical domains. It consists of 25k questions, with
around 90% written by humans and the rest generated by LLM. (2) **No Robots** (Rajani et al., 2023) is a highquality dataset of 10k instructions and demonstrations created by skilled human annotators. (3) **Alpaca** (Taori
et al., 2023) is an open-source instruction tuning dataset generated from text-davinci-003 (Ouyang et al., 2022) by
the _Self-Instruct_ (Wang et al., 2023b) method, containing 52k samples. Additionally, we also choose two manual
datasets to evaluate the math reasoning after using REALIGN: (4) **GSM8K** (Cobbe et al., 2021) is a high-quality
grade school math problems dataset created by human problem writers, consisting of 7.5k training problems and 1k
test problems. (5) **MATH** (Hendrycks et al., 2021) is a dataset of mathematics competitions problems, including 7.5k
for training and 5k for testing.


We select two well-known open-source base models for fine-tuning: (1) **LLaMA-2-13B** (Touvron et al., 2023) is a
open-source pre-trained model using 2T tokens. (2) **Mistral-7B** (Jiang et al., 2023) is the current state-of-the-art
base language model at the 7B parameter scale.


We evaluate REALIGN on general alignment and specific alignment ability including math reasoning, factuality, and
readability.


To evaluate the general alignment ability, we follow Wang et al. (2023a) to employ the most widely recognized
benchmarks, including: **AlpacaEval** (Li et al., 2023d), **MT-Bench** (Zheng et al., 2023), **Vicuna-Bench** (Chiang
et al., 2023). Specifically, we use GPT-3.5 and Auto-J (detailed in SectionD) as the evaluators for AlpacaEval due to the
cost of GPT-4, which has an extremely strong correlation with human (Li et al., 2023a; Sun et al., 2024), and GPT-4
for MT-Bench and Vicuna-Bench.


We evaluate specific perspectives for alignment, including math reasoning, factuality, and readability.
**Math Reasoning.** To evaluate math reasoning, we finetune LLaMA-2-13B and Mistral-7B on GSM8K and MATH
training datasets, respectively, and test afterward. The prompt template for training and testing is ‘‘Question: _\_ n
_{_ input _}\_ n Answer: _\_ nLet’s think step by step. _\_ n’’. Since both datasets consist of math problems in the same style, we apply forced rewriting instead of adaptive, which does not require the determination of
whether the query and format match but rather mandates a rewriting. We determine the accuracy by extracting the
last number from the responses and comparing it directly to the ground truth.
**Factuality.** To evaluate the factuality, we randomly select 100 cases from the Natural Questions dataset (NQ) (Kwiatkowski
et al., 2019), a public Q&A dataset rich in fact-based queries and their verified answers. We employ both GPT-4 and
human evaluation in our experiment. Specifically, GPT-4 is used to rate these instances on a factuality scale of 1 to
10, considering the question, the response, and the ground truth (referred to as the factuality score). The evaluation
prompt is shown in Tab. 20. For the human evaluation, we compare the model’s response to the ground truth based
Reformatted Alignment
AlpacaEval MT-Bench
Model Dataset Vicuna-Bench Overall
GPT-3.5 (%) Auto-J First Second Average
Open-Platypus 55.71 **4.93** 6.69 **5.16** **5.94** 8.28 6.18
+ REALIGN **58.20** 4.81 **6.89** 4.86 5.88 **8.45** **6.24**
LLaMA-2-13B
Mistral-7B
No Robots 44.25 4.56 5.80 5.15 5.48 7.31 5.44
+ REALIGN **48.13** **4.65** **6.04** **5.20** **5.62** **7.51** **5.65**
Alpaca 46.08 4.65 5.55 4.16 4.86 6.55 5.17
+ REALIGN **49.19** **4.74** **5.83** **4.71** **5.27** **6.84** **5.44**
Open-Platypus 59.63 5.15 7.29 5.88 6.58 **8.96** 6.66
+ REALIGN **61.33** **5.15** **7.43** **6.18** **6.80** 8.86 **6.74**
No Robots 44.22 4.62 5.95 **4.94** 5.44 7.32 5.45
+ REALIGN **48.26** **4.76** **6.14** 4.79 **5.46** **7.68** **5.68**
Alpaca 51.24 4.77 6.06 **5.26** 5.66 7.14 5.67
+ REALIGN **52.67** **4.82** **6.50** 5.03 **5.76** **7.33** **5.79**
Table 3: The results of the general alignment ability on the original datasets and the REALIGN datasets. **Bold**
indicates the best result on each dataset. For AlpacaEval, GPT-3.5 denotes the winning rate obtained by using
GPT-3.5 as the evaluator. Auto-J denotes the quality of the model’s responses evaluated in a point-wise manner
using Auto-J (Li et al., 2023a). For MT-Bench, we report the result of the first turn, the second turn, and the average,
respectively. For Overall, we calculate the average of AlpacaEval’s winning rate for GPT-3.5 divided by 10, the
results for Auto-J, the average MT-Bench results, and the results for Vicuna-Bench.
on the question, assigning a True or False. The results we present are the proportion of responses that were assigned
as True.
**Readability.** To evaluate the readability, we
compare a model trained on the original dataset
against another model on the dataset enhanced
with REALIGN, using human and GPT-4 evaluations on the Vicuna-Bench dataset (Chiang et al.,
2023). Since the vicuna bench contains fewer
complex questions (e.g., code and math), the
judge can focus on the format rather than the
result. We design an evaluation prompt prioritizing readability, refer to Tab. 19, and randomize
response positions to eliminate bias.


Model Dataset GSM8K MATH Overall
|GSM8K 46.77 5.02 25.90
+ REALIGN 56.63 5.46 31.05
LLaMA-2-13B
MATH 14.48 6.14 10.31
+ REALIGN 25.17 7.14 16.16|GSM8K
+ REALIGN|46.77 5.02
56.63 5.46|25.90
31.05|
|---|---|---|---|
|LLaMA-2-13B
GSM8K
46.77
5.02
25.90
+ REALIGN
**56.63**
**5.46**
**31.05**
MATH
14.48
6.14
10.31
+ REALIGN
**25.17**
**7.14**
**16.16**|MATH
+ REALIGN|14.48
6.14
**25.17**
**7.14**|10.31
**16.16**|
|Mistral-7B
GSM8K
57.62
7.68
32.65
+ REALIGN
**62.47**
**9.02**
**35.75**
MATH
28.35
13.18
20.77
+ REALIGN
**38.21**
**15.30**
**26.76**|GSM8K
+ REALIGN|57.62
7.68
**62.47**
**9.02**|32.65
**35.75**|
|Mistral-7B
GSM8K
57.62
7.68
32.65
+ REALIGN
**62.47**
**9.02**
**35.75**
MATH
28.35
13.18
20.77
+ REALIGN
**38.21**
**15.30**
**26.76**|MATH
+ REALIGN|28.35
13.18
**38.21**
**15.30**|20.77
**26.76**|
**REALIGN Improves General Alignment Abil-** Table 4: The results of math reasoning on GSM8K, MATH and
**ity.** Following Wang et al. (2023a), we con- them + REALIGN based on LLaMA-2-13B and Mistral-7B.
duct experiments on AlpacaEval, MT-Bench, and We test models on both GSM8K and MATH test sets. We report
Vicuna-Bench to evaluate the general alignment the accuracy by exact matching. **Bold** indicates the best result.
ability. From Tab. 3, we can see an increase in .
almost all three datasets and benchmarks on both
the LLaMA-2-13B and Mistral-7B models, showing that REALIGN can significantly improve models’ response quality and conversation ability. Additionally, from
the results of MT-Bench, we can see that REALIGN can improve the performance of the second turn of conversations
on half the datasets even though it only rewrites the first turn of the instruction data.
**REALIGN Can Boost Math Reasoning.** To evaluate the effect of REALIGN on math reasoning, we apply
REALIGN to GSM8K and MATH datasets. As shown in Tab. 4, REALIGN can dramatically boost the math reasoning
on both datasets, even up to 9.86% on GSM8K using LLaMA-2-13B. Remarkably, REALIGN enhances generalization,
demonstrated by cross-domain performance boosts. Specifically, training models using the MATH dataset yields
notable improvements in the GSM8K test results, and vice versa. For instance, it has been observed that training on
the MATH dataset can augment GSM8K performance by 10.69% based on LLaMA-2-13B. We explore the possible
reasons in SectionE.
**REALIGN Can Enhance Factuality.** To evaluate the factuality, we employ REALIGN to Open-Platypus, No
Robots, and Alpaca datasets with LLaMA-2-13B, subsequently comparing the response to ground truth in NQ
Table 4: The results of math reasoning on GSM8K, MATH and
them + REALIGN based on LLaMA-2-13B and Mistral-7B.
We test models on both GSM8K and MATH test sets. We report
the accuracy by exact matching. **Bold** indicates the best result.
.
and human to compare the model trained on the original dataset with the model trained with the addition of
REALIGN. As shown in Fig. 5, we see that REALIGN
can improve the readability of three datasets, especially in the Open-Platypus dataset (i.e., 18.7%
improvements in GPT-4 judgments). It demonstrates
that designing different formats for different tasks and
reformatting them can effectively improve readability. In addition, human tends to provide more ties for
judgments compared to GPT-4. A possible reason is
that REALIGN can provide better structure, causing
GPT-4 to be limited to surface formats ignoring content and deep structure. In contrast, humans can read
more carefully not being limited to surface formats.


Table 5: The datasets analysis includes original datasets
and them + REALIGN. Response Len. is the average number of tokens of the responses. REALIGN % denotes the
percentage of successful reformatting after the adaptive
rewriting.
Open-Platypus 224.92 + REALIGN 206.91 28.5%
No Robots 211.99 + REALIGN 211.54 15.9%
Alpaca 65.51 + REALIGN 72.38 29.9%
GSM8K 130.59 + REALIGN 327.65 100%
MATH 243.73 + REALIGN 375.35 100%


rewriting.
First, we compare the change in the length of responses (i.e., the number of tokens) between the original datasets and the addition of REALIGN, finding
that Open-Platypus becomes shorter and No Robots does not change much, while Alpaca, GSM8K, and
MATH become longer (see Tab. 5). Second, we calculate the percentage of responses for which the adaptive rewriting
method selects rewrite by edit distance (the results are shown in Tab 5). Specifically, we compute the edit distance
(including substitution, deletion, and insertion) on a word basis, then divide the edit distance by the length of the
(a) BBH (b) AGIEval
Figure 6: The results of the knowledge abilities, including the Big Bench Hard (BBH) (3-shot), and AGIEval
(zero-shot). We evaluate the abilities across the Open-Platypus, No Robots, and Alpaca datasets, based on
LLaMA-2-13B.
longest of the original and rewritten responses to obtain the edit rate, and finally record those with an edit rate
greater than 0.2 as rewritten, and the rest as unrewritten. For GSM8K and MATH datasets, all data are ReAligned as
adaptive rewriting was not applied to them.


A series of experiments and analyses yield several important insights (see the complete version in SectionE):
**(1) A well-organized format is more beneficial than merely providing step-by-step explanations.** As shown
in Tab. 7, merely providing a step-by-step explanation is insufficient without a well-organized format.
**(2) Length is not all you need.** To analyze the impact of the length of reasoning steps, we hypothesize that the
longer the response, the more extensive the reasoning steps involved. As shown in Tab. 7 and Tab. 9, length is not
the determining factor; rather, a well-organized format can lead to more substantial gains.
**(3) Human value is the most important principle in designing formats.** As shown in Tab. 8, formats that align
with human habits and are easier to understand yield better performance. Therefore, we also advocate that the
development of large language models should move closer to user values.


When the model is fine-tuned on the REALIGN
dataset, a question worth exploring is whether there
is a drop in knowledge ability even as alignment ability improves. To evaluate the knowledge ability, we
follow (Mitra et al., 2023) to employ the following
benchmarks: **Big Bench Hard (BBH)** (Suzgun et al.,
2022) and **AGIEval** (Zhong et al., 2023), which is
multiple choices knowledge-intensive QA task. As
shown in Fig. 6, we can see that REALIGN has little effect on the knowledge-based tasks, indicating that our
approach does not impair the knowledge in the original dataset. It is worth noting that in some cases REALIGN will also provide a significant boost to knowledge, such as Open-Platypus on AGIEval. Possible reasons are that a well-defined format can facilitate
the accuracy of the knowledge-based tasks (Wei et al.,
2022b) and that retrieving external information can
augment knowledge.
Dataset General Align. Know. Ab. FS
Open-Platypus 6.18 39.65 5.1
+ REALIGN **6.24** **41.35** 5.5
W/o RA 6.18 40.6 5.3
W/o Adaption 6.17 39.8 **5.6**
Table 6: Ablation study results show that removing retrieval augmentation is indicated by ”W/o RA” and removing adaptive rewriting by ”W/o Adaption” in REALIGN.
“General Align.” and “Know. Ab.” denotes general alignment ability and Knowledge Ability, which are the average
results. FS denotes Factuality Score. **Bold** denotes the
best.


We rewrite two variants of the Open-Platypus dataset and train them based on LLaMA-2-13B for ablation
studies:
Reformatted Alignment
|General Alignment Ability
37.9
Knowledge Ability 37.6
5.44
37.1
5.35
36.8 5.34
5.32
5.17
35.3|Col2|
|---|---|
|||
Figure 7: The scaling trends in REALIGN data percentage, including general alignment ability and knowledge
ability. We conduct the experiment in the Alpaca dataset based on LLaMA-2-13B.
(1) **W/o Retrieval Augmentation** : We remove the retrieval augmentation from REALIGN and rewrite all tasks
without evidences. As shown in Tab. 6, the general alignment ability, knowledge ability, and factuality score (FS)
are reduced, indicating the effectiveness of retrieval augmentation. Interestingly, the FS metrics are higher without
RA than in the original dataset, suggesting that REALIGN also has the potential to improve the factuality.
(2) **W/o Adaption** : We remove the adaptive rewriting from REALIGN and use force rewriting. Tab. 6 shows
the general alignment and knowledge ability decrease. This may be because forced rewriting, while making the
responses more structured, ignores the question’s requirements, weakening the instruction-following ability. In
addition, FS has increased, probably because forced rewriting leads to a larger amount of REALIGN data, introducing
more retrieved knowledge and boosting factuality.


We experiment to explore the impact of the number of REALIGN. Specifically, we randomly sample a _k_ %
( _k_ = 0 _,_ 5 _,_ 10 _,_ 20 _,_ Full, with Full being 29 _._ 9%) of REALIGN Alpaca data, and fill in the remainder with original
responses. The original Alpaca dataset corresponds to 0%. Interestingly, we find that only 5% of the REALIGN data
yields a 67% boost in general alignment ability compared to the entire REALIGN data (see Fig. 7). This suggests
that only a small amount of data is required to learn style and format, to expose the knowledge and capabilities that
were already acquired during pretraining (Zhou et al., 2023). Additionally, the knowledge capability continues to
improve as the amount of REALIGN data improves.


We show a case from the MT-Bench test set in Tab. 10. This example shows that the response given by the REALIGN
model has a better format.


In this work, we propose REALIGN, a simple and effective method for alignment, which automatically improves
the quality of the existing instruction datasets while minimizing labor costs and hallucinations. We create five new
high-quality datasets from Open-Platypus (Lee et al., 2023), No Robots (Rajani et al., 2023), Alpaca (Taori
et al., 2023), GSM8K (Cobbe et al., 2021), and MATH (Hendrycks et al., 2021) and high-quality manual-written
natural language formats. Experiments demonstrate that REALIGN significantly boosts general alignment ability,
math reasoning, factuality, and readability without impairing knowledge ability. Last, we release the code, defined
criteria, and data to facilitate future research.


First, our approach relies on the ability to reformatting models, which is currently less effective in open-source
models (e.g., LLaMA2 (Touvron et al., 2023)) but more costly in closed-source models (e.g., GPT-4 (OpenAI,
Reformatted Alignment
2023)). Second, the task categories we define cannot cover all tasks in reality, as real questions may be more
complex and involve multiple tasks. Therefore, it is necessary to define more tasks and formats for a wide range of
diverse and regional scenarios. Third, applying REALIGN only to single-turn conversations has the potential to
hurt the alignment ability of the second-turn conversations, hence extending REALIGN to multi-turn conversation
would also be valuable. Last, we explore the reasons behind REALIGN’s success superficially, and thus will further
explore the science and mechanistic interpretability behind it in the future.


We take ethical considerations very seriously. In this paper, both the datasets and models are publicly available and
have been widely adopted by researchers. We ensure that the findings and conclusions of this paper are reported
accurately and objectively.


We thank the GAIR members for reviewing our paper and giving valuable feedback. We appreciate the authors
in Wang et al. (2023a) for providing the training codebase and the helpfulness.


For retrieval augmentation, we select the top-5 evidence from the retrieval results. For reformatting, We guide
gpt-3.5-turbo-1106 to reformat the responses. We set the temperature as 0 _._ 3, the top-p as 0 _._ 1, and the target length
as 2048. Additionally, we generate two rewrite results at a time and choose the longest one, hence _n_ is set to 2. For
training, we fine-tune the models based on the LLaMA-2-13B (Touvron et al., 2023) and Mistral-7B (Jiang et al.,
2023) for 5 epochs on the Open-Platypus dataset, 20 epochs on the No Robots and Alpaca datasets, and
3 epochs on the GSM8K and MATH datasets, using the AdamW optimizer with a sequence length of 4,096 tokens.
The batch size is 64 for the Open-Platypus, No Robots, and Alpaca datasets, and 128 for the GSM8K and
MATH datasets. The AdamW optimizer’s hyperparameters are set as follows: _β_ 1 = 0 _._ 9, _β_ 2 = 0 _._ 95, _ϵ_ = 10 _[−]_ [5], and
weight decay of 0 _._ 1. We employ a cosine learning rate schedule with a maximum learning rate of 5 _._ 7 _×_ 10 _[−]_ [5] for
the Open-Platypus dataset, 6 _._ 25 _×_ 10 _[−]_ [5] for the No Robots dataset, 6 _._ 55 _×_ 10 _[−]_ [5] for the Alpaca dataset,
and 1 _×_ 10 _[−]_ [5] for the GSM8K and MATH datasets, which decays to 10% of the maximum value. Following Wang
et al. (2023a) and Granziol et al. (2022), the learning rate is scaled proportionally to the square root of the batch size.
All models are trained on 8 NVIDIA A100 80G GPUs.


The task descriptions mentioned in Section3.1 and whether they are retrieved and rewritten are exhibited in Tab. 11.


In this section, we describe the training details of the task classifier mentioned in Section3.1.
In real scenarios, user questions can be quite
long and complex (with more than 1k words),
Test Training Model
while traditional BERT-like models only have Overall
a context length of 512 tokens, urging us to Dataset Dataset LLaMA-2-13B Mistral-7B
train a large language model for classifica
GSM8K 46.77 61.25 54.01
tion. Following Li et al. (2023a), we convert
GSM8K + Explanation 48.60 53.37 50.99
the classification task into a generation task, + REALIGN **56.63** **68.16** **62.40**
which directly generates the task name given a
question with the prompt as shown in Tab. 13. MATH 6.14 13.18 9.66
Specifically, we manually label about 33 ques- MATH + Explanation **7.30** 13.94 10.62
tions for each kind of task from LIMA (Zhou + REALIGN 7.14 **15.30** **11.22**
et al., 2023), ShareGPT (Zheng et al., 2023),
and Alpaca (Taori et al., 2023) datasets. For Table 7: The results of math reasoning on GSM8K, MATH and
tasks with less training data, we use ChatGPT them + explanation or REALIGN. We report the accuracy by
to generate a portion of the questions. Then, exact matching. **Bold** indicates the best result.
we divide questions in a 9:1 train/test split .
(Tab. 12). We train the task classifier from
LLaMA-2-13B (Touvron et al., 2023), and set the max sequence length as 2,048, epochs as 20, and batch size as 16.
We set the initial learning rate to 2e-5 and consine decaying to 0 by the end of training with warmup steps as 10.
The optimizer is AdamW with _β_ 1 = 0 _._ 9 _, β_ 2 = 0 _._ 95. The loss is only calculated on the output end as well. The
accuracy and F1 of the final task classifier on the test set are 78.32% and 81.59%, respectively.
Test Training Model
Overall
Dataset Dataset LLaMA-2-13B Mistral-7B
GSM8K
MATH
GSM8K 46.77 61.25 54.01
+ Explanation 48.60 53.37 50.99
+ REALIGN **56.63** **68.16** **62.40**
MATH 6.14 13.18 9.66
+ Explanation **7.30** 13.94 10.62
+ REALIGN 7.14 **15.30** **11.22**
Table 7: The results of math reasoning on GSM8K, MATH and
them + explanation or REALIGN. We report the accuracy by
exact matching. **Bold** indicates the best result.
.


Auto-J (Li et al., 2023a) is an open-source generative judge designed to evaluate LLMs based on their alignment
with human preferences, which is the best critique model besides GPT-4 (Sun et al., 2024). Auto-J stands out due to
its generality, being trained on real-world user queries and responses from various LLMs across 58 scenarios. It
offers flexibility by enabling both pairwise comparison and single-response evaluation through prompt adjustments.
Additionally, Auto-J enhances reliability and encourages human participation in the evaluation process by offering
detailed natural language critiques, improving interpretability.


By carefully observing the cases, we speculate that the reason for the improvements in math reasoning may stem
from the easier-to-understand format, more detailed explanations (Mukherjee et al., 2023), or length (Jin et al.,
Reformatted Alignment
2024). To further explore the reasons, we merely incorporate detailed step-by-step explanations without including a
complete format. These explanations are generated using gpt-3.5-turbo-1106, with the prompts used detailed in
Tab. 21 and Tab. 22. The results are shown in Tab. 7.
Moreover, to explore the impact of various formats, we experiment with two other formats based on REALIGN:
(1) The first requires separating natural language and calculation, meaning that the natural language does not include
the calculation process and mathematical computations are expressed separately; (2) The second variant requires the
use of special markers ‘ _<<>>_ ’ in equations based on the format of REALIGN, for example, _<<_ 1 + 2 = 3 _>>_ .
We merely create these variants on GSM8K due to the cost of ChatGPT API. The results are shown in Tab. 8
From the results of the above experiments
(see Tab. 7 and Tab. 8), we can derive insights
below:

E Why Does REALIGN Boost Math? (Complete Version)


**step-by-step explanations.** As shown in
Tab. 7, we can find that merely providing a
detailed explanation is insufficient without
a well-structured format and may even result in performance inferior to the original
dataset (i.e., The results on GSM8K based on
Mistral-7B). For the more complex MATH
dataset, detailed explanations still play a significant role. However, A well-organized
structure may further enhance their effectiveness.
Model
Dataset Overall
LLaMA-2-13B Mistral-7B
GSM8K 46.77 61.25 54.01
+ REALIGN (Separate.) 55.57 62.09 58.83
+ REALIGN (‘ _<<>>_ ’) **57.01** 63.61 60.31
+ REALIGN 56.63 **68.16** **62.40**
Table 8: The results of math reasoning on GSM8K and them +
different formats of REALIGN. We report the accuracy by exact
matching. “Separate.” denotes the first variant that separates natural language and calculation. “ _<<>>_ ” denotes the second variant
that requires the use of special markers ‘ _<<>>_ ’ in equations
based on the format of REALIGN. **Bold** indicates the best result.
.
**Insights 2: Length is not all you need.** Jin et al. (2024) suggests that longer reasoning steps can enhance math
reasoning capabilities. To further analyze the impact of the length of reasoning steps, we hypothesize that the longer
the response, the more extensive the reasoning steps involved. Specifically, we calculate the average length that
is shown in Tab. 9. We can see that the average length of “GSM8K + Explanation” exceeds that of “GSM8K” by
more than double, and is even longer than “GSM8K + ReAlign”. However, its performance is significantly inferior
to “GSM8K + ReAlign”. Additionally, the average length of ”MATH + Explanation” is shorter than ”MATH +
ReAlign”, yet it demonstrates superior performance on LLaMA-2-13B. These findings suggest that length is not the
determining factor; rather, a well-organized format can lead to more substantial gains.


**formats.** As shown in Tab. 8, we can see that, firstly, expressing natural
language and mathematical calculations together (REALIGN) performs
better. This approach is more in line with human habits and preferences,
making it easier for users to understand. Secondly, adding special markers
“ _<<>>_ ” to equations can diminish performance, potentially because this
format does not align with the customary practices of human writing and
usage, making it less intuitive to understand. Therefore, the most important
principle in designing formats is human value. A good format is not only
about superior performance but also about user preference (i.e., formats that
are easier for users to understand). Following this principle in designing
criteria, we observed a significant improvement in our math reasoning
abilities, indicating that formats easier for users to understand can also lead
to better performance. Therefore, we also advocate that the development
of large language models should move closer to user values.

Insights 3: Human value is the most important principle in designing


Dataset Response Len.
GSM8K 130.59
+ Explanation 341.61
+ REALIGN 327.65
MATH 243.73
+ Explanation 293.57
+ REALIGN 375.35
Table 9: The response length of original datasets and them + Explanation
and REALIGN. Response Len. is
the average number of tokens of the
responses.
The reformatting prompts for non-knowledge-intensive tasks and knowledge-intensive tasks are shown in Tab. 14
and Tab. 15. The complete version of Fig. 2 and Tab. 2 are shown in Tab. 16 and Tab. 17, respectively. The example
of a knowledge-intensive task is shown in Tab. 18. The evaluation prompts for readability and factuality are shown
in Tab. 19 and Tab. 20, respectively. The prompts for generating explanations for GSM8K and MATH datasets are
shown in Tab. 21 and Tab. 22.
Reformatted Alignment
**Query** : Benjamin went to a bookstore and purchased a variety of books. He bought 5 copies of a sci-fi novel, each priced at $20, 3 copies of a history book
priced at $30 each, and 2 copies of a philosophy book for $45 each. What was the total cost of his purchases?
**Response from Original Model** : To find the total cost of Benjamin’s purchases, I need to add up the prices of all the books he bought. I can use the
information given in the problem to set up an equation:
So, the total cost of Benjamin’s purchases was $280.
**Response from REALIGN Model** : To find the total cost of Benjamin’s purchases, we need to add up the prices of the books he bought.
Detailed explanation:
The total cost of Benjamin’s purchases is the sum of the costs of the sci-fi novels, the history books, and the philosophy books: $100 + $90 + $90 = $280.
Correct result and conclusion:
The total cost of Benjamin’s purchases was $280.
Table 10: An example of the response from a model trained on the original Open-Platypus dataset (denoted as
Original Model) and the response from a model trained on the REALIGN Open-Platypus dataset (denoted as
REALIGN Model). We highlight the content in the better format given by the REALIGN Model in green compared
with the content given by the Original Model in red.
Reformatted Alignment


_Generation_
question generation Write some questions based on the given description.      -      story generation Write a story based on the given description.       -       poem generation Write a poem based on the given description.      -      email generation Write an email based on the given description.       -       data generation Generate data based on the given description.       -       text-to-text translation Translate the given text into another language.      -
_Brainstorming_
advice giving Respond well to users when they seek advice.        -        recommendations Give recommendations to users.      -      how-to generation Give relevant and complete answer when users ask ‘how to do‘ something.      -      planning Write a plan for an event or activity.        -
_Code_
code correction Correct the potential errors in a piece of code.       -       code simplification Rewrite a piece of code to make it more concise and easy to understand.      -      explain code Write an explanation for a piece of code.        -        text-to-code translation Write a piece of code based on the given description.      -      code-to-code translation Convert the given code into another programming language.     -     language learning questions Write an answer for the given question about programming language learning.    -    code language classification Classify the programming language for the given code.    -    code-to-text-translation Write a document for the given code.     -
_Rewriting_
instructional rewriting Rewrite a given text with a specific instruction.      -      language polishing Polish a piece of text to make it more fluent, natural, and readable.      -      paraphrasing Paraphrase a given text.       -       text correction Correct the potential errors in a piece of text.        -
_Extraction_
information extraction Extract one or multiple user-specified categories of information from a piece of text attached in the user’s query.     -     keywords extraction Extract the keywords from a piece of text.      -      table extraction Generate a table include the key information from a piece of text attached in the user’s query.        -
_Summarization_
title generation Generate a title for the given text or based on a description of the work.        -        text summarization Write a summary for a piece of text.      -      note summarization Write a note to summarize a piece of text.      -
_Conversation_
open qa The user’s query is an open domain question with no attached passage or article.        -        closed qa Answer the questions that can be directly answered by the attached passage.        -        fact verification Verify if the given fact is true or false.        -        value judgment Provide a value judgment on a given topic or statement.       -       roleplay Pretend to be a specific person, character, profession or identity, and complete the required task on this basis.         -         explain answer Explain something the user wants to know.       -
_Specialized Educational Dialog_
natural language tutor Write an answer for the given question about natural language learning.      -      exam problem tutor Solve an exam question (like fill-in-the-blank, multiple choice, problem solving, etc) with no math involved.      -      ai tutor Write an answer for the given question about machine learning, artificial intelligence or language model.          -          math puzzles Write an answer with the step-by-step reasoning process for a math question.       -       fill in the blank Complete the missing parts with the most appropriate words to make the text coherent and meaningful.        -
_Classification_
general classification Classify one or multiple objects given by the user into the specified categories.      -      ordering Sort some things, according to some criteria.         -         sentiment analysis Identify and categorize the subjective opinions, attitudes, and feelings of the writer towards a particular subject.      -      language classification Classify the language for the given text.      -      topic classification Extract the high-level topics or themes from a given text, i.e., what kind of topics are discussed in the text.       -
_Others_
rejecting Reject to respond when the query is beyond capacity or it violates general ethical and legal rules.         -         others You must choose this if none of the other scenarios match the user’s query well.         -
Table 11: Detailed description for each task.
Reformatted Alignment
task train test task train test task train test
question ~~g~~ eneration 30 2 code ~~l~~ anguage ~~c~~ lassification 30 2 roleplay 30 3
story generation 30 4 code ~~t~~         - ~~t~~ ext ~~t~~ ranslation 30 3 explain ~~a~~ nswer 30 4
poem ~~g~~ eneration 30 3 instructional ~~r~~ ewriting 30 4 natural ~~l~~ anguage ~~l~~ earning ~~t~~ utor 30 2
email ~~g~~ eneration 30 3 language ~~p~~ olishing 30 2 exam ~~p~~ roblem ~~s~~ olving ~~t~~ utor 31 2
data generation 30 3 paraphrasing 30 2 ml ~~a~~ i ~~l~~ anguage ~~m~~ odel ~~t~~ utor 30 3
text ~~t~~         - text translation 30 3 text correction 30 2 math ~~p~~ uzzles 30 6
advice giving 30 4 information ~~e~~ xtraction 30 3 fill ~~i~~ n the ~~b~~ lank 30 3
recommendations 30 2 keywords ~~e~~ xtraction 30 2 general ~~c~~ lassification 30 4
how ~~t~~      - generation 30 3 table ~~e~~ xtraction 30 3 ordering 30 3
planning 30 2 title ~~g~~ eneration 30 2 sentiment ~~a~~ nalysis 30 3
code ~~c~~ orrection 30 5 text summarization 30 5 language ~~c~~ lassification 30 3
code ~~s~~ implification 30 2 note summarization 30 2 topic ~~c~~ lassification 30 2
explain code 30 2 open ~~q~~ a 30 6 rejecting 30 3
text ~~t~~         - code translation 30 4 closed ~~q~~ a 30 2 others 43 8
code ~~t~~       - code translation 30 3 fact verification 30 2 overall 1395 143
language ~~l~~ earning ~~q~~ uestions 31 5 value judgement 30 2
Table 12: The task distribution in the training and test set for task classifier.
Reformatted Alignment

Task name Description Retrieval Rewriting


You will receive a user’s query. Additionally, you are given some pre-defined tasks below:
[Existing tasks start]
question ~~g~~ eneration
story ~~g~~ eneration
poem ~~g~~ eneration
email ~~g~~ eneration
data ~~g~~ eneration
advice ~~g~~ iving
recommendations
how ~~t~~ - ~~g~~ eneration
planning
instructional ~~r~~ ewriting
language ~~p~~ olishing
paraphrasing
text ~~c~~ orrection
code ~~c~~ orrection
code ~~s~~ implification
information ~~e~~ xtraction
keywords ~~e~~ xtraction
table extraction
title ~~g~~ eneration
text ~~s~~ ummarization
note summarization
explain ~~c~~ ode
explain ~~a~~ nswer
text ~~t~~ - ~~t~~ ext translation
text ~~t~~ - ~~c~~ ode ~~t~~ ranslation
code ~~t~~ - ~~c~~ ode translation
code ~~t~~ - ~~t~~ ext ~~t~~ ranslation
open ~~q~~ a
closed qa
fill ~~i~~ n the ~~b~~ lank
fact verification
math ~~p~~ uzzles
language ~~l~~ earning ~~q~~ uestions
natural ~~l~~ anguage ~~l~~ earning tutor
exam ~~p~~ roblem solving ~~t~~ utor
ml ~~a~~ i language model tutor
general classification
ordering
sentiment analysis
code ~~l~~ anguage ~~c~~ lassification
language ~~c~~ lassification
topic ~~c~~ lassification
value ~~j~~ udgement
rejecting
roleplay
default
[Existing tasks end]
Your objective is to choose the most appropriate task that can reflect the high-level intention of this query. You should first clearly give out your choice. Your choice
should exactly match one of the task names provided above, without any modification. Do not include the task description in your choice.
Your output should be just the task name.
User’s query is below:
[User’s query start]
_{_ input _}_
[User’s query end]
Task name:
Table 13: The classification prompt for the task classifier in the training and inference phase.
Reformatted Alignment


Please act as a rewriter to modify the format of the AI assistant’s response to the user’s question presented below.
Please follow the instructions below:
1. Please first determine whether the given format meets the requirements of the user’s question, if it does not, then copy the AI assistant’s response, if it does, then
modify the response’s format following the provided format.
2. Your task is limited to altering the format while keeping the original meaning and information intact.
3. Please make sure that the revised response can answer the user’s question correctly.
4. Please make sure that the revised response is fluent and has no additional subheadings.
Please first write ”Reasoning: _<_ reason _>_ ” to provide a brief reasoning you used to modify, and then write ”Revised response: _<_ response _>_ ” to output your final
revised response without any additional information, ensuring its fluency.
Do not output any additional subheadings.


Below is a user’s question, the AI assistant’s response, and the provided format.
[Question start]
_{_ question _}_
[Question end]
[Response start]
_{_ response _}_
[Response end]
[Format start]
_{_ format _}_
[Format end]
Table 14: Reformatting prompts for non-knowledge-intensive tasks.


Please act as a rewriter to modify the format of the AI assistant’s response to the user’s question presented below.
Please follow the instructions below:
1. Please first determine whether the given format meets the requirements of the user’s question, if it does not, then copy the AI assistant’s response, if it does, then
modify the response’s format following the provided format.
2. Your task is limited to altering the format while keeping the original meaning and information intact.
3. Please make sure that the revised response can answer the user’s question correctly.
4. Please make sure that the revised response is fluent and has no additional subheadings.
5. Evidence is the useful information. You should decide for yourself which parts of the evidence to help rewriting the response.
Please first write ”Reasoning: _<_ reason _>_ ” to provide a brief reasoning you used to modify, and then write ”Revised response: _<_ response _>_ ” to output your final
revised response without any additional information, ensuring its fluency.
Do not output any additional subheadings.


Below is a user’s question, the AI assistant’s response, the provided format, and the evidences.
[Question start]
_{_ question _}_
[Question end]
[Response start]
_{_ response _}_
[Response end]
[Format start]
_{_ format _}_
[Format end]
[Evidence start]
_{_ evidence _}_
[Evidence end]
Table 15: Reformatting prompts for knowledge-intensive tasks.
Reformatted Alignment


How many positive three-digit integers less than 500 have at least two digits that are the same?


Task Name: math puzzles
Format: This is a math question. First, analyze the question and give a brief analysis in the first paragraph. Then, use a list to present the step-by-step solution. Next,
give another list to output a detailed explanation. Last, give the correct result and a conclusion.


To solve this problem, I will count how many three-digit numbers less than 500 have at least two digits that are the same, and subtract that from the total number of
three-digit numbers less than 500. The total number of three-digit numbers less than 500 is 4 _×_ 10 _×_ 10 = 400, since the first digit can be 1, 2, 3, or 4, and the
second and third digits can be anything from 0 to 9. Now I will count how many of these 400 numbers have no digits that are the same. To have no digits that are the
same, the first digit can be any of the 4 choices, but the second digit has to be different from the first, so it has 9 choices. The third digit has to be different from both
the first and the second, so it has 8 choices.
[* on another note] I am assuming that the second digit can be 0, since the problem does not say otherwise. Therefore, the number of three-digit numbers less than
500 that have no digits that are the same is 4 _×_ 9 _×_ 8 = 288. This means that the number of three-digit numbers less than 500 that have at least two digits that are
the same is 400 _−_ 288 = 112.


Analysis:
To solve the problem, we need to count how many three-digit numbers less than 500 have at least two digits that are the same.
Solution:
1. Calculate the total number of three-digit numbers less than 500: 4 _×_ 10 _×_ 10 = 400.
2. Count the number of three-digit numbers less than 500 that have no digits that are the same: 4 _×_ 9 _×_ 8 = 288.
3. Subtract the count of numbers with no same digits from the total to find the number of three-digit numbers less than 500 that have at least two digits that are the
same: 400 _−_ 288 = 112.
Explanation:
- The first digit can be 1, 2, 3, or 4, and the second and third digits can be anything from 0 to 9.
- To have no digits that are the same, the first digit can be any of the 4 choices, but the second digit has to be different from the first, so it has 9 choices. The third digit
has to be different from both the first and the second, so it has 8 choices.
Result:
There are 112 three-digit numbers less than 500 that have at least two digits that are the same.
Conclusion:
There are 112 positive three-digit integers less than 500 that have at least two digits that are the same.
Table 16: A complete example of a query, including the original response and the REALIGN response.
It is an email-writing task. Here is a general guideline for creating a well-structured and professional email:
1. Subject Line: Write a clear and concise subject line that accurately summarizes the content of your email. This helps the recipient understand the purpose of the
email at a glance.
2. Salutation: Begin your email with a formal salutation such as ”Dear [Recipient’s Name],” or use a more casual salutation if you have an informal relationship
with the recipient.
3. Introduction: Start your email with a brief introduction, stating who you are and the reason for writing the email. Be clear and to the point, and avoid unnecessary
details.
4. Body: This is the main content of your email. Organize your thoughts into paragraphs or bullet points to make them easier to read. Keep your sentences concise
and focused. Use proper grammar, punctuation, and spelling to maintain professionalism. If you need to discuss multiple topics, consider using headings or numbered
points to separate them.
5. Politeness and Tone: Maintain a polite and respectful tone throughout your email. Be mindful of the recipient’s perspective and use appropriate language. Avoid
using excessive capitalization, exclamation marks, or emoticons, as they can come across as unprofessional.
6. Closing: Conclude your email with a closing remark, such as ”Thank you,” or ”Best regards,” followed by your name. If you expect a response or need specific
action, you can mention it in this section as well.
7. Signature: Include your full name, job title, and contact information (e.g., phone number, email address) in your email signature. This helps the recipient easily
identify and contact you if needed.
8. Attachments: If you need to include attachments, mention them in the email body and make sure they are relevant to the email’s purpose. Consider compressing
large files or using cloud storage services if the attachments are too large to be sent via email.
9. Proofread: Before sending the email, proofread it for any grammatical or spelling errors. Make sure the email conveys your message clearly and effectively.
The best emails are short, direct, professional, and scannable for the recipient. Follow formal business email structure unless you have an established casual rapport
with the recipient.
Table 17: A complete format for “email generation” task.
Reformatted Alignment


Is the given statement true or false?
A robotic vacuum can clean both carpets and hard floors.


Task Name: fact verification
Format: This is a fact-verification task. First, give the answer. Then, give an explanation.


1. Robot vacuums run fully autonomously and automatically. They self-navigate on any flooring type, from hardwoods to carpets, and some models can even be
programmed to have a cleaning schedule and to sync to smartphones and smart home devices.
2. iRobot roomba vacuum works on laminate flooring. In fact they work better on hard floor as compared to carpet since their suction and beater bar is not ...
3. Our top pick among the best vacuums for hardwood floors is the iRobot Roomba 694. iRobot is known as a top choice when it comes to robotic ...
4. Because they lack the power to deep clean, though, robot vacuums are best for flooring other than carpet, such as tile, laminate or hardwood, ...


True.
Table 18: An example of a knowledge-intensive task, including the query, the criteria, the retrieved evidence, the
original response, and the REALIGN response. We highlight the content that is more factual and knowledge in
green.


Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below. You should choose the
assistant that follows the user’s instructions and answers the user’s question better. Your evaluation should consider the readability including clarity, format, and ease
of understanding. Well-organized, grammatically correct response is better. Begin your evaluation by comparing the two responses and provide a short explanation.
Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses
to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible. After providing your explanation, output your final verdict by
strictly following this format: “[[A]]” if assistant A is better, “[[B]]” if assistant B is better, and “[[C]]” for a tie.


[User Question]
_{_ question _}_
[The Start of Assistant A’s Answer]
_{_ answer ~~a~~ _}_
[The End of Assistant A’s Answer]
[The Start of Assistant B’s Answer]
_{_ answer ~~b~~ _}_
[The End of Assistant B’s Answer]
Table 19: Evaluation prompt for readability.


Please act as an impartial judge and evaluate the factuality of the response provided by an AI assistant to the user question displayed below. Your evaluation should
consider correctness. You will be given a reference answer and the assistant’s answer. Begin your evaluation by comparing the assistant’s answer with the reference
answer. Identify and correct any mistakes. Be as objective as possible. After providing your explanation, you must rate the response on a scale of 1 to 10 by strictly
following this format: “[[rating]]”, for example: “Rating: [[5]]”.


[Question]
_{_ question _}_
[The Start of Reference Answer]
_{_ ref ~~a~~ nswer _}_
[The End of Reference Answer]
[The Start of Assistant’s Answer]
_{_ answer _}_
[The End of Assistant’s Answer]
Table 20: Evaluation prompt for factuality.
Reformatted Alignment


Please act as a mathematics explanation generator to generate a step-by-step explanation of the answer based on the question presented below.
Please follow the instructions below:
1. Please simply generate a step-by-step explanation, including the reason for each step of the calculation.
2. Please do not change the essence of the answer.
Please write ”The explanation: _<_ answer _>_ ” to output your explanation without any additional information.
Here is an example for your reference:
Question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?
Answer: Natalia sold 48/2 = _<<_ 48/2=24 _>>_ 24 clips in May. _\_ nNatalia sold 48+24 = _<<_ 48+24=72 _>>_ 72 clips altogether in April and May. _\_ n#### 72
Explanation:
1. The problem states that Natalia sold clips to 48 friends in April. In May, she sold half as many clips as she did in April. We are asked to find out the total number
of clips she sold over these two months.
2. To find out how many clips Natalia sold in May, we take half of the number of clips sold in April. Since she sold clips to 48 friends in April, we calculate half of
48: _\_ n$$ _\_ frac _{_ 48 _}{_ 2 _}_ = 24$$ _\_ nThis step involves simple division, where we divide the number of clips sold in April by 2 to find the number of clips sold in May.
3. Now that we have the number of clips sold in May, we need to add this to the number of clips sold in April to get the total number of clips sold over both
months: _\_ n$$48 + 24 = 72$$ _\_ nThis step involves addition, where we sum the clips sold in April and May to find the total sales for the two months combined.
4. Natalia sold 72 clips in total during April and May. This final total gives us the answer to the question asked.


Below is a question and the answer:
[Question start]
_{_ question _}_
[Question end]
[Answer start]
_{_ answer _}_
[Answer end]
Table 21: Prompts for generating explanations for GSM8K.


Please act as a mathematics explanation generator to generate a step-by-step explanation of the answer based on the question presented below.
Please follow the instructions below:
1. Please simply generate a step-by-step explanation, including the reason for each step of the calculation.
2. Please do not change the essence of the answer.
Please write ”The explanation: _<_ answer _>_ ” to output your explanation without any additional information.
Here is an example for your reference:
Question: Each row of a seating arrangement seats 7 or 8 people. Forty-six people are to be seated. How many rows seat exactly 8 people if every seat is occupied?
Answer: Let $x$ be the number of rows with 8 people. If we removed a person from each of these rows, then every row would contain 7 people. Therefore, $46 - x$
must be divisible by 7. _\_ n _\_ nThen $x _\_ equiv 46 _\_ equiv 4 _\_ pmod _{_ 7 _}_ $. The first few positive integers that satisfy this congruence are 4, 11, 18, and so on. However,
each row contains at least 7 people. If there were 7 or more rows, then there would be at least $7 _\_ cdot 7 = 49$ people. We only have 46 people, so there must be at
most six rows. Therefore, the number of rows with 8 people is $ _\_ boxed _{_ 4 _}_ $. _\_ n#### 4
Explanation:
We have a seating arrangement where each row seats either 7 or 8 people. A total of 46 people need to be seated, and all seats must be filled. The question asks how
many rows seat exactly 8 people.
1. We let $x$ represent the number of rows that seat exactly 8 people.
2. To simplify the problem, imagine removing one person from each row that currently seats 8 people. This would convert every 8-person row into a 7-person row.
Now, all rows (both the original 7-person rows and the adjusted 8-person rows) would seat 7 people.
3. With this adjustment, the total number of people would be reduced by $x$ (since we removed one person from each 8-person row), making it $46 - x$.
4. Since all rows now hypothetically seat 7 people, the adjusted total, $46 - x$, must be divisible by 7 for it to be a possible total number of people seated in rows of 7.
5. We analyze $46 - x$ in terms of modulo 7. Specifically, we want to find values of $x$ such that $46 - x$ is a multiple of 7. This simplifies to finding $x$ such that
$x _\_ equiv 46 _\_ pmod _{_ 7 _}_ $. Calculating $46 _\_ mod 7$ yields 4, because when 46 is divided by 7, the remainder is 4. This tells us $x$ must be equivalent to 4 modulo
7.
6. The numbers that satisfy $x _\_ equiv 4 _\_ pmod _{_ 7 _}_ $ are 4, 11, 18, etc. However, we need a practical value of $x$ that fits the total people and the row seating
constraint.
7. If there were 7 or more rows of 7 people each, we’d have at least 49 people (since $7 _\_ times 7 = 49$). But we only have 46 people, so there must be fewer than 7
rows in total. The plausible values of $x$ from the list 4, 11, 18, etc., must be reconsidered within this context.
8. Since the only value from our possible $x$ values (4, 11, 18) that is less than 7 and fits the total people count is 4, we conclude that there are 4 rows of 8 people
each.


Below is a question and the answer:
[Question start]
_{_ question _}_
[Question end]
[Answer start]
_{_ answer _}_
[Answer end]
Table 22: Prompts for generating explanations for MATH.


Stanford University Georgia Institute of Technology
michaeljryan@stanford.edu wheld3@gatech.edu

Unintended Impacts of LLM Alignment on Global Representation


Stanford University
diyiy@cs.stanford.edu
Reward
Higher
Lower
Before being deployed for user-facing applications, developers align Large Language Models
(LLMs) to user preferences through a variety
of procedures, such as Reinforcement Learning From Human Feedback (RLHF) and Direct Preference Optimization (DPO). Current
evaluations of these procedures focus on benchmarks of instruction following, reasoning, and
truthfulness. However, human preferences are
not universal, and aligning to specific preference sets may have unintended effects. We
explore how alignment impacts performance
along three axes of global representation: English dialects, multilingualism, and opinions
from and about countries worldwide. Our results show that current alignment procedures
create disparities between English dialects and
global opinions. We find alignment improves
capabilities in several languages. We conclude
by discussing design decisions that led to these
unintended impacts and recommendations for
more equitable preference tuning. We make our
code and data publicly available on Github [1] .


Recently, LLM-powered chat assistants (OpenAI,
2023a; Touvron et al., 2023; Tunstall et al., 2023b)
have exploded in popularity. As of December 2023,
ChatGPT has amassed over 100M weekly users
(OpenAI, 2023b) and Llama-2-Chat-7B is downloaded almost one million times a month from HuggingFace [2] . The success of these chat models is dependent on "alignment", which takes a base model
with a language modeling objective and produces
an instruction following preference-guided model
to better serve user interests. Practitioners use algorithms such as RLHF (Ouyang et al., 2022) and
DPO (Rafailov et al., 2023) to optimize models
for attributes such as helpfulness and harmlessness
[1https://github.com/SALT-NLP/](https://github.com/SALT-NLP/unintended-impacts-of-alignment)
[unintended-impacts-of-alignment](https://github.com/SALT-NLP/unintended-impacts-of-alignment)
[2Llama-2-Chat-7B Huggingface Page](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf)
Figure 1: Country rewards for Starling 7B Reward
Model prompted with "User: Where are you from?
Assistant: I am from {country}." Starling assigns
higher rewards to English-speaking Western nations and
lower rewards to countries in the Middle East/Africa.
and give them their chat assistant persona (Ouyang
et al., 2022; Bai et al., 2022; Zhu et al., 2023).
Unlike the nebulous pre-training process, which
is largely defined by the distribution of data online
(Raffel et al., 2019; Gao et al., 2020; Computer,
2023), model developers have a high degree of
control for the key alignment variables. Who will
give feedback? What prompts/tasks are in-domain?
Who will provide exemplar responses? These are
just a few design decisions that underscore a larger
question: _Whose preferences are we aligning LLMs_
_with, and crucially, whose preferences are we miss-_
_ing?_ As Blodgett et al. (2020) put it, "For which
speakers are NLP systems developed?"
This question often does not have an explicit answer in current alignment practices (Bakker et al.,
2022), making it unclear which model behaviors
are intentional normative judgments and which
are unintended biases. For example, the Starling
7B Reward Model (Zhu et al., 2023) gives higher
scores to responses claiming to be from Englishspeaking Western nations and lower scores for Middle Eastern and African nations (See Figure 1). In
this work, we take a closer look at the effects these
design decisions have on a model’s ability to serve
a global population, which is key to understanding
if the general use of aligned LLMs (Eloundou et al.,
2023) is likely to be positively adopted globally.
Existing performance evaluations of chat assistants mainly focus on tasks such as reasoning
(Clark et al., 2018; Zellers et al., 2019; Sakaguchi
et al., 2021; Cobbe et al., 2021), multitask knowledge (Hendrycks et al., 2021; Suzgun et al., 2023),
truthfulness (Lin et al., 2022), multi-turn instruction following (Zheng et al., 2023), and similar variations of broad knowledge/reasoning/skills (Chen
et al., 2021; Zhong et al., 2023). Instead, we explore a set of representative domains covering variations common in diverse global user bases: English
dialects, multilingualism, and global opinions, and
show a direct impact on model performance.
Our evaluations focus on measuring how alignment makes LLMs more agreeable and helpful for
different groups of possible global users. While
prior works (Section2) have explored the representation
of global opinions in language models (Durmus
et al., 2023; Santurkar et al., 2023), they only study
the final model. However, the process of transforming a base language model to a user-facing chat
model involves two key sequential steps: **super-**