School of Computer Science and Engineering
Southeast University
Nanjing, China
{jmingyang,xning,liubiao01,sqqiao,xgeng}@seu.edu.cn
ABSTRACT
Preference optimization is crucial for aligning large language models (LLMs)
with human values and intentions. A significant challenge in this process is
the distribution mismatch between pre-collected offline preference data and the
evolving model policy. Existing methods attempt to reduce this gap using static
heuristics or decoupled online sampling strategies, but they often fail to adapt to
the model’s dynamic learning state. To bridge this gap, we propose Meta-Weighted
Adaptive Preference Optimization (MetaAPO), a novel framework that dynamically
couples data generation with model training. MetaAPO employs a lightweight
meta-learner, as an “alignment gap estimator”, to evaluate the potential benefits of
on-policy sampling in relation to offline data. This guides targeted online generation
and assigns sample-wise meta-weights to the optimization objective, dynamically
balancing the quality and distribution of online and offline data. Experiments on
AlpacaEval 2, Arena-Hard and MT-Bench demonstrate that MetaAPO consistently
outperforms existing preference optimization approaches across various settings,
while reducing 42% in online annotation costs.
1 INTRODUCTION
Learning from human feedback is essential for aligning large language models (LLMs) with human
values and intentions (Ouyang et al., 2022), ensuring they are helpful, honest and harmless (Achiam
et al., 2023; Grattafiori et al., 2024). This advancement is primarily achieved in the model alignment
stage, which involves training reward models or LLMs directly on datasets curated from human
preferences. Reinforcement learning from human feedback (RLHF) (Stiennon et al., 2020; Ouyang
et al., 2022) is a widely used method for fine-tuning LLMs to achieve this goal. However, RLHF often
faces challenges, particularly in computational efficiency and training stability. Recently, alternative
offline algorithms like DPO (Rafailov et al., 2023), SimPO (Meng et al., 2024) and KTO (Ethayarajh
et al., 2024) have been explored. These methods reparameterize the RLHF objective to directly
optimize the policy using pre-collected offline preference datasets, simplifying the training process
and improving efficiency.
As alignment research progresses, recent studies have emphasized that the quality and distribution of
preference data are critical for effective model alignment (Zhou et al., 2023; Deng et al., 2025; Xia
et al., 2024; Ding et al., 2024). Although offline preference datasets are diverse and scalable, they
often suffer from a distribution mismatch between the pre-collected training data and current policy,
as the offline data is usually generated by a different model. This mismatch leads to out-of-distribution
(OOD) issues that hinder preference alignment performance (Yan et al., 2024; Xu et al., 2024; Tajwar
et al., 2024). To improve the quality and distribution of training data, existing methods for mitigating
distribution shifts generally fall into two categories. Some methods improve the reliability of offline
data by filtering out low-quality or OOD samples using manually designed metrics, such as average
validation loss, as exemplified by Selective DPO (Gao et al., 2025). Others attempt to mitigate
distribution shifts by constructing training data through online sampling (Zhang et al., 2024b; Ko
et al., 2024; Badrinath et al., 2024). For instance, Iterative DPO (Xiong et al., 2023; Guo et al.,
2024b) and SPPO (Wu et al., 2024) follow an iterative sampling-alignment loop: the current policy
Figure 1: **Left:** Overview of MetaAPO. MetaAPO employs a meta-learner to couple online data
generation (top) and model training (bottom). The meta-learner adaptively assigns weights by
evaluating offline data, guiding both targeted online sample generation and training on the weighted
combination of offline and online samples. **Right:** Performance (left y-axis, line plots) and online
generation and annotation ratio relative to Online DPO (right y-axis, bar plots) across training
iterations for different methods.
first generates responses, which are then annotated by a reward model to collect preference data for
direct preference alignment. The updated policy is subsequently used to repeat this process.
While human-annotated offline data typically offers high efficiency and quality, the distribution
mismatch between static datasets and the evolving model policy can negatively impact alignment
performance. On the other hand, model-generated data obtained through online sampling more
accurately reflects the current policy distribution, which helps address the distribution shifts. However,
online data often lacks diversity and quality due to its reliance on the model’s capabilities and current
alignment state, leading to preference data that may be inefficient or even noisy. This trade-off
underscores the need for adaptive mechanisms that can dynamically leverage the efficiency and
diversity of offline data while taking advantage of the distribution benefits of online data. However,
existing methods primarily depend on heuristic, manually designed metrics ( _e.g.,_ static thresholds)
for data selection and generation (Dong et al., 2024; Gao et al., 2025; Zhang et al., 2024b; Ji et al.,
2024), overlooking the interaction between the data sampling process and preference optimization.
An integrated approach that simultaneously accounts for data quality, distribution alignment, and the
dynamics of preference optimization is essential for more effective alignment.
Motivated by the above considerations, we propose Meta-Weighted Adaptive Preference Optimization
(MetaAPO), a novel framework that closely couples data sampling with model alignment through
a lightweight, learnable meta-learner. As demonstrated in Figure 1 (Left), the meta-learner acts as
an “alignment gap estimator”, predicting the potential improvements that online generations can
provide relative to offline data. This enables targeted online sampling, prioritizing prompts expected
to yield the highest alignment gains. In addition, the meta-learner adjusts the training process by
assigning sample-wise meta-weights to a hybrid loss function, adaptively balancing the contributions
of offline data and online samples according to their respective quality and distribution advantages.
This dynamic interaction ensures that the model continually adapts its learning focus to mitigate
distribution mismatches while avoiding redundant online sampling for already aligned data, thus
improving both training efficiency and performance. Empirical results demonstrate that MetaAPO
significantly improves alignment across various base models, outperforming both offline and online
approaches. Our contributions can be summarized as follows:
 - We propose MetaAPO, an extensible alignment framework compatible with various preference
optimization methods. It bridges the gap between online data generation and model training by
adaptively coupling these two stages, enabling a model-aware alignment process.
 - Our method employs a meta-learner that adaptively assigns sample-wise weights to data. These
weights serve two purposes: guiding online data sampling based on the model’s training needs to
mitigate distribution mismatches and adjusting the training by emphasizing beneficial samples.
 - Extensive experiments demonstrate that MetaAPO achieves superior alignment performance
compared to baselines, even reducing 42% online annotation requirements (Figure 1, Right).
Preprint
2 RELATED WORK
**Alignment with Preference Data.** RLHF is a widely used method for aligning LLMs with human
values and intentions (Stiennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022). While effective,
RLHF suffers from high computational costs and training instability, prompting the exploration
of alternative alignment strategies. One prominent direction is offline preference alignment (Azar
et al., 2024; Dong et al., 2023; Yuan et al., 2023; Lu et al., 2024), which directly trains models using
pre-collected datasets of human preferences, such as DPO (Rafailov et al., 2023), SimPO (Meng
et al., 2024) and KTO (Ethayarajh et al., 2024). These methods are attractive for their efficiency and
simplicity. However, a key limitation is the distribution mismatch between static training data and the
evolving outputs of the policy model, which often leads to poor generalization on out-of-distribution
responses (Xu et al., 2024; Yan et al., 2024; Yang et al., 2025; Tajwar et al., 2024). To overcome
this limitation, online preference alignment methods such as Iterative DPO (Xiong et al., 2023) and
SPPO (Wu et al., 2024) have been proposed. These approaches adopt an iterative training paradigm
in which each round involves generating new responses from the current policy and optimizing
the model using annotated preferences (Yuan et al., 2024; Ko et al., 2024; Chen et al., 2024). By
enabling exploration beyond the limitations of the initial static dataset, they can improve model
performance (Su et al., 2025; Xu et al., 2024; Tajwar et al., 2024). However, they primarily rely on
on-policy data and often overlook valuable offline preference datasets. Bridging both paradigms,
hybrid methods like SELM (Zhang et al., 2024b) integrate offline and online data by using a reward
model to compare responses from both sources, thereby constructing more diverse preference sets.
Nonetheless, these hybrid approaches lack strategic sampling and generation mechanisms, failing to
account for the distribution relationship between offline and online data, which leads to inefficient
resource utilization and suboptimal performance.
**Preference Data Selection and Augmentation.** These limitations underscore the growing importance
of strategic data sampling and targeted augmentation in preference alignment. The “less is more”
principle suggests that selecting high-quality and relevant data often leads to better outcomes than
relying on large volumes of unfiltered data (Zhou et al., 2023; Deng et al., 2025; Liu et al., 2023; Ding
et al., 2024). Consequently, various data selection methods aim to identify the most informative and
helpful samples from general-purpose datasets to improve training effectiveness (Wang et al., 2025;
Khaki et al., 2024; Badrinath et al., 2024; Yang et al., 2024b). For instance, Selective DPO (Gao et al.,
2025) employs loss-based filtering to discard uninformative or harmful samples, while MAP (Huang
et al., 2025) uses internal and external reward margins to guide sample inclusion. Beyond sample
selection, data augmentation further enhance alignment (Muldrew et al., 2024; Gou & Nguyen, 2024;
Guo et al., 2024a; Liu et al., 2024; Das et al., 2024). ADPO (Ji et al., 2024) filters offline data
based on a manual threshold and then performs online augmentation on the selected samples. The
Reward-Augmented method (Zhang et al., 2024a) augments the dataset by reversing preference pairs
under the guidance of prompts. DICE (Chen et al., 2024) and SeRA (Ko et al., 2024) combine data
augmentation with filtering methods, guided by length regularization and reward margins to enhance
training data. Despite these advancements, most prior work treats data sampling as an auxiliary step,
often decoupled from the optimization process and guided by static heuristics. In contrast, MetaAPO
proposes a framework that closely integrates online data generation with preference optimization via
adaptive meta-weighting, mitigating the distribution mismatch in preference alignment.
3 PRELIMINARIES
**Reinforcement Learning from Human Feedback.** RLHF aims to align the language model policy
_π_ _**θ**_ ( _·|_ _**x**_ ) with human preferences over prompts _**x**_ _∈X_ . A typical RLHF pipeline (Ouyang et al.,
2022) starts by training a reward model on a preference dataset _D_ = _{_ ( _**x**_ _,_ _**y**_ _w,_ _**y**_ _l_ ) _[i]_ _}_ _[N]_ _i_ =1 [, where each]
preference pair is annotated such that _**y**_ _w ≻_ _**y**_ _l_ indicates that _**y**_ _w_ is preferred over _**y**_ _l_ for prompt _**x**_ .
The reward model _r_ _**ψ**_ ( _**x**_ _,_ _**y**_ ) is trained using the Bradley–Terry (BT) model (Bradley & Terry, 1952),
by minimizing a negative log-likelihood loss that encourages larger reward gaps between preferred
and dispreferred responses:
_L_ RM( _**ψ**_ ) = _−_ E( _**x**_ _,_ _**yw**_ _,_ _**yl**_ ) _∼D_ [ _σ_ ( _r_ _**ψ**_ ( _**x**_ _,_ _**y**_ _w_ ) _−_ _r_ _**ψ**_ ( _**x**_ _,_ _**y**_ _l_ ))] _,_ (1)
where _σ_ ( _·_ ) denotes the logistic function. The policy _π_ _**θ**_ is then optimized via reinforcement learning
to maximize the expected reward while staying close to a reference policy _π_ ref :
max _π_ _**θ**_ [E] _**[x]**_ _[∼D][,]_ _**[y]**_ _[∼][π]_ _**[θ]**_ [(] _[·|]_ _**[x]**_ [)][[] _[r]_ _**[ψ]**_ [(] _**[x]**_ _[,]_ _**[ y]**_ [)]] _[ −]_ _[β]_ [D][KL][[] _[π]_ _**[θ]**_ [(] _**[y]**_ _[|]_ _**[x]**_ [)] _[||][π]_ [ref][(] _**[y]**_ _[|]_ _**[x]**_ [)]] _[,]_ (2)
Preprint
where _β_ controls the trade-off between reward maximization and deviation from the reference policy.
**Direct Preference Optimization.** While effective, RLHF can be complex, unstable, and computationally demanding. DPO (Rafailov et al., 2023) simplifies the process by directly optimizing the
policy using a supervised learning objective over preference data, without explicitly learning a reward
model. DPO reparameterizes the reward as _r_ ( _**x**_ _,_ _**y**_ ) = _β_ (log _π_ _[π]_ ref _**[θ]**_ [(] ( _**[y]**_ _**y**_ _[|]_ _|_ _**[x]**_ _**x**_ [)] ) [+ log] _[ Z]_ [(] _**[x]**_ [))][, where] _[ Z]_ [(] _**[x]**_ [)][ is]
the partition function and _π_ ref is typically the supervised fine-tuned (SFT) model. By minimizing the
negative log-likelihood of reward modeling in Eq. 1, DPO derives the following loss:
_π_ ref( _**y**_ _l|_ _**x**_ )
_L_ DPO( _**θ**_ ) = _−_ E( _**x**_ _,_ _**yw**_ _,_ _**yl**_ ) _∼D_
- log _σ_ _β_ log _[π]_ _**[θ]**_ [(] _**[y]**_ _[w][|]_ _**[x]**_ [)]
_[π]_ _**[θ]**_ [(] _**[y]**_ _[w][|]_ _**[x]**_ [)] _[π]_ _**[θ]**_ [(] _**[y]**_ _[l][|]_ _**[x]**_ [)]
_π_ ref( _**y**_ _w|_ _**x**_ ) _[−]_ _[β]_ [ log] _π_ ref( _**y**_ _l|_ _**x**_ )
��
_._ (3)
DPO supports both offline preference datasets _D_ offline ( _e.g.,_ curated in advance) and online datasets
_D_ online (generated by the policy model _π_ _**θ**_ ).
4 METHOD
In this section, we present the technical details of MetaAPO. The alignment process is conducted
within a single training epoch, where the dataset is divided into sequential subsets and processed
iteratively over _t ∈{_ 0 _,_ 1 _, . . ., T_ _}_ . In each iteration _t_, we start by performing meta-weighted adaptive
online sampling to generate online data from policy _π_ _**θ**_ ( _t−_ 1) (Section 4.1). The policy is then finetuned on the hybrid dataset using the meta-weighted objective to obtain the updated policy _π_ _**θ**_ ( _t_ )
(Section 4.2). Concurrently, the meta-learner is periodically updated to better estimate the potential
gain from online data (refer to Section 4.3). The overall workflow is illustrated in Algorithm 1.
4.1 META-WEIGHTED ADAPTIVE ONLINE SAMPLING
We begin with a model-aware sampling strategy for acquiring new preference data. This process
involves using a preference scoring function (derived from objectives like DPO) to assess the utility
of existing offline samples. Given a prompt _**x**_ and a pair of preferred and dispreferred responses
( _**y**_ _w,_ _**y**_ _l_ ), the instance-level preference score is computed as:
      _ℓ_ ( _**x**_ _,_ _**y**_ _w,_ _**y**_ _l_ ) = log _σ_ _β_ log _[π]_ _**[θ]**_ [(] _**[y]**_ _[w][|]_ _**[x]**_ [)]
_π_ ref( _**y**_ _l|_ _**x**_ )
_[π]_ _**[θ]**_ [(] _**[y]**_ _[w][|]_ _**[x]**_ [)] _[π]_ _**[θ]**_ [(] _**[y]**_ _[l][|]_ _**[x]**_ [)]
_π_ ref( _**y**_ _w|_ _**x**_ ) _[−]_ _[β]_ [ log] _π_ ref( _**y**_ _l|_ _**x**_ )
_,_ (4)
where a higher score _ℓ_ ( _·_ ) indicates stronger agreement between the current policy _π_ _**θ**_ and human
preferences, compared to a fixed reference policy _π_ ref. This scoring function is applied to each instance
in the offline preference dataset _D_ offline = _{_ ( _**x**_ _,_ _**y**_ _w_ [off] _[,]_ _**[ y]**_ _l_ [off][)] _[i][}]_ _i_ _[N]_ =1 [, producing scores] _[ ℓ]_ [off][(] _**[x]**_ _[,]_ _**[ y]**_ _w_ [off] _[,]_ _**[ y]**_ _l_ [off][)][.]
To enable the sampling process to dynamically adapt to the model’s current state and requirement,
we introduce a meta-learner that adaptively re-weights each offline sample based on its preference
score. The meta-learner _h_ _**ϕ**_, implemented as a two-layer MLP, maps the preference score _ℓ_ ( _·_ ) to a
weight _w_ = _h_ _**ϕ**_ ( _ℓ_ [off] ( _·_ )) _∈_ [0 _,_ 1] (Line 7 in Alg. 1). A high weight indicates that the sample aligns
well with the model’s current behavior, while a low weight suggests potential misalignment that may
require further online exploration.
This adaptive meta-weight _w_ subsequently guides the online sampling and generation phase. (Line
8-12 in Alg. 1). For each offline tuple _{_ _**x**_ _,_ _**y**_ _w_ [off] _[,]_ _**[ y]**_ _l_ [off] _[}][i]_ [, a value] _[ u][i]_ [ is sampled from][ Uniform][(0] _[,]_ [ 1)][.]
If _ui > wi_, the current policy _π_ _**θ**_ generates _K_ candidate responses _{_ _**y**_ 1 [on] _[, . . .,]_ _**[ y]**_ _K_ [on] _[}]_ [ for prompt] _**[ x]**_ [.]
These responses are then ranked by an external reward model to produce online preference pairs
_**y**_ pair [on] [= (] _**[y]**_ _w_ [on] _[,]_ _**[ y]**_ _l_ [on][)][. Finally, both the sampled offline and corresponding online preferences are merged]
into an augmentation set _D_ aug. This augmented dataset forms a dynamic training stream that evolves
with the model, enabling the model to better attend to regions where alignment gaps may exist.
4.2 META-WEIGHTED PREFERENCE OPTIMIZATION
During preference optimization, we employ the sample-wise meta-weight _w_ to dynamically balance
the influence of offline preference data and online model-generated samples (Line 16-17 in Alg. 1).
The overall objective is to minimize the meta-weighted preference loss:
            _L_ ( _**θ**_ ) = _−_ E( _**x**_ _,_ _**y**_ pairoff _[,]_ _**[y]**_ pair [on] [)] _[∼D]_ [aug] _w · ℓ_ _**θ**_ ( _**x**_ _,_ _**y**_ _w_ [off] _[,]_ _**[ y]**_ _l_ [off][)]
~~�~~ ~~��~~              adaptive offline loss
+ (1 _−_ _w_ ) _· ℓ_ _**θ**_ ( _**x**_ _,_ _**y**_ _w_ [on] _[,]_ _**[ y]**_ _l_ [on][)]
~~�~~ ~~�~~  - ~~�~~
adaptive online loss
_,_
(5)
Preprint
**Algorithm 1** Training Scheme of MetaAPO
1: **Input:** Offline preference dataset _D_ offline = _{_ ( _**x**_ _,_ _**y**_ _w_ [off] _[,]_ _**[ y]**_ _l_ [off][)] _[i][}]_ _i_ _[N]_ =1 [;]
2: Language model _π_ _**θ**_, reference model _π_ ref, reward model _R_, meta-learner _h_ _**ϕ**_
3: **Initialize:** Load model parameters _**θ**_, initialize meta-learner parameters _**ϕ**_
4: **for** each iteration **do**
5: Initialize augmentation dataset _D_ aug _←∅_ and meta buffer _B_ meta _←∅_
6: **for** each ( _**x**_ _,_ _**y**_ _w_ [off] _[,]_ _**[ y]**_ _l_ [off][)] _[i][ ∈D]_ [offline] **[ do]** _▷_ **Meta-Weighted Adaptive Online Sampling**
7: Compute meta-weight: _wi ←_ _h_ _**ϕ**_ ( _ℓ_ ( _**x**_ _,_ _**y**_ _w_ [off] _[,]_ _**[ y]**_ _l_ [off][))]
8: Sample _ui ∼_ Uniform(0 _,_ 1)
9: **if** _ui > wi_ **then**
10: Generate _K_ responses: _{_ _**y**_ 1 [on] _[, . . .,]_ _**[ y]**_ _K_ [on] _[} ←]_ _[π]_ _**[θ]**_ [(] _[· |]_ _**[ x]**_ [)]
11: Evaluate with _R_ : _{_ _**y**_ _l_ [on] _[,]_ _**[ y]**_ _w_ [on] _[} ←R]_ [(] _**[x]**_ _[,][ {]_ _**[y]**_ _j_ [on] _[}]_ _j_ _[K]_ =1 [)]
12: Add to augmentation set: _D_ aug _←D_ aug _∪{_ ( _**x**_ _,_ _**y**_ _w_ [off] _[,]_ _**[ y]**_ _l_ [off] _[,]_ _**[ y]**_ _l_ [on] _[,]_ _**[ y]**_ _w_ [on][)] _[i][}]_
13: **end if**
14: **end for**
15: **for** each batch _D_ batch _∼D_ aug **do** _▷_ **Meta-Weighted Model Training** (freeze _h_ _**ϕ**_ )
16: Compute sample-wise meta-weight: _wj ←_ _h_ _**ϕ**_ ( _ℓ_ ( _**x**_ _,_ _**y**_ _w_ [off] _[,]_ _**[ y]**_ _l_ [off][))]
17: Update language model by Eq. 5: _**θ**_ _←_ _**θ**_ _−_ _α∇_ _**θ**_ _L_ ( _**θ**_ )
18: Add batch data to meta buffer: _B_ meta _←B_ meta _∪D_ batch
19: **if** batch iteration % _T_ meta = 0 **then** _▷_ **Meta-Learner Update** (freeze _π_ _**θ**_ )
20: Update meta-learner by Eq. 6: _**ϕ**_ _←_ _**ϕ**_ _−_ _η∇_ _**ϕ**_ _L_ meta( _**ϕ**_ ; _B_ meta)
21: Reset meta buffer: _B_ meta _←∅_
22: **end if**
23: **end for**
24: **end for**
where _ℓ_ _**θ**_ ( _·_ ) denotes a standard preference alignment objective ( _e.g.,_ DPO in Eq. 4 or SimPO) used to
align the current policy _π_ _**θ**_ with the target preferences. The first term captures contributions from
offline preferences, while the second focuses on online model-generated preference samples.
This joint optimization enriches data diversity. The balance coefficient _w_ = _h_ _**ϕ**_ ( _ℓ_ [off] ( _·_ )) is predicted by
the meta-learner and indicates the confidence in offline preference data. When the model’s behavior
aligns well with the offline preference, the meta-learner assigns a higher weight _w_ to reinforce stable
learning from reliable offline human-labeled examples. Conversely, when misalignment is detected,
_w_ is reduced, prompting greater reliance on adaptive corrections from online feedback.
4.3 LEARNING TO WEIGHT: META-LEARNER FOR ALIGNMENT GAP ESTIMATION
The effectiveness of both adaptive online sampling (Section 4.1) and meta-weighted preference
optimization (Section 4.2) critically depends on the quality of the dynamically assigned weights
_w_ = _h_ _**ϕ**_ ( _ℓ_ [off] ( _·_ )). Instead of relying on pre-defined heuristics or static weights, we propose to learn the
weighting function _h_ _**ϕ**_ through a meta-learning approach. The goal is for _h_ _**ϕ**_ to act as an “alignment
gap estimator”, dynamically assessing the utility of each offline sample relative to potential online
preference data, and assigning weights that effectively guide the alignment process. This allows
training data to be weighted based on the model’s current state, enabling continuous adaptation
throughout the alignment process.
Specifically, the meta-learner _h_ _**ϕ**_ is trained in an alternating manner with the policy model _π_ _**θ**_ . At
intervals specified by a hyperparameter _T_ meta, the parameters of _π_ _**θ**_ are temporarily frozen, and _h_ _**ϕ**_ is
updated using data stored in the meta buffer _B_ meta (Lines 19–21 in Alg. 1). This buffer aggregates
preference data from multiple recent training batches of _π_ _**θ**_, enabling the meta-learner to derive a
more stable and generalized weighting strategy from accumulated experience. The meta-learner _h_ _**ϕ**_
is optimized by minimizing the following loss:
              _L_ meta( _**ϕ**_ ) = _−_ E( _**x**_ _,_ _**y**_ pairoff _[,]_ _**[y]**_ pair [on] [)] _[∼B]_ [meta] _h_ _**ϕ**_ ( _ℓ_ [off] ( _·_ )) _· ℓ_ ( _**x**_ _,_ _**y**_ _w_ [off] _[,]_ _**[ y]**_ _l_ [off][)]
~~�~~ ~~�~~                - ~~�~~
adaptive offline loss
+ (1 _−_ _h_ _**ϕ**_ ( _ℓ_ [off] ( _·_ ))) _· ℓ_ ( _**x**_ _,_ _**y**_ _w_ [on] _[,]_ _**[ y]**_ _l_ [on][)]
~~�~~  - ~~�~~  adaptive online loss
_,_
(6)
Preprint
where _ℓ_ ( _**x**_ _,_ _**y**_ _w_ [off] _[,]_ _**[ y]**_ _l_ [off][)][ (abbreviated as] _[ ℓ]_ [off][(] _[·]_ [)][) and] _[ ℓ]_ [(] _**[x]**_ _[,]_ _**[ y]**_ _w_ [on] _[,]_ _**[ y]**_ _l_ [on][)][ (abbreviated as] _[ ℓ]_ [on][(] _[·]_ [)][) denote the]
preference scores for the offline and online response pairs, respectively, for the same prompt _**x**_ .
**What does meta-learner update do?** For a mechanistic understanding of meta-learner, it is useful
to analyze the gradient of the loss function _L_ meta( _**ϕ**_ ). The gradient with respect to the meta-learner
parameters _**ϕ**_ can be written as:
_∇_ _**ϕ**_ _L_ meta( _**ϕ**_ ) = E _B_ meta �� _ℓ_ [on] ( _·_ ) _−_ _ℓ_ [off] ( _·_ )� _· ∇_ _**ϕ**_ ( _h_ _**ϕ**_ ( _ℓ_ [off] ( _·_ ))� _._ (7)
This gradient drives the adaptation of the weighting function _h_ _**ϕ**_ . When online samples yield higher
preference scores than offline samples ( _i.e., ℓ_ [on] ( _·_ ) _> ℓ_ [off] ( _·_ )), the meta-learner reduces the weight
assigned to offline data, encouraging exploration guided by online feedback. Conversely, when offline
preferences remain superior, their weights are increased to reinforce stable learning from reliable
human annotations. The full gradient derivation is provided in Appendix A.1.
Moreover, we provide a theoretical guarantee for the meta-learner _hϕ_ in Theorem 1. The result
demonstrates that the risk of the learned weighting function is provably close to the oracle risk, with
the alignment gap controlled by the hypothesis complexity and meta-buffer size.
**Theorem 1** (Generalization Bound for Meta-Learner) **.** _Let_ _h_ [ˆ] _**ϕ**_ _denote the meta-learner function_
_learned by minimizing the empirical meta-risk over the meta-buffer Bmeta of size m, and let h_ _[∗]_ _be the_
_oracle function that minimizes the true meta-risk over the hypothesis space H. Assume the meta-loss_
_Lmeta_ ( _**ϕ**_ ) _is bounded in_ [0 _, M_ ] _for any sample_ ( _**x**_ _,_ _**y**_ _pair_ _[off]_ _[,]_ _**[ y]**_ _pair_ _[on]_ [)] _[. Then, for any][ δ >]_ [ 0] _[, with probability]_
_at least_ 1 _−_ _δ, the following inequality holds:_
           _R_ ( _h_ [ˆ] _**ϕ**_ ) _≤_ _R_ ( _h_ _[∗]_ ) + 4 Rad _m_ ( _Lmeta_ ( _**ϕ**_ ) _◦H_ ) + _M_
2 ln(1 _m/δ_ ) _,_
_where R_ ( _h_ _**ϕ**_ ) _and_ _R_ [�] _m_ ( _h_ _**ϕ**_ ) _denote the true and empirical risk respectively, and_ Rad _m is the_
_Rademacher complexity of the meta-loss hypothesis. The detailed proof is provided in Appendix A.2._
Theorem 1 shows that as the meta-buffer size _m_ increases, the risk of the learned meta-learner _R_ ( _h_ [ˆ] _**ϕ**_ )
converges to that of the oracle function _R_ ( _h_ _[∗]_ ). And the Rademacher complexity term favors a simple
hypothesis space for better generalization. This justifies that a sufficient meta-buffer combined with a
simple meta-learner can achieve effective estimation and assign near-optimal weights.
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUP
**Datasets and Models.** We use the Ultrafeedback dataset (Cui et al., 2024), which contains 64K
preference annotations collected from four LLM-generated outputs and labeled preferences by GPT4. To demonstrate the general effectiveness of our method, we experiment with two supervised
fine-tuned models from different-sized base model: Llama-3.1-8B (Grattafiori et al., 2024) and
Qwen2.5-7B (Yang et al., 2024a). For online preference annotation, we train the reward models on
UltraFeedback using the corresponding base model to avoid any preference leakage. We also provide
a comparison with existing open-source reward models in Appendix C.2.
**Evaluation Metric.** We evaluate our method using three widely adopted automatic evaluation
benchmarks for LLMs: AlpacaEval 2 (Dubois et al., 2024), MT-Bench (Zheng et al., 2023), and
Arena-Hard (Li et al., 2024). All evaluations are conducted using GPT-4o-latest as the judge model.
For AlpacaEval 2, we report the raw win rate (WR) and the length-controlled win rate (LC) against
the reference model GPT-4o-2024-05-13. For Arena-Hard, we report the win rate (WR) and the
style-controlled win rate (SC), comparing our model against the GPT-4-Preview-1106 baseline. For
MT-Bench, we report the average multi-turn score (Score) assigned by GPT-4o-latest, which rates
each response on a 10-point scale.
**Baselines.** We evaluate the following baseline methods relevant to the setting of our method:
 - **Offline Alignment** : Methods that optimize models solely using offline human preference datasets,
such as DPO (Rafailov et al., 2023), IPO (Azar et al., 2024) and KTO (Ethayarajh et al., 2024).
This category also includes Selective DPO (Gao et al., 2025), which improves data quality by
filtering out out-of-distribution samples based on a hand-crafted metric.
Preprint
Table 1: Overall performance of our proposed MetaAPO method with Llama-3.1-8B and Qwen2.57B, compared with offline, online and hybrid baseline methods on AlpacaEval 2, Arena-Hard and
MT-Bench. The best results are highlighted in **bold** .
|Llama-3.1-8B Qwen2.5-7B<br>Models AlpacaEval 2 Arena-Hard MT-Bench AlpacaEval 2 Arena-Hard MT-Bench<br>WR(%) LC(%) WR(%) SC(%) Score WR(%) LC(%) WR(%) SC(%) Score|Col2|Col3|
|---|---|---|
|**SFT**|10.02<br>17.28<br>18.5<br>21.6<br>6.63|34.03<br>34.08<br>48.5<br>44.3<br>7.71|
_Offline Alignment_
 - **Online Alignment** : Methods that collect model-generated responses and align them using
preferences from an external reward model— _e.g.,_ Online DPO (Xiong et al., 2023) and PPO.
 - **Hybrid Alignment** : Methods that integrate both offline annotated preferences and online modelgenerated data to optimize alignment, as ADPO (Ji et al., 2024) and SELM (Zhang et al., 2024b).
**Implementation.** All experiments are conducted using 8 _×_ 80GB NVIDIA H100 GPUs with a batch
size of 128 and a maximum sequence length of 4096 tokens for 1 epoch. At each iteration round,
we sample responses from the current policy using different decoding configurations: temperature
_T_ = 1 _._ 0, _top_ - _p_ = 1 _._ 0 for the Llama3.1 setting, and _T_ = 0 _._ 8, _top_ - _p_ = 0 _._ 95 for the Qwen2.5 setting.
For each prompt, we generate _K_ = 8 diverse responses, which are then scored by a reward model
to construct preference pairs. The meta-learner is updated every _T_ meta = 8 steps. Additional details
about model configurations, evaluation procedure and hyperparameters are provided in Appendix B.
5.2 OVERALL PERFORMANCE
In Table 1, we present the overall results of MetaAPO compared to offline, online and hybrid
state-of-the-art alignment methods on the AlpacaEval 2, Arena-Hard and MT-Bench benchmarks.
**MetaAPO consistently outperforms preference optimization baselines.** Compared to offline alignment methods, which rely solely on fixed, pre-collected datasets, MetaAPO avoids the performance
degradation caused by distribution shifts between static data and evolving policies. Enhanced offline
approaches like Selective DPO, which attempt to mitigate this mismatch through data filtering, yield
only marginal improvements and still fall short of MetaAPO. Online and hybrid methods leverage
model-generated samples and online feedback to significantly outperform purely offline baselines.
While online and hybrid methods significantly outperform purely offline baselines by leveraging
on-policy feedback, they are often constrained by the quality and diversity of their own model generations. The heavy reliance on self-generated data can introduce noisy preferences. MetaAPO surpasses
these strong baselines by avoiding regions prone to noisy feedback and effectively leveraging diverse
offline data. We also compare performance across different reward models, preference optimization
methods and datasets in Appendix C.2, demonstrating the robustness of our approach.
**MetaAPO achieves superior performance with fewer online samples.** As shown in Figure 1
(Right), MetaAPO outperforms strong baselines such as Online DPO, SELM and PPO, while using
only 58% of the online generated and annotated samples on average compared to standard online
generation methods. This efficiency comes from a meta-weighted sampling mechanism, which
Preprint
Figure 2: **Left:** The dynamic changes in Offline Score (left y-axis) and Reward (right y-axis) across
training iterations. **Middle:** Scatter plot of offline preference score ( _ℓ_ off( _·_ )) versus the online-offline
score gap ( _ℓ_ on( _·_ ) _−_ _ℓ_ off( _·_ )). Points are colored by their sampling status: blue for _“Sampled”_ (selected
for online generation) and orange for _“Unsampled”_ . **Right:** Comparison of independent reward score
distributions (via kernel density estimation) for testset responses generated by DPO and MetaAPO.
prioritizes the most informative data points and skips unnecessary online generation for already
aligned or low-gain predicted samples. This adaptive data sampling significantly reduces labeling
costs and time consumption, cutting time by 80.1% compared to PPO and by 52.9% compared to
Online DPO (detailed in Appendix C.6).
5.3 FURTHER ANALYSES
**Training dynamics reveal an explore-integrate behavior.** To examine the dynamic interplay during
training, we tracked the offline preference score _ℓ_ [off] and generation reward on a held-out test set across
iterations. As shown in Figure 2 (Left), the average reward steadily increases (from 1 _._ 06 to 1 _._ 33)
while its standard deviation decreases (from 0 _._ 49 to 0 _._ 37), indicating that MetaAPO robustly enhances
generation quality under reward guidance. In contrast, the mean offline preference score drops sharply
at Iteration 2 before recovering, suggesting that MetaAPO does more than fit static offline data. The
drop reflects active exploration of response strategies diverging from offline preferences, while the
subsequent recovery in Iteration 3 reflects the integration of these discoveries with foundational
offline knowledge. Overall, these dynamics highlight MetaAPO’s adaptive behavior: it continuously
identifies and addresses gaps between the evolving policy and offline data.
**The dynamics are driven by a high-gain sampling strategy.** Figure 2 (Middle) shows MetaAPO’s
online sampling strategy based on two signals: the initial offline preference score _ℓ_ [off] ( _·_ ) (x-axis),
and the potential gain _ℓ_ [on] ( _·_ ) _−_ _ℓ_ [off] ( _·_ ) (y-axis). Each point represents an offline sample, marked
as _“Sampled”_ (blue) if selected for generation, or _“Unsampled”_ (orange) otherwise. Two trends
drive sampling. First, points with low _ℓ_ [off] ( _·_ ) (indicating weak alignment or OOD) receive smaller
meta-learner weights _w_, increasing their selection probability. Second, the meta-learner prioritizes
samples with high potential gain ( _ℓ_ [on] ( _·_ ) _> ℓ_ [off] ( _·_ )), concentrating its selection in the upper-left region
where these two factors converge. In contrast, samples with high offline scores or minimal online
gain are less likely to be selected, avoiding redundant computation.
**Effectiveness of the meta-learner.** A key component of MetaAPO is the meta-learner, a simple
two-layer MLP that effectively captures the model’s dynamic alignment state. Its effectiveness stems
from two factors. First, we observed a clear, learnable relationship between the offline score _ℓ_ [off] ( _·_ )
and potential online-offline gain _ℓ_ [on] ( _·_ ) _−_ _ℓ_ [off] ( _·_ ), as shown in Figure 2 (Left). This discovery allows
even a lightweight network to serve as a reliable "alignment gap estimator." Second, the meta-learner
is continuously updated alongside the policy, adapting its sampling and weighting decisions to the
evolving model state. Experiments with deeper networks and more complex inputs (Appendix C.5)
show that the two-layer MLP provides sufficient expressive power, confirming its efficiency and
effectiveness. Additional analyses further confirm the generalizability of MetaAPO. As shown in
Figure 2 (Right), its generated responses on the test set achieve higher scores distribution under an
independent reward model OpenRLHF/Llama-3-8b-rm-mixture.
5.4 ABLATION STUDY
To analyze the impact of each core component in MetaAPO, we conduct a series of ablation studies
under the Llama-3.1-8B setting with the DPO objective. Table 2 summarizes the effectiveness of
different sampling strategies, loss weighting mechanisms and the meta-learner.
Preprint
Table 2: Ablation study on key components of MetaAPO. We assess the impact of the sampling
strategy, loss weighting, and meta-learner. Metrics include AlpacaEval2, Arena-Hard, MT-Bench and
the proportion of annotations used during training.
|Model|AlpacaEval 2 Arena-Hard MT-Bench<br>WR(%) ↑ LC(%) ↑ WR(%) ↑ SC(%) ↑ Score ↑|Annotations<br>Ratio(%) ↓|
|---|---|---|
|**MetaAPO**<br>_w/_ Random Sampling<br>_w/_ Threshold Sampling<br>_w/_ All Sampling<br>_w/_ Uniform Loss Weighting<br>_w/o_ Meta-Learner|**47.48**<br>**43.21**<br>**43.9**<br>**40.8**<br>**7.56**<br>42.61<br>38.53<br>37.3<br>32.1<br>7.14<br>44.13<br>42.92<br>40.7<br>38.5<br>7.14<br>46.13<br>42.04<br>40.2<br>37.5<br>6.96<br>39.25<br>37.70<br>39.2<br>33.3<br>7.15<br>43.07<br>40.48<br>38.3<br>33.1<br>7.06|58.4<br>50.0<br>53.3<br>100.0<br>67.1<br>61.1|
**Effects of sampling strategies.** The choice of which offline samples to augment with online
responses is a critical factor to achieve efficient and effective alignment. We ablate three design
choices and keep the preference optimization and meta-learner training process consistent with the
original MetaAPO setup: (1) _Random Sampling_ selects half of the offline data uniformly for online
augmentation. (2) _Threshold Sampling_ selects samples whose initial preference scores fall below a
fixed threshold (resulting in a 53.3% annotation ratio). (3) _All Sampling_ augments the entire offline
dataset. As shown in Table 2, MetaAPO outperforms both random and threshold sampling across
all metrics. _All Sampling_ leverages a larger volume of data, yet the absence of consideration for
the dynamic distribution relationship between offline and online data during sampling results in
suboptimal alignment performance.
**Effects of loss weighting.** We evaluate the impact of meta-weighted preference optimization by
replacing the learned weights _w_ in Eq. 5 with a _uniform coefficient_ of 0 _._ 5 for all augmented samples,
assigning equal importance to offline and online preference pairs during training. Other components,
including adaptive sampling and meta-learner updates, remain unchanged. As shown in Table 2,
this modification leads to a significant performance drop with consistent declines on all benchmarks.
These results suggest that assigning equal importance forces the model to learn from low-margin
online data for already-aligned samples, which provides a weak signal and can destabilize learning,
underscoring the need for a dynamic balancing mechanism.
**Effects of Meta-learner.** We also evaluate the necessity of a trainable meta-learner by replacing
meta-learner function _h_ _**ϕ**_ with a fixed sigmoid-based heuristic ( _w/o Meta-Learner_ in Table 2). This
heuristic also uses offline preference scores as input but remains static during the meta-learner update
phase. Other components, including sampling and loss weighting mechanisms, were kept identical.
The results show performance drops across all metrics. For example, AlpacaEval 2 WR decreased
from 47 _._ 48% to 43 _._ 07%, and Arena-Hard WR dropped from 43 _._ 9% to 38 _._ 3%. Despite using more
online annotations (61 _._ 1% _vs._ 58 _._ 4%), the fixed heuristic underperformed, confirming the importance
of a learnable meta-learner for adaptive weighting and effective alignment. More ablation studies on
reward model annotations and offline data utilization are provided in Appendix C.2.
6 CONCLUSION
In this paper, we propose MetaAPO, a novel framework that bridges the gap between data generation
and model training in LLM alignment. The core idea is to leverage a meta-learner to assess the
alignment of the current policy and identify the potential for online exploration. To achieve this, we
introduce the techniques of _adaptive online sampling_, _meta-weighted optimization_ and _meta-learner_
_update_, which jointly enable dynamic integration of data and training. Experiments across three
alignment benchmarks validate the effectiveness of MetaAPO in consistently outperforming existing
alignment methods. While our current approach effectively utilizes preference scores as input, it may
be beneficial to explore additional input signals, such as gradient or representation-based features, to
further enhance flexibility and generalization.
Preprint
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman,
Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.
_arXiv preprint
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal
Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from
human preferences. In _International Conference on Artificial Intelligence and Statistics_, pp.
4447–4455. PMLR, 2024.
Anirudhan Badrinath, Prabhat Agarwal, and Jiajing Xu. Hybrid preference optimization: Augmenting
direct preference optimization with auxiliary objectives. _arXiv preprint
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain,
Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with
reinforcement learning from human feedback. _arXiv preprint
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method
of paired comparisons. _Biometrika_, 39(3/4):324–345, 1952.
Changyu Chen, Zichen Liu, Chao Du, Tianyu Pang, Qian Liu, Arunesh Sinha, Pradeep Varakantham, and Min Lin. Bootstrapping language models with dpo implicit rewards. _arXiv preprint_
_
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie,
Ruobing Xie, Yankai Lin, et al. Ultrafeedback: Boosting language models with scaled ai feedback.
In _International Conference on Machine Learning_, pp. 9722–9744. PMLR, 2024.
Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano, and Sayak Ray Chowdhury. Active preference
optimization for sample efficient rlhf. _arXiv preprint
Xun Deng, Han Zhong, Rui Ai, Fuli Feng, Zheng Wang, and Xiangnan He. Less is more: Improving
llm alignment via preference data selection. _arXiv preprint
Bosheng Ding, Chengwei Qin, Ruochen Zhao, Tianze Luo, Xinze Li, Guizhen Chen, Wenhan Xia,
Junjie Hu, Luu Anh Tuan, and Shafiq Joty. Data augmentation using llms: Data perspectives,
learning paradigms and challenges. In _Findings of the Association for Computational Linguistics_
_ACL 2024_, pp. 1679–1705, 2024.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao,
Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative
foundation model alignment. _arXiv preprint
Qingxiu Dong, Li Dong, Xingxing Zhang, Zhifang Sui, and Furu Wei. Self-boosting large language
models with synthetic preference data. _arXiv preprint
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled
alpacaeval: A simple way to debias automatic evaluators. _arXiv preprint
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model
alignment as prospect theoretic optimization. _arXiv preprint
Chengqian Gao, Haonan Li, Liu Liu, Zeke Xie, Peilin Zhao, and Zhiqiang Xu. Principled data
selection for alignment: The hidden risks of difficult examples. _arXiv preprint
2025.
Qi Gou and Cam-Tu Nguyen. Mixed preference optimization: Reinforcement learning with data
selection and better reference model. _arXiv preprint
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad
Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of
models. _arXiv preprint
Preprint
Hongyi Guo, Yuanshun Yao, Wei Shen, Jiaheng Wei, Xiaoying Zhang, Zhaoran Wang, and Yang Liu.
Human-instruction-free llm self-alignment with limited samples. _arXiv preprint
2024a.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre
Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online
ai feedback. _arXiv preprint
Kexin Huang, Junkang Wu, Ziqian Chen, Xue Wang, Jinyang Gao, Bolin Ding, Jiancan Wu, Xiangnan
He, and Xiang Wang. Larger or smaller reward margins to select preferences for alignment? _arXiv_
_preprint
Kaixuan Ji, Jiafan He, and Quanquan Gu. Reinforcement learning from human feedback with active
queries. _arXiv preprint
Saeed Khaki, JinJin Li, Lan Ma, Liu Yang, and Prathap Ramachandra. Rs-dpo: A hybrid rejection
sampling and direct preference optimization method for alignment of large language models. _arXiv_
_preprint
Jongwoo Ko, Saket Dingliwal, Bhavana Ganesh, Sailik Sengupta, Sravan Bodapati, and Aram
Galstyan. Sera: Self-reviewing and alignment of large language models using implicit reward
margins. _arXiv preprint
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez,
and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder
pipeline. _arXiv preprint
Biao Liu, Ning Xu, and Xin Geng. Progressively label enhancement for large language model
alignment. _arXiv preprint
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for
alignment? a comprehensive study of automatic data selection in instruction tuning. _arXiv preprint_
_
Chris Lu, Samuel Holt, Claudio Fanconi, Alex Chan, Jakob Foerster, Mihaela van der Schaar, and
Robert Lange. Discovering preference optimization algorithms with and for large language models.
_Advances in Neural Information Processing Systems_, 37:86528–86573, 2024.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a referencefree reward. _Advances in Neural Information Processing Systems_, 37:124198–124235, 2024.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. _Foundations of machine learning_ . MIT
press, 2018.
William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber. Active preference learning for
large language models. In _International Conference on Machine Learning_, pp. 36577–36590.
PMLR, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong
Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow
instructions with human feedback. _Advances in neural information processing systems_, 35:27730–
27744, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea
Finn. Direct preference optimization: Your language model is secretly a reward model. _Advances_
_in Neural Information Processing Systems_, 36:53728–53741, 2023.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford,
Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. _Advances in_
_neural information processing systems_, 33:3008–3021, 2020.
Xuerui Su, Yue Wang, Jinhua Zhu, Mingyang Yi, Feng Xu, Zhiming Ma, and Yuting Liu. Reveal the
mystery of dpo: The connection between dpo and rl algorithms. _arXiv preprint
2025.
Preprint
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano
Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of llms should leverage suboptimal,
on-policy data. In _International Conference on Machine Learning_, pp. 47441–47474. PMLR,
2024.
Yifan Wang, Runjin Chen, Bolian Li, David Cho, Yihe Deng, Ruqi Zhang, Tianlong Chen, Zhangyang
Wang, Ananth Grama, and Junyuan Hong. More is less: The pitfalls of multi-model synthetic
preference data in dpo safety alignment. _arXiv preprint
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play
preference optimization for language model alignment. _arXiv preprint
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less:
Selecting influential data for targeted instruction tuning. _arXiv preprint
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang.
Iterative preference learning from human feedback: Bridging theory and practice for rlhf under
kl-constraint. _arXiv preprint
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu,
and Yi Wu. Is dpo superior to ppo for llm alignment? a comprehensive study. In _International_
_Conference on Machine Learning_, pp. 54983–54998. PMLR, 2024.
Yuzi Yan, Yibo Miao, Jialian Li, Yipin Zhang, Jian Xie, Zhijie Deng, and Dong Yan. 3d-properties:
Identifying challenges in dpo and charting a path forward. _arXiv preprint
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li,
Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. _arXiv preprint_
_
Sen Yang, Leyang Cui, Deng Cai, Xinting Huang, Shuming Shi, and Wai Lam. Not all preference
pairs are created equal: A recipe for annotation-efficient iterative preference learning. In _Findings_
_of the Association for Computational Linguistics: EMNLP 2024_, pp. 6549–6561, 2024b.
Xiliang Yang, Feng Jiang, Qianen Zhang, Lei Zhao, and Xiao Li. Dpo-shift: Shifting the distribution
of direct preference optimization. _arXiv preprint
Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank
responses to align language models with human feedback. _Advances in Neural Information_
_Processing Systems_, 36:10935–10950, 2023.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason E
Weston. Self-rewarding language models. _ArXiv_, abs/2401.10020, 2024.
Shenao Zhang, Zhihan Liu, Boyi Liu, Yufeng Zhang, Yingxiang Yang, Yongfei Liu, Liyu Chen, Tao
Sun, and Zhaoran Wang. Reward-augmented data enhances direct preference alignment of llms.
_arXiv preprint
Shenao Zhang, Donghan Yu, Hiteshi Sharma, Han Zhong, Zhihan Liu, Ziyi Yang, Shuohang Wang,
Hany Hassan, and Zhaoran Wang. Self-exploring language models: Active preference elicitation
for online alignment. _arXiv preprint
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang,
Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and
chatbot arena. _Advances in Neural Information Processing Systems_, 36:46595–46623, 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia
Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. _Advances in Neural Information_
_Processing Systems_, 36:55006–55021, 2023.
Preprint
APPENDIX
In the appendix, we provide the following supplementary materials: (A) Details of the formulation
derivation, including meta-learner gradient derivation, theorem proof and SimPO objective formulation. (B) Implementation details of the experimental setup, including models and datasets, evaluation
benchmarks, baselines, model training and hyperparameters. (C) presents further experimental
analyses, including robustness of reward models, our tuning strategy for the meta-learner update
interval _T_ meta, deeper investigations into the meta-learner’s behavior with supporting visualizations,
and results from an extended ablation study. (D) provides qualitative case studies that illustrate
MetaAPO’s output quality in comparison to baseline methods.
A MATHEMATICAL DERIVATIONS AND FORMULATIONS
A.1 DERIVATION OF THE META-LEARNER LOSS GRADIENT
The meta-learner aims to capture the trade-off between offline and online preference and maximize
the expected weighted sum of these preference scores. Therefore, its update is guided by minimizing
the following loss:
               -               _L_ meta( _**ϕ**_ ) = _−_ E _B_ meta _h_ _**ϕ**_ ( _ℓ_ [off] ( _·_ )) _· ℓ_ [off] ( _·_ ) + (1 _−_ _h_ _**ϕ**_ ( _ℓ_ [off] ( _·_ ))) _· ℓ_ [on] ( _·_ ) _._ (A.1)
This expression can be reformulated to more effectively represent the optimal objective _h_ _**ϕ**_ ( _ℓ_ [off] ( _·_ )):
_L_ meta( _**ϕ**_ ) = _−_ E _B_ meta �� _ℓ_ [off] ( _·_ ) _−_ _ℓ_ [on] ( _·_ )� _· h_ _**ϕ**_ ( _ℓ_ [off] ( _·_ )) + _ℓ_ [on] ( _·_ )�
(A.2)
=E _B_ meta �� _ℓ_ [on] ( _·_ ) _−_ _ℓ_ [off] ( _·_ )� _· h_ _**ϕ**_ ( _ℓ_ [off] ( _·_ )) _−_ _ℓ_ [on] ( _·_ )� _._
Differentiating both sides of _L_ meta with respect to the meta-learner parameters _**ϕ**_ yields:
_∇_ _**ϕ**_ _L_ meta( _**ϕ**_ ) = E _B_ meta �� _ℓ_ [on] ( _·_ ) _−_ _ℓ_ [off] ( _·_ )� _· ∇_ _**ϕ**_ ( _h_ _**ϕ**_ ( _ℓ_ [off] ( _·_ ))� _._ (A.3)
This gradient governs the meta-learner’s update. The term ( _ℓ_ [on] ( _·_ ) _−_ _ℓ_ [off] ( _·_ )) can be seen as an
“advantage” signal, indicating how much better (or worse) the online score is compared to the offline
score for the same underlying prompt.
**Interpretation.** When online samples yield higher preference scores than their offline counterparts
( _i.e., ℓ_ [on] ( _·_ ) _> ℓ_ [off] ( _·_ )), the factor ( _ℓ_ [on] ( _·_ ) _−_ _ℓ_ [off] ( _·_ )) is positive. The gradient update will then tend to
adjust _**ϕ**_ such that _h_ _**ϕ**_ ( _ℓ_ [off] ( _·_ )) (the weight assigned to the offline sample) decreases. This encourages
more reliance on online exploration and data for instances where it proves more beneficial. Conversely,
when offline samples yield scores that are comparable to or better than online samples ( _i.e., ℓ_ [off] ( _·_ ) _≥_
_ℓ_ [on] ( _·_ )), the factor _ℓ_ [on] ( _·_ ) _−_ _ℓ_ [off] ( _·_ ) is zero or negative. This leads to updates that preserve or increase
the weight _h_ _**ϕ**_ ( _ℓ_ [off] ( _·_ )), thereby promoting stable exploitation of reliable, curated offline data.
A.2 PROOF OF THEOREM 1
We provide a complete proof of Theorem 1. First, we establish a uniform convergence bound between
the empirical and true risk of the meta-learner, using the Rademacher complexity of the meta-loss
hypothesis class. Then, we decompose the excess risk of the learned meta-learner relative to the
oracle into terms that can be bounded by the uniform convergence result. Finally, we obtain a
high-probability generalization guarantee for _h_ [ˆ] _**ϕ**_ as stated in the theorem.
A.2.1 PRELIMINARIES
**Definition 1** (Meta-Loss Function) **.** _For a meta-sample_ ( _**x**_ _,_ _**y**_ _pair_ _[off]_ _[,]_ _**[ y]**_ _pair_ _[on]_ [)] _[ and a meta-learner][ h]_ _**[ϕ]**_ _[, the]_
_meta-loss is defined as:_
_Lmeta_ ( _h_ _**ϕ**_ ; _**x**_ _,_ _**y**_ _pair_ _[off]_ _[,]_ _**[ y]**_ _pair_ _[on]_ [)] _[ ∈]_ [[0] _[, M]_ []] _[.]_
_where M is an upper bound on the loss. The meta-loss is always non-negative and does not exceed_
_M for any samples_ ( _**x**_ _,_ _**y**_ _pair_ _[off]_ _[,]_ _**[ y]**_ _pair_ _[on]_ [)] _[.]_
Preprint
**Definition 2** (True Risk and Empirical Risk) **.** _Let D be the distribution over meta-samples, and_
_Bmeta_ = _{_ ( _**x**_ _i,_ _**y**_ _i_ _[off][,]_ _**[ y]**_ _i_ _[on]_ [)] _[}]_ _i_ _[m]_ =1 _[a meta-buffer of size][ m][. The true and empirical risk of][ h]_ _**[ϕ]**_ _[ are defined]_
_as:_
_R_ ( _h_ _**ϕ**_ ) = E( _**x**_ _,_ _**y**_ _pairoff_ _[,]_ _**[y]**_ _pair_ _[on]_ [)] _[∼D]_ [[] _[L][meta]_ [(] _[h]_ _**[ϕ]**_ [)]] _[,]_ _R_ - _m_ ( _h_ _**ϕ**_ ) = _m_ [1]
_m_
- _Lmeta_ ( _h_ _**ϕ**_ ; _**x**_ _i,_ _**y**_ _i_ _[off][,]_ _**[ y]**_ _i_ _[on]_ [)] _[.]_
_i_ =1
**Definition 3** (Rademacher Complexity) **.** _For a function class F, the empirical Rademacher complex-_
_ity with respect to the meta-buffer Bmeta is:_
_m_
- _σif_ ( _**x**_ _i,_ _**y**_ _i_ _[off][,]_ _**[ y]**_ _i_ _[on]_ [)] ��� _Bmeta_ - _,_
_i_ =1
       Rad� _m_ ( _F_ ) = E _σ_ sup
_f_ _∈F_
_m_
_where σi are i.i.d. Rademacher variables. Its expectation over Bmeta defines_ Rad _m_ ( _F_ ) _. This quantity_
_captures the capacity/complexity of the meta-loss hypothesis class and is a standard tool in statistical_
_learning theory (Mohri et al., 2018)._
A.2.2 PROOF OF THEOREM 1
**Step 1: Uniform Convergence.** By standard results in statistical learning theory (Mohri et al.,
2018), for any function class _F_ bounded in [0 _, M_ ], with probability at least 1 _−_ _δ_ over the choice of
the meta-buffer _B_ meta:
ln(12 _m/δ_ ) _._ (A.4)
sup
_f_ _∈F_
        ��E[ _f_ ] _−_ E� _m_ [ _f_ ]�� _≤_ 2 Rad _m_ ( _F_ ) + _M_
**Step 2: Excess Risk Decomposition.** Let ∆= sup _h_ _**ϕ**_ _∈H |R_ ( _h_ _**ϕ**_ ) _−_ _R_ [�] _m_ ( _h_ _**ϕ**_ ) _|_ . For the empirical
minimizer _h_ [ˆ] _**ϕ**_ = arg min _h_ _**ϕ**_ _∈H_ _R_ [�] _m_ ( _h_ _**ϕ**_ ) and the oracle _h_ _[∗]_ = arg min _h_ _**ϕ**_ _∈H R_ ( _h_ _**ϕ**_ ), we have
_R_ ( _h_ [ˆ] _**ϕ**_ ) _−_ _R_ ( _h_ _[∗]_ ) = _R_ ( _h_ [ˆ] _**ϕ**_ ) _−_ _R_ [�] _m_ ( _h_ [ˆ] _**ϕ**_ )
         - ~~�~~         - ~~�~~
_≤_ ∆
+ _R_ [�] _m_ ( _h_ [ˆ] _**ϕ**_ ) _−_ _R_ [�] _m_ ( _h_ _[∗]_ )
 - ~~��~~  _≤_ 0
+ _R_ [�] _m_ ( _h_ _[∗]_ ) _−_ _R_ ( _h_ _[∗]_ )
~~�~~ ~~��~~ ~~�~~
_≤_ ∆
∆ 0 (A.5)
_≤_ 2 sup _|R_ ( _h_ _**ϕ**_ ) _−_ _R_ [�] _m_ ( _h_ _**ϕ**_ ) _|._
_h_ _**ϕ**_ _∈H_
**Step 3: Substituting the Uniform Bound.** Applying (A.4) to the meta-loss class _L_ meta( _**ϕ**_ ) _◦H_,
with probability at least 1 _−_ _δ_ :
~~�~~
ln(1 _/δ_ )
_._ (A.6)
2 _m_
sup _|R_ ( _h_ _**ϕ**_ ) _−_ _R_ [�] _m_ ( _h_ _**ϕ**_ ) _| ≤_ 2 Rad _m_ ( _L_ meta( _**ϕ**_ ) _◦H_ ) + _M_
_h_ _**ϕ**_ _∈H_
Combining with (A.5) yields the desired bound:
_R_ ( _h_ [ˆ] _**ϕ**_ ) _≤_ _R_ ( _h_ _[∗]_ ) + 4 Rad _m_ ( _L_ meta( _**ϕ**_ ) _◦H_ ) + _M_
This completes the proof.
A.3 METAAPO WITH SIMPO OBJECTIVE
2 ln(1 _/δ_ )
_._ (A.7)
_m_
The MetaAPO framework is designed to be largely agnostic to the choice of the underlying preference optimization loss, enabling it to be applied in conjunction with various alignment algorithms.
To showcase this flexibility, we demonstrate its integration with Simple Preference Optimization
(SimPO) (Meng et al., 2024), a reference-free alignment method. This integration results in the
MetaAPO-SimPO variant.
The original objective of SimPO involves minimizing the following loss function:
_[β]_ �� _._ (A.8)
_|_ _**y**_ _l|_ [log] _[ π]_ _**[θ]**_ [(] _**[y]**_ _[l][|]_ _**[x]**_ [)] _[ −]_ _[γ]_
_L_ SimPO( _π_ _**θ**_ ) = _−_ E( _**x**_ _,_ _**yw**_ _,_ _**yl**_ ) _∼D_
- - _β_
log _σ_
_β_
_|_ _**y**_ _w|_ [log] _[ π]_ _**[θ]**_ [(] _**[y]**_ _[w][|]_ _**[x]**_ [)] _[ −]_ _|_ _**y**_ _[β]_
Preprint
From this formulation, the corresponding instance-level preference score _ℓ_ ( _**x**_ _,_ _**y**_ _w_ [off] _[,]_ _**[ y]**_ _l_ [off][)][ in SimPO,]
used to quantify how well the current policy _π_ _**θ**_ aligns with the given preference pair ( _**y**_ _w_, _**y**_ _l_ ) for a
prompt _**x**_, is defined as the term whose negative log-likelihood is minimized in the SimPO objective:
      - _β_
_ℓ_ ( _**x**_ _,_ _**y**_ _w,_ _**y**_ _l_ ) = log _σ_
_[β]_  - _._ (A.9)
_|_ _**y**_ _l|_ [log] _[ π]_ _**[θ]**_ [(] _**[y]**_ _[l][|]_ _**[x]**_ [)] _[ −]_ _[γ]_
_β_
_|_ _**y**_ _w|_ [log] _[ π]_ _**[θ]**_ [(] _**[y]**_ _[w][|]_ _**[x]**_ [)] _[ −]_ _|_ _**y**_ _[β]_
When applying MetaAPO in conjunction with SimPO ( _i.e.,_ MetaAPO-SimPO), this SimPO-specific
preference score (as shown in Eq. A.9) is used in place of the DPO-based preference score within the
MetaAPO framework.
B EXPERIMENTAL SETUP
B.1 MODELS AND DATASET
For the SFT models used in training, we adopt existing open-source models that have been supervised
fine-tuned, specifically: allenai/Llama-3.1-Tulu-3-8B-SFT based on the Llama architecture, and
FuseAI/FuseChat-Qwen-2.5-7B-SFT based on the Qwen architecture. All methods are trained on
UltraFeedback-Binarized dataset, which contains 64K preference data.
**Reward models.** For the external reward model annotation, we trained the reward model for one
epoch on the UltraFeedback-Binarized dataset from the SFT model corresponding to the policy model.
This procedure ensures that any online improvement does not stem from potential leakage of the
reward model, thus providing a fully fair comparison between online and offline methods. We also
report experimental results using other open-source reward models in the Appendix.
**Meta-Learner architecture.** The meta-learner _h_ _**ϕ**_ is a compact two-layer MLP that maps the scalar
offline preference score _ℓ_ [off] ( _·_ ) to a weight _w ∈_ [0 _,_ 1]. It uses a 100-dimensional hidden layer with
Tanh activation, followed by a Sigmoid output. Due to its small size and simple input, the meta-learner
adds negligible overhead in both prediction and training.
B.2 EVALUATION BENCHMARKS
We evaluate our models on three representative alignment benchmarks: AlpacaEval 2(Dubois et al.,
2024), MT-Bench(Zheng et al., 2023), and Arena-Hard (Li et al., 2024). These benchmarks are
widely adopted for their comprehensive coverage of diverse tasks and their effectiveness in assessing
the subjective preference capabilities of LLMs.
 - **AlpacaEval 2** includes 805 instructions from five different datasets and assesses models using
two metrics: length-controlled (LC) win rate and raw win rate (WR). In this benchmark, GPT4o-2024-05-13 serves as the baseline model and GPT-4o-latest as the evaluator for the other
models.
 - **Arena-Hard** is a more challenging benchmark that closely aligns with the human preference
ranking from Chatbot Arena, a crowd-sourced platform for evaluating LLMs. It spans 250
high-quality topic clusters, including 500 well-defined technical problem-solving queries. We
report the Win Rate against GPT-4-0314 using GPT-4o as the judge model.
 - **MT-Bench** contains 80 multi-turn dialogues with 160 questions across eight categories, including
writing, roleplay, reasoning, math, coding, extraction, STEM, and humanities. Each response is
evaluated by GPT-4o on a scale from 1 to 10, with the average score reported for each dialogue
turn across the 80 dialogues. Different from the official setting, we follow the latest works to
adopt GPT-4o as the evaluator and baseline.
B.3 BASELINES.
To evaluate the effectiveness of MetaAPO, we compare it against a range of SOTA baseline methods
from current LLM alignment literature. These baselines include purely offline approaches, methods
that enhance offline data processing, techniques incorporating online data generation, and hybrid
strategies:
Preprint
 - **Offline Preference Alignment** : These methods, such as DPO (Rafailov et al., 2023),
SimPO (Meng et al., 2024), and IPO (Azar et al., 2024), directly learn from a fixed, pre-collected
preference dataset, typically by optimizing a log-likelihood objective.
 - **Selective DPO** (Gao et al., 2025): This approach aims to improve data quality by splitting
the dataset into training and validation sets. It calculates the average loss of all samples on the
validation set and then filters out samples presumed to be low-quality (those with higher validation
loss) to curate a sub-dataset for model training.
 - **Online DPO** (Xu et al., 2024): Online DPO follows an iterative process of data generation and
preference optimization. In each iteration, the current model generates multiple responses for
given prompts. These responses are subsequently evaluated by an external reward model to
identify the best (chosen, _**y**_ _w_ [on][) and worst (rejected,] _**[ y]**_ _l_ [on][) options, forming new preference pairs.]
The policy optimization then exclusively utilizes these self-generated preference pairs. Then the
policy model and reference model are updated and continue next iteration.
 - **ADPO** (Ji et al., 2024): Active DPO (ADPO) heuristically selects offline samples for online
augmentation if their preference loss falls below a fixed threshold. New online preference pairs
are generated for these selected prompts. These newly created online pairs are then used in
conjunction with the original offline samples for preference optimization via a standard DPO
algorithm.
 - **SELM** (Zhang et al., 2024b): Self-Exploring Language Model (SELM) operates with an iterative
generation and annotation cycle similar to Online DPO. However, when ranking the multiple
responses generated by the current policy (using a reward model), SELM distinctively includes
the original chosen _**y**_ _w_ [off] [and rejected] _**[ y]**_ _l_ [on] [responses from the offline dataset (for the same prompt)]
within the ranking pool. This method allows for a more integrated utilization of both offline and
newly generated online data to construct the training preference pairs.
 - **PPO** (Ouyang et al., 2022): Proximal Policy Optimization (PPO) is a foundational online
alignment method from Reinforcement Learning from Human Feedback. Unlike direct preference
methods, PPO first trains a separate reward model on a human preference dataset. Subsequently,
the policy model generates responses to prompts, and the RM scores these responses to provide
a reward signal. PPO then uses this reward to update the policy and critic model, aiming to
maximize the expected reward while a KL-divergence penalty keeps the policy from straying too
far from a reference model.
B.4 MODEL TRAINING
In our main experiments, we integrate MetaAPO into the iterative alignment framework (Ouyang
et al., 2022; Xu et al., 2024; Dong et al., 2024), which consists of two stages: data generation and
preference optimization. To obtain the policy and reference models, we first warm up the SFT model
on the UltraFeedback-Binarized dataset using the DPO loss. The warmed-up model serves as the
initial policy model and the SFT model is used as the reference model for MetaAPO training. For a
fair comparison, all models are trained for **1 epoch** over the dataset and run for 3 alignment iterations.
During each generation phase, we first employ the meta-learner to assign sampling probabilities for
the offline data. Based on these probabilities, a subset of offline preference data is sampled. For
each sampled prompt, the current policy model generates _K_ = 8 candidate responses. An external
reward model is then used to annotate scores for each prompt–response pair, from which we select
the highest and lowest scoring responses as the chosen and rejected samples, respectively.
In the optimization phase, we jointly leverage both the original offline preference pairs and the newly
generated online preference pairs for each augmented instance, forming the dataset _D_ aug. These pairs
are integrated through our meta-weighting mechanism, as defined in the preference optimization
objective (Eq. 5). Specifically, for each instance ( _**x**_ _,_ _**y**_ pair [off] _[,]_ _**[ y]**_ pair [on] [)] _[ ∈D]_ [aug][, the meta-learner computes a]
dynamic weight _w_ = _h_ _**ϕ**_ ( _ℓ_ off( _**x**_ _,_ _**y**_ _w_ [off] _[,]_ _**[ y]**_ _l_ [off][))][, which determines the relative importance of the offline]
component _ℓ_ _**θ**_ ( _**x**_ _,_ _**y**_ _w_ [off] _[,]_ _**[ y]**_ _l_ [off][)][ and the online component] _[ ℓ]_ _**[θ]**_ [(] _**[x]**_ _[,]_ _**[ y]**_ _w_ [on] _[,]_ _**[ y]**_ _l_ [on][)][ in the total loss.]
The policy model _π_ _**θ**_ is then updated for one epoch over the weighted dataset _D_ aug. Concurrently, the
meta-learner _h_ _**ϕ**_ is updated every _T_ meta steps using its dedicated meta-objective (Eq. 6), based on the
data stored in the meta buffer _B_ meta, as described in Algorithm 1.
Preprint
Table 3: Hyperparameters for Llama-3.1-8B and Qwen2.5-7B during generation and training.

Alignment through Meta-Weighted Online Sampling: Bridging the Gap
  between Data Generation and Preference Optimization

Junming Yang, Ning Xu, Biao Liu, Shiqi Qiao & Xin Geng

Junming Yang
Ning Xu
Biao Liu
Shiqi Qiao
Xin Geng


_Generation_
Temperature       - 1 0.8
Top-p      - 1 0.95
Rollout nums K 8
Max new token _L_ new 2048
_Training_
Learning rate _α_ 5e-7
Meta-learner lr _η_ 5e-3
Batch size B 128
Max prompt length _L_ prompt 2048
Max generation length _L_ gen 2048
Training max length _L_ train 4096
Reward model max length _L_ reward 4096
KL loss _β_ 0.1 (2.5 for SimPO)
Target reward margin _γ_ 0.6
Update Interval _T_ meta 8
Epochs      - 1
B.5 HYPERPARAMETERS
We list the key hyperparameters used in our experiments for both Llama-3.1-8B and Qwen2.5-7B
in Table 3. The generation settings ( _e.g.,_ temperature, top-p, and number of responses per prompt)
are adjusted to suit each model’s characteristics, while training-related parameters such as learning
rates, batch size, and sequence length are kept consistent to ensure fair comparison. For MetaAPO,
we adopt a moderate meta-learner update interval and use shared weighting hyperparameters across
all models and baselines.
For PPO training, we set the rollout batch size to 1024 to ensure a small distribution shift. All other
hyperparameters, such as the batch size and learning rate, are kept consistent with those used in the
other methods to enable a fair comparison.
**Evaluation.** We follow the generation protocols specified by each benchmark. The maximum
generation length is set to 4096, while the temperature is determined according to the default
configuration of the respective benchmarks.
C ADDITIONAL EXPERIMENTAL RESULTS AND ANALYSES
In this section, we provide extended experiments and analyses to further validate MetaAPO. (C.1)
examines its impact on general downstream tasks to ensure no degradation of broad capabilities.
(C.2) test robustness under different reward models and datasets. (C.4) analyzes the effect of the
meta-learner update interval and its dynamic behavior during training. Finally, we conduct extended
ablation studies (C.5) and report time efficiency comparisons (C.6).
C.1 PERFORMANCE ON GENERAL TASKS
We evaluated the aligned model’s performance on other tasks involving mathematics, logical reasoning, general knowledge and instruction following. To ensure the reliability of our results, we
followed the standard evaluation setup provided by the LM-Evaluation-Harness, where the number in
parentheses denotes the few-shot examples provided for each task. The results for Llama-3.1-8B are
Preprint
Table 4: We evaluated the performance of the Llama-3.1-8B aligned models on tasks related to
mathematics, logical reasoning, general knowledge, _et al_ . The numbers in parentheses indicate the
few-shot examples given for each task, and the results are presented in the table.
|Models|GSM8K(8) MMLU(5) ARC(25) EQBench(0) TruthfulQA(0) IFEval(0)|Average|
|---|---|---|
|SFT
DPO
Online DPO
SELM
MetaAPO|72.71
64.81
57.59
65.51
46.85
**70.98**
74.22
64.98
59.39
**67.58**
48.62
70.24
76.50
64.93
60.24
63.27
54.40
63.40
76.33
65.03
61.01
63.92
55.21
61.74
**77.18**
**65.15**
**61.43**
67.27
**55.47**
68.21|63.08
64.17
63.79
63.87
**65.79**|
Table 5: Overall performance of our proposed MetaAPO method with Llama-3.1-8B and Qwen2.57B, compared with offline and online baseline methods on AlpacaEval 2, Arena-Hard and MT-Bench
with the reward model OpenRLHF/Llama-3-8b-rm-mixture. The best and second-best results are
highlighted in **bold** and underline.
shown in the table, indicating that MetaAPO’s alignment does not lead to significant performance
degradation on general tasks.
C.2 EXPERIMENTS ON OTHER REWARD MODELS AND DATASETS.
We further evaluate MetaAPO under different reward models to examine its robustness and generalizability. Specifically, we adopt an open-source reward model, OpenRLHF/Llama-3-8b-rm-mixture,
which is trained on preference data disjoint from our alignment datasets. As shown in Table 5,
MetaAPO consistently outperforms both offline and online baseline methods across AlpacaEval
2, Arena-Hard, and MT-Bench. We note that online and hybrid methods inherently depend on the
accuracy and preference biases of the reward model, which can introduce variability in performance.
Nevertheless, under the same reward model, our approach achieves stable and superior improvements,
demonstrating the general applicability and robustness of MetaAPO.
We also evaluated Llama-3.1-8B trained on the Argilla/DPO-Mix-7k dataset, which consistently
demonstrates the stability and effectiveness of MetaAPO across different data sources.
C.3 TUNING STRATEGIES FOR META-LEARNER UPDATE INTERVAL _T_ META
In MetaAPO, we introduce a meta-learner update interval _T_ meta to accumulate experiences in the
meta buffer _B_ meta for more stable and generalizable updates to _h_ _**ϕ**_ . We investigate how varying _T_ meta
affects alignment performance.
As shown in Figure 3 (Left), the choice of _T_ meta significantly impacts performance on AlpacaEval
2 Win Rate (WR) and Length-Controlled win rate (LC). When _T_ meta is small (e.g., 1 or 2), _B_ meta
contains limited training data, leading to unstable or suboptimal updates of _h_ _**ϕ**_ and degraded alignment
performance. As _T_ meta gradually increases (e.g., to 4 and 8), the meta-learner benefits from a more
Preprint
Table 6: Performance comparison of different alignment methods on Argilla/DPO-Mix-7k dataset
with Llama-3.1-8B.
|Model|AlpacaEval2 WR (%) AlpacaEval2 LR (%) Arena-Hard WR (%) Arena-Hard SC (%)|
|---|---|
|SFT
Online DPO
SELM
PPO
MetaAPO|10.02
17.28
18.5
21.6
26.10
25.43
27.3
32.6
27.66
25.91
28.1
31.5
27.12
25.01
28.7
31.5
**28.56**
**27.27**
**30.4**
**34.0**|
|Input (Offline Loss)|Col2|Col3|Col4|Col5|Col6|
|---|---|---|---|---|---|
|~~Input (Offline Loss)~~||||||
|~~Input (Offline Loss)~~||||||
|~~Input (Offline Loss)~~||||||
|~~Input (Offline Loss)~~||||||
|~~Input (Offline Loss)~~||||||
|~~Input (Offline Loss)~~||||||
|~~Input (Offline Loss)~~||||||
|~~Input (Offline Loss)~~||||||
|~~Input (Offline Loss)~~||~~Input (O~~|~~ fline Loss)~~|~~ fline Loss)~~||
Figure 3: **Left** : Impact of meta-Learner update interval ( _T_ meta) on MetaAPO performance. Performance (AlpacaEval 2 WR/LC (%)) is shown for different _T_ meta values. **Right** : Meta-Learner
input-output relationship across training iterations.
comprehensive set of recent experiences stored in the buffer. This richer information allows for
more stable and generalizable updates to _h_ _**ϕ**_, leading to a consistent improvement in alignment
performance. The results indicate that performance tends to plateau or stabilize for _T_ meta _≥_ 8.
Balancing responsiveness and stability, we set _T_ meta = 8 in our main experiments. This observation
is consistent with Theorem 1, which guarantees improved generalization when the meta-learner is
updated with a sufficiently rich set of meta-samples.
C.4 FURTHER ANALYSIS OF META-LEARNER BEHAVIOR
To understand how the meta-learner _h_ _**ϕ**_ adapts during training, we visualize its input-output relationship over time (Figure 3, Right). The input is the offline preference score _ℓ_ off( _·_ ), and the output is the
weight _w_, which determines the likelihood of selecting a sample for online generation (lower _w_ →
higher sampling probability). At initialization (gray curve), the output resembles a scaled sigmoid:
highly negative scores (indicating misalignment) receive low weights, encouraging augmentation.
Positive scores receive moderate weights (about 0.5), allowing some exploration of well-aligned
samples. After the first iteration (blue dashed), the meta-learner becomes more selective: weights
for moderately negative and near-zero scores drop further, increasing focus on less-aligned samples.
Weights for positive scores start to decrease, reducing redundant augmentation. This trend continues
(red and green curves): weights for well-aligned samples steadily decline, while those for poorly
aligned ones remain low. By iteration 3, the function stabilizes, showing convergence to an effective
sampling policy. Overall, the meta-learner dynamically adjusts its strategy based on policy alignment,
focusing augmentation where it is most needed and avoiding overfitting to already aligned data.
C.5 EXTENDED ABLATION STUDY
We conduct additional ablation studies to further examine two key components of MetaAPO: (i) the
effect of its selective utilization strategy for offline samples, and (ii) the influence of the meta-learner’s
network complexity along with the incorporation of richer preference information. All experiments
Preprint
Table 7: Further ablation study on key components of MetaAPO. We assess the impact of the
offline samples and the influence of the meta-learner’s network and input features. Metrics include
AlpacaEval2, Arena-Hard, MT-Bench and the proportion of annotations used during training.
|Model|AlpacaEval 2 Arena-Hard MT-Bench
WR(%) ↑ LC(%) ↑ WR(%) ↑ SC(%) ↑ Score ↑|Annotations
Ratio(%) ↓|
|---|---|---|
|**MetaAPO**
_w/_ All Offine Samples
_w/_ 5-layers Meta-Learner
_w/_ Multi-Feature Input|50.19
**50.37**
**55.2**
**43.6**
**7.57**
**50.32**
38.40
50.8
40.1
7.23
47.28
45.62
46.0
39.7
7.41
49.33
49.17
53.1
41.3
7.56|**59.4**
62.7
63.0
61.4|
are carried out under the Llama-3.1-8B setting with the DPO objective, using the reward model
OpenRLHF/Llama-3-8b-rm-mixture. The results are summarized in Table 7.
**Impact of Utilizing All Offline Data.** In the standard MetaAPO setup, the augmented dataset _D_ aug
comprises tuples formed by pairing selected offline preference samples with their corresponding
online counterparts. Only those offline samples identified by the meta-learner as beneficial for
augmentation are included. The remaining offline samples, those not selected for online augmentation,
are excluded from the policy training phase.
To evaluate the effect of this selective inclusion, we tested a variant (denoted w/ _All Offline Samples_ ) in
which the training set was expanded to include all original offline samples. In this setting, augmented
pairs in _D_ aug are still weighted by _w_ and 1 _−_ _w_ as usual, while the additional unaugmented offline
samples are incorporated with a fixed weight of 1. As shown in Table 7, this broader inclusion leads
to a notable drop in overall performance compared to standard MetaAPO. The performance degrades
on AlpacaEval 2 LC, Arena-Hard WR and MT-Bench score. These results suggest that MetaAPO’s
selective augmentation strategy is more effective, likely because the policy is already sufficiently
trained on the offline data via DPO warm up. Consequently, the meta-learner can effectively identify
samples that are either well-aligned or less informative, thus excluding them from redundant retraining.
Including these may dilute the effect of the more informative online augmentations or risk overfitting
to noise and less impactful preferences.
**The Sufficiency of Meta-Learner.** To further investigate the capacity of the two-layer MLP, we
experimented with a deeper 5-layer meta-learner. As shown in Table 7, the two-layer network is
sufficient to map scalar preference scores to effective weights. Given the simplicity of the input
and output, increasing model complexity introduces a higher risk of overfitting without noticeable
performance gains. This observation is consistent with Theorem 1, which guarantees improved
generalization when the meta-learner with low complexity.
In addition, we investigate a more flexible input setting, where the meta-learner is provided with
_ℓ_ [off] ( _·_ ), log _π_ _[π]_ ref _**[θ]**_ [(] ( _**[y]**_ _**y**_ _[w]_ _w_ _[|]_ _|_ _**[x]**_ _**x**_ [)] ) [, and][ log] _π_ _[π]_ ref _**[θ]**_ [(] ( _**[y]**_ _**y**_ _[l]_ _l_ _[|]_ _|_ _**[x]**_ _**x**_ [)] ) [as inputs, denoted as] _[ Multi-Feature Input]_ [. As shown in Table 7,]
incorporating these richer inputs does not lead to further improvements. A possible explanation
is that, as discussed in Section 5.3, there exists a clear mapping between the offline score and the
potential gain. The single offline score input already captures this relationship, whereas introducing
additional inputs may require a more complex learning mechanism to effectively exploit the mapping.
C.6 TIME CONSUMPTION
Despite introducing an additional meta-learner module, MetaAPO remains highly efficient thanks to
its simple structure and scalar input, making both its training and inference extremely lightweight.
As shown in Table 8, we compare MetaAPO’s time cost against strong online baselines: Online
DPO and SELM. In the online sampling phase, MetaAPO completes in just 106 minutes on 8 _×_ H100
GPUs, more than 3 _×_ faster than Online DPO (337m) and SELM (363m). This substantial speedup
stems from MetaAPO’s adaptive sampling, which selectively augments only a subset of offline
samples—drastically reducing the number of costly response generations and reward model queries.
In the training phase, despite the added meta-learner and joint optimization over both offline and
online data, MetaAPO’s training time (59m) remains comparable to Online DPO (58m) and SELM
Preprint
Table 8: Comparison of time consumption in minutes for the sampling phase, training phase and
overall between MetaAPO, Online DPO and SELM.

Hyperparameters Notation Llama-3.1-8B Qwen2.5-7B


Online DPO 337 **58** 395 100
SELM 363 60 423 125
PPO  -  - 936 100
**MetaAPO** **106** 59 **186** 58
(60m), since fewer preference pairs are used. Overall, MetaAPO completes the entire process in just
186 minutes, less than half the time required by Online DPO (395m) and SELM (423m). Compared
to PPO, MetaAPO reduces the overall training time by **80.1%**, primarily due to eliminating the
costly sampling and annotation process as well as the additional critic model training. This highlights
MetaAPO’s scalability and efficiency in achieving high-quality alignment.
D CASE STUDY
We further demonstrate the effectiveness of MetaAPO through qualitative examples from AlpacaEval
2, a challenging benchmark for subjective instruction-following evaluation. We compare responses
generated by MetaAPO with those from WPO, Online DPO, and SELM. As shown in Figure 4,
when answering a concise factual question, WPO tends to over-provide information in a lengthy and
rigid manner. In contrast, MetaAPO delivers a more concise and direct response, better aligning
with human conversational preferences. In Figure 5, SELM fails to identify and explain all the
errors in the input transcription. MetaAPO, however, demonstrates more comprehensive and helpful
error correction. Lastly, in Figure 6, MetaAPO produces a more natural, engaging, and contextaware response than Online DPO, showcasing stronger interactivity and scenario alignment. Overall,
these examples illustrate that MetaAPO is capable of generating more helpful, user-friendly, and
human-aligned responses, highlighting its effectiveness in instruction alignment tasks.
Preprint
Figure 4: A concise alignment example.
Preprint
Figure 5: A comprehensive alignment example.
Figure 6: An engaging alignment example.

Method Sampling Time(m) Training(m) Overall(m) Annotation Ratio(%)