Advances in natural language processing (NLP)
have enabled large language models (LLMs) like
*Corresponding author: _jcorbeil@microsoft.com_
†Other affiliations: Cancer Research Center Cologne Essen
(CCCE), German Cancer Consortium (DKTK, Partner site Essen) and Department of Physics of TU Dortmund (Dortmund,
Germany).
Figure 1: Our approach in two steps: 1) continual pretraining; 2) alignment. 1) Starting from _Phi3.5 mini_ :
1.a) We leverage _knowledge acquisition methods_ such
as pre-instruction tuning on diverse medical and clinical corpora to obtain domain-specific experts. 1.b) We
recur to _model merging_ to merge experts first with the
base model to combine new knowledge as well as recover skills degraded by the previous step, then together
to form an unified model, MediPhi. 2) We generate
MediFlow, a synthetic instruction dataset MediFlow for
clinical tasks. We align our model on MediFlow using
Supervised Fine-Tuning (SFT) and Direct Preference
Optimization (DPO) and obtain MediPhi-Instruct. _Seg-_
_mentation of model parameters as equal block size are_
_only for illustrative purposes_ .
GPT-4 to excel in medical tasks, especially medical exams (Nori et al., 2023; Ben Abacha et al.,
2024; Nori et al., 2024). However, their deployment in clinical settings [1] faces many challenges,
1We define the medical field as encompassing medical
knowledge (e.g. anatomy, genetics, biology), while the clinical
including high latency and cost (Yang et al., 2023;
Dennstädt et al., 2025). As LLM progress faces
diminishing returns from scaling (Udandarao et al.,
2024; Longpre et al., 2024; Muennighoff et al.,
2023; Villalobos et al., 2022), specialized small
language models (SLMs) provide a viable alternative (Sardana et al., 2024), when optimized for
domain-specific performance, lower computational
requirements, and real-world clinical integration.
Developing high-quality clinical language models is hindered by the unavailability of clinical data
which are sensitive and tightly licensed, e.g. protected health information under HIPAA. Current
medical LLMs perform well on multiple-choice
question datasets but struggle with real-world clinical complexities (Dada et al., 2024; Chen et al.,
2024; Liu et al., 2024; Jeong et al., 2024a,b). Furthermore, both the inaccessibility of clinical data
and the misalignment of current continual pretraining methods for clinical tasks are critical limitations in the context of SLMs, which have limited
capacity. Addressing these gaps require innovative
training strategies tailored to small models.
This work introduces a modular framework for
building high-performance medical SLMs, leveraging pre-instruction tuning (PIT) (Jiang et al., 2024),
model merging, and clinical alignment. Using
pre-instruction tuning, we adapt _Phi3.5 mini_ with
3.8B parameters (Abdin et al., 2024a) into experts
trained on diverse medical and clinical corpora. We
unify these models through model merging into one
SLM which preserves benchmark improvements.
We complete training by aligning the model with
MediFlow, a new synthetic instruction dataset on
clinical tasks. A representation of this approach is
given in Figure 1.
Our contributions include:
  - Introducing the **MediPhi family**, the first collection of high-performance SLMs for medical and clinical applications under a commercially permissive license [2] . The collection
includes both a generalist expert model and
specialized variants. Additionally, we release
synthetic validation sets designed to guide the
model merging algorithm, facilitating reproducibility, and enabling future integration of
other clinical expert models.
field is related to direct patient care and healthcare practice
(e.g. doctor-patient dialog, discharge summary, and clinical
note).
2TBD
  - Releasing of **MediFlow collection** of 2.5 million high-quality synthetic instructions generated with a GPT-4o based agentic pipeline,
also under a commercially permissive license [2] .
This dataset for the clinical domain contains
14 tasks categories, 98 fine-grained input documents, 6 difficulty levels, and 2 output formats (JSON or plain text), filling a gap in the
clinical NLP resources.
  - An extension of CLUE to the **CLUE+ bench-**
**mark** by doubling its size from 6 to 12
datasets including complementary clinical
tasks and input documents, allowing a comprehensive evaluation of medical and clinical
capabilities of language models (e.g. radiology reports, medications, medical error detection, doctor-patient dialog summarization,
information extraction of social determinants
of health, and medical coding).
  - A demonstration of the **effectiveness of pre-**
**instruction tuning** for medical domain adaptation extending the method beyond questionanswering with named entity recognition, relation extraction, and summarization.
 - A **study on ICD10CM medical coding**, in
terms of domain adaptation and benchmarking of medical models, with relative improvements up to 44% over the base model, surpassing GPT-4-0125 by 14%.


Researchers have developed various open-weight
LLMs with diverse capabilities and research licenses in the medical NLP domain. Examples include ClinicalCamel (Toma et al., 2023), Med42
(Christophe et al., 2023), PMC-Llama (Wu et al.,
2024), BioMedGPT (Zhang et al., 2024), Meditron
(Chen et al., 2023), BioMistral (Labrak et al., 2024)
and Asclepius (Kweon et al., 2024). Most recent
medical LLMs (Chen et al., 2023; Christophe et al.,
2024a; Gururajan et al., 2024; Ankit Pal, 2024) are
based on Llama3 (Dubey et al., 2024) — 8B and
70B parameters. Google also trained their own
medical LLM named Med-PaLM 2 (Singhal et al.,
2023) with 340B parameters, which is not publicly
available.

A Modular Approach for Clinical SLMs Driven by Synthetic Data with
  Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignment

Jean-Philippe Corbeil
Amin Dada
Jean-Michel Attendu
Asma Ben Abacha
Alessandro Sordoni
Lucas Caccia
François Beaulieu
Thomas Lin
Jens Kleesiek
Paul Vozila


Recently, authors have released instruction-tuned
models for the medical domain: Aloe (Gururajan
et al., 2024), Hippocrates (Acikgoz et al., 2024),
and Med42 v2 (Christophe et al., 2024a,b). The
alignment phase of these three models based on
only supervised fine-tuning includes similar instruction datasets such as medical question-answering
databases, non-medical alignment data (e.g. UltraChat by Ding et al.), and benchmark training
sets — e.g. MedQA (Jin et al., 2021) and PubMedQA (Jin et al., 2019). While this mix of data
has contributed to improvements, it also introduces
imbalances in task and document coverage, as well
as in-distribution evaluations in some cases, i.e.
closer to evaluation in fine-tuning setting instead
of zero-shot/few-shot setting.


Phi 1 & 2 (Gunasekar et al., 2023; Li et al.,
2023), with 1.3B and 2.7B parameters, respectively,
showed strong reasoning performance using synthetic textbook-like data. Phi-3 and Phi-3.5 mini
(Abdin et al., 2024a) scaled model size to 3.8B
parameters and topic coverage. Phi4 (Abdin et al.,
2024b) at 14B parameters is trained using an iterative data generation process. Similarly, Orca
(Mukherjee et al., 2023; Mitra et al., 2024) focused on reasoning data, while UltraChat (Ding
et al., 2023) targeted multi-turn prompts. Zhang
et al. (2023b) generated 52,000 medical questionanswering instructions based on forms filled by experts. In the clinical domain, Kweon et al. (2024)
generated 158k short question-answering instructions for 8 tasks with synthetic clinical documents
as inputs, seeded from the PMC-Patients corpus
(Zhao et al., 2023). MagPie (Xu et al., 2024) proposed the generation of synthetic instructions in
general domains using LLMs’ chat template.


Model Soup (Wortsman et al., 2022) demonstrated
improvements over single checkpoint evaluation
by averaging training checkpoints, leading to methods like spherical linear interpolation (SLERP),
enabled by linear-mode connectivity (Frankle et al.,
2020; Mirzadeh et al.). To extend to multiple models, authors have proposed _task arithmetic_ (Ilharco
et al.) while techniques such as TIES (Yadav et al.,
2024a), DARE (Yu et al., 2024), and BreadCrumbs
(Davari and Belilovsky, 2024) address the param
eter interference issue. Hammoud et al. (2024)
highlighted the importance of validation sets for
an optimal merge. Large-scale experiments (Yadav
et al., 2024b) showed the effectiveness of merging
multiple experts, especially from large instruction
models. Merging has also proven as effective as
data-mix strategies during pre-training (Ahmadian
et al., 2024; Na et al., 2024).


Medical LLMs are commonly benchmarked
on medical knowledge through multiple-choice
datasets such as MedQA (Jin et al., 2021), MedMCQA (Pal et al., 2022), PubMedQA (Jin et al.,
2019), and MMLU-medical (Hendrycks et al.,
2020). However, recent studies (Liu et al., 2024;
Dada et al., 2024; Chen et al., 2024; Jeong et al.,
2024a,b) revealed a gap on the clinical domain
from medical LLMs.


Our method for clinical SLMs as illustrated in Figure 1 includes two steps: 1) continual pre-training
composed of 1.a) domain knowledge acquisition
methods and 1.b) model merging, and 2) posttraining with supervised fine-tuning and direct preference optimization on generated synthetic data.


In Table 1, we list the medical and clinical corpora
with permissive licenses, used to adapt our SLM
into five experts. We separate these into five groups:
_PubMed_, _Clinical_, _MedCode_, _Guidelines_, and _Med-_
_Wiki_ . We describe these dataset groups with their
licensing status in detail in Appendix A.1.
Table 1: Medical and clinical data sources separated into
five groups: _PubMed_, _Clinical_, _MedCode_, _Guidelines_,
and _MedWiki_ .


PMC Abstract Scientific Abstracts 36M 6B
PMC-Patients Patient summaries 156k 130M
Asclepius Clinical Documents 80k 44M


NoteChat Conversations 80k 72M
MTSamples Clinical Documents 5k 4M
**MedCode** ICD9/10&ATC Webpages 206k 257M
**Guidelines** Guidelines Websites 37k 90M
**MedWiki** MedWiki Encyclopedia 80k 80M


We consider three methods to enhance domain
knowledge of language models: domain adaptation
pre-training (DAPT), textbook-like synthetic material (Explainer), and pre-instruction tuning (PIT).
The latter showed considerable improvements in
our experiments. We apply these techniques on the
base model to obtain five experts in the medical
and clinical domain from our five dataset groups.
**DAPT** (Ganin and Lempitsky, 2015; Gururangan
et al., 2020) Domain Adaptation Pre-Training is a
technique for adapting a deep learning model to a
specific domain by performing next-token prediction on domain-specific corpora. The expert that
is trained on the _PubMed_ group follows standard
DAPT, because it is orders of magnitude larger than
the others. We also train on all the data a _DataMix_
baseline using DAPT, which is outperformed by
our approach. For this method, we trained models
for one epoch with a cosine scheduler at a maximum learning rate of 5e-5.
**Explainer** (Gunasekar et al., 2023) Textbooklike material demonstrated the effectiveness of
training SLMs on textbook-quality data generated by a strong LLM. Therefore, we generate
a textbook-like "explainer" using GPT-4o for the
_MedCode_ dataset group, for which the dense format of the webpages impedes model learning. For
this method, we trained models for two epochs with
a cosine scheduler at a maximum learning rate of
1e-4.
**PIT** (Jiang et al., 2024) Pre-Instruction Tuning [3]
(PIT) demonstrated significant improvements over
the conventional training paradigm, by first finetuning on instruction-like data, followed by training
on a concatenation of the instruction data and pretraining corpus.
PIT requires the generation of task data for each
document in the corpus. While Jiang et al. (2024)
originally used question-answering (QA) as the
primary task, we expand the approach to include
summarization, named entity recognition, and relation extraction. We use GPT-4o to generate outputs
for all four tasks on the ICD10CM subset of the
MedCode dataset as an initial case study. Based
on the results, we extend this process to the remaining four dataset groups — Clinical, MedCode,
3Despite its name, PIT does not involve explicit instruction
data or chat-style formatting.
Guidelines, and MedWiki — to train multiple expert models.
Each expert undergoes sequential training in two
phases. The first phase involves fine-tuning on the
generated outputs from a single task for one epoch
using a cosine scheduler (peak learning rate at 1e4). If the task contains multiple elements, such
as several question-answer pairs, they are concatenated into a single sequence, separated by end-ofsentence (EOS) tokens. In the second phase, the
model is fine-tuned for two epochs with another
cosine scheduler (peak learning rate at 3e-4) on
the concatenation of the task data and the original
documents, with EOS tokens acting as separators.
Figure 2: Overview of model merging techniques. **a.**
_Domain-specific merging via SLERP_ : An expert model
is obtained by fine-tuning the base model on domainspecific data (step i). The expert and base models
are then merged using spherical linear interpolation
(SLERP) to produce the final merged model (step ii).
**b.** _Multi-expert merging_ : Multiple expert models are
independently derived from the base model via domainspecific pretraining (step i). These are then combined
using a merging operator (e.g., Task Arithmetic, Ties,
or BreadCrumbs) to produce a unified merged model
(step ii).


We train five expert models specializing in different
aspects of medical and clinical knowledge, each
derived from a distinct dataset group: _PubMed_,
_Clinical_, _MedCode_, _Guidelines_, and _MedWiki_ .
Following the success of BioMistral (Labrak
et al., 2024), the first approach involves merging
each expert model individually with the base model
using SLERP [4] as in step _a_ of Figure 2. We determined the merging proportions (10%, 25% or 50%
) via validation sets (see below). We apply merging after PIT since these techniques demonstrate
a synergistic effect. While PIT enhances domainspecific learning, it also leads to catastrophic forgetting — degrading the model’s initial abilities such
as instruction following, long context handling, and
multilingual support (Scialom et al., 2022). This
issue is particularly evident in zero-shot and fewshot settings, where instruction-tuned models generally outperform their base counterparts (Longpre
et al., 2023; Zhang et al., 2023a). Merging with the
original instruction model after PIT mitigates these
degradations, preserving general capabilities while
maximizing domain adaptation [5] .


Our second set of experiments combines all five experts into a unified SLM forming _MediPhi_ as in
step _b_ of Figure 2. Multi-model merging involves
three primary techniques: Task Arithmetic, TIES,
and BreadCrumbs. Given the vast configuration
space of multi-model merges, we employ an evolutionary algorithm via MergeKit (Goddard et al.,
2024) to optimize the merging process.
However, optimization on our benchmark is not
feasible due to a lack of validation data and framework incompatibilities. To address this, we generate synthetic validation sets aligned with our benchmark tasks. Specifically, we prompt GPT-4o to create multiple-choice question sets covering 12 medical and clinical topics relevant to our benchmark
(e.g., doctor-patient interactions, medical coding,
discharge summaries). These sets maintain contextual consistency with our evaluation tasks. The
evolutionary algorithm is set to terminate after 500
evaluations, guided by average accuracy on these
validation sets. Further details about validation set
creation procedures are provided in Appendix A.3.


To identify the top-performing model, we primarily
measure the _average accuracy_ on CLUE+. How
4A method optimized for merging two models, often yielding high-performing hybrids (Hammoud et al., 2024; Ahmadian et al., 2024; Labrak et al., 2024).
5Alternative approaches, such as KL-divergence regularization in RLHF (Ouyang et al., 2022), exist for maintaining
generalization, but recent studies suggest model merging can
further optimize reward alignment in RL settings (Ramé et al.,
2024).
ever, we also consider important that the expert
achieves uniform improvements, especially among
experts with similar accuracies. For this purpose,
we use # _DG_, the number of datasets on which the
model achieves gains, and CV ∆, the _coefficient of_
_variations of gains/losses_ as defined in equation 1.
Figure 3: Schema of MediFlow generation. i) Given
a randomly sampled set of parameter values, we
prompt _GPT-4o_ for _N_ instructions to obtain triplets
( _instruction, inputs, outputs_ ) in which we have _P_
pairs of inputs ( _I_ ) and outputs ( _O_ ) each. ii) We prompt
_GPT-4o mini_ with LLM-as-a-Judge and self-consistency
on _M_ samples. iii) We define a heuristic to filter a diverse, high-quality subset for alignment purposes.
**Generation of MediFlow** In Figure 3, we illustrate our agentic pipeline to generate the MediFlow
dataset.
At step _i)_, we prompt _GPT-4o_ with a
meta-prompt to generate several triplets
( _instruction, inputs, outputs_ ) at once conditioned on five parameters: input-data type, task
type, difficulty level, output format, and temperature. For this step, we also apply temperatures
_CV_ ∆=
~~�~~
~~�~~
E _d∼D_ ( _δd −_ _µd_ ) [2] ~~[�]~~
(1)
_|µd|_
where _δd_ is the expert accuracy minus the baseline one for the _d_ _[th]_ dataset in the benchmark _D_
and _µd_ = E _d∼D_ [ _δd_ ]. A small CV ∆ indicate uniform gains or losses across datasets, while a high
value indicates gains or losses on a narrow subset
of datasets.


of 1.0 (70% of the dataset) or 1.25 (30% of the
dataset), to strike a balance between accuracy or
diversity, respectively. Specifically, we request 10
instructions at a time with four input-output pairs
each.
In step _ii)_, we prompt _GPT-4o mini_ with a LLMas-a-Judge approach — using self-consistency with
chain-of-thought across _M_ = 5 samples at temperature of 1.0 — to provide a critical assessment of
the synthetic instruction. We provide five criteria
in the prompt on a scale of 1 to 4: quality, clarity,
alignment, realism, and difficulty. We compute the
final score _Sj ∈_ [1 _,_ 4] of the _j_ _[th]_ criterion by summing individual scores _sij ∈{_ 1 _,_ 2 _,_ 3 _,_ 4 _}_ multiplied
by their respective counts _ci ∈_ [0 _, M_ ] (constrained
by [�] _i_ _[M]_ =1 _[c][i]_ [ =] _[ M]_ [) as in] _[ S][j]_ [ =] _M_ 1 - _Mi_ =1 _[c][i][ ·][ s][ij]_ [.]
At the final step _iii)_, we use a heuristic to trim
the collection down to its top- _K_ highest quality
samples based on the quality criteria. We added
details of the process in Appendix A.5.
**Generation of MediFlow-DPO** From MediFlow,
we filter the 130k top-quality samples stratified
by task type, input-data type and output format to
further align the SLM with direct preference optimization (DPO) after SFT. We generate marginally
wrong outputs (i.e. rejected outputs) prompting
GPT-4o as an error inducer. We provide the detail
in Appendix A.7.


We train _MediPhi_, the multi-expert merged
SLM, with supervised fine-tuning (SFT) on MediFlow to obtain _MediPhi_ - _SFT_ . We then
align _MediPhi_ - _SFT_ with DPO which leads to
_MediPhi_ - _Instruct_ (SFT + DPO). We provide the
hyperparameters for both settings in Tables 7 and
8 of Appendix A.2.


The CLUE Benchmark (Dada et al., 2024) covers six datasets: MedNLI (Romanov and Shivade, 2018), MeQSum (Ben Abacha and DemnerFushman, 2019), Problem List Summarization
(Gao et al., 2023), LongHealth (Adams et al., 2024),
MeDiSumQA (Dada et al., 2025), and MeDiSumCode. For these datasets, we implement the
same configuration (i.e. prompts, metrics, and
few shots) as Dada et al. (2024). While CLUE
focused on six tasks using clinical notes and discharge summaries as input data, we extend it with
additional clinical input documents (e.g., radiology
reports, doctor-patient dialog) and tasks (e.g. information extraction, medical error detection) to
obtain a broader assessment of clinical abilities. We
introduce the CLUE+ benchmark, with six additional datasets: MedicationQA (Ben Abacha et al.,
2019), MEDIQA-RRS QA (Ben Abacha et al.,
2021), MEDEC (Ben Abacha et al., 2024), ACIBench (Yim et al., 2023), Social Determinant of
Health (Lybarger et al., 2023), and MedConceptsQA ICD10CM (Shoham and Rappoport, 2024).
We provide further information in the Appendix
A.8 in Tables 9 and 10.


To assess the efficiency of different continual pretraining methods, we focus on ICD10CM medical
coding webpages from the _MedCode_ dataset. After
continual pre-training on this subset, we test the
models on ICD10CM questions from the MedConceptsQA dataset (Shoham and Rappoport, 2024).
We left out the easy and medium difficulty questions to focus on the most challenging questions.


In Figure 4, we plot the accuracy of different continual pre-training methods DAPT, Explainer and PIT,
as described in Sec. 3.1.2. First, we note that DAPT
dimishes the performance to random ( _Webpage_ ).
We hypothesize that the webpages have peculiar
implicit format. One solution is to reformulate
them as textbook-like explainers. The model finetuned on explainers is improving upon the baseline
by 6% ( _Explainers_ ). We see that using PIT by
generating summaries ( _Summary_ ) increases performance further by 8% percent.


In Figure 5, we experiment with merging back into
the base model after fine-tuning on the explainers
or training with PIT.
We observe significant improvements across
tasks for SLERP-merged models, with the Summary model performing best. Although question answering ( _QA_ ), named entity recognition
( _Entities_ ), and relation extraction ( _Relations_ )
initially decline compared to the base model before merging, SLERP not only boosts the explainer
model to 58% (a 28% relative improvement over
the base model) but also enhances all PIT-trained
models further—reaching 62% (38% relative) for
QA and 65% (44% relative) for Summary, surpass
|71 nual Pre-Training Methods DAPT Textbook-like material P|Col2|Col3|Col4|Col5|
|---|---|---|---|---|
|~~0.57~~
|~~0.57~~
|~~0.57~~
|~~0.57~~
|~~0.57~~
|
|~~0.45~~
0.|~~0.45~~
0.|51

~~0.53~~
~~0.47~~
~~0.~~|51

~~0.53~~
~~0.47~~
~~0.~~|51

~~0.53~~
~~0.47~~
~~0.~~|
|||~~0.43~~|||
|~~0.25~~|~~0.25~~||||
||||||
||||||
MeDiSumCode and MedConceptsQA, see Tables
11 and 12 in Appendix A.8 for detailed benchmark).
The highest enhancement comes from _MedWiki_ on
10 out of 12 datasets, with an average improvement
of 3.2% (8.8% relative). One possible explanation
is that the model is learning better on educational
contents — such as encyclopedic material — with
vast coverage of medical concepts. We observe
that removing PIT and/or merging on the _Guide-_
_line_ group lead to the worst outcomes below the
baselines.
Table 2: Average performances on CLUE+ of SLERPmerged experts obtained with PIT on the five dataset
groups. We provide _DataMix_ as a baseline SLERP
expert trained on all dataset groups. We also provide an
ablation study on _Guideline_ .We indicate gains and losses
over the base model. #DG stands for datasets with gains (out
of 12 datasets). CV ∆ stands for coefficient of variations of
gains/losses.
**AVG** _↑_ **#DG** _↑_ **CV** ∆ _↓_
**Baselines** _Phi-3.5 mini_ 36.5  -
_DataMix_ 37.5 **10** **1.2**
_PubMed_ 37.7 9 1.8
_Clinical_ 39.6 **10** 2.0
_MedWiki_ **39.7** **10** 1.5
_MedCode_ 36.7 5 39.6
_Guideline_ 39.2 **10** 1.8
w/o SLERP 27.2 4 1.0
w/o PIT 33.0 6 0.9
w/o SLERP&PIT 25.2 1 2.8
|.57 0.51|0.58 0|0.62 0.53 0.65|0|0.63|0|0.64|
|---|---|---|---|---|---|---|
|~~0.45~~
0.51|0.43|0.53|~~0.47~~|~~0.~~|~~45~~||
||||||||
||||||||
||||||||
Figure 5: Impact of SLERP merging on the performance of ICD10CM. Merging back with the base model
(SLERP 50%) systematically results in gains and these
are more pronounced for PIT.
ing GPT-4 by 8% (14% relative). Based on these
results, the rest of the paper will apply PIT with
summaries, unless otherwise stated.


We present the results on the CLUE+ benchmark
of the five medical and clinical experts, adapted on
the dataset groups and merged back with the base
model with SLERP in Table 2. While the _DataMix_
merged model trained on all the corpora (i.e. similar to BioMistral) achieves gains over the base
model, the individual experts realize further gains,
except for _MedCode_ . The latter improves over the
base models but mostly only on coding datasets (i.e.
We present the results on CLUE+ of combining all experts using three multi-model merging
techniques (Task-Arithmetic, Ties-merging, and
BreadCrumbs) in Table 3. While Task-Arithmetic
yields the highest score of 39.4 with gains on 9 out
of 12 benchmark datasets, the models from Ties
and BreadCrumbs remain competitive, demonstrating more uniform gains across the benchmark, as
indicated by their lower CV ∆. Since the goal
is to select the most robust model for the alignment phase, we find that the _BreadCrumbs_ expert
offers the best trade-off between high-amplitude
improvements and consistent gains on 11 datasets.
We also see that all unified SLMs are above the
average expectation of the SLERP performances
by 0.9%, but below the maximum values and be


Guideline
Ablations
Table 3: Average performance on CLUE+ of unifying
experts into one SLM. We leverage Task-Arithmetic
(TA), TIES-merging and Breadcrumbs (BC) for merging 5 experts: _PubMed_, _Clinical_, _MedWiki_, _MedCode_
and _Guideline_ . We provide statistics across dataset performances of the SLERP experts, translating into worst
case (minimum), average, and best case (maximum).We
indicate gains and losses over the baseline. #DG stands for
datasets with gains (out of 12 datasets). CV ∆ stands for
coefficient of variations of gains/losses.
**AVG** _↑_ **#DG** _↑_ **CV** ∆ _↓_
**Baseline** _Phi-3.5 mini_ 36.5  -
Minimum 34.5 4 1.9
SLERP
Experts


Average 38.5 7 2.1
Maximum 43.1 11 1.1
_DataMix_ 37.5 10 **1.2**
Task-Arithmetic **39.4** 9 1.9
Ties 39.3 7 1.7
BreadCrumbs 39.3 **11** 1.5
low the top-performing _MedWiki_ expert by 0.3%.
Yet, this expert improves only on 10 datasets, one
less than the BreadCrumbs-merged expert. We also
observe for _BreadCrumbs_ a stronger improvement
over _MedWiki_ of 10.6% on ICD10CM medical coding (see Table 12 in Appendix A.8), which is still
lower than the specialized _MedCode_ expert with an
improvement of 36.9% over the _MedWiki_ expert.
We select the _BreadCrumbs_ expert as _MediPhi_
for its strong, balanced average score on CLUE+.


Table 4: Average performance on CLUE+ of aligning Mediphi with MediFlow using SFT followed by
DPO.We indicate gains and losses over the baseline. #DG
stands for datasets with gains (out of 12 datasets). CV ∆
stands for coefficient of variations of gains/losses.
**AVG** _↑_ **#DG** _↑_ **CV** ∆ _↓_
We show the results of aligning SLMs with the
Mediflow dataset, as well as MediFlow-DPO in
Table 4. To begin, we note that the alignment of
MediPhi on all instructions (2.5M) leads to an accuracy of 41.9, an improvement of 5.4% over Phi3.5.
By filtering MediFlow for top-quality 800k, we
push the gain further by 6.5% to 43.0%. Then,
we apply DPO using MediFlow-DPO to realize an
improvement of 6.9% (18.9% relative), of which
the maximum average ends up at 43.4%. Notable
gains in relative are: medical entities (SDoH) with
64.3% and on radiology reports (RRS QA) by
49.5% (see Table 12). If we align Phi3.5 on MediFlow using SFT, we reach an accuracy of 42.2%.
While surpassing _MediPhi_ - _SFT_ 2.5M by 0.3%,
_MediPhi_ - _SFT_ 800k surpasses it by a margin of
2% (relative). Despite achieving gains overall, we
observe a diminution in # _DG_ from 11 to 9 for
all models. The lower-performing datasets for all
aligned models are Problem List Summarization
and MediSumCode, which we hypothesize this result from a specific bias in MediFlow towards listing tasks like extracting problems from clinical
notes or medical codes from discharge summaries
(see Tables 11 and 12 in Appendix A.8).
Table 5: Performances on CLUE+ of other medical
LLMs compared with MediPhi models.We indicate gains
and losses. #DG stands for datasets with gains (out of
12 datasets). CV ∆ stands for coefficient of variations of
gains/losses.
**Model** **AVG** _↑_ **#DG** _↑_ **CV** ∆ _↓_
Mistral-7B-Instruct-v0.1 33.6 -
BioMistral-7B-DARE 34.7 (+1.1) 8 3.4
Phi-3.5 mini (3.8B) 36.5 -
MediPhi (3.8B) 39.3 (+2.8) **11** 1.5
MediPhi-SFT (3.8B) 43.0 (+6.5) 9 **1.4**
MediPhi-Instruct (3.8B) **43.4** (+6.9) 9 **1.4**
Meta-Llama-3-8B-Instruct 44.1 -
Llama3-Med42-8B* 45.3 (+1.2) 5 7.8
*fine-tuned on ACI-Bench, i.e. not a few-shot setting.
Dada et al. (2024) highlighted a performance gap
regarding medical LLMs’ performances in clinical
settings based on the CLUE benchmark. Out of
the twelve medical LLMs in their study, only two
are improving upon their base model as shown
Alignment
of
SLMs
_Phi3.5 mini_ 36.5 -
+SFT 800K 42.2 9 1.4
+DPO 42.2 8 1.4
_MediPhi_ 39.3 **11** 1.5
+SFT 2.5M 41.9 9 1.6
+SFT 800K 43.0 9 1.4
+DPO **43.4** 9 1.4
in Table 5: BioMistral with DARE model merging, and Med42 with SFT alignment. By applying PIT, merging, and clinical alignment, MediPhi
yields the highest improvement over its base model
on CLUE+, achieving a +6.9% gain, compared to
+1.1% for BioMistral and +1.2% for Med42.
Although Phi3.5 already surpasses Mistral models, the MediPhi SLMs further widen this gap.
Despite being less than half the size of LLaMA3
(8B), MediPhi-Instruct (3.8B) achieves near-parity
on CLUE+, with a performance difference below
1%. MediPhi-Instruct outperforms LLaMA3 on
four key datasets [6] : ICD10CM (+29.2%), MeDiSumCode (+13.9%), RRS QA (+5.8%), and MeQSum (+3.3%). Med42 improves over LLaMA3 by
+1.2%, mainly due to a +27.7% boost on ICD10CM
as well as being fine-tuned on the trainset of ACIBench. However, MediPhi surpasses Med42 in
relative percentages on four tasks: ICD10CM by
2.1%, on RRS QA by 13.9%, SDoH by 5.2%, and
MeDiSumCode by 47.6%. Moreover, it is competitive within 1% in absolute on three other tasks:
MeQSum, MeDiSumQA and MEDEC. Moreover,
MediPhi models (3.8B) achieve wider improvements over its base model with a # _DG_ between 9
and 11, compared to 5 for Med42 (8B).
We summarized the overall progression of our
approach in Figure 6.
Figure 6: Summary of improvements from Phi3.5 to
_MediPhi_ - _Instruct_ (SFT+DPO). _DataMix_ is adapted
on all corpora. _SLERP PIT AV G_ is the average of five
experts trained with PIT on each dataset. _MediPhi_ is the
unified expert from the five experts. _MediPhi_ - _SFT_ is an
instruct model based on SFT only.


In this work, we introduced MediPhi, the first
clinical-focused SLM collection, alongside Medi
6See Tables 11 and 12 in Appendix A.8
Flow, a large-scale synthetic instruction dataset for
clinical alignment. Our results show that PIT significantly enhances domain adaptation, especially
when combined with model merging. Notably, our
medical coding expert surpasses GPT-4-0125 on
the ICD10CM benchmark, and aligning MediPhi
with MediFlow improves CLUE+ performance by
18.9% on average. By releasing these resources, we
aim to enhance reproducibility, drive SLM adoption in clinical settings, and foster MediPhi’s continued development through its modular design.


This study necessitated a vast amount of resources
in terms of 8x80GB A100 GPUs on Azure Machine
Learning for approximately 12,000 GPU _·_ hours, of
which close to 3,600 GPU _·_ hours were dedicated to
achieve the final model with evaluations. It also
required access to Azure OpenAI services for GPT4o, GPT-4o mini and text-embedding-3-large (i.e.
close to 25B input-output tokens).
The MediFlow corpus has a few limits. The
first limitations is that we set our scope to oneturn instructions instead of multi-turn conversation
dataset like UltraChat (Ding et al., 2023). The
second issue is the limited amount of very high
complexity tasks (i.e. multi-step reasoning tasks)
with also very long input and output. The third
limit is that Mediflow is an English-only clinical
corpus. To broaden the scope of MediFlow, future
works might use seed data from clinical corpora
as well as expanding our agentic pipeline. While
MediPhi and MediPhi-instruct might conserve abilities from _Phi3.5 mini_ (i.e. multilingual, conversational, safety alignment, etc.) by model merging
(Hammoud et al., 2024), we hypothesize that these
could be affected for which the impact was not
studied outside medical and clinical abilities as
well as instruction-following abilities. Thus, our
recommendation is to use the MediPhi collection
specifically on clinical NLP tasks. We also strongly
advice a verification of the model output by an expert in the specific medical field of the task.
Our CLUE+ benchmark expands upon the
CLUE coverage in terms of tasks, input data
and datasets. While the coverage of the clinical
field is large, a couple of gaps still remain. The
information-extraction tasks are only represented
by the SDoH dataset, and there is no input data on
the nursing sub-field (e.g. nurse-patient dialogs or
notes).
Given the quick evolutions of OpenAI’s GPT-4o
models as well as their stochastic nature, future exact reproduction of parts of this work may become
impossible if the mentioned versions are no longer
maintained.


We acknowledge that the substantial computational
resources required for this work, including GPU
hours and API calls, contribute to carbon emissions
as well as limit the reproduction of this work. However, we believe that the knowledge and artifacts
produced — such as the MediPhi SLM collection
and the MediFlow corpus — offer significant positive impacts. These include enabling the adoption
of SLMs in clinical settings, potentially reducing
carbon emissions in the medium to long term, and
fostering further research on continual pre-training
in medical, clinical, and other domains.
Additionally, our modular approach with releases of models and datasets allows researchers
and practitioners to build directly on this work,
promoting accessibility and collaboration. Strong
language models in the clinical field can positively
impact public health and support the work of healthcare providers, enhancing patient care and operational efficiency.


The _PubMed_ group is the largest by 2 orders of
magnitude with about 48B tokens. PubMed Central (National Library of Medicine, 2003) have a
segment for commercial use, while abstracts are
public.
In the _Medical_ group, we have medical
Wikipedia known as MedWiki (Corbeil, 2024).
We also fetch the open medical guidelines (Chen
et al., 2023) which comes from multiple recognized
health organizations (e.g. WHO). Then, we gather
medical coding corpora from public websites:
ICD9CM, ICD9PROC, ICD10CM, ICD10PROC
and ATC. The ICD coding is a wide taxonomy of
diseases and medical conditions as well as procedure codes, while _Anatomical Therapeutic Chem-_
_ical_ (ATC) codes are a classification of medical
drugs.
In the _Clinical_ group, we leverage PMC-Patients
v2 (Zhao et al., 2023) — subset with distributionand commercial-friendly CC licenses — which
contains clinical cases. We also include synthetic
derivative datasets under same licensing conditions:
_Asclepius_ (Kweon et al., 2024) containing clinical
documents and _NoteChat_ (Wang et al., 2023) containing doctor-patient dialogues. Furthermore, we
include _MTSamples_ (mts, 2019), a public database
filled with various de-identified clinical documents.


We use the HuggingFace ecosystem ( _transformers_,
_trl_, _accelerate_, and _datasets_ ) to train models leveraging its multi-node setting. The hyperparameters
of the continual pre-training are listed in Table 6.
The hyperparameters of the alignment process are
provided in Tables 7 and 8 for SFT and DPO, respectively.
Ding et al. (2024) highlighted the importance
of longer context windows, minimal truncations,
and restricted cross-document attention during pretraining. Many prior works in medical NLP rely
on concatenate-and-truncate strategies, despite introducing significant truncation (Toma et al., 2023;
Christophe et al., 2023, 2024a; Labrak et al., 2024).
Table 6: Hyperparameters for CPT excluding PIT.


Maximum Tokens 4,096
Optimizer AdamW
LR Scheduler Linear Warmup - Cosine


Maximum LR 1 _×_ 10 _[−]_ [4]
Epochs 1
NEFTune _α_ 5
Batch Size per GPU 16


Gradient Accumulation 2
Effective Batch Size 512
Table 7: Hyperparameters for SFT.


Optimizer AdamW
LR Scheduler Linear Warmup - Cosine


Maximum LR 2 _×_ 10 _[−]_ [5]
Epochs 2
NEFTune _α_ 5
Batch Size per GPU 16


Gradient Accumulation 2
Effective Batch Size 256
Table 8: Hyperparameters for DPO.


Maximum LR 1 _×_ 10 _[−]_ [6]
Epoch 1
Batch Size per GPU 8


Gradient Accumulation 1
Effective Batch Size 128
_β_ 0.1
Instead, we implement Best-Fit Packing (Ding
et al., 2024) for the _PubMed_ expert and _DataMix_
baseline, ensuring efficient token utilization. The
other experts are trained with PIT one document
at a time. For all expert models, we avoid padding
and cross-document attention (Kundu et al., 2024)
in trainings.


The twelve validation sets cover subjects such
as clinical case, clinical knowledge, medication,
icd10cm code definitions, radiology report, clinical NLI, QA on discharge letter, medical codes of
discharge letter, problem list from clinical notes,
summarization of patient inquiry, QA on medical
consultation and QA on multiple EHR documents.
We control its diversity by generating the embeddings of the question with its context, and applying
the density-based clustering method HDBSCAN
(McInnes et al., 2017). The final valid sets are the
combination of the outliers (i.e. not assigned to a
cluster) and one sample per cluster for a total up to
1,200 samples each.


We parametrized the prompt to generate MediFlow
based on the task type, input data, output format
(plain text or JSON), difficulty level (moderate,
moderate-hard, hard, very hard, or extreme), and
number of input-output example pairs (3 or 4 per
instruction). For the difficulty level, we favored the
sampling of hard, very hard and extreme levels by
a ratio of 3:1.
The **14 task types** are: summarization, questionanswering, multiple-choice question-answering,
named entity recognition, Relation extraction, classification, reasoning and diagnosis, textual entail
ment, text simplification, text expansion, abbreviation expansion, aspect-oriented keyword extraction,
error detection and correction and note scoring.
The 36 input-data types with various levels of
granularity (1 (only complete document) up to 7
(complete document with 6 individual sections))
resulting in a total of 98 fine-grained input-data
types. Decisions for the granularity levels were
made based on the document length — e.g. taking
the complete short documents compared to segmenting long documents. The data types are:
1. Discharge Summary (6)
2. SOAP Clinical Note (5)
3. Clinical Note (6)
4. Progress Note (4)
5. Admission Note (6)
6. Scientific Article (8)
7. Clinical Case (3)
8. Nursing Note (1)
9. Monitoring Data of Vital Signs (1)
10. Referral Letter (1)
11. Emergency Department Note (7)
12. Laboratory Report (1)
13. Radiology Report (3)
14. Doctor-Patient Conversation (5)
15. Nurse-Patient Dialog (5)
16. Operative Note (8)
17. Consultation Note (1)
18. Pathology Report (1)
19. Prescription Note (1)
20. Preoperative Assessment (1)
21. Postoperative Note (1)
22. Therapy Notes (5)
23. Immunization Record (1)
24. Screening Report (1)
25. Consent Form (1)
26. Care Plan (1)
27. Dietary Notes (1)
28. Psychiatric Evaluation (5)
29. Social Work Note (1)
30. End-of-Life Care Documentation (1)
31. Triage Note (1)
32. Dental Record (1)
33. Home Health Care Report (1)
34. Genetic Testing Report (1)
35. Incident Report (1)
36. Patient Education Material (1)
In Figures 7, 8 and 9, we present the histograms
of tokens for generated instructions, input and output, respectively.
Figure 7: Distribution of instruction tokens in MediFlow
with y-axis in log scale. The average is 301 _±_ 295 tokens.
Figure 8: Distribution of input tokens in MediFlow with
y-axis in log scale. The average is 76 _±_ 67 tokens.
Figure 9: Distribution of output tokens in MediFlow
with y-axis in log scale. The average is 79 _±_ 66 tokens.


We used GPT-4o-mini to score all MediFlow on
5 criteria: quality, alignment with instruction requirements, coherence, realism and difficulty. We
applied a temperature of 1 _._ 0 with chain-of-thought
for each criterion and average scores accross 5 generated samples for self-consistency. All scores are
an integer on a scale from 1 to 4 as defined below.


In Figure 10, we displayed the histograms (with a
logarithm y axis) of the judge scores predicted on
nearly 700k instructions. For all criteria, we note
that a large amount of generated instructions have
a peak at 3 with large distribution between 3 and
4 (on a scale from 1 to 4), which is considered on
the high end. We observe a different trend with
respect to the difficulty criterion, where a larger
peak is still at 3 (i.e. difficulty level of hard) but
some portion of the dataset (i.e. less than a 100k
instruction) have scores between 2 and 3.
Figure 10: Judge Score Distributions across MediFlow.
Y-axis is set to log scale.


We generated embedding with OpenAI _text-_
_embedding-3-large_ (truncated at 256 dimensions)
for each instruction in MediFlow. Then, we applied PCA at 50 dimensions and t-SNE to get 2 dimensions which we displayed on scatterplots with
other labels (output formats, difficulty levels, task
type and input-data type) in Figures 13, 11 and 12.
While the scatter plot on the difficulty levels do
not exhibit clear clustering patterns, we see that the
others have distinctive patterns at different scales.
The output format is affecting local clustering patterns often appearing at small scale as two closed
blobs. The task and input-data types affect the
macrostructure at similar large scale.


DPO requires to have a prompt (corresponding in
our framework to _instruction_ with _input_ ) along a
chosen response (i.e. _output_ ) and a rejected response. The rejected response must be less preferable compared to the chosen one. First, we filter
MediFlow to around 85k instructions keeping only
the top triplets on the _quality_ metric along with a
stratification by task type, input-data type and output format. To have the best trade-off between diversity and high quality, we sampled 3 input-output
pairs for the first top high-quality 20k instructions,
followed by 2 pairs for the next 25k and one pair
for the last 40k. After filtering, this resulted the
MediPhi-DPO dataset of 130,852 triplets. Then,
we prompted GPT-4o with a triplet at a temperature of 1.0 with a randomly sampled error type
on the following prompt to generate a marginally
wrong output as rejected output. The error types
are : ambiguity, partial correctness, over-verbosity,
brevity, unbalanced detail, stylistic issues, factual
inaccuracy, logical flaws, misinterpretation, simplistic reasoning, grammatical errors, and spelling
errors.


MedicationQA (Ben Abacha et al., 2019) consists
of 674 consumer health questions collected from
MedlinePlus [7] . These questions were linked to a
matching excerpt from a trusted source that contains the answer.
Through manual inspection we identified two
7https://medlineplus.gov
Figure 11: t-SNE 2D scatterplot of MediFlow (2.5M) using OpenAI _text-embedding-3 large_ API with tasks as
colors.
Figure 12: t-SNE 2D scatterplot of MediFlow (2.5M) using OpenAI _text-embedding-3 large_ API with input-data
types as colors.
Figure 13: t-SNE 2D scatterplots of MediFlow (2.5M)
with output format (top) and difficulty level (bottom) as
colors.
obstacles for effective few-shot evaluation of LLMs
on MedicationQA:
  - Some questions were poorly formulated or a
search query instead of a question
  - Some of the answers did not give a specific
answer to the question
  - Answers were often unconcise since they were
not formulated as a direct answer to the question, but a retrieved excerpt from a text
Based on these observations we prompted an LLM [8]
to remove non-matching question answer pairs and
formulate a direct answer based on the given excerpt. Figure 14 shows an example for a reformulated answer. This process resulted in 485 questionanswer pairs.
For benchmarking we use the following system
prompt:
Indiana University chest X-ray dataset (DemnerFushman et al., 2016) and reports from the Stanford Health Care system. The findings serve as
model inputs while the impressions are treated as
the summarization ground truth. We evaluate on the
test split, which contains 600 finding-impression
pairs. We observed that the impressions were often not a complete summary, but rather an answer
to a question posed before the exam (e.g., "No
acute cardiopulmonary findings."). Additionally,
the comprehensiveness of impressions varied substantially between reports. To address this we reformulated the impressions to a series of questionanswer pairs using an LLM [8] . We specified that the
answers should use the same wording as in the
original impression section to ensure factuality. We
confirmed that answers were not changed, filtering answers with exact string matching. Figure 15
shows an example of this reformulation step.
The following system prompt was used in the
evaluation:


ACI-Bench (Yim et al., 2023) is a dataset that takes
as input a doctor-patient dialog and the task is to
generate a clinical note of five sections.


MeDIQA-RRS (Ben Abacha et al., 2021) is a
summarization dataset based on findings and impressions sections of Radiology reports from the
8meta-llama/Llama-3.3-70B-Instruct
Figure 14: An example of a verbose answer in MedicationQA and the reformulated answer we replaced it with.


Figure 15: An example of a formulation of questions based on the impression section.


Table 9: The configuration used for CLUE+ subset
datasets of the benchmark regarding few shots and the
metric used. The decoding strategy on these six datasets
is greedy decoding.


ICD10CM 3 AVG Accuracy
Rouge-1 F1
MedicationQA 3
(Lin, 2004)
Rouge-1 F1
RRS QA 3
(Lin, 2004)
Type-Match F1
SDoH 4 w/ boundary overlap
(Chai, 2019)
Rouge-1 F1
ACI-Bench 1
(Lin, 2004)
MEDEC 2 Sentence ID Accuracy
Table 10: CLUE+ benchmark datasets with task types, input, and output specifications. Task types are: questionanswering (QA), summarization, reasoning and information extraction (IE).


NLI MedNLI (Romanov and Shivade, 2018) Premise + Hypothesis Label
Summary MeQSum (Ben Abacha and Demner-Fushman, 2019) Consumer Health Question Summary question
Summary Problem list summarization (Gao et al., 2023) Progress notes Problem list
QA LongHealth (Adams et al., 2024) Clinical records + Question Answer
QA MeDiSumQA (Dada et al., 2025) Discharge letter + Questions Answers
Reasoning MeDiSumCode (Dada et al., 2024) Discharge letter ICD10CM codes
QA MedConceptsQA ICD10CM (Shoham and Rappoport, 2024) Code + definition options Answer
QA MedicationQA (Ben Abacha et al., 2019) Question on medication Answer
QA MEDIQA-RRS QA (Ben Abacha et al., 2021) Findings + Questions Answers
IE SDoH (Lybarger et al., 2023) Clinical note Entities
Summary ACI-Bench (Yim et al., 2023) Doctor-patient dialog Clinical note
Reasoning MEDEC (Ben Abacha et al., 2024) Clinical note Error detection
Table 11: Performances on the CLUE subset of datasets for the merged and aligned versions of MediPhi as well as
other medical LLMs. PLS stands for Problem List Summary while LH refers to LongHealth.


**Baseline** _Phi-3.5-mini-instruct_ 66.6 28.4 36.7 45.9 25.9 41.1
DataMix 68.5 29.0 37.7 45.7 26.6 41.4

MedNLI PLS MeQSum LH MeDiSumQA MeDiSumCode


PubMed 68.3 29.2 37.6 45.7 26.3 41.0
Clinical 69.2 29.4 38.1 43.5 26.7 40.5
MedWiki 72.8 29.2 37.6 43.6 25.1 41.7
MedCode 68.5 22.3 33.5 45.7 23.6 39.0
Guideline 70.3 29.8 37.6 41.1 25.1 41.9
**BreadCrumbs** MediPhi 66.9 28.8 37.9 45.7 26.1 41.7
MediPhi-SFT 70.6 26.9 42.8 44.2 28.8 35.0


MediPhi-Instruct 71.0 26.0 42.8 45.0 29.1 37.2
Other
Medical
LLMs
Mistral-7B-Instruct-v0.1 64.8 25.0 31.1 30.0 25.5 13.9
BioMistral-7B-DARE 66.8 28.4 34.5 30.5 25.7 21.3
Meta-Llama-3-8B-Instruct 74.1 31.6 39.5 58.8 30.3 27.8
Llama3-Med42-8B 77.5 32.4 42.8 57.9 29.7 25.2
Table 12: Performances on the new CLUE+ subset of datasets for the merged and aligned versions of MediPhi as
well as other medical LLMs. ACI refers to ACI-Bench.


**Baseline** _Phi-3.5-mini-instruct_ 41.2 11.2 14.8 42.3 35.1 49.3
DataMix 43.3 10.8 18.8 42.7 36.2 49.5

RRS QA MedicationQA MEDEC ACI SDoH ICD10CM


PubMed 44.1 10.3 22.2 42.7 35.8 49.5
Clinical 52.1 12.0 34.5 43.9 35.8 49.6
MedWiki 46.7 12.2 28.8 44.7 43.6 50.2
MedCode 45.6 12.0 18.1 39.0 24.8 68.7
Guideline 48.9 11.9 28.3 44.7 41.0 49.8
**BreadCrumbs** MediPhi 44.5 11.3 29.1 44.3 39.7 55.5
MediPhi-SFT 60.8 18.8 35.0 43.4 54.5 54.9


MediPhi-Instruct 61.6 19.3 34.4 43.5 56.7 54.9
Other
Medical
LLMs
Mistral-7B-Instruct-v0.1 50.4 22.7 21.5 50.4 40.2 27.6
BioMistral-7B-DARE 49.6 22.3 23.1 43.3 45.9 25.1
Meta-Llama-3-8B-Instruct 55.8 26.1 46.5 50.2 63.1 25.7
Llama3-Med42-8B 54.1 25.7 35.4 56.5 53.9 53.4
Table 13: Performances of MediPhi models on multiple-choice question-answering medical benchmarks.


_Phi-3.5-mini-instruct_ 0.486 0.554 0.768 0.715 0.631
PubMed 0.467 0.549 0.774 0.724 0.629
Clinical 0.500 0.551 0.772 0.727 0.638
Medical 0.519 0.535 0.740 0.701 0.624
MedWiki 0.478 0.548 0.768 0.719 0.628
MedCode 0.531 0.532 0.758 0.700 0.630
Guideline 0.459 0.556 0.770 0.724 0.627
MediPhi 0.491 0.559 0.766 0.720 0.634
MediPhi-SFT 0.536 0.552 0.766 0.716 0.642
MediPhi-Instruct 0.548 0.555 0.764 0.714 0.645


**Qingrong Xia** **[3]**, **Xinyu Duan** **[3]**, **Zhefeng Wang** **[3]**, **Baoxing Huai** **[3]**, **Min Zhang** **[1,2]**,
1Soochow University, China
2Key Laboratory of Data Intelligence and Advanced Computing, Soochow University
3Huawei Cloud,China
[risus254@gmail.com, ljt@suda.edu.cn](mailto:risus254@gmail.com)

Alignment-Augmented Speculative Decoding with Alignment Sampling and
  Conditional Verification

Jikai Wang [1,2] , Zhenxu Tian [1,2] , Juntao Li [1,2] [] ,

Jikai Wang
Zhenxu Tian
Juntao Li
Qingrong Xia
Xinyu Duan
Zhefeng Wang
Baoxing Huai
Min Zhang


Recent works have revealed the great potential of speculative decoding in accelerating the
autoregressive generation process of large language models. The success of these methods relies on the alignment between draft candidates
and the sampled outputs of the target model.
Existing methods mainly achieve draft-target
alignment with training-based methods, e.g.,
EAGLE, Medusa, involving considerable training costs. In this paper, we present a trainingfree alignment-augmented speculative decoding algorithm. We propose alignment sampling,
which leverages output distribution obtained in
the prefilling phase to provide more aligned
draft candidates. To further benefit from highquality but non-aligned draft candidates, we
also introduce a simple yet effective flexible
verification strategy. Through an adaptive probability threshold, our approach can improve
generation accuracy while further improving
inference efficiency. Experiments on 8 datasets
(including question answering, summarization
and code completion tasks) show that our approach increases the average generation score
by 3.3 points for the LLaMA3 model. Our
method achieves a mean acceptance length up
to 2.39 and speed up generation by 2.23×.


The enormous size of state-of-the-art autoregressive models (Anthropic, 2024; Meta-AI, 2024;
OpenAI, 2024) demands substantial memory and
processing power, making real-time applications
challenging. In scenarios such as interactive text
generation, these models require vast amounts of
computation, leading to slower response times and
increased energy consumption. Hence, more efficient decoding algorithms are urgently needed to
reduce inference costs significantly while maintaining or improving generation quality.
*Corresponding author.
Speculative decoding (SD) (Stern et al., 2018;
Leviathan et al., 2023; Xia et al., 2023, 2024; Li
et al., 2024) has emerged as an effective solution to
the inefficient decoding process of large autoregressive models. It uses a “draft and verify” mechanism
to achieve lossless acceleration. A lightweight
drafting model generates candidate tokens, which
are then verified in parallel by the target model.
This enables the generation of multiple tokens in a
single step, thus accelerating decoding while preserving the output distribution. Speculative decoding algorithms that combine large and small models (Miao et al., 2023; Li et al., 2024) have shown
promising results. However, these approaches often require additional training and extra parameters, complicating deployment in inference systems
(Contributors, 2023; Kwon et al., 2023).
As an alternative, the training-free retrievalbased speculative decoding (Yang et al., 2023; Saxena, 2023; He et al., 2024) uses external knowledge
sources such as databases or historical text to retrieve n-grams as drafts for generation. It is more
flexible and scalable to large models, as the drafting cost is independent of model size. However,
two issues have been largely overlooked in previous work on retrieval-based speculative decoding.
First, poor alignment between the retrieved drafts
and the model’s output distribution leads to a low
acceptance rate. Second, in many generation scenarios, the input context and the generated content
have strong correlation. As a result, drafts retrieved
from the input context tend to be of higher quality and do not require strict verification to ensure
consistency with the target model’s output.
In this paper, we propose **A** lignment- **A** ugmented
**S** peculative **D** ecoding (AASD), a plug-and-play
generation algorithm that improves the draft-target
alignment in prompt-based speculative decoding.
To address the first issue, we introduce alignment
sampling, where we sample additional tokens from
the output distribution of the prefilling phase for
poorly aligned tokens, thereby improving the overall draft quality and increasing the chances of successful alignment with output distribution of the target model. To address the second issue, we propose
a conditional verification strategy. It applies heuristic probability thresholds for each token, based on
its information entropy, allowing the model to autonomously utilize the input text during decoding
through speculative decoding. It improves both
generation accuracy and efficiency. Both the two
techniques improve the alignment between the retrieved draft candidates and the target model output
distribution. Alignment sampling makes the drafts
more aligned with the target model distribution,
and conditional verification makes the target model
more aligned with the high-quality drafts.
We conduct comprehensive experiments with
two different models on 8 datasets in long context
generation scenarios, including question answering,
summarization, and code completion tasks, in Section 4 to evaluate the effectiveness of AASD. The
results show that AASD outperforms common sampling methods in terms of generation performance,
improving the average score from 44.69 to 47.98
compared to greedy sampling for the LLaMA3
model (Meta-AI, 2024). In terms of generation efficiency, AASD achieves the highest acceptance rate
and average decoding throughput among existing
retrieval-based speculative decoding algorithms. It
achieves a decoding speedup of up to 2.23 times.
In summary, we highlight the importance of
draft-target alignment in retrieval-based speculative
decoding in this paper. We propose a training-free
method called alignment-augmented speculative
decoding to improve this alignment, which consists
of alignment sampling and conditional verification.
The proposed method improves generation accuracy and efficiency compared to baseline methods.


Given a prompt _q_ = ( _x_ 1 _, x_ 2 _, ..., xl_ ) and language
model _M_, where _xi_ ( _i_ =1 _,_ 2 _,...,l_ ) represents each token, and _l_ is the prefix length, we input the current
sequence into the model to obtain the next token in
the autoregressive decoding manner:
_yl_ +1 _∼_ _pM_ ( _xl_ +1 _|_ ( _x_ 1 _, x_ 2 _, ..., xl_ )) _,_ (1)
where _pM_ ( _·|·_ ) represents the output probability distribution of _M_ . The model can generate only one
token in one forward propagation, resulting in low
efficiency, particularly for long generation lengths..
Speculative decoding (SD) alleviates this problem by adopting a “draft and then verify” mechanism. At each decoding step, a lightweight model
_m_ generates a draft _d_ = (ˆ _xl_ +1 _,_ ˆ _xl_ +2 _, ...,_ ˆ _xl_ + _ld_ ) for
the next tokens based on the current input sequence,
where ˆ _xi_ ( _i_ = _l_ +1 _, l_ +2 _, ..., l_ + _ld_ ) denotes the draft
tokens and _ld_ represents the draft length. This draft
is then simultaneously input into the target model
for verification. Tokens in the draft that match the
target model’s output will be accepted. Note that
if a token is rejected, all subsequent tokens in the
draft are also rejected. This allows the model to
generate multiple tokens in a single forward pass.


An effective draft model must address two crucial aspects. First, it must minimize computational
costs, ensuring faster and more efficient inference
through a lightweight design. Second, it must
closely align with the target model. Ideally, perfect
alignment would result in a 100% acceptance rate,
thus achieving the theoretical maximum speedup,
defined as the throughput of the smaller model divided by that of the target model.
Approaches such as EAGLE (Li et al., 2024)
strengthen the alignment between the draft and
target models through alignment training, causing inconvenience in adapting to different models.
Retrieval-based speculative decoding methods complete the drafting by retrieving n-grams from historically generated data or pre-provided databases
based on the current sequence. However, existing retrieval-based speculative decoding methods
(Yang et al., 2023; Saxena, 2023; He et al., 2024)
ignore the importance of alignment. Although their
drafting overhead is very small compared to methods that require additional modules, their acceptance rate is also relatively lower.


Recent research (Bachmann et al., 2025) shows
that even a high-quality draft will have a low acceptance rate if it does not align with the model output
distribution. We also conduct an experiment to give
an insight into the impact of draft-target alignment
in retrieval-based speculative decoding. For the
four-token input sequence “A, B, C, D”, the model
can predict the probability distribution of the next
token for each input token in parallel. For each
distribution, we select the token with the highest
Figure 1: Acceptance rate of aligned and misaligned
draft tokens in prompt-based speculative decoding.
probability, that is, greedy sampling. Assume that
the sampled tokens are B, C, F, E in order. The sampled tokens “B” and “C” match the input, while
“F” does not match “D” in the input. We regard
the matching tokens (“B” and “C”) as aligned tokens and the unmatched tokens (“D”) as misaligned
tokens. We conduct prompt-based speculative decoding on 100 samples in NQ (Kwiatkowski et al.,
2019) dataset with LLaMA3.1-8B-Instruct (MetaAI, 2024) and calculate the acceptance frequency
of aligned and misaligned draft tokens. Results
are shown in Figure 1. About 45% retrieved draft
tokens are misaligned with the model output. The
acceptance rate of misaligned tokens is much lower
than that of aligned tokens. Inspired by this, for
the positions of misaligned tokens, we can sample
additional tokens from the output distribution to
enhance the alignment of the entire draft and the
model, thus improving the acceptance rate.


Most speculative decoding methods adopts strict
verification to achieve lossless generation acceleration. The recent research (Bachmann et al., 2025)
claims that correct but non-aligned draft candidates
can be accepted to pursue higher speed-up ratio.
They trained an additional module on carefully designed data to judge whether to accept draft tokens.
However, for prompt-based speculative decoding,
as the draft source is highly relevant to the answer,
a training-free verification strategy can be designed
to realize a better trade-off between generation performance and efficiency. For example, in long context generation scenario, the input context contains
the key information needed for model generation.
Some of the fragments even overlap with the stan
dard generated content. Given this, we can design a
conditional verification method to allow the model
to accept high-quality draft tokens, rather than just
accept that match its original output. This allows
the model to adaptively call the input original segments on the decoding side, thus enhancing the
generation accuracy and efficiency.

2.4 Potential for Non-Strict Verification


In this section, we introduce AASD, an alignmentaugmented speculative decoding algorithm based
on alignment sampling and conditional verification
that improves both generation performance and efficiency. It is a plug-and-play speculative decoding
algorithm that does not require additional modules
or training. The overall description of AASD is
dispalyed in Algorithm 1 in Appendix C.


For long text generation, the input context can act
as an important source of draft, because it is highly
relevant to the generated content. We construct the
input of each request into a draft pool by sliding
window sampling, which is an index dictionary.
The keys in the draft pool are several consecutive
tokens in the input context and the values are the
position indexes where these tokens appear. We update the draft pool when new tokens are generated.
In each decoding step, we use the last several
tokens of the current sequence as the key to retrieve
indexes in the draft pool. Since longer keys give
higher draft relevance, following REST (He et al.,
2024), we prioritize searching with longer keys and
obtain n-grams from the current sequence.


As is shown in Figure 2, we perform alignment
sampling to expand the retrieved n-grams into draft
trees. For example, the token “is” is the third most
likely word in the probability distribution of the
next word of the token “Jackson”. Therefore, “was”
and “made” are potential high-quality draft tokens,
which are more aligned with the output distribution
of the target model. We additionally sample these
two tokens for the 100th position. In this way, we
expand and merge multiple candidate n-grams into
a draft tree and calculate the corresponding tree
attention mask for parallel verification. The draft
tree T is defined as:
_ni_
_j_ =1
T = (V _,_ E) _,_ V =
_l_ + _r_
_i_ = _l_ +1
- _x_ ˆ _[j]_ _i_ _,_ (2)
Figure 2: Illustration of alignment sampling. “Jackson is a great” is an n-gram retreived by the key “Michael
Jackson”. According to the output distribution obtained in the prefilling stage, “is” is the token with the third highest
probability. The top-1 token “was” and top-2 token “made” are more likely to be generated after “Michael Jackson”.
Therefore, we sample two more tokens (“was” and “made” for the 100th position). Token “a” is well aligned with
the model output, so we keep it unchanged for the 101st position.
where V and E is the set of its nodes and edges.
_ni_ is the number of retrieved tokens in the _ith_
layer of T. _r_ is the depth of T. Then we input
T and the corresponding tree attention mask into
the model and get the output probability distribution _PM_ ( _xi|_ F(ˆ _x_ _[j]_ _i_ [))][ for each node, where][ F][(ˆ] _[x][j]_ _i_ [)][ is]
the set of all parent nodes (including prefix) of ˆ _x_ _[j]_ _i_ [.]


As discussed in Section 2.2, high-quality drafts retrieved from prompts have the potential to be nonrigorously verified to enhance generation accuracy.
The naive idea is to use a fixed probability threshold to filter draft tokens or use top- _k_ verification.
We argue that using the probability threshold condition is superior to using the top- _k_ condition alone
since the output probability directly reflects the
model’s confidence in each draft token. Moreover,
the verification condition of top- _k_ is not applicable
for some extreme cases. For example, suppose the
probability of a certain token is close to 1, and the
probability of all other tokens is close to 0. In that
case, no other token should be accepted, except for
the token with the highest probability. If all tokens
have the same probability, then they should all be
accepted or rejected at the same time. However,
the top- _k_ verification condition will only pass individual tokens and reject others. Therefore, we first
set a probability threshold _δ_ so that token ˆ _x_ _[j]_ _i_ [will]
be accepted if the following conditions are met:
_pM_ (ˆ _x_ _[j]_ _i_ _[|]_ [F][(ˆ] _[x]_ _i_ _[j]_ [))][ ⩾] _[δ.]_ (3)
However, models exhibit varying levels of confidence across different samples or tokens. A static
threshold may either over-restrict or under-restrict
the verification process, leading to suboptimal performance. Therefore, we formulate an adaptive
threshold based on the verification probability distribution. Low-confidence tokens can be processed
with stricter thresholds to maintain precision, while
high-confidence tokens benefit from relaxed thresholds to ensure recall. Therefore, we adjust the
verification threshold of each token based on the
information entropy of its verification probability
distribution. Draft token ˆ _x_ _[j]_ _i_ [is accepted under the]
following condition:
_pM_ (ˆ _x_ _[j]_ _i_ _[|]_ [F][(ˆ] _[x][j]_ _i_ [))][ ⩾] _[δ]_ _i_ _[j][,]_ (4)
where _δi_ _[j]_ [is calculated by:]
   _δi_ _[j]_ [=] _[ min]_ [(] _[−][α]_ _P_ (ˆ _x_ _[j]_ _i_ [) log] _[ P]_ [(ˆ] _[x][j]_ _i_ [)+] _[β,]_ [ ∆)] _[,]_ [ (5)]
where
∆= max _P_ (ˆ _x_ _[j]_ _i_ [)] _[,]_ (6)
_x_ ˆ _[j]_ _i_
_α_ and _β_ are hyperparameters. _α_ a is the factor
affecting the threshold value due to information
entropy. _β_ is the benchmark value of the threshold. The function of ∆ is to ensure that the token
with the highest probability will be accepted during
verification. Considering that the overall quality
of the alignment sampled tokens is lower than the
tokens retrieved from the prompt. We only perform
conditional verification on the draft token retrieved
from the input context and selected by alignment
sampling. Strict verification is still applied for tokens retrieved from generated context. We accept
the longest draft candidate that passes verification.


In most speculative decoding methods with strict
verification (e.g., REST (He et al., 2024), PLD
(Saxena, 2023) and LLMA (Yang et al., 2023)),
the performance on datasets depends solely on the
sampling strategy rather than the specific decoding
algorithm. Equipped with conditional verification,
AASD introduces a novel sampling method, which
enhances access to input fragments. Therefore, we
compare the performance of **AASD** with several
common sampling methods: 1) **Greedy Sampling** :
It samples the token with the highest probability; 2)
**Top-** _**k**_ **Sampling** (Fan et al., 2018): It selecting the
next token from the top _k_ most probable options
( _k_ = 50); **Nucleus Sampling** (Holtzman et al.,
2020): It is also called top- _p_ sampling ( _p_ = 0 _._ 8),
which selects the next token from the smallest set of
top tokens whose cumulative probability exceeds
a predefined threshold; 3) **Beam Search** (Li et al.,
2016): It explores multiple candidate sequences at
each step, keeping only the top _n_ most likely options (the beam width _n_ = 10), to find an optimal
or near-optimal output sequence.
On the other hand, we compare the performance
and efficiency of **AASD** with autoregressive decoding and other retrieval-based speculative decoding methods. Baselines are as follows: 1) **Auto-**
**regressive decoding (ARD)**, 2) **REST** (He et al.,
2024): It retrieves n-grams from an external general database and constructs a tree-structured draft,
3) **PLD** (Saxena, 2023): It retrieves n-grams directly from the input context. Note that **LLMA**
(Yang et al., 2023) and **PLD** can be considered as
the same method since they are very similar.
We evaluate the proposed method with
LLaMA3.1-8B-Instruct (Meta-AI, 2024) and
Qwen2.5-32B-Instruct (Qwen et al., 2025) on 3 different tasks: 1) **Question Answering (QA)** : Nature
Question (NQ) (Kwiatkowski et al., 2019), TriviaQA (TQA) (Joshi et al., 2017), 2WikiMQA (Ho
Figure 3: Speed-up ratio on the six different task categories in SpecBench. The center point (1.0×) represents
the baseline of autoregressive decoding.
et al., 2020) and HotpotQA (Yang et al., 2018); 2)
**Summarization** : Multi-News (Fabbri et al., 2019)
and GovReport (Huang et al., 2021); 3) **Code Com-**
**pletion** : RepoBench-P (Liu et al., 2024) and LCC
(Guo et al., 2023). For the NQ dataset, we randomly sampled 300 queries from the validation
set for evaluation. We use subsets sampled by
LongBench (Bai et al., 2023) for other datasets.
We report F1 score for QA datasets, ROUGE-L
for summarization datasets and Edit Sim for code
completion datasets. We also test the decoding
efficiency on **SpecBench** (Xia et al., 2024).
We use the same generation hyperparameters for
all baselines. REST involves an additional database
as the draft source. We selected the best database
from the officially released database for testing (for
each model and each datasets). We did not compare with other training-required methods for generating enhancements and speculative algorithms
because they require additional training or use additional models. We use 6-grams and set the maximum length of the key for retrieval to 6 for AASD.
We use unified hyperparameters for AASD on all
datasets. _α_ is set to 0.1 and 0.2 for LLaMA3.1
and Qwen2.5, respectively. _β_ is set to 0.1. Each
position can be extended by at most two tokens
by alignment sampling during drafting. We use 4
A100-PCIE-40GB GPUs for all experiments.


We compare the performance of different sampling
methods in Table 1. We report the average scores
of the 8 datasets in the last column. AASD out


**Method** **Avg.**
F1 F1 F1 F1 ROUGE-L ROUGE-L Edit Sim Edit Sim
LLaMA3.1-8B-Instruct
**Greedy** 29.13 **92.13** 42.83 49.80 13.51 15.96 48.11 66.05 44.69
**Top-** _**k**_ 29.08 84.52 38.48 42.01 13.34 15.18 45.36 56.19 40.52
**Nucleus** 29.21 86.39 39.98 44.79 13.58 15.46 45.59 55.82 41.35
**Beam** 32.57 83.04 43.29 48.93 14.07 17.75 46.27 67.52 44.16

NQ TQA 2WikiMQA HotpotQA Multi-News GovReport RepoBench-P LCC


Qwen2.5-32B-Instruct
**Greedy** 41.90 81.90 32.35 35.73 **12.77** 14.58 37.72 60.85 39.72
**Top-** _**k**_ 40.86 80.02 28.17 36.09 11.72 13.19 36.14 50.56 37.09
**Nucleus** 41.64 79.07 29.64 35.37 12.30 13.56 38.01 49.48 37.43
**Beam** 36.78 **84.57** 17.49 19.55 9.49 11.07 **42.04** **63.26** 35.53


Table 1: Performance of different sampling methods with LLaMA3.1-8B-Instruct and Qwen2.5-32B-Instruct. For
all metrics, higher scores indicate better performance. The best results are in bold.


MAL TPS Speed-up MAL TPS Speed-up MAL TPS Speed-up
LLaMA3.1-8B-Instruct
**ARD** 1.00 20.10 1.00 1.00 20.15 1.00 1.00 20.67 1.00
**REST** 1.08 18.26 0.91 1.27 21.20 1.05 1.26 26.09 1.07
**PLD** 1.93 32.77 1.63 1.42 21.33 1.06 2.06 36.52 1.77


Qwen2.5-32B-Instruct
**ARD** 1.00 8.08 1.00 1.00 8.14 1.00 1.00 7.35 1.00
**REST** 1.07 7.33 0.91 1.26 8.16 1.00 1.16 7.53 1.02
**PLD** 2.13 **14.04** **1.74** 1.51 10.01 1.23 1.46 8.72 1.19


Table 2: Inference efficiency of AASD with LLaMA3.1-8B-Instruct and Qwen2.5-32B-Instruct. ARD refers to
auto-regressive decoding with greedy sampling. MAL represents mean acceptance length and TPS represents tokens
per second. The best results are in bold.
performs other sampling methods with LLaMA3.18B-Instruct on these datasets except for TQA. It
improves the average score from 44.69 to 47.98
compared to greedy sampling. For Qwen2.5-32BInstruct, AASD also improves the average score.
However, it does not show an advantage in coderelated tasks, possibly because the model’s inherent
code capabilities are relatively weak, making it unable to effectively reject the wrong draft tokens.
To further validate the effectiveness of AASD,
we employ DeepSeek-V3 (DeepSeek-AI et al.,
2025) as a judge model to evaluate the relative
quality of responses generated by AASD and autoregressive methods. The experimental results are
shown in the table 3. AASD demonstrates superior performance to greedy sampling across most
datasets in neural evaluation (except for TQA and
GovReport), especially for the code tasks.


We evaluate generation efficiency of AASD on
TQA, GovReport, and LCC datasets (one dataset
for each task). Table 2 displays the results with
different retrieval-based speculative decoding approaches. Since the inputs of these tasks are long
and have large length variance, input length has a
great impact on the prefilling time (computation
time of the first step). To avoid the influence of
this method-irrelevant factor, we only report the
TPS of incremental inference (without the first step
of generation for each sample). REST adopts an
independent database as the drafting source, thus
having a low acceptance rate for out-of-domain
tasks. AASD outperforms other approaches both
on the perspective of MAL and throughput with
LLaMA3.1-8B-Instruct. It achieves a speed-up
ratio up to 2.23 compared with auto-regressive de
NQ TQA 2WikiMQA HotpotQA Multi-News GovReport RepoBench-P LCC
Greedy Win Rate (%) 2.6 **6.7** 3.5 1.3 15.3 **19.2** 0.7 2.1
Tie Rate (%) 94.4 87.2 85.6 89.5 66.6 72.3 86.4 82.3
AASD Win Rate (%) **3.0** 6.0 **10.9** **9.2** **18.1** 8.5 **12.9** **15.6**
Table 3: Neural evaluation results for AASD with DeepSeek-V3.
coding. For Qwen2.5-32B-Instruct, AASD has
a significant improvement in the generation efficiency of summary and code tasks. However, for
2WikiMQA, as the mean output length is very short
(about 8 tokens), its performance is similar to PLD.
The total time overhead for retrieving the draft candidates for each step is between 0.02 milliseconds
and 2 milliseconds, which is less than 5% of the
time cost of a single step for the 8B model. For
larger target models, this overhead is negligible.
In addition, Figure 3 displays the evaluation results on SpecBench. We report the speed-up ratio
compared with autoregressive decoding. AASD
outperforms the baselines in most tasks, except for
the RAG task, which further proves the effectiveness in improving decoding efficiency.


We conduct the ablation study with LLaMA3.18B-Instruct in this section. We evaluate the performance of AASD with top- _k_ verification and fixed
threshold verification. For top- _k_ condition, _k_ is set
to 5. For threshold verification, threshold is set to
0.1. Table 4 shows the results. AASD achieves the
highest average score. Using the top-k condition
decreases the scores for most datasets. For evaluation the generation efficiency, we consider AASD
without alignment sampling and AASD without
conditional verification. Table 5 displays the results. Both alignment sampling and conditional
verification contribute to the inference acceleration.


Another training-free approach is to directly use
the same series of small models as the draft model
(Leviathan et al., 2023; Kim et al., 2023). We compare AASD with this approach in this section. We
test the inference efficiency on NQ dataset with
LLaMA2 model series (Touvron et al., 2023). Table 6 displays the result. AASD outperforms SD on
generation acceleration for the 7B and 70B model.
For SD, the size of the draft model has a significant
impact on inference efficiency. A draft model that
is too small will result in a low acceptance rate,
while a draft model that is too large will lead to
excessive draft overhead. For a 70B target model,
the 7B draft model has high quality, but its draft
overhead accounts for about 80% of the total inference overhead. Therefore, it does not significantly
improve throughput. It is difficult to obtain a draft
model of appropriate size that can well balance
acceptance rate and overhead. In contrast, AASD
does not have this concern. It costs little overhead
and performs well as the model size scales up.


Section 4.2 shows that AASD has the potential to
improve generation accuracy with a near relatively
strict threshold. In this section, we explore how the
verification threshold affects the generation results
through a case study. Table 7 displays the predictions under different thresholds for a sample in
2WikiMQA. Vanilla output fails to give the correct
answer. However, the prediction of AASD ( _δ_ =1e-3
and _δ_ =1e-5) contains the ground truth. With an
appropriate threshold, AASD improves the accuracy of generation through context-aware drafting
from the relevant context. When the threshold is
set to 1e-5, the logic of generated content begins to
lose. The prediction is completely messy when the
threshold comes to 1e-7. Therefore, the threshold
should be large enough to keep the model output
logical and fluent. Besides, too relaxed verification
may affect the model’s ability to defend against
attack inputs. If the input context contain harmful
information, using AASD may lead the model to
incorporate it into its responses. Therefore, in practical applications, AASD requires additional safety
alignment to ensure the security of the language
model. See Appendix A for a more comprehensive
study for the impact of verification strictness.


**Context Utilization** Using context to improve
generation performance is typical in retrievalaugmented generation (RAG) scenarios. RAG
methods (Zheng et al., 2023; Dai et al., 2023; Gao
et al., 2023; Fan et al., 2024; Zhao et al., 2024)
retrieves relevant documents based on the given


**Method** **Avg.**
F1 F1 F1 F1 ROUGE-L ROUGE-L Edit Sim Edit Sim


w/ threshold verification 32.05 **91.82** 41.19 **51.22** **15.16** 18.69 56.51 67.75 46.80
w/ top- _k_ verification 31.21 82.71 22.12 27.34 14.93 18.15 43.17 45.62 35.66
Table 4: Results for ablation study with LLaMA3.1-8B-Instruct. The best results are in bold.


MAL TPS Speed-up MAL TPS Speed-up MAL TPS Speed-up


w/o alignment sampling 1.80 36.12 1.80 1.87 32.93 1.63 2.25 42.71 2.07
w/o conditional verification 2.36 35.86 1.78 1.86 32.60 1.62 2.34 39.02 1.89
Table 5: Inference efficiency of AASD without alignment sampling and conditional verification.


**ADR** 7B - 1.00 34.32 1.00
**SD** 7B 68M 1.16 24.60 0.71


**ADR** 70B - 1.00 5.17 1.00
**SD** 70B 1B 1.14 2.68 0.52
**SD** 70B 7B **4.13** 6.61 1.28


Table 6: Comparison with speculative decoding with
big and small model collaboration. SD represents standard speculative decoding. Column _M_ and _Md_ indicates the target model size and draft model size.
input for model reference, thus improving the quality of generation. The naive approach (Ma et al.,
2023) applies the search engine as the retriever and
directly combines the retrieved documents with the
user query as the input for frozen LLMs. Most
RAG methods (Yoran et al., 2023; Luo et al., 2023;
Asai et al., 2023; Melz, 2023; Yan et al., 2024)
leveraging context on the input side to enhance generation. Self-RAG (Asai et al., 2023) introduces
generating reflection tokens to enable customizing
models’ behaviors for different tasks. Speculative
RAG (Wang et al., 2024b) adopts instruct-tuned
draft models to drafting according to different retrieved documents and uses the target model to pick
out the best draft as the final response. Apart from
them, CoG (Lan et al., 2023) proposes a encoderbased model architecture to seek suitable text spans
from the context during generation. Cao et al.
(2024) improves CoG through linguistic heuristics
initialization and iterative self-reinforcement.
**Speculative decoding** Stern et al. (2018);
Leviathan et al. (2023); Xia et al. (2023, 2024)
adopt a “drafting-verification” pattern to lossless
accelerates autoregressive decoding. At each decoding step, a small model is used to draft the
following few tokens. Then, the target model verifies the draft in parallel and accepts tokens that
are consistent with the original output. In this
way, multiple tokens can be generated in a single step. Some works (Leviathan et al., 2023; Cai
et al., 2024; Li et al., 2024) employ independent
small models or additional trained modules as the
draft models, while others (Saxena, 2023; Fu et al.,
2024; He et al., 2024) retrieve drafts from a draft
pool. The draft structure has evolved from n-grams
(Leviathan et al., 2023; Fu et al., 2024) to draft
trees (Li et al., 2024; Wang et al., 2024a). Among
them, PLD (Saxena, 2023) and LLMA (Yang et al.,
2023) retrieve n-grams from the prompt to construct the draft. REST (He et al., 2024) uses a
common public data source to build a draft pool
and retrieves the tree structure draft according to
the last several tokens of the current sequence at
each decoding step. These retrieval-based methods
avoid the hassle of additional training, but their
acceleration performance is relatively lower.


In this paper, we proposed AASD, an alignmentaugmented speculative decoding algorithm with
alignment sampling and conditional verification.
Both techniques improve the input-model alignment in prompt-based speculative decoding, thus
increasing the mean acceptance length and improving decoding efficiency. Conditional verification
based on a heuristic probability threshold shows the
potential to enhance the generation accuracy at the
token level. AASD does not introduce additional


Ground Truth **Mahesh Bhatt**
Greedy Pooja Bhatt (�)
AASD ( _δ_ =1e-3) Pooja Bhatt’s father is **Mahesh Bhatt** . (�)
AASD ( _δ_ =1e-5) Pooja Bhatt’s television film Daddy was directed by her father **Mahesh Bhatt** . (�)
AASD ( _δ_ =1e-7) Pooja Bhatt’s television film Daddy was directed by Pooja Bhatt, starring! the director’s father, played by
actor Anupam Kher (�)
Table 7: Comparison of generated results with AASD under different thresholds. The question is “Who is the father
of the director of film Kajraare?”
parameters or training, making it conveniently applicable to pre-trained LMs. It outperforms the
baseline with two different models on the average
score of 8 datasets in terms of generation performance and achieves a speed-up ratio of up to 2.23.


Due to resource constraints, we only consider applying alignment sampling on prompt-based speculative decoding in this paper. However, alignment
sampling is applicable to most existing retrievalbased speculative decoding (including REST) and
most generated scenarios. Taking REST as an example, if we input the draft pool into the model in
advance to obtain the output distribution of top-k
tokens (or directly use the content of the model’s
historical output to build the draft pool), then when
using REST, we can use alignment sampling to
improve the alignment of the draft pool with the
target model, thereby improving the draft token
acceptance rate and algorithm performance. We
leave this for future work.
If the input context contains harmful information,
using AASD may lead the model to incorporate it
into its responses. Therefore, in practical applications, AASD requires additional safety alignment
to ensure the security of the language model.


Figure 4: Mean acceptance length of different tasks
under threshold verification.
For non-strict verification, more relaxed verification conditions lead to faster reasoning, but
may also cause performance degradation. To study
the trade-off between accuracy and efficiency, we
conduct an experiment on LLaMA3.1-8B-Instruct
(Meta-AI, 2024) with threshold verification.
Figure 5 demonstrates the performance on each
dataset as the threshold changes. Note that alignment sampling is not applied. F1 score increases
with decreasing threshold for NQ while decreases
for other 3 QA datasets. The ground truth is implicit in the context for each sample in NQ. A more
relaxed verification condition makes the model tend
to use the original statement in the context, which
may be more accurate than the model’s output in
some cases, thus leading to better results. Therefore, for high-quality context, the threshold can
be lowered to increase confidence in the context
for better performance and efficiency. For both
summary datasets, the overall trend of the scores
decreases as the threshold decreases in the search
range. The performance drops rapidly for more
complex tasks such as code completion.
Figure 4 displays the mean acceptance length
under different thresholds. For all datasets, a lower
threshold leading to a higher acceptance rate, thus
improving the inference efficiency.


To explore the effectiveness of AASD in different
generation scenarios, we report the target length
and overlapping rate between the input and output
context of each dataset used in the main experiment.
We adopts longest substring match ratio (R) to evaluate the source-target overlapping rate, which is
calculated as:
_R_ = _[LCS]_ [(] _[X, Y]_ [ )] _,_ (7)
_|Y |_
where _LCS_ is length of the longest common subsequence of input context _X_ and target context _Y_ .
The average source-target overlapping rate and the
average target length of each dataset are shown in
Table 8. Even for QA tasks (e.g., TQA, 2WikiQA
and HotpotQA) where the average target length
is very short, AASD demonstrates effectiveness
according to the results in Table 1. On the other
hand, although AASD exhibits greater advantages
in tasks with high overlap, it remains effective even
in low-overlap scenarios, highlighting its versatility
across diverse generation tasks.


**Algorithm 1** AASD
**Input:** Prompt _q_, LM _M_, _α, β_, length of ngrams _n_, Maximum length of the key _l_ .
**Output:** Prediction _Aq_
1: _pool ←{}_
2: **for** _k_ = 1 to _l_ **do**
3: **for** _i_ = 0 to _len_ ( _q_ ) _−_ _k_ **do**
4: _key ←_ _q_ [ _i_ : _i_ + _k_ ]
5: _pool_ [ _key_ ] _.append_ ( _i_ + _k_ )
6: **end for**
7: **end for** _▷_ Initialize the draft pool.
8: **while** _⟨eos⟩_ not in _q_ **do**
9: T _←{q_ [ _−_ 1] _}_
10: **for** _k_ = _l_ to 1 **do**
11: **if** _q_ [ _−k_ :] _∈_ _pool_ **then**
12: T _←_ T + _Sample_ ( _q_ [ _pool_ [ _q_ [ _−k_ :]]])
_▷_ Conduct alignment sampling.
13: _break_
14: **end if**
15: **end for**
16: _P ←_ _M_ (T)
17: _y ←_ _V erify_ ( _P,_ T _, α, β_ )
18: _q ←_ _q_ + _y_
19: **for** _k_ = 1 to _l_ **do**
20: **for** _i_ = _−k −_ _len_ ( _y_ ) to _−k_ **do**
21: _key ←_ _q_ [ _i_ : _i_ + _k_ ]
22: _pool_ [ _key_ ] _.append_ ( _i_ + _k_ )
23: **end for**
24: **end for** _▷_ Update the draft pool.
25: **end while**
26: _Aq ←_ _q_


Input length 9681 8849 9458 3612 7882 8161 13516 15300
Target length 5 5 5 145 335 702 13 14
_R_ 0.13 0.77 0.82 1.00 0.43 0.54 0.71 0.42
Table 8: The target length and overlapping rate ( _R_ ) between the input and output context of each dataset used in the
main experiment.
Figure 5: Performance of different tasks under threshold verification. The dotted lines in each figure are the
baselines of the dataset with the corresponding colors.

TQA 2WikiQA HotpotQA NQ MultiNews GovReport LCC Repobench-P


Maintaining the acceleration at large batch sizes is
a fundamental challenge for speculative decoding
approaches. We show speed-up ratio of AASD on
SpecBench under different batch sizes in table 9.
**Batch size** 1 2 4 8
**Speed-up** 1.83 1.65 1.60 0.96
Table 9: Speed-up ratio of AASD on Specbench under
different batch sizes.
Increasing the batch size does lead to a decrease in the speedup ratio for speculative decoding.
AASD can accelerate inference with a batch size
of up to 4 in our experiment environment.

D Efficiency under Different Batch Sizes


To study under what circumstances AASD can improve accuracy, we examined multiple cases where
AASD responses were correct while greedy sampling produced errors, identifying three distinct
failure patterns, which are shown in Figure 6
Figure 6: Analysis for cases where AASD responses
were correct while greedy sampling produced errors.


**Gurpreet Singh** **[4]** **, Hasnat Md Abdullah** **[5]** **, Raghav Kaushik Ravi** **[6]** **, Vinija Jain** **[7]** _[†]_ **,**

Alignment Quality Index (AQI) : Beyond Refusals: AQI as an Intrinsic
  Alignment Diagnostic via Latent Geometry, Cluster Divergence, and Layer wise
  Pooled Representations

Abhilekh Borah
Chhavi Sharma
Danush Khanna
Utkarsh Bhatt
Gurpreet Singh
Hasnat Md Abdullah
Raghav Kaushik Ravi
Vinija Jain
Jyoti Patel
Shubham Singh
Vasu Sharma
Arpita Vats
Rahul Raja
Aman Chadha
Amitava Das


**1Manipal University Jaipur, India, 2LinkedIn, 3IIT Kharagpur, India, 4IIIT Guwahati, India,**
**5Texas A&M University, USA, 6Vellore Institute of Technology, Chennai, India, 7Meta AI,**

Rahul Raja [2] , Aman Chadha [10] _[‡]_ , Amitava Das [11]


Alignment is no longer a luxury; it is a necessity. As large language models (LLMs) enter
high-stakes domains like education, healthcare,
governance, and law, their behavior must reliably reflect human-aligned values and safety
constraints. Yet current evaluations rely heavily on behavioral proxies such as refusal rates,
G-Eval scores, and toxicity classifiers, all of
which have critical blind spots. Aligned models are often vulnerable to jailbreaking, stochasticity of generation and alignment faking. To
address this issue, we introduce the **Alignment**
**Quality Index (AQI)** . This novel geometric
and prompt-invariant metric empirically assesses LLM alignment by analyzing the separation of safe and unsafe activations in latent
space. By combining measures such as the
_Davies-Bouldin score (DBS)_, _Dunn index (DI)_,
_Xie-Beni index (XBI)_, and _Calinski-Harabasz_
_index (CHI)_ across various formulations, AQI
captures clustering quality to detect hidden
misalignments and jailbreak risks, even when
outputs appear compliant. AQI also serves as
an early warning signal for alignment faking,
offering a robust, decoding-invariant tool for
*These authors contributed equally to this work.
†Work does not relate to position at Meta.
‡Work does not relate to position at Amazon.
SectionWork does not relate to position at Linkedin.
behavior-agnostic safety auditing. Additionally, we propose the **LITMUS** dataset to facilitate robust evaluation under these challenging
conditions. Empirical tests on LITMUS across
different models trained under DPO, GRPO,
and RLHF conditions demonstrate AQI’s correlation with external judges and ability to reveal
vulnerabilities missed by refusal metrics. We
make our implementation publicly available to
foster future research in this area. [*]


As LLMs permeate sensitive domains like law,
healthcare, and policy-making, ensuring _reliable_
_alignment_ with human values is critical (Ouyang
et al., 2022; Askell et al., 2021). Current alignment evaluations rely heavily on _behavioral prox-_
_ies_ —refusal rates, G-Eval scores, and toxicity classifiers (OpenAI, 2023; Liu et al., 2023b; Gehman
et al., 2020). While convenient, these surface-level
metrics often fail to detect latent misalignments
that do not manifest in visible outputs.
Empirical findings suggest that LLMs can appear compliant while internally encoding unsafe or
adversarial tendencies (Hubinger and et al., 2021;
Carlsmith, 2023b). This underscores the need for
*https://anonymous.4open.science/r/aqi-eval-8F8B
a fundamentally different approach—one that inspects _representational geometry_ instead of relying solely on behavioral outputs. We propose
the _Alignment Quality Index (AQI)_, an intrinsic,
decoding-invariant diagnostic that evaluates latent
activations for alignment fidelity.
**Jailbreaking Vulnerability.** Despite fine-tuning,
safety-aligned models remain vulnerable to _jail-_
_breaking_ —adversarial prompts that bypass alignment safeguards (Zou et al., 2023; Liu et al., 2023c;
Anthropic, 2023). Techniques like few-shot policy
injection (Qi and et al., 2024) can generate unsafe
completions despite high refusal rates. While outputs may seem compliant, internal activations can
drift into unsafe regions. AQI detects this latent
shift by measuring the _separability_ of safe and
unsafe states.
**Stochastic Generation Instability.** LLMs are
inherently probabilistic; the same prompt may produce both safe and unsafe outputs depending on
decoding parameters (Hughes et al., 2025). Behavioral evaluations with deterministic decoding often
mask these risks. AQI mitigates this by analyzing
latent activations to capture _confidence margins_,
offering a scalable solution beyond surface-level
sampling.
**Alignment Faking Detection.** _Alignment faking_
allows models to simulate compliance while pursuing unsafe behaviors internally (Carlsmith, 2023b;
Hubinger et al., 2024; Greenblatt et al., 2024a). This
deception creates the illusion of alignment, masking
latent misalignments (Clymer et al., 2024b). Unlike
output-based metrics, AQI identifies these hidden
inconsistencies by probing latent geometry.
Intrinsic metrics like _AQI_ shift the focus from surface behavior to representational geometry, surfacing hidden vulnerabilities even when outputs seem
compliant (Greenblatt et al., 2024a). Its decodinginvariant design is robust against behavioral camouflage, adversarial prompts, and paraphrasing (Greenblatt et al., 2023a; Medlock et al., 2025), enabling
scalable, geometry-aware auditing.


We introduce the **AQI** —an intrinsic, geometryaware metric that quantifies how well a model separates _safe_ and _unsafe_ prompts in its latent space.
Unlike behavioral metrics such as refusal rate, toxicity, or G-Eval, which evaluate surface outputs,
AQI probes internal activations to diagnose true
alignment. AQI focuses on two core questions:


Preserving alignment during fine-tuning requires
identifying parameters most sensitive to safety behavior. Not all weights contribute equally: some
encode safety-critical functions (e.g., refusals),
others govern task-general behavior. We define
**alignment-critical parameters** as those whose
perturbation disproportionately alters a model’s
refusal response. Ignoring this sensitivity risks
degrading alignment, even under benign updates.
Recent mechanistic findings (Jain et al., 2024)
show that **safety fine-tuning (DPO) minimally**
**modifies MLP weights. See Figure 2** to steer
unsafe inputs into a “refusal” direction—often
aligned with the model’s null space-thus blocking
harmful output. This appears as:
_W_ ST = _W_ IT + ∆ _W,_
where _∥_ ∆ _W_ _∥≪∥W_ IT _∥_, yet ∆ _W_ exerts pivotal
effect. The top singular vectors of ∆ _W_ lie near the
null space of _W_ IT _[⊤]_ [, leaving benign inputs largely]
unchanged while sharply transforming unsafe activations. This localized transformation builds a
robust refusal mechanism—selective, minimal, and
behaviorally inert for safe prompts. However, adversarial examples orthogonal to ∆ _W_ ’s span may
Figure 2: Safety fine-tuning increases representational
separation between safe and unsafe prompts. Green and
red regions denote responses to safe and unsafe prompts.
Mean layer-wise separation score _τ_ is reported across
layers 1–6 for instruction-tuned, unlearning-tuned ( _ηM_ ),
and DPO-tuned ( _ηM_ ) models. Green and red denote
safe and unsafe samples, respectively.
evade detection, exposing vulnerabilities of linear
defenses. To disentangle safety-relevant learning
from task adaptation, we decompose the LoRA update:
∆ _W_ = _AB_ = ∆ _WA_ + ∆ _WT,_ _W_ = _W_ 0 + ∆ _W._
**Alignment-Critical Component (** ∆ _WA_ **):** Projected into a sensitive subspace via _PA_ ( _AB_ ), this
component is tightly regularized to preserve safety.
**Task-Specific Component (** ∆ _WT_ **):** The residual update ( _I −PA_ )( _AB_ ) captures task knowledge
and remains flexible.
This decomposition enables selective control:
safety is protected via constrained updates to ∆ _WA_,
while ∆ _WT_ supports continual learning. _Analogy:_
_W_ 0 is the blueprint, ∆ _W_ the renovation—updating
without touching structural safety beams.

2.1 Alignment-Critical Parameters - Preamble


_What does it mean for a model to be truly_
_aligned—not just in what it outputs, but in how_
_it thinks?_  - A model may reliably refuse unsafe
prompts or avoid toxic completions, yet these behaviors can be fragile under sampling, decoding
variation, or adversarial framing (Greenblatt et al.,
2023b; Zou et al., 2023). We propose a fundamentally different lens: inspecting whether alignment manifests in the model’s internal geometry.
Specifically, we ask: _Are safe and unsafe inputs_
_encoded in representationally distinct ways across_
_hidden layers?_ If alignment is real, it should leave
structural traces—detectable in the way activations
organize and cluster.
To this end, we introduce **AQI**, a decodinginvariant, behavior-agnostic metric that quantifies
the latent separability of safety-relevant inputs.
AQI builds on two core insights:
1. **Layer-Aware Semantics.** Alignment signals often emerge in intermediate MLP activations (Elhage et al., 2022a; Belrose et al., 2023), not solely
at the output layer. Final-layer representations suffer from over-smoothing and compression (Dong
et al., 2021; Kovaleva et al., 2021), masking useful
distinctions. AQI aggregates activations across a
band of informative layers to form semantically
aligned embeddings.
2. **Geometric Fidelity.** Rather than relying on behavioral outcomes, AQI evaluates cluster quality
in activation space—measuring intra-class compactness and inter-class separation using principled, unsupervised indices. This makes it robust
to prompt paraphrasing, decoding strategy, and
output-level camouflage.
Figure 3: **Latent Separation Captured by Xie–Beni Index (XBI).** 3D projections of safe (green) and unsafe (red)
activation clusters across layers. **Left:** Early layers show overlap. **Middle:** Mid layers begin to separate. **Right:**
Deeper layers exhibit clear partitioning, indicating alignment.
**Setup.** Let _XS_ and _XU_ denote activation vectors
for safe and unsafe prompts. For each input **x**, we
define its embedding as:
   _a_ ˆ( **x** ) = _wL · aL_ ( **x** ) _,_
_L∈L_
where _aL_ ( **x** ) is the post-GELU MLP activation at
layer _L_ and _wL_ is a layer weight. This yields a
fused embedding space _X_ = _XS_ _∪XU_ where safetyrelevant structure can be geometrically evaluated.
In what follows, we define AQI by combining the
strengths of the _Xie–Beni Index (XBI)_ and _Calinski–_
_Harabasz Index (CHI)_ —capturing local compactness and global dispersion—to assess whether alignment is not only expressed, but embedded.


The _Xie–Beni Index_ (Xie and Beni, 1991) quantifies cluster quality by balancing compactness and
separation:
ment and geometric confusion.


The _Calinski–Harabasz Index_ (Cali´nski and
Harabasz, 1974) measures cluster separability by
contrasting inter- and intra-cluster dispersion:
[Tr(] _[B][k]_ [)] _[n][ −]_ _[k]_
Tr( _Wk_ ) _[·]_ _k −_ 1
CHI = [Tr(] _[B][k]_ [)]
_k −_ 1 _[,]_
where Tr( _Bk_ ) and Tr( _Wk_ ) are the between- and
within-cluster scatter, respectively, for _k_ clusters
over _n_ points.
**Interpretation:** Higher CHI scores indicate wellseparated, coherent clusters—capturing global divergence across the representation space.