|ChrF++ Compared To MSA
0
5
0
5
0
5
0
FR ES AS D G H EI AL FX W US A IY ER AI B LX N S N N LE M M N RI G|Col2|Col3|Col4|hrF++ Comp|ared To MSA|Col7|Col8|
|---|---|---|---|---|---|---|---|
|FR
ES
AS
D
G
H
EI
AL
FX
W
US
A
IY
ER
AI
B
LX
N
S
N
N
LE
M
M
N
RI
G
0
5
0
5
0
5
0
ChrF++ Compared To MSA||||||||
|FR
ES
AS
D
G
H
EI
AL
FX
W
US
A
IY
ER
AI
B
LX
N
S
N
N
LE
M
M
N
RI
G
0
5
0
5
0
5
0
ChrF++ Compared To MSA||D
G
H|EI
AL
|FX
W
US
A
IY
ER
|AI
B
LX
N
S
N
N
LE
|M
M
N
RI
|G|
Figure 2: Average character-level similarity
(chrF++) between dialectal sentences and their
MSA counterparts, highlighting the high surface overlap that makes this a challenging generation task.
**Representational Geometry.** To understand _how_ models represent dialects, we analyze their internal geometry. We measure the **Geometric Separability** between sentence representations using
L2 and cosine distance, anchoring all comparisons to MSA representations. This allows us to quantify how distinct dialectal representations are from the high-resource standard. Furthermore, we
compute **Subspace Angles (SSA)** (M¨uller-Eberstein et al., 2023) to measure the alignment between
subspaces corresponding to different dialects. Smaller angles indicate greater alignment. This allows
Preprint. Work in Progress.
us to track how fine-tuning and our proposed interventions reshape the model’s internal organization
of linguistic information.
4.3 INFORMATION-THEORETIC PROBING
To complement the geometric analysis, we employ an information-theoretic variational linear probe
(Voita & Titov, 2020; M¨uller-Eberstein et al., 2023). The probe is a sparsity-regularized classifier
trained to identify a dialect from token-level representations. The resulting negative cross-entropy
provides a tight lower bound on the mutual information _I_ ( **h** [(] _[ℓ]_ [)] ; _Y_ ) between a model’s hidden states
and the dialect identity. This allows us to quantify how easily dialect-specific information can be
linearly decoded from the model’s representations, layer by layer, and how this changes during
training. Further details are in Appendix C.
4.4 CAUSAL INTERVENTION: ONLINE SUBSPACE DECOUPLING
To test the hypothesis that representational entanglement with a high-resource language harms lowresource generation, we introduce a novel training method: **Online Subspace Decoupling** . This
method acts as a causal intervention by actively discouraging dialectal representations from overlapping with the MSA subspace during fine-tuning.
The procedure is as follows:
1. **Identify MSA Subspace:** We train a variational linear probe (as in Sec. 4.3) to distinguish
MSA from all other dialects. We then use Singular Value Decomposition (SVD) on the
learned probe weights to extract an orthonormal basis **U** MSA for the MSA subspace and
form its projection matrix: **P** MSA = **U** MSA **U** _[⊤]_ MSA [.]
2. **Define Decoupling Loss:** During fine-tuning on the dialectal rewriting task, we add a
penalty term to the standard language modeling loss. This _decoupling loss_ penalizes the
magnitude of the projection of the model’s hidden states **H** onto the MSA subspace:
_L_ decouple = E [ _∥_ **HP** MSA _∥_ 2] (1)
The total loss is _L_ = _L_ LM + _λ L_ decouple, where _λ_ is a hyperparameter (we use 0.01).
Crucially, the probe is periodically retrained on fresh model checkpoints during fine-tuning. This **on-**
**line updating** of **P** MSA ensures that our intervention targets the evolving MSA subspace, enabling a
precise and adaptive causal manipulation of the model’s representational geometry. Training details
are in Appendix D.
4.5 EXPERIMENTAL SETUP
**Data.** All experiments use the MADAR 25 corpus (Bouamor et al., 2018), which contains 2,000
parallel sentences across 25 city-level Arabic dialects, MSA, English, and French. This fine-grained,
multi-dialect parallel resource is unique and enables our controlled study.
**Models.** We analyze a suite of state-of-the-art open-weight multilingual models: Jais-family 30B
(Sengupta et al., 2023), Gemma 3 1B (Team, 2025a), Aya expanse 8B (Dang et al., 2024), and Qwen
3 14B (Team, 2025b). For our causal intervention experiments, we deliberately select Gemma 3 1B.
Its smaller parameter count implies a more constrained representational space, making it a challenging and informative test case for the benefits of explicit subspace management. Furthermore,
its weaker baseline performance provides a clear opportunity to measure improvement from our
method.
5 RESULTS AND ANALYSIS
We now present our empirical investigation, which first diagnoses the representational pathologies
hindering dialectal generation in multilingual models and then validates our hypothesis with a causal
intervention.
Preprint. Work in Progress.
Figure 4: t-SNE of sentence representations. Higher-performing models (e.g., Qwen, Aya) exhibit clearer separation between dialectal clusters in their intermediate layers, unlike weaker models
(Gemma, Jais).
5.1 BASELINE: MODELS EXHIBIT POOR LOW-RESOURCE GENERATIVE CAPABILITIES
We first benchmark the zero-shot performance of several models on our dialectal rewriting task. As
shown in Figure 3, all models struggle significantly, with even the best-performing models, Qwen
3 14B and Aya Expanse 8B, achieving only modest chrF++ scores. Notably, performance does
not correlate with model scale or specialization; the largest, Arabic-centric Jais-family 30B model
is outperformed by smaller models. This poor performance, especially when contrasted with the
models’ strong capabilities in high-resource language pairs (MSA to English/French), points to a
more fundamental issue than a simple lack of capacity. We posit this stems from the model’s internal
representations.
5.2 DIAGNOSIS I: GEOMETRIC ANALYSIS LINKS PERFORMANCE TO REPRESENTATIONAL
SEPARATION
To investigate the underlying representational
geometry, we visualize the hidden states of parallel sentences using t-SNE (Figure 4). The visualizations reveal a striking pattern: stronger
models like Qwen and Aya learn to separate representations by dialect in their intermediate layers, whereas weaker models like
Jais and Gemma maintain entangled representations. This qualitative observation suggests a
link between a model’s ability to geometrically
isolate dialectal subspaces and its downstream
generative performance.
We quantify this by measuring the L2 and cosine distance between MSA and dialectal sentence representations across all layers (Figure 5). We observe that different distance met
Figure 3: Generative performance on the dialectal
rics capture different geometric properties: L2
rewriting task. All models perform poorly on low
distance reflects the degree of spatial separa
resource dialects compared to high-resource lan
tion, while cosine distance measures the align
guages, and performance does not correlate with
ment of subspaces. To substantiate the link to
model scale.
performance, we compute the layer-wise correlation between these distances and the chrF++ score (Figure 6). A consistent negative correlation
emerges between cosine distance and performance, especially in early-to-mid layers. This suggests that better alignment (lower cosine distance) in these layers is beneficial, likely facilitating the
transfer of semantic information from the high-resource MSA. Conversely, the relationship with L2
distance is more complex, with models like Aya benefiting from greater spatial separation in intermediate layers. This indicates a delicate balance: subspaces must be aligned enough for knowledge
transfer but separate enough to preserve unique dialectal features.
Preprint. Work in Progress.
Figure 5: Layer-wise L2 (Top) and Cosine (Bottom) distance between dialectal representations and
MSA. High-performing models show distinct geometric patterns, with Aya treating dialects more
like separate languages (high L2 distance).
Figure 6: Layer-wise Pearson correlation between representational distance from MSA (L2-Top,
Cosine-Bottom) and downstream generation performance. The consistent negative correlation with
cosine distance suggests that subspace alignment is beneficial.
5.3 DIAGNOSIS II: INFORMATION-THEORETIC EVIDENCE OF MSA’S REPRESENTATIONAL
DOMINANCE
The geometric analysis suggests entanglement with MSA is problematic. We further interrogate this
using information-theoretic probing during standard supervised fine-tuning (SFT) on the dialectal
rewriting task. We track the ELBO code length required to identify dialects from the model’s hidden
states (a proxy for how accessible this information is). As shown in Figure 7, standard fine-tuning
causes the code length for all dialects to increase slightly, as the model specializes for generation
rather than classification. However, the increase is **disproportionately large for MSA** . This indicates that the model is actively making MSA-specific information less linearly accessible, suggesting
its pre-trained MSA representation is oversized and detrimental to the dialectal generation task.
This “pruning” of the MSA subspace has a direct geometric consequence. As we fine-tune, the
Subspace Angle (SSA) between MSA and the dialectal subspaces consistently increases (Figure 8,
left). That is, the dialectal subspaces systematically drift away from the MSA subspace. Crucially,
this growing separation directly correlates with improvements in generation performance (Figure 8,
right).
Taken together, these analyses provide compelling correlational evidence for our central hypothesis:
the representational dominance of the high-resource standard language (MSA) actively hinders a
Preprint. Work in Progress.
|90
88
86
84
Dialect
Cairo
Rabat
Tunis
82 Beirut
Doha|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|
|---|---|---|---|---|---|---|---|---|
|82
84
86
88
90

Dialect
Cairo
Rabat
Tunis
~~Beirut~~
Doha|||||||||
|82
84
86
88
90

Dialect
Cairo
Rabat
Tunis
~~Beirut~~
Doha|||||||||
|82
84
86
88
90

Dialect
Cairo
Rabat
Tunis
~~Beirut~~
Doha|||||||||
|82
84
86
88
90

Dialect
Cairo
Rabat
Tunis
~~Beirut~~
Doha|Diale
Ca
Ra
Tu
|ct
iro
bat
nis
|||||||
|82
84
86
88
90

Dialect
Cairo
Rabat
Tunis
~~Beirut~~
Doha|~~B~~
Do|~~irut~~
ha|||||||
|Col1|Col2|Col3|Col4|Col5|Col6|
|---|---|---|---|---|---|
|.0
.5||||||
|.0
.5||||||
|.0
.5||||||
|.0
.5||||||
|.5
.0
.5|||||~~Dialec~~|
|.5
.0
.5|||||Cairo
Rabat
Tunis
|
|.0
|||||~~Beirut~~
Doha
English|
Figure 8: **(Left)** During SFT, the subspace angle (SSA) between MSA and dialects consistently
increases, indicating growing representational separation. **(Right)** This increase in separation correlates directly with improved chrF++ scores. This provides strong evidence that disentangling from
MSA is a key mechanism for improving dialectal generation.
model’s ability to generate text in related low-resource varieties. Fine-tuning implicitly alleviates
this by pushing dialectal representations away from the MSA subspace.
There are a few limitations to keep in mind of
our analyses so far. Namely, that they have
been observational and serve to build our hypothesis; the causal link is established specifically by the success of our decoupling intervention which we will discuss in Section 5.4.
Finally, the MADAR dataset, while unique in
its breadth of dialects, is composed of relatively
short sentences. This setting may not fully capture model behaviors on tasks requiring longerform generation, thereby defining the scope of
our current findings. We hope future work addresses this gap in data availability.
5.4 CAUSAL VALIDATION:
ONLINE SUBSPACE DECOUPLING BOOSTS
PERFORMANCE
|20.0|Col2|Col3|Col4|Col5|Col6|
|---|---|---|---|---|---|
|15.0
17.5
||||||
|15.0
17.5
||||||
|0
10000
20000
30000
40000
50000
60000
Checkpoint
0.0
2.5
5.0
7.5
10.0
12.5
||||||
|0
10000
20000
30000
40000
50000
60000
Checkpoint
0.0
2.5
5.0
7.5
10.0
12.5
||||||
|0
10000
20000
30000
40000
50000
60000
Checkpoint
0.0
2.5
5.0
7.5
10.0
12.5
||||||
|0
10000
20000
30000
40000
50000
60000
Checkpoint
0.0
2.5
5.0
7.5
10.0
12.5
||||||
|0
10000
20000
30000
40000
50000
60000
Checkpoint
0.0
2.5
5.0
7.5
10.0
12.5
||||||
|0
10000
20000
30000
40000
50000
60000
Checkpoint
0.0
2.5
5.0
7.5
10.0
12.5
||||||
Figure 7: Code Length evolution over baseline
training.
To move from correlation to causation, we test our hypothesis directly using our proposed **Online**
**Subspace Decoupling** method (Section 4.4). By adding an explicit penalty term that pushes dialectal hidden states out of the MSA subspace, we actively enforce the representational separation that
SFT appears to learn implicitly.
The results, shown in Figure 9, provide clear causal validation. Our intervention yields consistent
and significant performance gains across nearly all dialects, achieving a **+2.0 chrF++ average im-**
**provement** and gains as high as **+4.89** for specific dialects (Cairo) over a standard SFT baseline.
The dialects that benefit least (e.g., Tunis, Beirut) were already those with the highest initial separation from MSA (Figure 8), suggesting they were less affected by the entanglement problem. While
there is an expected trade-off, performance on MSA generation drops, the substantial boost for a
wide range of low-resource varieties confirms that mitigating representational dominance is a potent mechanism for improving generative capabilities. Visualizations (tSNE) in Appendix E confirm
that our method achieves a much greater degree of geometric separation than baseline SFT, directly
linking the intervention to its intended structural effect on the model’s internal representations.
Our analysis, while providing strong causal evidence, has several limitations that frame opportunities for future research. Our causal claim rests on intervention experiments within a single, albeit
complex, language family: Arabic. While we hypothesize that the underlying mechanism of repre
Preprint. Work in Progress.
|Col1|25.57 chrF++ Comparison by Dialect and Model Last Checkpoint|
|---|---|
|20
25|21.82

19.91
~~19.02~~

21.81

20.24
20.26
21.13
Last Checkpoint
~~Decoupled~~|
|Cairo
Doha
Beirut
Rabat
Tunis
Average
English (reverse)
0
5
10
15

chrF++ Score
16.92
18.49
17.94

18.43
17.78
15.75|16.92
18.49
17.94

18.43
17.78
15.75|
|Cairo
Doha
Beirut
Rabat
Tunis
Average
English (reverse)
0
5
10
15

chrF++ Score
16.92
18.49
17.94

18.43
17.78
15.75||
|Cairo
Doha
Beirut
Rabat
Tunis
Average
English (reverse)
0
5
10
15

chrF++ Score
16.92
18.49
17.94

18.43
17.78
15.75||
Figure 9: Our causal intervention (Online Decoupling, Orange) consistently improves performance
over baseline SFT (Blue) by actively enforcing representational separation.
sentational dominance is a general phenomenon, empirical validation on other language families is
necessary to confirm this.
6 CONCLUSION & FUTURE WORK
This work demonstrates that representational entanglement with a high-resource language is a critical and addressable bottleneck for generative modeling in closely-related, low-resource language varieties. Through a combination of geometric and information-theoretic analyses on Arabic dialects,
we provided evidence that the representational dominance of Modern Standard Arabic (MSA) hinders dialectal generation. We then moved from correlation to causation, introducing a novel _online_
_subspace decoupling_ method that actively and dynamically separates dialectal representations from
the MSA subspace during fine-tuning.
Our experiments provide the first causal evidence that explicitly managing this subspace overlap
yields substantial performance gains, up to +4.9 chrF++ on individual dialects and +2.0 on average,
validating our hypothesis. While our method was designed for hypothesis testing and is computationally intensive, its success illuminates a clear path forward. The results highlight the critical
importance of representational allocation in multilingual models and motivate future work in several key directions:
    - **Scalable and Efficient Methods:** Future research should focus on developing computationally cheaper alternatives that achieve similar decoupling effects. This includes designing parameter-efficient fine-tuning (PEFT) methods, such as subspace-aware adapters, or
formulating novel pre-training objectives that encourage a more balanced representational
space from the outset.
    - **Inference-Time Interventions:** A particularly promising avenue is to move beyond
training-based solutions. Inference-time techniques like activation steering or targeted
model editing could offer a more surgical and efficient approach. For instance, identifying
and patching neurons responsible for MSA-specific features could suppress the dominant
language’s influence on-the-fly, without requiring any gradient-based updates.
    - **Generalization to Other Languages:** A crucial next step is to investigate whether these
principles of representational entanglement generalize beyond Arabic. Applying our analytical framework and interventions to other language families with similar resource imbalances and orthographic overlap, such as the Czech-Slovak or Scandinavian language
continuums, is essential for establishing the broader utility of subspace management in
multilingual representation learning.
ACKNOWLEDGMENTS
This work was conducted during Ahmed Elshabrawy’s research internship at NICT, Japan. We
gratefully acknowledge the support and computational resources provided by NICT, Japan that made
this research possible.
Preprint. Work in Progress.
REFERENCES
Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max L¨ubbering, Johannes
Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, Charvi Jain, Alexander Weber, Lena Jurkschat, Hammam Abdelwahab, Chelsea John, Pedro Ortiz Suarez, Malte Ostendorff, Samuel Weinbach, Rafet Sifa, Stefan Kesselheim, and Nicolas Flores-Herr. Tokenizer choice for LLM training: Negligible or crucial? In Kevin Duh, Helena Gomez, and
Steven Bethard (eds.), _Findings of the Association for Computational Linguistics: NAACL 2024_,
pp. 3907–3924, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.findings-naacl.247. [URL https://aclanthology.org/2024.](https://aclanthology.org/2024.findings-naacl.247/)
[findings-naacl.247/.](https://aclanthology.org/2024.findings-naacl.247/)
Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat
Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, Kelly Marchisio, Max Bartolo, Sebastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Aidan Gomez, Phil Blunsom, Marzieh
Fadaee, Ahmet Ust¨un, and Sara Hooker. Aya 23: Open weight releases to further multilingual [¨]
[progress, 2024. URL https://arxiv.org/abs/2405.15032.](https://arxiv.org/abs/2405.15032)
Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer.
The MADAR Arabic dialect corpus and lexicon. In Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph
Mariani, H´el`ene Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga
(eds.), _Proceedings of the Eleventh International Conference on Language Resources and Eval-_
_uation (LREC 2018)_, Miyazaki, Japan, May 2018. European Language Resources Association
[(ELRA). URL https://aclanthology.org/L18-1535.](https://aclanthology.org/L18-1535)
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are
few-shot learners. _Advances in neural information processing systems_, 33:1877–1901, 2020.
Tyler Chang, Zhuowen Tu, and Benjamin Bergen. The geometry of multilingual language model
representations. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), _Proceedings of_
_the 2022 Conference on Empirical Methods in Natural Language Processing_, pp. 119–136,
Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis[tics. doi: 10.18653/v1/2022.emnlp-main.9. URL https://aclanthology.org/2022.](https://aclanthology.org/2022.emnlp-main.9)
[emnlp-main.9.](https://aclanthology.org/2022.emnlp-main.9)
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam
Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm:
Scaling language modeling with pathways. _arXiv preprint
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek,
Francisco Guzm´an, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. _arXiv preprint
John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian
Strub, Nathan Grinsztajn, Yannis Flet-Berliac, Acyr Locatelli, Hangyu Lin, Dwarak Talupuru,
Bharat Venkitesh, David Cairuz, Bowen Yang, Tim Chung, Wei-Yin Ko, Sylvie Shang Shi,
Amir Shukayev, Sammie Bae, Aleksandra Piktus, Roman Castagn´e, Felipe Cruz-Salinas, Eddie Kim, Lucas Crawhall-Stein, Adrien Morisot, Sudip Roy, Phil Blunsom, Ivan Zhang, Aidan
Gomez, Nick Frosst, Marzieh Fadaee, Beyza Ermis, Ahmet Ust¨un, and Sara Hooker. Aya ex- [¨]
[panse: Combining research breakthroughs for a new multilingual frontier, 2024. URL https:](https://arxiv.org/abs/2412.04261)
[//arxiv.org/abs/2412.04261.](https://arxiv.org/abs/2412.04261)
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson.
Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. _arXiv preprint
Preprint. Work in Progress.
Karima Kadaoui, Samar Magdy, Abdul Waheed, Md Tawkat Islam Khondaker, Ahmed El-Shangiti,
El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. TARJAMAT: Evaluation of bard
and ChatGPT on machine translation of ten Arabic varieties. In Hassan Sawaf, Samhaa ElBeltagy, Wajdi Zaghouani, Walid Magdy, Ahmed Abdelali, Nadi Tomeh, Ibrahim Abu Farha,
Nizar Habash, Salam Khalifa, Amr Keleg, Hatem Haddad, Imed Zitouni, Khalil Mrini, and Rawan
Almatham (eds.), _Proceedings of ArabicNLP 2023_, pp. 52–75, Singapore (Hybrid), December
2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.arabicnlp-1.6. URL
[https://aclanthology.org/2023.arabicnlp-1.6.](https://aclanthology.org/2023.arabicnlp-1.6)
Anjali Kantharuban, Ivan Vuli´c, and Anna Korhonen. Quantifying the dialect gap and its correlates
across languages. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Findings of the Associ-_
_ation for Computational Linguistics: EMNLP 2023_, pp. 7226–7245, Singapore, December 2023.
Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.481. URL
[https://aclanthology.org/2023.findings-emnlp.481.](https://aclanthology.org/2023.findings-emnlp.481)
Takeshi Kojima, Itsuki Okimura, Yusuke Iwasawa, Hitomi Yanaka, and Yutaka Matsuo. On the multilingual ability of decoder-based pre-trained language models: Finding and controlling languagespecific neurons. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), _Proceedings of the_
_2024 Conference of the North American Chapter of the Association for Computational Lin-_
_guistics: Human Language Technologies (Volume 1: Long Papers)_, pp. 6919–6971, Mexico
City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.
[naacl-long.384. URL https://aclanthology.org/2024.naacl-long.384.](https://aclanthology.org/2024.naacl-long.384)
Max M¨uller-Eberstein, Rob van der Goot, Barbara Plank, and Ivan Titov. Subspace chronicles: How linguistic information emerges, shifts and interacts during language model training. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Findings of the Association for_
_Computational Linguistics: EMNLP 2023_, pp. 13190–13208, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.879. URL
[https://aclanthology.org/2023.findings-emnlp.879/.](https://aclanthology.org/2023.findings-emnlp.879/)
El Moatez Billah Nagoudi, AbdelRahim Elmadany, Ahmed El-Shangiti, and Muhammad AbdulMageed. Dolphin: A challenging and diverse benchmark for Arabic NLG. In Houda Bouamor,
Juan Pino, and Kalika Bali (eds.), _Findings of the Association for Computational Linguistics:_
_EMNLP 2023_, pp. 1404–1422, Singapore, December 2023. Association for Computational Lin[guistics. doi: 10.18653/v1/2023.findings-emnlp.98. URL https://aclanthology.org/](https://aclanthology.org/2023.findings-emnlp.98)
[2023.findings-emnlp.98.](https://aclanthology.org/2023.findings-emnlp.98)
Hellina Nigatu, Atnafu Tonja, and Jugal Kalita. The less the merrier? investigating language representation in multilingual models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Find-_
_ings of the Association for Computational Linguistics: EMNLP 2023_, pp. 12572–12589, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.
findings-emnlp.837. [URL https://aclanthology.org/2023.findings-emnlp.](https://aclanthology.org/2023.findings-emnlp.837)
[837.](https://aclanthology.org/2023.findings-emnlp.837)
Maja Popovi´c. chrF: character n-gram F-score for automatic MT evaluation. In Ondˇrej Bojar,
Rajan Chatterjee, Christian Federmann, Barry Haddow, Chris Hokamp, Matthias Huck, Varvara
Logacheva, and Pavel Pecina (eds.), _Proceedings of the Tenth Workshop on Statistical Machine_
_Translation_, pp. 392–395, Lisbon, Portugal, September 2015. Association for Computational Lin[guistics. doi: 10.18653/v1/W15-3049. URL https://aclanthology.org/W15-3049.](https://aclanthology.org/W15-3049)
Hassan Sajjad, Ahmed Abdelali, Nadir Durrani, and Fahim Dalvi. AraBench: Benchmarking dialectal Arabic-English machine translation. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.),
_Proceedings of the 28th International Conference on Computational Linguistics_, pp. 5094–5107,
Barcelona, Spain (Online), December 2020. International Committee on Computational Linguis[tics. doi: 10.18653/v1/2020.coling-main.447. URL https://aclanthology.org/2020.](https://aclanthology.org/2020.coling-main.447)
[coling-main.447.](https://aclanthology.org/2020.coling-main.447)
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´c, Daniel Hesslow, Roman
Castagn´e, Alexandra Luccioni, Franc¸ois Yvon, Matthias Gall´e, et al. Bloom: A 176b-parameter
open-access multilingual language model. _arXiv preprint
Preprint. Work in Progress.
Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William
Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong
Han, Sondos Mahmoud Bsharat, Alham Fikri Aji, Zhiqiang Shen, Zhengzhong Liu, Natalia
Vassilieva, Joel Hestness, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, Hector Xuguang Ren, Preslav Nakov, Timothy Baldwin, and Eric Xing. Jais and jais-chat: Arabiccentric foundation and instruction-tuned open generative large language models, 2023.
Cheril Shah, Yashashree Chandak, Atharv Mahesh Mane, Benjamin Bergen, and Tyler A. Chang.
Correlations between multilingual language model geometry and crosslingual transfer performance. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti,
and Nianwen Xue (eds.), _Proceedings of the 2024 Joint International Conference on Computa-_
_tional Linguistics, Language Resources and Evaluation (LREC-COLING 2024)_, pp. 4059–4066,
[Torino, Italia, May 2024. ELRA and ICCL. URL https://aclanthology.org/2024.](https://aclanthology.org/2024.lrec-main.361)
[lrec-main.361.](https://aclanthology.org/2024.lrec-main.361)
[Gemma Team. Gemma 3. 2025a. URL https://goo.gle/Gemma3Report.](https://goo.gle/Gemma3Report)
[Qwen Team. Qwen3 technical report, 2025b. URL https://arxiv.org/abs/2505.09388.](https://arxiv.org/abs/2505.09388)
Elena Voita and Ivan Titov. Information-theoretic probing with minimum description length. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), _Proceedings of the 2020 Conference on_
_Empirical Methods in Natural Language Processing (EMNLP)_, pp. 183–196, Online, November
2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.14. URL
[https://aclanthology.org/2020.emnlp-main.14/.](https://aclanthology.org/2020.emnlp-main.14/)
Weixuan Wang, Barry Haddow, Minghao Wu, Wei Peng, and Alexandra Birch. Sharing matters:
[Analysing neurons across languages and tasks in llms, 2024. URL https://arxiv.org/](https://arxiv.org/abs/2406.09265)
[abs/2406.09265.](https://arxiv.org/abs/2406.09265)
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey,
Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa,
Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa,
Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google’s neural
machine translation system: Bridging the gap between human and machine translation. _CoRR_,
[abs/1609.08144, 2016. URL http://arxiv.org/abs/1609.08144.](http://arxiv.org/abs/1609.08144)
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya
Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In
Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven
Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), _Proceedings of the 2021_
_Conference of the North American Chapter of the Association for Computational Linguistics:_
_Human Language Technologies_, pp. 483–498, Online, June 2021. Association for Computational
[Linguistics. doi: 10.18653/v1/2021.naacl-main.41. URL https://aclanthology.org/](https://aclanthology.org/2021.naacl-main.41)
[2021.naacl-main.41.](https://aclanthology.org/2021.naacl-main.41)
Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. MultiVALUE: A framework for cross-dialectal English NLP. In Anna Rogers, Jordan Boyd-Graber,
and Naoaki Okazaki (eds.), _Proceedings of the 61st Annual Meeting of the Association for Com-_
_putational Linguistics (Volume 1: Long Papers)_, pp. 744–768, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.44. [URL https:](https://aclanthology.org/2023.acl-long.44)
[//aclanthology.org/2023.acl-long.44.](https://aclanthology.org/2023.acl-long.44)
A TOKENIZER FERTILITIES
To check how dialects are handled differently to MSA at the most basic level by models we examine
various tokenizer fertilities. We include tokenizers of all the model families we explore. Furthermore, we train custom WordPiece tokenizers with an 8,000 vocabulary size (Wu et al., 2016) on
Preprint. Work in Progress.
Dialect #Unique Words
DOH 12,651
BEI 15,083
MSA 12,197
CAI 14,611
RAB 16,136
TUN 15,437
Table 2: Unique words
Figure 10: Tokenizer fertilities of various language model families, as well as a WordPiece tokenizer
trained from scratch on a specifc dialectal set. The red bar corresponds to MSA.
the train and dev set of the MADAR 25 corpus of 6 different dialects/varieties (each having 10,000
sentences in total).
As seen in Figure 10, there are several interesting observations. First, it is clear that there is a divide
between MSA, English, and French and the non-standard dialects of Arabic which have significantly
higher fertilities.
Furthermore, Across **all** tokenizers, dialects tend to have much higher fertilities to MSA. Interestingly, even for tokenizers trained on a single dialect, the fertility of MSA tends to be close or even
lower than the dialect it was trained on. This seems to indicate that MSA serves a lowest common
denominator for Arabic tokenizers whereas dialects can have more foreign/disjoint vocabularies
(i.e. most dialectal varieties will feature a minimum MSA lexicon with other elements which are not
shared). This is consistent with the linguistic intuition and is supported by the unique word counts
of the train sets of all the dialectal tokenizers show below:
Interestingly, we note however that training on a single dialect seems to lead to lower fertilites on all
other dialects. Especially when comparing to the LLM and MSA-only tokenizers observed. This is
indicative that a majority of the training data used to train the multilingual tokenizers of all models
observed is lacking in dialectal data.
This highlights that before any processing is done by models dialectal words are inherently treated
differntly at the lowest stage of tokenization. Furthermore, training tokenizers on MSA data only is
not sufficient for tokenizers to capture dialectal words sufficiently.
Preprint. Work in Progress.
|City|Code|
|---|---|
|Rabat
Fes
Algiers
Tunis
Sfax
Tripoli
Benghazi
Cairo
Alexandria
Aswan
Khartoum
Jerusalem
Amman
Salt
Beirut
Damascus
Aleppo
Mosul
Baghdad
Basra
Doha
Muscat
Riyadh
Jeddah
Sana’a|RAB
FES
ALG
TUN
SFX
TRI
BEN
CAI
ALX
ASW
KHA
JER
AMM
SAL
BEI
DAM
ALE
MOS
BAG
BAS
DOH
MUS
RIY
JED
SAN|
Table 3: City Names and Their Codes
B CITY NAMES TO DIALECT CODE
C MORE INFORMATION ABOUT PROBING
To complement geometric subspace analysis, we adopt an information-theoretic variational linear
probe (Voita & Titov, 2020; M¨uller-Eberstein et al., 2023) to quantify how much dialect identity information is recoverable from token-level model representations. For a given token, let
_{_ **h** [(0)] _, . . .,_ **h** [(] _[ℓ]_ [)] _} ∈_ R _[d]_ denote its hidden states from all _ℓ_ layers, including the non-contextualized
layer 0. The probe computes a learned weighted average over layers:
**h** _[′]_ =
_ℓ_
- _αi_ **h** [(] _[i]_ [)] _,_
_i_ =0
where _**α**_ _∈_ R _[ℓ]_ are learned combination weights.
This aggregated representation is fed to a linear classifier with weight matrix _**θ**_ _∈_ R _[d][×][c]_ for _c_ dialect
classes. Following Voita & Titov (2020), each weight _w_ in _**θ**_ is drawn from a normal distribution
_w ∼N_ ( _zµ, z_ [2] _σ_ [2] ) _,_
where the scaling factor _z_ is also drawn from
_z ∼N_ ( _µz, σz_ [2][)] _[.]_
The pair ( _w, z_ ) is given a joint normal–Jeffreys prior
_γ_ ( _w, z_ ) _∝|z|_ _[−]_ [1] _N_ ( _w |_ 0 _, z_ [2] )
which encourages sparsity by pushing weights toward zero with low variance.
The probe parameters ( _**α**_ _,_ _**θ**_ ) are trained to minimize
_L_ = CE( _y,_ ˆ _y_ ) + _β D_ KL( _q_ ( _**θ**_ ) _∥_ _γ_ ( _**θ**_ )) _,_
Preprint. Work in Progress.
where CE is the cross-entropy loss for one-vs-rest dialect classification, and the KL term regularizes _**θ**_ toward the sparsity-inducing prior. This objective maximizes compression while preserving
predictive accuracy, yielding a layer-combined, token-level estimate of recoverable dialect identity
information. The one-vs-rest objective hones in on dialect specific information that can help the
model discern between similar dialects and offers counter-examples. We construct the training set
for each dialect/variety/language by taking all the target’s sentences in MADAR 26’s training set, we
construct an equal number of counter-examples from all the other dialects and languages. We make
this data available (anonymized). We include training hyperparameters for the probes in Table 4.


Model name google/gemma-3-1b-pt
KL weight 1.0
Number of epochs 30 (for analysis)
15 (for decoupling training)
Early stopping patience 5
Table 4: Training hyperparameters for variational probe experiments.
D ONLINE DECOUPLING TRAINING DETAILS
This appendix outlines the key design decisions underlying our online MSA subspace decoupling
method, as well as the exact hyperparameters used in our experiments.
D.1 DESIGN CHOICES
**Projection Subspace Estimation.** We estimate the MSA subspace using a _variational linear_
_probe_ trained on a Modern Standard Arabic (MSA) vs. non-MSA dialect identification task over
the MADAR corpus. We recover the subspace basis from the learned probe parameters using Singular Value Decomposition (SVD) of the parameter matrix _θ_ MSA. The number of retained singular
vectors equals the probe’s latent dimension.
**Online Updating.** Rather than estimating the MSA subspace once before training, we periodically retrain the probe on the current model checkpoint during fine-tuning. This ensures that the
projection matrix **P** MSA remains synchronized with the evolving hidden representation geometry.
The projection matrix is updated every _N_ update gradient steps.
**Layer Aggregation.** Hidden representations from all layers are combined using a learned set of
attention weights _α ∈_ R _[L]_ [+1] from the variational probe. This allows the method to focus the decoupling penalty on layers most predictive of MSA features.
**Penalty Formulation.** We penalize the _ℓ_ 2 norm of the projection of the aggregated hidden states
onto the MSA subspace:
_L_ decouple = E [ _∥_ **HP** MSA _∥_ 2] _,_ (2)
where **H** are the contextual hidden states and **P** MSA is the projection matrix.
**Loss Weighting.** The decoupling penalty is scaled by a coefficient _λ_ and added to the standard
causal language modeling loss:
_L_ = _L_ LM + _λ · L_ decouple _._ (3)
**Bidirectional Training Data.** To encourage symmetric modeling of both MSA _→_ dialect and
dialect _→_ MSA directions, we construct bidirectional rewriting prompts for each sentence pair.
D.2 HYPERPARAMETERS
E DECOUPLING RESULTS FOR ALL DIALECTS
In Figure 11, we show the comparison of the baseline SFT and the decoupling method for all 25 dialects. The overwhelming majority of dialects benefit from our novel training. Aleppo, Alexandria,
Preprint. Work in Progress.

When Alignment Hurts: Decoupling Representational Spaces in Multilingual Models

Ahmed Elshabrawy
Hour Kaing
Haiyue Song
Alham Fikri Aji
Hideki Tanaka
Masao Utiyama
Raj Dabre


Base model google/gemma-3-1b-pt
Tokenizer Matching HF tokenizer (pad ~~t~~ oken = eos ~~t~~ oken)
Batch size (per device) 1
Gradient accumulation steps 4
Max sequence length 512
Optimizer AdamW (via HF Trainer default)
Learning rate 5 _×_ 10 _[−]_ [5] (default HF schedule)
Loss coefficient _λ_ 0.01
Probe update steps _N_ update 500
Probe training epochs 15
Probe dataset anonymous dataset (DID-MSA)
Probe input type Sequence-level dialect identification
Number of probe classes 2 (MSA vs. non-MSA)
Projection estimation SVD on _θ_ MSA
Subspace dimensionality Full rank of _θ_ MSA
Layer aggregation Learned attention weights _α_
Early stopping patience 3 epochs (validation loss)
Early stopping threshold 0.01
Train/validation split 90% / 10%
Table 5: Hyperparameters used in online MSA decoupling experiments.
|25|21.88 23.64 21.82 22.55 22.5822.11 23.00 21.98 21.6021.3021.67 24.59 21.66 22.89 23.88 23.81 21.81 22.88 25.57 21.93 24.7324.75 23.0623.21 21.45 24.80 22.53 22.60 22.58 Last Checkpoint Decoupled|
|---|---|
|0|20.59
21.88
20.4120.1620.4320.07
18.49
21.04
16.92
20.61
21.82

21.20

20.93
18.58
19.91

18.3718.22
21.20
17.94
20.59
18.43
21.6021.3021.67
21.66
20.24
21.81
21.93
21.45
20.26

~~18.59~~

17.78

15.75|
|0
5||
|0
5||
|Aleppo
exandria
Algiers
Amman
Aswan
Baghdad
Basra
Beirut
Benghazi
Cairo
amascus
Doha
Fes
Jeddah
erusalem
hartoum
Mosul
Muscat
Rabat
Riyadh
Salt
Sanaa
Sfax
Tripoli
Tunis
Average
 (reverse)
0
5||
Figure 11: Baseline ChrF++ (Blue) vs. our novel online decoupling method (Orange) on Multiple
dialects and on English to MSA translation.
Fes, Rabat, Sfax, and Tunis seem to benefit the least from the method seeing barely any changes
when compared to baseline SFT. Algiers seems to be the only dialect (aside from MSA) that is actively hurt from the intervention. It is difficult to explain why exactly it stands out; however, given
that the overwhelming majority dialects seem to benefit from the intervention it seems to be overall
beneficial to DiaMT.
F TSNE POST SFT TRAINING
Preprint. Work in Progress.
Figure 12: Visualizing internal model representations every 5 layers across 2 different sentences
(Sentence 0 in blue and Sentence 1 in orange) written in 6 varieties of Arabic using t-SNE. Layer
spaces are approximated with ellipses with a color gradient from layer 0 (blue) to the last layer (red).
Left is our novel decoupling training, center is the base model, right is the baseline SFT training on
DiaMT.


**Houxing Ren** [1] **Zimu Lu** [1] **Weikang Shi** [1] **Haotian Hou** [2] _[,]_ [5] **Yunqiao Yang** [1]
**Ke Wang** [1] **Aojun Zhou** [1] **Junting Pan** [1] _[,]_ [3] **Mingjie Zhan** [2][*] **Hongsheng Li** [1] _[,]_ [3] _[,]_ [4][*]
1CUHK MMLab 2SenseTime Research 3CPII under InnoHK
4Shanghai AI Laboratory 5Beihang University
renhouxing@gmail.com zhanmingjie@sensetime.com hsli@ee.cuhk.edu.hk

Alignment with Fill-In-the-Middle for Enhancing Code Generation

Houxing Ren
Zimu Lu
Weikang Shi
Haotian Hou
Yunqiao Yang
Ke Wang
Aojun Zhou
Junting Pan
Mingjie Zhan
Hongsheng Li


The code generation capabilities of Large
Language Models (LLMs) have advanced applications like tool invocation and problemsolving. However, improving performance in
code-related tasks remains challenging due to
limited training data that can be verified with
accurate test cases. While Direct Preference
Optimization (DPO) has shown promise, existing methods for generating test cases still
face limitations. In this paper, we propose a
novel approach that splits code snippets into
smaller, granular blocks, creating more diverse
DPO pairs from the same test cases. Additionally, we introduce the Abstract Syntax
Tree (AST) splitting and curriculum training
method to enhance the DPO training. Our
approach demonstrates significant improvements in code generation tasks, as validated
by experiments on benchmark datasets such
as HumanEval (+), MBPP (+), APPS, LiveCodeBench, and BigCodeBench. Code and
[data are available at https://github.com/](https://github.com/SenseLLM/StructureCoder)
[SenseLLM/StructureCoder.](https://github.com/SenseLLM/StructureCoder)


Large Language Models (LLMs) have significantly enhanced applications like tool invocation
and mathematical problem-solving (OpenAI, 2023;
Touvron et al., 2023; Jiang et al., 2024; Yang et al.,
2024). To improve LLM performance, a common approach involves two stages: supervised finetuning (SFT) and alignment. The alignment phase
is particularly effective, with Direct Preference Optimization (DPO) (Rafailov et al., 2023) gaining
popularity for its simplicity and effectiveness. DPO
has been successfully applied in fields like dialogue (Dubey et al., 2024), faithfulness (Bi et al.,
2024), and reasoning (Lai et al., 2024).
However, despite its success in several domains,
DPO has shown limited improvement in code generation tasks, and in some cases, its application
*Corresponding author.
may even be counterproductive (Xu et al., 2024).
This limitation may stem from the scarcity of training data containing test cases (Zhang et al., 2024a),
as seen in training sets like APPS (Hendrycks et al.,
2021), which include only 5,000 samples. To address this, PLUM (Zhang et al., 2024a) and CodeDPO (Zhang et al., 2024b) have proposed methods
for constructing test cases to expand the training
dataset for DPO. However, these approaches still
face criticism for their inability to fully guarantee the accuracy of test samples, which limits the
achievable performance improvements.
To overcome these challenges, we propose a
novel approach that makes more efficient use of
the limited, high-quality data available. Inspired
by recent advancements in the mathematics domain (Lightman et al., 2024; Lu et al., 2024b,a;
Lai et al., 2024), we decompose code snippets into
smaller, more granular blocks. This allows for
the construction of more detailed and diverse DPO
pairs. By treating each block as a target for prediction, the model can fine-tune on smaller code
segments, enabling more effective learning and potentially improving performance in code generation tasks. Furthermore, this approach generates a
larger number of DPO pairs, making better use of
the available training set.
Building on these insights, we introduce a
novel method called StructureCoder, which focuses on maximizing the utility of limited training data. Specifically, we leverage the fill-in-themiddle (FIM) approach (Bavarian et al., 2022), an
important capability of code LLMs for code completion, to generate fine-grained DPO pairs. By
using FIM, we can divide a code snippet into multiple blocks by Abstract Syntax Tree (AST), each
serving as a target, and then prompt the model to
generate the missing block. Test cases are used
to evaluate the correctness of the generated block,
thereby constructing accurate DPO pairs. Additionally, we propose a curriculum training method that
Figure 1: A preference pair case in the code generation field. The left is the correct response, and the right is the
incorrect response. The only difference between the two responses is in Line 16.
organizes the training set according to the depth
of the target block. This method progressively increases training difficulty, leading to better performance.
Our contributions can be summarized as follows:
 - We propose the use of FIM to enhance DPO
for code LLMs, enabling the effective use of
limited training data with test cases.
 - We introduce a method for generating accurate
FIM data through AST-based block segmentation, and a curriculum training strategy to organize the training set by target block depth,
leading to improved performance.
 - We conduct extensive experiments on HumanEval (+), MBPP (+), APPS, LiveCodeBench, and BigCodeBench to demonstrate
the effectiveness of the proposed method in
code-related tasks.


In this section, we introduce the fundamental components of DPO and FIM, followed by an analysis
of the limitations of DPO in code generation tasks.


DPO (Rafailov et al., 2023) offers a solution that
bypasses reward model training, instead directly
fine-tuning the LLM using preference pairs. The
DPO loss is defined as
_LDP O_ ( _yw, yl, x_ ) =
where _yw_ is the preferred response and _yl_ is the
dispreferred response.


When performing FIM (Bavarian et al., 2022) training, we swap the middle and the suffix segments.
Specifically, after splitting, it moves the suffix segment before the middle segment:
code _→_ (pre, mid, suf) _→_ (pre, suf, mid) _._
Then concatenate the three pieces using special
tokens as
<PRE> pre <SUF> suf <MID> mid <EOT> _._
After that, we use the format to train a causal language model and use the prefix and suffix segments
to predict the middle segment in the inference
stage.


Prior work (Rafailov et al., 2024) demonstrates that
DPO can implicitly learn token-level reward functions within the Markov Decision Process framework of large language models. This enables DPOtrained models to assign differentiated rewards to
individual tokens, effectively identifying those associated with factual inaccuracies while maintaining consistent rewards elsewhere. However, accurately capturing such fine-grained reward signals
requires substantial training data, as the quality and
scale of the dataset critically influence the learned
reward and the performance.
_−_ log _σ_ _β_ log _[π][θ]_ [(] _[y][w][ |][ x]_ [)]
_πref_ ( _yl | x_ )
_[π][θ]_ [(] _[y][w][ |][ x]_ [)] _[π][θ]_ [(] _[y][l][ |][ x]_ [)]
_πref_ ( _yw | x_ ) _[−]_ _[β]_ [ log] _πref_ ( _yl | x_
_,_
This ability introduces two challenges for DPO
in the code field. On the one hand, there are limited
training sets containing test cases. On the other
hand, the preferred and dispreferred responses often share a similar structure, with only minimal differences in detailed expression. This phenomenon
causes DPO to need a larger training data set to
learn token-level rewards. This is because when
constructing DPO training data, a model is used to
generate multiple responses, which are then evaluated against test cases. This method often results
in correct responses and incorrect responses being
nearly identical, differing only in small details such
as a specific expression, an if-block, or a function.
To illustrate the consequences of this scenario,
we present an extreme example, as shown in Figure 1. In this case, we assume that only a single
segment differs between a correct response _yw_ and
an incorrect response _yl_ . These sequences are defined as
_yw_ = _{_ pre _,_ mid _w,_ suf _},_
_yl_ = _{_ pre _,_ mid _l,_ suf _}._
Here, the prefix is the same for both sequences, so
the loss for the prefix is zero. The DPO loss can
then be expressed as
_L_ ( _yw, yl, x_ ) = _LDPO_ (mid _w|_ suf _,_ mid _l|_ suf _, x|_ pre) _,_
where _|_ denotes concatenation. Let _xp_ = _x|_ pre,
_xw_ = _x|_ pre _|_ mid _w_, and _xl_ = _x|_ pre _|_ mid _l_ . The argument of the log sigmoid function in the loss can
then be separated into two components:
_β_ log _[π][θ]_ [(][mid] _[w][ |][ x][p]_ [)]
_[π][θ]_ [(][mid] _[w][ |][ x][p]_ [)] _[π][θ]_ [(][mid] _[l][ |][ x][p]_ [)]
_πref_ (mid _w | xp_ ) _[−]_ _[β]_ [ log] _πref_ (mid _l | xp_
_πref_ (mid _l | xp_ ) _[,]_
_β_ log _[π][θ]_ [(][suf] _[ |][ x][w]_ [)]
_πref_ (suf _| xl_ ) _[.]_
_[π][θ]_ [(][suf] _[ |][ x][w]_ [)] _[π][θ]_ [(][suf] _[ |][ x][l]_ [)]
_πref_ (suf _| xw_ ) _[−]_ _[β]_ [ log] _πref_ (suf _| xl_
The first term corresponds to the loss for the middle
segment, which is the key component of the overall
loss. The second term, however, has a negative
impact, as it implies the model should not generate
the suffix based on _xl_ . However, this is an issue because a prior error is unrelated to the rest segment.
For instance, as shown in Figure 1, even though
there’s an error in Line 16, the remaining function
should still be generated like this.
As demonstrated, it is inappropriate to calculate
the DPO loss on the suffix, as it leads to negative
rewards for correctly generated tokens. Unfortunately, only a small fraction of the generated code
typically contains errors; the majority is correct.
This implies the need for a large dataset with test
cases to make up for the negative impact of the
DPO loss on the suffix.


In Section 2, we analyze the impediments of DPO
with limited code training data. In this section,
we introduce a novel approach that effectively mitigates these weaknesses. Unlike previous methods (Zhang et al., 2024a,b), which focus on expanding the training dataset, we emphasize optimizing the utilization of the limited available data.
We begin by demonstrating how to leverage FIM
to avoid the detrimental part of DPO loss. Next,
we describe how to construct more accurate FIM
data. Finally, we provide an overview of the entire
training pipeline.


To alleviate the negative impact of the DPO loss
on the suffix part, we proposed to combine FIM
and DPO to enhance the code generation performance. The FIM ability of code LLMs allows them
to generate the middle segment based on the prefix
and suffix segments, enabling us to control what is
included in the model response flexibly.
As shown in Figure 2, given a code training
case, which includes a code problem _q_, a reference solution _c_ to solve the problem, and several test cases _t_ to test any generation, we first
select consequent snippets from the golden code
_n_ times to construct the target snippets _M_ =
_{m_ 1 _, m_ 2 _, . . ., mn}_ . This process also generate the
corresponding prefixes _P_ = _{p_ 1 _, p_ 2 _, . . ., pn}_ and
suffixes _S_ = _{s_ 1 _, s_ 2 _, . . ., sn}_ . Then we construct
the FIM prompt with the target snippets, prefixes,
and suffixes:
<PRE> Convert( _q_ ) _pi_ <SUF> _si_ <MID> _,_ (1)
where “Convert” denotes a function converting the
text prompts into comments in the code. Then,
we task the code LLM to generate _m_ generations for each prefix and suffix pair: _G_ =
_{g_ 1 [(1)] _[, . . ., g]_ 1 [(] _[m]_ [)] _, . . ., gn_ [(1)] _[, . . ., g]_ _n_ [(] _[m]_ [)] _, }_ based on the
prompt. Finally, we concatenate the generation
with the corresponding prefix and suffix, _e.g., p_ 1 _|_
_g_ 1 [(1)] _| s_ 1, and use the test cases _t_ to verify the
generation.
For each target snippet, we select a correct response for each incorrect response, based on the
```
 <MID>
```


```
                                              assert check_prime_area (5, 5) == "YES"
```


```
return False
```
Figure 2: An overview of our FIM-style preference modeling process. A concrete example (right) illustrates the
task of completing an is_prime function, with correctness judged via downstream function tests.
minimum edit distance, to construct the preference
pair. Then we apply two prompt formats to finetune the model. One is the FIM format:
<PRE> Convert( _q_ ) _pi_ <SUF> _si_ <MID> _gi_ [(] _[j]_ [)] _[.]_
(2)
Here, only _gi_ [(] _[j]_ [)] is regarded as the response and
included in the DPO loss. Another format is the
Chat format:
User: _q._ Assistant: _pi gi_ [(] _[j]_ [)] _si._ (3)
Also, only _gi_ [(] _[j]_ [)] is calculated in the DPO loss. This
format is designed to preserve the model’s chat capabilities. During training, we randomly choose
one of two formats for each sample by drawing
from a Bernoulli distribution with probability _α_,
which determines the proportion of samples using
each format. This strategy enables the model to
be fine-tuned on shorter code snippets, thereby improving learning efficiency.


As discussed in Section 2, errors in the middle
segment are independent of those in the suffix segment. Therefore, it is essential to select appropriate
consequent code snippets from the golden label.
Figure 3: Illustration of our AST-based segmentation
strategy. This segmentation ensures each middle segment is syntactically and semantically coherent, enabling more effective FIM-based fine-tuning.
The ideal snippets should have the following characteristics to ensure effective model learning and
improved performance:
 - The selected code snippets should be distinct
blocks to maintain clear boundaries between
different functional units, ensuring that each
segment functions independently without relying on adjacent ones.
 - The selected code snippets should encompass a
variety of code structures to enable the model
to learn a broad range of coding patterns.
```
def is_prime(n):
  if n <= 1:
    return False
  if n <= 3:
    return True
  if n % 2 == 0 or n % 3 == 0:
    return False
i = 5
  while i * i <= n:
    if n % i == 0 or n % (i + 2) == 0:
     return False
i += 6
  return True
def check_prime_area(x, y):
  area = x * x - y * y
  return "YES" if not is_prime(area) else "NO"
```
_**If**_
_**If**_
_**If**_
_**If**_ _**While**_
_**Func**_
_**Func**_
Inspired by previous work (Gong et al., 2024),
we parse a code into an Abstract Syntax Tree (AST).
As illustrated in Figure 3, we first parse the golden
label into an AST, then select several blocks as the
target code snippets. In this approach, we consider
only four node types in the AST: _i.e.,_ if block, for
block, while block, and function block.
By analyzing the AST, we can extract syntactically complete and functionally independent code
blocks, which reduces the risk of including partial
or entangled code, ensures that the middle segments are self-contained and coherent, and exposes
the model to a broader range of distinct structural
patterns. For instance, as shown in Figure 3, the
first three if blocks handle distinct conditions and
thus operate independently, focusing solely on their
respective logic and ignoring others. Additionally,
the four selected node types represent the core
building blocks of Python code, capturing a diverse range of structures commonly encountered in
Python programming.


To effectively leverage all the blocks selected
from AST, we propose a curriculum learning strategy (Bengio et al., 2009) that orders the training
data based on the length of code snippets. The
goal is to enable the model to first focus on mastering token-level rewards for shorter, simpler code
snippets before gradually progressing to longer,
more complex ones, and finally the whole codes.
This strategy leverages the inherent structure of
code, where shorter snippets tend to contain simpler constructs and fewer dependencies, making
them easier to learn at the token level.
Specifically, we first sort the training samples by
the number of lines in the corresponding code snippets. During the initial training stages, the model
is fine-tuned on shorter code snippets, which typically have fewer tokens and simpler logic. This
allows the model to learn the fundamental code
patterns without being overwhelmed by the complexity of longer code blocks. As the training procedure, we progressively introduce longer snippets,
which feature more intricate logic, dependencies,
and structures.
In summary, the training begins by selecting target code snippets by AST segmentation from the
golden code, using FIM to generate middle segments based on the prefix and suffix. Then the
training data is ordered by code snippet length, with
the model first learning from shorter, simpler code
and progressively moving to longer, more complex
snippets, ensuring a structured and efficient learning process. Finally, we fine-tune the code LLMs
with the sorted training data and randomly select
the prompt format (FIM format or chat format) by
drawing from a Bernoulli distribution with probability _α_ in each training step.


In this section, we construct extensive experiments
to demonstrate the effectiveness of the proposed
method and analyze the proposed method.


**Training Dataset.** Our training dataset is the
APPS training set (Hendrycks et al., 2021), which
is collected from different open-access coding websites such as Codeforces, Kattis, and more. The
training set includes 5,000 samples and each sample contains a query, several golden labels, and
several test cases.
**Test Dataset.** We evaluate our method on HumanEval (Chen et al., 2021) and MBPP (Austin
et al., 2021), two of the most widely used benchmarks for code generation. Considering the insufficiency of test cases in these benchmarks, Liu et al.
(2023) proposed HumanEval+ and MBPP+, which
contain 80×/35× more tests. We also evaluate our
method on APPS (Hendrycks et al., 2021) and LiveCodeBench (Jain et al., 2024), two datasets with
more difficult code problems. In addition, we evaluate our method on BigCodeBench (Zhuo et al.,
2024), which challenged LLMs to invoke multiple
function calls as tools.
**Implementation Details.** We test our methods
on Qwen2.5-Coder-Instruct 1.5B/3B/7B (Hui et al.,
2024). We first use Black [1] to format all golden
labels in the training set, and then use AST [2] to
parse the formatted code and extract all desired
blocks, _i.e.,_ if block, for block, while block, function block. Then we use the code LLMs to generate
5 responses based on the FIM prompt, with top_p
= 0.95 and temperature = 0.7. After that, we use
the test cases to evaluate the generated responses
and select one preferred response and one dispreferred response (if there is no correct generation or
no incorrect generation, we will discard this case).
For training, we fine-tune all models for 3 epochs.
[1https://github.com/psf/black](https://github.com/psf/black)
[2https://docs.python.org/3/library/ast.html](https://docs.python.org/3/library/ast.html)
HumanEval MBPP
BigCodeBench
Method
APPS LiveCodeBench
Base Plus Base Plus Full Hard
Closed-Source Models
Claude-3.5-Sonnet 92.1 86.0 91.0 74.6 - 37.1 45.3 23.6
GPT-4o 92.1 86.0 86.8 72.5 - 33.0 50.1 25.0
o1-mini 97.6 90.2 93.9 78.3 - 54.1 46.3 23.0
o1-preview 95.1 88.4 93.4 77.8 - 42.5 49.3 27.7
Open-Source Models
DS-Coder-1.3B 65.9 60.4 65.3 54.8 5.7 4.1 22.8 3.4
Qwen2.5-Coder-1.5B 70.7 66.5 69.2 59.4 7.7 6.7 25.2 4.1
w/ DPO 74.4 69.5 73.5 63.8 9.3 7.8 28.6 6.8
w/ KTO 74.4 70.1 74.1 62.4 9.4 9.4 28.0 7.4
w/ Focused-DPO 72.6 68.3 72.8 62.7 9.2 9.1 **29.2** 7.4
w/ StructureCoder **75.6** _[‡]_ **71.3** _[‡]_ **75.7** _[∗†‡]_ **64.8** _[∗†‡]_ **9.9** _[∗†‡]_ **10.8** _[∗‡]_ **29.2** _[∗†]_ **8.1** _[∗‡]_
Qwen2.5-Coder-3B 84.1 80.5 73.6 62.4 10.9 10.4 35.8 14.2
w/ DPO 84.8 80.5 75.4 64.0 12.7 10.8 37.4 12.8
w/ KTO 81.1 77.4 73.3 63.0 12.4 10.8 36.6 11.5
w/ Focused-DPO **86.0** 81.1 **75.7** 64.3 **13.9** 10.8 37.9 14.5
w/ StructureCoder **86.0** _[∗†]_ **82.3** _[∗†]_ **75.7** _[∗†]_ **64.6** _[∗†‡]_ **13.9** _[∗†]_ **13.4** _[∗†‡]_ **38.4** _[∗†‡]_ **16.9** _[∗†‡]_
CodeLlama-7B 40.9 33.5 54.0 44.4 4.3 7.1 21.9 3.4
DS-Coder-6.7B 74.4 71.3 74.9 65.6 8.5 10.4 35.5 10.1
Qwen2.5-Coder-7B 88.4 84.1 83.5 71.7 16.2 16.8 41.0 18.2
w/ DPO 88.4 84.1 83.1 71.4 19.8 17.5 **41.8** 19.1
w/ KTO 89.0 84.1 85.4 72.5 19.0 16.9 41.0 18.2
w/ Focused-DPO 89.0 84.8 85.4 72.5 20.0 17.9 41.1 18.2
w/ StructureCoder **90.9** _[∗†‡]_ **87.2** _[∗†‡]_ **85.7** _[∗]_ **74.3** _[∗†]_ **20.1** _[∗†]_ **18.3** _[∗†‡]_ **41.8** _[†]_ **19.6** _[∗†‡]_
Table 1: Pass@1 accuracy on HumanEval (+), MBPP (+), APPS, LiveCodeBench (July 2024 - January 2025),
and BigCodeBench. The best results of each base are in bold, and results unavailable are left blank. The results
re-evaluated on our end are marked with an underline. _∗_, _†_, and _‡_ denote that our method significantly outperforms
DPO, KTO, and Focused-DPO at the level of 0.05, respectively. Note that all results of the LiveCodeBench have
been re-evaluated due to the dataset update. We also report the re-evaluated results of the BigCodeBench of
Qwen2.5-Coder-1.5B-Inst because the reproduced results differed significantly from the original paper.
We employ an RMSProp optimizer (Graves, 2013)
with a learning rate of 1e-6, a 0.05 warm-up ratio, and a cosine scheduler. We set the batch size
as 128 and the max sequence length as 2048. We
set the hyperparameter _α_ = 0 _._ 5. To efficiently
train the computationally intensive models, we simultaneously employ VLLM (Kwon et al., 2023),
DeepSpeed (Rajbhandari et al., 2020) and Flash
Attention (Dao, 2023). On 8 NVIDIA A800 80GB
GPUs, the experiments on 1.5B, 3B, and 7B models
take 8 hours, 10 hours, and 16 hours, respectively.


**Baseline.** We employ DPO (Rafailov et al.,
2023), KTO (Ethayarajh et al., 2024), and Focused
DPO (Zhang et al., 2025) as our baseline. For the
DPO and KTO, we use the chat format to generate
5 responses for each problem. For Focused-DPO,
we follow the paper to generate 10 responses and
select the 5000 samples with the longest common
prefix and suffix. The other pipeline and hyperparameters are the same as the implementation details
of StructureCoder.
**Results.** Table 1 shows the pass@1 accuracy of
different models on HumanEval, MBPP, APPS,
LiveCodeBench, and BigCodeBench. Based on the
results, we have the following findings:
(1) The proposed StructureCoder consistently
outperforms standard DPO and KTO across all
HumanEval MBPP
BigCodeBench
Method
APPS LiveCodeBench
Base Plus Base Plus Full Hard
StructureCoder 75.6 71.3 **75.7** **64.8** **9.9** **10.8** **29.2** **8.1**
w/ DPO 74.4 69.5 73.5 63.8 9.3 7.8 28.6 6.8
w/ DPO (Data Equal) 66.5 59.8 71.4 59.0 2.0 4.1 25.9 6.1
w/o AST 75.0 69.5 74.1 62.4 9.9 9.7 28.8 3.4
w/o Curriculum **78.7** **73.2** 73.8 64.3 9.8 9.0 28.3 7.4
w/ _α_ = 0 75.0 70.7 74.1 62.4 9.6 8.6 28.7 8.1
w/ _α_ = 1 78.7 72.0 74.1 64.0 9.6 9.7 29.1 8.1
w/ suf loss 77.4 70.7 75.4 64.6 9.8 8.2 29.0 7.4
w/ pre & suf loss 76.2 70.7 74.9 64.6 9.7 8.1 29.1 7.4
w/o suf 73.2 69.5 75.1 62.4 9.8 9.7 29.1 **8.1**
Table 2: Ablation results based on Qwen2.5-Coder-1.5B-Instruct. The score is Pass@1 accuracy.
models on a variety of code-related tasks. Specifically, for Qwen2.5-Coder-1.5B-Instruct, StructureCoder surpasses DPO by an average of 1.5 points
across all test sets. Similarly, for Qwen2.5-Coder3B-Instruct and Qwen2.5-Coder-7B-Instruct, StructureCoder outperforms DPO by an average of 1.6
points across all test sets. The similar increase in
different sizes demonstrates the effectiveness of the
proposed method.
(2) The proposed StructureCoder shows a more
significant improvement over DPO on most test
sets compared to the APPS test set. For Qwen2.5Coder-1.5B-Instruct, StructureCoder exceeds DPO
by an average of 1.5 points on all test sets except
for the APPS test set, where the improvement is
only 0.6 points. This suggests that the proposed
method enables the model to learn more diverse
code patterns and achieve better generalization.
(3) The proposed StructureCoder demonstrates
comparable performance to Focused-DPO on
Qwen2.5-Coder-7B-Instruct, but achieves a significant improvement on Qwen2.5-Coder-1.5BInstruct. This is because generations from the
weaker model tend to have shorter comment prefixes and suffixes, which diminishes the effectiveness of Focused-DPO. In contrast, our method does
not rely on aligning specific features of the generations, leading to consistent improvements across
all three models.


Here, we check how each component contributes
to the final performance. We prepare two group
variants of our method based on Qwen2.5-Coder1.5B-Instruct: (1) The first group is related to DPO.
w/ DPO denotes constructing training data w/o
FIM. w/ DPO (Data Equal) denotes training with
more epochs to ensure the total training samples are
the same as StructureCoder. (2) The second group
is related to the proposed components. w/ AST
denotes selecting target code snippets randomly.
w/ Curriculum denotes training with random order.
w/ _α_ = 0 denotes training only with the standard
format. w/ _α_ = 1 denotes training only with the
FIM format.
Table 2 presents the pass@1 accuracy for various
model variants. Based on the results, we have the
following findings:
(1) As shown, all variants perform worse than
StructureCoder across most test sets, except for
HumanEval. These results suggest that all components ( _i.e.,_ AST enhanced FIM, curriculum learning, and mixing FIM training format and Chat format) are critical for enhancing performance.
(2) Furthermore, w/ DPO (Data Equal) performs worse than w/ DPO, indicating that DPO
benefits more from diverse, large-scale training
data than from simply increasing the number of
epochs. Increasing epochs can lead to overfitting.
In contrast, StructureCoder effectively generates
more diverse training data through FIM, resulting
in improved performance.
(3) Lastly, w/ Curriculum and w/ _α_ = 1 outperform StructureCoder on HumanEval. A potential
explanation for this is that the HumanEval test
cases differ significantly from those in the APPS,
making fine-tuning with longer code during the
final stage detrimental to HumanEval performance.


As we analyzed in Section 2, the DPO loss on the
prefix segment does not impact the optimization,
but the DPO loss on the suffix segment harms the
optimization. Here, we conduct experiments to
check the effect of loss on the prefix segment and
the suffix segment. Specifically, we prepare three
variants based on Qwen2.5-Coder-1.5B-Instruct.
w/ suf loss computes the loss on both the middle
and the suffix segments. w/ pre & suf loss computes the loss on all segments. w/o suf remove
suffix in the whole pipeline, _a.k.a.,_, we only use
the query and the prefix to prompt the model to
generate various generations, effectively setting the
suffix to be empty.
Table 2 presents the pass@1 accuracy for the
variants. As shown in the table, w/ suf loss significantly performs worse than StructureCoder on
LiveCodeBench, which is the most difficult test set.
This indicates that the DPO loss on the suffix segment hurts the optimization. However, w/ suf loss
and w/ pre & suf loss perform similarly. This indicates that the DPO loss on the prefix segment does
not impact the optimization. This phenomenon is
less noticeable on other datasets because they are
simpler. The difficulty of LiveCodeBench highlights the impact of the DPO loss on the suffix
segment, while simpler datasets don’t show the
same effect. Finally, w/o suf performs worse than
StructureCoder in most settings. This is because
concatenating the middle and suffix spans results
in overly long input sequences, which may hinder
the model’s ability to focus on critical code blocks.
This observation aligns with our theoretical insights
discussed in Section 2.3.

4.3.2 Effect of Loss on Prefix and Suffix


Previous works have introduced LLMs for the
code domain. OpenAI introduced Codex (Chen
et al., 2021), and Google introduced PaLMCoder (Chowdhery et al., 2023). There are also
several open-source LLMs for the code domain,
such as CodeGen (Nijkamp et al., 2023), Incoder (Fried et al., 2023), SantaCoder (Allal et al.,
2023), StarCoder (Li et al., 2023), StarCoder2 (Lozhkov et al., 2024), CodeGeeX (Zheng et al.,
2023), Code Llama (Rozière et al., 2023), and
DeepSeek-Coder (Guo et al., 2024). Recently,
Deepseek-Coder-V2 (DeepSeek-AI et al., 2024)
and Qwen2.5-Coder (Hui et al., 2024) have been
proposed, which achieve performance close to that
of closed-source models.
Instruction tuning is demonstrated to be an effective method to improve the ability of LLMs
in specific domains. Code Alpaca (Chaudhary,
2023) applied Self-Instruct (Wei et al., 2022) to
fine-tune LLMs with ChatGPT-generated instructions. WizardCoder (Luo et al., 2023) proposed
Code Evol-Instruct, which evolves Code Alpaca
data using the ChatGPT to generate more complex and diverse datasets. MagiCoder (Wei et al.,
2023), WaveCoder (Yu et al., 2023), and InverseCoder (Wu et al., 2024) proposed some methods
to make full use of source code. OpenCodeInterpreter (Zheng et al., 2024), AutoCoder (Lei et al.,
2024) and ReflectionCoder (Ren et al., 2024b) proposed to utilize a compiler to enhance code LLMs.


Reinforcement learning-based alignment methods have been shown to improve LLM performance. DPO (Rafailov et al., 2023), though widely
adopted, provides limited gains in code generation tasks (Xu et al., 2024). PLUM (Zhang et al.,
2024a) uses GPT-4 to generate test cases for code
validation and ranking. CodeDPO (Zhang et al.,
2024b) applies a PageRank-inspired algorithm for
iterative preference scoring. StepCoder (Dou et al.,
2024) employs PPO (Schulman et al., 2017) to
optimize general code generation, and CodeOptimise (Gee et al., 2024) focuses on improving
runtime efficiency. Concurrent with our work,
Focused-DPO (Zhang et al., 2025) also identified
the issue of common prefixes and suffixes. However, their approach heavily depends on model generations that contain these patterns, whereas our
method eliminates such dependency.
Unlike previous DPO-based approaches to code
generation, our approach focuses on maximizing
the utility of limited data to construct more effective preference pairs. While existing methods often rely on larger datasets or complex augmentation strategies, we take a data-efficient approach
by making the most of even small-scale datasets.
By introducing FIM (Bavarian et al., 2022), we
can efficiently extract high-quality preference pairs,
significantly improving the alignment of code generation models while utilizing a limited amount of
labeled data.


In this paper, we introduced a novel approach to
enhance code generation performance for LLMs
using the fill-in-the-middle (FIM). By generating
fine-grained DPO pairs from limited test cases and
employing a curriculum training method, we maximized the utility of available data, leading to improved model performance. Our experiments on
benchmarks like HumanEval (+), MBPP (+), APPS,
LiveCodeBench, and BigCodeBench demonstrated
the effectiveness of our approach in code-related
tasks. Future work could explore further refinements, such as alternative segmentation strategies
or additional data sources.


One key limitation of our approach is that it requires the model to possess strong Fill-In-theMiddle (FIM) capabilities, which are essential
for effectively generating fine-grained DPO pairs.
However, most current models, after supervised
fine-tuning, may struggle to maintain high FIM
performance. As a result, the effectiveness of our
method is closely tied to the underlying model’s
proficiency in FIM, a capability that is not yet universally available in existing LLMs. This reliance
on FIM limits the applicability of our approach to
models that have been specifically optimized for
this task. Another limitation is that our method
is currently focused solely on the code generation
domain. While we believe that similar phenomena
may exist in other domains, our approach has not
yet been tested outside the code generation field.
Further research would be required to explore the
transferability to other closed-question tasks.


The model utilized in this paper, Qwen2.5Coder (Li et al., 2023), is licensed for academic research purposes [3] . Furthermore, the data employed
in this study, APPS (Hendrycks et al., 2021), is also
licensed for academic research purposes [4] .


This project is funded in part by National Key R&D
Program of China Project 2022ZD0161100, by the
Centre for Perceptual and Interactive Intelligence
(CPII) Ltd under the Innovation and Technology
[3https://github.com/QwenLM/Qwen2.5-Coder](https://github.com/QwenLM/Qwen2.5-Coder)
[4https://github.com/hendrycks/apps](https://github.com/hendrycks/apps)
Commission (ITC)’s InnoHK, and in part by NSFCRGC Project N_CUHK498/24.


To evaluate the structural diversity of training data
constructed for our method, we performed a statistical analysis of code blocks extracted via Abstract
Syntax Tree (AST) parsing. Specifically, we retained four categories of syntactic nodes as middle
targets for the FIM task: if, for, while, and def
(function definitions). The figure below presents
the distribution of these node types in the final
training dataset.
As shown in Figure 4, among all segments used
in FIM training, if blocks constitute the largest proportion (41.54%), followed by for loops (32.09%),
def (function) blocks (18.63%), and while loops
(7.74%). This distribution indicates that the selected code snippets exhibit substantial structural
variety, with conditional (if) and iterative (for
and while) constructs dominating the sample pool.
Such a mix helps the model learn frequent control
flow patterns more effectively in FIM tasks. Although while and def blocks appear less frequently,
they still contribute essential control and functional
semantics, ensuring the structural completeness of
the training examples.
To construct the training dataset, we follow
a systematic pipeline that parses source code
into abstract syntax trees, generates fill-in-themiddle (FIM) prompts, evaluates model completions against test cases, and derives preference pairs
from completions. The detailed procedure is outlined in Algorithm 1.


Fill-in-the-Middle (FIM) (Bavarian et al., 2022) has
been shown to preserve the model’s ability to generate code effectively (Li et al., 2023; Rozière et al.,
2023; Ren et al., 2024a), while offering several
distinct advantages that enhance overall model performance and robustness. By requiring generation
conditioned on both preceding and succeeding context, FIM strengthens contextual understanding and
promotes deeper semantic reasoning. Rather than
replacing standard code generation, FIM acts as
a powerful complement—particularly well-suited
for practical scenarios such as code refactoring,
patching, and bug fixing. Its bidirectional formulation improves generalization and equips models
to handle complex code dependencies more effec
Figure 4: Distribution of extracted code blocks based on
AST node types. This reflects the syntactic diversity in
the training corpus and validates the representativeness
of selected blocks.
tively. Our empirical results further demonstrate
that FIM-based training leads to notable gains in
code generation quality, underscoring its value as
a targeted and practical enhancement to existing
methodologies.


While our proposed method shares some similarities with Step-DPO regarding loss construction and
data generation, it offers distinct improvements in
the following aspects:
(1) Step-DPO relies on labeling intermediate
reasoning steps either through human annotation,
which is prohibitively expensive, or through automated labeling methods such as GPT-4, which
introduces potential biases as highlighted by recent
studies (DeepSeek-AI et al., 2025). In contrast, our
proposed method utilizes FIM to automatically generate and assess intermediate reasoning fragments
based on objective evaluation criteria from test samples. By eliminating the reliance on costly human
annotation and avoiding the biases associated with
automated labeling tools, our method offers an unbiased alternative for training and evaluating reasoning capabilities in large language models.
(2) Annotating individual steps in Step-DPO can
be inherently ambiguous, especially for long Chainof-Thought (CoT). An incorrect intermediate step
can sometimes contribute meaningfully to reaching
a correct final answer — for example, models like
o1 often make a mistake in early reasoning steps,
such as miscalculating a quantity or misunderstanding part of the problem, but then later detect the
inconsistency in their own logic and correct for it
in the final answer. In such cases, although the
intermediate step is technically incorrect, it still
plays a functional role in producing the right out
come. This makes it difficult to definitively label
individual steps as preferred or dispreferred. Our
approach overcomes this issue by fixing the suffix
of the code, requiring the generated intermediate
code (the middle segment) to coherently bridge the
given prefix and suffix. This design fundamentally
eliminates ambiguity in annotation, clearly delineating preferred and dispreferred middle segments.


To thoroughly assess the Fill-In-the-Middle (FIM)
capabilities of our proposed method, we conducted
a series of targeted evaluations using the APPS
benchmark. Specifically, we curated a set of 22,044
samples by randomly masking contiguous code
spans from the reference solutions. The models
were then tasked with accurately reconstructing
these masked segments solely based on the surrounding context, thereby simulating realistic code
completion scenarios.
As summarized in Table 3, the application of our
method consistently improves FIM performance
across all evaluated model scales. Notably, our
approach achieves the highest Pass@1 accuracy
in each model category, surpassing several strong
baselines. These findings underscore the robustness
and generalizability of our method in enhancing
middle-span code generation, demonstrating its potential as a principled strategy for strengthening
contextual reasoning in large language models for
programming tasks.


Here, we use a case from Qwen2.5-Coder1.5B-Instruct, Qwen2.5-Coder-3B-Instruct and
Qwen2.5-Coder-7B-Instruct to show the effectiveness of the proposed method. Specifically, we compute the DPO reward ( _r_ ( _s, a_ ) = _β_ log _πθ_ ( _s|a_ ) _−_
_β_ log _πref_ ( _s|a_ )) for each token in both correct and
incorrect responses. The DPO is used as a baseline
for comparison.
As shown in Figure 5, the error is introduced in
Line 2, and the proposed method, StructureCoder,
successfully identifies the token corresponding to
the erroneous statement ( _x ∗_ _x_ and _x ∗_ _y_ ). For
Qwen2.5-Coder-3B-Instruct, as shown in Figure 6,
the proposed StructureCoder successfully assigns
high reward to the token corresponding to the erroneous statement ( _len_ ( _bits_ ) _−_ 1 and _len_ ( _bits_ )). For
Qwen2.5-Coder-7B-Instruct, as shown in Figure 7,


Qwen-2.5-Coder-1.5B-Inst 27.1
w/ DPO 27.5
w/ KTO 27.4
w/ Focused-DPO 27.4
w/ StructureCoder **28.6**
Qwen-2.5-Coder-3B-Inst 31.1
w/ DPO 31.5
w/ KTO 31.6
w/ Focused-DPO 31.7
w/ StructureCoder **32.9**
Qwen-2.5-Coder-7B-Inst 35.7
w/ DPO 35.6
w/ KTO 35.9
w/ Focused-DPO 36.1
w/ StructureCoder **36.7**
Table 3: Pass@1 accuracy on the FIM task.
the proposed StructureCoder effectively highlights
he erroneous statement ( _n_ + _i_ and _n −_ _i_ ). These
demonstrate the ability of our method to localize
specific errors and focus attention on the relevant
details. On the other hand, the DPO fails to detect these errors, highlighting its limitation in handling errors with high precision, especially when
the training data is sparse. While the DPO might
perform well with abundant training examples, its
performance drops in the presence of limited data,
making it less effective for tasks requiring finegrained error detection in these conditions.
(a) Credit assignment Qwen2.5-Coder-1.5B-Instruct w/ DPO.
(b) Credit assignment Qwen2.5-Coder-1.5B-Instruct w/ StructureCoder.
Figure 5: Credit assignment with different methods. Due to the limited space, we omit the previous function, only
keeping the key function. The left is the correct response and the right is the incorrect response (error is introduced
in Line 2). Each token is colored corresponding to the DPO implicit reward (darker is higher).
(a) Credit assignment Qwen2.5-Coder-3B-Instruct w/ DPO.
(b) Credit assignment Qwen2.5-Coder-3B-Instruct w/ StructureCoder.
Figure 6: Credit assignment with different methods on Qwen2.5-Coder-3B-Instruct. The left is the correct response
and the right is the incorrect response (error is introduced from the last token of Line 4). Each token is colored
corresponding to the DPO implicit reward (darker is higher).
(a) Credit assignment Qwen2.5-Coder-7B-Instruct w/ DPO.
(b) Credit assignment Qwen2.5-Coder-7B-Instruct w/ StructureCoder.
Figure 7: Credit assignment with different methods on Qwen2.5-Coder-7B-Instruct. The left is the correct response
and the right is the incorrect response (error is introduced in Line 6). Each token is colored corresponding to the
DPO implicit reward (darker is higher).
**Algorithm 1:** Training Data Construction Pipeline
**Input:** _D_ = _{_ ( _q, c, t_ ) _}_, where _q_ is a question, _c_ is the corresponding code, and _t_ are test cases
**Output:** Final training set ready for fine-tuning


Parse _c_ into an Abstract Syntax Tree (AST);
Extract target code blocks _M_ = _{m_ 1 _, m_ 2 _, . . ., mn}_ ;


Extract prefix _pi_ and suffix _si_ ;
Construct FIM prompt: <PRE> Convert( _q_ ) _pi_ <SUF> _si_ <MID>;
Generate _m_ completions _Gi_ = _{gi_ [(1)] _, . . ., gi_ [(] _[m]_ [)] _}_ ;


_full_ _ _code ←_ _pi_ + _gi_ [(] _[j]_ [)] + _si_ ;
Execute _full_ _ _code_ on test cases _t_ ;


Find closest correct _g_ [+] by edit distance;
Form preference pair ( _g_ [+] _, g_ _[−]_ );

foreach _incorrect completion g_ _[−]_ do


Sample prompt format using Bernoulli distribution ( _α_ );

foreach _preference pair_ ( _g_ [+] _, g_ _[−]_ ) do


Positive _←_ <PRE> Convert( _q_ ) _pi_ <SUF> _si_ <MID> _g_ [+] ;
Negative _←_ <PRE> Convert( _q_ ) _pi_ <SUF> _si_ <MID> _g_ _[−]_ ;


Positive _←_ User: _q_ ; Assistant: _pi_ + _g_ [+] + _si_ ;
Negative _←_ User: _q_ ; Assistant: _pi_ + _g_ _[−]_ + _si_ ;


Sort training samples by length of _g_ [+] (short _→_ long);
**return** Final training set;