AliceAndNoob/babylm2026-strict-gptbert-simper

GPT-BERT + CDI-Curriculum + SimPER v2 — BabyLM 2026 Strict Track

A hybrid GPT-BERT language model submitted to the BabyLM 2026 Challenge (Strict Track).

Pre-trained with a 75% MLM / 25% CLM objective using a CDI-based lexical curriculum on an optimised 100M-word English corpus, then post-trained with SimPER v2 (Simple Preference Optimization with Entropy Regularization) using 30,000 task-specific preference pairs targeting grammar, entity tracking, and multi-container reasoning.

---

Model Description

• Architecture: GPT-BERT hybrid encoder (12 layers, 768 hidden, 12 heads, ~125M parameters)
• Pre-training objective: 75% Masked LM + 25% Causal LM with a 2-phase CDI lexical curriculum
• Post-training: SimPER v2 — 30K specialised preference pairs targeting grammar + entity tracking
• Track: BabyLM 2026 — Strict (≤100M words training data, ≤1B word exposures total)
• Total word exposure: ~0.964B words (well within the 1B budget)

---

Training Data

We constructed an optimised 100M-word corpus by combining official BabyLM 2026 Strict data (76.17M words) with two supplementary public sources:

Source	Domain	Words
CHILDES	Child-directed speech	—
Gutenberg (official BabyLM)	Books / literature	—
OpenSubtitles	Dialogue / subtitles	—
BNC-spoken	Spoken British English	—
Switchboard	Telephone conversations	—
Official BabyLM 2026 Strict total		76.17M
swifte/gutenberg_english	Additional literary texts	9.42M
wikimedia/wikipedia (2021-11-01 en)	Encyclopedia (replaces Simple Wikipedia)	14.41M
Grand total		~100M

All sources were filtered to English-only documents, whitespace-normalised, and document-separated by newlines. Wikipedia replaces Simple Wikipedia to increase factual density.

¹ The distribution of parts of the BNC Text is permitted under the fair dealing provisions of copyright law (see the BNC website).

CDI Lexical Curriculum

Inspired by child language acquisition research (MacArthur-Bates CDI norms), we apply a two-phase lexical curriculum during pre-training:

• Phase 1 (first 10% of steps): Sentences drawn from documents whose vocabulary skews toward the earliest-acquired English words (AoA percentile < 10%), giving the model child-like early exposure to high-frequency, high-familiarity content.
• Phase 2 (remaining 90% of steps): Standard random shuffled documents from the full corpus.

This curriculum improves AoA Pearson r by +0.5 pp over the flat-sampling baseline and provides a stronger foundation for entity tracking tasks.

SimPER v2 Post-training Data

30,000 preference pairs were generated using three targeted categories:

Category	Count	Construction
Grammar — subject-verb agreement	~8,200	Rule-based minimal pairs
Grammar — tense	~5,200	Rule-based minimal pairs
Grammar — article	~1,600	Rule-based minimal pairs
Entity tracking — simple 2-container	5,000	Template-based closed-vocabulary
Entity tracking — move_contents (7 boxes, bulk moves)	5,000	Programmatic simulation
Entity tracking — ambiref (adjective-qualified items)	5,000	Programmatic simulation

The move_contents and ambiref categories were added in v2, directly targeting the hardest entity-tracking sub-tasks from Kim & Schuster (ACL 2023). Rejected answers represent temporal state errors — the model predicts the initial box state rather than the post-move state — matching the most common failure mode observed in preliminary evaluation.

No additional text was introduced; all pairs are derived from closed-vocabulary templates. Total word exposure from SimPER v2 post-training: 1.76M words (0.002B).

---

Training Procedure

Pre-training Hyperparameters

Hyperparameter	Value
Architecture	GPT-BERT hybrid (encoder-only)
Total Parameters	~125M
Attention Heads	12
Hidden Layers	12
Hidden Size	768
Intermediate Size	2,560
Vocab Size	24,000
Tokenizer	BabyLM English baseline tokenizer (24k vocab)
Max Sequence Length	512
Optimizer	LAMB
Peak Learning Rate	7e-3
LR Schedule	Cosine with warmup and cooldown
Warmup Proportion	1.6%
Cooldown Proportion	1.6%
Global Batch Size (sequences)	1,024
Avg Tokens per Step	81,920
Sequence Curriculum	Phase 1 (0–80% steps): 128 tokens; Phase 2 (80–100%): 256 tokens
Lexical Curriculum	CDI Phase 1 (0–10% steps): AoA-sorted docs; Phase 2 (10–100%): random
MLM : CLM Ratio	75% : 25%
Mask Probability	30% → 15% (annealed)
EMA Decay	0.999
Weight Decay	0.1
Z-loss Weight	1e-4
Max Gradient Norm	2.0
Total Training Steps	16,372
Effective Epochs	~10
Random Seed	42
GPU Hours (training)	~22
GPU Hours (total development)	~250

SimPER v2 Post-training Hyperparameters

Hyperparameter	Value
Preference Pairs	30,000 (15K grammar + 5K entity-simple + 5K move_contents + 5K ambiref)
Loss	SimPER (arXiv:2502.00883)
Optimizer	AdamW
Learning Rate	1e-5
Batch Size (sequences)	32
Training Steps	300
Warmup Steps	20
Anchor Weight (α)	0.05
Max Sequence Length	128
Final accuracy (chosen > rejected)	63.4%

---

Evaluation

We use the official BabyLM 2026 evaluation pipeline with Mean Token Prediction (MNTP) scoring. BLiMP and BLiMP-Supplement use filtered datasets (training-data-overlapping items excluded). Entity Tracking is the 3-dataset average: regular (2-container), move_contents (7-box bulk moves), and ambiref (adjective-qualified items).

Overall Score

Metric	Value	Weight in TextAvg
BLiMP	74.99	1/8
BLiMP-Supplement	63.75	1/8
EWoK	54.70	1/8
Entity Tracking	45.89	1/8
COMPS	55.90	1/8
Reading (SPR+ET avg)	2.21	1/8
AoA (Pearson r × 100)	17.24	1/8
GLUE (7-task avg)	67.72	1/8
TextAvg	47.80

Zero-Shot Tasks

Task	Metric	Score
BLiMP	Accuracy	74.99
BLiMP-Supplement	Accuracy	63.75
EWoK	Accuracy	54.70
Entity Tracking (avg)	Accuracy	45.89
— regular (2-container)	Accuracy	48.42
— move_contents (7-box)	Accuracy	46.48
— ambiref (adj-qualified)	Accuracy	42.75
COMPS	Accuracy	55.90
Reading — Eye-tracking	Correlation	3.10
Reading — SPR	Correlation	1.32
AoA (Word Surprisal)	Pearson r × 100	17.24

GLUE Fine-tuning

Task	Metric	Score
BoolQ	Accuracy	68.99%
MultiRC	Accuracy	64.81%
RTE	Accuracy	64.75%
WSC	Accuracy	61.54%
MRPC	Accuracy	82.35%
QQP	Accuracy	77.96%
MNLI	Accuracy	53.63%
GLUE Average		67.72%

---

Technical Specifications

Model Architecture

The model combines:

• Bidirectional masked attention (BERT-style) for 75% of training tokens
• Causal language modelling objective on the remaining 25% of tokens (GPT-style)
• Relative position embeddings (bucket size 32) instead of absolute positional encodings
• EMA (Exponential Moving Average) weights used at inference

Inference uses the AutoModel / AutoModelForMaskedLM interface via the custom modeling_gpt_bert_eval.py included in this repository.

Compute Infrastructure

Hardware: 4 × NVIDIA GeForce RTX 5090 (32 GB each)

Software:

• Python 3.10
• PyTorch 2.x
• HuggingFace Transformers 5.x
• 4-GPU data-parallel distributed training (NCCL)

---

Citations

@article{simper2025,
  title   = {SimPER: Simple Preference Optimization with Entropy Regularization},
  author  = {Anonymous},
  journal = {arXiv preprint arXiv:2502.00883},
  year    = {2025}
}

@inproceedings{kim2023entity,
  title     = {Entity Tracking in Language Models},
  author    = {Kim, Najoung and Schuster, Tal},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL)},
  year      = {2023}
}

@inproceedings{babylm2024,
  title     = {The BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora},
  author    = {Warstadt, Alex and Mueller, Aaron and Choshen, Leshem and Wilcox, Ethan and
               Zhuang, Chengxu and Ciro, Juan and Mosquera, Rafael and Paranjabe, Bhargavi and
               Williams, Adina and Linzen, Tal and Cotterell, Ryan},
  booktitle = {Proceedings of the BabyLM Challenge at CoNLL 2023},
  year      = {2024}
}

@article{fenson1994,
  title   = {Variability in Early Communicative Development},
  author  = {Fenson, Larry and Dale, Philip S. and Reznick, J. Steven and Bates, Elizabeth and
             Thal, Donna J. and Pethick, Stephen J.},
  journal = {Monographs of the Society for Research in Child Development},
  volume  = {59},
  number  = {5},
  year    = {1994}
}

@article{you2019lamb,
  title   = {Large Batch Optimization for Deep Learning: Training {BERT} in 76 minutes},
  author  = {You, Yang and Li, Jing and Reddi, Sashank and Hseu, Jonathan and Kumar, Sanjiv and
             Bhojanapalli, Srinadh and Song, Xiangru and Demmel, James and
             Talwalkar, Ameet and Hsieh, Cho-Jui},
  journal = {arXiv preprint arXiv:1904.00962},
  year    = {2019}
}

@article{warstadt2020blimp,
  title   = {{BLiMP}: The Benchmark of Linguistic Minimal Pairs for English},
  author  = {Warstadt, Alex and Parrish, Alicia and Liu, Haokun and Mohananey, Anhad and
             Peng, Wei and Wang, Sheng-Fu and Bowman, Samuel R.},
  journal = {Transactions of the Association for Computational Linguistics},
  volume  = {8},
  pages   = {377--392},
  year    = {2020}
}

---

Model Card Authors

Gan Wang — Xi'an Jiaotong-Liverpool University