Avelina/lovelace-medium-alpha1

Lovelace Medium Alpha1

551M parameter Transformer-XL style model trained on 100B tokens of The Pile!

This model was originally trained for the "Direct Prefrence Heads" paper, but will also be used as the basis for much of my future research. All code used to train and run these models is available here: https://github.com/Avelina9X/direct-preference-heads and our paper is available here: https://arxiv.org/abs/2405.20053

Model Architecture

Name	Value
Total Parameters	551M
Non-Embedding Parameters	512M
Vocab Size	50272
$d_{\text{vocab}}d\_\backslash text\{vocab\}$	768
$d_{\text{model}}d\_\backslash text\{model\}$	1536
$n_{\text{layers}}n\_\backslash text\{layers\}$	18
FFN Activation	SwiGLU
$d_{\text{ffn}}d\_\backslash text\{ffn\}$	4096
Attention Type	Full
Positon Embedding	Reversed RoPE with ABF
$n_{\text{heads}}n\_\backslash text\{heads\}$	24
$d_{\text{key}}d\_\backslash text\{key\}$	64
Trained Context	2048
Trained Memory	2048
Max Inference Context	4096

Model Collection

Model	Link
Pre-Trained Model	lovelace-medium-alpha1
Fine-Tuned Model	lovelace-medium-alpha1-sft
DPH Aligned Model	lovelace-medium-alpha1-dph