eliebak/very-smollm-corpus
Viewer
•
Updated
•
4.58M
•
2
•
2
Elie Bakouch
eliebak
AI & ML interests
Training LLM's @ 🤗
Recent Activity
liked
a model
about 18 hours ago
Motif-Technologies/Motif-2.6B
posted
an
update
about 19 hours ago
Motif 2.6B tech report is pretty insane, first time i see a model with differential attention and polynorm trained at scale!
> It's trained on 2.5T of token, with a "data mixture schedule" to continuously adjust the mixture over training.
> They use WSD with a "Simple moving average" averaging the last 6 ckpt every 8B token.
> They trained on Finemath, Fineweb2, DCLM, TxT360.
> Lot of details in the finetuning data they used, for instance they used EvolKit and did some "dataset fusion" to have more compressed knowledge into the data.
> They mention they also tried Normalized GPT, QK-Norm and Cross Layer Attention.
https://huggingface.co/Motif-Technologies/Motif-2.6B
liked
a model
about 19 hours ago
Motif-Technologies/activation
Organizations
