Hugging Face's logo
Elie Bakouch's picture
80 140 156

Elie Bakouch

eliebak
ebernabe's profile picture jamesmanconvile42's profile picture naturelizer's profile picture
·

AI & ML interests

Training LLM's @ 🤗

Recent Activity

liked a model about 22 hours ago
Motif-Technologies/Motif-2.6B
posted an update about 22 hours ago
Motif 2.6B tech report is pretty insane, first time i see a model with differential attention and polynorm trained at scale! > It's trained on 2.5T of token, with a "data mixture schedule" to continuously adjust the mixture over training. > They use WSD with a "Simple moving average" averaging the last 6 ckpt every 8B token. > They trained on Finemath, Fineweb2, DCLM, TxT360. > Lot of details in the finetuning data they used, for instance they used EvolKit and did some "dataset fusion" to have more compressed knowledge into the data. > They mention they also tried Normalized GPT, QK-Norm and Cross Layer Attention. https://huggingface.co/Motif-Technologies/Motif-2.6B
liked a model about 23 hours ago
Motif-Technologies/activation
View all activity

Organizations

Hugging Face's profile picture HuggingFaceBR4's profile picture Hugging Face H4's profile picture Blog-explorers's profile picture Hugging Face Smol Models Research's profile picture huggingPartyParis's profile picture Nanotron Research's profile picture MLX Community's profile picture Hugging Face SMOL's profile picture FineData's profile picture HuggingFaceFW-Dev's profile picture StarCoder2 Data's profile picture Hugging Face Discord Community's profile picture LLHF's profile picture llmc's profile picture SLLHF's profile picture Argilla Warehouse's profile picture nltpt's profile picture smol-explorers's profile picture Open Science's profile picture Hugging Face Science's profile picture open/ acc's profile picture Open R1's profile picture smol-ablations's profile picture SmolEvalData's profile picture