simonko

might be releasing BananaMind 2 Ultra soon, 1.1B
Just a hint

yall are so good at making small models that i think you would easily make even a 100 expert 0.2b per expert model

im probably worse than llama3.1 in making models, i always forget something and it allways comes out bad lol

Maybe 230M with low batch or 360M with very low but higher no

well i remember 0.5b working in fp16, maybe ill be able to train something larger in fp8 later.

didnt even know yall used 16gb vram, i have a 4090 (24gb), so idk what that can even do with maxxed out vram

@simonko912 That 0.41B scale is actually a sweet spot for practical edge models. Getting basic coding, math, and instruction following at 410M with a clean Llama arch is solid work. Do you have a link to the model card or repo? Would love to test it out.

Its really not the best, but yes you can try. (for some reason sometimes the model doesnt stop, so set "[INST]" as stopword if you have issues)
https://huggingface.co/simonko912/simllama-1.1-instruct

@simonko912 Honestly, Chinchilla-optimal or undertrained beats over-baked weights any day. At least your parameters can still learn new things. What models or architectures are you working on lately? Always cool connecting with others experimenting in this space.

Just recently released a 0.41b model with the llama arch and llama tokenizer recently, trained it a bit on fineweb and then instruct tuned on a 3gb dataset for 2 epoches. Not the best but knows basic coding, math sometimes, some instructions and etc
How many tokens total?

not sure about tokens, never really mesure that but i used the llama 3 32k tokenizer, so you can probably figure it out

@simonko912 Honestly, Chinchilla-optimal or undertrained beats over-baked weights any day. At least your parameters can still learn new things. What models or architectures are you working on lately? Always cool connecting with others experimenting in this space.

Just recently released a 0.41b model with the llama arch and llama tokenizer recently, trained it a bit on fineweb and then instruct tuned on a 3gb dataset for 2 epoches. Not the best but knows basic coding, math sometimes, some instructions and etc

Llama overtrains, qwen overtrains, kimi overtrains deepseek overtrains

my are undertrained probs