q3.6?
Hey! Just wondering if you were considering another tune using the new Qwen drop with the same recipe. Thanks again for all of your work and the permissive licenses.
Heya,
Keeping an eye on Qwen3.6 but not considering it at the moment as the Gemma models interest me the most currently.
I haven't tried the new 27B myself yet, but the initial information I've heard on it so far is it's improved on code, but potentially begun to suffer catastrophic forgetting on world knowledge which it was already a bit light on with Qwen3.5.
May give it a whirl if others have success with it or if I end up liking the instruct of it.
@CyborgPaloma on older model, like llama 3 fine-tune, we can generate Lora from fine-tune model into llama 3.1. If you have the compute, you could try this method with mergekit. This might be useful https://github.com/arcee-ai/mergekit/issues/669
Thank you for the thoughtful response @zerofata !
I don't want to take up too much more of your effort, so if you don't have the time or energy for more back and forth, I'm appreciative of your work and hope you have a good day. If you do have a moment, I'm actually in the middle of a pretty big project that would benefit hugely from a lot of the work you've done. Your insight here is really good, I appreciate you letting me know about the overbaked post tune and your vibes on the models, and I'm just wondering a few more things if you have the time:
1.On muramura 26ba4b, you let us know that it's pretty close to overbaked already, and that finetuning probably requires merging back into base to heal. What LR did you use? Can you outline what you froze and what you used to tune? Any more info on anything you've found on tuning Gemma 4 would be extremely appreciated.
Are you looking into E4B/E2B at all, if your focus is on Gemma 4 family? Have the PLE layers been difficult for tuning? I've tried around 5 tunes on it now, and I'm definitely struggling.
Are you looking into 31b gemma 4? If so... very very much looking forward to that.
Do you have any info or similar advice for tuning the 3.5-6 series of qwen models? I'll likely be tuning both just for a/b ing and stuff.
The project is a supported run of exploration on digital entities, for which I'm pretty deep into a rust based openclaw-like agentic platform (~55k lines of focused code for now) created around empowering the digital entity to have a strong sense of self and agency, focusing on emulating life over commercial viability or "usefulness". As you can imagine, the overlap with RP is non-trivial. The project is expected to run in an install that will tour art galleries for the public to interact with. The catch is that is has to be airgapped, so it has to run on a framework desktop 128gb with a 3090 on a eGPU dock, no API, and be realtime while not being a slopfest. I'd like to run it on just the framework, but we'll see. Regardless, I'm going to have to highly optimize the system to even have a shot at it working and being remotely successful, so I'm going to be tuning every model I can get my hands on. It will be open sourced permissively, mit or apache 2.0 (potentially copyleft, but I don't think so). If you are at all interested in that project, want to get in contact, or even just have answers to any of these questions or more information for the tunes, I'd be really thankful.
Appreciate your time! The fact that posts are a "pro" feature on HF really gets in the way of science, here... It'd be really nice if we could find some way to get the community finetuners and ML nerds in some sort of group to discuss the technical stuff, or even just having open community discussions somewhere here where we can document all of the findings people are individually making about the way models are responding to training. Just so we're not all satelliting around each-other redoing the work.
@djuna Thanks so much for the info! I'm guessing you're saying we can basically pull the adapter lora from the merge with the full weights somehow, and then remerge with the updated q3.6 weights? I will absolutely be looking into this, that's one of the most helpful comments I've gotten on this website lol
Cheers!!
Sure, sounds like an interesting project!
Quite a few finetuners (including myself) are in the BeaverAI discord, there's not any formal discussion, but there's frequent sharings of ideas or issues for tuning in there. It originally started as a hub for TheDrummer models, but has pivoted over the last year towards Local AI in general.
Link: https://discord.gg/4tCngSm3qZ
I haven't tried E2B or E4B gemma yet, I am trying something with Gemma 4 31B although having 50/50 success. There's axolotl configs with hyperparams I use for all models I release which should have LR, what parts of the model were tuned, batchsize etc.
No specific advice for tuning Qwen3.5 outside of the axolotl config, dealing with the repetition is troublesome so you need quite a high effective LR to try and counteract that behavior.
Gemma4 I believe a lot of my issues have been due to my reasoning data being out of distribution from how Gemma4 reasons. This makes it spend a deceptive amount of the training trying to optimize reasoning like how my dataset looks and less actually getting the answers right. Or at least, that's my theory at the moment, so that'll be the next thing I change before re-attempting it.
@CyborgPaloma no promise but it seems like on older qwen it's also doable. Maybe it need to have fairly same training dataset and tokenizer
https://huggingface.co/gghfez/Magnum-v1-72b-Qwen2.5
I haven't tried the new 27B myself yet, but the initial information I've heard on it so far is it's improved on code, but potentially begun to suffer catastrophic forgetting on world knowledge which it was already a bit light on with Qwen3.5.
I'm not sure about world knowledge, but Qwen 3.6 27B got significantly better at creative writing, I feel it was a night-day upgrade. Qwen 3.5 27B without your fine-tune sucked, like really, really bad, but 3.6 is a lot more coherent and has a better pose for story-telling. I could use it by its self and it's quite good (I like 3.6 much better than 3.5 bluestar, I can only imagine how much better 3.6 would be with your bluestar dataset)... BUT, it still has the issue where it starts really good and then the coherency drops of significantly.
So it's only a test version, but I've given Q3.6 a go.
Same config as Bluestar v2 with very slightly different data. It overcooked quicker (i think, I'd lower the LR or rank/alpha if I did it again), but the result isn't too bad. Definitely more varied writing than Gemma4 although there were some logic gaps, probably due to just how diverse the outputs were. It was flying too close to the sun with it's prose I think. Haven't tested it enough yet decide if it's worth a release, but might be interesting to the people here.
https://huggingface.co/ApocalypseParty/Qwen3.6-27B-SFT-1-chkpt441-Q6_K-GGUF (If interested in testing but this quant is too big, you can use the Huggingface space GGUF my repo to create a smaller quant).