Is the model trainable?
Lora, Lokrs, etc
Important, important thing for making the model relevant on actual use.
Thankss
Until they release the Encoder, it's not.
yes: https://docs.simpletuner.io/quickstart/MINIMAX_MUSIC/ better answer: not really in a useful manner just yet. don't waste your compute budgets on this; try ACE-Step or HeartMuLa instead.
LyCORIS and PEFT LoRA trainable via SimpleTuner, with lyrics + caption validation libraries too. though that's not easily configured via the webui just yet (the validation library), the example prompts and captions from their website are included as a built in validation prompt library if you're doing eg. a broad finetune and want to validate a number of separate concepts.
it works with distillation too, so, the next project i'm on with MM Music is to throw AnyFlow at it and bring it down to a decent 4-step generation pipeline in the flow transformer.
what we're limited by is the lack of RVQ encoder's semantic audio tokens from raw audio. i think we could reverse engineer one though if so inclined, i'm not sure whether i'd be going against the license to do so.
Nice.. Its something.
Would it make any sense to opensource a model to then not share a fundamental piece? A model that is not trainable/conditionable is pretty much a toy.
The RVQ encoder is required for properly fine-tuning the model. Without it, you can maybe adjust some tone and pitches, though you would be doing so 100% without proper conditioning. It is like training an image model but instead of captions, you are just typing pure nonsense random text in the encoder for the conditioning. I mean sure, you CAN train it that way. But you will not get good results. The RVQ encoder is absolutely needed.
that's not true, look at how the pipeline actually works. the depth decoder gives the model its tokens from text. the encoder just does it from raw audio.
the inference path mirrors training. you're not required to supply RVQ encoder output (audio references) at inference time. why would you think it'd be mandatory at training?
our training target for the flow matching objective is the VAE audio latents, not the RVQ encoder output.
we wouldn't even be able to run the model for inference if the encoder were required for training.
@bghira You are attempting to train on non aligned tokens. During inference, they are aligned because they were trained to be that way, with the RVQ tokenizer, something you will be breaking with your "training" method.. You are just swapping out a non aligned target and breaking the alignment and saying you trained it. It is like training an LLM without a tokenizer. You have your output tokens, but you are feeding complete garbage on the input.
Oh this is exciting! Following through on your ask now. I'll update in a few hours after if trains!
@scragnog — pooled-v4 is done, and your architectural read was right. We warm-started SimpleTuner's 169m-v4 final and
ran the same fine-tune we used for the 41M: the pooled 10,959-track corpus, unchanged architecture/loss/μP/trainer,
12 epochs at 3e-4 polynomial. On the synthetic holdout with the replay harness (which reproduces v4-base's published
0.8748 cosine / 0.432 sem top-1 exactly when we run it as control): replay conditioning cosine 0.8748 → 0.8936,
semantic top-1 0.4323 → 0.4701, semantic top-5 0.8056 → 0.8449, acoustic top-1 0.0730 → 0.0862. The corpus effect
composed cleanly with the causal depth decoder — it now leads every community encoder on every axis we can measure,
including both channels where our pooled-41M and base-v4 each held an edge before.
Weights + config + raw eval JSONs: Mothersuperior/open-rvq-encoder-minimax-music3-169m-pooled-v4. We'd love to see it
through your real-audio gate — on current evidence it should hold up, but your 13-track album eval is the test that
matters, and semantic diversity on real recordings (no oversmoothing) is the property we most want confirmed. If it
passes, the corpus + recipe are fully public, so the reference encoder stays reproducible end to end.
https://huggingface.co/Mothersuperior/open-rvq-encoder-minimax-music3-169m-pooled-v4
Yep, nailed it! Definitely the best reconstruction so far.
- Pascal 41M — semantic CE 8.45 (top-1 5.8%), acoustic CE 6.76 (top-1 2.3%)
- pooled-v3 41M — semantic CE 7.26 (top-1 8.5%), acoustic CE 6.35 (top-1 4.5%)
- SimpleTuner v4 169M — semantic CE 7.16 (top-1 8.9%), acoustic CE 5.64 (top-1 5.8%)
- pooled-v4 169M — semantic CE 6.84 (top-1 10.9%), acoustic CE 5.37 (top-1 7.5%)
Aaaaaand no oversmoothing — semantic diversity is bit-for-bit level with v4 (1949 vs 1945 unique on burn, top-1 share unchanged), acoustic diversity slightly higher. The corpus effect and the architecture effect composed cleanly on our data too — both of the community's independent improvements stack.
Latest test on this encoder, can be compared to the original V4 above.
At this point, would it even benefit us for me to create another 8k corpus examples continuing with my diversity prompt forge and doing the loop one more time with near 20k corpus?
Some more examples in different genres.
Seems like it falls apart on alt-metal track the worst, likely because of the wall-of-sound.
Original
Reconstruction
Original
Reconstruction
Original
Reconstruction
Original
Reconstruction
At this point, would it even benefit us for me to create another 8k corpus examples continuing with my diversity prompt forge and doing the loop one more time with near 20k corpus?
I think so yes. I'm considering doing something similar here through Modal.
I think so yes. I'm considering doing something similar here through Modal.
Cool! I'll spin mine back up and make some more examples with a slightly shifted focus on trying to improve alt rock and metal.. which is good cause that's what I want this shit for any way :D
@Mothersuperior we have a Discord group where there's a few of us working through the problem if you're up for joining - https://discord.gg/8UccZvHzB