Open to Work: The Musical — distilling a 20B lyricist into a 0.5B model that runs on CPU

Community Article

Published June 15, 2026

Upvote

Tanya K

tanya8997

build-small-hackathon

Community Article · Build Small Hackathon, Track 2: An Adventure in Thousand Token Wood

I graduated recently, and I have spent more hours than I would like inside the job-hunt loop: same PDF, same cover letter, tweak a bullet, hit Apply, refresh an inbox that does not refresh back. The market is brutal, and the feedback loop is mostly silence.

So when I found a hackathon whose thesis was, “take it back to when models were small enough to tinker with and the vibe was hopeful,” I made the most unserious possible answer to the most serious problem in my life: an app that turns your résumé into a parody musical about wanting one specific job.

You upload a résumé, a job description, a photo, and optionally a short voice sample. The app writes custom lyrics, composes the music, sings it, optionally converts the vocal into your voice, and lip-syncs the whole thing to your photo. One slider controls how unhinged it gets, from 1: Actually Sendable to 10: HR Has Left the Chat.

This is what I built, and what I learned making it run small.

How it works

The basic flow is:

Upload a résumé and job description.
Add a photo, and optionally a 10-second voice sample.
Drag the Sendability Slider and choose a genre.
Render a shareable parody music video.

The pipeline looks like this:

RÉSUMÉ ─┐
JOB ────┼─► THE PRODUCER ──► lyrics ──► ACE-Step ──► song ──► demucs ──► SadTalker ──► video
GENRE ──┤   draft→critique        diffusion          stem split    lip-sync
SLIDER ─┘   →revise loop          music gen

                                seed-VC ──► optional voice conversion

Everything runs inside the Hugging Face Space: ZeroGPU for the heavier stages, CPU for the tiny lyricist path, and no external inference APIs.

Technical details

The agent: a writer that critiques its own drafts

The lyrics are not one-shotted. The “Producer,” powered by openai/gpt-oss-20b, runs a small reflection loop:

read the materials → set the sendability level → choose a genre
→ draft the lyrics → critique the draft → revise
→ pass style tags to the music model

The critique step is the interesting one. The model reads its own draft back against the target Sendability level using a strict rubric: SCORE, KEEP, and FIX. Then it rewrites accordingly.

The critic and the writer are the same model under different system prompts. The loop is simply:

generate → evaluate → revise

Every step is logged and published as a dataset, so the behavior is inspectable rather than hidden inside the demo.

Two implementation details mattered a lot:

Structured output plus a forgiving parser. The model is asked to return a clean GENRE / TITLE / LYRICS format, with [verse] and [chorus] tags. Small models drift, so the parser is forgiving: if the format wobbles, it salvages the useful parts instead of failing the whole run.

Reasoning-channel handling. gpt-oss uses OpenAI’s Harmony-style separation between reasoning and final output. For this project, I had to keep only the final performance text. Otherwise, the model’s out-loud deliberation can leak into the lyrics, which is funny once and broken every time after.

Distilling the Understudy

This is the part I am proudest of.

The 20B Producer writes strong lyrics, but it needs a GPU. For instant slider previews, I wanted a path that did not touch GPU quota at all. So I built an understudy: a model 40× smaller that learned to imitate the Producer.

The technique is teacher-student knowledge distillation. The 20B teacher generated the training set that did not exist: around 1,500 synthetic parody songs across fictional careers, genres, and sendability levels.

Then I LoRA fine-tuned openbmb/MiniCPM4-0.5B to imitate it.

LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
    target_modules="all-linear",
)

# 3 epochs, one A100, ~30 minutes, eval loss 0.718

After training, I merged the adapters, converted the model to a 4-bit Q4_K_M GGUF, and serve it through llama.cpp on plain CPU.

In Tiny Mode, the 0.5B Understudy replaces the 20B Producer entirely. The whole résumé → song → video pipeline runs at about 2.8B parameters, with every model under 4B.

One real gotcha: the 0.5B model kept parroting memorized lines. Because it was trained on the teacher’s calibration examples, some outputs repeated those examples instead of adapting to the résumé. The fix was a mix of decoding controls, including repeat penalties and presence penalties, plus an anchor-echo detector. If the output contains a known memorized phrase, the app regenerates with a hotter temperature.

The cleaner long-term fix is data-side: curate those anchor examples out of the fine-tuning set.

The rest of the stack

The full system uses a small set of specialized models:

openai/gpt-oss-20b: the full Producer / teacher model
openbmb/MiniCPM4-0.5B: the base model for the CPU-running Understudy
ACE-Step: diffusion-based music generation
demucs: source separation for vocal and instrumental stems
Plachta/Seed-VC: optional zero-shot voice conversion
SadTalker: headshot animation and lip-sync
vikhyatk/moondream2: photo vibe read

The slider does not only rewrite the lyrics. It also changes the style tags passed to the music model, so the generated music shifts with the words.

Inference architecture

The runtime is ZeroGPU plus CPU.

The heavy stages run through ZeroGPU. The instant-preview path, powered by the Understudy, never touches a GPU.

In my setup, ACE-Step was more stable when re-initialized inside each GPU call rather than at app startup. That means each full render pays an initialization cost, but avoids the startup and worker-state issues I was hitting during deployment.

What broke

Nothing small was actually easy.

ZeroGPU and ACE-Step needed careful handling. In my setup, torch.cuda.is_available() and ACE-Step’s startup CUDA checks did not play nicely with the ZeroGPU environment. The result was an NVML-related failure during initialization. The fix was to avoid initializing the model too early, then initialize it inside the GPU call.

The newer transformers stack interacted badly with the model init path. ACE-Step’s quantizer called .item() during initialization, while part of the model-loading path was on the meta device. That combination failed, so I had to patch the init path to get the model to load cleanly.

Dependency Tetris was real. I originally wanted OpenBMB’s MiniCPM-V for the photo read, but it pinned an older transformers version that could not coexist with the version ACE-Step needed. That was a version conflict, not a model-quality problem. So the photo read moved to moondream2, and MiniCPM stayed where it mattered most: the Understudy.

Most of the weekend was not spent on the glamorous part. It was spent making older vision code, a diffusion music model, and a CPU-quantized LLM coexist inside one Space without setting each other on fire.

What I learned

A hard parameter cap can be a feature, not just a constraint. It forced better design decisions: what is the smallest model that can do this job, and how do I connect the pieces so the whole thing still feels fun?

Distillation is the unlock for tiny end-to-end systems. A 0.5B model trained on a 20B model’s synthetic output was good enough to carry the lyricist role for instant previews, and it runs on CPU.

Small models need babysitting. Repetition, anchor-parroting, and format drift are real. The craft is in the guardrails: decoding controls, tolerant parsers, and regenerate-on-echo checks.

Most of a small-model build is integration. The fine-tune was not the hardest part. The hard part was making all the models coexist in one user-facing app.

Try it

Try the Space here:

Open to Work: The Musical

Drop in a résumé, drag the slider, and see what your application sounds like at level 10.

Thank you

Thank you to Hugging Face and Gradio, the organizers of the Build Small Hackathon, for a prompt that made small models feel exciting again. The “think small, stay hopeful” brief genuinely shaped the build.

Thank you to OpenBMB for MiniCPM4-0.5B, the model that makes Tiny Mode possible. The Understudy is your model, fine-tuned, quantized, and running on CPU.

Thank you to Modal for making the risky parts easy to test before they touched the Space: the synthetic dataset generation, the fine-tune, the CPU wheel, and the model-integration experiments.

And thank you again to Hugging Face for Spaces and ZeroGPU, which made it possible to run the whole thing in one place without hiding the demo behind external inference APIs.