NemotronLabs-VoiceChat-11B — MLX (4bit)

MLX conversion of nvidia/NVIDIA-NemotronLabs-VoiceChat-11B, a duplex speech-to-speech model that listens and speaks at the same time. 11.1 B parameters, converted from the original float32 release (44.4 GB) for Apple Silicon.

tier	size	quantization
bf16	22.2 GB	none
8bit	13.9 GB	8-bit, LLM + vocab heads only
4bit (this repo)	9.2 GB	4-bit, LLM + vocab heads only

What is inside

The conversion keeps all four components of the model, complete and in the original layout.

component	parameters	description
language backbone	7.71 B	Nemotron-H hybrid — 56 layers: 27 Mamba-2, 25 MLP, 4 grouped-query attention. Hidden size 4480, vocabulary 131072
vocabulary heads	1.76 B	token embedding, LM head, and a function-calling head
speech encoder	0.61 B	Conformer encoder with a mel front end
speech generation	1.00 B	Gemma-3 TTS backbone, mixture-of-Gaussians head, neural audio codec, 31-stage residual VQ

Duplex operation runs at 80 ms frames (12.5 frames/s), 16 kHz input and 22.05 kHz output.

What runs in MLX today

Two parts of the model are directly usable, with runnable examples below:

the language backbone, via mlx-lm's nemotron_h implementation
the audio codec — waveform to residual-VQ codes and back

The Conformer speech encoder and the full duplex loop are not yet implemented in MLX. The weights for them are present and complete in this repository.

Example 1 — text generation with the language backbone

pip install mlx mlx-lm
python mlx/extract_llm.py --src . --dst ./voicechat-llm
python mlx/chat_example.py --model ./voicechat-llm --prompt "The capital of France is"

The source model ships no text tokenizer, so extract_llm.py pairs the backbone with a Nemotron-H base tokenizer (vocabulary 131072) and verifies the size matches before writing.

Example 2 — audio codec round trip

pip install mlx numpy
python mlx/codec_example.py --repo . --wav input.wav --out output.wav

Encodes a waveform to 512-dimensional latents and 31-stage codes, then decodes back to audio. Reconstruction is about 8 dB SNR at 12.5 frames per second, and runs roughly 6x faster than real time on an M4 Max.

Quantization

The language backbone and the three vocabulary heads are quantized to 4 bits (group size 64) — 9.47 B of the 11.10 B parameters. The entire speech path is left in bfloat16: the audio codec, the mixture-of-Gaussians head, the Conformer encoder and the RNN-T decoder. Those are small tensors whose error would land directly on audio quality for a negligible size saving.

Mean relative error against the bf16 tier, measured per tensor: 9.69 %.

Measured performance

Apple M4 Max, 68.7 GB unified memory, language backbone only, greedy decoding:

	load	peak memory	first token	throughput
bf16	3.1 s	17.96 GB	646 ms	23.4 tok/s
8-bit	1.9 s	10.22 GB	803 ms	33.2 tok/s

Audio codec: about 6x real time.

Notes

Text-only prompting is outside the model's training distribution. It was trained to emit interleaved text and audio-codec tokens, so generations may end with audio-side tokens such as <SPECIAL_12>.
The codec expects 22.05 kHz mono audio. Other rates decode at the wrong speed.
Every tensor from the source checkpoint is present; the conversion is verified for completeness and for finite values in all tiers.

License and attribution

Released under OpenMDW-1.1, following the original model. Base model and architecture by NVIDIA; see nvidia/NVIDIA-NemotronLabs-VoiceChat-11B. The MLX audio-codec implementation follows NVIDIA NeMo Speech.