vibethinker-3b-optiq-5bpw-mlx

MLX quantization of WeiboAI/VibeThinker-3B for Apple Silicon.

Variant: OptiQ mixed-precision (target 5.0 bpw)
Disk size: 2150 MB
Quantized by: sahilchachra

About this quantization

Unlike uniform 4-bit quantization (which forces every layer onto the same bit grid and often collapses reasoning at low bit widths), this model was quantized with mlx-optiq using per-layer KL-sensitivity analysis:

1. A small calibration set (32 samples spanning prose, multi-step reasoning, code, and constraint-following instructions) is run through the FP16 reference and through trial quantizations of each layer.
2. The output drift per layer is measured. Layers whose outputs are most affected by quantization (typically the final attention projections, the lm_head, and a few middle blocks) get more bits; layers that tolerate aggressive quantization get fewer.
3. The final assignment hits the target average bits-per-weight while keeping the bits where they matter. This trades off precision unequally so the average comes out near the target (5.0 bits/weight), but the bits that matter most for output fidelity stay high.

Quantization config

• Method: optiq_mixed_precision (mlx-optiq)
• Target bits/weight: 5.0
• Achieved bits/weight: 5.013
• Candidate bits: [3, 4, 6, 8]
• Group size: 64
• Sensitivity reference: bf16
• Calibration: 32-sample 4-domain mix (prose + reasoning + code + constraints)

Per-layer bit allocation

252 quantizable components total. OptiQ allocated bits non-uniformly based on KL sensitivity:

Bits	Components	Share
8-bit	73	29.0%
6-bit	101	40.1%
4-bit	59	23.4%
3-bit	19	7.5%
Total	252	100.0%

Benchmark results

Evaluated on Apple M5 Pro with MLX. Model loaded once; performance and quality measured in a single pass.

Performance

	This model	FP16 baseline
Decode tok/s (steady-state)	57.46	45.34
Prefill tok/s (steady-state)	388.6	341.91
Decode tok/s (avg, long traces)	69.39	44.08
Peak memory (GB)	3.207	6.936
Disk size (MB)	2150	5902

Warmed, short-prompt, chat-templated, thinking disabled. Represents steady-state decode for typical chat use; long thinking traces will be slower due to KV-cache growth.

Quality

Benchmark	This model	FP16 baseline	n
MATH-500 (math reasoning)	90.0% (answered 29/30)	93.3% (answered 30/30)	30
AIME 2024 (competition math)	80.0%	76.7%	30
IFEval (instruction following)	61.4%	61.4%	44
HumanEval (code, pass@1)	83.3%	73.3%	30
MMLU (knowledge, accuracy)	50.0%	40.0%	50

MATH-500 per-level accuracy

Level	This model	FP16 baseline
level 1	100.0%	100.0%
level 2	100.0%	100.0%
level 3	83.3%	83.3%
level 4	83.3%	100.0%
level 5	83.3%	83.3%

Context scaling (decode tok/s)

Context length	Decode tok/s
~128 tokens	58.4
~256 tokens	59.4
~512 tokens	59.5
~1024 tokens	58.8

Prompt format & recommended settings

VibeThinker-3B is a reasoning model (built on Qwen2.5-3B). It thinks inside a <think>…</think> block and then writes its final answer, so the way you call it matters:

• Always apply the chat template (tokenizer.apply_chat_template(..., add_generation_prompt=True)). Passing a raw string skips the <|im_start|> / <|im_end|> markers the model was trained on.
• Give it room to think. It can emit thousands of reasoning tokens before the answer — use max_tokens of at least 8192 (16384+ for hard competition math). Small caps cut it off mid-thought.
• For math, ask for the final answer in \boxed{} — that's how it was trained and how the benchmarks below were scored. Parse the text after </think> for the answer.
• Context length: up to 64K tokens.
• Stop token: these MLX variants set eos_token_id to include <|im_end|> (151645) so generation halts cleanly at the turn boundary.

Recommended sampling (from the original model card): temperature=1.0, top_p=0.95, top_k disabled.

The benchmark numbers above were produced with greedy decoding (temperature 0) for reproducibility — so a variant's score reflects quantization damage, not sampling noise. For everyday use, the recommended sampling settings give better, more diverse reasoning.

Usage

pip install mlx-lm

from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

model, tokenizer = load("sahilchachra/vibethinker-3b-optiq-5bpw-mlx")

messages = [{"role": "user",
             "content": "Let x be the number of ways to ... . Give the final answer in \\boxed{}."}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=False,
)

# Reasoning needs a large token budget; sampling per the model card.
sampler = make_sampler(temp=1.0, top_p=0.95)
response = generate(
    model, tokenizer, prompt=prompt,
    max_tokens=16384, sampler=sampler, verbose=True,
)
# `response` contains a <think>...</think> trace followed by the final answer.

All variants in this collection

Model	Variant
sahilchachra/vibethinker-3b-mxfp4-mlx	Block float MX FP4
sahilchachra/vibethinker-3b-mxfp8-mlx	Block float MX FP8
sahilchachra/vibethinker-3b-optiq-5bpw-mlx	OptiQ mixed-precision (target 5.0 bpw) ← this model

Notes

• Requires Apple Silicon (M1 or later) with MLX
• Benchmarks run on Apple M5 Pro, 24 GB unified memory
• License: see WeiboAI/VibeThinker-3B for the original model's license

Original model

See WeiboAI/VibeThinker-3B for full model details and intended use.