README.md

metadata

license: apache-2.0
base_model: Qwen/Qwen3.8-27B
tags:
  - mlx
  - mtplx
  - quantized
  - multi-token-prediction
  - speculative-decoding
  - qwen3.5
  - apple-silicon
pipeline_tag: text-generation

Qwen3.8-27B-MTPLX-8bit

MTP weights kept. This build does not strip the model's multi-token-prediction head — it is preserved at full bf16 precision (mtp_policy: keep_bf16) alongside the 8-bit quantized body, so native speculative decoding still works.

An 8-bit MLX build of Qwen/Qwen3.8-27B for Apple Silicon, produced with mtplx forge. Unlike a standard mlx_lm.convert quantization, this build preserves the model's native multi-token-prediction (MTP) head instead of discarding it, so it runs with real speculative decoding on mtplx — not just a quantized weight dump.

Why this build exists

Qwen3.8-27B is a hybrid architecture (Gated DeltaNet linear attention interleaved with Gated Attention, 16 groups of 3:1). Standard MLX conversion paths (mlx_lm.convert, and MLX inference engines built around dense-attention assumptions) silently strip the model's 15 MTP tensors during conversion — the model still loads and generates, but you lose the native speculative-decoding speedup entirely, often without any error or warning.

This build's recipe uses mtp_policy: keep_bf16, which retains the MTP block in full precision alongside the quantized body, and validates it with mtplx's tune verification suite (long-code-uncapped, 2048 max tokens) before shipping.

Quantization recipe

Parameter	Value
Body precision	8-bit affine
Group size	64
MTP head	kept at bf16 (not quantized)
Source	Qwen/Qwen3.8-27B (bf16 native)

8-bit affine at this group size is close to lossless relative to the bf16 source — this is the recommended variant when you have the memory headroom (64GB+ unified memory) and want maximum output fidelity rather than the smallest footprint.

Requirements

• Apple Silicon Mac (M-series), 64GB+ unified memory recommended
• mtplx — required to use the native MTP speculative-decode path. Standard mlx-lm can load this model's body weights but will not use the MTP head.

Usage

mtplx quickstart --model johninthepool/Qwen3.8-27B-MTPLX-8bit --port 8020 \
  --reasoning off --paged-kv-quantization q8

--reasoning off is recommended for agentic/tool-calling workloads — leaving reasoning on causes this model to spend a large number of tokens per turn on hidden thinking before it produces usable output.

--paged-kv-quantization q8 quantizes the KV cache to 8-bit, giving substantially larger usable context at negligible quality cost. Qwen3.8-27B natively supports up to 262,144 tokens of context (extendable to ~1M with YaRN).

What you get vs. what you give up

You get: near-lossless output quality relative to the source bf16 weights, native speculative decoding via the preserved MTP head, long-context serving via paged KV quantization.

You give up relative to the 4-bit sibling: roughly double the disk/memory footprint (~30GB vs ~17GB). If footprint or raw throughput matters more than peak fidelity, see johninthepool/Qwen3.8-27B-MTPLX-4bit.

Provenance

Built from the original Qwen/Qwen3.8-27B weights with no fine-tuning, distillation, or calibration pass applied — this is a direct affine (round-to-nearest) quantization of the release weights, with the MTP head kept unquantized. No behavioral changes beyond quantization noise are expected relative to the source model.