---
license: apache-2.0
base_model: Qwen/Qwen3.8-27B
tags:
  - mlx
  - mtplx
  - quantized
  - multi-token-prediction
  - speculative-decoding
  - qwen3.5
  - apple-silicon
pipeline_tag: text-generation
---

# Qwen3.8-27B-MTPLX-8bit

> **MTP weights kept.** This build does not strip the model's multi-token-prediction head — it is preserved at full bf16 precision (`mtp_policy: keep_bf16`) alongside the 8-bit quantized body, so native speculative decoding still works.

An 8-bit MLX build of [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) for Apple Silicon, produced with [mtplx](https://github.com/philipjohnbasile/mtplx) forge. Unlike a standard `mlx_lm.convert` quantization, this build **preserves the model's native multi-token-prediction (MTP) head** instead of discarding it, so it runs with real speculative decoding on mtplx — not just a quantized weight dump.

## Why this build exists

Qwen3.8-27B is a hybrid architecture (Gated DeltaNet linear attention interleaved with Gated Attention, 16 groups of 3:1). Standard MLX conversion paths (`mlx_lm.convert`, and MLX inference engines built around dense-attention assumptions) silently strip the model's 15 MTP tensors during conversion — the model still loads and generates, but you lose the native speculative-decoding speedup entirely, often without any error or warning.

This build's recipe uses `mtp_policy: keep_bf16`, which retains the MTP block in full precision alongside the quantized body, and validates it with mtplx's `tune` verification suite (`long-code-uncapped`, 2048 max tokens) before shipping.

## Quantization recipe

| Parameter | Value |
|---|---|
| Body precision | 8-bit affine |
| Group size | 64 |
| MTP head | kept at bf16 (not quantized) |
| Source | `Qwen/Qwen3.8-27B` (bf16 native) |

8-bit affine at this group size is close to lossless relative to the bf16 source — this is the recommended variant when you have the memory headroom (64GB+ unified memory) and want maximum output fidelity rather than the smallest footprint.

## Requirements

- Apple Silicon Mac (M-series), 64GB+ unified memory recommended
- [mtplx](https://github.com/philipjohnbasile/mtplx) — required to use the native MTP speculative-decode path. Standard `mlx-lm` can load this model's body weights but will not use the MTP head.

## Usage

```bash
mtplx quickstart --model johninthepool/Qwen3.8-27B-MTPLX-8bit --port 8020 \
  --reasoning off --paged-kv-quantization q8
```

`--reasoning off` is recommended for agentic/tool-calling workloads — leaving reasoning on causes this model to spend a large number of tokens per turn on hidden thinking before it produces usable output.

`--paged-kv-quantization q8` quantizes the KV cache to 8-bit, giving substantially larger usable context at negligible quality cost. Qwen3.8-27B natively supports up to 262,144 tokens of context (extendable to ~1M with YaRN).

## What you get vs. what you give up

**You get:** near-lossless output quality relative to the source bf16 weights, native speculative decoding via the preserved MTP head, long-context serving via paged KV quantization.

**You give up relative to the 4-bit sibling:** roughly double the disk/memory footprint (~30GB vs ~17GB). If footprint or raw throughput matters more than peak fidelity, see [johninthepool/Qwen3.8-27B-MTPLX-4bit](https://huggingface.co/johninthepool/Qwen3.8-27B-MTPLX-4bit).

## Provenance

Built from the original `Qwen/Qwen3.8-27B` weights with no fine-tuning, distillation, or calibration pass applied — this is a direct affine (round-to-nearest) quantization of the release weights, with the MTP head kept unquantized. No behavioral changes beyond quantization noise are expected relative to the source model.