canada-quant

/

GLM-5.2-W4A16-MTP

like 17

Follow

Canada Quant Labs 25

Hi @madeby561 — short answer: no custom vLLM image is needed to load k_norm. Those are the DSA-indexer / MLA norm tensors, and they're kept at BF16 in this artifact (the W4A16 quant only touches the routed experts), so they load as ordinary weights through GLM-5.2's own modeling code. You just need a recent stock vLLM (≥ 0.23, which has native GlmMoeDsaForCausalLM) and --trust-remote-code. That part is the same regardless of GPU.

The validated setup is Hopper / H200: stock vLLM ≥ 0.23 on a CUDA ≥ 12.8 toolchain (12.9 recommended — upstream's vllm/vllm-openai:glm52-cu129 image is the easy path; the DSA indexer JIT-compiles DeepGEMM kernels that need ≥ 12.8). Serve with --enable-expert-parallel, since the asymmetric W4A16 MoE needs expert parallelism:

  --tensor-parallel-size 8 --enable-expert-parallel \
  --kv-cache-dtype fp8 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
  --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
  --max-model-len 1048576 --gpu-memory-utilization 0.90 --trust-remote-code```


We're benchmarking on 8× RTX PRO 6000 (Blackwell / SM120) right now and will post that config + numbers as soon as we have them. The k_norm loading answer above is unchanged on that hardware — if anything extra is needed there it'll be on the INT4 Marlin expert-kernel side, not k_norm.

If you do hit a k_norm-specific error, it's almost always an older vLLM (< 0.23) or a missing --trust-remote-code — share the traceback + your vLLM/CUDA versions and I'll help.

As noted, I can't post this for you from this session (no reachable HF write token, and the HF MCP here is read-only). Paste it into the discussion, or set HF_TOKEN in the environment and I'll post it via the HF API. Want me to also drop this into a short FAQ on the model card / README on the claude/k-norm-vllm-loading-wsc2k6 branch?

A RTX Pro 6000 config would be amazing. Thank you for the reply

Hi @madeby561 — as promised, here's the validated RTX PRO 6000 config. We ran this model on 8× RTX PRO 6000 (96 GB, SM120) with MTP speculative decoding and cudagraphs both working, zero token corruption, and quality at parity with our H200 results (GSM8K 0.948, IFEval 0.909/0.920, MATH-500 0.954 math-verify, RULER@32K/64K 0.918/0.826). Throughput: 50 tok/s @ c=1 up to 989 tok/s @ c=64.

The full recipe (patched-image Dockerfile, complete docker run command, and why each flag is needed) is now in the model card under "Serving on Blackwell — 8×RTX PRO 6000 (SM120)". The short version:

1. Patch the image. SM120 has no sparse-MLA kernel for GLM-5.2's DSA head layout, so DSA sparse attention must be disabled (is_v32 = False in deepseek_v2.py) and the model served through dense TRITON_MLA; three more one-liners skip the now-orphaned indexer weights at load time. All four patches (applied on top of the official vllm/vllm-openai:glm52 image) are in the card as a copy-paste Dockerfile.

2. Serve with env VLLM_USE_DEEP_GEMM=0 VLLM_MOE_USE_DEEP_GEMM=0 NCCL_P2P_DISABLE=1 and:

--tensor-parallel-size 8 --enable-expert-parallel \
--attention-backend TRITON_MLA \
--speculative-config '{"method":"mtp","num_speculative_tokens":1}' \
--compilation-config '{"cudagraph_mode":"PIECEWISE","cudagraph_capture_sizes":[2,4,8,16,32,64],"max_cudagraph_capture_size":64}' \
--max-model-len 131072 --max-num-seqs 64 --gpu-memory-utilization 0.92 \
--reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
--disable-custom-all-reduce --trust-remote-code

Gotchas that cost us days, so you don't repeat them:

• Keep the KV cache bf16 — don't pass --kv-cache-dtype fp8. fp8 KV overflows SM120's per-SM shared memory in the TRITON_MLA kernel (102,400 > 101,376 bytes) and the server won't come up.
• cudagraph_mode must be PIECEWISE — FULL decode graphs + MTP silently produce degenerate output on TRITON_MLA (its decode kernel only handles single-token queries; MTP's verify step sends multi-token queries). PIECEWISE routes them through the prefill path correctly and is still ~10× faster than eager at c=1. This is an upstream kernel gap (vllm-project/vllm#21505), not a config bug.
• MTP num_speculative_tokens=1 on this backend, with cudagraph capture sizes divisible by (1 + k) = 2.

Expectation-setting: single-stream (~50 tok/s) is memory-bandwidth-bound on this card vs ~126 tok/s on H200 — where it shines is concurrent serving (989 tok/s @ c=64, ~2.2× what nvidia/GLM-5.2-NVFP4 without MTP did on the same box). And the k_norm answer from earlier is unchanged: no custom loading logic needed for those tensors, the patches above are only about DSA/indexer on SM120.

Enjoy, and let us know how it runs on your setup!

run in A100 is okay?