DeepSeek-R1-Distill-Qwen-7B-NVFP4A16

NVFP4A16 quantization of deepseek-ai/DeepSeek-R1-Distill-Qwen-7B, produced with llm-compressor and published only after passing an automated accuracy gate against the bf16 baseline (see Certification).

Model overview

• Scheme: NVFP4A16 - 4-bit NVFP4 weights, 16-bit (bf16) activations (weight-only).
• Why weight-only: activations are transient (never stored), so keeping them at 16-bit costs ~nothing on disk - weights dominate size. Fully 4-bit activation quantization (W4A4) is widely reported to cost token-level quality; we have not measured that ourselves, and the one W4A4 candidate assay did gate failed the perplexity bar at +12.55% and was rejected. You get almost all the compression at a quality cost this card measures rather than estimates.
• Format: compressed-tensors; loads natively in vLLM (no --quantization flag).

Use with vLLM

from vllm import LLM, SamplingParams

llm = LLM(model="uist-labs/DeepSeek-R1-Distill-Qwen-7B-NVFP4A16")
prompts = ["Give me a short introduction to large language models."]
params = SamplingParams(temperature=0.6, top_p=0.95, max_tokens=256)
for out in llm.generate(prompts, params):
    print(out.outputs[0].text)

Hardware requirements

This checkpoint uses weight-only NVFP4: it runs via vLLM's FP4 Marlin kernel and does not require Blackwell GPUs or native FP4 tensor cores. On non-Blackwell hardware the 4-bit weights are dequantized to 16-bit for the GEMM, trading some compute throughput for the smaller memory footprint (the KV cache and activations are 16-bit either way, so the saving is on the linear weights).

Validated (measured through assay's gate): Ada (sm_89: e.g. L4, RTX 4090). Other >= sm_80 (Ampere / Hopper) GPUs are expected to work by the same weight-only Marlin path but are not yet independently validated here. Turing (sm_75) is excluded: a known vLLM issue makes the NVFP4 Marlin GEMM emit incorrect output on Turing, so we do not claim it until that is fixed and measured.

Creation

• Base snapshot: deepseek-ai/DeepSeek-R1-Distill-Qwen-7B @ 916b56a44061 - upstream repos are mutable, so the certificate names the exact commit it describes; the staged weights were verified against the recipe's pinned sha256s before quantization.
• Tool: llm-compressor, scheme NVFP4A16, targets Linear, ignore lm_head.
• Calibration: 512 samples from HuggingFaceH4/ultrachat_200k at 2048-token sequences.
• Pipeline: UIST Labs' assay benchmark-gated quantization pipeline (https://github.com/uistlabs/assay/tree/v0.6.1) - quantize -> benchmark -> gate -> publish.

Evaluation

Measured with lm-evaluation-harness (vLLM backend) on the bf16 baseline and this checkpoint. retention is quantized / baseline (higher is better; for perplexity, lower raw value is better).

task	metric	baseline	quantized	delta	+/-stderr	retention
aime24_avg	exact_match,avg	0.5437	0.5083	-0.0354	0.0281	0.9349
aime25_avg	exact_match,avg	0.3979	0.3646	-0.0333	0.0235	0.9162
minerva_math500	math_verify,none	0.8360	0.8740	+0.0380	0.0177	1.0455
gpqa_diamond_cot_zeroshot	exact_match,flexible-extract	0.3788	0.3434	-0.0354	0.0375	0.9067
wikitext	word_perplexity	31.2853	31.4296	+0.1442	-	1.0046

Gate: PASS - no task regressed beyond k=2 paired stderr (* = statistically significant regression at the recipe's k) No task showed a statistically significant regression.

Methodology and limitations

• Apples-to-apples deltas. The bf16 baseline and this checkpoint were evaluated with the identical harness and settings, so the delta/retention columns are a fair like-for-like comparison - which is what a quantization gate should measure: change from the original, honestly.
• Chat-mode evaluation. Tasks are evaluated with the model's chat template applied. The template shifts the absolute scores on both the baseline and the quantized model, so read the deltas, not the absolute values. The comparison stays valid because both sides are evaluated with identical settings.
• Scope: the certified quantity is the delta, not the absolute score. Both sides were measured in the same run against the same software stack, which is the only condition under which these two columns are comparable. Because this battery samples its answers, rerunning it on the same stack moves absolute scores by up to roughly the standard errors shown in the table above. Absolute scores can move further across harness or library versions, so a number here will not necessarily reproduce elsewhere, while the delta under identical conditions is what was certified. For the same reason we do not compare against a stored baseline from an earlier run: that would fold stack drift into the measurement.
• Per-task fail thresholds (this run). The gate fails a task only when its measured drop exceeds k=2 times the paired standard error of the per-item score differences. On this run those thresholds were: aime24_avg (avg@16, the mean of 16 samples per item) 5.6 pts; aime25_avg (avg@16, the mean of 16 samples per item) 4.7 pts; minerva_math500 3.5 pts; gpqa_diamond_cot_zeroshot 7.5 pts. A measured drop at or below a task's threshold passed as statistically indistinguishable from zero - so this certification does not assert that no regression exists below that size. Threshold width tracks each task's sampling noise, not its importance: a wide threshold means this run had limited resolving power on that task. Separately, the perplexity criterion is an independent hard bar (increase no more than 3%) that does not depend on this test at all.
• How the standard error is computed. This is a paired test: both evaluations score the identical items, so the standard error is computed from the per-item score differences (quantized minus baseline, item by item) rather than by combining the two sides' independent standard errors. Pairing credits the correlation the two sides share through item difficulty; the per-side stderr column in the table is informative only. These are the thresholds the gate actually enforced.
• One-sided upper confidence bound on the true regression. At the same k=2, measured drop + k*SE per task: aime24_avg 9.2 pts; aime25_avg 8.0 pts; minerva_math500 -0.3 pts; gpqa_diamond_cot_zeroshot 11.0 pts. This is the non-inferiority claim this certification makes: the data are consistent with a true per-task regression of at most these sizes.
• A sound quantization can still fail on noise. The per-task test is one-sided at k=2, a nominal false-alarm rate of about 2% per task, and every task is tested - so the run-level chance is higher than any single task's. We do not quote a run-level figure: the tasks share one checkpoint and are not independent. We accept those odds in the withholding direction, since a false alarm costs us a release while a missed regression would cost you a bad checkpoint.
• Run-level generation settings: temperature 0.6, top_p 0.95. These apply where a task generates its answer, and have no effect on any task the harness scores by loglikelihood. The baseline and this checkpoint were measured with identical settings.
• Retention near or above 100% means "no measurable loss," not "better." Where a task ticks up, that is sampling noise - a finite benchmark set is a sample, not the whole population - so read the whole table as "indistinguishable from the original," not as an improvement.
• Weight-only tradeoff. Weights are 4-bit; activations stay 16-bit. That keeps quality close to the original at nearly the full disk-size saving, at some inference-speed cost versus a fully 4-bit (W4A4) variant. We have not benchmarked a W4A4 build of this model, so we make no claim about where it lands on that curve.
• Bias, risks, and inherited behavior. This is a quantization of deepseek-ai/DeepSeek-R1-Distill-Qwen-7B and inherits its capabilities, biases, and limitations. The gate above certifies accuracy retention on the listed benchmarks only; it does not measure bias, safety, or any behavior those benchmarks do not cover, and quantization is not guaranteed to preserve what was not measured. For intended use, safety, and ethical considerations, refer to the base model's card.

Certification

UIST Labs publishes a quantized checkpoint only if it clears a hard, stated accuracy bar against its own bf16 baseline - we would rather withhold a release than ship an unverified one. This checkpoint passed all of:

• No statistically significant per-task accuracy regression: one-sided paired test, a task fails only if its drop exceeds k=2 times the standard error of the per-item score differences
• Perplexity increase <= 3%

The deltas above are the actual measured numbers, not vendor estimates. This gate runs on every UIST Labs quantization release.

Citation

If you use this checkpoint, please cite both this quantized release and the base model.

@misc{uistlabs_deepseek_r1_distill_qwen_7b_nvfp4a16,
  title        = {DeepSeek-R1-Distill-Qwen-7B-NVFP4A16: benchmark-gated NVFP4 quantization of DeepSeek-R1-Distill-Qwen-7B},
  author       = {{UIST Labs}},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/uist-labs/DeepSeek-R1-Distill-Qwen-7B-NVFP4A16}},
  note         = {Quantized and certified with the assay pipeline (https://github.com/uistlabs/assay/tree/v0.6.1); published only after passing an automated accuracy gate against the bf16 baseline.}
}

Please also cite the base model, deepseek-ai/DeepSeek-R1-Distill-Qwen-7B.

Produced by assay v0.6.1 - UIST Labs