sleepyeldrazi/DeepSeek-v4-Flash-REAP-K180-NVFP4

DeepSeek V4 Flash — REAP K180 Mixed NVFP4 (NVFP4 + Q2_K + Q8)

REAP-pruned DeepSeek V4 Flash at K180 (180 routed experts per MoE layer, 30% pruning), with a hybrid mixed-precision quantization targeting NVIDIA DGX Spark (GB10 Blackwell, sm_121). Supports full 1M token context on a single 128 GB DGX Spark via managed memory + FP8-packed KV cache.

At a Glance

Base model	DeepSeek V4 Flash
Source checkpoint	deepseek-ai/DeepSeek-V4-Flash (MXFP4/MXFP8 mixed precision)
Pruning method	REAP (Router-weighted Expert Activation Pruning) — Cerebras Research
Routed experts	180 per layer (down from 256)
Hash-preserved	Layers 0-2 (256 experts each — hash-routed layers must remain full-width)
Pruned	Layers 3-42 (180 experts each)
Format	ds4-compact-v2 GGUF with NVFP4 multi-tensor convention
File size	~99 GiB
Measured throughput	~12 t/s on DGX Spark (single-residency, managed memory, short context)
Max context	1M tokens (FP8-packed KV, ~116 GiB peak resident)

Mixed Quantization Structure

This model uses a hybrid quant strategy optimized for DGX Spark's bandwidth profile (273 GB/s LPDDR5X, ~97 GB/s managed-memory serving path):

Component	Quant	bpw	BW (GB10)
Routed experts — gate (w1)	NVFP4 (e2m1 + e4m3 per-16 + fp32 scale_2)	4.50	~140 GB/s
Routed experts — up (w3)	NVFP4 (e2m1 + e4m3 per-16 + fp32 scale_2)	4.50	~140 GB/s
Routed experts — down (w2)	Q2_K	2.625	~160 GB/s
Attention (q, k, v, o)	Q8_0	8.5	~228 GB/s
Shared experts (w1, w2, w3)	Q8_0	8.5	~228 GB/s
Output head	Q8_0	8.5	~228 GB/s
Token embeddings	F16	16	~208 GB/s
Norms, HC base/scale	F32	32	~247 GB/s
HC fn weights	F16	16	~208 GB/s

Why this mix?

The bottleneck in MoE decode is gate+up expert projection (~2/3 of expert bytes). Standard GGUF quants use IQ2_XXS here. NVFP4 reads more bytes (4.50 vs 2.06 bpw) but offers substantially better precision at the same bit budget, with dequant bandwidth at ~140 GB/s on GB10. In practice, NVFP4 decode is slightly slower than IQ2_XXS due to the higher byte volume, but the precision gain is well worth the small throughput cost. Down experts stay Q2_K (already saturating at ~160 GB/s). Attention and shared experts stay Q8_0 for quality.

NVFP4: lossless MXFP4→NVFP4 conversion

The HF source checkpoint stores experts in MXFP4 (e2m1 nibbles + e8m0 per-32 block scale). NVFP4 is e2m1 + e4m3 per-16 + per-expert fp32 scale_2. The e2m1 nibbles are identical — the conversion is a scale-only transform. No weight requantization, no quality loss from the quantization step. The NVFP4 weights are bit-identical to the MXFP4 originals; only the scale format changes.

Domain Split (Calibration)

8,000 prompts · 5.0M tokens · calibration on DGX Spark (NVIDIA GB10) at 4,096 token context. REAP activation_energy_sum2 score metric.

Domain	Share
Coding & development	35–40%
Agentic tool-calling	16%
Research & knowledge	15–20%
Math & science	10–15%
Design & planning	5–10%
Trivia & general QA	3–5%

How to Run

⚠️ Standard llama.cpp / GGUF runtimes will NOT load this model. It uses a custom NVFP4 multi-tensor GGUF convention (.nvfp4_weight + .nvfp4_scale_2) plus NVFP4 CUDA kernels, managed-memory serving, and FP8-packed KV cache.

Required Runtime

Clone and build the custom ds4 engine:

git clone https://github.com/sleepyeldrazi/ds4-nvfp4-spark
cd ds4-nvfp4-spark
make

The managed-memory path (required for K180)

The DGX Spark (GB10) has hardware ATS — the GPU reads CPU-allocated memory directly through the host page tables, coherently, no copy. The ds4-nvfp4-spark runtime uses cudaMallocManaged + cudaMemAdvise(SetReadMostly + SetPreferredLocation=device) + cudaMemPrefetchAsync to achieve ~97 GB/s single-residency serving, avoiding the 2× duplication of the original ds4's cudaMemcpy span cache. This saves ~10 GiB of RAM but costs 1-3 t/s depending on context length — the fault-and-migrate overhead scales with working set churn.

Without DS4_CUDA_MANAGED_MODEL=1, K180 will OOM.

Inference (DGX Spark / NVIDIA GB10)

# Short context — managed memory required
DS4_CUDA_MANAGED_MODEL=1 ./ds4 --cuda \
  -m DeepSeek-V4-Flash-REAP-K180-hybrid.gguf --ctx 32768

# 256K context — managed memory + FP8 KV
DS4_KV_TURBO=1 DS4_CUDA_MANAGED_MODEL=1 ./ds4 --cuda \
  -m DeepSeek-V4-Flash-REAP-K180-hybrid.gguf --ctx 262144

# Full 1M context — managed memory + FP8 KV
DS4_KV_TURBO=1 DS4_CUDA_MANAGED_MODEL=1 ./ds4 --cuda \
  -m DeepSeek-V4-Flash-REAP-K180-hybrid.gguf --ctx 1048576

API Server

DS4_KV_TURBO=1 DS4_CUDA_MANAGED_MODEL=1 ./ds4-server --cuda \
  -m DeepSeek-V4-Flash-REAP-K180-hybrid.gguf \
  --host 0.0.0.0 --port 17777 --ctx 1048576

Memory Budget (DGX Spark, 128 GB unified memory)

Resident figures include the ~7 GiB fixed UVM page-tracking overhead. The FP8 KV cache at 584 bytes/row (vs 2048 for FP32) saves ~7.2 GiB at 1M context.

Context	KV storage	K180 resident	Fits?
Short (8K)	FP32	~118 GiB	✅
32K	FP32	~120 GiB	✅
256K	FP32	~128 GiB	⚠️ (borderline)
256K	FP8	~122 GiB	✅
1M	FP8	~116 GiB	✅
1M	FP32	~130 GiB	❌

At 1M context with FP8 KV, K180 peaks at 116 GiB — under the 128 GB physical limit with ~12 GiB headroom for the OS. The UVM overhead (7 GiB) is a fixed cost, not reclaimable by Linux THP (UVM memory isn't THP-eligible).

Confirming single residency

# After loading, verify no duplication:
nvidia-smi  # "Memory-Usage: Not Supported" is expected on iGPU
# Check process RSS — should be ~model size + KV + overhead, not 2×:
grep VmRSS /proc/$(pgrep ds4)/status

How It Was Built

1. Source: deepseek-ai/DeepSeek-V4-Flash official MXFP4/MXFP8 mixed-precision checkpoint
2. REAP plan: 8,000 prompts → ds4 imatrix on DGX Spark → activation_energy_sum2 per-expert scores → top-180 per layer (layers 3–42), layers 0–2 preserved at 256
3. NVFP4 repack: gate+up expert e2m1 nibbles copied losslessly from MXFP4; e8m0 per-32 scales converted to e4m3 per-16 + fp32 scale_2; down experts kept as Q2_K
4. Copy policy: attention, shared experts, output head → Q8_0; embeddings → F16; norms/HC → original precision
5. Emission: single-pass GGUF generation via deepseek4-quantize (NVFP4 emission, REAP pruning, type-change tracking)

No fine-tuning. Purely structural expert removal + lossless NVFP4 repack. Weights are unmodified — a subset of the original MXFP4 experts.

Why REAP + Hybrid NVFP4?

Standard GGUF quants (IQ2_XXS, Q2_K, Q4_K) are compact but sacrifice precision at low bit widths. NVFP4 trades some speed for substantially better quality: it reads more bytes per weight (4.50 vs 2.06 bpw) but preserves the lossless MXFP4→NVFP4 mapping, meaning no requantization error on expert weights. The result is a model that prioritizes precision over raw throughput — slightly slower decode than an all-IQ2_XXS quant, but with dramatically better weight fidelity on the expert path.

Combined with REAP (30% expert pruning), managed memory, and FP8-packed KV:

• Fits K180 on a single 128 GB DGX Spark
• Serves full 1M token context
• Preserves attention quality (Q8_0)
• Has no quantization quality loss on experts (lossless MXFP4→NVFP4)

The GB10 has no TMEM, no WGMMA, no tcgen05 — it's consumer Blackwell (sm_121), not datacenter Blackwell. The NVFP4 __dp4a kernel (~140 GB/s) is the pragmatic v1; a native mma.mxf4 kernel (projected ~186–210 GB/s) is the next step.

For complete bandwidth analysis and kernel benchmarks, see the accompanying cuda_debug/ANALYSIS.md in the runtime repo.

Variants

Variant	Experts	Size	Best for
K128	128 (50% pruned)	76 GiB	Comfortable fit, 1M ctx
K150	150 (41% pruned)	86 GiB	Better quality, ~256K ctx
K180 (this)	180 (30% pruned)	99 GiB	Best quality, 1M ctx, managed memory required

Acknowledgments

• DeepSeek — DeepSeek V4 Flash base model
• antirez — ds4 engine and GGUF quants
• Cerebras Research — REAP expert pruning
• NVIDIA — DGX Spark (GB10) hardware
• eouya2 — reap-for-ds4 tooling