Instructions to use utarn/DeepSeek-V4-Flash-0731-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use utarn/DeepSeek-V4-Flash-0731-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="utarn/DeepSeek-V4-Flash-0731-NVFP4")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("utarn/DeepSeek-V4-Flash-0731-NVFP4")
model = AutoModelForCausalLM.from_pretrained("utarn/DeepSeek-V4-Flash-0731-NVFP4", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use utarn/DeepSeek-V4-Flash-0731-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "utarn/DeepSeek-V4-Flash-0731-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "utarn/DeepSeek-V4-Flash-0731-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/utarn/DeepSeek-V4-Flash-0731-NVFP4

SGLang

How to use utarn/DeepSeek-V4-Flash-0731-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "utarn/DeepSeek-V4-Flash-0731-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "utarn/DeepSeek-V4-Flash-0731-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "utarn/DeepSeek-V4-Flash-0731-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "utarn/DeepSeek-V4-Flash-0731-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use utarn/DeepSeek-V4-Flash-0731-NVFP4 with Docker Model Runner:
```
docker model run hf.co/utarn/DeepSeek-V4-Flash-0731-NVFP4
```

Fails to load on RTX PRO 6000 Blackwell (sm120): DeepGEMM `t.dim()==N` in the ue8m0 block-FP8 path

by tlanderso - opened 9 days ago

Discussion

tlanderso

9 days ago

Hardware: 4x RTX PRO 6000 Blackwell Workstation (compute 12.0 / sm120), CUDA 13.2, TP=4.
Runtimes tried: stock vLLM 0.23.0, and a DeepSeek-V4-aware vLLM fork (0.11.2.dev ... cu132).

The checkpoint loads cleanly — arch DeepseekV4ForCausalLM, ModelOpt MIXED_PRECISION experts-only NVFP4, scale_fmt=ue8m0 detected.

On sm120 the working config is:

--kv-cache-dtype fp8 (the fp8_ds_mla layout asserts on auto: "DeepseekV4 fp8_ds_mla layout only supports fp8 kv-cache, got auto").
MoE backend must be flashinfer_cutlass: b12x refuses because the model sets swiglu_limit=10.0 and that backend doesn't apply the clamp ("moe_backend='b12x' does not apply the SwiGLU clamp"); flashinfer_trtllm reports "does not support current device cuda" (sm120); marlin is available. flashinfer_cutlass loads and applies the clamp.

Blocker: on the first forward (memory profiling / cudagraph capture), DeepGEMM asserts:

RuntimeError: Assertion error (.../jit_kernels/impls/../heuristics/../../utils/layout.hpp:39): t.dim() == N

This reproduces on both runtimes, and VLLM_USE_DEEP_GEMM=0 does not avoid it (DeepGEMM is invoked from a path that flag doesn't gate). It appears tied to the ue8m0 block-FP8 scale layout of the non-expert (attention/dense) tensors.

Question: Is the ue8m0 block-FP8 scale format expected to work with current DeepGEMM on sm120, or does it need a different scale_fmt/packing? Any known-good runtime + commit? Happy to test fixes — I can reproduce quickly and provide full logs.

utarn

Owner 9 days ago

•

edited 9 days ago

ue8m0 block-FP8 scale format does not work with current DeepGEMM on sm120. I need to use another backend. I have uploaded the fixed version for marlin backend and sm120 support.

tlanderso

9 days ago

Update: the W4A16 / Marlin re-quant (c2a8a2f) fixes the load crash — it now loads and serves on sm120. The one remaining blocker is a DeepGEMM sm120 gap, not the checkpoint. Thanks for the fast turnaround! 🙏

Results on 4× RTX PRO 6000 (sm120), vLLM 0.23.0, TP4:

The ue8m0 → W4A16_NVFP4 switch worked. No more DeepGEMM t.dim()==N. The engine loads all 48 shards (~39 GB/GPU), auto-selects the Marlin NVFP4 MoE backend, allocates a ~4.76M-token fp8 MLA KV pool, and reaches Application startup complete.

A few things were needed to get there (sharing in case it's useful for other sm120 users or a future tweak):

--kv-cache-dtype fp8 is mandatory — the DeepseekV4 attention only ships the fp8_ds_mla MLA layout on CUDA (assert kv_cache_dtype.startswith("fp8")), regardless of kv_cache_quant_algo: null.
Attention wo_a/wo_b are quantized W4A16_NVFP4, but vLLM's DeepseekV4 o_proj (deep_gemm_fp8_o_proj, both the FlashMLA and FlashInfer paths) is block-fp8-only — it reads wo_a.weight_scale_inv, which W4A16 weights don't carry (AttributeError: ... has no attribute 'weight_scale_inv'). Might be worth considering whether the o-projection weights should stay block-fp8 (or bf16) for compatibility with the current attention kernels — as-is it needs an engine-side bf16-dequant o_proj to run.
Two attention fusions (fused_wqa_wkv←wq_a+wkv, and gate_up_proj←w1+w3) aren't in vLLM's packed_modules_mapping, so _resolve_quant_algo returns None and they build unquantized → uint8-vs-bf16 load-shape mismatch. This looks like a vLLM-side gap rather than a checkpoint issue.

Final blocker (a compiled-kernel gap, upstream of this repo): at decode, the DSA sparse indexer calls DeepGEMM get_paged_mqa_logits_metadata / fp8_fp4_paged_mqa_logits, which assert Unsupported architecture (deepgemm/csrc/apis/attention.hpp:219) on sm120. These paged-MQA-logits kernels have no Blackwell/sm120 build, so sparse-attention decode can't run yet on this GPU (and VLLM_USE_DEEP_GEMM=0 doesn't gate this path).

Net: the checkpoint itself is in great shape — it loads and serves. The wall now is DeepGEMM's lack of sm120 sparse-attn kernels. For anyone else on sm120 in the meantime, the GGUF route (unsloth's llama.cpp build) works today. Thanks again!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment