Instructions to use z-lab/Qwen3.8-27B-DFlash2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use z-lab/Qwen3.8-27B-DFlash2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="z-lab/Qwen3.8-27B-DFlash2")

# Load model directly
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("z-lab/Qwen3.8-27B-DFlash2")
model = AutoModel.from_pretrained("z-lab/Qwen3.8-27B-DFlash2", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use z-lab/Qwen3.8-27B-DFlash2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "z-lab/Qwen3.8-27B-DFlash2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "z-lab/Qwen3.8-27B-DFlash2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/z-lab/Qwen3.8-27B-DFlash2

SGLang

How to use z-lab/Qwen3.8-27B-DFlash2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "z-lab/Qwen3.8-27B-DFlash2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "z-lab/Qwen3.8-27B-DFlash2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "z-lab/Qwen3.8-27B-DFlash2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "z-lab/Qwen3.8-27B-DFlash2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use z-lab/Qwen3.8-27B-DFlash2 with Docker Model Runner:
```
docker model run hf.co/z-lab/Qwen3.8-27B-DFlash2
```

Made Sglang + Dflash2 + Docker

by AIReach - opened 11 days ago

Discussion

AIReach

11 days ago

Dflash2 has not yet installed in the latest Sglang official docker images. I made this repo to build and server Dflash2 embedded Sglang containers.
https://github.com/xycjscs/sglang-dflash2

On my dual 5090s machine, following speeds are achieved.

Performance (2× RTX 5090, TP=2)

Measured from production logs of this image and start.sh flags. Prefill for large requests uses whole-request wall time.

Prefill

	tok/s
Large requests (≥2k tokens, n=93), mean	4391
Median	3736
p10 / p90	2257 / 7331
Token-weighted	4583
Longest request (108k tokens)	3731 (29 s)
Short turns (<2k tokens, median)	~470 (includes scheduling gaps)

The first chunk of each large request is slowed by Triton JIT (one logged chunk at 25.6 tok/s). Later chunks return to 4k–5k tok/s.

Decode (DFlash2)


Mean / median / p90 / peak	172 / 156 / 280 / 542 tok/s
Mean accept length	3.5 tokens / step (8 draft tokens)
Accept rate	~0.40 (p10 0.20 / p90 0.59)

Throughput barely drops with context length (hybrid GDN linear attention):

Context	tok/s
0–20k	182
40–60k	162
120–140k	201
140–160k	151

Samples at 100–150 tok/s line up with accept rate 0.15–0.25 (draft misses, close to raw decode).

AIReach changed discussion status to closed 11 days ago

AIReach changed discussion status to open 11 days ago

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment