Instructions to use amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2")
model = AutoModelForMultimodalLM.from_pretrained("amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2

SGLang

How to use amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2 with Docker Model Runner:
```
docker model run hf.co/amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2
```

Model Overview

Model Architecture: Qwen3_5MoeForConditionalGeneration
- Input: Text, Image, Video
- Output: Text
Supported Hardware Microarchitecture: AMD MI350 / MI355
ROCm: 7.2.0
PyTorch: 2.9.1
Transformers: 5.3.0
Operating System(s): Linux
Inference Engine: SGLang
Model Optimizer: AMD-Quark (v0.12)
- Quantized layers: All MoE experts in the language model, including the shared expert (the shared expert is also fused into the MoE kernel for faster decode).
- Weight quantization:
  - self_attn and linear_attn Perchannel, FP8E4M3, Static
  - MOE OCP MXFP4, Static
- Activation quantization:
  - self_attn and linear_attn Pertoken, FP8E4M3, Dynamic
  - MOE OCP MXFP4, Dynamic

This checkpoint extends the routed-expert MXFP4 quantization by also quantizing the shared expert to MXFP4 and fusing it into the routed MoE kernel (FSE: fused shared expert). Compared with keeping the shared expert in bf16, this further reduces the bf16 footprint and improves decode throughput, with no measurable accuracy loss on GSM8K (see Evaluation).

Model Quantization

The model was quantized from Qwen/Qwen3.5-397B-A17B using AMD-Quark. Weights and activations are quantized to OCP MXFP4.

Quantization scripts:

MODEL_PATH="Qwen/Qwen3.5-397B-A17B"
OUTPUT_DIR="amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2"

cd /path/to/Quark

python3 examples/torch/language_modeling/llm_ptq/quantize_quark.py \
  --model_dir "${MODEL_PATH}" \
  --quant_scheme mxfp4 \
  --layer_quant_scheme '*linear_attn*' ptpc_fp8 \
  --layer_quant_scheme '*self_attn*' ptpc_fp8 \
  --no_trust_remote_code \
  --model_export hf_format \
  --skip_evaluation \
  --multi_gpu balanced \
  --exclude_layers "lm_head" "*mlp.gate" "mtp.*" "model.visual.*" "*shared_expert_gate*" \
  --output_dir "${OUTPUT_DIR}"

For further details or issues, please refer to the AMD-Quark documentation or contact the respective developers.

Evaluation

The model was evaluated on the GSM8K benchmark using the SGLang framework with lm-evaluation-harness, following the InferenceX SGLang evaluation recipe (local-chat-completions with the chat template applied, 5-shot, greedy). The baseline is the original Qwen/Qwen3.5-397B-A17B checkpoint, evaluated with the identical recipe on SGLang.

Accuracy

Benchmark	Qwen/Qwen3.5-397B-A17B	amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2 (this model)	Recovery
gsm8k (flexible-extract, 5-shot)	97.71	97.24	99.55%

GSM8K is essentially lossless after quantizing the shared expert to MXFP4 and fusing it into the routed MoE kernel.

Reproduction

The GSM8K results were obtained on SGLang following the InferenceX SGLang lm-eval recipe.

Serve the model with SGLang:

export SGLANG_USE_AITER=1
export SGLANG_USE_AITER_UNIFIED_ATTN=1
export AITER_FLYDSL_FORCE=1
export SGLANG_MAMBA_SSM_DTYPE=bfloat16
export SGLANG_USE_AITER_FP8_PER_TOKEN=1

python3 -m sglang.launch_server \
    --model-path amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2 \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 30000 \
    --tensor-parallel-size 2 \
    --attention-backend aiter \
    --mem-fraction-static 0.8 \
    --model-loader-extra-config '{"enable_multithread_load": true}' \
    --watchdog-timeout 1200 \
    --disable-radix-cache \
    --reasoning-parser qwen3 \
    --page-size 16 \
    --enable-aiter-allreduce-fusion

Run lm-evaluation-harness against the running server (chat-completions endpoint, chat template applied, 5-shot, greedy). This uses InferenceX's gsm8k.yaml, whose only change vs. the stock lm-eval task is a doc_to_text that asks the model to end its response with #### <number> so the reasoning model's final answer is parsed correctly:

lm_eval --model local-chat-completions --apply_chat_template \
  --tasks gsm8k.yaml \
  --num_fewshot 5 \
  --model_args "model=amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2,base_url=http://127.0.0.1:30000/v1/chat/completions,num_concurrent=64,tokenized_requests=False,max_length=16384" \
  --gen_kwargs "max_tokens=12288,temperature=0,top_p=1"

License

Downloads last month: 3,157

Safetensors

Model size

203B params

Tensor type

BF16

F8_E4M3

Model tree for amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2

Base model

Qwen/Qwen3.5-397B-A17B

Quantized

(72)

this model