Qwen3-VL-Embedding-8B-W8A16 (Quadux) — multimodal

This is a W8A16 quantization of Qwen/Qwen3-VL-Embedding-8B, produced by Quadux IT GmbH with llm-compressor for direct serving via vLLM.

⚠️ Hardware: NVIDIA GPU (CUDA) only

This W8A16 quant relies on the Marlin kernel family, which is implemented for CUDA only. vLLM's CPU backend cannot load this model — it accepts only uint4 (W4A16) or compressed-tensors INT8 W8A8, not the uint8b128 format W8A16 produces. Tested with vLLM v0.20.0 — see issue #33970 for the kernel-level discussion. For CPU deployment, run the original BF16 source model directly (no quantization needed — Strix Halo / recent Intel with AVX-512+BF16 perform well).

Overview

Property	Value
Model size on disk	14 GB (vs. ~16 GB BF16 original)
Format	compressed-tensors (W8A16)
Native embedding dimension	4096
Matryoshka dimensions	[128, 256, 384, 512, 768, 1024, 1536, 2048, 3072, 4096]
Max sequence length	32768 tokens
Languages	100+ (multilingual)
Target hardware	CUDA GPUs. Not for CPU. Multimodal: text, images, video.

Why use this quant

• 50% smaller on disk than the BF16 source (9 GB vs ~17 GB for 8B)
• Negligible quality loss for retrieval: cosine ≥ 0.998 vs BF16 source on 20 realistic German texts (mails, invoices, engineering data), Top-1-nearest-neighbour agreement 20/20.
• Production-ready on any CUDA GPU with compressed-tensors support (vLLM, SGLang). Not for vLLM CPU — for CPU deployment use the original BF16 source model directly (Strix Halo / Zen5 with AVX-512+BF16 runs at ~17 emb/s for 8B).

Quick start with vLLM

docker run --rm -p 1236:1236 \
    --gpus all \
    -v $(pwd)/models:/models:ro \
    vllm/vllm-openai:latest \
    --model QuaduxIT/Qwen3-VL-Embedding-8B-W8A16 \
    --served-model-name qwen3-embedding \
    --port 1236 --host 0.0.0.0 \
    --trust-remote-code \
    --runner pooling \
    --convert embed \
    --hf-overrides='{"is_matryoshka":true,"matryoshka_dimensions":[128,256,384,512,768,1024,1536,2048,3072,4096]}'

Then call the OpenAI-compatible endpoint:

curl -s -X POST http://localhost:1236/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3-embedding","input":"Sample text","dimensions":1024}'

The response is OpenAI-conformant; the server slices the native 4096-dim vector to your requested dimensions and re-normalises to L2 = 1.

Quantization details

• Algorithm: Round-to-Nearest with llm-compressor QuantizationModifier
• Scheme: W8A16
• Group size: 128 (per-channel within groups)
• Calibration: 128 samples from HuggingFaceH4/ultrachat_200k, max_seq_length=2048
• Layers excluded from quantization: lm_head, embed_tokens
  • vision_tower, visual, patch_embed, merger, video_model (the entire vision side stays in FP16 — its activation distribution is too different from the language model's for AWQ scales to work well)
• Output format: compressed-tensors (vLLM-native, no special flags needed)
• Tooling version: llm-compressor>=0.4.0,<0.7
• Recipe file: recipe.yaml is shipped with the model and documents the exact quantization configuration.

Quality vs. BF16 source

Verified on the Quadux internal multimodal fixture (8 German texts + 5 Wikimedia Commons images covering animals, landmarks, historical documents, and food):

Text Embeddings

Metric	Value
Cosine similarity vs BF16 source (median)	0.9987
Cosine similarity worst case	0.9986
Top-1 nearest-neighbour agreement	8/8 (100%)

Multimodal (Text + Image) Embeddings

Metric	Value
Cosine similarity vs BF16 source (median)	0.975
Cosine similarity worst case	0.967

Text embeddings are functionally indistinguishable from BF16 for retrieval. Image embeddings show slightly larger drift (2-3% cosine) because the quantized text-decoder processes the larger vision-token sequences (500-1500 tokens per image vs ~30 for typical text), so quantization errors accumulate more over the longer sequence. The vision tower itself is kept in FP16 — see "Quantization details" below — so its outputs are bit-identical to the BF16 reference; the drift comes purely from the text-decoder consuming those vision tokens.

For Top-K image retrieval and multimodal RAG the W8A16 vector is production-ready. For strict cosine-threshold filters (e.g. ">0.99 match required") the BF16 source is the better choice.

Important: vLLM v0.20.0 contains a known bug in the Qwen3-VL deepstack handling that causes every multimodal embedding request to crash the engine (PR #40932 was merged ~12 hours after the v0.20.0 tag was cut). Until vLLM v0.20.1 ships, run with the patched image we use internally:

# 12-line patch removing the broken boundary check.
# See https://github.com/vllm-project/vllm/pull/40932 for details.
docker pull vllm/vllm-openai:v0.20.0
# apply patch in /usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_vl.py

Text-only embeddings work without the patch.

Hardware compatibility

Backend	Status
vLLM CUDA (Marlin / compressed-tensors)	✅ verified
vLLM CPU	❌ not supported (W8A16 / uint8b128 not in CPU kernels)
TensorRT-LLM	likely (untested by us)
SGLang	likely (compressed-tensors support)

For CPU deployment, use the original BF16 source model directly via vllm-openai-cpu — Qwen3 inference on AVX-512+BF16 hardware (Zen5 Strix Halo, recent Intel) runs at usable throughput (~17 emb/s for 8B in our tests).

Files

Qwen3-VL-Embedding-8B-W8A16/
├── config.json
├── tokenizer.json
├── tokenizer_config.json
├── special_tokens_map.json
├── chat_template.jinja
├── recipe.yaml             # quantization recipe (reproducibility)
├── model.safetensors.index.json
├── model-00001-of-00002.safetensors
└── model-00002-of-00002.safetensors

License

This model and its base model are licensed under the Apache License 2.0. The Apache 2.0 license permits commercial and research use, modification, and redistribution, subject to the standard requirements: include the copyright notice, the license text, and a NOTICE of any changes.

• Base model license: Apache 2.0 — see the Qwen3-Embedding model card for the original license text.
• This quantization: Apache 2.0 (same terms as the base model).
• Modifications by Quadux IT GmbH: weight-only post-training quantization to the W8A16 scheme. No retraining, no fine-tuning, no architectural changes.

If you redistribute this model, you must include the Apache 2.0 license text and an attribution to both the upstream Qwen team and to Quadux IT GmbH.

Citation

The original Qwen3 Embedding work — please cite this if you publish results using this model:

@article{qwen3vlembedding,
  title  = {Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking},
  author = {Li, Mingxin and Zhang, Yanzhao and Long, Dingkun and Chen, Keqin and Song, Sibo and Bai, Shuai and Yang, Zhibo and Xie, Pengjun and Yang, An and Liu, Dayiheng and Zhou, Jingren and Lin, Junyang},
  journal = {arXiv preprint arXiv:2601.04720},
  year   = {2026}
}

If you want to cite the quantization specifically:

@misc{{quadux_Qwen3-VL-Embedding-8B-W8A16},
  author       = {{Quadux IT GmbH}},
  title        = {Qwen3-VL-Embedding-8B-W8A16 (Quadux) — multimodal},
  year         = {{2026}},
  publisher    = {{Hugging Face}},
  howpublished = {{\\url{{https://huggingface.co/QuaduxIT/Qwen3-VL-Embedding-8B-W8A16}}}},
  note         = {{Post-training W8A16 quantization of Qwen/Qwen3-VL-Embedding-8B}}
}

---

About Quadux IT GmbH

Software for engineering offices and accounting pipelines. Custom RAG infrastructure for internal Quadux deployments — these quants are the text-embedding side of that stack, released to the community as infrastructure we'd otherwise pay vendors for.

Find more at quadux.it · contact info@quadux.it

Imprint

Quadux IT GmbH · Schulstr. 3 · 37139 Adelebsen · Germany Registered Göttingen, HRB 206773 · VAT ID DE353975332 · DUNS 344198559 Managing Director: Walter Hoffmann