File size: 1,006 Bytes
d6f1700 51f432b d513d71 4c089e3 d513d71 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
---
base_model:
- Qwen/Qwen2.5-VL-7B-Instruct
---
This is an example model demonstrating how to run the AutoRound format for a visual language model on vLLM. Some visual modules have been quantized to 8-bit precision.
this pr https://github.com/vllm-project/vllm/pull/21802 is required.
~~~bash
vllm serve Intel/Qwen2.5-VL-7B-Instruct-int4-mixed-AutoRound --dtype bfloat16 --max-model-len 10000
~~~
~~~bash
curl --noproxy '*' http://localhost:8001/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "Intel/Qwen2.5-VL-7B-Instruct-int4-mixed-AutoRound",
"messages": [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"
}
},
{
"type": "text",
"text": "请描述这张图"
}
]
}
],
"max_tokens": 512
}'
~~~ |