add vllm source code part.
#19
by
asherszhang
- opened
README.md
CHANGED
|
@@ -266,7 +266,7 @@ docker run --rm --ipc=host \
|
|
| 266 |
--gpus=all \
|
| 267 |
-it \
|
| 268 |
-e VLLM_USE_V1=0 \
|
| 269 |
-
--entrypoint python
|
| 270 |
-m vllm.entrypoints.openai.api_server \
|
| 271 |
--host 0.0.0.0 \
|
| 272 |
--tensor-parallel-size 4 \
|
|
@@ -275,6 +275,12 @@ docker run --rm --ipc=host \
|
|
| 275 |
--trust_remote_code
|
| 276 |
```
|
| 277 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 278 |
|
| 279 |
#### Tool Calling with vLLM
|
| 280 |
|
|
@@ -296,6 +302,8 @@ These settings enable vLLM to correctly interpret and route tool calls generated
|
|
| 296 |
|
| 297 |
vLLM reasoning parser support on Hunyuan A13B model is under development.
|
| 298 |
|
|
|
|
|
|
|
| 299 |
### SGLang
|
| 300 |
|
| 301 |
#### Docker Image
|
|
|
|
| 266 |
--gpus=all \
|
| 267 |
-it \
|
| 268 |
-e VLLM_USE_V1=0 \
|
| 269 |
+
--entrypoint python hunyuaninfer/hunyuan-a13b:hunyuan-moe-A13B-vllm \
|
| 270 |
-m vllm.entrypoints.openai.api_server \
|
| 271 |
--host 0.0.0.0 \
|
| 272 |
--tensor-parallel-size 4 \
|
|
|
|
| 275 |
--trust_remote_code
|
| 276 |
```
|
| 277 |
|
| 278 |
+
### Source Code
|
| 279 |
+
Support for this model has been added via this [PR 20114](https://github.com/vllm-project/vllm/pull/20114 ) in the vLLM project.
|
| 280 |
+
|
| 281 |
+
You can build and run vLLM from source after merging this pull request into your local repository.
|
| 282 |
+
|
| 283 |
+
|
| 284 |
|
| 285 |
#### Tool Calling with vLLM
|
| 286 |
|
|
|
|
| 302 |
|
| 303 |
vLLM reasoning parser support on Hunyuan A13B model is under development.
|
| 304 |
|
| 305 |
+
|
| 306 |
+
|
| 307 |
### SGLang
|
| 308 |
|
| 309 |
#### Docker Image
|