Update README.md (#31)
Browse files- Update README.md (c3e22d3f9f15649c42fb3d61ed29e609d85d0339)
README.md
CHANGED
@@ -126,7 +126,11 @@ We recommend that you use Mistral-Small-3.2-24B-Instruct-2506 in a server/client
|
|
126 |
1. Spin up a server:
|
127 |
|
128 |
```
|
129 |
-
vllm serve mistralai/Mistral-Small-3.2-24B-Instruct-2506
|
|
|
|
|
|
|
|
|
130 |
```
|
131 |
|
132 |
**Note:** Running Mistral-Small-3.2-24B-Instruct-2506 on GPU requires ~55 GB of GPU RAM in bf16 or fp16.
|
|
|
126 |
1. Spin up a server:
|
127 |
|
128 |
```
|
129 |
+
vllm serve mistralai/Mistral-Small-3.2-24B-Instruct-2506 \
|
130 |
+
--tokenizer_mode mistral --config_format mistral \
|
131 |
+
--load_format mistral --tool-call-parser mistral \
|
132 |
+
--enable-auto-tool-choice --limit-mm-per-prompt '{"image":10}' \
|
133 |
+
--tensor-parallel-size 2
|
134 |
```
|
135 |
|
136 |
**Note:** Running Mistral-Small-3.2-24B-Instruct-2506 on GPU requires ~55 GB of GPU RAM in bf16 or fp16.
|