File size: 491 Bytes
04deaba |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
## Usage: Serving with SGLang
Download the weights. You can replace `/local/grok-2` with any other folder name you prefer.
```
hf download xai-org/grok-2 --local-dir /local/grok-2
```
Launch a server.
```
python3 -m sglang.launch_server --model /local/grok-2 --tokenizer-path /local/grok-2/tokenizer.tok.json --tp 8 --quantization fp8 --attention-backend triton
```
Send a request.
```
python3 -m sglang.test.send_one --prompt "Human: What is your name? <|separator|>\n\nAssistant:"
``` |