README.md

Usage: Serving with SGLang

Download the weights. You can replace /local/grok-2 with any other folder name you prefer.

hf download xai-org/grok-2 --local-dir /local/grok-2

Launch a server.

python3 -m sglang.launch_server --model /local/grok-2 --tokenizer-path /local/grok-2/tokenizer.tok.json --tp 8 --quantization fp8 --attention-backend triton

Send a request.

python3 -m sglang.test.send_one --prompt "Human: What is your name? <|separator|>\n\nAssistant:"