yarikdevcom/Seed-OSS-36B-Instruct-GGUF

How to build:

sudo apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first

How to run

./llama.cpp/build/bin/llama-server -hf yarikdevcom/Seed-OSS-36B-Instruct-GGUF:Q3_K_M --ctx-size 4096 --n-gpu-layers 99 --temp 1.1 --top-p 0.95 --port 8999 --host 0.0.0.0 --flash-attn --cache-type-k q8_0 --cache-type-v q8_0

All credits to this PR, I just applied changes from one of the comments. Based on this PR https://github.com/ggml-org/llama.cpp/pull/15490