colefuoco00/gemma-4-12B-it-assistant-GGUF

Gemma 4 12B IT Assistant - GGUF Q8_0

This repository contains a GGUF Q8_0 conversion of google/gemma-4-12B-it-assistant.

This GGUF is intended to be used as an MTP / speculative draft model with a compatible Gemma 4 main model. At the time of upload, this requires the gemma4-mtp llama.cpp branch:

https://github.com/am17an/llama.cpp/tree/gemma4-mtp

Usage

This model is not intended to be run as a standalone chat model. It is intended to be loaded as the draft model with -md.

Required llama.cpp branch:

https://github.com/am17an/llama.cpp/tree/gemma4-mtp

Example llama-server command:

llama-server.exe -m models\gemma-4-12B-it-Q6_K.gguf ^
    -md models\gemma-4-12B-it-assistant-Q8_0.gguf ^
    --spec-type draft-mtp ^
    --spec-draft-n-max 2 ^
    --draft-p-min 0.0 ^
    -ngl 99 ^
    -c 262144 ^
    -np 1 ^
    -fa on ^
    --reasoning off ^
    -ctk q8_0 ^
    -ctv q8_0 ^
    --ctx-checkpoints 128 ^
    --temp 0 ^
    --top-p 0.95 ^
    --top-k 40 ^
    --presence-penalty 0.0 ^
    --min-p 0.00

Benchmark

Benchmark was run with llama-server.exe from the gemma4-mtp llama.cpp branch.

Setting	Value
Main model	gemma-4-12B-it-Q6_K.gguf
Draft model	gemma-4-12B-it-assistant-Q8_0.gguf
Required branch	am17an/llama.cpp gemma4-mtp
Runs	3 prompts x 5 measured repeats per mode, with 1 warmup per prompt
Generation length	256 tokens
Context	262144
GPU layers	99
Flash attention	on
KV cache	Q8_0 / Q8_0
Temperature	0

Generation throughput

Mode	Short gen tok/s	Medium gen tok/s	Long gen tok/s	Mean gen tok/s	Mean speedup vs baseline
Baseline	52.08	51.94	51.10	51.71	-
MTP draft n=1	57.79	56.27	58.21	57.42	+11.0%
MTP draft n=2	57.83	59.45	61.54	59.61	+15.3%
MTP draft n=3	52.55	57.32	65.28	58.38	+12.9%

In this benchmark, --spec-draft-n-max 2 produced the best average generation throughput across the three prompt types. --spec-draft-n-max 3 was fastest on the long prompt but weaker on the short prompt.

Prompt-processing throughput

Mode	Short prompt tok/s	Medium prompt tok/s	Long prompt tok/s
Baseline	1194.85	2041.71	2840.91
MTP draft n=1	1097.50	1943.36	2545.60
MTP draft n=2	1133.98	1947.88	2542.42
MTP draft n=3	1102.01	1965.93	2572.39

Long-prompt prompt-processing values use the first full-ingest/warmup run. Repeated long-prompt runs appeared to reuse cached/prefix state and reported partial prompt-eval timings, so those rows were excluded from the prompt-processing table.

Limitations

These numbers are local throughput measurements from one hardware/software setup. They are not model quality benchmarks and may not generalize to other GPUs, drivers, llama.cpp commits, CUDA versions, context lengths, quantization settings, or sampling settings.

This draft model should be used with a compatible Gemma 4 main model and a llama.cpp build that supports draft-mtp.

Conversion notes

Converted from Hugging Face safetensors to GGUF using the am17an/llama.cpp gemma4-mtp branch.

The upstream config.json used Gemma4UnifiedAssistantForCausalLM / gemma4_unified_assistant. For compatibility with the converter branch, the local config was patched to:

{
  "architectures": ["Gemma4AssistantForCausalLM"],
  "model_type": "gemma4_assistant"
}

The tokenizer config was also patched by replacing list-valued extra_special_tokens with {} to avoid a local Transformers tokenizer loading error during GGUF export.

Source model

Original model: google/gemma-4-12B-it-assistant