Instructions to use ibm-granite/granite-docling-258M with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ibm-granite/granite-docling-258M with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="ibm-granite/granite-docling-258M")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("ibm-granite/granite-docling-258M")
model = AutoModelForMultimodalLM.from_pretrained("ibm-granite/granite-docling-258M", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ibm-granite/granite-docling-258M with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ibm-granite/granite-docling-258M"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-docling-258M",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/ibm-granite/granite-docling-258M

SGLang

How to use ibm-granite/granite-docling-258M with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ibm-granite/granite-docling-258M" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-docling-258M",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ibm-granite/granite-docling-258M" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-docling-258M",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use ibm-granite/granite-docling-258M with Docker Model Runner:
```
docker model run hf.co/ibm-granite/granite-docling-258M
```

Request for ONNX version

#21

by harisnaeem - opened Sep 21, 2025

Discussion

harisnaeem

Sep 21, 2025

I have tried to convert this model to ONNX using the current Hugging Face ONNX conversion workflow (via transformers.js/scripts/convert.py and optimum.exporters.onnx). Unfortunately, the conversion fails because this model uses the idefics3 architecture, which is currently not supported by optimum.exporters.onnx.

Could the maintainers or the model authors provide an official ONNX version? Having an ONNX version would greatly help with deployment in environments that require ONNX Runtime and improve inference efficiency.

ldenoue

Sep 24, 2025

It would be nice yes, I second that idea. We could run the model in a browser environment for example.

gabegoodhart

IBM Granite org Sep 24, 2025

I have some initial work on this up in: https://github.com/gabe-l-hart/optimum-onnx. This is entirely done using our internal developer assistant (see the commit message for the full prompt). I haven't done anything to validate either the code changes or the output model, but the auto-evaluated maximum delta is fairly low, so I think it should be somewhat close. I'm not at all familiar with how VLMs are handled in ONNX runtime environments, in particular the preprocessing portions. For llama.cpp, there are some significant preprocessing changes that are needed before the model will perform well, so it very well may be the case that the same is needed here.

glamberson

Sep 28, 2025

Hi, I have generated an ONNX format of this model here: https://huggingface.co/lamco-development/granite-docling-258M-onnx using @gabegoodhart 's provided work. Check it out!

Xenova

Oct 4, 2025

Posting here for others to take a look at: https://huggingface.co/onnx-community/granite-docling-258M-ONNX
:)

harisnaeem

Oct 4, 2025

Posting here for others to take a look at: https://huggingface.co/onnx-community/granite-docling-258M-ONNX
:)

Shalom Joshua 🖖🏽, thanks for the onnx version 😊

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment