Instructions to use lmms-lab/llava-onevision-qwen2-7b-si with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use lmms-lab/llava-onevision-qwen2-7b-si with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="lmms-lab/llava-onevision-qwen2-7b-si")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForCausalLM

processor = AutoProcessor.from_pretrained("lmms-lab/llava-onevision-qwen2-7b-si")
model = AutoModelForCausalLM.from_pretrained("lmms-lab/llava-onevision-qwen2-7b-si", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use lmms-lab/llava-onevision-qwen2-7b-si with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "lmms-lab/llava-onevision-qwen2-7b-si"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lmms-lab/llava-onevision-qwen2-7b-si",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/lmms-lab/llava-onevision-qwen2-7b-si

SGLang

How to use lmms-lab/llava-onevision-qwen2-7b-si with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "lmms-lab/llava-onevision-qwen2-7b-si" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lmms-lab/llava-onevision-qwen2-7b-si",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "lmms-lab/llava-onevision-qwen2-7b-si" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lmms-lab/llava-onevision-qwen2-7b-si",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use lmms-lab/llava-onevision-qwen2-7b-si with Docker Model Runner:
```
docker model run hf.co/lmms-lab/llava-onevision-qwen2-7b-si
```

LLaVA-OneVision

Play with the model on the LLaVA OneVision Chat.

Model Summary
Use
Limitations
Training
License
Citation

Model Summary

The LLaVA-OneVision models are 0.5/7/72B parameter models trained on LLaVA-OneVision, based on Qwen2 language model with a context window of 32K tokens.

Repository: LLaVA-VL/LLaVA-NeXT
Project Website: llava-onevision.lmms-lab.com
Paper: LLaVA-OneVision
Point of Contact: Bo Li
Languages: English, Chinese

Use

Intended use

The model was trained on LLaVA-OneVision Dataset and have the ability to interact with images, multi-image and videos.

Feel free to share your generations in the Community tab!

Generation

We provide the simple generation process for using our model. For more details, you could refer to Github.

# pip install git+https://github.com/LLaVA-VL/LLaVA-NeXT.git
from llava.model.builder import load_pretrained_model
from llava.mm_utils import get_model_name_from_path, process_images, tokenizer_image_token
from llava.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN, DEFAULT_IM_START_TOKEN, DEFAULT_IM_END_TOKEN, IGNORE_INDEX
from llava.conversation import conv_templates, SeparatorStyle

from PIL import Image
import requests
import copy
import torch

import sys
import warnings

warnings.filterwarnings("ignore")
pretrained = "lmms-lab/llava-onevision-qwen2-7b-si"
model_name = "llava_qwen"
device = "cuda"
device_map = "auto"
tokenizer, model, image_processor, max_length = load_pretrained_model(pretrained, None, model_name, device_map=device_map)  # Add any other thing you want to pass in llava_model_args

model.eval()

url = "https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true"
image = Image.open(requests.get(url, stream=True).raw)
image_tensor = process_images([image], image_processor, model.config)
image_tensor = [_image.to(dtype=torch.float16, device=device) for _image in image_tensor]

conv_template = "qwen_1_5"  # Make sure you use correct chat template for different models
question = DEFAULT_IMAGE_TOKEN + "\nWhat is shown in this image?"
conv = copy.deepcopy(conv_templates[conv_template])
conv.append_message(conv.roles[0], question)
conv.append_message(conv.roles[1], None)
prompt_question = conv.get_prompt()

input_ids = tokenizer_image_token(prompt_question, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0).to(device)
image_sizes = [image.size]


cont = model.generate(
    input_ids,
    images=image_tensor,
    image_sizes=image_sizes,
    do_sample=False,
    temperature=0,
    max_new_tokens=4096,
)
text_outputs = tokenizer.batch_decode(cont, skip_special_tokens=True)
print(text_outputs)

Training

Model

Architecture: SO400M + Qwen2
Pretraining Stage: LCS-558K, 1 epoch, projector
Mid Stage: A mixture of 4.7M high-quality synthetic data, 1 epoch, full model
Final-Image Stage: A mixture of 3.6M single-image data, 1 epoch, full model
OneVision Stage: A mixture of 1.6M single-image/multi-image/video data, 1 epoch, full model
Precision: bfloat16

Hardware & Software

GPUs: 256 * Nvidia Tesla A100 (for whole model series training)
Orchestration: Huggingface Trainer
Neural networks: PyTorch

Citation

@article{li2024llavaonevision,
      title={LLaVA-OneVision}, 
}

Downloads last month: 231

Safetensors

Model size

8B params

Tensor type

BF16

Model tree for lmms-lab/llava-onevision-qwen2-7b-si

Finetunes

4 models

Dataset used to train lmms-lab/llava-onevision-qwen2-7b-si

Collection including lmms-lab/llava-onevision-qwen2-7b-si

LLaVA-OneVision

Collection

a model good at arbitrary types of visual input • 17 items • Updated Sep 17, 2025 • 32

Paper for lmms-lab/llava-onevision-qwen2-7b-si

LLaVA-OneVision: Easy Visual Task Transfer

Paper • 2408.03326 • Published Aug 6, 2024 • 61

Evaluation results

accuracy on AI2D
self-reported

81.600
accuracy on ChartQA
self-reported

78.800
accuracy on DocVQA
self-reported

89.300
accuracy on InfoVQA
self-reported

69.900
accuracy on MathVerse
self-reported

26.900
accuracy on MathVista
self-reported

56.100
accuracy on MMBench
self-reported

81.700
score on MME-Perception
self-reported

1626.000

lmms-lab
/

llava-onevision-qwen2-7b-si

LLaVA-OneVision

Table of Contents

Model Summary

Use

Intended use

Generation

Training

Model

Hardware & Software

Citation

Model tree for lmms-lab/llava-onevision-qwen2-7b-si

Dataset used to train lmms-lab/llava-onevision-qwen2-7b-si

Collection including lmms-lab/llava-onevision-qwen2-7b-si

LLaVA-OneVision

Paper for lmms-lab/llava-onevision-qwen2-7b-si

LLaVA-OneVision: Easy Visual Task Transfer

Evaluation results