SmolVLM2-500M-Video-Instruct-vqav2

This model is a fine-tuned version of HuggingFaceTB/SmolVLM2-500M-Video-Instruct on the jinaai/table-vqa dataset.

Model description

This model is a SmolVLM2-500M-Video-Instruct model fine-tuned for Visual Question Answering on table images using the jinaai/table-vqa dataset. It was fine-tuned using QLoRA for efficient training on consumer GPUs.

Intended uses & limitations

This model is intended for Visual Question Answering tasks specifically on images containing tables. It can be used to answer questions about the content of tables within images.

Limitations:

• Performance may vary on different types of images or questions outside of the table VQA domain.
• The model was fine-tuned on a small subset of the dataset for demonstration purposes.
• The model's performance is dependent on the quality and nature of the jinaai/table-vqa dataset.

Training and evaluation data

The model was trained on a subset of the jinaai/table-vqa dataset. The training dataset size is 800 examples, and the test dataset size is 200 examples.

Training procedure

The model was fine-tuned using the QLoRA method with the following configuration:

• r=8
• lora_alpha=8
• lora_dropout=0.1
• target_modules=['down_proj','o_proj','k_proj','q_proj','gate_proj','up_proj','v_proj']
• use_dora=False
• init_lora_weights="gaussian"
• 4-bit quantization (bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16)

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 50
• num_epochs: 1

Direct Use

import torch
from peft import PeftModel, PeftConfig
from transformers import AutoProcessor, Idefics3ForConditionalGeneration, BitsAndBytesConfig
from PIL import Image
import requests

# Define the base model and the fine-tuned adapter repository
base_model_id = "HuggingFaceTB/SmolVLM2-500M-Video-Instruct"
adapter_model_id = "Susant-Achary/SmolVLM2-500M-Video-Instruct-vqav2"

# Load the processor from the base model
processor = AutoProcessor.from_pretrained(base_model_id)

# Load the base model with quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

model = Idefics3ForConditionalGeneration.from_pretrained(
    base_model_id,
    quantization_config=bnb_config,
    device_map="auto"
)

# Load the adapter and add it to the base model
model = PeftModel.from_pretrained(model, adapter_model_id)

# Prepare an example image and question
# You can replace this with your own image and question
url = "/content/VQA-20-standard-test-set-results-comparison-of-state-of-the-art-methods.png"
image = Image.open(url)
question = "What is in the image?"

# Prepare the input for the model
messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Answer briefly."},
            {"type": "image"},
            {"type": "text", "text": question}
        ]
    },
    {
        "role": "assistant",
        "content": [
            {"type": "text", "text": None}
        ]
    }
]

prompt = processor.apply_chat_template(messages, add_generation_prompt=False)
inputs = processor(text=[prompt], images=[image], return_tensors="pt").to(model.device) # Move inputs to model device

# Generate a response
generated_ids = model.generate(**inputs, max_new_tokens=100)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]

# Print the generated response
print(generated_text)

Framework versions

• PEFT 0.16.0
• Transformers 4.53.2
• Pytorch 2.7.1+cu126
• Datasets 4.0.0
• Tokenizers 0.21.2
• bitsandbytes 0.46.1
• num2words