Instructions to use Aayush672/Finetuned-llama3.2-Vision-Model with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Aayush672/Finetuned-llama3.2-Vision-Model with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Aayush672/Finetuned-llama3.2-Vision-Model")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("Aayush672/Finetuned-llama3.2-Vision-Model", dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use Aayush672/Finetuned-llama3.2-Vision-Model with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Aayush672/Finetuned-llama3.2-Vision-Model"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Aayush672/Finetuned-llama3.2-Vision-Model",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Aayush672/Finetuned-llama3.2-Vision-Model

SGLang

How to use Aayush672/Finetuned-llama3.2-Vision-Model with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Aayush672/Finetuned-llama3.2-Vision-Model" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Aayush672/Finetuned-llama3.2-Vision-Model",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Aayush672/Finetuned-llama3.2-Vision-Model" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Aayush672/Finetuned-llama3.2-Vision-Model",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Aayush672/Finetuned-llama3.2-Vision-Model with Docker Model Runner:
```
docker model run hf.co/Aayush672/Finetuned-llama3.2-Vision-Model
```

Finetuned Llama 3.2 Vision for Product Description Generation

A fine-tuned version of Meta's Llama-3.2-11B-Vision-Instruct model specialized for generating SEO-optimized product descriptions from product images, names, and categories.

Model Details

Model Description

This model generates concise, SEO-optimized product descriptions for e-commerce applications. Given a product image, name, and category, it produces mobile-friendly descriptions suitable for online marketplaces and product catalogs.

Developed by: Aayush672
Model type: Vision-Language Model (Multimodal)
Language(s): English
License: MIT
Finetuned from model: meta-llama/Llama-3.2-11B-Vision-Instruct

Model Sources

Repository: Aayush672/Finetuned-llama3.2-Vision-Model
Base Model: meta-llama/Llama-3.2-11B-Vision-Instruct

Uses

Direct Use

The model is designed for generating product descriptions in e-commerce scenarios:

Product catalog automation
SEO-optimized content generation
Mobile-friendly product descriptions
Marketplace listing optimization

Example Usage

from transformers import AutoModelForVision2Seq, AutoProcessor
from PIL import Image

model = AutoModelForVision2Seq.from_pretrained("Aayush672/Finetuned-llama3.2-Vision-Model")
processor = AutoProcessor.from_pretrained("Aayush672/Finetuned-llama3.2-Vision-Model")

# Prepare your inputs
image = Image.open("product_image.jpg")
product_name = "Wireless Bluetooth Headphones"
category = "Electronics | Audio | Headphones"

prompt = f"""Create a Short Product description based on the provided ##PRODUCT NAME## and ##CATEGORY## and image.
Only return description. The description should be SEO optimized and for a better mobile search experience.

##PRODUCT NAME##: {product_name}
##CATEGORY##: {category}"""

messages = [{
    "role": "user",
    "content": [
        {"type": "text", "text": prompt},
        {"type": "image", "image": image}
    ]
}]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=text, images=[image], return_tensors="pt")

output = model.generate(**inputs, max_new_tokens=200, temperature=0.7)
description = processor.tokenizer.decode(output[0], skip_special_tokens=True)

Out-of-Scope Use

General conversation or chat applications
Complex reasoning tasks
Non-commercial product descriptions
Content outside e-commerce domain

Training Details

Training Data

The model was fine-tuned on the philschmid/amazon-product-descriptions-vlm dataset, which contains Amazon product images with corresponding names, categories, and descriptions.

Training Procedure

Fine-tuning Method

Technique: LoRA (Low-Rank Adaptation) with PEFT
Target modules: q_proj, v_proj
LoRA rank (r): 8
LoRA alpha: 16
LoRA dropout: 0.05

Training Hyperparameters

Training regime: bf16 mixed precision with 4-bit quantization (QLoRA)
Number of epochs: 1
Batch size: 8 per device
Gradient accumulation steps: 4
Learning rate: 2e-4
Optimizer: AdamW (torch fused)
LR scheduler: Constant
Warmup ratio: 0.03
Max gradient norm: 0.3
Quantization: 4-bit with double quantization (nf4)

Hardware & Software

Quantization: BitsAndBytesConfig with 4-bit precision
Gradient checkpointing: Enabled
Memory optimization: QLoRA technique
Framework: Transformers, TRL, PEFT

Bias, Risks, and Limitations

Limitations

Trained specifically on Amazon product data, may not generalize well to other e-commerce platforms
Limited to English language descriptions
Optimized for mobile/SEO format, may not suit all description styles
Performance depends on image quality and product visibility

Recommendations

Test thoroughly on your specific product categories before production use
Consider additional fine-tuning for domain-specific products
Implement content moderation for generated descriptions
Validate SEO effectiveness for your target keywords

Environmental Impact

Training utilized quantized models (4-bit) to reduce computational requirements and carbon footprint compared to full-precision training.

Technical Specifications

Model Architecture

Base Architecture: Llama 3.2 Vision (11B parameters)
Vision Encoder: Integrated multimodal architecture
Fine-tuning: LoRA adapters (trainable parameters: ~16M)
Quantization: 4-bit with double quantization

Compute Infrastructure

Training: Optimized with gradient checkpointing and mixed precision
Memory: Reduced via 4-bit quantization and LoRA
Inference: Supports both quantized and full precision modes

Citation

@misc{finetuned-llama32-vision-product,
  title={Fine-tuned Llama 3.2 Vision for Product Description Generation},
  author={Aayush672},
  year={2025},
  howpublished={\url{https://huggingface.co/Aayush672/Finetuned-llama3.2-Vision-Model}}
}

Model Card Contact

For questions or issues, please open an issue in the model repository or contact the model author.

Downloads last month: -; Downloads are not tracked for this model. How to track

Model tree for Aayush672/Finetuned-llama3.2-Vision-Model

Base model

meta-llama/Llama-3.2-11B-Vision-Instruct

Adapter

(351)

this model

Aayush672
/

Finetuned-llama3.2-Vision-Model