keeeeenw
/

MicroLlava

@@ -14,42 +14,46 @@ base_model:
 - google/siglip-so400m-patch14-384
 ---
-# MicroLLaVA-siglip-so400m
-A compact vision language model that you can pretrain and finetune on a single consumer GPU such as NVIDIA RTX 4090 with 24G of VRAM.
-## TLDR
-| Item            | Detail |
-|-----------------|--------|
-| Framework       | Transformers + PyTorch |
-| Checkpoint type | `safetensors` |
-| LLM             | [`keeeeenw/MicroLlama`](https://huggingface.co/keeeeenw/MicroLlama) (about 300M parameters) |
-| Vision tower    | [`siglip-so400m-patch14-384`](https://huggingface.co/google/siglip-so400m-patch14-384) |
-| Hardware used   | Single NVIDIA RTX 4090 |
-| Training stack  | No DeepSpeed required |
-| Intended tasks  | Visual Question Answering, caption-style prompts |
 ---
-## Introduction
-MicroLLaVA is a [TinyLLaVA Factory](https://github.com/TinyLLaVA/TinyLLaVA_Factory) based model that pairs a very small language model [`keeeeenw/MicroLlama`](https://huggingface.co/keeeeenw/MicroLlama) with an efficient SigLIP vision encoder.
-The goal is to create a vision language model that almost anyone can train and iterate on with one consumer GPU.
 - **Language model**: [`keeeeenw/MicroLlama`](https://huggingface.co/keeeeenw/MicroLlama) with ~300M parameters
 - **Vision encoder**: [`siglip-so400m-patch14-384`](https://huggingface.co/google/siglip-so400m-patch14-384)
 - **Training codebase**: [TinyLLaVA Factory](https://github.com/TinyLLaVA/TinyLLaVA_Factory) with additional changes in my fork: [Custom fork with training tweaks](https://github.com/keeeeenw/TinyLLaVA_Factory)
-Because of its compact size, this model can be trained entirely on a single NVIDIA RTX 4090 without DeepSpeed.
-Pretraining on **LAION-CC-SBU-558K** took about **5 hours** on a single NVIDIA RTX 4090 without DeepSpeed.
-Supervised finetuning on all datasets from the TinyLLaVA Factory guide (except `ocr_vqa`) took about **12 hours** on the same GPU.
 ---
-## Quick start
 ```python
 from transformers import AutoTokenizer, AutoModelForCausalLM
@@ -58,115 +62,123 @@ hf_path = 'keeeeenw/MicroLlava-siglip-so400m-patch14-384-base-finetune'
 model = AutoModelForCausalLM.from_pretrained(hf_path, trust_remote_code=True)
 # model.cuda() # turn on cuda as needed by the model runs fairly quickly on CPU.
 config = model.config
-tokenizer = AutoTokenizer.from_pretrained(hf_path, use_fast=False, model_max_length = config.tokenizer_model_max_length,padding_side = config.tokenizer_padding_side)
-prompt="What are the things I should be cautious about when I visit here?"
-image_url="https://llava-vl.github.io/static/images/view.jpg"
-output_text, genertaion_time = model.chat(prompt=prompt,
-                                          image=image_url,
-                                          tokenizer=tokenizer)
 print('model output:', output_text)
 print('runing time:', genertaion_time)
 ```
-### Example Usage
-**Input Image:**
 ![Llava Input Image Example](https://llava-vl.github.io/static/images/view.jpg "Llava Input Image Example")
-**Prompt:** "What are the things I should be cautious about when I visit here?"
-**Model Output:**
 ```
 When I visit the beach at the waterfront, I should be cautious about several things. First, I should be cautious about the water, as it is a popular spot for boating and fishing. The water is shallow and shallow, making it difficult for boats to navigate and navigate. Additionally, the water is not a suitable surface for boating, as it is too shallow for boating. Additionally, the water is not suitable for swimming or fishing, as it is too cold and wet. Lastly, I should be cautious about the presence of other boats, such as boats that are parked on the beach, or boats that are not visible from the water. These factors can lead to potential accidents or accidents, as they can cause damage to the boat and the other boats in the water.
 ```
-### Implementation Notes
-For inference, I created a custom class [modeling_tinyllava_llama.py](https://huggingface.co/keeeeenw/MicroLlava-siglip-so400m/blob/main/modeling_tinyllava_llama.py) which:
 - Loads the same chat template as the TinyLlava model for TinyLlama
 - Connects the LLM to the vision tower
 - May require additional dependencies such as PyTorch and Transformers library
 ---
-## Evaluation
-Evaluation results will be added in the coming days.
-### VQAv2 Results
-| Split | Yes/No | Number | Other | Overall |
-|-------|--------|---------|-------|---------|
-| test-dev | 65.08 | 28.97 | 29.32 | **44.01** |
-#### Evaluation Details
 - **Dataset**: VQAv2 (Visual Question Answering v2.0)
 - **Challenge**: [VQA Challenge 2017](https://eval.ai/web/challenges/challenge-page/830/)
 - **Split**: test-dev
-- **Overall Accuracy**: 44.01%
-#### Performance Breakdown
 - **Yes/No Questions**: 65.08% - Performance on binary questions
 - **Number Questions**: 28.97% - Performance on counting/numerical questions
 - **Other Questions**: 29.32% - Performance on open-ended questions
 - **Overall**: 44.01% - Weighted average across all question types
-Planned tests include:
 - VQAv2 test set (instead of test-dev)
-- and datasets from [TinyLlava evaluation](https://tinyllava-factory.readthedocs.io/en/latest/Evaluation.html)
-Community contributions with benchmark results are welcome and encouraged.
 ---
-## Intended uses and limitations
-**Intended uses**
-- Rapid experimentation for vision-language research on limited hardware
-- Educational demonstrations for students and hobbyists
-- Starting point for domain-specific finetuning
-**Limitations**
-- The small LLM size and compact vision encoder may limit reasoning depth and OCR performance
-- Performance can vary significantly depending on the image domain and quality
-- The model includes minimal safety filtering and refusal behavior — downstream applications should implement their own safeguards
-> ⚠️ This model should not be used for applications that may cause harm or have significant safety, financial, legal, or medical implications without thorough human review.
 ---
-## Reproducibility
 For reproducibility, please visit my fork of [TinyLLaVA_Factory](https://github.com/keeeeenw/TinyLLaVA_Factory), which follows the exact same pre-training and fine-tuning steps as the original implementation.
-### Key Differences
-**Pre-training Modifications:**
 To support training on a single GPU, I modified several hyperparameters:
-- `gradient_accumulation_steps`: 2 → 8
-- `learning_rate`: 1e-3 → 2.5e-4
-- `warmup_ratio`: 0.03 → 0.06
-The original hyperparameters were too aggressive for pre-training, causing training loss to increase after some time. With the updated hyperparameters, pre-training loss remained stable, which is expected for LLaVA's first stage where we align the LLM output with ViT features.
-**Fine-tuning Changes:**
-- All major hyperparameters remain the same as the original
-- Used `bfloat16` precision instead of `float16` for improved numerical stability
-- The current model version does not use `ocr_vqa` due to difficulties downloading all required images for fine-tuning
-### Training Setup
 - **Hardware**: Single GPU configuration
-- **Precision**: bfloat16 (fine-tuning), modified from original float16. For pre-training, I used float16 which is the same configuration as the original TinyLlava model.
 - **Stages**: Two-stage training following LLaVA methodology
-  1. Pre-training: Vision-language alignment with stable loss
-  2. Fine-tuning: Task-specific adaptation
 ---
-## Citation
 ```bibtex
 @misc{wang2024microllama,
@@ -177,24 +189,30 @@ The original hyperparameters were too aggressive for pre-training, causing train
 }
 ```
-## License
-This model is released under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).
-You are free to use, modify, and distribute this model and its derivatives, provided that you comply with the terms of the license.
-If you use this model in your research or applications, please credit the original authors and clearly indicate any modifications you have made.
-> **Note**: Ensure that the datasets used for pretraining or finetuning also allow redistribution of derived model weights.
 ---
-## Acknowledgements
 This work builds upon the efforts of many in the open-source AI community:
 - **[TinyLLaVA Factory](https://github.com/TinyLLaVA/TinyLLaVA_Factory)** maintainers and contributors for creating the training framework
-- **[`keeeeenw/MicroLlama`](https://huggingface.co/keeeeenw/MicroLlama)** I am also the creator of MicroLlama. Please help support my work!
 - **SigLIP** authors for the efficient vision encoder architecture
 - Contributors to **LAION-CC-SBU-558K** and other datasets used in pretraining and finetuning
-- The Hugging Face ecosystem for hosting, tools, and community support

 - google/siglip-so400m-patch14-384
 ---
+# 🔥 MicroLLaVA-siglip-so400m
+> **A compact vision language model that you can pretrain and finetune on a single consumer GPU such as NVIDIA RTX 4090 with 24G of VRAM.**
+---
+## ⚡ TLDR
+| 📋 **Item** | 🔧 **Detail** |
+|-------------|---------------|
+| **Framework** | Transformers + PyTorch |
+| **Checkpoint type** | `safetensors` |
+| **LLM** | [`keeeeenw/MicroLlama`](https://huggingface.co/keeeeenw/MicroLlama) (about 300M parameters) |
+| **Vision tower** | [`siglip-so400m-patch14-384`](https://huggingface.co/google/siglip-so400m-patch14-384) |
+| **Hardware used** | Single NVIDIA RTX 4090 |
+| **Training stack** | No DeepSpeed required |
+| **Intended tasks** | Visual Question Answering, caption-style prompts |
 ---
+## 🚀 Introduction
+**MicroLLaVA** is a [TinyLLaVA Factory](https://github.com/TinyLLaVA/TinyLLaVA_Factory) based model that pairs a very small language model [`keeeeenw/MicroLlama`](https://huggingface.co/keeeeenw/MicroLlama) with an efficient SigLIP vision encoder.
+🎯 **The goal**: Create a vision language model that almost anyone can train and iterate on with one consumer GPU.
+### 🧠 **Model Components**
 - **Language model**: [`keeeeenw/MicroLlama`](https://huggingface.co/keeeeenw/MicroLlama) with ~300M parameters
 - **Vision encoder**: [`siglip-so400m-patch14-384`](https://huggingface.co/google/siglip-so400m-patch14-384)
 - **Training codebase**: [TinyLLaVA Factory](https://github.com/TinyLLaVA/TinyLLaVA_Factory) with additional changes in my fork: [Custom fork with training tweaks](https://github.com/keeeeenw/TinyLLaVA_Factory)
+### ⏱️ **Training Times**
+Because of its compact size, this model can be trained entirely on a single NVIDIA RTX 4090 **without DeepSpeed**.
+- **Pretraining** on LAION-CC-SBU-558K: **~5 hours** ⚡
+- **Supervised finetuning** on all TinyLLaVA Factory datasets (except `ocr_vqa`): **~12 hours** 🔥
 ---
+## 💻 Quick Start
 ```python
 from transformers import AutoTokenizer, AutoModelForCausalLM
 model = AutoModelForCausalLM.from_pretrained(hf_path, trust_remote_code=True)
 # model.cuda() # turn on cuda as needed by the model runs fairly quickly on CPU.
 config = model.config
+tokenizer = AutoTokenizer.from_pretrained(
+    hf_path,
+    use_fast=False,
+    model_max_length=config.tokenizer_model_max_length,
+    padding_side=config.tokenizer_padding_side
+)
+prompt = "What are the things I should be cautious about when I visit here?"
+image_url = "https://llava-vl.github.io/static/images/view.jpg"
+output_text, genertaion_time = model.chat(
+    prompt=prompt,
+    image=image_url,
+    tokenizer=tokenizer
+)
 print('model output:', output_text)
 print('runing time:', genertaion_time)
 ```
+---
+## 🖼️ Example Usage
+### 📸 **Input Image:**
 ![Llava Input Image Example](https://llava-vl.github.io/static/images/view.jpg "Llava Input Image Example")
+### 💬 **Prompt:**
+*"What are the things I should be cautious about when I visit here?"*
+### 🤖 **Model Output:**
 ```
 When I visit the beach at the waterfront, I should be cautious about several things. First, I should be cautious about the water, as it is a popular spot for boating and fishing. The water is shallow and shallow, making it difficult for boats to navigate and navigate. Additionally, the water is not a suitable surface for boating, as it is too shallow for boating. Additionally, the water is not suitable for swimming or fishing, as it is too cold and wet. Lastly, I should be cautious about the presence of other boats, such as boats that are parked on the beach, or boats that are not visible from the water. These factors can lead to potential accidents or accidents, as they can cause damage to the boat and the other boats in the water.
 ```
+### 🔧 **Implementation Notes**
+For inference, I created a custom class [`modeling_tinyllava_llama.py`](https://huggingface.co/keeeeenw/MicroLlava-siglip-so400m/blob/main/modeling_tinyllava_llama.py) which:
 - Loads the same chat template as the TinyLlava model for TinyLlama
 - Connects the LLM to the vision tower
 - May require additional dependencies such as PyTorch and Transformers library
 ---
+## 📊 Evaluation
+### 🏆 VQAv2 Results
+| **Split** | **Yes/No** | **Number** | **Other** | **Overall** |
+|-----------|------------|------------|-----------|-------------|
+| test-dev | **65.08** | **28.97** | **29.32** | **🎯 44.01** |
+#### 📈 **Evaluation Details**
 - **Dataset**: VQAv2 (Visual Question Answering v2.0)
 - **Challenge**: [VQA Challenge 2017](https://eval.ai/web/challenges/challenge-page/830/)
 - **Split**: test-dev
+- **Overall Accuracy**: **44.01%**
+#### 🎯 **Performance Breakdown**
 - **Yes/No Questions**: 65.08% - Performance on binary questions
 - **Number Questions**: 28.97% - Performance on counting/numerical questions
 - **Other Questions**: 29.32% - Performance on open-ended questions
 - **Overall**: 44.01% - Weighted average across all question types
+### 🔜 **Planned Evaluations**
 - VQAv2 test set (instead of test-dev)
+- Datasets from [TinyLlava evaluation](https://tinyllava-factory.readthedocs.io/en/latest/Evaluation.html)
+Community contributions with benchmark results are welcome and encouraged! 🤝
 ---
+## 🎯 Intended Uses and Limitations
+### ✅ **Intended Uses**
+- **🔬 Rapid experimentation** for vision-language research on limited hardware
+- **🎓 Educational demonstrations** for students and hobbyists
+- **🚀 Starting point** for domain-specific finetuning
+### ⚠️ **Limitations**
+- The small LLM size and compact vision encoder may limit **reasoning depth** and **OCR performance**
+- Performance can **vary significantly** depending on the image domain and quality
+- The model includes **minimal safety filtering** and refusal behavior — downstream applications should implement their own safeguards
+> ⚠️ **Important**: This model should not be used for applications that may cause harm or have significant safety, financial, legal, or medical implications without thorough human review.
 ---
+## 🔬 Reproducibility
 For reproducibility, please visit my fork of [TinyLLaVA_Factory](https://github.com/keeeeenw/TinyLLaVA_Factory), which follows the exact same pre-training and fine-tuning steps as the original implementation.
+### 🔧 **Key Differences**
+#### **🎯 Pre-training Modifications:**
 To support training on a single GPU, I modified several hyperparameters:
+- `gradient_accumulation_steps`: **2 → 8**
+- `learning_rate`: **1e-3 → 2.5e-4**
+- `warmup_ratio`: **0.03 → 0.06**
+*The original hyperparameters were too aggressive for pre-training, causing training loss to increase after some time. With the updated hyperparameters, pre-training loss remained stable, which is expected for LLaVA's first stage where we align the LLM output with ViT features.*
+#### **🎨 Fine-tuning Changes:**
+- All major hyperparameters remain **the same** as the original
+- Used `bfloat16` precision instead of `float16` for **improved numerical stability**
+- The current model version does **not use `ocr_vqa`** due to difficulties downloading all required images for fine-tuning
+### 🛠️ **Training Setup**
 - **Hardware**: Single GPU configuration
+- **Precision**: `bfloat16` (fine-tuning), modified from original `float16`. For pre-training, I used `float16` which is the same configuration as the original TinyLlava model.
 - **Stages**: Two-stage training following LLaVA methodology
+  1. **Pre-training**: Vision-language alignment with stable loss
+  2. **Fine-tuning**: Task-specific adaptation
 ---
+## 📝 Citation
 ```bibtex
 @misc{wang2024microllama,
 }
 ```
+---
+## 📄 License
+This model is released under the [**Apache License 2.0**](https://www.apache.org/licenses/LICENSE-2.0).
+You are **free to use, modify, and distribute** this model and its derivatives, provided that you comply with the terms of the license.
+If you use this model in your research or applications, please **credit the original authors** and clearly indicate any modifications you have made.
+> **📌 Note**: Ensure that the datasets used for pretraining or finetuning also allow redistribution of derived model weights.
 ---
+## 🙏 Acknowledgements
 This work builds upon the efforts of many in the open-source AI community:
 - **[TinyLLaVA Factory](https://github.com/TinyLLaVA/TinyLLaVA_Factory)** maintainers and contributors for creating the training framework
+- **[`keeeeenw/MicroLlama`](https://huggingface.co/keeeeenw/MicroLlama)** I am also the creator of MicroLlama. Please help support my work! ⭐
 - **SigLIP** authors for the efficient vision encoder architecture
 - Contributors to **LAION-CC-SBU-558K** and other datasets used in pretraining and finetuning
+- The **Hugging Face ecosystem** for hosting, tools, and community support 🤗
+---
+### 🌟 **Star this model if you find it useful!** 🌟