File size: 5,260 Bytes
0fb6d3c 50fd814 0fb6d3c 50fd814 0fb6d3c 50fd814 0fb6d3c bccfc28 0fb6d3c 50fd814 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 |
---
license: mit
tags:
- vision-language
- mixture-of-experts
- text-generation
- vision-transformer
- pytorch
model_index:
- name: SparseFusion
results:
- task:
type: text-generation
dataset:
name: Custom Caption Dataset
type: custom
metrics:
- name: Validation Loss
type: loss
value: 0.8
---
# SparseFusion
**SparseFusion** is a multimodal Mixture-of-Experts (MoE) model integrating a Vision Transformer (ViT) and transformer decoder for image-conditioned text generation. It is built entirely in PyTorch and extends [SeeMOE](https://github.com/AviSoori1x/seemore).
---
## 🧠 Model Details
- **Name**: SparseFusion
- **Author**: Derrick Kirimi ([GitHub](https://github.com/DerrickKirimi) · [LinkedIn](https://www.linkedin.com/in/derrick-kirimi-22a470175/) · [Hugging Face](https://huggingface.co/Aptheos))
- **Model Type**: Vision-Language Model
- **Architecture**:
- Vision Encoder: ViT (96×96 images, 16×16 patches, 512-dim patch embeddings)
- Decoder: Transformer with MoE layers (8 layers, 128-dim, 8 heads)
- MoE Setup: 8 experts, top-2 routing, expert capacity control
- Token Fusion: Concatenation of image tokens and character-level encoded text
- **License**: Apache 2.0
- **Repository**: [GitHub - DerrickKirimi/SparseFusion](https://github.com/DerrickKirimi/SparseFusion)
---
## 🌟 Intended Use
- **Primary Use Case**: Image-conditioned text generation for educational and research experimentation
- **Intended Users**: ML researchers, students, developers
- **Out-of-Scope Uses**: Not suitable for deployment in production or for generating harmful content
---
## 🏋️♂️ Training & Evaluation
### 📅 Dataset
- **Text**: Tiny Shakespeare (character-level)
- **Images**: 300 synthetic image-caption pairs
### ⚙️ Training
- Trained for 2 epochs on **Google Colab (1 GPU, 12 GB VRAM)**
- Logging via **Weights & Biases (wandb)**
### 📊 Hyperparameters
```yaml
epochs: 2
batch_size: 16
learning_rate: 0.001
n_embd: 128
n_head: 8
n_layer: 8
num_experts: 8
top_k: 2
expert_capacity: 32
img_size: 96
patch_size: 16
```
### 📈 Evaluation
- **Validation Loss**: 0.8 after 2 epochs
- **Summary**:
- Generates basic coherent text
- Shows 15% improvement in expert utilization with routing control and load balancing
---
## 🚀 Usage
### 📦 Installation
```bash
pip install torch torchvision transformers huggingface_hub wandb
```
### 🔄 Inference
```python
import torch
import pickle
from PIL import Image
import torchvision.transforms as transforms
from huggingface_hub import hf_hub_download
# Load vocabulary mappings
stoi = pickle.load(open(hf_hub_download("Aptheos/SparseFusion", "stoi.pkl"), "rb"))
itos = pickle.load(open(hf_hub_download("Aptheos/SparseFusion", "itos.pkl"), "rb"))
encode = lambda s: [stoi[c] for c in s]
decode = lambda l: ''.join([itos[i] for i in l])
# Define model architecture
model = VisionMoELanguageModel(
n_embd=128, image_embed_dim=512, vocab_size=len(stoi), n_layer=8,
img_size=96, patch_size=16, num_heads=8, num_blks=3,
emb_dropout=0.1, blk_dropout=0.1, num_experts=8, top_k=2, expert_capacity=32
)
model.load_state_dict(torch.load(hf_hub_download("Aptheos/SparseFusion", "vision_moe_model.pth")))
model.eval().to("cuda")
# Preprocess image
transform = transforms.Compose([
transforms.Resize((96, 96)),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])
image = transform(Image.open("example.jpg")).unsqueeze(0).to("cuda")
prompt = torch.tensor([encode("A photo of")], dtype=torch.long).to("cuda")
# Generate text
generated = model.generate(image, prompt, max_new_tokens=50)
print(decode(generated[0].tolist()))
```
To run on CPU:
```python
model.eval().to("cpu")
image = image.to("cpu")
prompt = prompt.to("cpu")
```
---
## ⚠️ Limitations & Biases
### Limitations
- The model generates incoherent text (e.g., `"A photo ofiecp ntti<pad><pad>..."`) due to training on a small, synthetic dataset of 300 identical images with simplistic captions.
- Vision encoder (ViT) is **not pre-trained**, reducing visual feature quality.
- Character-level tokenization limits text fluency and introduces `<pad>` tokens.
- Limited training time (2 epochs) restricts deep multimodal learning.
### Biases
- Synthetic captions create bias toward repetitive language structures.
- Lack of diverse image inputs may bias the model’s visual representation.
---
## 🔭 Future Work
- Train on larger datasets (e.g., COCO, Flickr30k) for better generalization
- Use pre-trained ViT backbone (e.g., `timm/vit_small_patch16_224`)
- Implement subword tokenization (e.g., SentencePiece, BPE)
- Add modality type embeddings and rotary positional embeddings (RoPE)
- Visualize expert routing and attention patterns for interpretability
- Increase training epochs and perform hyperparameter tuning
---
## 📄 License
Licensed under the **MIT License** for open research and educational use.
---
## 📚 Citation
```bibtex
@misc{sparsefusion2025,
author = {Derrick Kirimi},
title = {SparseFusion: A Multimodal Mixture-of-Experts Model},
year = {2025},
url = {https://huggingface.co/Aptheos/SparseFusion}
}
``` |