|
|
--- |
|
|
language: |
|
|
- en |
|
|
base_model: |
|
|
- openai/clip-vit-large-patch14 |
|
|
tags: |
|
|
- emotion_prediction |
|
|
- VEA |
|
|
- computer_vision |
|
|
- perceptual_tasks |
|
|
- CLIP |
|
|
- EmoSet |
|
|
--- |
|
|
# Don’t Judge Before You CLIP: Visual Emotion Analysis Model |
|
|
|
|
|
This model is part of our paper: |
|
|
*"Don’t Judge Before You CLIP: A Unified Approach for Perceptual Tasks"* |
|
|
It was trained on the *EmoSet dataset* to predict emotion class. |
|
|
|
|
|
## Model Overview |
|
|
|
|
|
Visual perceptual tasks, such as visual emotion analysis, aim to estimate how humans perceive and interpret images. Unlike objective tasks (e.g., object recognition), these tasks rely on subjective human judgment, making labeled data scarce. |
|
|
|
|
|
Our approach leverages *CLIP* as a prior for perceptual tasks, inspired by cognitive research showing that CLIP correlates well with human judgment. This suggests that CLIP implicitly captures human biases, emotions, and preferences. We fine-tune CLIP minimally using LoRA and incorporate an MLP head to adapt it to each specific task. |
|
|
|
|
|
## Training Details |
|
|
|
|
|
- *Dataset*: [EmoSet](https://vcc.tech/EmoSet) |
|
|
- *Architecture*: CLIP Vision Encoder (ViT-L/14) with *LoRA adaptation* |
|
|
- *Loss Function*: Cross Entropy Loss |
|
|
- *Optimizer*: AdamW |
|
|
- *Learning Rate*: 0.0001 |
|
|
- *Batch Size*: 32 |
|
|
|
|
|
## Performance |
|
|
|
|
|
The model was trained on the *EmoSet dataset* using the common train, val, test splits and exhibits *state-of-the-art performance compared to previous methods. |
|
|
|
|
|
## Usage |
|
|
|
|
|
To use the model for inference: |
|
|
|
|
|
```python |
|
|
from torchvision import transforms |
|
|
import torch |
|
|
from PIL import Image |
|
|
device = torch.device("cuda" if torch.cuda.is_available() else "cpu") |
|
|
# Load model |
|
|
model = torch.load("EmoSet_clip_Lora_16.0R_8.0alphaLora_32_batch_0.0001_headmlp.pth").to(device).eval() |
|
|
# Load an image |
|
|
image = Image.open("image_path.jpg").convert("RGB") |
|
|
# Preprocess and predict |
|
|
def Emo_preprocess(): |
|
|
transform = transforms.Compose([ |
|
|
transforms.Resize(224), |
|
|
transforms.CenterCrop(size=(224,224)), |
|
|
transforms.ToTensor(), |
|
|
# Note: The model normalizes the image inside the forward pass |
|
|
# using mean = (0.48145466, 0.4578275, 0.40821073) and |
|
|
# std = (0.26862954, 0.26130258, 0.27577711) |
|
|
]) |
|
|
return transform |
|
|
image = Emo_preprocess()(image).unsqueeze(0).to(device) |
|
|
with torch.no_grad(): |
|
|
emo_label = model(image).item() |
|
|
print(f"Predicted Emotion: {emo_label}") |
