language:
- en
base_model:
- openai/clip-vit-large-patch14
tags:
- emotion_prediction
- VEA
- computer_vision
- perceptual_tasks
- CLIP
- EmoSet
Don’t Judge Before You CLIP: Visual Emotion Analysis Model
This model is part of our paper:
"Don’t Judge Before You CLIP: A Unified Approach for Perceptual Tasks"
It was trained on the EmoSet dataset to predict emotion class.
Model Overview
Visual perceptual tasks, such as visual emotion analysis, aim to estimate how humans perceive and interpret images. Unlike objective tasks (e.g., object recognition), these tasks rely on subjective human judgment, making labeled data scarce.
Our approach leverages CLIP as a prior for perceptual tasks, inspired by cognitive research showing that CLIP correlates well with human judgment. This suggests that CLIP implicitly captures human biases, emotions, and preferences. We fine-tune CLIP minimally using LoRA and incorporate an MLP head to adapt it to each specific task.
Training Details
- Dataset: EmoSet
- Architecture: CLIP Vision Encoder (ViT-L/14) with LoRA adaptation
- Loss Function: Cross Entropy Loss
- Optimizer: AdamW
- Learning Rate: 0.0001
- Batch Size: 32
Performance
The model was trained on the EmoSet dataset using the common train, val, test splits and exhibits *state-of-the-art performance compared to previous methods.
Usage
To use the model for inference:
from torchvision import transforms
import torch
from PIL import Image
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# Load model
model = torch.load("EmoSet_clip_Lora_16.0R_8.0alphaLora_32_batch_0.0001_headmlp.pth").to(device).eval()
# Load an image
image = Image.open("image_path.jpg").convert("RGB")
# Preprocess and predict
def Emo_preprocess():
transform = transforms.Compose([
transforms.Resize(224),
transforms.CenterCrop(size=(224,224)),
transforms.ToTensor(),
# Note: The model normalizes the image inside the forward pass
# using mean = (0.48145466, 0.4578275, 0.40821073) and
# std = (0.26862954, 0.26130258, 0.27577711)
])
return transform
image = Emo_preprocess()(image).unsqueeze(0).to(device)
with torch.no_grad():
emo_label = model(image).item()
print(f"Predicted Emotion: {emo_label}")