README.md

---
language: en
license: apache-2.0
library_name: tensorflow
tags:
  - tensorflow
  - keras
  - emotion-recognition
  - vgg19
  - ckplus
  - rafdb
  - fine-tuning
  - computer-vision
  - deep-learning
  - facial-expression
  - affective-computing
  - tflite
model-index:
  - name: emotion_vgg19_model
    results:
      - task:
          type: image-classification
        dataset:
          type: dataset
          name: CK+ & RAF-DB
        metrics:
          - name: accuracy
            type: float
            value: 0.7751
    inference: "Supports TensorFlow and TensorFlow Lite inference"
---

# 🧠 Emotion Recognition Model – VGG19 (Fine-Tuned on CK+ and RAF-DB)

## 📘 Overview

This repository provides a fine-tuned **VGG19-based Emotion Recognition model** trained using a combination of **CK+** and **RAF-DB** datasets. The model is designed to classify human facial emotions into seven categories and has been optimized for both performance and size (TensorFlow and TensorFlow Lite versions available).

The model is a key module in a broader AI system for **emotion-aware human-computer interaction**, providing robust real-time emotion inference.

---

## 🧩 Model Architecture

The model is based on **VGG19** pre-trained on ImageNet, and fine-tuned in **two stages**:

1. **Stage 1 – Frozen Base Training (10 Epochs):**

   - The convolutional base (VGG19) was frozen.
   - Only newly added dense layers were trained.
   - Purpose: Train classifier layers without disrupting pre-trained features.

2. **Stage 2 – Unfrozen Base Fine-Tuning (30 Epochs):**
   - The base model was unfrozen and fine-tuned with a low learning rate.
   - Purpose: Enhance generalization and feature learning for emotion-specific characteristics.

---

## 📊 Datasets

Two publicly available datasets were used for training and evaluation:

1. **[CK+ Dataset (Kaggle)](https://www.kaggle.com/datasets/shareef0612/ckdataset)**
2. **[RAF-DB Dataset (Kaggle)](https://www.kaggle.com/datasets/shuvoalok/raf-db-dataset)**

### Dataset Preparation

- Combined both datasets for richer emotion diversity.
- Applied **class balancing** through **image augmentation** (rotation, flips, brightness, zoom, and shift).
- Final dataset distribution was uniform across emotion classes.

---

## ⚙️ Training Configuration

| Parameter         | Description                     |
| ----------------- | ------------------------------- |
| **Base Model**    | VGG19 (Pre-trained on ImageNet) |
| **Optimizer**     | Adam                            |
| **Learning Rate** | 1e-4 (unfrozen phase)           |
| **Loss Function** | Sparse Categorical Crossentropy |
| **Batch Size**    | 32                              |
| **Epochs**        | 40 (10 + 30)                    |
| **Image Size**    | 224x224                         |

---

## 📈 Performance Summary

| Metric       | Training | Validation | Testing |
| ------------ | -------- | ---------- | ------- |
| **Accuracy** | 97.42%   | 81.26%     | 77.51%  |
| **Loss**     | 0.0910   | 1.0053     | 1.4182  |

### Classification Report

| Class | Precision | Recall | F1-Score |
| ----- | --------- | ------ | -------- |
| 0     | 0.72      | 0.65   | 0.68     |
| 1     | 0.39      | 0.46   | 0.42     |
| 2     | 0.46      | 0.51   | 0.49     |
| 3     | 0.95      | 0.84   | 0.89     |
| 4     | 0.67      | 0.89   | 0.76     |
| 5     | 0.79      | 0.67   | 0.72     |
| 6     | 0.83      | 0.74   | 0.78     |

**Overall Accuracy:** 77.51%  
**Weighted F1-Score:** 0.78

---

## 🖼️ Visualizations

### 1. Training Accuracy and Loss

- **Graph 1:** Training and Validation Accuracy vs Epochs
  ![Training Accuracy](images/Accuracies.png)
  _Training and validation accuracy over 40 epochs._

- **Graph 2:** Training and Validation Loss vs Epochs
  ![Training Loss](images/Losses.png)
  _Training and validation loss over 40 epochs._

### 2. Dataset Distributions

- **Graph 3:** Original Dataset Class Distribution
  ![Original Dataset Distribution](images/Original_Class_Distribution.png)
  _Class distribution of the original combined dataset._

- **Graph 4:** Balanced (Augmented) Dataset Class Distribution
  ![Balanced Dataset Distribution](images/Balanced_Class_Distribution.png)
  _Class distribution after augmentation and balancing._

### 3. Evaluation Visuals

- **Graph 5:** Multi-Class ROC Curves (AUC per class)
  ![ROC Curves](images/ROC.png)
  _Multi-class ROC curves with AUC values._

- **Graph 6:** Confusion Matrix (Heatmap)
  ![Confusion Matrix](images/Confusion_Matrix.png)
  _Confusion matrix heatmap on test set._

- **Graph 7:** Sample Test Results (Subplots of 5 predictions per class)
  ![Sample Test Results](images/Sample_Test_Results.png)
  _Sample predictions (5 images per class) showing model performance._

These visualizations clearly demonstrate model learning stability, class balance, and classification performance across emotions.

---

## 🧩 Model Files

| File                             | Description                                                        |
| -------------------------------- | ------------------------------------------------------------------ |
| `emotion_vgg19_model.h5`         | Original fine-tuned TensorFlow model (≈230 MB)                     |
| `emotion_vgg19_optimized.tflite` | Optimized TensorFlow Lite model (≈19.2 MB) for mobile/edge devices |

---

## 🧰 Inference Example

```python
import tensorflow as tf
from tensorflow.keras.preprocessing import image
import numpy as np

# Load original model
model_path = 'emotion_vgg19_model.h5'
model = tf.keras.models.load_model(model_path)

# Prepare input
img = image.load_img('test_face.jpg', target_size=(224, 224))
input_data = np.expand_dims(image.img_to_array(img) / 255.0, axis=0)

# Run inference with original model
pred = model.predict(input_data)
classes = ['Angry', 'Disgust', 'Fear', 'Happy', 'Neutral', 'Sad', 'Surprise']
print("Original Model Prediction:", classes[np.argmax(pred)])

# Load TFLite optimized model
tflite_model_path = 'emotion_vgg19_optimized.tflite'
interpreter = tf.lite.Interpreter(model_path=tflite_model_path)
interpreter.allocate_tensors()
input_index = interpreter.get_input_details()[0]['index']
output_index = interpreter.get_output_details()[0]['index']
interpreter.set_tensor(input_index, input_data.astype(np.float32))
interpreter.invoke()
output = interpreter.get_tensor(output_index)
print("TFLite Model Prediction:", classes[np.argmax(output)])
```

---

## 🚀 Key Features

- Dual-dataset fine-tuning (CK+ + RAF-DB)
- Balanced training set with augmentation
- Strong generalization (77.5% test accuracy)
- Mobile-optimized TFLite version (19.2 MB)
- Suitable for real-time emotion-aware applications

---

## 🏷️ Tags

`emotion-recognition` `vgg19` `facial-expression` `deep-learning` `tensorflow` `tflite` `ckplus` `rafdb` `computer-vision` `affective-computing` `multimodal-ai` `fine-tuning`

---

## 📄 Citation

```bibtex
@misc{pasindu_sewmuthu_abewickrama_singhe_2025,
	author       = { Pasindu Sewmuthu Abewickrama Singhe },
	title        = { vgg19-emotion-recognition-ckplus-rafdb (Revision dad246e) },
	year         = 2025,
	url          = { https://huggingface.co/PSewmuthu/vgg19-emotion-recognition-ckplus-rafdb },
	doi          = { 10.57967/hf/6651 },
	publisher    = { Hugging Face }
}
```

---

## 👤 Author & Model Info

**Author:** P.S. Abewickrama Singhe  
**Developed with:** TensorFlow + Keras  
**License:** Apache-2.0  
**Date:** October 2025  
**Email:** [apsewmuthu@gmail.com](mailto:apsewmuthu@gmail.com)