<jupyter_start><jupyter_text>![image.png](../assets/data_prep_wf.png)  Raw Data - FER2013* https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data* 48 X 48 gray scale images* 28,709 training samples* 3,589 validation data* 3,589 test data* 7 emotion labels (0=Angry, 1=Disgust, 2=Fear, 3=Happy, 4=Sad, 5=Surprise, 6=Neutral)* ~60-65% human accuracy in labelling![image.png](../assets/test_raw_data.png) Data Preparation is Critical* State of the Art accuracy using a Convolutional Neural Network (CNN) model directly on this raw data is **63%** [1].* State of the Art accuracy using a slightly modified Convolutional Neural Network (CNN) model after processing the data is **83%** [2].  Processed Data - FER+Follow the instructions listed in the README file - https://github.com/TalkAI/facial-emotion-recognition-gluonbefore-you-start* FER+ has new corrected labels* FER+ has 8 emotions - (0: 'neutral', 1: 'happiness', 2: 'surprise', 3: 'sadness', 4: 'anger', 5: 'disgust', 6: 'fear',7: 'contempt')* Image augmentations:    * Crop faces in the images – bounding box in the FER+ dataset    * Scale image size from 48 X 48 -> 64 X 64    * Shift image    * Flip image    * Rotate (angle) image    * Normalize the pixels in the image | Emotion  | Train  | Val  |  Test ||---|---|---|---|| neutral  |8733   |  1180 | 1083  || happiness  | 7284  | 862  | 892  || surprise  | 3136  | 411  | 394  || sadness  | 3022  | 348  | 382  || anger  | 2098  |  289 |  269 || disgust  | 116  | 25  | 16  || fear  | 536  | 60  | 86  || comtempt  | 120  | 16  | 15  |<jupyter_code># 8 Emotions we want to recognize
emotion_table = {0: 'neutral',
                 1: 'happiness',
                 2: 'surprise',
                 3: 'sadness',
                 4: 'anger',
                 5: 'disgust',
                 6: 'fear',
                 7: 'contempt'}
processed_train_images = np.load('../data/fer_train_processed_images.npy')
processed_train_labels = np.load('../data/fer_train_processed_labels.npy')
print(processed_train_images.shape, processed_train_labels.shape)
from matplotlib import pyplot as plt
plt.imshow(processed_train_images[987].reshape(64,64), cmap='gray')
processed_test_images = np.load('../data/fer_test_processed_images.npy')
processed_test_labels = np.load('../data/fer_test_processed_labels.npy')
processed_val_images = np.load('../data/fer_val_processed_images.npy')
processed_val_labels = np.load('../data/fer_val_processed_labels.npy')
print(processed_test_images.shape, processed_test_labels.shape)
print(processed_val_images.shape, processed_val_labels.shape)
# Sets context to GPU, if available. Otherwise sets to CPU.
# Note that training on one GPU takes approx. 1.1 minute per epoch, and may take longer on CPU

ctx = mx.gpu() if mx.context.num_gpus() else mx.cpu()<jupyter_output><empty_output><jupyter_text>![image.png](../assets/model_training_prep_wf.png)  Step 1 – Construct the Neural Network  13 layer VGGNet presented in the paper [2]![image.png](../assets/network.png)<jupyter_code># We use HybridSequential network type to able to save the trained model as symbols and params.
# More Info - https://mxnet.incubator.apache.org/tutorials/gluon/save_load_params.html
net = gluon.nn.HybridSequential()

# Construct 13 layer VGGNet suggested in the paper
with net.name_scope():
    net.add(gluon.nn.Conv2D(channels=64, kernel_size=3, padding=(1,1), activation='relu'))
    net.add(gluon.nn.Conv2D(channels=64, kernel_size=3, padding=(1,1), activation='relu'))
    net.add(gluon.nn.MaxPool2D(pool_size=2, strides=2))
    net.add(gluon.nn.Dropout(0.25))
    
    net.add(gluon.nn.Conv2D(channels=128, kernel_size=3, padding=(1,1), activation='relu'))
    net.add(gluon.nn.Conv2D(channels=128, kernel_size=3, padding=(1,1), activation='relu'))
    net.add(gluon.nn.MaxPool2D(pool_size=2, strides=2))
    net.add(gluon.nn.Dropout(0.25))
    
    net.add(gluon.nn.Conv2D(channels=256, kernel_size=3, padding=(1,1), activation='relu'))
    net.add(gluon.nn.Conv2D(channels=256, kernel_size=3, padding=(1,1), activation='relu'))
    net.add(gluon.nn.Conv2D(channels=256, kernel_size=3, padding=(1,1), activation='relu'))
    net.add(gluon.nn.MaxPool2D(pool_size=2, strides=2))
    net.add(gluon.nn.Dropout(0.25))
    
    net.add(gluon.nn.Conv2D(channels=256, kernel_size=3, padding=(1,1), activation='relu'))
    net.add(gluon.nn.Conv2D(channels=256, kernel_size=3, padding=(1,1), activation='relu'))
    net.add(gluon.nn.Conv2D(channels=256, kernel_size=3, padding=(1,1), activation='relu'))
    net.add(gluon.nn.MaxPool2D(pool_size=2, strides=2))
    net.add(gluon.nn.Dropout(0.25))
    
    net.add(gluon.nn.Flatten())
    net.add(gluon.nn.Dense(1024, activation='relu'))
    net.add(gluon.nn.Dropout(0.5))
    net.add(gluon.nn.Dense(1024, activation='relu'))
    net.add(gluon.nn.Dropout(0.5))
    net.add(gluon.nn.Dense(8))
# We Hybridize the HybridSequential network to able to save the trained model as symbols and params.
# More Info - https://mxnet.incubator.apache.org/tutorials/gluon/save_load_params.html
net.hybridize()<jupyter_output><empty_output><jupyter_text>Step 2 – Initialize the parameters in Neural Network<jupyter_code>net.collect_params().initialize(mx.init.Xavier(), ctx=ctx)
nd.waitall()
# Use MXBOARD here to visualize network
x = mx.sym.var('data')
sym = net(x)
mx.viz.plot_network(sym)<jupyter_output><empty_output><jupyter_text>Step 3 – Prepare the Trainer with optimizer<jupyter_code>softmax_cross_entropy = gluon.loss.SoftmaxCrossEntropyLoss()
batch_size = 32
trainer = gluon.Trainer(net.collect_params(), 'sgd', {'learning_rate': 0.0025, 'momentum': 0.9})<jupyter_output><empty_output><jupyter_text>Step 4 – Prepare the model evaluation strategy<jupyter_code>def evaluate_accuracy(data_iterator, net):
    acc = mx.metric.Accuracy()
    for i, (data, label) in enumerate(data_iterator):
        data = data.as_in_context(ctx)
        label = label.as_in_context(ctx)
        output = net(data)
        predictions = nd.argmax(output, axis=1)
        acc.update(preds=predictions, labels=label)
    return acc.get()[1]<jupyter_output><empty_output><jupyter_text>Step 5 – Prepare data loaders<jupyter_code>train_labels = np.argmax(processed_train_labels, axis=1)
val_labels = np.argmax(processed_val_labels, axis=1)

train_data = gluon.data.DataLoader(gluon.data.ArrayDataset(processed_train_images, train_labels), batch_size = batch_size, shuffle=True)
val_data = gluon.data.DataLoader(gluon.data.ArrayDataset(processed_val_images, val_labels), batch_size = batch_size)<jupyter_output><empty_output><jupyter_text>Step 6 – Train the Neural Network<jupyter_code>epochs = 25

train_accuracies = []
losses = []
val_accuracies = []

for e in range(epochs):
    batch = 0
    for i, (data, label) in enumerate(train_data):
        data = data.as_in_context(ctx)
        label = label.as_in_context(ctx)
        with autograd.record():
            output = net(data)
            loss = softmax_cross_entropy(output, label)
            
        loss.backward()
        trainer.step(data.shape[0])
        curr_loss = nd.mean(loss).asscalar()
        batch +=1        

    val_accuracy = evaluate_accuracy(val_data, net)
    train_accuracy = evaluate_accuracy(train_data, net)
    
    losses.append(curr_loss)
    train_accuracies.append(train_accuracy)
    val_accuracies.append(val_accuracy)
    
    print("Epoch %s. Loss: %s, Train_acc %s, Val_acc %s" % (e, curr_loss, train_accuracy, val_accuracy))<jupyter_output>Epoch 0. Loss: 1.53916, Train_acc 0.348692353763, Val_acc 0.369790034472
Epoch 1. Loss: 1.66221, Train_acc 0.384627670194, Val_acc 0.387652773425
Epoch 2. Loss: 1.39836, Train_acc 0.399401078059, Val_acc 0.402381698527
Epoch 3. Loss: 1.52955, Train_acc 0.461768816131, Val_acc 0.482607333124
Epoch 4. Loss: 1.40021, Train_acc 0.514154521861, Val_acc 0.528674396741
Epoch 5. Loss: 1.67981, Train_acc 0.56027151128, Val_acc 0.571921027891
Epoch 6. Loss: 1.30359, Train_acc 0.617248951887, Val_acc 0.632090253839
Epoch 7. Loss: 1.00052, Train_acc 0.666200838491, Val_acc 0.683798182388
Epoch 8. Loss: 0.847253, Train_acc 0.694270313436, Val_acc 0.707615167659
Epoch 9. Loss: 1.05446, Train_acc 0.713835096826, Val_acc 0.715763083673
Epoch 10. Loss: 0.860204, Train_acc 0.732241964464, Val_acc 0.734565966782
Epoch 11. Loss: 0.632866, Train_acc 0.75016969455, Val_acc 0.740206831714
Epoch 12. Loss: 0.545957, Train_acc 0.760670792573, Val_acc 0.737386399248
Epoch 13. Loss: 0.79762, Train_acc 0.779157516[...]<jupyter_text>Step 7 - Evaluate on Test Data<jupyter_code># Test accuracy

acc = mx.metric.Accuracy()
test_labels = np.argmax(processed_test_labels, axis=1)
data_iterator = gluon.data.DataLoader(gluon.data.ArrayDataset(processed_test_images, test_labels), batch_size = 32)
for i, (data, label) in enumerate(data_iterator):
    data = data.as_in_context(ctx)
    label = label.as_in_context(ctx)
    output = net(data)
    predictions = nd.argmax(output, axis=1)
    acc.update(preds=predictions, labels=label)
print("Test Accuracy - ", acc.get()[1])
# for plotting purposes
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt

epochs = range(len(train_accuracies))

f = plt.figure(figsize=(12,6))
fg1 = f.add_subplot(121)
fg2 = f.add_subplot(122)

fg1.set_xlabel('epoch',fontsize=14)  
fg1.set_title('Loss over Training')
fg1.grid(True, which="both")
fg1.plot(epochs, losses)

fg2.set_title('Comparing accuracy')
fg2.set_xlabel('epoch', fontsize=14)
fg2.grid(True, which="both")

p1, = fg2.plot(epochs, train_accuracies)
p2, = fg2.plot(epochs, val_accuracies)
fg2.legend([p1, p2], ['training accuracy', 'validation accuracy'],fontsize=14)
# Example Inference
idx = 98
plt.imshow(processed_test_images[idx].reshape(64,64), cmap='gray')
print("Actual Emotion - ", emotion_table[test_labels[idx]])
# Perform Inference
output = net(mx.nd.array(processed_test_images[idx].reshape(1,1,64,64)).as_in_context(ctx))
print("Predicted Emotion - ", emotion_table[nd.argmax(output, axis=1).asnumpy()[0]])<jupyter_output>Predicted Emotion -  happiness<jupyter_text>![image.png](../assets/inference_wf.png)  Step 8 - Export the model for Production<jupyter_code># Export the model for production deployment.
# There will be 2 files exported: 
# 1) gluon_ferplus-symbol.json => Contains the network definition
# 2) gluon_ferplus-0000.params => Contains the weights in the network
net.export('fer')<jupyter_output><empty_output><jupyter_text>Step 9 - Install pre-requisites for model servingWe will need the PyPi packages listed below to test model server locally, and to perform image pre-processing prior to the model inference.<jupyter_code>!pip install mxnet-model-server
!pip install model-archiver
!pip install scikit-image
!pip install opencv-python<jupyter_output><empty_output><jupyter_text>Step 10 - Export a model archiveWe will now prepare the model archive, which encapsulates all of the resources required by MMS for serving the model. To learn more about the model archive, visit [MMS Export docs](https://github.com/awslabs/mxnet-model-server/blob/master/docs/export.md).<jupyter_code># As a first step, we will download the pre-trained model. 
# You can skip this step if you have just trained your model, but then you will need to copy the model files into ferplus directory
!mkdir fer
%cd fer
!curl -O https://s3.amazonaws.com/mxnet-demo-models/models/fer/fer-0000.params
!curl -O https://s3.amazonaws.com/mxnet-demo-models/models/fer/fer-symbol.json
# Now let's pull in and review the other resources needed for the model archive
%cp ../../model_archive_resources/* .
# We define the model's input and output type and shape via signature.json
!cat signature.json
# We define the model's class label names via synset.txt
!cat synset.txt
# And lastly, we define custom code for request handling via python code other auxiliary files
!cat fer_service.py
# Let's package everything up into a Model Archive bundle
% cd ..
!model-archiver --model-name fer --model-path ./fer --handler fer_service:handle
!ls -l fer.mar<jupyter_output><empty_output><jupyter_text>Step 11 - Serving the model archive with MXNet Model ServerNow that we have the model archive ready, we can start the server and ask it to setup HTTP endpoints to serve the model, emit real-time operational metrics and more.To learn more about serving, check out the [MMS Serving docs](https://github.com/awslabs/mxnet-model-server/blob/master/docs/server.md).To start the server and load the FER model on startup, go to the console and run:  `$ mxnet-model-server --models fer=fer.mar --model-store .`<jupyter_code># Check out the health endpoint to make sure model server is running
!curl http://127.0.0.1:8080/ping
# Call MMS management API to see list of loaded models
!curl http://localhost:8081/models
# Query the loaded FER model
!curl http://localhost:8081/models/fer<jupyter_output><empty_output><jupyter_text>Now let's make a prediction request with a test image:![Neutral](../assets/neutral-sad.png)<jupyter_code>!curl -X POST http://127.0.0.1:8080/fer/predict -T "../assets/neutral-sad.png"<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup