Facial Expression Recognition with Keras

Kaggle announced facial expression recognition challenge in 2013. Researchers are expected to create models to detect 7 different emotions from human being faces. However, recent studies are far away from the excellent results even today. That’s why, this topic is still satisfying subject.

Scarlett Johansson

Dataset

The both training and evaluation operations would be handled with Fec2013 dataset. Compressed version of the dataset takes 92 MB space whereas uncompressed version takes 295 MB space. There are 28K training and 3K testing images in the dataset. Each image was stored as 48×48 pixel. The pure dataset consists of image pixels (48×48=2304 values), emotion of each image and usage type (as train or test instance).


Neural Networks Fundamentals in Python

Suppose that the dataset is already loaded under the data folder. Herein, we can read the dataset content as mentioned below.

with open("/data/fer2013.csv") as f:
content = f.readlines()

lines = np.array(content)

num_of_instances = lines.size
print("number of instances: ",num_of_instances)

Learning Procedure

Deep learning dominates computer vision studies in recent years. Even academic computer vision conferences are closely transformed into Deep Learning activities. Herein, we would apply convolutional neural networks to tackle this task. And we will construct CNN with Keras using TensorFlow backend.

We’ve already loaded the dataset before. Now, train and test set can be stored into dedicated variables.

x_train, y_train, x_test, y_test = [], [], [], []

for i in range(1,num_of_instances):
 try:
  emotion, img, usage = lines[i].split(",")

  val = img.split(" ")
  pixels = np.array(val, 'float32')

  emotion = keras.utils.to_categorical(emotion, num_classes)

  if 'Training' in usage:
   y_train.append(emotion)
   x_train.append(pixels)
  elif 'PublicTest' in usage:
   y_test.append(emotion)
   x_test.append(pixels)
 except:
  print("", end="")

Time to construct CNN structure.

model = Sequential()

#1st convolution layer
model.add(Conv2D(64, (5, 5), activation='relu', input_shape=(48,48,1)))
model.add(MaxPooling2D(pool_size=(5,5), strides=(2, 2)))

#2nd convolution layer
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(AveragePooling2D(pool_size=(3,3), strides=(2, 2)))

#3rd convolution layer
model.add(Conv2D(128, (3, 3), activation='relu'))
model.add(Conv2D(128, (3, 3), activation='relu'))
model.add(AveragePooling2D(pool_size=(3,3), strides=(2, 2)))

model.add(Flatten())

#fully connected neural networks
model.add(Dense(1024, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(1024, activation='relu'))
model.add(Dropout(0.2))

model.add(Dense(num_classes, activation='softmax'))

We can train the network. To complete the training in less time, I prefer to implement learning with randomly selected trainset instances. That is the reason why train and fit generator used. Also, loss function would be cross entropy because the task is multi class classification.

gen = ImageDataGenerator()
train_generator = gen.flow(x_train, y_train, batch_size=batch_size)

model.compile(loss='categorical_crossentropy'
, optimizer=keras.optimizers.Adam()
, metrics=['accuracy']
)

model.fit_generator(train_generator, steps_per_epoch=batch_size, epochs=epochs)

Fit is over. We can evaluate the network.

train_score = model.evaluate(x_train, y_train, verbose=0)
print('Train loss:', train_score[0])
print('Train accuracy:', 100*train_score[1])

test_score = model.evaluate(x_test, y_test, verbose=0)
print('Test loss:', test_score[0])
print('Test accuracy:', 100*test_score[1])

I’ve got the following results not to fall into overfitting. I faced with overfitting when I increase the epoch.

Test loss: 2.27945706329
Test accuracy: 57.4254667071

Train loss: 0.223031098232
Train accuracy: 92.0512731201

Confusion Matrix

Sure, accuracy should not express right impression for multi class classification problems. Confusion matrix of this model is demonstrated below. Lines represent actual values whereas columns state predictions. I mean that there are 467 angry instances in testset. We can classify 214 angry items correctly. On the other hand, we classified 9 items as disgust but these items are actual angry ones.

Angry Disgust Fear Happy Sad Surprise Neutral
Angry 214 9 53 30 67 8 86
Disgust 10 24 9 2 6 0 5
Fear 45 2 208 29 89 45 78
Happy 24 0 40 696 37 18 80
Sad 65 3 83 56 285 10 151
Surprise 7 1 42 27 9 303 26
Neutral 45 2 68 65 88 8 331

Basically, scikit-learn produces that confusion matrix.


from sklearn.metrics import classification_report, confusion_matrix

pred_list = []; actual_list = []

for i in predictions:

pred_list.append(np.argmax(i))

for i in y_test:

actual_list.append(np.argmax(i))

confusion_matrix(actual_list, pred_list)

Testing

Let’s try to recognize facial expressions of custom images. Because only error rates don’t express anything.

img = image.load_img("/data/pablo.png", grayscale=True, target_size=(48, 48))

x = image.img_to_array(img)
x = np.expand_dims(x, axis = 0)

x /= 255

custom = model.predict(x)
emotion_analysis(custom[0])

x = np.array(x, 'float32')
x = x.reshape([48, 48]);

plt.gray()
plt.imshow(x)
plt.show()

Emotions stored as numerical as labeled from 0 to 6. Keras would produce an output array including these 7 different emotion scores. We can visualize each prediction as bar chart.

def emotion_analysis(emotions):
objects = ('angry', 'disgust', 'fear', 'happy', 'sad', 'surprise', 'neutral')
y_pos = np.arange(len(objects))

plt.bar(y_pos, emotions, align='center', alpha=0.5)
plt.xticks(y_pos, objects)
plt.ylabel('percentage')
plt.title('emotion')

plt.show()

If you watch the famous Netflix series Narcos, then you would be familiar with the following picture. The following picture of Pablo Escobar is taken in a police station when he was taken into custody. It seems that the model we’ve constructed can successfully recognize Pablo in happy mood.

pablo-facial-expression
Pablo Escobar’s facial expression

Secondly, we will test the scene of Marlon Brando acting in Godfather as Don Corleone. Corleone cries at dead body of his son’s elbow. It seems that the model can recognize Brando’s facial expression, too.

marlon-brando-facial-expression
Marlon Brando’s facial expression

What’s more, Hugh Jackman comes to my mind as always angry figure. That’s why, I would like to test him. Especially, I choose a picture of Jackman from X-Men as Wolverine. Result seems very successful.

hugh-jackman-facial-expression
Hugh Jackman’s facial expression

Finally, art authorities still cannot come to mutual agreement for Mona Lisa’s emotion. Network says that Mona Lisa is in neutral mood.

mona-lisa-facial-expression
Da Vinci’s Mona Lisa’s facial expression

Real time solution

Besides, we can apply emotion analysis on a video streaming or web cam capturing. I’ve written a dedicated blog post about this subject. Its demo is shown below.

Web cam

Me and my colleagues try to act all emotion classes. As seen, this implementation runs very fast.

Video streaming

This video is captured during the testimony of Zuckerberg after Cambridge Analytica scandal. Facebook lost 134 billion dollar after this news. This makes unhappy anyone just like Mark as detected below.

Conclusion

So, we’ve constructed a CNN model to recognize facial expressions of human beings. Model produces 57% accuracy on test set. That can be acceptable because winner of kaggle challenge has got 34% accuracy.

Processing detected faces instead of the entire image would increase accuracy. That’s a little trick. I crop the faces manually before running network.

The entire code of the project is pushed on GitHub. Also, you might want to apply transfer learning and use pre-trained weights. Pre-trained weights and pre-constructed network structure are pushed on GitHub, too.

If you interested in this post, you might be interested in deep face recognition.

59 Comments

  1. Hi,blogposter.Thanks a lot for your wonderful sharing . It help me a lot .But I am haveing some problem when i tried to run facial-expression-recognition.py scripts in your project. I have downloaded the whole project from your github account.
    when read the data from the fer2013.csv I got the below error
    Traceback (most recent call last):
    File “C:/Users/liuxiongcheng/PycharmProjects/untitled/Facial_Expression_Recognition/facial-expression-recognition.py”, line 49, in
    pixels = np.array(val, ‘float32’)
    ValueError: could not convert string to float:

  2. Hello, it seems that content of Fec2013 file is changed. Additional cite and reference lines added into the raw file. We can handle this with adding try – catch mechanism. Would you change the code as illustrated below?

    for i in range(1,num_of_instances):

    try:
    emotion, img, usage = lines[i].split(“,”)

    val = img.split(” “)

    pixels = np.array(val, ‘float32’)

    emotion = keras.utils.to_categorical(emotion, num_classes)

    if ‘Training’ in usage:
    y_train.append(emotion)
    x_train.append(pixels)
    elif ‘PublicTest’ in usage:
    y_test.append(emotion)
    x_test.append(pixels)
    except:
    print(“”, end=””)

  3. Thanks for sharing your experience, but I wanted to know your view on training the inception v3 for facial expression recognition (I mean not loading the pre-trained weights and training the inception v3 from scratch).

    1. Training inception for emotion analysis would most probably produce much more successful results than the design in this post. But, inception V3 has very complex structure. That’s why, you must implement learning with highly costly GPUs for a long time.

      1. Hi Serengil,

        I have Nvidia 1080 GTX GPU so no worry for me. I have seen the google code https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/ for training the inception model but i only go the 422% test accuracy. Below are the steps i followed
        1. First converted the FER2013 in to jpg images with emotion types as directory.
        2. Next cloned the tensorflow-for-poests-2 github project and retrained

        Below are the few lines
        INFO:tensorflow:2018-05-09 20:38:24.267195: Step 3940: Cross entropy = 1.316952
        INFO:tensorflow:2018-05-09 20:38:24.305648: Step 3940: Validation accuracy = 40.0% (N=100)
        INFO:tensorflow:2018-05-09 20:38:24.709390: Step 3950: Train accuracy = 52.0%
        INFO:tensorflow:2018-05-09 20:38:24.709500: Step 3950: Cross entropy = 1.369833
        INFO:tensorflow:2018-05-09 20:38:24.749251: Step 3950: Validation accuracy = 40.0% (N=100)
        INFO:tensorflow:2018-05-09 20:38:25.148167: Step 3960: Train accuracy = 51.0%
        INFO:tensorflow:2018-05-09 20:38:25.148296: Step 3960: Cross entropy = 1.320472
        INFO:tensorflow:2018-05-09 20:38:25.187060: Step 3960: Validation accuracy = 44.0% (N=100)
        INFO:tensorflow:2018-05-09 20:38:25.584200: Step 3970: Train accuracy = 51.0%
        INFO:tensorflow:2018-05-09 20:38:25.584305: Step 3970: Cross entropy = 1.347709
        INFO:tensorflow:2018-05-09 20:38:25.623482: Step 3970: Validation accuracy = 55.0% (N=100)
        INFO:tensorflow:2018-05-09 20:38:26.021101: Step 3980: Train accuracy = 49.0%
        INFO:tensorflow:2018-05-09 20:38:26.021240: Step 3980: Cross entropy = 1.375533
        INFO:tensorflow:2018-05-09 20:38:26.076670: Step 3980: Validation accuracy = 49.0% (N=100)
        INFO:tensorflow:2018-05-09 20:38:26.482145: Step 3990: Train accuracy = 46.0%
        INFO:tensorflow:2018-05-09 20:38:26.482256: Step 3990: Cross entropy = 1.417852
        INFO:tensorflow:2018-05-09 20:38:26.521648: Step 3990: Validation accuracy = 47.0% (N=100)
        INFO:tensorflow:2018-05-09 20:38:26.884866: Step 3999: Train accuracy = 44.0%
        INFO:tensorflow:2018-05-09 20:38:26.884979: Step 3999: Cross entropy = 1.445659
        INFO:tensorflow:2018-05-09 20:38:26.925073: Step 3999: Validation accuracy = 45.0% (N=100)
        INFO:tensorflow:Final test accuracy = 41.5% (N=3619)
        INFO:tensorflow:Froze 2 variables.
        Converted 2 variables to const ops.

          1. Thank you, If you dont mind give your mail ID to ask questions on other different topics. Because im started learning machine learning

  4. Hi blockposter thank you for sharing the post.

    Fec2013 file not showing any image cannot open at all after download.

    First need download opencv or not? to read, resize, convert grayscale
    Need install numpy?

    Keras or tensor flow need to install?
    Keras is one lib that inside tensor flow?

    What to start first? I view many webpage and github code.
    All get me confuse, i really not knowing how to start the project at all,
    spending many days to search but not understand.
    I am new in python but need to do my FYP project TT.

    Before running your code need to apply transfer learning and use pre-trained weights?
    Wht to do to apply transfer learning and use pre-trained weights?
    running the code that given then run your code?

    Pre-trained weights (facial_expression_model_weights.h5) file to be download for?
    After click and dowload cannot be open.

    1. Hello,

      1- To read Fec2013, you need to install numpy but you do not have to install OpenCV.
      2- Yes, you must install keras and tensorflow because in this post keras code pushed
      3- Please follow steps mentioned only in this post. If something confuse you, then please contact.
      4- It depends. I recommend you to train the dataset instead of applying transfer learning
      5- Once, you trained a network and understand how system works, you might apply tarnsfer learning. And yes, facial_expression_model_weights.h5 refers to pre-trained weights. Would you try different browser? I can download it in chrome.

      1. Thank you a lot, really. I hv installed the numpy, keras, n dowload the fer2013 file n covert it to .csv. However, the facial_expression_model_weights.h5 i am able to download in chrome but unable to open it. Which apps u use to open/view it or just direct download and apply only.

        Beisdes, there is problem when i copy yr code until line:

        for i in range(1,num_of_instances):
        try:
        emotion, img, usage = lines[i].split(“,”)

        val = img.split(” “)
        pixels = np.array(val, ‘float32’)

        emotion = keras.utils.to_categorical(emotion, num_classes)

        if ‘Training’ in usage:
        y_train.append(emotion)
        x_train.append(pixels)
        elif ‘PublicTest’ in usage:
        y_test.append(emotion)
        x_test.append(pixels)
        except:
        print(end=” +”)

        It direct jump to except statement print out ++++++

        Output:
        no of instances: 35918
        instance length: 2304
        ++++++++++++++++

        Or I need to direct copy all the codes from start to end and just put in.

        1. Downloading facial_expression_model_weights.h5 is enough. You don’t have to open it. It’s a binary file. Your python code will consume it.

          Jumping except block would not be a problem. It seems you can read valid lines in fec2013 because you can dump num of instances.

          1. Thank you very much. Isn’t over fitting means the test accuracy is lower than train accuracy?

            For epoch 5 I get the result of:
            Train loss: 1.103590385833875
            Train accuracy: 58.18036155919678

            Test loss: 1.2256663155509224
            Test accuracy: 53.552521594173896

            For epoch 7:
            Train loss: 1.0120716843428483
            Train accuracy: 62.001462955971306

            Test loss: 1.1952817713726063
            Test accuracy: 54.332683201984096

            Is it over fitting, if is how can it solve?

          2. No, in your case, both train and test accuracy increase during epochs. If your train accuracy increases, meanwhile your test accuracy decreases, then this means that you fall into overfitting.

  5. Hello. Sefik Serengil
    Thanks for you wonderful tutorial.
    I get an error when trying to fit the model.
    Error when checking target: expected dense_33 to have 2 dimensions, but got array with shape (50, 1, 7)
    I tried to debug without success, what could be the cause

  6. Yes im using Tensorflow

    Layer (type) Output Shape Param #

    conv2d_91 (Conv2D) (None, 44, 44, 64) 1664

    max_pooling2d_19 (MaxPooling (None, 20, 20, 64) 0

    conv2d_92 (Conv2D) (None, 18, 18, 64) 36928

    conv2d_93 (Conv2D) (None, 16, 16, 64) 36928

    average_pooling2d_37 (Averag (None, 7, 7, 64) 0

    conv2d_94 (Conv2D) (None, 5, 5, 128) 73856

    conv2d_95 (Conv2D) (None, 3, 3, 128) 147584

    average_pooling2d_38 (Averag (None, 1, 1, 128) 0

    flatten_20 (Flatten) (None, 128) 0

    dense_46 (Dense) (None, 1024) 132096

    dropout_30 (Dropout) (None, 1024) 0

    dense_47 (Dense) (None, 1024) 1049600

    dropout_31 (Dropout) (None, 1024) 0

    dense_48 (Dense) (None, 7) 7175

  7. Im running the same code. The only difference,I used a smaller data set of 200 rows from the original dataset because of i have no GPU

    1. Actually, you do not have to have a GPU. model.fit_generator() command trains network randomly selected 256 instances instead of 60000. In this way, you could train the network in a short time. Would you please try to run exactly the same code?

  8. Model fit generator still failing.
    Error when checking target: expected dense_3 to have 2 dimensions, but got array with shape (256, 1, 7)

    I’m using the same code without any change

    1. You might have a trouble because of your keras version. My keras version is 2.1.5, and tensorflow 1.6.0.


      import keras
      import tensorflow as tf
      print("keras: ",keras.__version__)
      print("tensorflow: ",tf.__version__)

      Would you try to update your keras if you are using lower version by running the command
      pip install git+git://github.com/fchollet/keras.git –upgrade –no-deps

    1. I add confusion matrix section in the post. That section includes both results and how to produce it in python. Thank you for your question.

  9. Thanks a lot Sefik Serengil for such a beautiful work. The code you have given plots the emotion versus percentage. What should we do if we want to find the emotion with the highest percentage?

    1. You can just change emotion analysis function as illustrated below

      def emotion_analysis(emotions):
      print(objects[np.argmax(emotions)])

  10. This is a very well documented and clean code from tutorial point of view.
    I still had 1 doubt though,

    why do you have to convert the ’emotion’ values to categorical when it actually already had values as 0,1,2,3,.. (i.e. categorical values) ??

    Also when I skip the line `emotion = keras.utils.to_categorical(emotion, num_classes)` there’s actually an error as

    `Error when checking target: expected dense_3 to have shape (7,) but got array with shape (1,)`

    So I figured its actually needed but couldn’t understand the need as to why, when the values are already categorical??

    Please explain. Thanks

    1. We have emotions 0 to 6 but we would like to classify emotion of an image. I mean that if we would not change these labels as 0 to 6, then our problem would be regression problem (left side on the image). Yes, in this case, neural networks can handle but it will predict continuous outputs. But now network would produce decimal outputs from 0 to 6. What would emotion be if output is 5.5?

      That’s why, we would like to define problem as classification problem (right side on the image). In this case, network would produce one of outputs as 1, and rest of outputs as 0. You can think that output nodes are like bulb. Only one bulb would flash on. And, emotion would be the index of bulb flashed on.

      Regression vs Classification

      Still, you can define this problem as regression problem to understand. In this case, deactivate keras.utils.to_categorical command first. Then, change number of outputs of the model as model.add(Dense(1, activation=’softmax’)). This modification would handle the error you have faced with.

      I hope this explanation is understandable.

  11. Hi Serengil,

    Am trying to execute this code and am getting very less accuracy(20%). It is taking so much of time to execute this in windows. New to machine learning please help me with this. Below is the console data obtained.

    2018-08-01 09:13:20.260051: W T:\src\github\tensorflow\tensorflow\core\framework\allocator.cc:108] Allocation of 126877696 exceeds 10% of system memory.

    2/256 […………………………] – ETA: 1:06:24 – loss: 1.9237 – acc: 0.23052018-08-01 09:13:21.361092: W T:\src\github\tensorflow\tensorflow\core\framework\allocator.cc:108] Allocation of 126877696 exceeds 10% of system memory.

    3/256 […………………………] – ETA: 1:05:05 – loss: 1.8963 – acc: 0.2396
    4/256 […………………………] – ETA: 1:03:17 – loss: 1.8855 – acc: 0.2432
    5/256 […………………………] – ETA: 1:02:36 – loss: 1.8748 – acc: 0.2305
    6/256 […………………………] – ETA: 1:02:53 – loss: 1.8663 – acc: 0.2240
    7/256 […………………………] – ETA: 1:02:29 – loss: 1.8629 – acc: 0.2176

    1. Your loss seems decreasing in every step. This means that you are correct way. But this operation requires high computation power. I have run this on a GPU. You have to wait until epochs completed.

  12. Thank you for the reply.

    keeping batch_size = 50
    epochs = 1 am getting
    Train loss: 1.8260875169359456
    Train accuracy: 25.13149186666202
    Test loss: 1.8270909512531261
    Test accuracy: 24.937308442878273

    Keeping batch_size to 256 is also giving me the same result. All the custom images results are misclassified to happy. Please let me know where am going wrong.

    1. If you’re using the same CNN as the one above, I would really increase the epochs size, as 1 is very small and would result in a lower accuracy. I have managed to get a 57% model with a different architecture by increasing epochs to 35, and batch size to 125. As long as your test accuracy increase with training accuracy, there is no overfitting problem

  13. Thank you for your guidance but I can’t work on the confusion matrix. May i know what is the prediction variable stand for? I view the origin code at github, it state predictions = model.predict(x_test), but it not works. It comes out error of and the data content in predList is suspicious only 0. I try out other code too for import panda but also fail it display only one line in predList with 0 output.

        1. Please update both your keras and tensorflow version. I put my environment information below.

          python –version
          Python 3.5.5 :: Anaconda, Inc.

          >>> import tensorflow
          >>> print(tensorflow.__version__)
          1.9.0
          >>> import keras
          Using TensorFlow backend.
          >>> print(keras.__version__)
          2.2.0

    1. Sure, keras provides VGG model as an out-of-the-box function. You might add some additional neural networks layers to cover these facial expressions.

  14. i ran the code from github but i am getting this error,
    ValueError Traceback (most recent call last)
    in ()
    42 #——————————
    43 #data transformation for train and test sets
    —> 44 x_train = np.array(x_train, ‘float32’)
    45 y_train = np.array(y_train, ‘float32’)
    46 x_test = np.array(x_test, ‘float32’)

    ValueError: setting an array element with a sequence.

    1. Please confirm your environment version. I run the code for the following versions. If you still have a trouble for these versions, then please contact me again.

      python –version
      Python 3.5.5 :: Anaconda, Inc.

      >>> import tensorflow
      >>> print(tensorflow.__version__)
      1.9.0
      >>> import keras
      Using TensorFlow backend.
      >>> print(keras.__version__)
      2.2.0

    1. Please share the error message. Before, could you confirm your environment is same as mine. Python 3.5.5, Tensorflow 1.9.0 and Keras 2.2.0

  15. Hello.I ran the same code as u provided on github but with 10 epochs.I got 66% trining accurqcy and 56% test accuracy.But the predictions on random images is very absurd.Even the predictions using webcam are very absurd.Is this an environment/version issue?
    python version-3.5
    keras-2.2.0
    tf version=1.11.0

  16. I ran the same code as u provided on github but with 10 epochs.I got 66% trining accurqcy and 56% test accuracy.But the predictions on random images is very absurd.Even the predictions using webcam are very absurd.Is this an environment/version issue?
    python version-3.5
    keras-2.2.0
    tf version=1.11.0

  17. I ran the same code as u provided on github but with 10 epochs.I got 66% trining accurqcy and 56% test accuracy.But the predictions on random images is very absurd.Even the predictions using webcam are very absurd.Is this an environment/version issue?
    python version-3.5
    keras-2.2.0
    tf version=1.11.0

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.