Kaggle announced facial expression recognition challenge in 2013. Researchers are expected to create models to detect 7 different emotions from human being faces. However, recent studies are far away from the excellent results even today. That’s why, this topic is still satisfying subject.
Dataset
The both training and evaluation operations would be handled with Fec2013 dataset. Compressed version of the dataset takes 92 MB space whereas uncompressed version takes 295 MB space. There are 28K training and 3K testing images in the dataset. Each image was stored as 48×48 pixel. The pure dataset consists of image pixels (48×48=2304 values), emotion of each image and usage type (as train or test instance).
🙋♂️ You may consider to enroll my top-rated machine learning course on Udemy
Suppose that the dataset is already loaded under the data folder. Herein, we can read the dataset content as mentioned below.
with open("/data/fer2013.csv") as f: content = f.readlines() lines = np.array(content) num_of_instances = lines.size print("number of instances: ",num_of_instances)
Learning Procedure
Deep learning dominates computer vision studies in recent years. Even academic computer vision conferences are closely transformed into Deep Learning activities. Herein, we would apply convolutional neural networks to tackle this task. And we will construct CNN with Keras using TensorFlow backend.
We’ve already loaded the dataset before. Now, train and test set can be stored into dedicated variables.
x_train, y_train, x_test, y_test = [], [], [], [] for i in range(1,num_of_instances): try: emotion, img, usage = lines[i].split(",") val = img.split(" ") pixels = np.array(val, 'float32') emotion = keras.utils.to_categorical(emotion, num_classes) if 'Training' in usage: y_train.append(emotion) x_train.append(pixels) elif 'PublicTest' in usage: y_test.append(emotion) x_test.append(pixels) except: print("", end="")
Time to construct CNN structure.
model = Sequential() #1st convolution layer model.add(Conv2D(64, (5, 5), activation='relu', input_shape=(48,48,1))) model.add(MaxPooling2D(pool_size=(5,5), strides=(2, 2))) #2nd convolution layer model.add(Conv2D(64, (3, 3), activation='relu')) model.add(Conv2D(64, (3, 3), activation='relu')) model.add(AveragePooling2D(pool_size=(3,3), strides=(2, 2))) #3rd convolution layer model.add(Conv2D(128, (3, 3), activation='relu')) model.add(Conv2D(128, (3, 3), activation='relu')) model.add(AveragePooling2D(pool_size=(3,3), strides=(2, 2))) model.add(Flatten()) #fully connected neural networks model.add(Dense(1024, activation='relu')) model.add(Dropout(0.2)) model.add(Dense(1024, activation='relu')) model.add(Dropout(0.2)) model.add(Dense(num_classes, activation='softmax'))
We can train the network. To complete the training in less time, I prefer to implement learning with randomly selected trainset instances. That is the reason why train and fit generator used. Also, loss function would be cross entropy because the task is multi class classification.
gen = ImageDataGenerator() train_generator = gen.flow(x_train, y_train, batch_size=batch_size) model.compile(loss='categorical_crossentropy' , optimizer=keras.optimizers.Adam() , metrics=['accuracy'] ) model.fit_generator(train_generator, steps_per_epoch=batch_size, epochs=epochs)
Fit is over. We can evaluate the network.
train_score = model.evaluate(x_train, y_train, verbose=0) print('Train loss:', train_score[0]) print('Train accuracy:', 100*train_score[1]) test_score = model.evaluate(x_test, y_test, verbose=0) print('Test loss:', test_score[0]) print('Test accuracy:', 100*test_score[1])
I’ve got the following results not to fall into overfitting. I faced with overfitting when I increase the epoch.
Test loss: 2.27945706329 Test accuracy: 57.4254667071 Train loss: 0.223031098232 Train accuracy: 92.0512731201
Confusion Matrix
Sure, accuracy should not express right impression for multi class classification problems. Confusion matrix of this model is demonstrated below. Lines represent actual values whereas columns state predictions. I mean that there are 467 angry instances in testset. We can classify 214 angry items correctly. On the other hand, we classified 9 items as disgust but these items are actual angry ones.
Angry | Disgust | Fear | Happy | Sad | Surprise | Neutral | |
Angry | 214 | 9 | 53 | 30 | 67 | 8 | 86 |
Disgust | 10 | 24 | 9 | 2 | 6 | 0 | 5 |
Fear | 45 | 2 | 208 | 29 | 89 | 45 | 78 |
Happy | 24 | 0 | 40 | 696 | 37 | 18 | 80 |
Sad | 65 | 3 | 83 | 56 | 285 | 10 | 151 |
Surprise | 7 | 1 | 42 | 27 | 9 | 303 | 26 |
Neutral | 45 | 2 | 68 | 65 | 88 | 8 | 331 |
Basically, scikit-learn produces that confusion matrix.
from sklearn.metrics import classification_report, confusion_matrix pred_list = []; actual_list = [] for i in predictions: pred_list.append(np.argmax(i)) for i in y_test: actual_list.append(np.argmax(i)) confusion_matrix(actual_list, pred_list)
Face detection
Images are already cropped and just facial area are focused on in the train set. This is not a must but we should detect faces of the custom testing images and feed just facial areas to the neural networks model. This will increase the accuracy dramatically.
There are several face detection solutions. OpenCV offers haar cascade and single shot multibox detector (SSD). Dlib offers Histogram of Oriented Gradients (HOG) and Max-Margin Object Detection (MMOD). Finally Multi-task Cascaded Convolutional Networks (MTCNN) is a common solution for face detection. Herein, haar cascade and HoG are legacy methods whereas SSD, MMOD and MTCNN are deep learning based modern solutions. You can see the detection performance of those models in the following video.
Here, you can watch how to use different face detectors in Python.
Here, retinaface is the cutting-edge face detection technology. It can even detect faces in the crowd and it finds facial landmarks including eye coordinates. That’s why, its alignment score is very high.
Testing
Let’s try to recognize facial expressions of custom images. Because only error rates don’t express anything.
img = image.load_img("/data/pablo.png", grayscale=True, target_size=(48, 48)) x = image.img_to_array(img) x = np.expand_dims(x, axis = 0) x /= 255 custom = model.predict(x) emotion_analysis(custom[0]) x = np.array(x, 'float32') x = x.reshape([48, 48]); plt.gray() plt.imshow(x) plt.show()
Emotions stored as numerical as labeled from 0 to 6. Keras would produce an output array including these 7 different emotion scores. We can visualize each prediction as bar chart.
def emotion_analysis(emotions): objects = ('angry', 'disgust', 'fear', 'happy', 'sad', 'surprise', 'neutral') y_pos = np.arange(len(objects)) plt.bar(y_pos, emotions, align='center', alpha=0.5) plt.xticks(y_pos, objects) plt.ylabel('percentage') plt.title('emotion') plt.show()
If you watch the famous Netflix series Narcos, then you would be familiar with the following picture. The following picture of Pablo Escobar is taken in a police station when he was taken into custody. It seems that the model we’ve constructed can successfully recognize Pablo in happy mood.
Secondly, we will test the scene of Marlon Brando acting in Godfather as Don Corleone. Corleone cries at dead body of his son’s elbow. It seems that the model can recognize Brando’s facial expression, too.
What’s more, Hugh Jackman comes to my mind as always angry figure. That’s why, I would like to test him. Especially, I choose a picture of Jackman from X-Men as Wolverine. Result seems very successful.
Finally, art authorities still cannot come to mutual agreement for Mona Lisa’s emotion. Network says that Mona Lisa is in neutral mood.
Real time solution
Besides, we can apply emotion analysis on a video streaming or web cam capturing. I’ve written a dedicated blog post about this subject. Its demo is shown below.
Web cam
Me and my colleagues try to act all emotion classes. As seen, this implementation runs very fast.
Video streaming
Remember the testimony of Mark Zuckerberg after Cambridge Analytica scandal. Facebook lost 134 billion dollar after this news. This makes unhappy anyone just like Mark as detected below.
Conclusion
So, we’ve constructed a CNN model to recognize facial expressions of human beings. Model produces 57% accuracy on test set. That can be acceptable because winner of kaggle challenge has got 34% accuracy.
Processing detected faces instead of the entire image would increase accuracy. That’s a little trick. I crop the faces manually before running network.
The entire code of the project is pushed on GitHub. Also, you might want to apply transfer learning and use pre-trained weights. Pre-trained weights and pre-constructed network structure are pushed on GitHub, too.
This post covers my custom design for facial expression recognition task. I can improve the accuracy from 57% to 66% with Auto-Keras for the same task.
If you interested in this post, you might be interested in deep face recognition.
Python library
Herein, deepface is a lightweight facial analysis framework covering both face recognition and demography such as age, gender, race and emotion. If you are not interested in building neural networks models from scratch, then you might adopt deepface. It is fully open-source and available on PyPI. You can make predictions with a few lines of code. It also supports real-time implementations as well.
Here, you can watch a how to apply facial attribute analysis in python with a just few lines of code.
You can run deepface in real time with your web cam as well.
Like this blog? Support me on Patreon
Hi,blogposter.Thanks a lot for your wonderful sharing . It help me a lot .But I am haveing some problem when i tried to run facial-expression-recognition.py scripts in your project. I have downloaded the whole project from your github account.
when read the data from the fer2013.csv I got the below error
Traceback (most recent call last):
File “C:/Users/liuxiongcheng/PycharmProjects/untitled/Facial_Expression_Recognition/facial-expression-recognition.py”, line 49, in
pixels = np.array(val, ‘float32’)
ValueError: could not convert string to float:
Hello, it seems that content of Fec2013 file is changed. Additional cite and reference lines added into the raw file. We can handle this with adding try – catch mechanism. Would you change the code as illustrated below?
for i in range(1,num_of_instances):
try:
emotion, img, usage = lines[i].split(“,”)
val = img.split(” “)
pixels = np.array(val, ‘float32’)
emotion = keras.utils.to_categorical(emotion, num_classes)
if ‘Training’ in usage:
y_train.append(emotion)
x_train.append(pixels)
elif ‘PublicTest’ in usage:
y_test.append(emotion)
x_test.append(pixels)
except:
print(“”, end=””)
Thanks for sharing your experience, but I wanted to know your view on training the inception v3 for facial expression recognition (I mean not loading the pre-trained weights and training the inception v3 from scratch).
Training inception for emotion analysis would most probably produce much more successful results than the design in this post. But, inception V3 has very complex structure. That’s why, you must implement learning with highly costly GPUs for a long time.
Hi Serengil,
I have Nvidia 1080 GTX GPU so no worry for me. I have seen the google code https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/ for training the inception model but i only go the 422% test accuracy. Below are the steps i followed
1. First converted the FER2013 in to jpg images with emotion types as directory.
2. Next cloned the tensorflow-for-poests-2 github project and retrained
Below are the few lines
INFO:tensorflow:2018-05-09 20:38:24.267195: Step 3940: Cross entropy = 1.316952
INFO:tensorflow:2018-05-09 20:38:24.305648: Step 3940: Validation accuracy = 40.0% (N=100)
INFO:tensorflow:2018-05-09 20:38:24.709390: Step 3950: Train accuracy = 52.0%
INFO:tensorflow:2018-05-09 20:38:24.709500: Step 3950: Cross entropy = 1.369833
INFO:tensorflow:2018-05-09 20:38:24.749251: Step 3950: Validation accuracy = 40.0% (N=100)
INFO:tensorflow:2018-05-09 20:38:25.148167: Step 3960: Train accuracy = 51.0%
INFO:tensorflow:2018-05-09 20:38:25.148296: Step 3960: Cross entropy = 1.320472
INFO:tensorflow:2018-05-09 20:38:25.187060: Step 3960: Validation accuracy = 44.0% (N=100)
INFO:tensorflow:2018-05-09 20:38:25.584200: Step 3970: Train accuracy = 51.0%
INFO:tensorflow:2018-05-09 20:38:25.584305: Step 3970: Cross entropy = 1.347709
INFO:tensorflow:2018-05-09 20:38:25.623482: Step 3970: Validation accuracy = 55.0% (N=100)
INFO:tensorflow:2018-05-09 20:38:26.021101: Step 3980: Train accuracy = 49.0%
INFO:tensorflow:2018-05-09 20:38:26.021240: Step 3980: Cross entropy = 1.375533
INFO:tensorflow:2018-05-09 20:38:26.076670: Step 3980: Validation accuracy = 49.0% (N=100)
INFO:tensorflow:2018-05-09 20:38:26.482145: Step 3990: Train accuracy = 46.0%
INFO:tensorflow:2018-05-09 20:38:26.482256: Step 3990: Cross entropy = 1.417852
INFO:tensorflow:2018-05-09 20:38:26.521648: Step 3990: Validation accuracy = 47.0% (N=100)
INFO:tensorflow:2018-05-09 20:38:26.884866: Step 3999: Train accuracy = 44.0%
INFO:tensorflow:2018-05-09 20:38:26.884979: Step 3999: Cross entropy = 1.445659
INFO:tensorflow:2018-05-09 20:38:26.925073: Step 3999: Validation accuracy = 45.0% (N=100)
INFO:tensorflow:Final test accuracy = 41.5% (N=3619)
INFO:tensorflow:Froze 2 variables.
Converted 2 variables to const ops.
Well but I actually do not know what tensorflow for poets do. Besides, I haven’t train inception model for a custom problem. But I remember style transfer topic consumes similar network VGG. You might find it from https://github.com/keras-team/keras/blob/master/examples/neural_style_transfer.py .7
In parallel, I will research on this topic and publish it as a post i this blog.
Thank you, If you dont mind give your mail ID to ask questions on other different topics. Because im started learning machine learning
You can find my mail in About-Me section.
Hi blockposter thank you for sharing the post.
Fec2013 file not showing any image cannot open at all after download.
First need download opencv or not? to read, resize, convert grayscale
Need install numpy?
Keras or tensor flow need to install?
Keras is one lib that inside tensor flow?
What to start first? I view many webpage and github code.
All get me confuse, i really not knowing how to start the project at all,
spending many days to search but not understand.
I am new in python but need to do my FYP project TT.
Before running your code need to apply transfer learning and use pre-trained weights?
Wht to do to apply transfer learning and use pre-trained weights?
running the code that given then run your code?
Pre-trained weights (facial_expression_model_weights.h5) file to be download for?
After click and dowload cannot be open.
Hello,
1- To read Fec2013, you need to install numpy but you do not have to install OpenCV.
2- Yes, you must install keras and tensorflow because in this post keras code pushed
3- Please follow steps mentioned only in this post. If something confuse you, then please contact.
4- It depends. I recommend you to train the dataset instead of applying transfer learning
5- Once, you trained a network and understand how system works, you might apply tarnsfer learning. And yes, facial_expression_model_weights.h5 refers to pre-trained weights. Would you try different browser? I can download it in chrome.
Thank you a lot, really. I hv installed the numpy, keras, n dowload the fer2013 file n covert it to .csv. However, the facial_expression_model_weights.h5 i am able to download in chrome but unable to open it. Which apps u use to open/view it or just direct download and apply only.
Beisdes, there is problem when i copy yr code until line:
for i in range(1,num_of_instances):
try:
emotion, img, usage = lines[i].split(“,”)
val = img.split(” “)
pixels = np.array(val, ‘float32’)
emotion = keras.utils.to_categorical(emotion, num_classes)
if ‘Training’ in usage:
y_train.append(emotion)
x_train.append(pixels)
elif ‘PublicTest’ in usage:
y_test.append(emotion)
x_test.append(pixels)
except:
print(end=” +”)
It direct jump to except statement print out ++++++
Output:
no of instances: 35918
instance length: 2304
++++++++++++++++
Or I need to direct copy all the codes from start to end and just put in.
Downloading facial_expression_model_weights.h5 is enough. You don’t have to open it. It’s a binary file. Your python code will consume it.
Jumping except block would not be a problem. It seems you can read valid lines in fec2013 because you can dump num of instances.
Thank you very much. Isn’t over fitting means the test accuracy is lower than train accuracy?
For epoch 5 I get the result of:
Train loss: 1.103590385833875
Train accuracy: 58.18036155919678
Test loss: 1.2256663155509224
Test accuracy: 53.552521594173896
For epoch 7:
Train loss: 1.0120716843428483
Train accuracy: 62.001462955971306
Test loss: 1.1952817713726063
Test accuracy: 54.332683201984096
Is it over fitting, if is how can it solve?
No, in your case, both train and test accuracy increase during epochs. If your train accuracy increases, meanwhile your test accuracy decreases, then this means that you fall into overfitting.
I have the same problem described below here
the training accuracy is higher than the testing accuracy , I supposed that it’s overfitting.
how can I solve that ?
I tend to use earlystopping and modelcheckpoint to avoid overfitting. ModelCheckpoint saves the weights for the best epoch based on validation loss whereas earlystopping terminates training if validation loss wouldn’t decrease for 200 epochs. You should pass both earlystopping and modelcheckpoint to fit command as callbacks parameters as illustrated below.
from keras.callbacks import ModelCheckpoint,EarlyStopping
checkpointer = ModelCheckpoint(
filepath=’model.hdf5′
, monitor = “val_loss”
, verbose=1
, save_best_only=True
, mode = ‘auto’
)
eStop = EarlyStopping(monitor=’val_loss’
, patience=200
, verbose=1)
model.fit(
train_x, train_y
, epochs = 5000
, verbose = 1
, validation_data = (test_x, test_y)
, callbacks = [eStop, checkpointer]
)
BTW, if you still want to apply transfer learning for this case, I recommend you to read this blog post: https://sefiks.com/2019/02/13/apparent-age-and-gender-prediction-in-keras/
The trick is that you lock earlier layers and don’t update weights of these layers. In this way, you have the outcome of pre-trained models. For example, early layers of the inception model is responsible for detecting edges. Locking early layers provides you to detect edges to. The last 3 or 4 layers are free to update weights. In this way, inception model customized for your custom problem.
Hello. Sefik Serengil
Thanks for you wonderful tutorial.
I get an error when trying to fit the model.
Error when checking target: expected dense_33 to have 2 dimensions, but got array with shape (50, 1, 7)
I tried to debug without success, what could be the cause
You are using TensorFlow backend? And would you run model.summary() command share the result?
Yes im using Tensorflow
Layer (type) Output Shape Param #
conv2d_91 (Conv2D) (None, 44, 44, 64) 1664
max_pooling2d_19 (MaxPooling (None, 20, 20, 64) 0
conv2d_92 (Conv2D) (None, 18, 18, 64) 36928
conv2d_93 (Conv2D) (None, 16, 16, 64) 36928
average_pooling2d_37 (Averag (None, 7, 7, 64) 0
conv2d_94 (Conv2D) (None, 5, 5, 128) 73856
conv2d_95 (Conv2D) (None, 3, 3, 128) 147584
average_pooling2d_38 (Averag (None, 1, 1, 128) 0
flatten_20 (Flatten) (None, 128) 0
dense_46 (Dense) (None, 1024) 132096
dropout_30 (Dropout) (None, 1024) 0
dense_47 (Dense) (None, 1024) 1049600
dropout_31 (Dropout) (None, 1024) 0
dense_48 (Dense) (None, 7) 7175
Would you confirm that you are running the same code in the following repository: https://github.com/serengil/tensorflow-101/blob/master/python/facial-expression-recognition.py
You might put your code in a GitHub repo and share link if you still have a trouble.
Im running the same code. The only difference,I used a smaller data set of 200 rows from the original dataset because of i have no GPU
Actually, you do not have to have a GPU. model.fit_generator() command trains network randomly selected 256 instances instead of 60000. In this way, you could train the network in a short time. Would you please try to run exactly the same code?
Model fit generator still failing.
Error when checking target: expected dense_3 to have 2 dimensions, but got array with shape (256, 1, 7)
I’m using the same code without any change
You might have a trouble because of your keras version. My keras version is 2.1.5, and tensorflow 1.6.0.
import keras
import tensorflow as tf
print("keras: ",keras.__version__)
print("tensorflow: ",tf.__version__)
Would you try to update your keras if you are using lower version by running the command
pip install git+git://github.com/fchollet/keras.git –upgrade –no-deps
That was the problem. I was using old versions of keras and tensorflow. Thank you very much
Hello, can you provide a function to display the confusion matrix for this study? Thank you.
I add confusion matrix section in the post. That section includes both results and how to produce it in python. Thank you for your question.
Thanks a lot Sefik Serengil for such a beautiful work. The code you have given plots the emotion versus percentage. What should we do if we want to find the emotion with the highest percentage?
You can just change emotion analysis function as illustrated below
def emotion_analysis(emotions):
print(objects[np.argmax(emotions)])
This is a very well documented and clean code from tutorial point of view.
I still had 1 doubt though,
why do you have to convert the ’emotion’ values to categorical when it actually already had values as 0,1,2,3,.. (i.e. categorical values) ??
Also when I skip the line `emotion = keras.utils.to_categorical(emotion, num_classes)` there’s actually an error as
`Error when checking target: expected dense_3 to have shape (7,) but got array with shape (1,)`
So I figured its actually needed but couldn’t understand the need as to why, when the values are already categorical??
Please explain. Thanks
We have emotions 0 to 6 but we would like to classify emotion of an image. I mean that if we would not change these labels as 0 to 6, then our problem would be regression problem (left side on the image). Yes, in this case, neural networks can handle but it will predict continuous outputs. But now network would produce decimal outputs from 0 to 6. What would emotion be if output is 5.5?
That’s why, we would like to define problem as classification problem (right side on the image). In this case, network would produce one of outputs as 1, and rest of outputs as 0. You can think that output nodes are like bulb. Only one bulb would flash on. And, emotion would be the index of bulb flashed on.
Still, you can define this problem as regression problem to understand. In this case, deactivate keras.utils.to_categorical command first. Then, change number of outputs of the model as model.add(Dense(1, activation=’softmax’)). This modification would handle the error you have faced with.
I hope this explanation is understandable.
Hi Serengil,
Am trying to execute this code and am getting very less accuracy(20%). It is taking so much of time to execute this in windows. New to machine learning please help me with this. Below is the console data obtained.
2018-08-01 09:13:20.260051: W T:\src\github\tensorflow\tensorflow\core\framework\allocator.cc:108] Allocation of 126877696 exceeds 10% of system memory.
2/256 […………………………] – ETA: 1:06:24 – loss: 1.9237 – acc: 0.23052018-08-01 09:13:21.361092: W T:\src\github\tensorflow\tensorflow\core\framework\allocator.cc:108] Allocation of 126877696 exceeds 10% of system memory.
3/256 […………………………] – ETA: 1:05:05 – loss: 1.8963 – acc: 0.2396
4/256 […………………………] – ETA: 1:03:17 – loss: 1.8855 – acc: 0.2432
5/256 […………………………] – ETA: 1:02:36 – loss: 1.8748 – acc: 0.2305
6/256 […………………………] – ETA: 1:02:53 – loss: 1.8663 – acc: 0.2240
7/256 […………………………] – ETA: 1:02:29 – loss: 1.8629 – acc: 0.2176
Your loss seems decreasing in every step. This means that you are correct way. But this operation requires high computation power. I have run this on a GPU. You have to wait until epochs completed.
Thank you for the reply.
keeping batch_size = 50
epochs = 1 am getting
Train loss: 1.8260875169359456
Train accuracy: 25.13149186666202
Test loss: 1.8270909512531261
Test accuracy: 24.937308442878273
Keeping batch_size to 256 is also giving me the same result. All the custom images results are misclassified to happy. Please let me know where am going wrong.
If you’re using the same CNN as the one above, I would really increase the epochs size, as 1 is very small and would result in a lower accuracy. I have managed to get a 57% model with a different architecture by increasing epochs to 35, and batch size to 125. As long as your test accuracy increase with training accuracy, there is no overfitting problem
Thank you for your guidance but I can’t work on the confusion matrix. May i know what is the prediction variable stand for? I view the origin code at github, it state predictions = model.predict(x_test), but it not works. It comes out error of and the data content in predList is suspicious only 0. I try out other code too for import panda but also fail it display only one line in predList with 0 output.
We may not work on same tf / keras version. What is your current version and error message?
my current version is Anaconda3 v 4.3.1, keras v1.80, error message: function confusion matrix at 0x000001E7B391D048
Please update both your keras and tensorflow version. I put my environment information below.
python –version
Python 3.5.5 :: Anaconda, Inc.
>>> import tensorflow
>>> print(tensorflow.__version__)
1.9.0
>>> import keras
Using TensorFlow backend.
>>> print(keras.__version__)
2.2.0
i type wrong is tf 1.8.0 ,keras 2.1.6
Can you fine-tune VGG16 for this task?
Sure, keras provides VGG model as an out-of-the-box function. You might add some additional neural networks layers to cover these facial expressions.
i ran the code from github but i am getting this error,
ValueError Traceback (most recent call last)
in ()
42 #——————————
43 #data transformation for train and test sets
—> 44 x_train = np.array(x_train, ‘float32’)
45 y_train = np.array(y_train, ‘float32’)
46 x_test = np.array(x_test, ‘float32’)
ValueError: setting an array element with a sequence.
Please confirm your environment version. I run the code for the following versions. If you still have a trouble for these versions, then please contact me again.
python –version
Python 3.5.5 :: Anaconda, Inc.
>>> import tensorflow
>>> print(tensorflow.__version__)
1.9.0
>>> import keras
Using TensorFlow backend.
>>> print(keras.__version__)
2.2.0
im getting this error when i execute the train_generator line. can you help in fixing this
Please share the error message. Before, could you confirm your environment is same as mine. Python 3.5.5, Tensorflow 1.9.0 and Keras 2.2.0
My error
https://prnt.sc/lc3rml
Python: 3.5
Tensorflow: 1.11.0
Keras: 2.2.4
Can you share the whole code?
Can this code be tuned to work for multiple faces in a video ?
Yes, of course! I’ve already handled it. Please read this post: https://sefiks.com/2018/01/10/real-time-facial-expression-recognition-on-streaming-data/
https://pastebin.com/ZaW5ciJ4
its actually the same code shown on this website, still have a look. im having an issue while executing gen.flow() as shown in the picture, link https://prnt.sc/lc3rml
It seems that x_train and y_train are inconsistent. I cannot access the code repository you shared. You might put your code on GitHub.
BTW, you try to run the code from this repo: https://github.com/serengil/tensorflow-101/blob/master/python/facial-expression-recognition.py ? Or just copy the parts from the blog post?
i ran the github code as it is and it runs perfectly, the only problem is it doesnt predict input images correctly, always shows the wrong output.
image1: https://imgur.com/F2pl137 output: http://prntscr.com/ldh0qm
image2: http://gg.gg/cb68g output: http://prntscr.com/ldh26p
what does monitor_testset_results does? i havent executed that part.
You should try to extract face area first just like in this example: https://github.com/serengil/tensorflow-101/blob/master/dataset/pablo.png . You can use opencv’s face detection module for this duty. I’ve mentioned this topic in this post: https://sefiks.com/2018/01/10/real-time-facial-expression-recognition-on-streaming-data/ .
Hi Sefik ,
Is the deepface emotional analysis and the model you trained have the same accuracy ?
Or is deepface much better than the trained model?
They are totally same
Hello.I ran the same code as u provided on github but with 10 epochs.I got 66% trining accurqcy and 56% test accuracy.But the predictions on random images is very absurd.Even the predictions using webcam are very absurd.Is this an environment/version issue?
python version-3.5
keras-2.2.0
tf version=1.11.0
I do not think so. If you got 56% test accuracy, then you should get same accuracy on random images. BTW, another blog post exists for real time facial expression recognition on this blog. Face detected first and then emotion predicted. You should look that post.
Link: https://sefiks.com/2018/01/10/real-time-facial-expression-recognition-on-streaming-data/
I ran the same code as u provided on github but with 10 epochs.I got 66% trining accurqcy and 56% test accuracy.But the predictions on random images is very absurd.Even the predictions using webcam are very absurd.Is this an environment/version issue?
python version-3.5
keras-2.2.0
tf version=1.11.0
I’ve already replied this question. It seems that it repeats several times. To sum up, this post mentioned detected faces emotions but when you feed webcam capture you ask for undetected picture’s emotion. Please look at the following block post. In there, we detect faces on the captured image first and then ask for emotions.
https://sefiks.com/2018/01/10/real-time-facial-expression-recognition-on-streaming-data/
I ran the same code as u provided on github but with 10 epochs.I got 66% trining accurqcy and 56% test accuracy.But the predictions on random images is very absurd.Even the predictions using webcam are very absurd.Is this an environment/version issue?
python version-3.5
keras-2.2.0
tf version=1.11.0
I’ve already replied this question. It seems that it repeats several times. To sum up, this post mentioned detected faces emotions but when you feed webcam capture you ask for undetected picture’s emotion. Please look at the following block post. In there, we detect faces on the captured image first and then ask for emotions.
https://sefiks.com/2018/01/10/real-time-facial-expression-recognition-on-streaming-data/
lines=np.array(content)
MemoryError: Unable to allocate array with shape (35888,) and data type <U9230
How to resolve this error?
This is likely due to your system’s overcommit handling mode. Are there other processes running in memory?
how to print colored images in output while plotting graphs rather than greyscale
it plots it colorful by default as illustrated below. You just need to normalize it in [0, 1]. Dividing to 255 handles this normalization.
img = (test_df.iloc[i][‘pixels’].reshape([224, 224, 3])) / 255
plt.imshow(img)
plt.show()
Hi Sefik,
How can I convert the data into csv file ?
Which data?
The Fer data. but it’s okey the problem is solved.
Actually I’m having trouble with the code of Real Time Facial Expression Recognition on Streaming Data. it throws the error: ‘numpy.ndarray’ object has no attribute ‘img_to_array’
You should import “from keras.preprocessing import image” .That’s the whole code: https://github.com/serengil/tensorflow-101/blob/master/python/emotion-analysis-from-video.py
from sklearn.metrics import classification_report, confusion_matrix
pred_list = []; actual_list = []
for i in predictions:
pred_list.append(np.argmax(i))
for i in y_test:
actual_list.append(np.argmax(i))
confusion_matrix(actual_list, pred_list)
In this code what is the value of predictions? pls help me
It is what model.predict returns