We’ve recently studied OpenCV face recognition module. It wraps some legacy techniques. On the other hand, deep learning based models are state-of-the-art for face recognition tasks nowadays. We can build deep neural networks models in OpenCV with its dnn module as well. The library lets you to build external models for Tensorflow, Caffe, Torch, Darknet and ONNX. In this post, we are going to build OpenFace model within OpenCV to apply face recognition tasks. Besides, we will put opencv in the middle of a face recognition pipeline.
We’ve already built OpenFace model in Keras. Raw OpenFace model is built in Torch. Notice that it was not PyTorch. OpenCV dnn module offers a pretty interface to consume torch models directly.
🙋♂️ You may consider to enroll my top-rated machine learning course on Udemy
Pipeline
A modern face recognition pipeline consists of 4 common stages: detect, align, represent and verify.
Detect and align
Feeding detected faces increases the model accuracy dramatically. Besides, just alignment increases model accuracy 1% based on the Google FaceNet research. Luckily, deepface package can handle the both detection and alignment steps with a line of code.
from deepface.commons import functions img1_path = "img1.jpg" img2_path = "img2.jpg" img1 = functions.detectFace(img1_path, target_size=(96, 96))[0] img2 = functions.detectFace(img2_path, target_size=(96, 96))[0]
We resized image pair to 96×96 pixels which is the input size of OpenFace model.
On the other hand, face detection can be done with many solutions such as OpenCV, Dlib or MTCNN. OpenCV offers haar cascade, single shot multibox detector (SSD). Dlib offers Histogram of Oriented Gradients (HOG) and Max-Margin Object Detection (MMOD). Finally, MTCNN is a popular solution in the open source community as well. Herein, SSD, MMOD and MTCNN are modern deep learning based approaches whereas haar cascade and HoG are legacy methods. Besides, SSD is the fastest one. You can monitor the detection performance of those methods in the following video.
Here, you can watch how to use different face detectors in Python.
You can find out the math behind alignment more on the following video:
Besides, face detectors detect faces in a rectangle area. So, detected faces come with some noise such as background color. We can find 68 different landmarks of a face with dlib. In this way, we can get rid of any noise of a facial image.
Representation
OpenFace model will be responsible for the representation stage. We firstly need to build the CNN model. Here, you can find the pre-trained models of OpenFace. It has 4 different alternatives. We will use its default version – nn4.small2.v1 – because it gets the highest accuracy (92% accuracy and 97% AUC) among others.
import cv2 #4.3.0
model = cv2.dnn.readNetFromTorch("openface_nn4.small2.v1.t7")
Now, we will feed these processed pairs to model.
img1_blob = cv2.dnn.blobFromImage(img1) img2_blob = cv2.dnn.blobFromImage(img2) model.setInput(img1_blob) img1_representation = model.forward() model.setInput(img2_blob) img2_representation = model.forward()
OpenFace represents faces to 128 dimensional vectors.
Distance and similarity
We’ve represented faces to 128 dimensional vectors in the representation step. Now, we need to find the distance between vectors. The easiest ways are euclidean distance and cosine similarity.
def findCosineDistance(source_representation, test_representation): a = np.matmul(np.transpose(source_representation), test_representation) b = np.sum(np.multiply(source_representation, source_representation)) c = np.sum(np.multiply(test_representation, test_representation)) return 1 - (a / (np.sqrt(b) * np.sqrt(c))) def findEuclideanDistance(source_representation, test_representation): euclidean_distance = source_representation - test_representation euclidean_distance = np.sum(np.multiply(euclidean_distance, euclidean_distance)) euclidean_distance = np.sqrt(euclidean_distance) return euclidean_distance
Verification
We’ve represented faces as vectors and known how to compare the similarity of two vectors.
euclidean_distance = findEuclideanDistance(img1_representation[0], img2_representation[0]) cosine_distance = findCosineDistance(l2_normalize(img1_representation[0]), l2_normalize(img2_representation[0]))
We expect that face pairs of same person should have a low distance whereas face pairs of different person should have a high distance. We need to set a threshold here. My experiments show that euclidean distance should be 0.60 and cosine distance should be 0.25 as thresholds. To find out how to determine the best threshold value, you can read this blog post: Fine Tuning The Threshold in Face Recognition.
if euclidean_distance < 0.60: return True else: return False
Tests
Pipeline returns the following results for both positive and negative pairs based on the decided threshold.
Deepface package
We mentioned the all required stages of a face recognition pipeline and consume OpenCV for representation stage. This might be boring for programmers. Deepface offers you to run a face recognition pipeline with a few lines of code as well. It wraps the state-of-the-art face recognition models including VGG-Face, Google FaceNet,OpenFace, Facebook DeepFace and DeepID.
Large scale face recognition
We’ve actually mentioned face verification concept. I mean that we determine two face pairs are same person or not. Finding a face in a database requires to apply face verification several times. This task is handled in deepface as well.
Notice that face recognition has O(n) time complexity and this might be problematic for millions or billions level data. Herein, approximate nearest neighbor (a-nn) algorithm reduces time complexity dramatically. Spotify Annoy, Facebook Faiss and NMSLIB are amazing a-nn libraries. Besides, Elasticsearch wraps an NMSLIB and it comes with highly scalability. You should run deepface within those a-nn libraries if you have really large scale data base.
Real time
Finally, we can apply face recogntion in real time as well.
Conclusion
So, we’ve mentioned how to run a modern face recognition pipeline within opencv. Even though opencv does not support state-of-the-art face recognition models, it offers a pretty interface to load and run externally. In this post, we’ve built OpenFace model which is originally built in Torch. Beyond face recogntion, OpenCV offers a common interface to consume external models.
The source code of this study is pushed to GitHub. You can support this study by starring⭐️ the repo.
Support this blog if you do like!
