Face Alignment for Face Recognition in Python within OpenCV

Face alignment is an early stage of the modern face recognition pipeline. Google declared that face alignment increases the accuracy of its face recognition model FaceNet from 98.87% to 99.63%. This is almost 1% accuracy improvement. Similar to face detection which is also the earlier stage of the pipeline, we can apply 2D face alignment within OpenCV in Python easily.

Vlog

You can either continue to read this tutorial or watch the following video. They both cover the face alignment from scratch topic with same procedures.

---

🙋‍♂️ You may consider to enroll my top-rated machine learning course on Udemy

---

Face detectors

In this post, we will use OpenCV’s haar cascade method to detect faces. This is very traditional method based on adaboost algorithm. Even though it is legacy, it works well.

Still there are more modern approaches existing in the literature. OpenCV offers haar cascade and Single Shot Multibox Detector (SSD). Dlib offers Histogram of Oriented Gradients (HOG) and a CNN based Max-Margin Object Detection (MMOD) and finally Multi-task Cascaded Convolutional Networks (MTCNN) is a common solution for face detection. You can monitor the detection performance of those methods in the following video. SSD and MTCNN seem to be more robust than Haar and Dlib HoG.

SSD can process 9.20 frames per second whereas fps rates are 6.50 for Haar cascade, 1.57 for Dlib Hog, 1.54 for MTCNN. In other words, SSD is the fastest one. 

Here, you can watch how to use different face detector backends with just a few lines of code.

Besides, retinaface offers a cutting-edge technology for face detection.

Initializing environment

OpenCV needs the exact path of its haarcascade configurations. We will use both frontal face and eye detection modules.

import cv2
face_detector = cv2.CascadeClassifier("haarcascade_frontalface_default.xml")
eye_detector = cv2.CascadeClassifier("haarcascade_eye.xml")

Reading image

We will work the the following image of Angelina Jolie.

img = cv2.imread("angelina.jpg")
img_raw = img.copy()

Face detection

We will just focus on the detected face and ignore out of the area.

---

---

faces = face_detector.detectMultiScale(img, 1.3, 5)
face_x, face_y, face_w, face_h = faces[0]

img = img[int(face_y):int(face_y+face_h), int(face_x):int(face_x+face_w)]
img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

Eye detection

OpenCV offers to detect eyes as well. It expects the gray version of the image.

eyes = eye_detector.detectMultiScale(gray_img)

index = 0
for (eye_x, eye_y, eye_w, eye_h) in eyes:
   if index == 0:
      eye_1 = (eye_x, eye_y, eye_w, eye_h)
   elif index == 1:
      eye_2 = (eye_x, eye_y, eye_w, eye_h)

   cv2.rectangle(img,(eye_x, eye_y),(eye_x+eye_w, eye_y+eye_h), color, 2)
   index = index + 1

We stored eye locations in 1st and 2nd eye variables. Now, we should decide which one is left eye and which one is right eye. X locations are stored in the 1st item of the tuple. So, small one will be the left eye.

if eye_1[0] < eye_2[0]:
   left_eye = eye_1
   right_eye = eye_2
else:
   left_eye = eye_2
   right_eye = eye_1

Coordinates of eye

Top left point is the (0, 0) point in OpenCV. Detected eye includes 4 values. The following illustration show these values.

We need the center of detected eyes. Notice that 0 index refers to x, 1 index refers to y, 2 index refers to w and 3 index refers to h. Let’s draw a line between the center of two eyes also.

left_eye_center = (int(left_eye[0] + (left_eye[2] / 2)), int(left_eye[1] + (left_eye[3] / 2)))
left_eye_x = left_eye_center[0]; left_eye_y = left_eye_center[1]

right_eye_center = (int(right_eye[0] + (right_eye[2]/2)), int(right_eye[1] + (right_eye[3]/2)))
right_eye_x = right_eye_center[0]; right_eye_y = right_eye_center[1]

cv2.circle(img, left_eye_center, 2, (255, 0, 0) , 2)
cv2.circle(img, right_eye_center, 2, (255, 0, 0) , 2)
cv2.line(img,right_eye_center, left_eye_center,(67,67,67),2)

We exactly need the angle between horizontal line of the drawn line. Because we will rotate the image based on this angle. In this picture left eye is above than the right eye. That’s why, we will rotate the image based on the inverse direction of clock. On the other hand, we would rotate the image based on the clock direction if right eye were above than the right eye.

if left_eye_y < right_eye_y:
   point_3rd = (right_eye_x, left_eye_y)
   direction = -1 #rotate same direction to clock
   print("rotate to clock direction")
else:
   point_3rd = (left_eye_x, right_eye_y)
   direction = 1 #rotate inverse direction of clock
   print("rotate to inverse clock direction")

cv2.circle(img, point_3rd, 2, (255, 0, 0) , 2)

cv2.line(img,right_eye_center, left_eye_center,(67,67,67),2)
cv2.line(img,left_eye_center, point_3rd,(67,67,67),2)
cv2.line(img,right_eye_center, point_3rd,(67,67,67),2)

Little trigonometry

We can find the length of 3 edges of the triangle with euclidean distance.

def euclidean_distance(a, b):
x1 = a[0]; y1 = a[1]
x2 = b[0]; y2 = b[1]
return math.sqrt(((x2 - x1) * (x2 - x1)) + ((y2 - y1) * (y2 - y1)))
a = euclidean_distance(left_eye_center, point_3rd)
b = euclidean_distance(right_eye_center, left_eye_center)
c = euclidean_distance(right_eye_center, point_3rd)

Besides, cosine rule says that

cos(A) = (b² + c² – a² ) / (2bc)

We’ve already calculated the edge lengths of the triangle. So, we can apply cosine rule as well. Finding the angle from its cosine value requires to call inverse trigonometric functions. Luckily, numpy does it for us. However, arc cos function returns angle in radian unit. We need to find its 180 times over pi value to find in degree.

cos_a = (b*b + c*c - a*a)/(2*b*c)
print("cos(a) = ", cos_a)

angle = np.arccos(cos_a)
print("angle: ", angle," in radian")

angle = (angle * 180) / math.pi
print("angle: ", angle," in degree")

We’ve built a right angle triangle. In other words, one angle is 90 degree. Sum of the other two angles will be 90 degree as well. If we will rotate image to clock direction, then we need to rotate the image 90 minus found angle. You can understand why if you look at the following demonstration.

---

---

if direction == -1:
   angle = 90 - angle

Now, we know the required angle to rotate.

from PIL import Image
new_img = Image.fromarray(img_raw)
new_img = np.array(new_img.rotate(direction * angle))

Testings

If we test the explained logic in this post to the picture above, results seem very satisfactory.

Conclusion

So, we’ve mentioned how to align faces in Python with OpenCV from scratch. We’ve used a little trigonometry to do this task. We can of course rotate image 1 degree until both eyes are horizontal but this will increase the complexity of the solution. On the other hand, this approach offer linear time to align faces.

I pushed the source code of this study to the GitHub. You can support this study by starring⭐️ the GitHub repo as well.

Python library

Herein, deepface is a lightweight facial analysis framework covering both face recognition and demography such as age, gender, race and emotion. It also offers you to detect and align faces with a single line of code. It is fully open-sourced and available on PyPI.

from deepface import DeepFace
import matplotlib.pyplot as plt
face_objs = DeepFace.extract_faces(img_path = "angelina.jpg")
for face_obj in face_objs:
   img = face_obj["face"]
   plt.imshow(img)
   plt.show()

Here, you can find a hands-on video tutorial as well.

---

Like this blog? Support me on Patreon

---