Facial Landmarks for Face Recognition with Dlib

Sefik Serengil

Dlib can incredibly find 68 different facial landmark points including chin and jaw line, eyebrows, nose, eyes and lips. We can extract exact facial area based on those landmark points beyond rough face detection. This will increase the accuracy of face recognition models dramatically because we will discard any noise in this way.

Extracting faces with landmarks

Vlog

You can either continue to read this tutorial or watch the following video to follow Normalization in Face Recognition with Dlib Facial Landmarks. They both cover the same procedures.

🙋‍♂️ You may consider to enroll my top-rated machine learning course on Udemy

Decision Trees for Machine Learning

Objective

We are going to run facial landmarks detector with Dlib in Python. Landmark points are jaw and chin, eyes and eyebrows, inner and outer area of lips and nose.

Requirements

We need to load the landmark detector externally even if dlib was installed. The following code block will guide you where to install the required file and how to load it.

import dlib
#http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2
face_detector = dlib.get_frontal_face_detector()
landmark_detector = dlib.shape_predictor("shape_predictor_68_face_landmarks.dat")
Reading the image

You can read an image with either dlib or opencv. Here, dlib loads images in RGB format whereas opencv loads BGR. So, if you load it via opencv, then you also need to reconvert it to RGB as shown below.

img_path = "deepface/tests/dataset/img1.jpg"

#read with dlib
img = dlib.load_rgb_image(img_path)

#read with opencv
#img = cv2.imread(img_path)[:,:,::-1]
Face detection

Finding facial landmarks requires to detect faces first. Dlib has its own out-of-the-box frontal face detector module.

faces = face_detector(img, 1)

We’ve already known that Dlib HoG is a robust one among common face detectors.

Landmark detections

Then, we will pass the detected faces to landmark detector.

landmark_tuple = []
for k, d in enumerate(faces):
   landmarks = landmark_detector(img, d)
   for n in range(0, 27):
      x = landmarks.part(n).x
      y = landmarks.part(n).y
      landmark_tuple.append((x, y))
      cv2.circle(img, (x, y), 2, (255, 255, 0), -1)

Here, there are 68 landmark points. However, I am interested in only boundaries of facial area. So, I will discard eyes, nose and mouth related points. That’s why, I built a for loop in range of 0 and 27.

Landmark points

Here, you can see the facial landmarks of an image of Angelina Jolie.



Landmarks of Angelina Jolie

Notice that I stored the landmark point coordinates in the landmark tuple variable. My route will start the point 17 and end at point 27. In this way, I can visit all points of the boundaries of facial landmarks respectively.

routes = []

for i in range(15, -1, -1):
   from_coordinate = landmark_tuple[i+1]
   to_coordinate = landmark_tuple[i]
   routes.append(from_coordinate)

from_coordinate = landmark_tuple[0]
to_coordinate = landmark_tuple[17]
routes.append(from_coordinate)

for i in range(17, 20):
   from_coordinate = landmark_tuple[i]
   to_coordinate = landmark_tuple[i+1]
   routes.append(from_coordinate)

from_coordinate = landmark_tuple[19]
to_coordinate = landmark_tuple[24]
routes.append(from_coordinate)

for i in range(24, 26):
   from_coordinate = landmark_tuple[i]
   to_coordinate = landmark_tuple[i+1]
   routes.append(from_coordinate)

from_coordinate = landmark_tuple[26]
to_coordinate = landmark_tuple[16]
routes.append(from_coordinate)
routes.append(to_coordinate)

Let’s draw a line in the routes.

for i in range(0, len(routes)-1):
   from_coordinate = routes[i]
   to_coordinate = routes[i+1]
   img = cv2.line(img, from_coordinate, to_coordinate, (255, 255, 0), 1)
Drawing the face boundaries
Extract facial area

Now, we can discard the background of an image based on facial landmarks. I will feed just routes because it stores the facial landmarks respectively.

mask = np.zeros((img.shape[0], img.shape[1]))
mask = cv2.fillConvexPoly(mask, np.array(routes), 1)
mask = mask.astype(np.bool)

out = np.zeros_like(img)
out[mask] = img[mask]

plt.imshow(out)
Discarding background

In this way, we can just focus on the facial area. Black pixels represent 0 in RGB and they will be neutral element in my training stage. Thus, the accuracy of my face recognition model will increase because I ignored any noise in the image.

BTW, face swap studies are mainly based on this approach as well.

Sketch drawing

We can draw a sketch with dlib facial landmarks detector as well.

Background blurring

In one of my previous post, I blurred the background of an image except detected face. However, face detection returns a rectangle area and it won’t be sensitive.

Background blurring by detected area

Herein, we can blur the image except the facial landmark boundaries. In this way, we can focus on just facial area and we also have more sensitive results.

I will use the blur_img function mentioned in this post. 

blurred_img = blur_img(img, factor = 60)
blurred_img[mask] = img[mask]
plt.imshow(blurred_img)
Sensitive background blurring

Notice that facial area is clear whereas the image itself is blurred. This seems more pretty.



Bonus

Dlib can find 68 facial landmarks in 2D space. On the other hand, MediaPipe can find 468 landmarks in 3D space. Recommend you to read this tutorial: Deep Face Detection with MediaPipe. You can see its real-time implementation below.

Instead of focusing on just the facial area, we can remove the background as well. This is very similar to zoom style virtual background setup. Here, we will assign black or white background. You can see its real-time implementation below.

Conclusion

So, we have mentioned how to find facial landmarks with Dlib. This will ignore the noises of images and this will increase the accuracy of face recognition pipeline.

I pushed the source code of this study as a notebook to GitHub. You can support this study if you star⭐️ the repo.

Like this blog? Support me on Patreon

Buy me a coffee