Dlib can incredibly find 68 different facial landmark points including chin and jaw line, eyebrows, nose, eyes and lips. We can extract exact facial area based on those landmark points beyond rough face detection. This will increase the accuracy of face recognition models dramatically because we will discard any noise in this way.
Vlog
You can either continue to read this tutorial or watch the following video to follow Normalization in Face Recognition with Dlib Facial Landmarks. They both cover the same procedures.
🙋♂️ You may consider to enroll my top-rated machine learning course on Udemy
Objective
We are going to run facial landmarks detector with Dlib in Python. Landmark points are jaw and chin, eyes and eyebrows, inner and outer area of lips and nose.
Requirements
We need to load the landmark detector externally even if dlib was installed. The following code block will guide you where to install the required file and how to load it.
import dlib
#http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2
face_detector = dlib.get_frontal_face_detector()
landmark_detector = dlib.shape_predictor("shape_predictor_68_face_landmarks.dat")
Reading the image
You can read an image with either dlib or opencv. Here, dlib loads images in RGB format whereas opencv loads BGR. So, if you load it via opencv, then you also need to reconvert it to RGB as shown below.
img_path = "deepface/tests/dataset/img1.jpg" #read with dlib img = dlib.load_rgb_image(img_path) #read with opencv #img = cv2.imread(img_path)[:,:,::-1]
Face detection
Finding facial landmarks requires to detect faces first. Dlib has its own out-of-the-box frontal face detector module.
faces = face_detector(img, 1)
We’ve already known that Dlib HoG is a robust one among common face detectors.
Landmark detections
Then, we will pass the detected faces to landmark detector.
landmark_tuple = []
for k, d in enumerate(faces):
landmarks = landmark_detector(img, d)
for n in range(0, 27):
x = landmarks.part(n).x
y = landmarks.part(n).y
landmark_tuple.append((x, y))
cv2.circle(img, (x, y), 2, (255, 255, 0), -1)
Here, there are 68 landmark points. However, I am interested in only boundaries of facial area. So, I will discard eyes, nose and mouth related points. That’s why, I built a for loop in range of 0 and 27.
Here, you can see the facial landmarks of an image of Angelina Jolie.
Notice that I stored the landmark point coordinates in the landmark tuple variable. My route will start the point 17 and end at point 27. In this way, I can visit all points of the boundaries of facial landmarks respectively.
routes = [] for i in range(15, -1, -1): from_coordinate = landmark_tuple[i+1] to_coordinate = landmark_tuple[i] routes.append(from_coordinate) from_coordinate = landmark_tuple[0] to_coordinate = landmark_tuple[17] routes.append(from_coordinate) for i in range(17, 20): from_coordinate = landmark_tuple[i] to_coordinate = landmark_tuple[i+1] routes.append(from_coordinate) from_coordinate = landmark_tuple[19] to_coordinate = landmark_tuple[24] routes.append(from_coordinate) for i in range(24, 26): from_coordinate = landmark_tuple[i] to_coordinate = landmark_tuple[i+1] routes.append(from_coordinate) from_coordinate = landmark_tuple[26] to_coordinate = landmark_tuple[16] routes.append(from_coordinate) routes.append(to_coordinate)
Let’s draw a line in the routes.
for i in range(0, len(routes)-1): from_coordinate = routes[i] to_coordinate = routes[i+1] img = cv2.line(img, from_coordinate, to_coordinate, (255, 255, 0), 1)
Extract facial area
Now, we can discard the background of an image based on facial landmarks. I will feed just routes because it stores the facial landmarks respectively.
mask = np.zeros((img.shape[0], img.shape[1])) mask = cv2.fillConvexPoly(mask, np.array(routes), 1) mask = mask.astype(np.bool) out = np.zeros_like(img) out[mask] = img[mask] plt.imshow(out)
In this way, we can just focus on the facial area. Black pixels represent 0 in RGB and they will be neutral element in my training stage. Thus, the accuracy of my face recognition model will increase because I ignored any noise in the image.
BTW, face swap studies are mainly based on this approach as well.
Sketch drawing
We can draw a sketch with dlib facial landmarks detector as well.
Background blurring
In one of my previous post, I blurred the background of an image except detected face. However, face detection returns a rectangle area and it won’t be sensitive.
Herein, we can blur the image except the facial landmark boundaries. In this way, we can focus on just facial area and we also have more sensitive results.
I will use the blur_img function mentioned in this post.
blurred_img = blur_img(img, factor = 60) blurred_img[mask] = img[mask] plt.imshow(blurred_img)
Notice that facial area is clear whereas the image itself is blurred. This seems more pretty.
Bonus
Dlib can find 68 facial landmarks in 2D space. On the other hand, MediaPipe can find 468 landmarks in 3D space. Recommend you to read this tutorial: Deep Face Detection with MediaPipe. You can see its real-time implementation below.
Instead of focusing on just the facial area, we can remove the background as well. This is very similar to zoom style virtual background setup. Here, we will assign black or white background. You can see its real-time implementation below.
Conclusion
So, we have mentioned how to find facial landmarks with Dlib. This will ignore the noises of images and this will increase the accuracy of face recognition pipeline.
I pushed the source code of this study as a notebook to GitHub. You can support this study if you star⭐️ the repo.
Support this blog if you do like!
