Are you using virtual backgrounds in your online meetings? The answer is probably yes. We can join meetings with such cool backgrounds such as libraries with thousands of books. Have you ever wonder how virtual background works in zoom? Zoom handles this very fast even in real-time. In this post, we are going to mention background segmentation with mediapipe and virtual background with opencv and numpy in python.
Summary
Once you read this tutorial, then you will be able to use mediapipe to segment backgrounds and opencv – numpy pair to set virtual backgrounds in python. A real-time implementation of those work is shown in the following video.
🙋♂️ You may consider to enroll my top-rated machine learning course on Udemy
Vlog
You can either continue to read this tutorial or watch the following video. They both cover the zoom style virtual background setup, background blurring and removing the background cases.
Input images
I am going to use the following image as input. Then, I am going to modify the background of this image.
The White House would be a cool background, wouldn’t it?
Let’s read these images first.
import cv2
#Ref: https://www.pexels.com/photo/portrait-photo-of-woman-in-white-crew-neck-shirt-8090149
img = cv2.imread("pexels-cottonbro-8090149.jpg")
background_img = cv2.imread("white-house.jpg")
The both base image and background image should be same size.
background_img = cv2.resize(background_img, (img.shape[1], img.shape[0]))
Background segmentation
This technology is called as selfie segmentation in mediapipe. It is available under mediapipe solutions.
import mediapipe as mp mp_selfie_segmentation = mp.solutions.selfie_segmentation selfie_segmentation = mp_selfie_segmentation.SelfieSegmentation(model_selection=0)
Once selfie segmentation object is built, we are able to pass the input image to segmentator. Then, it will return the segmentation mask.
results = selfie_segmentation.process(cv2.cvtColor(img, cv2.COLOR_BGR2RGB)) segmentation_mask = results.segmentation_mask
Here, segmentation mask must have the same shape with the base image. We will determine a pixel belongs to background or person itself if its value is greater than a threshold value. We set 0.1 to the threshold in this experiment.
condition = np.stack((segmentation_mask,) * 3, axis=-1) < 0.1
The larger the threshold is, the more precise background segmentation is.
Classification
I am going to set the background to gold color and person itself to white. Condition already stores True for selfie pixels whereas False for background pixels. Numpy where function can make this replacement easily.
BG_COLOR = (255, 215, 0) # gold MASK_COLOR = (255, 255, 255) # white fg_image = np.zeros(img.shape, dtype=np.uint8) fg_image[:] = MASK_COLOR bg_image = np.zeros(img.shape, dtype=np.uint8) bg_image[:] = BG_COLOR output_image = np.where(condition, fg_image, bg_image) plt.imshow(output_image)
We can clearly extract person itself and the background as seen.
Setting a color to background
Let’s paint the background of the base image.
BG_COLOR = (255, 215, 0) # gold output_image = np.where(condition, img[:, :, ::-1], BG_COLOR) plt.imshow(output_image)
We can clear the background and feed no noise facial images to facial recognition pipelines. This is the principle of normalization in facial recognition. Alternatively, we can focus on the just facial area with facial detection module of MediaPipe. I recommend you read this post: Deep Face Detection with MediaPipe
Virtual background
Now, I am going to use the picture of the White House as a virtual background.
output_image = np.where(condition, img[:, :, ::-1], background_img[:, :, ::-1]) plt.imshow(output_image)
Blurring the background
In our case, the base image has a gray background but it could have a panaroma. We can blur the background if we have the condition variable. Let’s blur the white house background.
blurred_image = cv2.GaussianBlur(img, (55,55), 0) output_image = np.where(condition, img[:, :, ::-1], blurred_image) plt.imshow(output_image)
Conclusion
So, we have mentioned how to use virtual background in python similar to zoom meetings. Mediapipe handles segmentation stage and then we are able to modify the background with regular opencv and numpy.
I pushed the source code of this study to GitHub. You can support this work if you star⭐️ the repo🙏
Like this blog? Support me on Patreon
