Artistic Style Transfer for Videos

Sefik Serengil

State-of-the-art states the highest level in English. Because art is outpouring of human intelligence and sense of aesthetics. Artistic style transfer is a fascinating study in deep learning. We can transform ordinary photos to masterpieces. Running code creates a real art. In this post, we will mention how to make our own Loving Vincent style movie.

style-transfer-in-videos
A style transfer video

Transformation over iterations

Style transfer is proposed to transform images. Base image becomes more similar to style image over iterations. It is long process even if you have a GPU. Moreover, we could not allocate GPU memory for high resolution images – we have to work on cpu cores in this case.

🙋‍♂️ You may consider to enroll my top-rated machine learning course on Udemy

Decision Trees for Machine Learning

The trick is that the longer working time you provide, the better it is.

Videos

You might watch the movie Loving Vincent. Even though the movie is created by real artists, we can transfer the artistic style of an art to videos. Because, a video roughly consists of 24 frames (or images) per second.

This video is transformed from this original video. Created art is really fascinating, right?

How?

We should apply artistic style transfer for all frames of a video. This means that we need to handle 1440 images for a 60 seconds long video. Remember that applying artistic style transfer for a single image is a long process. My experiments show that 100 epoch requires almost one hour for a HD image (1920×1080) if you have a Tesla P100 GPU. This means that you might spend a month for a half minute video! On the other hand, Loving Vincent is created by 120 artists and 65K paintings.

Video to images

We need to extract frames of a video in the fist step. The following program extracts the frames of myvideo.mp4 and saves to frames folder in the same directory. OpenCV handles to extract frames.

import cv2
import os

resolution = (1920, 1080)

def video_to_frames(video, path_output_dir):
   vidcap = cv2.VideoCapture(video)
   frame = 0
   while vidcap.isOpened():
      success, image = vidcap.read()
      if success:
         image = cv2.resize(image, resolution)
         cv2.imwrite(os.path.join(path_output_dir, '%d.png') % frame, image)
         frame += 1
      else:
         break
   cv2.destroyAllWindows()
   vidcap.release()

video_to_frames('myvideo.mp4', 'frames')

Applying artistic style transfer

Frames of the video is stored in the frames folder. Now, we will apply artistic style transfer to all of those images. You should read this blog post to handle this duty. Also, the source code of this duty is already pushed to GitHub. I skip this step because it alrady has a dedicated blog post.

Images to video

Artistic style transfer applied images will be saved as images, too. We need to create a video based on those images. OpenCV handles to create video based on saved images. The following program will create a minute long video (1440 frames in the for loop / 24 frames per second).

import cv2
import numpy as np

frame_width = 1920; frame_height = 1080

fps = 24
out = cv2.VideoWriter('outpy.avi',cv2.VideoWriter_fourcc('M','J','P','G'), fps, (frame_width,frame_height))

for i in range(0, 1440+1):
   source = "%d.png" % (i)
   source_frame = cv2.imread(soure)

   try:
      out.write(source_frame)
   except:
      print(soure," does not exist")

   # Display the resulting frame
   #cv2.imshow('frame', source_frame)

   # Press Q on keyboard to stop recording
   if cv2.waitKey(1) & 0xFF == ord('q'):
     break

# When everything done, release the video capture and video write objects
out.release()

# Closes all the frames
cv2.destroyAllWindows()

So, we’ve mentioned how to apply artistic style transfer to videos. Even though it is a very long process, I really enjoy its products. I spent a long time to create the following masterpieces. My favorite style image is Vincent van Gogh’s Starry Night. My experiments show that combination of an image including sea, sky and Starry Night would be spectacular.



Into The Forest

Ethnical argument

Recently, Lex Fridman from MIT raised an argument. Who is going to be the copyright owner of images created by AI? The owner of the data set that the network was trained on? In this case, data set would be the base photo or video. Or designer of the network architecture? In this case, style transfer code is run on VGG network and it is designed by Oxford Visual Geometry Group. The person running the code? I’m going to hold the copyright in this case. Or AI system itself? I’ve focused on this subject deeply in the following video. So, what do you think? Who is going to be the owner of the copyright?

Like this blog? Support me on Patreon

Buy me a coffee