Dance Moves of Deep Learning Activation Functions

Neither convolution nor recurrent layers of deep learning enable non-linearity. Activation functions enable neural networks to become non-linear. An activation function has two main roles. One is the function itself involving in feed forward step. Two is the derivative of the function involving in backpropagation step. You can think feed forward as prediction and backpropagation as training or learning.

sample-activation-functions-square
Activation Functions

Perceptron was raised in 50’s. Step function is adopted in that architecture. However, those architecture can just generalize linear problem. Perceptron evolved to neural networks in time. Activation functions evolved as well. Researchers have adopted sigmoid and tanh for a long time.


๐Ÿ™‹โ€โ™‚๏ธ You may consider to enroll my top-rated machine learning course on Udemy

Decision Trees for Machine Learning

regular-perceptron
Legacy Perceptron

These functions cause gradients to be vanishing in complex structures. In other words, neural networks models cannot learn anything.ย Simpler functions such as ReLU come to the throne surprisingly. Nowadays, a new activation function is raised.

I have been studying activation functions in this blog for a long time. You can find activation function related studies here. Posts cover the function itself, its derivative calculation, pros and cons.

I spent the much of time to design a featured image. Because it should attract audiences fast. Herein, I think that expressing an activation function as a dance move would be funny. Then, I like this idea and continue to design new activation functions as new dance moves. I inspired this from the study of Michael Gralmannin in Imaginary: Open Mathmematics.

Now, there are tens of the most common activation functions described deeply in this blog. Recently, I animate the featured images of activation function posts and put a funny background music (Run Amok by Kevin MacLeod, Licensed under Creative Commons: By Attribution 3.0 License).

The result video would be very funny. I hope this would contribute beginners to learn these concepts easier.


Like this blog? Support me on Patreon

Buy me a coffee