Logarithm of Sigmoid As a Neural Networks Activation Function

Previously, we’ve reviewed sigmoid function as activation function for neural networks. Logarithm of sigmoid states it modified version. Unlike to sigmoid, log of sigmoid produces outputs in scale of (-∞, 0]. In this post, we’ll mention how to use the logarithmic sigmoid in feedforward and backpropagation in neural networks.

Transfer Function

y = log(1/(1+e^-x))

---

🙋‍♂️ You may consider to enroll my top-rated machine learning course on Udemy

---

Notice that log(x) denotes base-2 log in computer science, base-e log in mathematical analysis and  base-10 log in logarithm tables. Any log base can be refered to in this equation. That’s why, I would modify the equation to more generalized form.

y = log_b(1/(1+e^-x))

Logarithm base change rule states that log_bx = log_cx / log_cb. We would apply the base change rule to the equation for the base-e (or natural logarithm).

y = log_b(1/(1+e^-x)) = log_e(1/(1+e^-x)) / log_e(b) = ln(1/(1+e^-x)) / ln(b)

Now, we would apply logarithm quotient rule to the dividend.

y = ln(1/(1+e^-x)) / ln(b) = [ln(1) – ln(1+e^-x)] / ln(b)

Notice that natural logarithm of 1 is equal to 0.

y = [ln(1) – ln(1+e^-x)] / ln(b) = [0 – ln(1+e^-x)] / ln(b) = – ln(1+e^-x) / ln(b)

---

---

Derivative

Now, it is time to derive.

y = – ln(1+e^-x) / ln(b)

dy/dx = d(- ln(1+e^-x) / ln(b))/dx

We can move the constant terms ln(b) and -1 outside of the derivative.

dy/dx = (-1/ln(b)) . d(ln(1+e^-x))/dx

Notice that derivative of ln(x) is equal to 1/x

dy/dx = (-1/ln(b)) . d(ln(1+e^-x))/dx = (-1/ln(b)). (1/1+e^-x). d(1+e^-x)/dx

dy/dx = (-1/ln(b)) . (1/1+e^-x). e^-x . (-1) = (1/ln(b)).(1/1+e^-x).e^-x

dy/dx = (e^-x)/(ln(b) . (1+e^-x))

To simplify the derivative, we would multiply both dividend and divisor to e^x

---

---

dy/dx = (e^-x)/(ln(b) . (1+e^-x)) = (e^-x.e^x)/(ln(b).(1+e^-x).e^x) = 1 / (ln(b).(e^x+1))

To sum up, activation function and derivative for logarithm of sigmoid is demonstrated below.

y = log_b(1/(1+e^-x))

dy/dx = 1 / (ln(b).(e^x+1))

Natural Logarithm of Sigmoid

We’ve produced generalized form for derivative of logarithm of sigmoid. We would change b to e to calculate the derivative of natural logarithm of sigmoid. Then, derivative would be in simpler form.

y = log_e(1/(1+e^-x)) = ln(1/(1+e^-x)

dy/dx = 1 / (ln(e).(e^x+1))

Notice that logarithm of the base is equal to 1. Similarly ln(e) is equal to 1, too.

dy/dx = 1 / (ln(e).(e^x+1)) = 1 / (e^x+1)

To sum up, activation function and derivative for natural logarithm of sigmoid is illustrated below.

---

---

y = ln(1/(1+e^-x)

dy/dx = 1 / (e^x+1)

Let’s dance

These are the dance moves of the most common activation functions in deep learning. Ensure to turn the volume up 🙂

---

Support this blog if you do like!

    

---