Feature Importance in Logistic Regression for Machine Learning Interpretability

Feature importance is a common way to make interpretable machine learning models and also explain existing models. That enables to see the big picture while taking decisions and avoid black box models. We’ve mentioned feature importance for linear regression and decision trees before. Besides, we’ve mentioned SHAP and LIME libraries to explain high level models such as deep learning or gradient boosting. In this post, we will find feature importance for logistic regression algorithm from scratch.

Ian Goodfellow shows the sigmoid function in this PhD defense very funnily.

---

🙋‍♂️ You may consider to enroll my top-rated machine learning course on Udemy

---

Vlog

You can either watch the following video or read this tutorial. They both cover the feature importance of logistic regression algorithm within python for machine learning interpretability and explainable ai.

Linear regression

Logistic regression is derived from linear regression. The main difference between those algorithms is that linear regression produces continuous outputs whereas logistic regression produces binary outputs for classification with probabilities. Remembering the feature importance in logistic regression is important to understand feature importance in logistic regression. I summarized the feature importance in logistic regression in the following video.

Logistic regression is linear

Logistic regression is mainly based on sigmoid function. The graph of sigmoid has a S-shape. That might confuse you and you may assume it as non-linear funtion. But that is not true. Logistic regression is just a linear model. That’s why, Most resources mention it as generalized linear model (GLM).

Actually, logistic regression is very similar to the perceptron. We just used the identity function in perceptron as an activation. You can drop the activation layer in perceptron because it is a dummy layer. In logistic regression, activation function becomes sigmoid function.

Remember that hidden layers make multilayer perceptrons (or neural networks) non-linear. Notice that there is no hidden layer in logistic regression. In other words, we cannot summarize the output of a neural networks in terms of a linear function but we can do it for logistic regression.

So, it is easy to explain linear functions naturally.

Data set

We are going to build a logistic regression model for iris data set. Its features are sepal length, sepal width, petal length, petal width. Besides, its target classes are setosa, versicolor and virginica. However, it has 3 classes in the target and this causes to build 3 different binary classification models with logistic regression. To make it simple, I will drop virginica classes in the data set and make it to binary data set.

Pre-processing

Let’s remember the logistic regression equation first.

---

---

z = w₀ + w₁x₁+ w₂x₂+ w₃x₃ + w₄x₄

y = 1 / (1 + e^-z)

x1 stands for sepal length; x2 stands for sepal width; x3 stands for petal length; x4 stands for petal width.

The output y is the probability of a class. If it gets closer to 1, then the instance will be versicolor whereas it becomes setosa when the proba gets closer to 0.

The output is unitless. If the left side of the equation has no unit, then its right side must not have unit as well. Let’s focus on the z equation. x₁ term stands for sepal length and its unit is centimeters. To make the equation z unitless, the multiplication of x₁ and w₁ has to be unitless as well.

We can divide the x₁ term to the standard deviation to get rid of the unit because the unit of standard deviation is same with its feature. Alternatively, we can feed x1 as is and find w₁ first. We know that its unit becomes 1/centimeters in this case. If we multiply the w₁ term to the standard deviation of the x₁ then it works as well. I prefer to apply first one in this study.

Loading the data set

Luckily, sklearn offers iris data set as an out-of-the-box function.

from sklearn.datasets import load_iris
import pandas as pd

feature_names = ["sepal_length", "sepal_width", "petal_length", "petal_width"]

x, y = load_iris(return_X_y=True)
df = pd.DataFrame(x, columns = feature_names)
df['target'] = y
print(df.head())

As I mentioned before, I’m going to drop the virginica classes in the data set to make it binary classification problem.

#0: setosa, 1: versicolor, 2: virginica
df = df[df['target'] != 2]

Normalize inputs

I’m going to walk over the columns, and divide each instance to the standard deviation of the column. In this way, features becomes unitless.

for feature_name in feature_names:
    df[feature_name] = df[feature_name] / df[feature_name].std()

Some researchers subtracts the mean of the column to each instance first, then divide it to the standard deviation. They will both work.

---

---

Modelling

We’ve finished the pre-processing the data set. We have the unitless features and binary class values in the target. We can build logistic regression model now.

from sklearn.linear_model import LogisticRegression
model = LogisticRegression(random_state=0).fit(df[feature_names].values, df['target'].values)

score = model.score(df[feature_names].values, df['target'].values)
print(score)

I got 100% accuracy for 100 instances. Of course, that’s the training set accuracy and I should split the data set into train, test and validation but this is an experimental study and I skip those stages. I am not interested in overfitting in this study.

Prediction

Built model stores intercept and coefficients already. Let’s focus on those parameters to understand the algorithm well.

w0 = model.intercept_[0]
w = w1, w2, w3, w4 = model.coef_[0]

equation = "y = %f + (%f * x1) + (%f * x2) + (%f * x3) + (%f * x4)" % (w0, w1, w2, w3, w4)
print(equation)

Logistic regression model has the following equation:

y = -0.102763 + (0.444753 * x1) + (-1.371312 * x2) + (1.544792 * x3) + (1.590001 * x4)

Let’s predict an instance based on the built model.

idx = 99
x = df.iloc[idx][feature_names].values
y = model.predict_proba(x.reshape(1, -1))[0]
print(y[1])

Prediction of the 100th instance (notice that index starts with 0) is 0.9782192589879745 based on the predict proba function. We can find the same value based on the equation.

import math
def sigmoid(x):
    return 1 / (1 + pow(math.e, -x))

result = 0
result += w0
for i in range(0, 4):
    result += x[i] * w[i]
result = sigmoid(result)
print(result)

This calculates the result 0.9782192589879745 as well.

Interpretability

We will use coefficient values to explain the logistic regression model. However, coefficients are not directly related to importance instead of linear regression.

Being positive class over being negative class could be expressed as below for a binary classification task.

---

---

P(y = 1) / P(y = 0) = P(y = 1) / (1 – P(y = 1))

Remember that we express the probability with logistic function

P(y = 1) / (1 – P(y = 1)) = [ 1 / (1 + e^-z) ] / [1 – (1 / (1 + e^-z))]

P(y = 1) / (1 – P(y = 1)) = [ 1 / (1 + e^-z) ] / [(1 + e^-z – 1) / (1 + e^-z)] = 1 / e^-z = e^+z

Let’s put the z term in the equation

P(y = 1) / P(y = 0) = e^(w₀ + w₁x₁+ w₂x₂+ w₃x₃ + w₄x₄)

BTW, we call the left side of this equation odds.

Let’s focus on a specific feature. E.g. x3. What happens to prediction when you make a change on x3 by 1 unit. I mean that I will change x3 to (x3 + 1). This is very similar to the definition of derivative.

odd_{(x3 -> x3+1)} / odd = e^(w₀ + w₁x₁+ w₂x₂+ w₃(x₃+1) + w₄x₄) / e^(w₀ + w₁x₁+ w₂x₂+ w₃x₃ + w₄x₄)

Remember that e^a / e^b = e^(a-b). I will apply this rule to the equation above.

---

---

odd_{(x3 -> x3+1)} / odd = e^(w₀ + w₁x₁+ w₂x₂+ w₃(x₃+1) + w₄x₄ – (w₀ + w₁x₁+ w₂x₂+ w₃x₃ + w₄x₄))

odd_{(x3 -> x3+1)} / odd = e^(w₀ + w₁x₁+ w₂x₂+ w₃(x₃+1) + w₄x₄ – w₀ – w₁x₁– w₂x₂ – w₃x₃ – w₄x₄)

odd_{(x3 -> x3+1)} / odd = e^(w₃(x₃+1) – w₃x₃) = e^(w₃x₃+w₃ – w₃x₃)

odd_{(x3 -> x3+1)} = e^w3

So, if we increase the x₃ feature one unit, then the prediction will change e to the power of its weight. We can apply this rule to the all weights to find the feature importance.

Feature importance

So, we will calculate the Euler number to the power of its coefficient to find the importance.

feature_importance = pd.DataFrame(feature_names, columns = ["feature"])
feature_importance["importance"] = pow(math.e, w)
feature_importance = feature_importance.sort_values(by = ["importance"], ascending=False)

from sklearn.linear_model import LogisticRegression
ax = feature_importance.plot.barh(x='feature', y='importance')
plt.show()

To sum up, the strongest feature in iris data set is petal width. An increase of the petal width feature by one unit increases the odds of being versicolor class by a factor of 4.90 when all other features remain the same.

Conclusion

So, we’ve mentioned how to explain built logistic regression models in this post. Even though its equation is very similar to linear regression, we can co-relate weights as power of e number.

Special thanks to Christoph Molnar, the author of the book – Interpretable Machine Learning: A Guide for Making Black Box Models Explainable to help me to understand this calculation.

I pushed the source code of this study to GitHub. You can support this study if you star⭐️ the repo.

---

---

Bonus

If you like this content, feature importance in decision trees might attract your attention, too. Unsimilar to linear regression and logistic regression, decision trees do this in their own way. I summarized the feature importance in decision trees topic in the following video.

---

Support this blog if you do like!

    

---