A Gentle Introduction to Auto-Keras

Finding the correct network structure and hyper-parameters is a totally black box in deep learning. There is no rule or formula for designing. Practitioners mostly adapt to apply heuristic approaches. This requires machine learning expertise. Newly trend in machine learning world is automated machine learning. Automated machine learning or self service AI aims to find the best model without ML expertise. In this way, we can by-pass all modelling steps of data scientists. AutoML idea is basically based on brute force. This makes sense for people out of data science world and data scientists. Today, Google, H2O, Auto-WEKA, Auto-Sklearn – they all exist in the market with pre-release versions. Herein, Auto-Keras is an unpaid alternative to these self-service AI solutions. It is supported by Keras community as well.

It is very easy to use. All you need is just to feed your data set and execution time.

---

🙋‍♂️ You may consider to enroll my top-rated machine learning course on Udemy

---

Data set

I feed FER 2013 data set. This data set stores facial expression labels for face photos. The data set consists of 28709 instances for training set and 3589 instances for public test set. Each instance consists of 48×48 sized image. In other words, there are 2304 features. Same procedure with this repository is applied to load data.

num_classes = 7 #angry, disgust, fear, happy, sad, surprise, neutral

with open("fer2013.csv") as f:
    content = f.readlines()

lines = np.array(content)

num_of_instances = lines.size

x_train, y_train, x_test, y_test = [], [], [], []

for i in range(1,num_of_instances):
    try:
        emotion, img, usage = lines[i].split(",")
        val = img.split(" ")
        pixels = np.array(val, 'float32')

	#auto-keras expects labels as numeric instead of one hot encoded
        #emotion = keras.utils.to_categorical(emotion, num_classes)

        if 'Training' in usage:
            y_train.append(emotion)
            x_train.append(pixels)
        elif 'PublicTest' in usage:
            y_test.append(emotion)
            x_test.append(pixels)
    except Exception as ex:
        print(ex)

#------------------------

x_train = np.array(x_train, 'float32')
y_train = np.array(y_train, 'float32')
x_test = np.array(x_test, 'float32')
y_test = np.array(y_test, 'float32')

x_train /= 255 #normalize inputs between [0, 1]
x_test /= 255

x_train = x_train.reshape(x_train.shape[0], 48, 48, 1)
x_train = x_train.astype('float32')
x_test = x_test.reshape(x_test.shape[0], 48, 48, 1)
x_test = x_test.astype('float32')

print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')

Auto-keras

Here, I set time limit to 24 hours. This is the default configuration.

import autokeras as ak

model = ak.ImageClassifier(path="/automodels/", verbose=True)
model.fit(x_train, y_train, time_limit=60*60*24)
model.final_fit(x_train, y_train, x_test, y_test, retrain=True)

My custom design was able to get 57% accuracy. Auto-keras can get 66.23% accuracy. Increasing time limit might contribute to improve the accuracy but as-is comes with almost 10 percent improvement and this is very satisfactory.

score = model.evaluate(x_test, y_test)
print("accuracy on test set is ",100*score)

This is a classification problem and pure accuracy couldn’t give an idea to evaluate the system.

autokeras_predictions = autokeras_model.predict(x_test)
from sklearn.metrics import classification_report, confusion_matrix
confusion_matrix(y_test, autokeras_predictions)

Auto-Keras generates the following confusion matrix. X-axis is predictions whereas y-axis is actual values.

	angry	disgust	fear	happy	sad	surprise	neutral	Recall
angry	233	9	49	23	90	11	52	50%
disgust	12	31	1	2	5	0	5	55%
fear	30	3	189	21	130	41	82	38%
happy	7	1	11	778	22	17	59	87%
sad	35	4	51	24	391	9	139	60%
surprise	8	1	20	18	13	333	22	80%
neutral	13	1	29	49	90	3	422	70%
Precision	69%	62%	54%	85%	53%	80%	54%	acc=66%

24-hours-run can complete 41 different models.

Preprocessing the images.
Preprocessing finished.
Initializing search.
Initialization finished.
+----------------------------------------------+
|               Training model 0               |
+----------------------------------------------+
No loss decrease after 5 epochs.
Saving model.
+--------------------------------------------------------------------------+
|        Model ID        |          Loss          |      Metric Value      |
+--------------------------------------------------------------------------+
|           0            |   5.586169672012329    |  0.48439999999999994   |
+--------------------------------------------------------------------------+
+----------------------------------------------+
|               Training model 1               |
+----------------------------------------------+
No loss decrease after 5 epochs.
Saving model.
+--------------------------------------------------------------------------+
|        Model ID        |          Loss          |      Metric Value      |
+--------------------------------------------------------------------------+
|           1            |   3.8872979521751403   |         0.6352         |
+--------------------------------------------------------------------------+
...
+----------------------------------------------+
|              Training model 40               |
+----------------------------------------------+
No loss decrease after 5 epochs.
Saving model.
+--------------------------------------------------------------------------+
|        Model ID        |          Loss          |      Metric Value      |
+--------------------------------------------------------------------------+
|           40           |   3.762257432937622    |         0.6532         |
+--------------------------------------------------------------------------+
+----------------------------------------------+
|              Training model 41               |
+----------------------------------------------+
Epoch-10, Current Metric - 0.668:  59%|█████████████▌         | 130/221 [01:24<01:02,  1.45 batch/s]Time is out.

Auto-keras stores model related files into the automodels folder. I specified this folder while initializing model. best_model.txt contains the text “best model: 29”.

Individual predictions

Making predictions are very similar to regular Keras.

---

---

predictions = model.predict(x_test)

instances = 0
correct_ones = 0

for i in range(0, len(predictions)):
    print("Prediction: ",predictions[i],", Actual: ",y_test[i])

    if predictions[i] == y_test[i]:
        correct_ones = correct_ones + 1

    instances = instances + 1

Exporting and restoring Auto-Keras models

Restoration is naturally enabled in Auto-Keras. We can re-use the best model by exporting auto-keras model and pickle from file functions respectively.

model.export_autokeras_model('best_auto_keras_model.h5')

When I load this auto-keras model in a different notebook, I can get same accuracy score for same data set. This means that exporting auto-keras model stores both network structure and final weights. Notice that restored model is different than Keras!

from autokeras.utils import pickle_from_file

model = pickle_from_file("best_auto_keras_model.h5")

score = model.evaluate(x_test, y_test)
print(score)

predictions = model.predict(x_test)

Exporting and restoring Keras models

We can export regular Keras models in Auto-Keras, too.

model.export_keras_model('best_keras_model.h5')

Now, I need to call Keras functions to load models.

from keras.models import load_model
keras_model = load_model('best_keras_model.h5')

Remember that summarizing is enabled in Keras.

It seems Auto-Keras builds a model consisting of 501 layers and 7M parameters. This is really amazing! Here can be found the exported Keras model structure.

keras_model.summary()
layers = keras_model.layers
index = 1
for layer in layers:
   print(index,"- ",layer.name)
   index = index + 1

It seems that auto-keras just exports network structure and excludes weights. Because predictions must be same in previous step if weights are same. This means that regular keras model must be re-trained.

keras_predictions = keras_model.predict(x_test)

correctly_classified = 0
for i in range(0, keras_predictions.shape[0]):
    print("actual: ",y_test[i]," - prediction: ",np.argmax(keras_predictions[i]))

    if y_test[i] == np.argmax(keras_predictions[i]):
        correctly_classified = correctly_classified + 1

This disappoints me because exporting model except weights doesn’t come in handy. Maybe this is because Auto-Keras is now pre-release version. I report this to developers. I’ll update this post if this issue is resolved.

Conclusion

Automated machine learning is a promising field in machine learning world. However, all alternatives exist in the market with pre-release versions. Besides, the most powerful one – Google AutoML costs 20 dollars per hours. This wouldn’t be adopted for personal usage. On the other hand, you should have a powerful computational power (GPU) hand it to open source alternative such as Auto-Keras. All things considered, it seems that we need to come a long way to have artificial general intelligence.

I pushed to jupyter notebook of this post to GitHub. I will also shared the model including both network structure and pre-trained weights for auto-keras on Google Drive because of file size limit soon.

---

---

---

Like this blog? Support me on Patreon

---