A Gentle Introduction to Auto-Keras

Finding the correct network structure and hyper-parameters is a totally black box in deep learning. There is no rule or formula for designing. Practitioners mostly adapt to apply heuristic approaches. This requires machine learning expertise. Newly trend in machine learning world is automated machine learning. Automated machine learning or self service AI aims to find the best model without ML expertise. In this way, we can by-pass all modelling steps of data scientists. AutoML idea is basically based on brute force. This makes sense for people out of data science world and data scientists. Today, Google, H2O, Auto-WEKA, Auto-Sklearn – they all exist in the market with pre-release versions. Herein, Auto-Keras is an unpaid alternative to these self-service AI solutions. It is supported by Keras community as well.

google-automl
Google AutoML

It is very easy to use. All you need is just to feed your data set and execution time.


🙋‍♂️ You may consider to enroll my top-rated machine learning course on Udemy

Decision Trees for Machine Learning

Data set

I feed FER 2013 data set. This data set stores facial expression labels for face photos. The data set consists of 28709 instances for training set and 3589 instances for public test set. Each instance consists of 48×48 sized image. In other words, there are 2304 features. Same procedure with this repository is applied to load data.

num_classes = 7 #angry, disgust, fear, happy, sad, surprise, neutral

with open("fer2013.csv") as f:
    content = f.readlines()

lines = np.array(content)

num_of_instances = lines.size

x_train, y_train, x_test, y_test = [], [], [], []

for i in range(1,num_of_instances):
    try:
        emotion, img, usage = lines[i].split(",")
        val = img.split(" ")
        pixels = np.array(val, 'float32')

	#auto-keras expects labels as numeric instead of one hot encoded
        #emotion = keras.utils.to_categorical(emotion, num_classes)

        if 'Training' in usage:
            y_train.append(emotion)
            x_train.append(pixels)
        elif 'PublicTest' in usage:
            y_test.append(emotion)
            x_test.append(pixels)
    except Exception as ex:
        print(ex)

#------------------------

x_train = np.array(x_train, 'float32')
y_train = np.array(y_train, 'float32')
x_test = np.array(x_test, 'float32')
y_test = np.array(y_test, 'float32')

x_train /= 255 #normalize inputs between [0, 1]
x_test /= 255

x_train = x_train.reshape(x_train.shape[0], 48, 48, 1)
x_train = x_train.astype('float32')
x_test = x_test.reshape(x_test.shape[0], 48, 48, 1)
x_test = x_test.astype('float32')

print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')

Auto-keras

Here, I set time limit to 24 hours. This is the default configuration.

import autokeras as ak

model = ak.ImageClassifier(path="/automodels/", verbose=True)
model.fit(x_train, y_train, time_limit=60*60*24)
model.final_fit(x_train, y_train, x_test, y_test, retrain=True)

My custom design was able to get 57% accuracy. Auto-keras can get 66.23% accuracy. Increasing time limit might contribute to improve the accuracy but as-is comes with almost 10 percent improvement and this is very satisfactory.

score = model.evaluate(x_test, y_test)
print("accuracy on test set is ",100*score)

This is a classification problem and pure accuracy couldn’t give an idea to evaluate the system.

autokeras_predictions = autokeras_model.predict(x_test)
from sklearn.metrics import classification_report, confusion_matrix
confusion_matrix(y_test, autokeras_predictions)

Auto-Keras generates the following confusion matrix. X-axis is predictions whereas y-axis is actual values.

angry disgust fear happy sad surprise neutral Recall
angry 233 9 49 23 90 11 52 50%
disgust 12 31 1 2 5 0 5 55%
fear 30 3 189 21 130 41 82 38%
happy 7 1 11 778 22 17 59 87%
sad 35 4 51 24 391 9 139 60%
surprise 8 1 20 18 13 333 22 80%
neutral 13 1 29 49 90 3 422 70%
Precision 69% 62% 54% 85% 53% 80% 54% acc=66%

24-hours-run can complete 41 different models.

Preprocessing the images.
Preprocessing finished.
Initializing search.
Initialization finished.
+----------------------------------------------+
|               Training model 0               |
+----------------------------------------------+
No loss decrease after 5 epochs.
Saving model.
+--------------------------------------------------------------------------+
|        Model ID        |          Loss          |      Metric Value      |
+--------------------------------------------------------------------------+
|           0            |   5.586169672012329    |  0.48439999999999994   |
+--------------------------------------------------------------------------+
+----------------------------------------------+
|               Training model 1               |
+----------------------------------------------+
No loss decrease after 5 epochs.
Saving model.
+--------------------------------------------------------------------------+
|        Model ID        |          Loss          |      Metric Value      |
+--------------------------------------------------------------------------+
|           1            |   3.8872979521751403   |         0.6352         |
+--------------------------------------------------------------------------+
...
+----------------------------------------------+
|              Training model 40               |
+----------------------------------------------+
No loss decrease after 5 epochs.
Saving model.
+--------------------------------------------------------------------------+
|        Model ID        |          Loss          |      Metric Value      |
+--------------------------------------------------------------------------+
|           40           |   3.762257432937622    |         0.6532         |
+--------------------------------------------------------------------------+
+----------------------------------------------+
|              Training model 41               |
+----------------------------------------------+
Epoch-10, Current Metric - 0.668:  59%|█████████████▌         | 130/221 [01:24<01:02,  1.45 batch/s]Time is out.

Auto-keras stores model related files into the automodels folder. I specified this folder while initializing model. best_model.txt contains the text “best model: 29”.

auto-keras-models
Auto-keras models

Individual predictions

Making predictions are very similar to regular Keras.





predictions = model.predict(x_test)

instances = 0
correct_ones = 0

for i in range(0, len(predictions)):
    print("Prediction: ",predictions[i],", Actual: ",y_test[i])

    if predictions[i] == y_test[i]:
        correct_ones = correct_ones + 1

    instances = instances + 1

Exporting and restoring Auto-Keras models

Restoration is naturally enabled in Auto-Keras. We can re-use the best model by exporting auto-keras model and pickle from file functions respectively.

model.export_autokeras_model('best_auto_keras_model.h5')

When I load this auto-keras model in a different notebook, I can get same accuracy score for same data set. This means that exporting auto-keras model stores both network structure and final weights. Notice that restored model is different than Keras!

from autokeras.utils import pickle_from_file

model = pickle_from_file("best_auto_keras_model.h5")

score = model.evaluate(x_test, y_test)
print(score)

predictions = model.predict(x_test)

Exporting and restoring Keras models

We can export regular Keras models in Auto-Keras, too.

model.export_keras_model('best_keras_model.h5')

Now, I need to call Keras functions to load models.

from keras.models import load_model
keras_model = load_model('best_keras_model.h5')

Remember that summarizing is enabled in Keras.

It seems Auto-Keras builds a model consisting of 501 layers and 7M parameters. This is really amazing! Here can be found the exported Keras model structure.

keras_model.summary()
layers = keras_model.layers
index = 1
for layer in layers:
   print(index,"- ",layer.name)
   index = index + 1

It seems that auto-keras just exports network structure and excludes weights. Because predictions must be same in previous step if weights are same. This means that regular keras model must be re-trained.

keras_predictions = keras_model.predict(x_test)

correctly_classified = 0
for i in range(0, keras_predictions.shape[0]):
    print("actual: ",y_test[i]," - prediction: ",np.argmax(keras_predictions[i]))

    if y_test[i] == np.argmax(keras_predictions[i]):
        correctly_classified = correctly_classified + 1

This disappoints me because exporting model except weights doesn’t come in handy. Maybe this is because Auto-Keras is now pre-release version. I report this to developers. I’ll update this post if this issue is resolved.

Conclusion

Automated machine learning is a promising field in machine learning world. However, all alternatives exist in the market with pre-release versions. Besides, the most powerful one – Google AutoML costs 20 dollars per hours. This wouldn’t be adopted for personal usage. On the other hand, you should have a powerful computational power (GPU) hand it to open source alternative such as Auto-Keras. All things considered, it seems that we need to come a long way to have artificial general intelligence.

I pushed to jupyter notebook of this post to GitHub. I will also shared the model including both network structure and pre-trained weights for auto-keras on Google Drive because of file size limit soon.






Support this blog if you do like!

Buy me a coffee      Buy me a coffee


26 Comments

  1. I was trying to load the model from autokeras that you provided in the google doc link but when i run the python for emotion detection using this model “fer_autokeras_model.h5” it says :
    OSError: Unable to open file (file signature not found)
    what should I do?

    1. You load the model as illustrated below?

      from autokeras.utils import pickle_from_file
      autokeras_model = pickle_from_file(“fer_autokeras_model.h5”)

      Please try to download h5 file again if you see same error. It might be corruption while downloading.

      1. i just downloaded the model and then used it as
        model=load.model(“fer_autokeras_model.h5”)
        i’ve tried a few times but still no luck. do i keep trying downloading it or it might be something else?

        1. You are loading the model in a wrong way. This is not a regular keras model. It is an auto-keras model and you have to load the model as demonstrated below.

          from autokeras.utils import pickle_from_file
          autokeras_model = pickle_from_file(“fer_autokeras_model.h5”)

          Code sample is pushed to GitHub (https://github.com/serengil/tensorflow-101/blob/master/python/Auto-Keras.ipynb). Block 31 with ”
          Auto-Keras Model Restoration” header. It might help you how to load model.

          1. I’m sorry to keep asking questions but I’m facing another problem and it says:
            TypeError Traceback (most recent call last)
            in
            65
            66 emotion = “”
            —> 67 for i in range(len(predictions[0])):
            68 emotion = “%s %s%s” % (emotions[i], round(predictions[0][i]*100, 2), ‘%’)
            69

            TypeError: object of type ‘numpy.float32’ has no len()

            i can’t find anywhere any solutions that could work on this

          2. Would you try predictions[0].shape or shape[0] instead of length function?

  2. Hello there, I was trying Autokeras with facial expression detection, but it gets stuck while training model 2.
    during normal execution the program takes almost all device resources, and check it from time to time, finally i find it stuck at some point during model 2 training, and all device resources are released.
    so i guess it’s not working, and seems to have just stopped.
    I tried to rerun it but it got stuck again, i searched about it online, and i found it quite frequent for Autokeras to stuck with other as well, but didn’t find a solution for it any where.
    So what can i do to solve this?
    note: i’m running with CPU only.
    this is a link for code i used : https://ideone.com/30a7oD

    1. Spending time on fit or final_fit line? Could you remove the final_fit line and rerun it again?

      1. Sorry for late reply, i tried this solution and still stuck in same location, seems like it gets suck in fit not final_fit.

        1. It seems that you did everything what I’ve done. Could you please open an issue on auto-keras GitHub repository? It might be OS related issue.

  3. I was trying to load the AutoKeras model as below:

    from autokeras.utils import pickle_from_file
    autokeras_model = pickle_from_file(“fer_autokeras_model.h5”)

    but I got the following error.

    AttributeError: Can’t get attribute ‘ImageDataTransformer’ on

    What can I do to solve this?

    1. Sorry, the complete message was this:

      AttributeError: Can’t get attribute ‘ImageDataTransformer’ on module ‘autokeras.preprocessor’ from ‘/home/ubuntu/anaconda3/envs/tensorflow-101/lib/python3.7/site-packages/autokeras/preprocessor.py’

      1. This most probably happens because of version imcompability. Could you try install autokeras with the command “pip install autokeras==0.3.7”?

  4. Hey,
    First of all, thanks you for this good work !

    I can use the auto-keras model of the google drive link (“fer_autokeras_model.h5”), but when I get a prediction, I only have an array of one numpy.float32 : the index of the emotion I presume.

    How can I have the array of predictions percentage like in “emotion-analysis-from-video.py” please?

    1. It is very easy task. The following code block will help you.

      class_names = {
      0: ‘angry’,
      1: ‘disgust’,
      2: ‘fear’,
      3: ‘happy’,
      4: ‘sad’,
      5: ‘surprise’,
      6: ‘neutral’
      }

      autokeras_predictions = autokeras_model.predict(x_test)

      for i in range(0, len(autokeras_predictions)):
      prediction_classes = autokeras_predictions[i]

      max_index = np.argmax(prediction_classes)
      print(“emotion: “,class_names[max_index])

      1. Yes thx,
        But it’s exactly what I’m doing, but “autokeras_predictions = autokeras_model.predict(x_test)” return only an array of one element (len(autokeras_predictions) == 1), not the array of probabilities of 7 expressions… but only the index of the detected emotion
        Does it come from the model you “fer_autokeras_model.h5” ?

          1. Okay, so it’s not possible with auto-keras to have the probabilities of all expressions for now, that’s right ?

          2. I think unfortunately there is not because we fed target values as numeric values instead of multi classes in training step.

  5. Hello,
    I have a question about the function of auto-keras.Recently, I find auto-keras can process 1D CNN,like ImageRegressor1D. I want it has two different 1D input and one output value. Is it possible to achieve this?
    Also, thanks for your great work.

  6. Hello,
    I wanted to see your model, but the link to the Google Drive with the model doesn’t work. Google Drive says that URL doesn’t exist. Can you provide the model in another way?
    Thank you for your answer in advance.

Comments are closed.