Neural networks are like swiss army knifes. They can solve both classification and regression problems. Surprisingly, they can also contribute unsupervised learning problems. Today, we are going to mention autoencoders which adapt neural networks into unsupervised learning.
Road map
A more complex data set will be covered in this post whereas a simpler data has been covered in the following video. Autoencoding layer has 2 outputs. In this way, we can show results in a 2-dimensional graph. Results are very satisfactory!
🙋♂️ You may consider to enroll my top-rated machine learning course on Udemy
Autoencoder Neural Networks
They are actually traditional neural networks. Their design make them special. Firstly, they must have same number of nodes for both input and output layers. Secondly, hidden layers must be symmetric about center. Thirdly, number of nodes for hidden layers must decrease from left to centroid, and must increase from centroid to right.
The key point is that input features are reduced and restored respectively. We can say that input can be compressed as the value of centroid layer’s output if input is similar to output. I said similar because this compression operation is not lossless compression.
Left side of this network is called as autoencoder and it is responsible for reduction. On the other hand, right side of the network is called as autodecoder and this is in charge of enlargement.
Let’s apply this approach to handwritten digit dataset. We’ve already applied several approaches for this problem before. Even though both training and testing sets are already labeled from 0 to 9, we will discard their labels and pretend not to know what they are.
Constructing network
Let’s construct the autoencoder structure first. As you might remember, dataset consists of 28×28 pixel images. This means that input features are size of 784 (28×28).
model = Sequential() model.add(Dense(128, activation='relu', input_shape=(784,))) model.add(Dense(32, activation='relu')) model.add(Dense(128, activation='relu')) model.add(Dense(784, activation='sigmoid'))
Autoencoder model would have 784 nodes in both input and output layers. What’s more, there are 3 hidden layers size of 128, 32 and 128 respectively. Based on the autoencoder construction rule, it is symmetric about the centroid and centroid layer consists of 32 nodes.
Training
We’ll transfer input features of trainset for both input layer and output layer.
model.compile(loss='binary_crossentropy', optimizer='adam') model.fit(x_train, x_train, epochs=3, validation_data=(x_test, x_test))
Both train error and validation error satisfies me (loss: 0.0881 – val_loss: 0.0867). But it would be concrete when it is applied for a real example.
Testing
def test_restoration(model):
decoded_imgs = model.predict(x_test)
get_3rd_layer_output = K.function([model.layers[0].input], [model.layers[1].output])
for i in range(2):
print("original: ")
plt.imshow(x_test[i].reshape(28, 28))
plt.show()
#-------------------
print("reconstructed: ")
plt.imshow(decoded_imgs[i].reshape(28, 28))
plt.show()
#-------------------
print("compressed: ")
current_compressed = get_3rd_layer_output([x_test[i:i+1]])[0][0]
plt.imshow(current_compressed.reshape(8, 4))
plt.show()
Even though restored one is a little blurred, it is clearly readable. Herein, it means that compressed representation is meaningful.
We do not need to display restorations anymore. We can use the following code block to store compressed versions instead of displaying.
def autoencode(model): decoded_imgs = model.predict(x_test) get_3rd_layer_output = K.function([model.layers[0].input], [model.layers[1].output]) compressed = get_3rd_layer_output([x_test]) return compressed com = autoencode(model)
Clustering
Notice that input features are size of 784 whereas compressed representation is size of 32. This means that it is 24 times smaller than the original image. Herein, complex input features enforces traditional unsupervised learning algorithms such as k-means or k-NN. On the other hand, including all features would confuse these algorithms. The idea is that you should apply autoencoder, reduce input features and extract meaningful data first. Then, you should apply a unsupervised learning algorithm to compressed representation. In this way, clustering algorithms works high performance whereas it produces more meaningful results.
unsupervised_model = tf.contrib.learn.KMeansClustering( 10 , distance_metric = clustering_ops.SQUARED_EUCLIDEAN_DISTANCE , initial_clusters=tf.contrib.learn.KMeansClustering.RANDOM_INIT) def train_input_fn(): data = tf.constant(com[0], tf.float32) return (data, None) unsupervised_model.fit(input_fn=train_input_fn, steps=5000) clusters = unsupervised_model.predict(input_fn=train_input_fn) index = 0 for i in clusters: current_cluster = i['cluster_idx'] features = x_test[index] index = index + 1
Surprisingly, this approach puts the following images in the same cluster. It seems that clustering is based on general shapes of digits instead of their identities.
So, we’ve mentioned how to adapt neural networks in unsupervised learning process. Autoencoders are trend topics of last years. They are not the alternative of supervised learning algorithms. Today, most data we have are pixel based and unlabeled. Some mechanisms such as mechanical turk provides services to label these unlabeled data. This approach might help and fasten to label unlabeled data process. Finally, source code of this post is pushed to GitHub.
Like this blog? Support me on Patreon
