Auto-encoding features is a useful technique in unsupervised deep learning. It not only helps in latent feature learning but can also significantly reduce dimensionality by reducing noise.
Autoencoder (single layered) : An autoencoder takes the raw input, passes it through a hidden layer and tries to reconstruct the same input at the output layer. So basically, it works like a single layer neural network where instead of predicting labels you predict the input at the output layer. Hence the loss you calculate is between raw input you provided and your predicted input at the output layer. Minimizing this loss should then lead to more or less the same input being predicted at the output. A simple intuition behind using autoencoder is to learn latent features from the raw features while retaining the capability to produce the raw input back from the latent features which you get at the hidden layer. Denoising refers to the intentional addition of noise to the raw input before providing it to the network. Some of them are gaussian and masking noises. But when you calculate loss, it should be between the predicted input and the original input only (not the noisy input).
Stacked (Denoising) Autoencoder It is an autoencoder which has multiple layers except that its training is not the same as that of a multi layered neural network. The process, in short, is carried out as follows:
You provide noise to the input. After propagating through one hidden layer, the output is generated and loss is calculated between this output and the original input. You continue the training until it reaches convergence. Then finally you pass the full data through this network and collect the data present in the current hidden layer. This is your new input. You take this latent input and pass noise to it and follow the same procedure thereafter with the successive hidden layers. Ultimately after you are done with the last layer, the data collected in this last hidden layer is now your new data. The whole process is also referred to as layer-wise training.
I wrote a simple library for stacked autoencoder that will do the whole job for you. We use it for some of our projects at Artifacia. You can use it too in your own projects and let me know know what you build using it!
My main motivation behind building a library for deep denoising autoencoders was to facilitate the process of auto-encoding features with ease.
Auto-encoding is kind of similar to a feed-forward neural network except it is trained in a layer by layer fashion and more hidden features are learned successively. The implementation of libsdae is very high level. All you have to do is just call the method with proper parameters.
from deepautoencoder import StackedAutoEncoder model = StackedAutoEncoder(dims=[5,6], activations=['relu', 'relu'], noise='gaussian', epoch=[10000,500], loss='rmse', lr=0.007, batch_size=50, print_step=2000) # usage 1 - encoding same data result = model.fit_transform(x) # usage 2 - fitting on one dataset and transforming (encoding) on another data model.fit(x) result = model.transform(np.random.rand(5, x.shape))
More technical documentation is provided in the readme of the repository
Proper choice of loss functions - like for data that consists of bit vectors ([0, 0, 1, ...]), cross-entropy will be a better option whereas for real-valued data root-mean-square error is more appropriate.
Since the last layer trained in the autoencoder is the feature that we carry forward for our next steps, such as classification, the choice we make for the dimension of this layer is very crucial.
libsdae is completely flexible in the sense that it reduces the frequent process of autoencoding features to a simple one library call, entitles you to denoise the network with two noises (mask and gaussian) currently. In a fast growing world where people readily want to make predictive models as fast as possible, applying libsdae at the start of the ML pipeline with suitable parameters will generate more latent features for your task without any hassle.
I started working on libsdae with the intention of contributing to the popular machine learning framework TensorFlow by Google even before joining Artifacia. I didn't have resources back then and it was hard to maintain it all by myself. It is now a part of Artifacia Open Source projects and now I have a team that wants to support its further development with the community.
P.S. The library is fully written fully using the TensorFlow and hence TensorFlow (>=0.7) is required and the library supports both python 2.7x and 3.x.
Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion by P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio and P. Manzagol (Journal of Machine Learning Research 11 (2010) 3371-3408)