We show how nonlinear embedding algorithms popular for use with shallow semi-supervised learning techniques such as kernel methods can be applied to deep multi-layer architectures, either as a regularizer at the output layer, or on each layer of the architecture. This provides a simple alternative to existing approaches to deep learning whilst yielding competitive error rates compared to those methods, and existing shallow semi-supervised techniques.

We then go on to generalize this approach to take advantage of sequential data: for images, and text.

For images, we take advantage of the temporal coherence that naturally exists in unlabeled video recordings. That is, two successive frames are likely to contain the same object or objects. We demonstrate the effectiveness of this method in a semi-supervised setting on some pose invariant object and face recognition tasks.

For text, we describe a unified approach to tagging: a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, and semantic roles. State-of-the-art performance is attained by learning word embeddings using a text specific semi-supervised task called a language model.

Joint work with: Ronan Collobert, Frederic Ratle, Hossein Mobahi, Pavel Kuksa and Koray Kavukcuoglu.