Item — HN Reader

For those of you who want to learn the nuts and bolts of deep neural networks, Andrew Ng's tutorial on Unsupervised Feature Learning and Deep Learning is getting older but still great: http://ufldl.stanford.edu/wiki/index.php/UFLDL_Tutorial

The best research results from 2014 and 2013 make less use of the unsupervised techniques than initially expected, so I would start by focusing on the below sections, which focus more on supervised learning with deep neural networks:

Sparse Autoencoder: Neural Networks, Backpropagation Algorithm

Building Deep Networks for Classification: Deep Networks: Overview, Fine-tuning Stacked AEs

Working with Large Images: Feature extraction using convolution

You'll need some background in matrix algebra, calculus, and probability to understand this. Having taken a previous machine learning course, although not strictly necessary, is probably extremely helpful--I'd recommend taking any standard course on ML on Coursera or Udacity, or going through any standard textbook.

EDIT: I almost forgot that Michael Nielsen (who wrote the standard textbook on quantum computation) is also writing a free online textbook on Neural Networks and Deep Learning. Chapters 1-4 are currently available and would get you pretty far: http://neuralnetworksanddeeplearning.com/

2 comments

Bookwise, Yoshua Bengio, Aaron Courville, and Ian Goodfellow are nearly finished with their MIT Press book on deep learning: http://www.iro.umontreal.ca/~bengioy/dlbook/ . It is pretty strong on the true theory of what is going on in deep networks, and has fairly good intuition for how and why things work. Paired with the deep learning tutorials http://www.deeplearning.net/tutorial/, as well as the content from UFLDL it is a pretty strong foundation for advanced study.

Michael's book seems to target a more introductory level - a beginner might be better off to start with that, follow with Andrew Ng's ML course, which has a section on neural nets including an assignment implementing backpropagation, then continue with the deep learning book and the {deep learning, UFLDL} tutorials. This should be solid enough to at least read most of the cutting edge work and papers, if that is the aim.

Hugo Larochelle's youtube course https://www.youtube.com/playlist?list=PL6Xpj9I5qXYEcOhn7Tqgh... and Hinton's coursera course https://www.coursera.org/course/neuralnets are also great references.

Didn't realize Yoshua & co had a book coming! That would definitely be the one to read.

BTW, for anybody who wants to learn machine learning in general, Kyle's blog also seems to be packed full of clear explanations with working demo code: http://kastnerkyle.github.io/

Very nice!

Thanks for checking it out! I am planning to add a few deep learning related posts during the holidays. The recent results for NLP, captioning and speech using encoder/decoder models are just too cool not to demo.

Thanks for the resources, my current hobby project is attempting to recreate deep learning in Clojure, have been burning through material, cheers!