Really nice paper.

In the 1980s at SAIC, we spent a small fortune designing custom (Harvard memory architecture) hardware to do back-prop neural networks relatively fast, but fast back then was about 23m/sec FP operations. My droid phone could probably do more than that now :-)