In a recent talk[1], Bill Dally of Nvidia suggests using actual logarithms, as they're more consistent in relative error than floats, which is especially relevant at low precision as increasingly used in AI. Multiplications then become simple additions, but then addition becomes more complicated, as in general it needs lookup tables. This is the same mechanism as a slide rule, and also reminds me of the way the Yamaha DX7 works. The lookup tables can be expensive, and the talk contains an idea how to batch them when adding up the results of many many multiplications (as is also often the case for AI workloads).
back
1 comments
I would guess that additions are pretty important in deep learning workloads. For backpropagation, accumulator on the big delta matrix is very important.