Item — HN Reader

What? Yes the modelling technique falls under "deep learning" but it still very much processes language and language only, making it NLP.

Yes yes, language modelling ends up being surprisingly powerful at scale, but that doesn't make it not language modelling.

1 comments

A couple of years of this LLM AI hype train has blinded people to what was actually surprising about LLMs. The surprise wasn't that you could make a language model and it wasn't that a language model could generate text. Those are both rather pedestrian observations, and their implementations are trivial. The surprise of LLMs was that contemporary hardware could scale this far and that an un-curated training set turns out to contain a statistically significant amount of truth. Deep learning was interesting because we didn't expect that amount of computation to be feasible at this time in human history, not because nobody had ever thought of it before.

The surprise of the LLM AI was that they were somewhat truthful at all.

The AI revolution has mostly been a hardware revolution. I studied AI in the 1990s, so I knew about neural networks and backpropagation, so when suddenly everybody was talking about "Deep Learning", I wanted to know what was different about it. Turns out: not much. It's mostly just plain old backpropagation on a much larger scale because we have more powerful hardware.

Of course there have still been plenty of meaningful innovations, like the transformer/attention thing, but it's mostly the fact that affordable graphics cars offer massively-parallel floating point calculations which turns out to be exactly what we need to scale this up. That and the sheer amount of data that's become available in the age of the Internet.

>The AI revolution has mostly been a hardware revolution.

It's certainly important but this reads as overly simplistic to me. All the hardware we have today won't make an SVM or a random forest scale the way transformers do.

Neural networks and backpropagation were known in the 90s too.