Item — HN Reader

Back in the day it was a bit of a cliche to bring up “clever Hans”, the horse that could do math, when talking about machine learning. He couldn’t do math but he read some cues from his handler of pick the write answers, the handler iirc wasn’t in on it.

The point of the story was that classifiers can be right for the wrong reasons and almost inevitably are. At least there’s zero guarantee that the reason for making the prediction matches the human or “real” reason why it’s correct.

LLMs are classifiers, there is absolutely no reason to assume they’re any different, regardless of any reasoning tokens they emit. They do what their handler wants to see, that’s all, and that’s what they’re trained to do.

People often take this as a knock against them. It isn’t, it’s just the reality of neural network classifiers. The results speak for themselves and don’t depend on whether they “actually” reason, but all evidence says they don’t, or at least there’s no special reason why they would.

5 comments

Reasoning is a term that's like the term intelligence in that it's highly complex and much less well defined than we'd like it to be, especially as we're trying to compare algorithms to human behaviors.

To begin to explore this we have to imagine a reasoning oracle, that when presented with a question that can be reasoned will immediately present a correct answer. Immediately from this definition we can see that humans are not perfect reasoning oracles. Also with definition of reasoning, any agent with generalized capabilities would quickly reach some kind of singularity or solution to the universe as much as is possible within this universe.

We've established that we are imperfect reasoners, the next question ends up being how do you measure reasoning capabilities? This comes with a second question of when does following an algorithm become reasoning? Animals for example have a large amount of instinctual reasoning as an inherit part of their architecture that self trains when presented with the correct environment. What does the gradient between low level reasoning and high level reasoning look like?

Coming back to humans, we know of countless example of where human reasoning was right for the wrong reasons. We also have seen many different paths to the same solution. For the egotistical it would be easy to say "well if you didn't do it they way I did, you are wrong", but that's not very scientific. Defining the topology of the problem is a more scientific way of looking at it, but quickly runs into P=NP and Kolmogorov complexity issues.

The last thing I'll touch on is why did human society begin to rapidly change after we started following the scientific method. Looking at the method in hindsight, it seems like a pattern that a reasoning agent should have been able to distil and follow around the time it reaches what we consider higher level reasoning. Therefor I'm highly resistant to consider human reasoning "real" reasoning versus any other possible path to reasoning. Human reasoning is simply what we are comfortable with. This also leads me to consider the worrying possibility there are far more efficient and faster means of reasoning that I as a human will be very uncomfortable with.

You bring up good questions, but you are making an assumption here:

> The last thing I'll touch on is why did human society begin to rapidly change after we started following the scientific method. Looking at the method in hindsight, it seems like a pattern that a reasoning agent should have been able to distil and follow around the time it reaches what we consider higher level reasoning.

Why human society changed at the time the scientific method was codified is a confluence of things and I don't think the scientific method should be given an outsized role. The scientific method goes hand in hand with peer review, publishing of methods, a common scientific language, an educated leisure class, and the development of tools for observation. The method itself is just a practical way to utilize these developments and is as much a result of the situation as any other technology. It did not get people to think differently about problems; we had navigated seas, moved armies, built wonders, produced math and calculated the circumference of the globe just fine without it.

Arguably it was the abandoning of the alchemist tradition of secrecy that was the revolution that enabled our modern world more than a specific codification of method.

LLMs are not classifiers. A classifier is an algorithm or neural net that assigns a label from a fixed set of labels to an input.

You can broaden the definition of classifier to anything that internally divides its input space into regions, but that definition would include every neural network, whether biological or artificial. So it's not very meaningful, and certainly doesn't give any insight into how they differ from humans.

It's pretty meaningful actually. Because biological neurons don't sit alone. There's a third player in every synapse: the astrocyte. And astrocytes form a large syncitium, a collection of cells that are "open" to each other through gap junctions. This syncitium has gotten more extensive over evolution, and human astrocytes (which are larger and more complex) when transplanted in mice, make the mice smarter at maze tasks etc.

So every neural network is actually entertained in an astrocytic syncitium. Every synapse is being "listened to" and modulated by it's astrocytic end foot (which forms the third arm of what we now call the tripartite synapse).

If neurons are about breaking down signals, the astrocytes are the slow, steady integrators that (potentially) can serve as the global workspace that scientists have been theorizing but not yet found.

So there actually is a problem with modern neural networks, in so far as comparisons with human brains go: they're missing the design spec for literally half of it.

Which is fine. Connectionist models come from a time when the height of neuroscience said neurons were the bees knees and glia were just support staff. Today, astrocytes, and other glia, are forcefully back in the conversation. And they don't seem to do anything like neural networks.

But a LLM literally does assign a token from a fixed set of tokens (its vocab) to an input, rinse and repeat, until the stop token.

Classifiers have been giving logits since decades ago.

Sure, but then there's no such thing as a network that isn't a classifier. Every physically computable function that terminates in finite time will map an input to a fixed set of outputs. And it goes against the common usage, where in machine learning we talk about classifiers, regressors, generative models, etc. as different things. They all become classifiers.

The parent commenter was trying to draw some insight from LLMs being classifiers that wouldn't apply equally to everything else.

You couldn't possibly successfully do clever hans for novel mathematical proofs and counter-examples that don't exist yet, however here we are.

Generalization in successful results is enough to close this argument for me.

Aren't humans also classifiers? Where, precisely, is the dividing line between a sufficiently large model and an intelligence?

Is that all humans are? When clockwork was the frontier, people thought the brain was akin to clockwork (which is why the mechanical turk fooled people). By analogy, now that classifiers are the frontier, we think the brain is a classifier. Our model of the mind is whatever the most complex artifact of our era is .

Just to be clear: I do think the brain classifies. But I wonder if it does other operations as well that we don't fully understand.

The mechanical clock or a von-neumann based computer were never as similiar as neural networks.

We can find similiar structures in brain and in neural networks.

The real question is only if the complexity of the structure of the brain is critical and how we can mimik it or if we can make it appear through learning architecture.

> They do what their handler wants to see, that’s all, and that’s what they’re trained to do.

This sounds like a tautological argument that doesn't really mean anything. Hypothetically, in a near future a super-powerful AI could bring something truly astonishing (say, practical fusion power, or cure all cancer) and one could still argue "They do what their handler wants to see, that's all."

So what does that even mean.

They are already producing truly astonishing results. The unit distance problem and Jacobian conjecture are world-famous problems that stood unsolved for 80 and 87 years. Entire careers have been spent on them. Yitang Zhang for example spent his entire PhD trying and failing to prove the latter in the 2-dimensional case.