That’s just conflating reasoning and “parallel reconstruction” - there is such thing as reasoning, and I agree it’s probably less common in decision making, but it’s fundamental to many tasks where we figure out a solution, like writing an algorithm, math proofs, etc.
back
5 comments
Is it really?
I think the assumption that there is some sort of formal ‘reasoning’ process that is inherently superior to whatever ‘non-reason-based’ decision process ‘non-reasoning’ systems use is not immediately justifiable.
Most reasoning is ultimately based in heuristics: this is the right thing to do next because it has worked in the past. This is the right thing to do because our intuition tells us doing it will lead to this desirable outcome.
When you’re formulating a math proof, sure, the steps you are laying out make up a ‘reasoned’ argument but how you choose what steps to take is wholly intuitive. Why assume the contrapositive to begin this proof? It’s worked in the past and it might work here.
It’s all pattern matching trained on a reward function.
It's true that humans are not perfectly logical actors, but even so, most humans won't tell you seriously that the swimming pools in the Titanic are empty, or that glue is a great pizza topping.
At the end of the day, there's a difference between showing your work for a math problem by writing down each logical step as you work through it, vs. just blindly copying both the answer and the "show your work" part from your classmates.
That humans can fail at math, or at other kinds of logic problems, is immaterial. It's still an entirely different process, and it leads to different results.
The fact that humans often fail because they're actively prioritizing something else can also be for the best. Like, it's reasonable to get a math problem wrong because you're distracted by argueing classmates who might be about to start a fight right next to you. Or you might realize that the premise of a question itself is wrong. Or you might logically know that the Earth goes around the sun, but decide to pretend otherwise because it's not worth dying over.
Humans are always balancing priorities, and it doesn't always make sense to judge their success by a single metric like "do these math problems and show your work within 10 minutes". LLMs do not have these conflicts, even when we would want them to.
There is explicit reasoning where we externalise steps, but it is slow and cumbersome. The point is that when we make a decision without externalising each step, and then afterward explain why and how we made that decision, there is no evidence that explanation is true, and a lot that it is a retrospective fiction.
Sure, but the same must be true for these models as well. They must be doing some form of reasoning, even if it's not formally correct and it relies instead on all kind of half-assed heuristics which only work inside certain unexamined boundaries, which leads to wrong conclusions when those boundaries are being trespassed, etc. And even if it's not the reasoning they report when asked. But all this can't be that different to how humans reason most of the time either.
I suspect humans have other ways that help with error correction and guiding the reasoning effort, but that's another story.
> They must be doing some form of reasoning, even if it's not formally correct and ...
Surely it is worthwhile to attempt to understand the details of that? And if we seek human equivalent performance then it is reasonable to wonder if the reasoning achieved to date is the "correct" sort.
Maybe, but I suspect that a lot of the actual things that produce the algorithm, proof, etc are not actually explicit reasoning. Q: "Why did you try that vs some other thing at this point?" A:"seemed right to do"/"kinda looks like other stuff I've seen before"/"first thing that came to mind" etc.
Why pursue this one avenue vs some other, typically its cause it seemed more promising, and the person can come up with reasons but did they explicitly verbalize a fully sound chain of thinking at the time? Probably not.
Not to say that the explicit thinking, or writing things down, isn't important. But if I examine the process by which I develop a proof or something, there's a lot of vague hunches, blind alleys, etc that come along the way. And, many of the blind alleys etc probably aren't actually that important in the end for me finding the right answer -- if you were to trim that part out of my own internal reasoning trace but left the rest intact, I'd still get the right answer because, well, it was a blind alley.
Exactly. Very little of what we do is "explicit thinking", because it's slow to have to develop and verbalise those arguments step by step. Most of it is "hunches" that we then retroactively justify if asked, and those retroactive justifications were demonstrated not to be trustworthy by Sperry's experiments. We don't even have evidence to suggest that humans can reliably introspect our past decisions, but we do have evidence to suggest we readily manufacture them.
Of course, but it is rare.