I think the discussion has to be more nuanced than this. "LLMs still can't do X so it's an idiot" is a bad line of thought. LLMs with harnesses are clearly capable of engaging with logical problems that only need text. LLMs are not there yet with images, but we are improving with UI and access to tools like figma. LLMs are clearly unable to propose new, creative solutions for problems it has never seen before.
back
8 comments
> LLMs are clearly unable to propose new, creative solutions for problems it has never seen before.
LLMs are incredibly useful but I'm not sure about this statement.
It is proposing stuff that I haven't seen before, but I don't know about it is new or creative from the entirety of collective human knowledge.
I'm not sure if you misread the statement you quoted or I'm misreading yours, but it doesn't sound like you're really disagreeing with their point. Did you miss the "un" in "unable", or am I misunderstanding you as also saying that you don't consider them to be creative?
Yeah it was my bad, thanks for pointing it out. For some reason I read that as "able", can't unsee it or understood how it happened.
> LLMs with harnesses are clearly capable of engaging with logical problems that only need text.
To some extent. It's not clear where specifically the boundaries are, but it seems to fail to approach problems in ways that aren't embedded in the training set. I certainly would not put money on it solving an arbitrary logical problem.
> To some extent. It's not clear where specifically the boundaries are, but it seems to fail to approach problems in ways that aren't embedded in the training set. I certainly would not put money on it solving an arbitrary logical problem.
In what way can you falsify this without having the LLM be omniscient? We have examples of it solving things that are not in the training set - it found vulnerabilities in 25 year old BSD code that was unspotted by humans. It was not a trivial one either.
Here's an odd example of testing, but I design very complex board and card games, and LLMs are terrible at figuring out whether they make sense or really even restating the rules in a different wording.
I thought they would be ideal for the job, until I realized that it would just pretend that the rules worked because they looked like board game rules. The more you ask it to restate, manipulate or simulate the rules, the more you can tell that it's bluffing. It literally thinks every complicated set of rules works perfectly.
> it found vulnerabilities in 25 year old BSD code that was unspotted by humans.
I don't think the age of the code makes the problem more complex. Finding buffers that are too small is not rocket science, bothering to look at some corner of some codebase that you've never paid attention to or seen a problem with is. AI being infinitely useful (cheap) to sic on pieces of codebase nobody ever carefully looks at is a great thing. It's not genius on the part of the AI.
Re: cheap - Anthropic’s write-up said it cost $20,000 of runs to find that bug (and a few others). So not that cheap compared to other tools - more similar in cost to human review/pentest, but probably more exhaustive.
> This was the most critical vulnerability we discovered in OpenBSD with Mythos Preview after a thousand runs through our scaffold. Across a thousand runs through our scaffold, the total cost was under $20,000 and found several dozen more findings.
They don’t talk about the other findings, so I’m guessing they are minor.
> Here's an odd example of testing, but I design very complex board and card games, and LLMs are terrible at figuring out whether they make sense or really even restating the rules in a different wording.
I'm positive that they are perfectly fine and will a pretty good job. Did you actually try it?
Eh, I can see their point, I think. The models can restate the rules differently, I'm sure, but it sounds like the GP is saying that LLMs can't tell whether the rules are well-balanced.
It would be interesting to see some example problems along those lines. Design some games with complex rules, including one or two of the most subtle game-wrecking bugs you can think of, and ask the models if they can spot them.
In fact that sounds more interesting the more I think about it. Intensive RL on that sort of thing might generalize in... let's say useful ways.
I would love to see examples but I think we won’t. I’m happy to be proven wrong that an llm will do worse than a fairly smart human (without prior experience in the board game).
I'm just saying I'd rather hire a human that can be reasoned with than rely on software that can't be. At least where reasoning is involved.
Granted, I don't do a lot of needle-in-the-haystack work like finding vulnerabilities where search will naturally dominate.
Also, I imagine most reasoning involved in exploits will be found in the training sets—there are only so many patterns of exploitation found in formal languages.
Solving arbitrary logical problems seems to be equivalent to solving the halting problem so you are probably wise not to make that bet.
> LLMs are not there yet with images
https://genai-showdown.specr.net/image-editing
There's been a lot of progress there, it's just that an LLM that's best for, say coding, isn't going to be also the best for image edit.
To be clear, image generation models are not in general LLMs although most now use an LLM as a text encoder.
Harnesses could have solved things like the bathroom remodel, maybe, but the main point about how LLMs don't understand is the key here. You can make chatgpt better at rendering 3d scenes but you can't make it think, not really. Reasoning was only ever a feedback loop.
Anyone who has worked with LLMs has experienced all the issues he talks about here, we're either optimistic and imagine they'll be fixed, or we're pessimistic and we say they are inherent to the nature of the technology and will never be fixed
> LLMs with harnesses are clearly capable of engaging with logical problems that only need text.
All of the LLMs are bad at music. They get intervals wrong. They list unsuited songs.
I would not trust them with any domain until proven otherwise.
> LLMs with harnesses are clearly capable of engaging with logical problems that only need text.
> LLMs are clearly unable to propose new, creative solutions for problems it has never seen before.
How do you reconcile this with this article that the author linked? It's not a novel problem, and it's only text: https://medium.com/the-generator/one-word-answers-expose-ai-...
I guess it's a form of engagement to give a wildly wrong answer, but I'm not convinced that the extra nuance you've introduced is really all that nuanced either.
The author of the medium article specifically hobbled the models to stop them thinking it through and got a wrong answer but that would happen with humans too and doesn't prove much.
I would argue that most humans would either give the correct answer or just say "I don't know". Some of them might confidently give the wrong answer, but humans will readily refuse to follow instructions in plenty of circumstances where they decide they aren't worthwhile. LLMs don't do this, and I'd argue that the ability to reject premises is fundamental to engaging with things in a truly logical way.
More nuanced than what? It’s just impossible to read the article as claiming "LLMs are idiots". Its whole point is that LLMs are simultaneously astonishingly smart and utterly stupid, and critically the boundary between those is complex and unintuitive which makes trusting their output so precarious.
> "LLMs still can't do X so it's an idiot"
Let’s be careful. That’s a straw man. I don’t know anyone who says that. Aphyr says in the article that AIs can do things. But they have been marketed as “intelligent,” and I agree with Aphyr that the word is suggesting way more than AIs currently deliver. They do not reason and they do not think and are not truly intelligent. As the article says, they are big wads of linear algebra. Sometimes, that’s useful.
> They do not reason
How do you disprove it?
We know that they do not reason because we know the algorithm behind the curtain. The model is generating the next token via model weights and some randomness. That’s all. It not reasoning. Sometimes it has an appearance of reasoning, but not if you know how it works. It doesn’t matter that the model manufacturer marketing department slaps a “Reasoning!” sticker on the side of the model. It’s not actually doing that. As an analogy, sometimes a stage magician in Las Vegas makes it seem that he’s making a woman disappear and a tiger appear in her place, but we all know that’s not what is really happening; It’s just a clever trick.
Well, could you define what reasoning actually means? What would an AI need to do to be considered capable of reasoning? What is the core difference between what we do that is considered reasoning verse what AI currently does that is not considered reasoning?
To be clear, I am not making a statement as to whether AI reasons or not. Its just slippery to say something isn't or can't do X when we can't really define X. Perhaps if we can put it down as an outcome rather than an, in my opinion, currently impossible to accurately define characteristic of a thing.
In many examples, LLMs betray the fact that they are not reasoning, because when provided with problems that can be solved with the ability to reason, they fail.
Even in this discussion someone provided an example of coming up with board game rules. LLMs found all board game rules valid, because they looked and sounded like board game rules. Even when they were not.
In short, You can learn a subject, you can make a mental model of it, you can play with it, and you can rotate or infer new things about it.
LLMs are more analogous to actors, who have learnt a stupendous amount of lines, and know how those lines work.
They are, by definition, models of language.
IF you want a better version - GENAI needs to be able to generate working voxels of hands and 3D objects just from images.
I don’t believe the board game rules example. I think this would be a piece of cake for an llm. I’m happy to be proven wrong here if you share an example.
This is the user I took the example from: https://news.ycombinator.com/item?id=47689648#47696789
>We know that they do not reason because we know the algorithm behind the curtain.
In other words, we didn't put the "reasoning algorithm" in LLMs therefore they do not reason. But what is this reasoning algorithm that is a necessary condition for reasoning and how do you know LLMs parameters didn't converge on it in the process of pre-training?
Model parameters are weights, not algorithms. The LLM algorithm is (relatively) fixed: generate the next token according to the existing context, the model weights, and some randomization. That’s it. There is no more algorithm than that. The training parameters can shift the probabilities for predicting a token given the context, but there’s no more to it than that. There is no “reasoning algorithm” in the weights to converge to.
This overly reductive description of LLMs misses the forest for the trees. LLMs are circuit builders, the converged parameters pick out specific paths through the network that define programs. In other words, LLMs are differentiable computers[1]. Analogous to how a CPU is configured by the program state to execute arbitrary programs, the parameters of a converged LLM configure the high level matmul sequences towards a wide range of information dynamics.
Statistics has little relevance to LLM operation. The statistics of the training corpus imparts constraints on the converged circuit dynamics, but otherwise has no representation internally to the LLM.
> LLMs are circuit builders
I think they are circuit "approximators". In other words, a result of a glorified linear regression..
And the brain is just a complicated chemical reaction.
what if eventually the physical mechanism for human consciousness becomes fully understood? does understanding that process mean we are no longer intelligent?
What is reasoning?