back
7 comments
Seeing the dramatic differences in scores just going from high to xhigh is just another demonstration of the bitter lesson: Just keep scaling search and learning. We are probably going to need a lot more GPUs.
These aren’t raw base models they are the result of a ton of RLHF and various adjustments.

Bitter lesson wildly overstated in this context.

rlhf = reinforcement learning from human feedback

(had to look it up)

More RLHF is in fact scaling.
Not sure where I implied they are "raw base models" and not sure what "various adjustments" means here or how "ton of RLHF" contradicts anything. If we look at research for open source models, "adjustments" usually come in the form of efficiency gains which directly contributes to the ability to scale or synthetic data pipelines to increase the dataset and increasing the context window.
Not just RLHF but also RLVR, and isn't that the litter lesson though?

My sense of the Sutton Dwarkesh interview was that he was calling out that he didn't mean just longer datasets, but rather learning through exploration and that's exactly RL.

The scaling with reasoning models is more and more with things like verifiable rewards (coding and math), in line with bitter lesson and also Sutton invented lots of modern RL.
RLHF is an increasingly small part of training though? From what I understand most of the capability gain is in RLVR
Nah, the last few generations have more RLVR in the data mix. Which is more CPU intensive and very much amenable to the bitter lesson as you can reduce the loss by doing more rollouts in your tool environment.
While I think this is true, remember as we get more efficient we just decide to scale even bigger. So more GPUs, and more efficient.

I agree with the sibling comment, effiency is probably the more important component at this point. We are hitting not just a practical engineering roadblock for scaling with current technology, I think we have definitely hit a financial and logistical roadblock for up scaling with the number of GPUs (on an immediate basis)

There goes my plan to buy a PC for the next decade
The whole of knowledge work is being automated. We've barely begun to see the GPU build out. This is just the start.

I'd imagine they're going to 10x this, maybe 100x this.

I actually think we're in a strange situation with AI compute.

Right now, we have models that are statistical models of language, with a world model and reasoning "falling out" of a lot of effort.

It's like we've made something that's a little bit intelligent, and now we're trying to amplify that trick to create something that's quite intelligent. And - don't get me wrong - it works.

But it's also super, super inefficient. We're having machines "think out loud" to compensate for the quality of their thought processes. We elongate the path to make up for the progress made on a given step.

I tink there's probably a much smarter way of doing things that will require qualitative architectural (and quite possibly hardware) innovations. Right now we're on the path to a Dyson sphere: that's probably not going to be necessary once we figure out a smarter way to think.

Kind of refreshing though that the "throw more processing at it" scaling we saw in the 90s has returned in a different way. For a while we were really bottlenecked in our advances by relatively low levels of parallelism (most software used by your average user doesn't scale cleanly with more than a few threads).
Not always, in some cases, changing to a higher reasoning makes the AI doubt itself too much, and skip over the correct answer by overcomplicating the problem and polluting the context.

It would be nice to see on which categories of problems the extra thinking makes it better and on which it makes it worse.

This shows up in OpenAI's graphs on their announcement page. There is a peak performance datapoint in the graphs past which (to the right on the graph indicating more resources spent) peformance declines. And it's on every graph on that page!
I think I have this problem but with my human brain.
> We are probably going to need a lot more GPUs.

Or a breakthrough in algorithms etc.

The human brain, heck all bio brains, are proof that you don't need a lot of power or size for intelligence.

The human brain has 80 billion neurons and a 100 trillion synapses. I think you're underselling the processing power of that warm chunk of meat.

The real message of the last 15 years has actually been the opposite: if you throw enough processing power at it, intelligence emerges.

20 watts for inference AND training!
I mean, theoretically you can solve every finitary problem with a brute force solution...

Richard Sutton specifically states that the search has to be smart. We know that the brain uses recurrent connections and is shallow. I think a lot more money has to go into architecture. Feed Forward transformers can only scale so far

Or a lot better efficiency.
And dozens of data centers in every state so tokens are dirt cheap.
Or a new model. The human brain does far more with far less.
This isn’t really how it works anymore. Agents rely heavily on tool use and the agentic harness to perform tasks. Pre-training is no longer very effective.
> Dramatic difference

Isn't this just the difference between getting 0 right and getting 1 right?

I said a few months ago, "man, Opus is great, but sometimes when talking with it I have the feeling like, this thing should be about 10 times bigger."

When Mythos was announced after that, I was pleasantly surprised to hear about it. But when it turned out to be only two times bigger, I was a little disappointed!

(I am even more disappointed with the safety filters, but that's kind of a separate discussion... "Fortunately" I find that I can usually edit my prompt by single character and get through...)

We’re definitely going to need a lot of Gpu’s
And a lot more electricity to power them.
We have it slightly ahead of Fable in our multi-agent coding evaluations.

Fable's main advantage is that its average solution size is smaller. However, GPT 5.6 Sol is a substantial improvement from GPT 5.4/5.5 which would write verbose, defensive code. 31KB for GPT 5.4/5.5 down to 26KB for GPT 5.6 Sol, with better performance for Sol.

Fable scores slightly lower, but with an average solution size of 12.2 KB.

Data at https://gertlabs.com/rankings?mode=oneshot_coding

This looks like a good benchmark. Time and time again I keep giving OpenAI models the chance to win me back, but Opus (and Fable especially) just writes more elegant code and is a significantly more productive rubber duck for interactive discussions. I feel vindicated seeing your description of verbose and defensive code, and I’m a bit disappointed that 5.6 Sol’s solution is still >5x longer than the human solution and 2x as verbose as Fable’s. Do you have any insight whether any of that is comments?

I wonder why nobody has tried to optimize for actual code size or complexity metric, or at least why I haven’t seen more benchmarks that display this. GPT5.5 just keeps pushing more and more pointless indirection into every function it writes in my main project, it’s borderline negative productivity.

P.S. I’d be curious to see Cursor’s composer models in there, they seem to be among the best performing low cost models: https://artificialanalysis.ai/articles/cursor-composer-2-5-c...

The human solutions are all written in Python, which creates a significant length bias, whereas the AI models are assigned to create solutions randomly distributed across 11 relevant programming languages, most of which are inherently more verbose than Python.

I have not broken down the comment/code ratio, but that's actually a really interesting idea for a metric.

I would also like to test Cursor, but our policy is to only test models available on public routers for now.

Really almost all benchmarks I look at have a cost per task column, which is basically the code size metric if you take an extra step
Seems quite kind to Gemini models.
Gemini models struggle with agentic coding/tool use/exploration, but they are actually quite smart in one-shot reasoning. They're not as far behind as people think. It's mostly post-training and productization issues, which are easier to fix than pre-training/mid-training issues.

Agentic coding data: https://gertlabs.com/rankings?mode=agentic_coding

This is the first I have herd of this benchmark. Can someone explain how it in any way indicates how close we are to "AGI"?

Replay of Sol attempting the game: https://arcprize.org/replay/83543d22-8e1e-439a-8809-129ff1d9...

It seems a weird and arbitrary challenge for a language model to be expected to perform. It also seems like there are some harness/visual issues even in the first few steps, where it states that it hasn't moved when it clearly has.

> Can someone explain how it in any way indicates how close we are to "AGI"?

I think it is historical name. At some point when benchmarking was very undeveloped, this was targeting abstract reasoning and generalization, hence AGI.

the problems are general and abstract, domain specific knowledge and memorization don't help. Figuring out the rules, the goal, the controls, and how to solve in a reasonable budget all indicate some level of general ability.
I'm surprised it is that low. Are not all top AI labs "cheating" and workaround LLMs's low sample efficiency by hiring people to generate more data points - similar problems with answers, so they can train models on those and improve scores? A good benchmark for general intelligence probably should be a complete black box, no sample data given/leaked at all.
Oh it's because the bench is lying. You need to pass each level without failing, if you fail a level, it count as you "lost" the minigame. The fact it start to get a score means it managed to get a 100% score on one of the minigame.
it seems the older models were capped at 10kusd for the runs though?
> Cost/task: $25.1K

Yikes

Very interesting. My prediction is that Mythos would outperform Sol.

Also what does this tell about Yann LeCuns whole world model theory? Bro has been going on and on about it. He has made multiple wrong predictions on the trajectory of LLMs.

At some point his claim should be fully falsified no?

Mythos probably wouldn't, otherwise they'd have included it in their release. Next version of Mythos probably will though.

And yeah.. Reality has not been kind to LeCun.

Falsifying Yann Lecun isn't exactly a priority for anyone seriously working in this space.
Mythos doesn't appear to be on the verified leaderboard for ARC-AGI 3
Notice how neither him, nor Ilya, nor Mira shipped anything relevant recently

It's telling

“Bro” spent most of his career in the wilderness because everybody thought ML/NN/etc were a dead end.

I’d not wager against him having at one one more break though architecture before he retires.