▲ 643 points
back
295 comments
It's cool and I'm glad it sounds like it's getting more reliable, but given the types of things people have been saying GPT-5 would be for the last two years you'd expect GPT-5 to be a world-shattering release rather than incremental and stable improvement.
It does sort of give me the vibe that the pure scaling maximalism really is dying off though. If the approach is on writing better routers, tooling, comboing specialized submodels on tasks, then it feels like there's a search for new ways to improve performance(and lower cost), suggesting the other established approaches weren't working. I could totally be wrong, but I feel like if just throwing more compute at the problem was working OpenAI probably wouldn't be spending much time on optimizing the user routing on currently existing strategies to get marginal improvements on average user interactions.
I've been pretty negative on the thesis of only needing more data/compute to achieve AGI with current techniques though, so perhaps I'm overly biased against it. If there's one thing that bothers me in general about the situation though, it's that it feels like we really have no clue what the actual status of these models is because of how closed off all the industry labs have become + the feeling of not being able to expect anything other than marketing language from the presentations. I suppose that's inevitable with the massive investments though. Maybe they've got some massive earthshattering model release coming out next, who knows.
It reminds me of the latest, most advanced steam locomotives from the beginning of the 20th century.
They become extremely complex and sophisticated machines to squeeze a few more percent of efficiency compared to earlier models. Then diesel, and eventually electric locomotive arrived, much better and also much simpler than those late steam monsters.
I feel like that's where we are with LLM: extremely smart engineering to marginally improve quality, while increasing cost and complexity greatly. At some point we'll need a different approach if we want a world-shattering release.
The quiet revolution is happening in tool use and multimodal capabilities. Moderate incremental improvements on general intelligence, but dramatic improvements on multi-step tool use and ability to interact with the world (vs 1 year ago), will eventually feed back into general intelligence.
I mostly use Gemini 2.5 Pro. I have a “you are my editor” prompt asking it to proofread my texts. Recently it pointed out two typos in two different words that just weren’t there. Indeed, the two words each had a typo but not the one pointed out by Gemini.
The real typos were random missing letters. But the typos Gemini hallucinated were ones that are very common typos made in those words.
The only thing transformer based LLMs can ever do is _faking_ intelligence.
Which for many tasks is good enough. Even in my example above, the corrected text was flawless.
But for a whole category of tasks, LLMs without oversight will never be good enough because there simply is no real intelligence in them.
> It does sort of give me the vibe that the pure scaling maximalism really is dying off though
Really though, why put all your eggs in one basket? That's what I've been confused about for awhile. Why fund yet another LLMs to AGI startup. Space is saturated with big players and has been for years. Even if LLMs could get there that doesn't mean something else won't get there faster and for less. It also seems you'd want a backup in order to avoid popping the bubble. Technology S-Curves and all that still apply to AI
Though I'm similarly biased, but so is everyone I know with a strong math and/or science background (I even mentioned it in my thesis more than a few times lol). Scaling is all you need just doesn't check out
I agree, we have now proven that GPUs can ingest information and be trained to generate content for various tasks. But to put it to work, make it useful, requires far more thought about a specific problem and how to apply the tech. If you could just ask GPT to create a startup that'll be guaranteed to be worth $1B on a $1k investment within one year, someone else would've already done it. Elbow grease still required for the foreseeable future.
In the meantime, figuring out how to train them to make less of their most common mistakes is a worthwhile effort.
Performance is doubling roughly every 4-7 months. That trend is continuing. That's insane.
If your expectations were any higher than that then, then it seems like you were caught up in hype. Doubling 2-3 times per year isn't leveling off my any means.
>you'd expect GPT-5 to be a world-shattering release rather than incremental and stable improvement.
Compared to the GPT-4 release which was a little over 2 years ago (less than the gap between 3 and 4), it is. The only difference is we now have multiple organizations releasing state of the art models every few months. Even if models are improving at the same rate, those same big jumps after every handful of months was never realistic.
It's an incremental stable improvement over o3, which was released what? 4 months ago.
To be fair, this is one of the pathways GPT-5 was speculated to take as far back at 6 or so months ago - simply being an incremental upgrade from a performance perspective, but a leap from a product simplification approach.
At this point it's pretty much given it's a game of inches moving forward.
> Maybe they've got some massive earthshattering model release coming out next, who knows.
Nothing in the current technology offers a path to AGI. These models are fixed after training completes.
Year on year, progress is indeed a bit incremental. But seen over a five year period the progress is actually stupidly amazing.
In practical terms, Gpt 5 is a nice upgrade over most other models. We'll no doubt get lots of subjective reports how it was wrong or right or worse than some other model for some chats. But my personal (subjective) experience so far is that it just made it possible for me to use codex on more serious projects. It still gets plenty of things wrong. But that's more because of a lack of context than hallucination issues. Context fixes are a lot easier than model improvements. But last week I didn't bother and now I'm getting decent results.
I don't really care what version number they slap on things. That is indeed just marketing. And competition is quite fierce so I can understand why they are overselling what could have been just chat gpt 4.2 or whatever.
Also discussions about AGI tend to bore me as they seem to escalate into low quality philosophical debates with lots of amateurs rehashing ancient argument poorly. There aren't a hell of a lot new arguments that people come up with at this point.
IMHO we don't actually need an AGI to bootstrap the singularity. We just AIs to be good enough to come up with algorithmic optimizations, breakthroughs and improvements at a steady pace. We're getting quite close to that and I wouldn't be surprised to learn that OpenAI's people are already eating their own dogfood in liberal quantities. It's not necessary for AIs to be conscious in order to come up with the improvements that might eventually enable such a thing. I expect the singularity might be more of a phase than a moment. And if you know your boiling frog analogy, we might be smack down in the middle of that already and just not realize it.
Five years ago, it was all very theoretical. And now I'm waiting for codex to wrap up a few pull requests that would have distracted me for a week each five years ago. It's taking too long and I'm procrastinating my gained productivity away on HN. But what else is new ;-).
Things have moved differently than what we thought would happen 2 years ago, but lest we forget what has happened in the meanwhile (4o, o1 + thinking paradigm, o3)
So yeah, maybe we are getting more incremental improvements. But that to me seems like a good thing, because more good things earlier. I will take that over world-shattering any day – but if we were to consider everything that has happened since the first release of gpt-4, I would argue the total amount is actually very much world-shattering.
I for one am pretty glad about this. I like LLMs that augment human abilities - tools that help people get more done and be more ambitious.
The common concept for AGI seems to be much more about human replacement - the ability to complete "economically valuable tasks" better than humans can. I still don't understand what our human lives or economies would look like there.
What I personally wanted from GPT-5 is exactly what I got: models that do the same stuff that existing models do, but more reliably and "better".
> It's cool and I'm glad it sounds like it's getting more reliable, but given the types of things people have been saying GPT-5 would be for the last two years you'd expect GPT-5 to be a world-shattering release rather than incremental and stable improvement.
Are you trying to say the curve is flattening? That advances are coming slower and slower?
As long as it doesn't suggest a dot com level recession I'm good.
The next step will be for OpenAI to number their releases based on year (ala what Windows did once innovation ran out)
It seems like few people are referencing the improvements in reliability and deception. If the benchmarks given generalize, what OpenAI has in GPT-5 is a cheap, powerful, _reliable_ model -- the perfect engine to generate high quality synthetic data to punch through the training data bottleneck.
I'd expect that at some level of reliability this could lead to a self-improvement cycle, similar to how a powerful enough model (the Claude 4 models in Claude Code) enables iteratively converging on a solution to a problem even if it can't one-shot it.
No idea if we're at that point yet, but it seems a natural use for a model with these characteristics.
> but given the types of things people have been saying GPT-5 would be for the last two years
My reading is more that unit economics are starting to catch up with the frontier labs, rather than "scaling maximalism is dying". Maybe that is the same thing.
Isn’t reasoning, aka test-time compute, ultimately just another form of scaling? Yes it happens at a different stage, but the equation is still 'scale total compute > more intelligence'. In that sense, combining their biggest pre-trained models with their best reasoning strategies from RL could be the most impactful scaling lever available to them at the moment.
"They claim impressive reductions in hallucinations. In my own usage I’ve not spotted a single hallucination yet, but that’s been true for me for Claude 4 and o3 recently as well—hallucination is so much less of a problem with this year’s models."
This has me so confused, Claude 4 (Sonnet and Opus) hallucinates daily for me, on both simple and hard things. And this is for small isolated questions at that.
"Are you GPT5" - No I'm 4o, 5 hasnt been released yet. "It was released today". Oh you're right, Im GPT5. You have reached the limit of the free usage of 4o
The aggressive pricing here seems unusual for OpenAI. If they had a large moat, they wouldn't need to do this. Competition is fierce indeed.
"GPT-5 in the API is simpler: it’s available as three models—regular, mini and nano—which can each be run at one of four reasoning levels: minimal (a new level not previously available for other OpenAI reasoning models), low, medium or high."
Is it actually simpler? For those who are currently using GPT 4.1, we're going from 3 options (4.1, 4.1 mini and 4.1 nano) to at least 8, if we don't consider gpt 5 regular - we now will have to choose between gpt 5 mini minimal, gpt 5 mini low, gpt 5 mini medium, gpt 5 mini high, gpt 5 nano minimal, gpt 5 nano low, gpt 5 nano medium and gpt 5 nano high.
And, while choosing between all these options, we'll always have to wonder: should I try adjusting the prompt that I'm using, or simply change the gpt 5 version or its reasoning level?
Can anyone explain to me why they've removed parameter controls for temperature and top-p in reasoning models, including gpt-5? It strikes me that it makes it harder to build with these to do small tasks requiring high-levels of consistency, and in the API, I really value the ability to set certain tasks to a low temp.
Despite the fact that their models are used in hiring, business, education, etc this multibillion company uses one benchmark with very artificial questions (BBQ) to evaluate how fair their model is. I am a little bit disappointed.
Good to know -
> Knowledge cut-off is September 30th 2024 for GPT-5 and May 30th 2024 for GPT-5 mini and nano
> but for the moment here’s the pelican I got from GPT-5 running at its default “medium” reasoning effort:
Would been interesting to see a comparison between low, medium and high reasoning_effort pelicans :)
When I've played around with GPT-OSS-120b recently, seems the difference in the final answer is huge, where "low" is essentially "no reasoning" and with "high" it can spend seemingly endless amount of tokens. I'm guessing the difference with GPT-5 will be similar?
So, "system card" now means what used to be a "paper", but without lots of the details?
It’s fascinating and hilarious that pelican on a bicycle in SVG is still such a challenge.
Practically the first thing I do after a new model release is try to upgrade `llm`. Thank you, @simonw !
> a real-time router that quickly decides which model to use based on conversation type, complexity, tool needs, and explicit intent
This is sort of interesting to me. It strikes me that so far we've had more or less direct access to the underlying model (apart from the system prompt and guardrails), but I wonder if going forward there's going to be more and more infrastructure between us and the model.
This new naming conventions, while not perfect are alot clearer and I am sure will help my coworkers.
Basically repeats what it's been out through the usual PR channels, just paraphrased.
No mention about the (missing) elephant on the room, where are the benchmarks?
@simonw has been compromised. Sad.
METR of only 2 hours and 15 minutes. Fast takeoff less likely.
One key element missing from all these model cards are the model size/number of parameters. Without that info we are in the dark. We can't predict the future of AI. How does the intelligence scale with the increasing #parameters? Is there a limit? Should we attribute incrementally better metrics to larger model size or other techniques? Do they announce the full model they trained or a smaller version that is economically viable for the market conditions? If they double the model size will it be a Professor-level intelligence, a super-human level intelligence or a couple of phds level intelligence?
I was excited for GPT-5, but honestly, it feels worse than GPT-4 for coding.
I'm curious what platform people are using to test GPT-5? I'm so deep into the claude code world that I'm actually unsure what the best option is outside of claude code...
This is key info from the article for me:
> -------------------------------
"reasoning": {"summary": "auto"} }'
Here’s the response from that API call.
https://gist.github.com/simonw/1d1013ba059af76461153722005a0...
Without that option the API will often provide a lengthy delay while the model burns through thinking tokens until you start getting back visible tokens for the final response.
I wonder if there are people who find Sonnet 3.5 to be faster and good enough for programming instead of moving on to reasoning models (sometimes I do go to Gemini 2.5 pro if Sonnet doesn't cut it).
It seems to be trained to use tools effectively to gather context. In this example against 4.1 and o3 it used 6 in the first turn in a pretty cool way (fetching different categories that could be relevant). Token use increases with that kind of tool calling but the aggressive pricing should make that moot. You could probably get it to not be so tool happy with prompting as well.
This "system card" thing seems to have suddenly come out of nowhere. Signs of a cult forming. Is it just what we'd normally call a technical write up?
Only a third cheaper than Sonnet 4? Incrementally better I suppose.
> and minimizing sycophancy
Now we're talking about a good feature! Actually one of my biggest annoyances with Cursor (that mostly uses Sonnet).
"You're absolutely right!"
I mean not really Cursor, but ok. I'll be super excited if we can get rid of these sycophancy tokens.
Simon, as always, I appreciate your succinct and dedicated writeup. This really helps to land the results.
The improved bicycling pelican of course could be overfitting / benchmark-cheating...
Its basically opus 4.1 ... but cheaper?
Whoa this looks good. And cheap! How do you hack a proxy together so you can run Claude Code on gpt-5?!
> Definitely recognizable as a pelican
right :-D
We're supposed to get AGI next year and this is all they have. We are not getting real AI from LLMs people, wake up, we're in a bubble.