>> the error rate of frontier LLMs on textbook highschool/college level mathematics is going to be extremely low.
The scientific basis for this statement is unclear.
back
2 comments
AIME and HMMT are high school math olympiad competitions now used to benchmark AI. Frontier models score 95-99%
They score 99% but then can’t explain _why_ their solution works with any accuracy. Not as useful for learning as one might think.
are you referring to this section?
> Verification is different from generation: Models scoring 98 can solve problems but can't always explain why their approach works at the level a human mathematician would.
It doesn't say it can't explain why with any accuracy, it just says it can't *always* explain at the level of a mathematician, but most of us don't have such a mathematician at our beck and call to answer our questions anyways (thinking of the perspective of a self-learner outside of formal education)
It’s not just math. Anecdotally, LLMs struggle the same way with software engineering where the code they write is correct (compiles and passes tests), but reasoning is wrong often enough to eliminate most trust in these models’ ability to explain codebases or even features they themselves produce. It’s not about the _level_ of the supposed intelligence where a model struggles to summarize things succinctly or simply enough (responding to the “pocket mathematician” comment) or can’t grasp certain concepts at all (if so, how tf is it able to apply them?). It’s that by their design LLMs have no concept of truth and no concept of causality. They guess with every single inference and it’s very hard as a user to understand which guesses are more or less certain, since, you know, confidence ratings aren’t part of these models’ design either.
>The scientific basis for this statement is unclear.
Is it? Actually this seems like one of the MOST clear aspects of LLM performance we can measure.