Fundamentally aren’t they concluding that tasks assigned to software developers (human or otherwise) are often incomplete, self contradictory or worse? This is the world in which their tool must play. I’m unsympathetic.
back
4 comments
Agreed - "underspecified prompts" being listed as a failure of the tooling is not a strong case. Even interns can understand ambiguous asks with a bit of help, and understand when they need to stop and ask instead of just carrying on. They are often working fairly independently on ambiguous tasks before the end of an internship, too.
So is the argument that frontier models are not just junior engineers, but first-month interns with no capability of progressing beyond that level?
> Even interns can understand ambiguous asks with a bit of help
This is not a case of an ambiguous task. This is literally trying to judge a model based on information it cannot possibly know, like trying to judge someone based on whether they know what I have hidden in my backpack. In the real world an intern could look at unit tests or ask for feedback, but that is not the case in a benchmark.
In a real job, you would be allowed to see the test case that failed and tweak your code (or more likely the poorly written test).
If you let a modern LLM do even the first, they’d crush this specific benchmark.
What is interesting is understanding how LLMs are able to beat 70+% on this benchmark or getting some of the poorly framed questions right? Are they implicitly learning the test writers style? Are the solutions leaking into their training set?
Perhaps reassuring is that even Fable stalls out at ~72% (on the hidden set which OpenAI did not run this analysis on), so perhaps training on the bench is not happening in anything but the most indirect ways.
I care a lot because small open models can never learn idiosyncrasies like this, so I really want good ways to judge models fairly.
EDIT: Humm OpenAI is muddying the water a bit. Only 20%ish of problems are broken in ways that are unfair to the agent, 4-10% are broken in favorable ways, so the benchmark ceiling is probably closer to 80-85%
The more subtle point is that there's a gap between the task and its verification. e.g. if you have an open-ended / under-specified prompt, the verification needs to be able to handle all potential solutions.
So you can have a very narrow task prompt that's easy to verify (but likely too simple of a challenge). Or a more realistic task prompt that's much harder to verify. And likely harder to both build the robust verifier and run it cheaply.
A substantial portion of software engineering -- and the fundamental jobs of a proper Product Owner and UX Designer -- is to turn "vague ideas about what we need to do" into "this widget, on this page, it should work like this"
It's not a pipeline, it's an ongoing conversation within any functional team, but this requires buy-in from management, who is often selected for "line must go up this quarter no matter the cost" over "hey, wouldn't it be cool if this company was still a going concern in twenty years?"
Variance in time horizons explains a lot of corporate behaviour.
And it’s rational. We all have limited careers.
I think that all makes a bit more sense as we get older. Optimising for short time horizons is not what I strive for, but explains things.
This was also my thought, and I think holds true of the ones with invisible requirements that aren't stated up front and are only captured in tests. Oh, you need to rework your solution to handle requirements nobody mentioned before? Well, me too.
Yeah, it's testing a different thing than what the benchmark claims to test, but it's also accidentally testing something more real-world applicable than a clean benchmark would be, so hey.
(EDIT: That is, if the agent is allowed to see the failed tests and iterate. If not, then yeah, that's just a problem. And either way, the ones with tests that just encode a particular solution's implementation details, thereby demanding that your solution have some rando internal details, are junkier. That's not a situation you'd run into in reality.)