back

by matheusmoreira·3d ago·view on hn ↗
> The simplest explanation is that 'Fable-level' doesn't mean anything; it's just hype, and there's not much difference in capability.

Couldn't be further from the truth. The models can be tested and statistically evaluated.

I ran a massive Fable max code review on my lone lisp codebase. Now that I have switched to OpenAI, I decided to run an equivalent review using Sol max and compare them. I'm keeping all data so I can thoroughly evaluate their performance in multiple areas such as correctness, rigor, performance, security, maintainability, consistency, among others.

Fable pass is 100% done and I'm around 70% done with the Sol pass. Preliminary results are already becoming clear: Sol is capable of reproducing around 70% to 90% of Fable's performance. Haven't tested open weight models but I'd wager they have the same performance as Sol if not lower.

It seems Fable is still king, I'm afraid. It's undeniable that OpenAI is providing huge value here: up to 90% Fable performance at multiple times the usage on a subscription than what Anthropic offers us is a phenomenal deal. However, if one desires the best model, to me it looks like Fable is still it.

1 comments
You sound very certain, but so do all the people who disagree with you, and they've got their own private benchmarks.

You'll forgive me if I remain unconvinced.

If I sounded certain, it was not intentional. I made sure to hedge my statistical claims with "seems" and "looks like". I'm no AI lab, I'm just a random subscription user trying to get the most value out of them.

I'm just saying it's not wise to simply put all these models in the same bucket and say any differences are due to vibes or hype. They are clearly different. We can and should scrutinize the testing methodology but it's not exactly fair to just ignore the results.

I don't intend for my benchmark to be private. The core component of my test is my parallel code review skill which is already on my GitHub. I'll be publishing the results on my website when it's done. Anyone could take the skill and reproduce the test using multiple models against any codebase out there, then analyse the depth of each model's findings.