Item — HN Reader

I am used to seeing technical papers from ieee, but this is an opinion piece? I mean, there is some anecdata and one test case presented to a few different models but nothing more.

I am not necessarily saying the conclusions are wrong, just that they are not really substantiated in any way

5 comments

To be fair, it's very rare that articles praising the power of AI coding assistants are ever substantiated, either.

In the end, everyone is kind of just sharing their own experiences. You'll only know whether they work for you by trying it yourself.

> You'll only know whether they work for you by trying it yourself.

But at the same time, even this doesn't really work.

The lucky gambler thinks lottery tickets are a good investment. That does not mean they are.

I've found very very limited value from these things, but they work alright in those rather constrained circumstances.

And you can't try it out without for the most part feeding the training machine for at best free.

Codex and Claude Code allows you to opt out of model training.

Perhaps you don't believe OpenAI and Anthropic when they say this, but it is a requirement upon which most enterprise contracts are predicated.

Are there a lot of products or services you can try out without using the product or service?

Without sending info to their servers, yes.

This is the Spectrum magazine; the lighter fare. https://en.wikipedia.org/wiki/IEEE_Spectrum

Yeah I saw the ieee.org domain and was expecting a much more rigorous post.

This may be a situation where HackerNews' shorthand of omitting the subdomain is not good. spectrum.ieee.org appears to be more of a newsletter or editorial part of the website, but you wouldn't know that's what this was just based on the HN tag.

I've been on this site for over a decade now and didn't know this. That's a genuinely baffling decision given how different content across subdomains can be.

On the other hand, “ieee spectrum” is directly at the top of the page, then “guest article.”

Well, as much as I'm sure HN is a special place ;) , it is well documented that a lot of people on the internet just read the headlines

Articles? Headlines? I go right to the comments!

Maybe an exception could be made here, like HN does for medium.com.

And the example given was specific to OpenAI models, yet the title is a blanket statement.

I agree with the author that GPT-5 models are much more fixated on solving exactly the problem given and not as good at taking a step back and thinking about the big picture. The author also needs to take a step back and realize other providers still do this just fine.

He tests several Claude versions as well

Ah you're right, scrolled past that - the most salient contrast in the chart is still just GPT-5 vs GPT-4, and it feels easy to contrive such results by pinning one model's response as "ideal" and making that a benchmark for everything else.

and they are using OpenAI models, who haven't had a successful training run since Ilya left, GPT 5x is built on GPT 4x, not from scratch aiui

I'm having a blast with gemini-3-flash and a custom copilor replacement extension, it's much more capable than Copilot ever was with any model for me and a personalized dx with deep insights into my usage and what the agentic system is doing under the hood.

can you talk a little more about your replacement extention? I get copilot from my worksplace and id love to know what I can do with it, ive been trying to build some containerized stuff with copilot cli but im worried I have to give it a little more permissions than im comfortable with around git etc

I post a lot about it on Bluesky

https://bsky.app/profile/verdverm.com

The container stuff that backs it is built on Dagger

https://github.com/hofstadter-io/hof/tree/_next/examples/env

The entire extension and agent framework is in that repo too

extensions/vscode and lib/agent

I let my agent do whatever because I know exactly what it can and can't do. For example, it can use git, but cannot push, and any git changes are local to its containerized environment and don't get exported back to my filesystem where I do real git work. I could create an envelope where they could push git, and more likely I'll give them something where they can call GitHub ali, that's really more useful anyway