Item — HN Reader

> And yet it’s not.

That was my point as well. That it hasn't been output, even though it could be done by a talented solo developer given enough time, and that current LLMs definitely aren't able to do so.

> The problem is that they don’t generate acceptable code, they generate code that needs to be edited to be acceptable.

You've never had an LLM output a one line bugfix that is correct to the point where you don't have to edit it?

To make things more concrete, here's an example from the creator of Redis on how he utilizes LLMs in programming: https://antirez.com/news/164

1 comments

> You've never had an LLM output a one line bugfix that is correct to the point where you don't have to edit it?

I have. I’ve also had IDEs and static analysers do the same thing. I can also take my car out of gear and have it roll down a hill but that doesn’t mean it can run without fuel. Only a sith deals in absolutes, and in the general cases LLMs don’t generate acceptable code.

My experience is that when I ask for a clear and well defined problem, of the scale of "add motion blur (linear, spin, and zoom) to the filters menu; include standard dialog box (see existing design) for user input on all options", this works something like 90% of the time, is obvious garbage 5% of the time (in my experience, when it claimed to be writing "unit tests" it was actually performing regexes on the source code), and is subtly wrong the other 5% of the time.

If you use the planning mode, and your first move in the project is "write plan to reimplement photoshop" then you blindly say "continue" until the plan is done, then you get 0.9^{number of features} success, which of course on the scale of photoshop is going to be a failure. But this is still in one sense a 10x speedup in that 9 times out of 10 you're only doing code review, not having to re-write it. But code review is a real thing, so it's 10x on writing code not 10x on delivery.

Try it out, like seriously, learn to use it well. Spent a few weeks with it. You will not say these things if you were an experienced user of these tools. Saying their code is "unacceptable" is a skill issue. Describe what you consider "acceptable code" and watch it produce it in copious amounts. They don't have one mode, one setting, they can generate whatever the F you want in whatever style you deem "acceptable". You're completely in control.

That said, I watched many of my - generally pretty clever - colleagues struggle mightily with this. I can't put my finger on it yet. Regular "programming" - typing BS syntax one character at a time - always felt astronomically boring to me so I'm one of the guys happy with these tools. Not happy with how it will fuck up society though, but that's uh.. yeah.

As always in these discussions, I think people compare apples and oranges. LLMs are great with in-distribution solutions for solved problems with a lot of relevant prior material and in established technologies. Frontend stuff works great, for instance.

But for novel solutions, complex business logic, things deeply integrated with external systems... the code generation quickly turns terrible and useless. Especially if it's in anything but Python, Java, or JS.

Most of these differences in results end up being about differences in application. LLMs suck at out-of-distribution material, inherently.