▲ 120 points
back
117 comments
> We are 4 years into this ‘revolution’ and the industry has almost nothing to show for it. Software has not improved in quality,
Apple, Mozillia, and Firefox just released a record number of bug fixes because of AI. So this assertion that Software as not improved in quality is arguably if not provably false. There are 1000s of other examples. I have my own. Personal projects that were stopped because of dependency rot, having 100s of bugs fixed, nearly all dependencies removed, and lots of tests added just by asking. I know tons of others with similar experiences
> I do not feel like I’m falling behind by not using them
Obviously you are correct about "how you feel". But, this argument to me is more like a horse owner claming they don't feel like they're failling behind to trucks that can carry 10x, 100x, 1000x more than their horse. Sure, maybe the horse can go places the truck can't. For most uses the truck is better than the horse.
> The LLM-generated PRs I get are still rubbish.
I can't argue with that. I haven't seed the LLM-generated PRs "you" get. I can say though the LLM-generated bug reports I've seen are 1000x better than human ones. Human bug report "your software breaks, fix it!". LLM bug report "There is a bug in this specfic part of the code for these specific reasons and here's a repo I wrote to demonstrate the bug. Here's instructions on how to run the repo. There's the code to fix the bug. And, here's a test to show the bug is fixed."
Is it perfect? No, sometimes the bug isn't real because the code can only be reached in situtations that can never happen. But still, 7 of 8 times it's a real bug and the bug report are the best I've ever received.
you can go through this list: https://www.firefox.com/en-US/firefox/150.0/releasenotes/ and for those very few issues not still blocked you can read the examples
>For most uses the truck is better than the horse.
I think you're talking past the author here, who says:
>By not leaning into LLMs I’ve given myself the room to specialise in a smaller set of skills. So far that conscious decision is working out well, and I remain confident about my job security.
"Most uses" might mean boilerplate and simple frontend work that LLMs can do easily because it's formulaic and doesn't require much creativity. OP is simply not doing that type of work.
I don't think anyone sane is really claiming 100x or 1000x speedups anymore. Most people I talk to who use LLMs are closer to 1.5 or 2. Because their job is not mostly boilerplate, there is simply not that much room to be improved.
>Apple, Mozillia, and Firefox just released a record number of bug fixes because of AI.
A hidden metric here is the number of new bugs created by these fixes. If an LLM creates 10x bugs but create 5x more new bugs, then it is not really an improvement. Because that is 5x more bugs that the user has to observe and report (unless it is a security vulnerability, which the LLMs might detect) before it can be fixed.
LLMs are very helpful as a debugging aid, yes, but in large part because the fixes tend to be small and verifiable. That this does not carry over to many other use cases is the crux of the problem.
I myself use them to accelerate programming tasks, so I'm not anywhere near as pessimistic as the author, but the claimed multiples of productivity definitely haven't materialized for me.
> Obviously you are correct about "how you feel".
I posted the link. I didn't write it. My name is Liam Proven, thus "lproven", while they are Joshua Barretto, thus "jsbarretto".
Not the 1000x more productive numbers again please. Is this a meme or something? If you said +17% I might have taken you seriously.
>productivity
I recently experienced some cognitive decline due to chronic health issues. This is unfortunate timing because I am shipping a game with online multiplayer for the first time. So "brain works properly" is kind of a basic job requirement! But you work with what you got...
Some day I have the energy to do programming "manually", but on many days, the difference between "tasks get done" and "I am just staring at the screen reading the same code over and over again" is "ask the LLM to do it."
So I have had to shift my identity from "the guy who does the thing" to "the guy who ensures the thing has been done."
That being said, I do check the diffs carefully. (I tried a more hands-off approach a few months ago, and that was a pretty bad idea.) And I've learned to make the diffs as small as possible: more digestible for me, and less room for the AI to go off the rails.
Nice bonus to small, well defined changes is that small, fast, cheap models can handle them. (Having a great time with Luna lately, even on Low reasoning effort, which I wasn't expecting at all.)
By checking the diffs carefully, you're showing how to use an LLM to add value instead of creating a 17000-line PR that nobody with any sense will read.
LLMs are really good at making it harder to ignore ignorant people, and you've demonstrated that you're NOT part of the problem.
My story is a bit similar. Recently, I broke my wrist due to a fall and have also been suffering from eye strain. I couldn’t spend much time “working” on screen as much as I need. And so I ask LLMs to do a major chunk of work, review and test the changes, course correct things to align with architecture that I’ve outlined and get things done as if I were working normally.
>> We are 4 years into this ‘revolution’ and the industry has almost nothing to show for it.
If you actually used LLMs instead of preaching against them from the sidelines with no real experience, you'd know such statements are wrong.
This is the real problem with many of the anti LLM folks - they tell us about LLMs and also tell us they don't use them.
"That thing cannot do the stuff you say it can. And there is no way I will ever try it. But I know for sure I am right."
> If you actually used LLMs instead of preaching against them from the sidelines with no real experience, you'd know such statements are wrong.
What about the people who do have a lot of LLM experience and agree with OP?
> That thing cannot do the stuff you say it can. And there is no way I will ever try it. But I know for sure I am right.
If someone claims to be able to fly by strapping bird wing shaped pieces of plywood with feathers glued on to their arms, do you need to personally jump off a tower with them to say they don't work, or can you look at the results of others attempting it and draw conclusions based on that? LLM proponents are making claims about their capabilities which can relatively easily be checked without using LLMs yourself. To pick an example where LLM proponents are correct, anyone who says LLMs can't generate syntactically valid code can be proven wrong fairly easily by producing an example of syntactically valid, LLM generated code.
Further, if you read the rest of the paragraph you responded to, it's clear that this claim is about the state of the industry as a whole. "Software has not improved in quality, got faster, become cheaper to produce (when you exclude the mountain of poor-quality demoware that no reputable organisation would touch with a barge pole), or become more capable." Whether this is true or not is something that can be evaluated without ever having prompted yourself, or even arguably without being a developer at all.
> This is the real problem with many of the anti LLM folks
(BTW, I am the OP: I posted this, but I didn't write the article.)
No, the real problem here is that you do not understand the anti-LLM arguments.
I won't touch LLMs for anything except translation. But my anti-LLM stance is nothing whatsoever to do with whether they work or not (and that is even without the extreme torturing of the word "work" that the advocates do.)
My objection is based on the terrible crimes committed in building them (from theft), training them (at catastrophic ecological cost), in running them (ditto), in the effects on their individual users (cumulative brain damage), on their corporate users (economic destruction), and so on.
If they work is incidental. The believers like what LLMs do. For me, by my definitions, LLMs do not work. They extrude content that I never ever want to see, or read, or here, or want to execute. But for the botlickers, that is enough: they extrude a product. The botlickers want product.
I don't care. I don't want the product, but more to the point, I want the machines making the product not to exist... and if the price of that is all the people using the machines to never work again, and all the companies using them to go bankrupt, I would be absolutely delighted.
> If you actually used LLMs instead of preaching against them from the sidelines with no real experience, you'd know such statements are wrong.
Unlike the author, I'm an LLM optimist; but this take doesn't make sense to me. The thing you show cannot be "hey look at this LLM-based workflow, that's so much better than last year".
Where is the big browser, operating system, or other piece of work that has been able to quickly compete with the existing entries in the market due to LLMs?
The only one that I can think of off the top of my head is the rewrite of Bun in Rust. I think the fact that Claude Code runs on millions of devices on top of that rewrite is the sort of impressive demonstration that would prove that LLMs have lasting, systemic effects -- but it is telling that the only one I can think of was made by the extremely well funded lab that also happens to build the LLM.
> If you actually used LLMs instead of preaching against them from the sidelines with no real experience, you'd know such statements are wrong.
I have. Which is how I know that such statements are correct.
And when you show them evidence of the incredible things that can be done they stick their fingers in their ears and say "I'm not listening. I'm not listening. looping ad infinitum. OR they start saying "Oh yeah, prove it with hard numbers." when nothing about software productivity has ever been able to be proven with hard numbers. It becomes a semantic word game that you cannot win because nothing you say will get them to believe because they don't want to believe. The LLM haters are the most dogmatic of unbelievers.
Which I find weird - the greatest fun for me is making a computer do something awesome. I learned to program because I want that outcome.
I respect if you love the art and craft of hand programming but it's something else entirely to simply deny that LLM's are what they are - which is to say so amazing it is really beyond belief and they are getting more so every year.
If you're a hand programmer forever then fine, just don't make a fool of yourself by saying that everyone is lying about LLMs, to justify your choices.
I see this pattern in a lot of things.
- people who preach against PV solar... while people I actually know broke even 10 or more years ago and have been happily getting "free power" for a lot of their needs
- people who preach against electric cars... while people I actually know are driving them with no "range anxiety" or "replaced main batteries", but their non-engineer spouse/kids are driving the cars fine too.
- people who preach that tesla self driving isn't viable/etc... while lots of people are using tesla to drive themselves around for > 90% of their driving
>The not-so-quiet goal of the LLM push by big tech was to homogenise intellectual labour such that it can be made fungible. When labour is fungible you can swap it out whenever you like for an alternative, decreasing the bargaining power of workers. Software development was one of the last ‘guild fields’ and now we are seeing its industrialisation occur.
I think that the industrialization of software development already happened decades ago. The common professional software development methodologies, both waterfall and sprint-based, ARE the industrialization of software development. Isn't modern "agile", sprint-based software development methodology directly influenced by Japanese manufacturing methods? I don't know when software development actually was a guild field, and my personal experience doesn't reach back this far, but based on what I've read at least, it seems to me that even back in the 1990s it was already not a guild field.
A "guild field" is not a well-defined matter, but I think it's still the easiest engineering field to sit at home on your personal gear and learn skills that can get you hired in a real job with a future without attending any professional school for the field.
In fact the four-year degree you'll get from school is getting increasingly distant from the skills I actually want out of a new grad. It's not impossible to bridge the gap or anything but my transition into the commercial realm in the early 2000s was a cakewalk compared to the sheer number of things I'm asking a new grad to learn as soon as they're settled in at their desk... source control, CI/CD, bug trackers, devops, and that's just the beginning of that list not the end.
The last bullet point is the most important. The industry is most hyped about AI so that they can pay software developers $70k instead of paying them $200k.
I don’t really agree that AI can’t make development faster, though. What the author describes as the negative AI outcome is blind vibecoding. There are many other ways to use it, and even the basic “enhanced autocomplete” is a net benefit, especially since that functionality is dirt cheap or free.
To the author’s point about there not being enough evidence to support pro-AI claims — Additionally, there are studies and articles coming out showing those people are wrong and AI only makes people barely more efficient. Not the 10x tech CEOs are claiming while slashing thousands of jobs
https://getdx.com/blog/ai-productivity-gains-are-10-percent-...
LLMs have not improved the output quality of high level engineers — may have increased volume, though. One of the biggest things it has done however is given nontechnical people the ability to code MVPs in a very short amount of time. Yes, the code is ugly, nonsensical, and weird to look at. Can you build quick and dirty prototypes in a few hours? Yes.
All AI hallucinates, and therefore is unpredictable and unreliable. 1.5 trillion dollars to prove the obvious. It has no place in production use. IMHO at least. They want so badly to get rid of all of us, and replace us with something that doesn't complain, doesn't have kids, or outside lives or want ick, vacations. Most of you are paycheck to paycheck, statistically speaking, so you have a vested interest in how this all turns out.
edit: I get it, it's not a popular opinion.
But am I wrong?
If the problem of hallucination in AI has been solved, maybe I missed it?
> The not-so-quiet goal of the LLM push by big tech was to homogenise intellectual labour such that it can be made fungible. When labour is fungible you can swap it out whenever you like for an alternative, decreasing the bargaining power of workers. Software development was one of the last ‘guild fields’ and now we are seeing its industrialisation occur.
This is really well said. I had not put my finger on it before. Maybe this is a next move in the saga to avoid labor unions in engineering.
>The not-so-quiet goal of the LLM push by big tech was to homogenise intellectual labour such that it can be made fungible. When labour is fungible you can swap it out whenever you like for an alternative, decreasing the bargaining power of workers.
Especially if you replace people with robots that don't need days off and have the people train their replacements.
idk. I get more stuff done, faster, that I wouldn’t have been able to. You have to use it judiciously and with supervision - I can see how letting it get out of control would negate the benefits. But with good use, its undeniable.
> Software development was one of the last ‘guild fields’ and now we are seeing its industrialisation occur.
Doesn’t really support the rest of the argument. If you’re a skeptic and think it’s all hype, there’s nothing to worry about.
I have started using AI a bit, but I'm still trying to be careful in that if I ever lose access I can still perform my job at pre-AI ability. My understanding is that tokens are still heavily subsidized, and when that well runs dry I don't want it to be some nightmare to go back to the way I was doing my work in like 2022.
Its wild people still have this attitude.
Like... have you used these tools at all?
You don't have to go full vibe coding to see the clear value they provide and its order of magnitude leap over the tools we had before...
- you forgot the biggest and most important point of all!
- Everyone and their mother using an LLM is slowly and steadily destroying their brain cells and abilities in the process while creating a learned dependence
- Give it about 5 yrs, mark my words. The studies ll come out saying "95% programmers cannot write a hello world without an LLM now"
- Now someone smart ll hop in on my comment and bring calculators into the argument and I have answer waiting for them that those do (not able to put a direct link here for some reason)
I think the fundamental rebuttal to this is merely that, we’re still so early, the $1.5 trillion number is irrelevant for developing technologies in the same way two women can’t birth a baby in 4.5 months
Anecdotally, I've had an iPhone for 15 years, and that whole time I've dreamed of apps - some silly, some useful, that I would tell people "Imagine if there was an app that did x…". In those 15 years I learned to program and built websites and web apps, and built very basic web apps to use on my phone but shipped zero iOS apps.
Thanks to Claude, as a hobby, I've built and shipped 4 iOS apps in the last month. None of them are world-changing, but that's 4 in the last month thanks to Claude and zero in the 15 years before it. I'm clearly getting my subscription's worth out of Claude and squeezing more juice out of my own devices. It's a fun hobby! (And I'm sure if I used Android I could do more and customize my own device even more!)
>We should all know by now that code is an input to the software development process and not an output: yet even the most forward-looking organisations still tactitly treat lines of code as a productivity metric
What exactly is this referring to? My thinking here is "code is a hypothesis", and you don't know if it's solid or not until it contacts the real world. (And ideally tests, and maybe a proof or two.) And what we're seeing lately is that the hypotheses often get disproven decades later (longstanding kernel bugs etc.)
I wonder how much AI was used by the human submission PRs. Maybe they also just use AI, but just in the way you should: as a tool and not a replacement. And the people that use it in the wrong way are of course much more, making it harder to see the people who are actually benefitting from it.
I agreed right upto the last quote. That made the post political, similar to the ones being written by the AI evangelists.
The thing about complex systems is that they are not predictable. Right now the answer is whatever you think is right, and a lot of scenarios seem plausible.
It is a fun intellectual moment trying to make sense, but maybe only time will tell how the dynamics work out.
I have a similar experience that I expected several competitors to popup in my app niche, and so far only one has come up, and after some digging, they already had 8+ apps in another niche that had some overlap, so they would be able to complete the competing one without AI.
My apps are also not super complicated.
I think an argument can be made that some software has gotten more secure.
I don’t think it can be denied that the models do show an ability to find security vulnerabilities that may have otherwise been missed
The author claims that software has become no more secure - despite the avalanche of security fixes appearing for all software everywhere.
Never mind the more subtle issues.
Seems like a lot of people who aren’t very good remain skeptical.
This is equally as biased as those who claim AI can just code all night autonomously for you. You don't have to use it to generate "fully LLM-generated PRs". It's true GitHub is flooded with poor-quality LLM slop, but that is not the only way to use it.
It's ironic the author links to Naur's paper because I use LLMs to dig through code faster, question my assumptions, review my drafts, or generate drafts for my review. It's not because I am bad at it, I used to be very proud of being able to do this better than others. Anyway, this does not lead to more "lines of code" as the author suggests and instead it can lead to higher quality code.
FWIW, I also don't believe you can let LLMs make critical judgement by somehow stringing together more LLMs. And I am also not making a claim on whether the $1.5T investment is proportional to its benefits. But to say "industry has nothing to show for it", or to bring up strawmen like "LLM-generated PRs" and "lines of code", makes me question whether the author has tried studying the applications of LLMs.
are open ai, anthropic, zai, deepseek, cursor etc not new players? that benchmark seems like one defined to never be good enough
the way i see, LLM's in their current shape are neither good enough to replace software engineers but they have become a necessary part of everyone tool kit. ergo, actual companies themselves aren't getting much benefit because their competitors are using the same tools and they can't fire their human labor either just it's just an extra cost. At my org, i see good engineers use LLM's in wise ways and bad engineers just let it write slop for them. What gets to me is the verbosity and noise though, i consider brevity, and curation to be hallmarks of intelligence but most people at work just spit out so many words in their design docs or code comment these days.
The reality is new technology that works, will get used for what it best works for.
There are things that LLMs remain uniquely suited for, while the echo chambers go for everything.
There's no requirement to use new technology. Maybe what you do, or how you do it is just fine, and that's OK.
Skepticism today can remind me of someone who went into a cave for 3-4 years with a dated take of LLMs on day 1 and not remember software improves, and LLMs are software.
There are no shortage of uniquely working solutions with LLMs if sone seeks to find them, and also the self-validating echo chamber that says that it doesn't if that's desired. Which parts of software development, or how software development may better happen differently with the involvement of AI remains to be seen.
"Well, actually, vim doesn't make you faster"
it's not as good as the hype, but it's definitely revolutionary. i am still blown away at what I can achieve in 20 minutes with a couple of agents
AI skeptics hate him. Here's how he doubled his productivity using AI with one simple trick.
You have to stop thinking like an employee or a worker bee, and start thinking like a boss.
Guess what, the work you give your boss isn't 100% gold either. And sometimes you are preoccupied, or tired, so what you deliver isn't the best of all possible deliverables. Sometimes you hallucinate, i.e. you are dead wrong when you think you are dead right.
This is the job of a boss: take fallible employees and turn what they produce into something actually valuable. The boss lets the grunts do the grunt work, and then they combine the output together, and do the due diligence, etc etc.
It's an entirely new skillset. But learning the new skillset isn't the real blocker--the real blocker is that you've got to stop thinking like a grunt and start thinking like an executive.
WHAT is it that you really want to do? Frame that question as clearly and as concretely as you can. Then farm out thee work to human or AI grunts, and be prepared to do your due diligence on the result.