▲ 303 points
back
270 comments
You know, sometimes I feel that all this discourse about AI for coding reflects the difference between software engineers and data scientists / machine learning engineers.
Both often work with unclear requirements, and sometimes may face floating bugs which are hard to fix, but in most cases, SWE create software that is expected to always behave in a certain way. It is reproducible, can pass tests, and the tooling is more established.
MLE work with models that are stochastic in nature. The usual tests aren't about models producing a certain output - they are about metrics, that, for example, the models produce the correct output in 90% cases (evaluation). The tooling isn't as developed as for SWE - it changes more often.
So, for MLE, working with AI that isn't always reliable, is a norm. They are accustomed to thinking in terms of probabilities, distributions, and acceptable levels of error. Applying this mindset to a coding assistant that might produce incorrect or unexpected code feels more natural. They might evaluate it like a model: "It gets the code right 80% of the time, saving me effort, and I can catch the 20%."
This has been about 50% of the time my experience as well. There are very good SWE who know how to use ML in real systems, and then there are the others who believe through and through it will replace well understood systems developed by subdomain experts.
As a concrete example, when I worked at Amazon, there were several really good ML-based solutions for very real problems that didn't have classical approaches to lean on. Motion prediction from grid maps, for example, or classification from imagery or grid maps in general. Very useful and well integrated in a classical estimation and control pipeline to produce meaningful results.
OTOH, when I worked at a startup I won't name, I was berated over and over by a low-level manager for daring to question a learning-based approach for, of all things, estimating orientation of a stationary plane over time. The entire control pipeline for the vehicle was being fed flickering, jumping, adhoc rotating estimate for a stationary object because the entire team had never learned anything fundamental about mapping or filtering, and was just assuming more data would solve the problem.
This divide is very real, and I wish there was a way to tease it out better in interviewing.
> So, for MLE, working with AI that isn't always reliable, is a norm. They are accustomed to thinking in terms of probabilities, distributions, and acceptable levels of error. Applying this mindset to a coding assistant that might produce incorrect or unexpected code feels more natural. They might evaluate it like a model: "It gets the code right 80% of the time, saving me effort, and I can catch the 20%."
And given the current climate, the MLE's feel empowered for force their mindset onto others groups where it doesn't fit. I once heard a senior architect at my company ranting about that after a meeting: my employer sells products where accuracy and correctness have always been a huge selling point, and the ML people (in a different office) didn't seem to get that and thought 80-90% correct should be good enough for customers.
I'm reminded of the arguments about whether a 1% fatality rate for a pandemic disease was small or large. 1 is the smallest integer, but 1% of 300 million is 3 million people.
You're talking about deterministic behavior vs. probabilistic behavior and yes some discourse lines up with what you describe.
I don't think it's the case with this article. It focuses on the meta-concerns of people doing software engineering and how AI fits into that. I think he hits it on the head when he talks about Program Entropy.
A huge part of building a software product is managing entropy. Specifically, how you can add more code and more people while maintaining a reasonable forward velocity. More specifically, you have to maintain a system so you make it so all of those people understand how all the pieces fit together and how to add more of those pieces. Yes, I can see AI one day making this easier but right now, it oftentimes makes entropy worse.
There are so many use cases where 90% correct answers are absolutely not enough.
Nobody would have much of a problem with that, if a flurry of people with vested interests wouldn't try to convince us all that that is not the case, and AI is good to go for absolutely everything.
The absurdity of this assumption is so outrageous that it becomes even hard to counter it with logic. It's just a belief-based narrative whose delivery has been highly successful so far in commanding insane investments, and as a travestee for profit-oriented workforce optimizations.
SWE use probability all the time. Rearchitect around that race condition or reduce its footprint? How long will this database call make, p99? A/B tests. Etc.
I’m currently getting my masters in AI (lots of ML) and as a SWE, it’s definitely a new muscle I’m growing. At the same time, I can think about MLE in isolation and how it fits within the larger discipline of SWE. How I can build robust pipelines, integrate models into applications, deploy models within larger clusters, etc. I think there are many individuals which are pure MLE and lack the SWE perspective. Most critically, lots of ML people in my program aren’t computer people. They are math people or scientists first. They can grok the ML but grokking SWE without computer affinity is difficult. I see true full-stack being an understanding of low-level systems, back-front architecture, deployment, and now MLE. Just need to find someone who will compensate me for bringing all that to the table. Most postings are still either for SWE or PhD in MLE. Give me money!! I know it all
Yea, the problem is that most people expect things to be absolute and correct outside of engineering.
I love the gray areas and probabilities and creativity of software...but not everyone does.
So the real danger is in everyone assuming the AI model is, must be, and always will be correct. They misunderstand the tool they are using (or directing others to use).
Hmm. It's like autopilot on the Tesla. You aren't supposed to take your hands off the wheel. You're supposed to pay attention. But people use it incorrectly. If they get into an accident, then people want to blame the machine. It's not. It's the fault of person who didn't read the instructions.
I often wonder if society will readjust its expectation of programs or even devices. Historically, machines of all kinds were difficult to design and manufacture.. the structure was hard set (hence the name) but at the same time, society fantasize about adaptive machines, hyper adaptive, multipurpose, context-aware.. which if pushed high, is not far from the noisy flexibility of ML.
Yup. All the people I've worked with through my career post 2020 (AI/ML types) have been AI first and AI native. They're the first users of cursor - a year before it went mainstream.
Sorry not sorry that the rest of the world has to look over their shoulders.
I strongly agree with both the premise of the article, and most of the specific arguments brought forth. That said, I've also been noticing some positive aspects of using LLMs in my day-to-day. For context, I've been in the software trade for about three decades now.
One thing working with AI-generated code forces you to do is to read code -- development becomes more a series of code reviews than a first-principles creative journey. I think this can be seen as beneficial for solo developers, as in a way, it mimics / helps learn responsibilities only present in teams.
Another: it quickly becomes clear that working with an LLM requires the dev to have a clearly defined and well structured hierarchical understanding of the problem. Trying to one-shot something substantial usually leads to that something being your foot. Approaching the problem from a design side, writing a detailed spec, then implementing sections of it -- this helps to define boundaries and interfaces for the conceptual building blocks.
I have more observations, but attention is scarce, so -- to conclude. We can look at LLMs as a powerful accelerant, helping junior devs grow into senior roles. With some guidance, these tools make apparent the progression of lessons the more experienced of us took time to learn. I don't think it's all doom and gloom. AI won't replace developers, and while it's incredibly disruptive at the moment, I think it will settle into a place among other tools (perhaps on a shelf all of its own).
Remember when 3d printing was going to replace all manufacturing? Anybody?
AI is closer to this sentiment than it is to the singularity.
LLMs are amazing at writing code and terrible at owning it.
Every line you accept without understanding is borrowed comprehension, which you’ll repay during maintenance with high interest. It feels like free velocity. But it's probably more like tech debt at ~40 % annual interest. As a tribe, we have to figure out how to use AI to automate typing and NOT thinking.
> Input Risk. An LLM does not challenge a prompt which is leading...
(Emphasis mine)
This has been the biggest pain point for me, and the frustrating part is that you might not even realize you're leading it a particular way at all. I mean it makes sense with how LLMs work, but a single word used in a vague enough way is enough to skew the results in a bad direction, sometimes contrary to what you actually wanted to do, which can lead you down rabbit holes of wrongness. By the time you realize, you're deep in the sludge of haphazardly thrown-together code that sorta kinda barely works. Almost like human language is very vague and non-specific, which is why we invented formal languages with rules that allow for preciseness in the first place...
Anecdotally, I've felt my skills quickly regressing because of AI tooling. I had a moment where I'd reach out to it for every small task from laziness, but when I took a real step back I started realizing I'm not really even saving myself all that much time, and even worse is that I'm tiring myself out way quicker because I was reading through dozens or hundreds of lines of code, thinking about how the AI got it wrong, correcting it etc. I haven't measured, but I feel like in grand totality, I've wasted much more time than I potentially saved with AI tooling.
I think the true problem is that AI is genuinely useful for many tasks, but there are 2 camps of people using it. There are the people using it for complex tasks where small mistakes quickly add up, and then the other camp (in my experience mostly the managerial types) see it shit out 200 lines of code they don't understand, and in their mind this translates to a finished product because the TODO app that barely works is good enough for an "MVP" that they can point to and say "See, it can generate this, that means it can also do your job just as easily!".
To intercept the usual comments that are no doubt going to come flooding in about me using it wrong or trying the wrong model or whatever, please read through my old comment [1] for more context on my experience with these tools.
> it doesn't reason about ideas, diagrams, or requirements specifications. (...) How often have you witnessed an LLM reduce the complexity of a piece of code?
> Only humans can decrease or resist complexity.
It's funny how often there's a genuine concept behind posts like these, but then lots of specific claims are plainly false. This is trivial to do: ask for simpler code. I'm using that quite often to get a second opinion and get great results. If you don't query the model, you don't get any answer - neither complex or simple. If you query with default options, it's still a choice, not something inherent to the idea of LLM.
I'm also having a great time converting code into ideas and diagrams and vice versa. Why make the strong claims that people contradict in practice every day now?
I think you could make similar arguments about mapping technology like Google and Apple Maps -- that using them decreases people's skills in navigating the physical world, atrophying our sense of direction and geography.
And actually, that's not wrong. People really do often struggle to navigate these days if they don't have the crutch of something like Google Maps. It really has changed our relationship to the physical world in many ways.
But also, a lot of people weren't especially good at navigation before? The overall average ability of people being able to get from Point A to Point B safely and reliably, especially in areas they are unfamiliar with, has certainly increased dramatically. And a small subset of people who are naturally skilled at geography and navigation have seen their own abilities complemented, not replaces, by things like Google Maps.
I think AI will end up being similar, on a larger scale. Yes, there are definitely some trade offs, and some skills and abilities will decrease, but also many more people will be able to do work they previously couldn't, and a small number of people will get even better at what they do.
Let's be real, 70% of staff are phoning in their jobs so badly that an AI often is just as good if not better.
The real struggle will be, the people phoning it in are still going to be useless, but with AI. The rest will learn and grow with AI.
"[AI] is not capable of working at a conceptual level".
I wonder where did author got that feeling. What recent LLMs proved time and time again is that they are definitely able to work at conceptual level (by correctly translating concepts from one language to another depending on the context for example). Saying it doesn't "understand" the concepts as humans do is a different thing. It wouldn't "understand" pain, because it has no experience of it. But humans constantly talk about thing they've never personally experienced (and indeed maybe they shouldn't, but that's another topic).
It's occurred to me more and more that I need to repurpose 90s.dev into a non-AI community, focused on the ancient art of writing software well, and welcoming to all who honed that craft. What would it need to start with? Forum? Mailing list? Aggregate blog from multiple authors like hackernoon?
[edit] Makeshift mailing list for those interested: https://github.com/sdegutis/90s.dev/issues/2 (subscribe or comment to "sign up for emails" on the topic)
> How often have you witnessed an LLM reduce the complexity of a piece of code?
I would add that AI often makes far too clever code as well, and would defer to Kernighan's law: “Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.”
> If you're a skilled, experienced engineer and you fear that AI will make you unemployable, adopt a more nuanced view. LLMs can't replace human engineering.
I don't think LLMs were ever meant to completely replace human engineering, at least in the way we think of engineering as a craft. But the truth is the world is changing, and with LLMs/AI, the goalposts are changing, too. With massive economies of scale and huge output, the goal is less and less good engineering and more and more pure output.
Put it another way, the post-AI world considers a reduction of quality to be an acceptable tradeoff for the massive short-term gains. Moreover, although quantity over quality is not exactly a new concept, what AI does is magnify and distill that concept as a prime directive.
Code doesn't have to work well to be profitable given how cheaply in can be produced, and that's a general effect of technology that can reach global audiences, and AI is the apex of that technology.
I don't think there is a nuanced view: AI is an all-around bad thing and its positives in the short-term will be vastly dwarfed by its negatives except for those who have become specialized at concentrating wealth at the expense of the common good.
Is AI perhaps an opportunity for years of paid cleanup and re-design work that those of us who have dedicated decades to learning the craft of software engineering will be able to cash in on?
Using AI effectively is like riding a horse or getting a monkey to do things for you. There's absolutely skill involved in getting them to do what you want, and used incorrectly they can create far more mess than any individual human. But we can also use them to achieve things that'd be almost impossible for us to do alone.
> LLMs as they currently exist cannot master a theory, design, or mental construct because they don't remember beyond their context window. Only humans can can gain and retain program theory.
False.
> An LLM is a token predictor. It works only at the level of text. It is not capable of working at a conceptual level: it doesn't reason about ideas, diagrams, or requirements specifications.
False.
Anyone who have spent time in machine learning or reinforcement learning understands that models are projections of higher dimension concepts on to lower dimensions as weights.
I disagree with the author here. His point can be summarized as "LLMs produce bad code, LLMs lead to the teams's average skills decreasing and they can't maintain large codebases over time".
Yet nowhere he addresses the #1 flaws to his position: rate of improvement of the technology, and its promise to deliver on saved money and gained speed.
In all the companies I've seen engineering leadership hardly really gives a shit about things OP says are important. They just care that customers are happy, the system is stable, and its malleable enough to allow for pivots when need be.
Good discussions & documentation about architecture before starting the work that gets peer-reviewed + A non-stupid engineer putting on good guardrails around the LLM's output + the extensive unit test suites in CD/CI + peer reviews on the PRs = all downsides near eliminated while all upside gained.
This is how we work at my company today (health startup). Google and Meta also boast publicly +30% of new lines of code are AI-generated in their companies today. That the state of *today*; assume in 5 years these AIs are 10x better... I simply cannot foresee a world where LLM-assisted coding is not the de-facto way be a software engineer.
I think it's more of an issue where the term "engineering" means something very different in the software world than it does in the rest of the world but I find it a bit difficult to take any such article seriously when it makes statements like: "LLMs can't replace human engineering." as a response to "If you're a skilled, experienced engineer and you fear that AI will make you unemployable, adopt a more nuanced view."
If you are able to make such deductions then you should also be able to deduce that almost nobody employed in a "software engineering" role is doing any actual engineering.
The article assumes that software companies are hiring software engineers (where "engineer" actually means what it does everywhere else) when in reality most software companies are not hiring any kind of actual engineer.
And if engineers can't be replaced by AI, but you're not actually an engineer, can you be replaced by AI?
Now I don't know the answer for sure, but I'd say for most people in "software engineering" roles the answer is still no, at least for now. But the reasons definitely can't have anything to do with whether AI can do engineering.
As a final note: I'd recommend anyone in a software engineering role, who thinks they do "actual engineering", to actually study some actual engineering discipline. You most likely have the necessary level of intelligence to get started. But you will quickly find that electrical engineering, electronics engineering, mechanical engineering, structural engineering, drainage engineering, etc, are nothing like your actual day to day job in _very fundamental_ ways.
I'm really appreciating some of the writing en AI that is coming out now. Beyond simply "vibe coding is bad" but advancing arguments about the specific risks.
One thing that I've noticed is that AI has made it even more abundantly obvious that the low IQs of middle-managers are the main problem.
They have a great faith in AI (which is understandable), but they're constantly realising that:
a) they don't understand any of the problems enough to even being prompting for a solution
b) the AI can explain our code but the manager still won't understand
c) the AI can rephrase our explanations and they still won't understand.
Traditionally middle-managers probably consoled themselves with the idea that the nerds can't communicate well and coding is a dumb arcane discipline anyway. But now that their machine god isn't doing a better job than we are of ELI5ing it, I think even they're starting to doubt themselves.
More than AI, learn Discrete Math and Number Theory. Calculus it's fine, but it's more suited for mechanics/system control/continous stuff. Sets and relations will weight far more on how to sort and classify your data in a really fast way. The Nyxt browser uses that really well, no AI needed, both for user search and data clustering.
> I don't think anyone believes that a computer program is literally their companion
Quoth the makers of Claude:
> AI systems are no longer just specialized research tools: they’re everyday academic companions.
> https://www.anthropic.com/news/anthropic-education-report-ho...
To call Anthropic's opener brazen, obnoxious, or euphemistic would be an understatement. I hope it ages like milk, as it deserves, and so embarrasses the company that it eventually requires a corrective foreword or retraction.
This claim is talking out of proportion, LLMs do push back:
> Input Risk. An LLM does not challenge a prompt which is leading or whose assumptions are flawed or context is incomplete. Example: An engineer prompts, "Provide a thread-safe list implementation in C#" and receives 200 lines of flawless, correct code. It's still the wrong answer, because the question should have been, "How can I make this code thread-safe?" and whose answer is "Use System.Collections.Concurrent" and 1 line of code. The LLM is not able to recognize an instance of the XY problem because it was not asked to.
When I prompt Gemini 2.5 Pro with "Provide a thread-safe list implementation in C#" it _does_ push back and suggest using the standard library instead (in addition to providing the code of course). First paragraph of the LLM response:
> You can achieve a thread-safe list in C# by using the lock statement to synchronize access to a standard List<T>. Alternatively, you can use concurrent collection classes provided by .NET, such as ConcurrentBag<T> or ConcurrentQueue<T>, depending on the specific access patterns you need.
I'm quite big on using LLMs in coding, and aside from the inflammatory graphic at the top and the comment about loss of joy (I haven't lost my joy at all personally), I agree with the entire rest of the post completely. (For context I've been programming for 30 years now and still love it.)
I'm still trying to figure out how to produce a codebase using LLMs and end up as a expert in the system at the end of it while still coming out ahead. My hope is I can be more of an expert a bit faster than before, not less of an expert a lot faster.
It feels within reach to me as long as there's frequent periods of coming to a complete understanding of the code that's been produced and reshaping it to reduce complexity. As well as strong technical guidance as input to the LLM to begin with.
I think there's still a lot of learning to do about how to use these things. For example, I've tried LLM-heavy things in lua, C, and C#. The LLM was great at producing stuff that works (at first) in lua, but lua was miserable and the code ended up a huge mess that I can't be bothered to become an expert in. The LLM was really tripped up on C and I didn't make it that far, I didn't want to watch it fight the compiler so hard. C# has been great, the LLM is reasonably effective and I have an easy time consuming and reshaping the LLM code.
I've always liked static type systems but I like them even more now, in part because they help the LLM produce better code, but mostly because they make it a lot easier to keep up to speed on code that was just produced, or to simplify it.
LLMs are great tools for assisting learning - but this is not an area where it's easy to extract large profits from users interested in learning. For example, if you want to understand the risks of floating point calculations given the limitations of computer hardware by analyzing how LAPACK and BLAS manage the problem, you can have an LLM write up a whole course syllabus on the subject and work your way through examples provided by the LLM with pretty good certainty that it's not hallucinating you into a corner. This is a well-studied topic, so one feels fairly confident that the LLM, properly prompted, is going to give you good information and if you cross-check with a textbook you won't be surprised.
In practice, I find this approach reduces productivity in favor of gaining a deeper understanding of how things work - instead of just naively using LAPACK/BLAS based libraries, one 'wastes time' diving into how they work internally, which previously would have been very opaque.
These are tools, it's up to you how you use them. Kind of like compilers, really.
Great write up. The writing style reminds me of my new favorite phrase “imitation and innovation” the author has right amount of quotes and comparable new thoughts. This keeps the reader engaged and not feel like how much more we have to go. I am 95% sure LLM cannot write like this, may be it can if the author wrote more pieces :)
> program building is an entropy-decreasing process...program maintenance is an entropy-increasing process, and even its most skillful execution only delays the subsidence of the system into unfixable obsolescence
> Only humans can decrease or resist complexity.
For a simple program, maintenance is naturally entropy-increasing: you add an `if` statement for an edge case, and the total number of paths/states of your program increases, which increases entropy.
But in very large codebases, it's more fluid, and I think LLMs have the potentially to massively _reduce_ the complexity by recommending places where state or logic should be decoupled into a separate package (for example, calling a similar method in multiple places in the codebase). This is something that can be difficult to do "as a human" unless you happen to have worked in those packages recently and are cognizant of the pattern.
I use LLMs for annoying shit that used to take an inordinate amount of time. For example an analyst gives me a CSV, it has some data clean up issues, eventually it needs to become a series of SQL inserts.
What was a day of script writing becomes 15 minutes of prompt engineering to clean up the CSV and convert it into the proper statements. Massive time savings!
I believe one can vibe code a basic app or script. Like building one of those little trinkets people put on websites, like calculators or mini games.
But I agree, the LLM can’t build, say, Google Chrome, or AWS Cognito, or Netflix, or anything that actually has serious billion dollar value. It’s not a “when” question, because these are not “solvable” problems.
This subject has become such a Rorschach Blot. Some blame their middle managers, some blame their coworkers, some blame folks from this field or that. I'm guessing that's not a new thing to complain about, it's just a new reason.
I call it fast-food software, but that doesn’t mean everyone holds the tool in the same way, the collaboration and use of an assistant to research or modify specific things is the key value-add.
But many people will use unrestrained tool-enabled agents to do all of their work. And so be it, there’s already bad software out there, always has been. It’s easier to make and distribute software. It’s high calorie and tastes great and is FAST.
But you have to hope it will be more of a tool than a full meal.
Power-accelerated imperfection. Your great-grandfather's combustion engine might run great on vegetable oil, but if you put vegetable oil in a race car, bad things will happen. The more powerful your tools, the more dangerous you are. But, that's the cynical way of looking at things. My prefered outlook is tools are great, and being an adult is great, and when you put these two things together: that's even better.
My most recent ai related peeve is people releasing some ai tool and claim that it's "democratising x". "democratising art", "democratising education" -- yeah giving all your data to and relying on a centralised model run by a fascist mega corp is so totally democratic. Maybe can't blame them -- that is what they actually mean based on "democracy" in the US etc though.
> It's still the wrong answer, because the question should have been, "How can I make this code thread-safe?" and whose answer is "Use System.Collections.Concurrent" and 1 line of code.
I often found myself adding "use built-in features if they exists", just because because of this type of scenario.
It unsettles me that some people feel okay always accepting AI code, even when it "works".
> LLMs as they currently exist cannot master a theory, design, or mental construct because they don't remember beyond their context window.
That's not categorically true: if a theory/design fits in their context window it's possible that they _can_ master it.
LLMs shine for simple tasks with little context. There are plenty of tasks like that.
AI drive me crazy! For simple things they are ok, but with complex problems, they can't solve it and my blood pressure explodes. ;D
If you have a new unknown, tool, things go wrong in two ways. For example a hammer.
If it works, like to hit a nail, you end up smashing everything in sight. If it fails, like digging a garden, you end up thinking it is stupid.
But there is a third case.
You use it to do something that you did not know you could do before. Like to planish metal.
People are experiencing the first case and second.
> If you're a skilled, experienced engineer and you fear that AI will make you unemployable, adopt a more nuanced view. LLMs can't replace human engineering.
Ah, but you see, without any promise of pudding: why eat the meat? I do the 'human engineering' part only to do... the rest of the engineering.
I wish more people could understand this. Yes, LLMs help me learn faster, brainstorm ideas and so on, but to say that it can generate code and hence you can do complex things easily does not make sense.
For me, writing code has never ever been the challenge. Deciding what to write has always been the challenge.
In principle, mental constructs don’t need to remain in your head. They could be documented. But writing documentation, reading it, and maintaining it takes time.
With LLM assistance, it might become easier to maintain a Markdown file containing the “theory” of a program. But what should be in the file?
> LLMs give me finished thoughts, polished and convincing, but none of the intellectual growth that comes from developing them myself
Even if all the wonders were true that people love to believe in about LLMs, you cannot get around this argument.
The author seems to have an inflated notion of what developing software is about. Most software doesn't require "perspicacity". Think about Civil Engineering. In a very few cases you are designing a Golden Gate Bridge. The vast majority of the time you are designing yet another rural bridge over a culvert. The first case requires deep investigation into all of the factors involving materials strength, soil dynamics, ... In the second case your role as an engineer is to run the standard calculations so that it meets legal standards and isn't going to fall apart in a couple of years.
We all like to think the we are grand architects finely honing software for the ages, when in reality we are really just specifying that the right grade of gravel is being used in the roadbed so that you don't get potholes.