Item — HN Reader

by jeffreyrogers·6d ago·view on hn ↗

I was fairly skeptical of agentic coding before I used it for a real product. Although I still have to be heavily involved in planning the code that LLMs write for me, they can write code much faster than I can, and they know more about edge cases than I do, so they can handle edge cases/subtle bugs that I would have missed. I have been paid to write code at every level of the stack from assembly to frontend javascript, but I'm not equally good at all those areas. In some areas I can still outperform LLMs, but for areas I'm weak they do a much better job than I would have.

I still think of what I'm doing as software engineering, and I'm glad that I had many years of professional and hobby development before using agents since I think that's given me the ability to make good architectural decisions (and helps me resteer the LLMs when they want to do something suboptimal), but my involvement in actually writing code is quickly going to zero. That said, they aren't perfect and they still introduce bugs, but I believe the quality of my current product is higher than what I would have created pre-agentic coding.

Things I've found helpful in keeping quality high:

- Visual regression tests (detect UI bugs before you commit them)

- Fuzz testing of interfaces and app behavior

- Automatically add regression tests for any bug that I/the LLM fixes

- Logging/alerting that tracks an errors/invariant violations triggered in the app

- Performance metrics that are surfaced in a dashboard.

All of these are very easy to add since the LLM can create this infrastructure for you. The fuzz testing in particular is something very few products I've previously worked on have since most people don't know how to implement it. I ran the fuzzers for a few minutes and they quickly caught multiple subtle bugs that I was not aware of.

This is a real product that helps a real, non-VC funded service business, and although I could have made something similar myself it would have taken me a lot longer, be harder to use, and probably be less reliable.

Edit: while it's true that you can quickly blow through the $20/month plan, the $200/month plan allows you to get a lot done and is basically sufficient for my needs. It's also very cheap when you consider what it would cost to pay someone to do similar work.

Adding too many tests was a real issue before LLMs. You end up in situations where you when you add a feature there's 200 (out of e.g. 10k) tests that fail and you have to figure out which of them should fail and you need to fix them and which are actual bugs.

I haven't run into this yet. My test failures have either been real or have been triggered by an (intentional) breaking change. The later does require updating the tests, but I think catching the real bugs is worth that tradeoff. I haven't found spurious failures to be a big problem (I've had a few but rewriting the tests that have this problem has eliminated it for me). My codebase is a pretty modular rails app, which I think helps with this.

I have worked on other projects where flaky tests are a problem (and generally cause developers to ignore and submit anyways), but so far I've avoided that. My experience with flaky tests is that they're typically due to poor modularity or to subsystems that other teams can modify. This project exists in a monorepo and I'm the sole developer so that's not a problem here.

I disagree, I say the more tests the merrier, if you can afford the time cost. It's fairly rare to see faulty tests in my experience. Yes you can have hundreds failing tests and the code still works 100% - because those tests are testing API usage in a way the actual application never uses. But that's still a bug, it's just a future bug waiting to happen that you caught early. I wouldn't delete those tests.