Item — HN Reader

What size project an AI can complete on its own is IMHO not an interesting question. The AIs and harnesses are getting good enough that quite large things can be built, primarily as a function of cost.

The interesting question in my view is what is the largest system an AI can maintain over a long time horizon -- while keeping a coherent architecture that adapts and responds well to change. All the while remaining performant, scalable, and understandable to humans. I am unaware of what is the currently longest-lived vibe coded project but it cannot be more than a few years old.

So to my mind, we don't have very much real data yet.

2 comments

I think reproducing existing software and building a new product are quite different tasks.

In a benchmark, there are usually clear tests and a correct reference. In actual product development, requirements are often unclear, and we do not always know what the correct result is.

Passing tests is also not enough to confirm security, maintainability, or operability. I would like to see a benchmark where an AI continues changing the same product for several months. It would be interesting to see whether the architecture remains understandable and safe after many changes.

isn't this a harness engineering problem at its core (and the human equivalent being a product org problem)?

I don't believe it is. If we think of AI-maintained software as automated gardening / automated agriculture, the harness is all the machinery that can plow, sow, water, monitor, harvest, etc. There still has to be a human stewarding the overall ecosystem and tending the garden. The human is 'on top' of loop rather than 'in it' but I don't think a harness by itself can be a good steward of architecture.

Product org problems also is very context-dependent. Product orgs can work significantly different, in some orgs humans desire to be deep in the details, in other orgs the product folks are happy to let engineering drive the boat beyond a certain level of complexity. So one size harness can't necessarily fit all.