back
6 comments
Using Opus to judge the outcome of "good code" isnt exactly the sign of a great experiment.
I don't know why you would run an agent loop without TDD? Should you write code without test coverage? So something must run the tests to be sufficient anyway?

> TLDR; Based on Opus's judgment of the quality of the outcomes, there was no clearly discernable difference based on TDD workflow versus no TDD workflow. On the contrary, more than once Opus ranked the non-TDD workflow solutions slightly higher in design and test quality. There was also no meaningful difference in mutation scores across the solutions.

That's really surprising. Was there a difference in cost?

Does it matter whether you ask the agent to write the tests first or last? Is one way TDD and the other not?

Without property testing or fuzzing or formal methods, could the code metrics in this experiment have been sufficient?

Perhaps there's a TDD gauntlet loop that's as justified as tests for QC; https://news.ycombinator.com/item?id=49261545

> Does it matter whether you ask the agent to write the tests first or last? Is one way TDD and the other not?

Yes. TDD = Test _Driven_ Development, meaning you write the test first, then the implementation.

If you write the test after, that's just called "development" afaik :)

I've only done TDD in very limited cases, like when you're implementing against a known spec. For "regular" work I'm usually still figuring out the shape and code interface for what I'm building, so writing tests first is impractical and wasteful.

Readme-driven-development is more fun: write the readme (ie: the intention) first.

But then what do you call coding without tests?

This says - if coding with an LLM - to write the tests before you commit.

Is there a name for AI looping until there's test coverage? Did this study require >98% test coverage? Line or branch coverage?

What will coverage-guided fuzzing fuzz if there's 100% test coverage?

Perhaps dev codebase comprehension is a significant unmeasured external benefit of writing tests first TDD, but LLMs don't train online so all you win is the context cost?

  I don't know why you would run an agent loop without TDD? Should you write code without test coverage? So something must run the tests to be sufficient anyway?
You can have the agent write them after. The article is specifically about TDD, meaning writing a small test that will fail, making it pass, (refactoring), and repeating until the feature is finished.

  Was there a difference in cost?
The author claims TDD used 3x tokens, but points out it might not be 3x the cost due to caching. It's near the end of the article.
A strict tests-must-be-written-ahead-of-time TDD advocate might say that writing the tests last is an error like p-hacking (choosing the threshold of statistical significance after the experiment) is an error.

Having always suspected that test-first TDD was most correct like adhering to the Scientific Method is most correct, I suspect that there are unmeasured differences between the results. But perhaps test-first TDD is a sham in AI and IRL