Getting AI to work in complex codebases

> Sean proposes that in the AI future, the specs will become the real code. That in two years, you'll be opening python files in your IDE with about the same frequency that, today, you might open up a hex editor to read assembly.

> It was uncomfortable at first. I had to learn to let go of reading every line of PR code. I still read the tests pretty carefully, but the specs became our source of truth for what was being built and why.

This doesn't make sense as long as LLMs are non-deterministic. The prompt could be perfect, but there's no way to guarantee that the LLM will turn it into a reasonable implementation.

With compilers, I don't need to crack open a hex editor on every build to check the assembly. The compiler is deterministic and well-understood, not to mention well-tested. Even if there's a bug in it, the bug will be deterministic and debuggable. LLMs are neither.

All I know is that it works. On the greenfield project the code is simple enough to mostly just run `/create_plan` and skip research altogether. You still get the benefit of the agents and everything.

The key is really truly reviewing the documents that the AI spits out. Ask yourself if it covered the edge cases that you're worried about or if it truly picked the right tech for the job. For instance, did it break out of your sqlite pattern and suggest using postgres or something like that. These are very simple checks that you can spot in an instant. Usually chatting with the agent after the plan is created is enough to REPL-edit the plan directly with claude code while it's got it all in context.

At my day job I've got to use github copilot, so I had to tweak the prompts a bit, but the intentional compaction between steps still happens, just not quite as efficiently because copilot doesn't support sub-agents in the same way as claude code. However, I am still able to keep productivity up.

-------

A personal aside.

Immediately before AI assisted coding really took off, I started to feel really depressed that my job was turning into a really boring thing for me. Everything just felt like such a chore. The death by a million paper cuts is real in a large codebase with the interplay and idiosyncrasies of multiple repos, teams, personalities, etc. The main benefit of AI assisted coding for me personally seems to be smoothing over those paper cuts.

I derive pleasure from building things that work. Every little thing that held up that ultimate goal was sucking the pleasure out of the activity that I spent most of my day trying to do. I am much happier now having impressed myself with what I can build if I stick to it.

But let's assume you're much better than average at understanding code by reviewing it -- you have another frustrating experience to get through with AI. Pre-AI, let's say 4 days of the week are spend writing new code, while 1 day is spent fixing unforseen issues (perhaps incorrect assumption) that came up after production integration or showing things to real users. Post-AI, someone might be able to write those 4 days worth of code in 1 day, but making decisions about unexpected issues after integration doesn't get compressed -- that still takes 1 day.

So post-AI, your time switches almost entirely from the fun, creative act of writing code to the more frustrating experience of figuring out what's wrong with a lot of code that is almost correct. But you're way ahead -- you've tested your assumptions much faster, but unfortunately that means nearly all of your time will now be spent in a state of feeling dumb and trying to figure out why your assumptions are wrong. If your assumptions were right, you'd just move forward without noticing.

It starts with /feature, and takes a description. Then it analyzes the codebase and asks questions.

Once I’ve answered questions, it writes a plan in markdown. There will be 8-10 markdowns files with descriptions of what it wants to do and full code samples.

Then it does a “code critic” step where it looks for errors. Importantly, this code critic is wrong about 60% of the time. I review its critique and erase a bunch of dumb issues it’s invented.

By that point, I have a concise folder of changes along with my original description, and it’s been checked over. Then all I do is say “go” to Claude Code and it’s off to the races doing each specific task.

This helps it keep from going off the rails, and I’m usually confident that the changes it made were the changes I wanted.

I use this workflow a few times per day for all the bigger tasks and then use regular Claude code when I can be pretty specific about what I want done. It’s proven to be a pretty efficient workflow.

[0] GitHub.com/iambateman/speedrun

I made specs for every part of the code in a separate folder and that had in it logs on every feature I worked on. It was an API server in python with many services like accounts, notifications, subscriptions etc.

It got to the point where managing context became extremely challenging. Claude would not be able to determine business logic properly and it can get complex. e.g. if you want to do a simple RBAC system with an account and profile with a junction table for roles joining an account with profile. In the end what kind of worked was I had to give it UML diagrams of the relationship with examples to make it understand and behave better.

Also quite funny that one of the latest commits is "ignore some tests" :D

Sorry, had they effectively estimated that an engineer should produce 4-6KLOC per day (that's before genAI)?

the key part was really just explicitly thinking about different levels of abstraction at different levels of vibecoding. I was doing it before, but not explicitly in discrete steps and that was where i got into messes. The prior approach made check pointing / reverting very difficult.

When i think of everything in phases, i do similar stuff w/ my git commits at "phase" levels, which makes design decision easier to make.

I also do spend ~4-5 hours cleaning up the code at the very very end once everything works. But its still way faster than writing hard features myself.

> We've gotten claude code to handle 300k LOC Rust codebases, ship a week's worth of work in a day, and maintain code quality that passes expert review.

This seems more like delegation just like if one delegated a coding task to another engineer and reviewed it.

> That in two years, you'll be opening python files in your IDE with about the same frequency that, today, you might open up a hex editor to read assembly (which, for most of us, is never).

This seems more like abstraction just like if one considers Python a sort of higher level layer above C and C a higher level layer above Assembly, except now the language is English.

Can it really be both?

An hour for 14 lines of code. Not sure how this shows any productivity gain from AI. It's clear that it's not the code writing that is the bottleneck in a task like this.

Looking at the "30K lines" features, the majority of the 30K lines are either auto-generated code (not by AI), or documentation. One of them is also a PoC and not merged...

This is exactly right. Our role is shifting from writing implementation details to defining and verifying behavior.

I recently needed to add recursive uploads to a complex S3-to-SFTP Python operator that had a dozen path manipulation flags. My process was:

* Extract the existing behavior into a clear spec (i.e., get the unit tests passing).

* Expand that spec to cover the new recursive functionality.

* Hand the problem and the tests to a coding agent.

I quickly realized I didn't need to understand the old code at all. My entire focus was on whether the new code was faithful to the spec. This is the future: our value will be in demonstrating correctness through verification, while the code itself becomes an implementation detail handled by an agent.

But.. I hate this. I hate the idea of learning to manage the machine's context to do work. This reads like a lecture in an MBA class about managing certain types of engineers, not like an engineering doc.

Never have I wanted to manage people. And never have I even considered my job would be to find the optimum path to the machine writing my code.

Maybe firmware is special (I write firmware)... I doubt it. We have a cursor subscription and are expected to use it on production codebases. Business leaders are pushing it HARD. To be a leader in my job, I don't need to know algorithms, design patterns, C, make, how to debug, how to work with memory mapped io, what wear leveling is, etc.. I need to know 'compaction' and 'context engineering'

I feel like a ship corker inspecting a riveted hull

1. Break down the feature or bug report into a technical implementation spec. Add in COT for the splits. 2. Verify the implementation spec. Feed reviews back to your original agent that has created the spec. Edit, merge, integrate feedback. 3. Transform implementation spec into an implementation plan - logically split into modules look at dependency chain. 4. Build, test and integrate continuously with coding agents 5. Squash the commits if needed into a single one for the whole feature.

Generally has worked well as a process when working on a complex feature. You can add in HITL at each stage if you need more verification.

For larger codebases always maintain an ARCHITECTURE.md and for larger modules a DESIGN.md

Please kill me now

But the write up troubles me. If I'm reading correctly, he did 1 bugfix (approved and merged) and then 2 larger PRs (1 merged, 1 still in draft over a month later). That's an insanely small sample size to draw conclusions from.

How can you talk like you've just proven the workflow works "for brownfield codebases"? You proved it worked for 2/3 tasks in 2 codebases, one failure (we can't say it works until the code is shipped IMO).

It'd be nice if the article included the cost for each project. A 35k LOC change in a 350k codebase with a bunch of back and forth and context rewriting over 7 hours, would that be a regular subscription, max subscription, or would that not even cover it?

2. Write down the principles and assumptions behind the design and keep them current

In other words, the same thing successful human teams on complex projects do! Have we become so addicted to “attention-deficit agile” that this seems like a new technique?

Imagine, detailed specs, design documents, and RFC reviews are becoming the new hotness. Who would have thought??

I've been working on something what I call Micromanaged Driven Development https://mmdd.dev and wrote about it at https://builder.aws.com/content/2y6nQgj1FVuaJIn9rFLThIslwaJ/...

I'm in a similar search and I'm stoked to see that many people riding the wave of coding with AI is moving in this direction.

Lots of learning ahead.

So with this approach of spending 3 hours on planning without verification in code, that's too hard for me.

I agree the context compaction sounds good. But I'm not sure if an md file is good enough to carry the info from research to plan and implementation. Personally I often find the context is too complex or the problem is too big. I just open a new session to resolve a smaller, more specific problem in source code, then test and review the source code.

The hierarchy of leverage concept is great! Love it. (Can't say I like the 1 bad line of CLAUDE.md is 100K lines of bad code; I've had some bad lines in my CLAUDE.md from time to time - I almost always let Claude write it's own CLAUDE.md.).

- Mr. Snarky

It's super effective with the right guardrails and docs. It also works better on languages like Go instead of Python.

An abstraction for this that seems promising to me for its completeness and size is a User Story paired with a research plan(?).

This works well for many kinds of applications and emphasizes shipping concrete business value for every unit of work.

I wrote about some of it here: https://blog.nilenso.com/blog/2025/09/15/ai-unit-of-work/

I also think a lot of coding benchmarks and perhaps even RL environments are not accounting for the messy back and forth of real world software development, which is why there's always a gap between the promise and reality.

Only if AI code generation is correct 99.9% of the time and almost never hallucinates. We trust compilers and don't read assembly code because we know it's deterministic and the output can never be wrong (barring bugs and certain optimization issues, which are rare/one-time fixes). As long as generated code is not doing what the original "code" (in this case, specs) doing, humans need to go back to fix things themselves.

Edit: typo.

With this I find that most of the shenanigans of manual context window managing with putting things in markdown files is kind of unnecessary.

You still need to make it plan things, as well as guide the research it does to make sure it gets enough useful info into the context window, but in general it now seems to me like it does a really good job with preserving the information. This is with Sonnet 4

YMMV

> I had to learn to let go of reading every line of PR code

Ah. And I’m over here struggling to get my teammates to read lines that aren’t in the PR.

Ah well, if this stuff works out it’ll be commoditized like the author said and I’ll catch up later. Hard to evaluate the article given the authors financial interest in this succeeding and my lack of domain expertise.

Research helps with complex implementations and for brownfield. But it isn't always needed - simple bugfixes can be one-shot!

So all AI workflows could be expressed with some number "N" of "document expansion phases":

N(0): vibe coding.

N(1): "write a spec then implement it while I watch".

N(2): "research then specify". At this point you start to get serious steerability.

What's N(3) and beyond? Strategy docs, industry research, monetization planning? Can AI do these too, all of it ending up in git? Interesting to muse on.

It's kind of like if you could chat with Repomix or Gitingest so they only pull the most relevant parts of your codebase into a prompt for planning, etc

I'm a paying RepoPrompt user but not associated in any other way.

I've used it in conjunction with Codex, Claude Code, and any other code gen tool I have tried so far. It saves a lot of tokens and time (and headaches)