▲ 217 points
back
108 comments
I kind of have a different idea of agents. I totally believe in a deterministic scaffold but I really think that an agent should be as deterministic as possible - the more code, the better.
Think of a typical loop we may ask of Claude Code today (assume we are not using TDD): run some test suite with fail fast mode, diagnose if the failure is due to recent feature changes (pass reference to backend/frontend, github issues, PRD,...). Ask CC to decide if test failed due to feature change and then update the test. Perhaps ask CC to use sub-agent to investigate and fix (if deemed so). Commit each fix, move on to next.
I know, this has so many ways to make blunder but I am talking about the agent here, not our error-prone test maintenance. What if we had an agent that had context of your codebase, deterministically ran test suite, linter, hooks, etc. The "English" prompt would become a code loop with the LLM only brought in to decide if a test has failed because of feature change. Also, we can extract git log, JIRA and what not.
Each tool here is real code. Executable code that calls others and only prompts when they meet edge cases. Edge cases are defined but we can now accelerate the maintenance of these tools using agents themselves. But the system is built on "programs that do one thing and do it well" and then reach out to an LLM for its specific edge case. The agent is how these executables work with each other.
100% agree that the more deterministic code the better up to the limit where you need the LLM's ability to be non-deterministic to kick in.
There is this ACM blog post called "Manual Work is a Bug" [0] that was originally written to help humans automate processes using code. I find it just as applicable today as when it was written. You and the LLM look at what has to be done and then figure out the scripts/tools to make it happen. You then tie those tools into a system.
The more I use the above the more it makes sense and the worse the whole "just commit the prompt" seems like nonsense.
In the age of agents, I feel like BDD is more important than ever since it describes the behavior, and if it does so well it’ll make it much easier for these tools to pick up correct behavior.
Gherkin style tests also come to mind
I’m definitely on the deterministic code train as well. All of my success for long running tasks has been around wrapping the agentic harness (cc, codex-cli, etc.) in a deterministic workflow with deterministic gates. We need a name for this outer layer. In my mind that is the true harness because it constrains the agents failure mode. I think flow engineering has been proposed. Maybe it’s the agentic exoskeleton?
What you are describing is precisely my goal with my agent framework.
One of the meta-processes designed in is pushing automated processes, both defined and discovered, down as far as possible. "Down" here means as far towards the metal as reasonable. So automate the automatable stuff, and leave the LLMs to do stuff LLMs are actually good at.
A trivial example is 'handle this bugfix ticket'. Many actions in a bugfix are pre-defined, for example a git commit at the end of the ticket. So Maelstrom will, at the end of a bugfix workflow, will force a git commit from the LLM that did the implementation. The LLM never even sees the git command, it just fills in a JSON field with a commit summary, and the workflow handles the commit.
If we really have intelligent LLMs, then I would guess they are going to inflate their token rates, which when someone sees "token costs" they should think "invisibly variable consulting rates".
I just did a complex (for me) task: I needed to wrap a 2015 build of Dosbox Daum, a 32 bit binary, in an AppImage. Claude kept finding incremental bugs, and I went through two cycles of depletion of my token rate with Claude. It kept getting close, but..... something was off each time.
So I took the Claude output and Chatgippity polished it off with a few more rounds. I then wondered how much Claude was "just showing enough" to try to hook me into subscribing.
That said, LLMs were quite useful, and I learned a lot about ELF binaries, and extracting dependencies. It's the ideal task: a breadth/obscure task that is documented but poorly explained, that I wouldn't have easily been able to do without LLMs.
Anyway, back to the article, do we really want arbitrary-billing silent tasks running? Like AWS billing spikes are bad enough to lose sleep over.
Also, if you want quiet rebellion against AI, developers should shove as much busywork on AI to overwhelm the AI budgets for your orgs, because it is very apparent to me that you can keep the LLMs doing lots of hardening, testing, redundacy, and optimization tasks with larger and larger and larger token windows and burn those tokens baby.
One of my policies for agentic coding is to spend much effort in developing tests, coded tests not LLM based vibes. My projects have around 1:1 LOC between code and tests. Tests are like skin, when the skin is pricked it hurts, agents need to feel pain too.
OP's idea "everything is a text file" is good and I use it too. My plans are saved as task.md files, numbered and named. Work items are checkboxes inside the file, closed work items are checked and a comment is added on the same line to provide feedback about the implementation.
I also keep a current-state-of-the-world document, it should be <20KB of text, keep the essential decisions and intents. Loading it allows resuming in <30s.
Something I never saw anyone else do - I save all user messages in a chat_log.md file which is referenced for intent alignment and state recovery. I consider the chat log on the one hand, and coded tests on the other hand as the two walls, the agent works in the mid section between them.
https://horiacristescu.github.io/claude-playbook-plugin/docs...
I believe there should be easy ways to move logic between prompts and code in a smooth ways. Moving from prompts to code is for getting deterministic, fast, well defined and cheap execution - the other way is when you want to quickly extend your system or when you want to relax a pre-condition (which is extending if you look into it deeply).
There are some inroads into this vision - but I haven't seen anything build directly for this (beside my own experiment).
I have some 'vibe noted' notes on this: https://zby.github.io/commonplace/notes/unified-calling-conv..., https://zby.github.io/commonplace/notes/rlm-tendril-and-llm-...
I know it's a type of blasphemy here, but deterministic workflows such as what you describe is where langgraph really shines imo.
I love the flexibility of agents in a regular harness but mostly for investigation, planning etc.
Once I actually have my plan/spec, it's the same process every time, it needs to be as deterministic as possible, using agents as tools throughout the process.
Yada yada yada introduction done so I can drop the link to what I'm building which is exactly that
I keep pushing back open sourcing it but it's truly close to ready and will be fully free to use.
It's the most advanced deterministic agentic orchestrator on the planet.
I think AI usage is going to fall into exactly this pattern the same way the frontend + API has become the "obvious" approach. We mostly operate with known workflows and just need a touch of LLM magic to determine when to do what.
Right now we mostly YOLO prompts with some docs/skills in the mix but I think it will start to look more like internal MCPs, with tools the LLM can string together. I think the reality will be most tasks end up serviceable by Haiku-level LLM, not Fable.
You might like this: https://github.com/deepclause/deepclause-sdk.
It’s a DSL I’ve been working on to encode mixed deterministic/probabilisitic agent behavior.
I don't think we're necessarily in disagreement here: I agree that determinism should be taken as far as possible, but once we zoom out of the software engineering world I think things tend to get a lot less easy to automate if that makes sense.
agreed. i built this for a wide variety of background workflows that have been running seamlessly for many weeks now. it's an overbuilt personal project but i think pretty dialed in and very flexible https://stepwise.run/
This post is chock-full of soft ideas. They make no meaningful steps toward "a harness that can do anything". The suggestion is to replace a small node application with specific tooling, with a vm "to give it more capabilities". This is what Agent sandboxes are, already. Making the sandbox the harness, doesn't achieve a concrete goal.
I find it refreshing to read a blog of someone just noodling on an idea rather than pitching a new product.
I feel achieved in running pi in containers!
the current bottlenecks for agents are actually bottlenecks of the LLMs, rather than the agents themselves.
I spent 6 months frantically working fulltime on a harness, and when it was finished, I just stopped using it and went back to Codex CLI. I've also stopped using all the fancy features: subagents, planning mode. I've also stopped worrying about context, since I just let codex hit compaction. Somehow this doesn't seem to hurt much. Weird.
harnesses i suspect are better for noncoding tasks like wrapping claude code to be a personal assistant... but im still procrastinating on building mine... if only i had a personal assistant to tell me how much time i should spend on building a personal assistant wrapper for claude
You adapted to how your tool works rather than try to shape it into what it isn't. So far all the nondeterministic (prompt based) workflows I have tried have been meh in an "I can't tell if it's any better but it sure is more complicated and less portable" way.
Current tools. Opencode and whatever cli i can't avoid (like claude code for my first month which I don't thinking I'll renew) usually accessed using Paseo for it's excellen mobile client.
I disagree with the idea that file is a good metaphor for LLM. Files have seek and byte streams, which is just an unneeded abstraction for LLM. The LLM doesn't need to seek or jump to the middle of a file, if you store and organize your data properly.
Why force the LLM to use files over vector database or key-value stores, just because it's a design principal for UNIX (which is designed for human users, not LLMs.)
What has been the most helpful when developing harnesses:
> When in doubt, simplify. Remove, trim and minimize. Reproduce issues in as small cases as possible, understand the full design completely, there is no shortcuts for this.
I still find it hard to develop harnesses because you can't really test many turns with an llm in the middle. I suppose doing LLM as judge is one way to start to tackle this kind of thing.
Love the Unix philosophy and the buffs mentioned of the Linux FS. Lean, transparent, and auditability-first is exactly the direction harness' should continue in.
Something I am convinced of though, there probably isn't a single `best` harness for all tasks. Different workloads will likely perform better with certain combinations of model + harness, especially when we are talking about token budgeting and cost tracking.
Ambiance feels like a great base “kernel” to build those variants on top of, rather than the one true harness.
How much do the labs post-train on the harness inputs & outputs? That's a critical piece to understand if a "generic" harness is at all possible
I really like this idea and the way you mapped the concepts to unix primitives. Indeed llms are already "unix native". I've been experimenting with similar event driven workflows using k8s primitives but that's one level up the stack. This makes a whole lot of sense to me in terms of organizing a shared mental model. Will definitely check it out. Thanks for the good work.
>I've been thinking about how to free LLMs from the chat pane for a few years now.
LLM's are language models, you can absolutely control them with bash scripts and deterministic code, there are plenty of frameworks that already do that, and a great engineer will use them, but LLMS are at their most powerful when a user can give the agent an input and the model can run its ReAct loop. Wanting to free an LLM from a chat pane is like wanting to free email from the thread model, or closer, removing the chat window to DM a friend or colleague.
>what can we learn from the before-fore times, when people used to actually write code?
Treat an agent like a human writing code. Give them the best context, give them the best tools. This is why harnesses are overly complicated, because they need to guide the model through the context and tools it has available in a way that is efficient. A good harness is not incompatible with the Unix philosphy, it can do one thing well (interfacing with LLMs and giving them access to filesystems and compute), it will heavily use bash, stringing commands together with the cli tools that it knows (it's context) that it has, and and LLM will naturally handle text streams because that is what it does best.
>Everything is a File
If you want things to be deterministic why resort to plaintext? Wouldn't we want as much as possible to be typed? A computer can parse json which is what you want if you are trying to make your harness as deterministic as possible.
>It watches our FS for changes with cursors on textfiles,
Wow. What is your monthly token bill? I don't know how that would use less tokens than a 30 minute heartbeat, which as you mention will already use a lot of tokens. Why not have it notify your agent after a certain amount of files have been changed, or certain files you deem important?
It seems like this user works at a 12 week programmer retreat and seems to post their cohort's blog posts about the projects they work on.
It’s interesting to me how many people articulate things like the “Preiminary Truths” section as if they are novel insights.
Effective people managers (of whom I would not specifically consider myself) have known these tenets for as long as history. “Be concise”, “state your intent clearly”, funny how these are touted as novel “strategies” with which to expertly direct AI.
I don’t agree that “everything is a file”. Files are arrays of bytes. For an LLM, everything is a vector of tokens/embeddings.
An aphorism I recently heard: "All sufficiently advanced technology eventually becomes a web browser".
… seems apt especially in the context of the progression from chat-windows to harnesses and onwards to “harnesses that can do anything”.
I think domain specific harnesses are already surpassing generic harnesses. I also think software development is its own domain.
My harness is a Claude Code plugin with its own brainstorming, adr, and planning skills with associated review and interview skills. Behavioral testing related to acceptance criteria is built in. Everything in my harness is gated to prevent ratholes.
I recently inflated a docker container to execute a set of work with Claude in unsafe mode and immediately saw problems with everything it was doing…and then I realized I had not installed my harness.
Running Claude without an engineering harness is like driving a car without brakes or a steering wheel.
Like many others in also building a wrap… sorry I meant a harness for Claude and codex, and though I’m initially going into the route of “VFx node editor, but with prompts” - why not go full meta and actually have your harness bootstrap another harness / flavor of a given tool or combination thereof as a generative output.
Current project: https://sxp.studio/apps/subjectivezero
Mapping agent concepts to the Unix environment sounds like a great idea in general, but I get off the train at adopting the FHS, which is an ancient relic that has no place in a green-field system. I don't know exactly what shape it should take, but something along the lines of Nix seems more appropriate. Or maybe Plan 9.
“That can do anything” gives me Zombo.com vibes. I would be more interested if it seemed to do particular things well.
What's with this "harness" word people have been trying to adopt lately? Are we all going rock climbing?
Oh, it’s macOS only. The site didn’t say anything about it, or maybe I’m just blind. I had to look at the repo to see.
Why are we not just using Claude Code or Codex on our machine and using this thing? Real question...
Truly enjoyed this article, but hate to admit one thing: I'm so burned by AI slop that once I noticed this seems to have been "assisted" by an LLM, I got irrationally angry because I felt tricked.
It hit me at the end, when I read this:
> The idea behind Ambiance is simple: the model's priors [...] Everything else here is just in service of that.
I've noticed "in service of" take off similar to "load-bearing" with LLMs, and the whole structure just pattern matched to Claude for me.
I went back and scanned it over again, and noticed several other tells:
* "Think of U/L [Unix / Linux] as a motivating analogy rather than a direct comparison." * "Priors" _and_ "a priori" used in the same article. * "A real kernel […]. The Ambiance Kernel […]. The Kernel […]." — LLMs love this pattern.
To be abundantly clear, *I'm not calling this AI SLOP*; it's obvious that a human put a lot of thought into this, gives a shit about the topic, and shared interesting ideas leading to a productive discussion. I really did like it.
There's just something empty or hollow about the LLM style; paraphrasing Joni Mitchell "there's something lost and some thing gained" [1] when using AI. Your writing, your code is more consistent, more structured, more planned out — generally better but in a way that loses the character behind human writing.
[1]: From "Both Sides Now", a great song about looking something from two perspectives: youthful innocence, and jaded cynicism. Listen to the original 1969 version first, and then the 2000 remake as you can tell she's singing from the respective perspectives. Deeply meaningful song!
> that can do anything
isn't that a tall claim?
I was hoping this was a climbing harness that could service other trades too
Just one more harness bro, one more harness I swear one more harness and we will have solved AGI bro
I use AI all the time
But
I don't want the AI to summarize my email for me.
I don't want the AI to summarize my calendar for me.
I don't want the AI to summarize my Zoom call for me.
Thanks