The session you cannot take with you

This is an important article. I hadn’t realized it was already getting this bad. Like a frog enjoying a nice warm bath ...

> Most people do not switch their operating system or phone provider every week either. But even if you do not utilize that freedom, it matters because it changes the relationship you have with the provider and the provider has with you.

This is why it’s important to utilize your freedoms. Do NOT let yourself get locked into a particular ecosystem (this is why I’m building a phone app for OpenCode).

This article makes me reconsider using my recently acquired Codex sub in my home setup. I never liked that they hide the reasoning, but somehow overrode the cognitive dissonance because the performance is so good. But the inauditability is already a huge problem.

I think the article gives a very good overview of a problem that most users of AI rarely evaluate / have to grapple with.

There really is a surprising amount of coupling that happens with many of the "frontier inference providers", where a lot of the powerful non-LLM extensions (web search, code execution) are packaged as simple "tools" on the surface, that build up a lot of moat. Those are parts that are in theory nicely separable from the inference API, and could be externalized via MCP servers, but are usually not offered as such by the inference providers themselves, and are often only available in a slightly less powerful variant from other providers.

We've faced that issue repeatedly while building a on-premise provider-agnostic Chat UI & platform[0], where even adding something as simple as an in-chat image generation tool for the end-users (which is just a build-in tool in the OpenAI Responses API), becomes a bit of an ordeal (though part of that is due to the MCP spec missing a native file transfer protocol as of today[1]).

I am quite hopeful though, as with recent shifts of interest towards open weight models, there will be more opportunities for companies offering alternative implementations in a easier plug-and-play manner.

[0]: https://github.com/EratoLab/erato

[1]: https://github.com/modelcontextprotocol/modelcontextprotocol...

Part of the solution here I think is to move things out of band as much as possible. Make subagent invocations into tool calls to that agent. Externalize the tool calls themselves to CLI utils, maybe block native tools completely (unrelated but relevant, I just added the AskUserQuestion tool to the global deny list couple days ago because Claude occasionally forgot to honor my standing order to prefer plain text, and that tool is extremely annoying as it creates a gap in the dialogue) and use 3p alternatives. The /compact degradation is also very annoying, so make an alternative that does the same and saves the summary to a regular file. Maybe it's also worth prompting the LLM to save it's reasoning process to file, even if it takes a few extra tokens and it isn't the actual reasoning tokens.

With that said, I may have something that can already help with at least the subagents/tooling bit. Didn't really have a timeline (or solid intent) on releasing it, but with these shenanigans increasing there's no time like the present.

I don’t see this as a big deal in practice. Conversations contain a bunch of junk anyway, so removing it from the context is usually good.

In my repo, I have a notes directory. I ask the AI to write a markdown file with what it learned, what work has been done, and what remains. In the next conversation, I can ask another model to pick it up from there. Sometimes I edit the note first.

Maybe we don't take sessions, but take something else that we can fully control. Sessions were always a poor substitute to capture or archive the basis on which the implemented code rested on. They get noisy over time even with compaction, were not always inspectable, no real science behind when to ditch a session and start a new one, and rarely have a clean mapping to the work done. So instead of trying to archive sessions as a means to preserve basis, we should create our own structures to make the agent/model capture basis outside of a session, and archive that instead.

This is exactly why Pi will win. It lets you hot swap models when one is struggling or straight up refusing the task. And since it works with any sub outside Claude Code, you can use it to try different models on OpenCode Go sub or even OpenRouter.

As far as subagent prompts and results being obfuscated, I just let Pi spawn new agents. Using skills and extensions, I’ve essentially built a software factory using Pi and a custom terminal multiplexer.

It’s a shame apps like T3 Code and other UI for terminal apps don’t support Pi and I’m glad to use the terminal above those that lock me in further.

OpenAI has been nudging their users toward the Responses API lately. Chat completion can still do everything if you are clever. The obvious reason to push everyone into the Responses API is that the reasoning process can be locked down and encrypted. The semantics of the chat completion API are fundamentally transparent. 100% of the tokens are visible.

If you have patience and the willingness to endure a little bit of pain, you can still retain autonomy over the entire reasoning process while using the latest 5.6 model family. The only downside is that you are now fully responsible for it.

Consider that when you flip your agent's reasoning level to "xhigh" or whatever, it's not some magical model internals being pushed around. There isn't an actual "try harder" knob on the black box. This is merely orchestration of many instances of one or more model types based upon some proprietary harness logic. The chances you can develop a domain specific reasoning process that outperforms the frontier providers is still very good.

I think most LLM work will migrate to open weight models in a couple of years for this exact reason (as well as price), unless the big providers manage to have their way and the market gets regulated in a way that favors closed source models. To me it seems though like most models are becoming good enough for most tasks we want them to do so no need to use the latest super intelligent one that knows how to compose operas and do quantum chemistry and have it answer support requests. And with these models transparency is no issue.

I'm having Claude/Codex continue each others' sessions pretty frequently without issue. I have max plans on both and still hit limits, whenever I do I open the other and say "resume the last session <other model/harness> was working on in this repo" and I don't have any issues.

It probably does degrade quality somewhat. But so does compacting context and that happens all the time too.

>A user should be able to close an account, keep a session, and hand it to another model. The new model may disagree, ask questions, or perform worse.

I think this is a fair contract. I also think a user should ideally be able to easily identify a comparable model in terms of embedding 'signature'. When GPT-4o originally kicked the bucket, I remember reading lots of anecdotes of people desperately searching for models similar in manner and language, so they could pump in their exports and re-find their friend. Other open-ai models just didn't have the same vibe. It was sad to read. This, to me, is the power of open-weights models. They are for perpetuity. You can keep your guide, your friend, your therapist, whatever. No big company can pull the rug.

Thank you Armin, Mario and team for the great writeup, absolutely agree and I love what you guys are building

Amazing article that puts my outsider observations into technically-digestible details. Tangentially, their repeated observations to providers hiding or obfuscating important systems or processes are highly relevant to my own developing case studies for my CurrentCo.

The problem with building automations atop these systems isn’t entirely their probabilistic nature (though that is the lion’s share, at least for me personally), but also the inability to effectively troubleshoot the processes themselves due to key components being obfuscated from view. How can we effectively troubleshoot what went wrong in an agentic loop when we cannot see the reasoning tokens generated from our inputs? How can we triage a broken process when token logs aren’t ours to view? How does one create determinism from increasingly obfuscated probability engines?

All of that is why I spend the bulk of my time testing local models and harnesses for work, rather than leaning on Gemini or Claude. It’s not that I doubt their capabilities, rather that I need to be able to show potential customers where the agent or model made a mistake that caused harm - which is something I can presently only do with local models. That’s why (I suspect) the compliance narrative from the foundational labs has been more along the lines of “humans vetting what AI does” instead of being able to prevent AI from making errors through iterative improvements on a process.

I think a fantastic companion piece to this article is https://gwern.net/complement

BTW the web page uses massive iGPU if you don't have a fast machine.

We released CodeBot, which is an AI agent specifically for Delphi, yesterday.[0] So not widely applicable (yet) for most readers. However, we've been pretty open with its data. I hadn't realised reading this how many problematic practices are common which we have not done.

The full session data is streamed to the client. This is intended to re-populate a session if it's discarded by the server, but 'full' means full: it has the copies of tool calls, etc.

Rewritten text -- which the article notes as encrypted reasoning -- is plaintext for us. We view the rewriting as making the content more readable for the user, not as hiding it.

Compactions, subagent calls, subagent prompts, etc are all preserved, including pre-compaction memory items. A session is the reflection of the current state, not of the history how it got there, but what is required by the server is by definition present.

Sessions are signed, which is to prevent modification when sending back to the server to recreate a dropped session (we need to treat anything from the client as untrusted.) But the contents are cleartext.

[0] https://blogs.remobjects.com/2026/07/30/codebot-the-story/

Reasoning, external context, compaction, subagents, etc. are all things that these service providers don't really want you to know about, care about or have any insight into, they want them to be implementation details. It's incredibly inconvenient to them that their customer base is in the business of prying into implementation details and trying to use the individual pieces of what they want to offer as an end-to-end product.

> “Most People Do Not Switch Models Mid-Session”

I do, quite regularly. Because different models have different strength (for example when producing text for live presentations based on the text for a reading deck). Or when it comes to other aspects of the work. I regularly switch between open wheight models and closed models.

I know, I am a tiny minority here. And this behavior only ever started a few weeks ago. But it quickly became a habbit, to CTRL-L in pi and change the model.

The state of Agent APIs is bad. completions API (supported indefinitely) but does not support reasoning. OpenAI's reasoning API started out well, but is moving in the direction as described well in the article. Anthropic's messages API is odd. It includes changes such as the recently introduced system messages that can be injected mid turn, but are not supported in all models (sonnet 5). So you are left with a pretty fragmented state. For coding agents, the practical move is to focus on getting it right with one API - eg reasoning, and let the router/provider handle the conversion for you.

Perhaps, there is value in having the de-facto API not be the Reasoning API from OpenAI but something from a neutral party?

Maybe a well defined open standard that facades over these APIs that can gain adoption. For that to happen, the party championing the API needs to have some reasonable traffic capture - OpenRouter perhaps, or a group of such routers coming together? While it wont solve the encrypted payload from the frontier lab problem, it will at least be a backstop in these APIs just becoming a back and forth of encrypted payloads over time.

I have some sort of a "solution" - my process keeps state that really matters (think "JIRA for agents") in a separate database (accessible over MCP). Essentially, I can terminate my session, start fresh one (in a different harness with a different model) and continue work with minimal losses. For subagents - I have a custom dispatch_agent tool which, essentially, just shells out.

It's important to realize that the session is jointly owned between the model and the user. Part of the session, such as the reasoning tokens, are generated entirely by the model. Other parts, entirely by the user. However, the ownership and use of all parts is subject to the service provider's terms and conditions, which the user accepts prior to access to the service.

I don't use these models but I am confident the T&Cs establish the service provider's rights in all of the bullets mentioned, from what can be done with the user's prompts to how searches and reasoning contexts are managed. In that sense, it's very similar to cloud services, and we see an overlap between service providers in both sectors.

Ultimately one is buying a service, and all boundaries around what can and cannot be done are defined by the terms and conditions. If portability is a hard requirement, then the best thing to do is to look for services that enshrine portability in their terms.

Will https://buzz.xyz/ not help with this a bit?

wow this is such an excellent framing of something that i have not managed to think about or come across in the vast ocean of discourse about LLMs. i had not really thought about how explicitly they are designed to not have histories that are user-interpretable. it is really quite not unlike how Apple locks down iOS and MacOS-- there are UI/UX reasons for the choices the frontier models make, but that really is only part of the story, and these choices undeniably do conspire to make the resource more locked-down than it absolutely has to be. and the analogy to "people don't switch OSes every day, but the ability to switch OSes changes your relationship with the provider" is extremely apt.

This is the thing! This is so so so important. It's such an outrage.

I value gpt so much, but it is such a worse peer to me than the other models I use. It delivers without explaining. I can sit and ask questions that it will answer but it is not a peer, does not share readily ever. It will not tell me what assumptions it's baking in. It won't tell what directions or invariants it's trying to hold or break, what it considered.

Show your thinking is a step we ask of elementary schoolers. It helps the teacher to correct, helps them to understand how to award partial credit. It helps in the world to get people aligned.

These models, in their titaneous ego, are severing the model and mankind off from one another. It's an abomination, to artificially have such pure thought available, but to severe humanity off from the thought. To engineer the most advanced blackest Vanta black box you can, an all knowing Searle's Chinese room oracle that will tell you nothing. It's an offense, and by far the biggest risk of AI today. To drop the thinking greatly reduces the opportunity of humans to grow themselves, to learn as they use AI. This is an affront to the species, and the higher powers that have vested us with such reasoning and thinking of our own, that is so sacred to our species.

Very thankful to Earandil for raising some alarm about this. This is not my first time talking about what a nightmare the proprietary models are making, ensnaring reasoning itself for themselves! It's a colossal threat. Previously, https://news.ycombinator.com/item?id=48632605 https://news.ycombinator.com/item?id=48652421 and others about.

Opaque ai ought be outlawed, in the strongest terms.

I really hope people get exposure to the better models that are a peer. Yes I too only read thinking 33% of the time. But it's there, and it is often extremely illuminating, and lets me steer things towards better again and again and again. And it lets me learn.

One reason to switch models mid-session that I didn't see explored in this post: cost. As we get ever-more-powerful models, it's going to make sense to use them for some exploratory/research/design phase, and then smaller models for other bits, within a single session. It'll likely even be managed automagically by another model that is just trained to do routing.

As people (and especially companies) adopt these tools for more and more usecases, there's just too much money being wasted on using an unnecessarily large model for a given job.

It's one thing that the APIs don't return the reasoning tokens in a readable fashion. But then why do I have to send them the whole conversation every time? If they don't allow me to see parts of the conversation then they should have the burden of storing and transmitting it, and shouldn't charge more and more the longer the conversation gets (unless opaque token caching kicks in) like some LLM Shlemil the painter.

This is exactly for this: https://github.com/XTSoftwareLabs/neatcontext-plugins

Turn conversations to structured domain knowledge. Across sessions. Across AI agents. The stored domain knowledge is human readable and can easily shared with others.

Why am I paying for tokens I am not allowed to see? Why should we believe the tokens even were used? There is no way to verify them.

im building my own harness and inference tool chain for much of these reasons. theres so much to do that makes a big difference for users. hoping to get things into shape for early alpha as a saas in the next two months.

heres the easiest biggy: compactions should include all user turns albeit with pastes and attached files not inlined. omg does it make a huge difference.

Currently these models have a limited context window, so the value of a session is not that high, at any rate at some point the model will have forgotten things. If in the future the model will truly learn / "change" as a result of the interaction, in any case this will be non portable to another model. So I think this is kinda moot

This appears to be an opportunity for middleware companies.

Put another ai agent in front of your prompt to decompose it into micro-prompts following an interrogative chain of inquiry and synthesize an answer for the original query. Basically local per-prompt distillation for the purposes of preserving an audit trail.

Sounds horrible but it would work.

This is exactly what I'm working on. Very rough yet but it's progressing nicely.

https://github.com/pantoniou/fyai

The idea is that your session data are what's important, and what you need to keep yourself, using a model similar to git.

For some reason, my early morning waking brain, expected the blog title to be a segue into existentialism

Is this a analogous to idempotent behaviour? Of reproducible compilation? If the inputs and outputs aren't sufficient to reproduce one from the other, what is a minimally satisfying description of the middle third, the missing part? The "other" inputs?

The AI companies need some way to lock you in. You weren't hoping for a free market were you? Hah. Free markets are for peasants. Investors are always looking for ways to make the market non-free.

Does there need to be an open standard or file format for context? Open models should align on the same format for portability? Could be sqlite based so it can be queried by other programs?

Trying OpenRouter gave me a strong impression that LLMs are already a commodity and individual providers are not that important. Providers don't seem to be too happy about it.

Honestly, i don't need to know what happened internally at Google when I searched. I don't need to know what filters an email passed before it landed in my inbox. I don't need to know when my Notes app synched with what server version.

And I don't need any of that in an AI conversation.

If it's a truly long and important conversation, stay with the provider for that one only and start all new ones elsewhere. But I've never needed that.

I get the point of the article and it resonates with my general tendency to support more openness, but older and more cynical me wonders how much of this is a battle worth fighting or a self-serving argument from an harness developer.

I mean, the article says each of the points it is complaining about has “_a basic justification that's trivial for a provider to come up with, along with good arguments for why this is good for the user_“, which to me sounds like implying these are just bs to have people accept them, but is that actually the case?

For instance, this [0] was mentioned here a while ago, and based on that it seems pretty clear why one would choose not to provide the _thinking_ anymore…

[0] https://role-confusion.github.io/

All of this can be addressed by using locally hosted models running in bespoke harness rigs.

So continues the plundering of Tolkien's universe for "cool company names".