▲ 696 points
back
307 comments
>We take a trace produced by a frontier model, replay it into a weaker sibling, jailbreak the weaker model, ...
Ha! I've been wondering if replaying across models would work, ever since https://blog.cryptographyengineering.com/2026/05/29/fooling-...
I'm honestly rather curious if this was intentionally allowed, it's the sort of validation that's easy to miss (particularly if you're wading into the vibe waters). Seems like something that'd be absolutely riddled with possibilities for shenanigans.
Author of the blog post here: just for the record, I did try replaying against other models. I replayed from GPT 5.5 to 5.5-mini. It "worked" (replay not rejected) but mini didn't hand me the plaintext of the reasoning. I'm not a jailbreaking expert, so I didn't really "push" on it very hard. I also didn't pursue Opus/Haiku for reasons I don't really recall. These folks took it all the way. I'm bummed it didn't work for me, but having these folks prove the channel out is a very good consolation prize.
PS Here's a conversation I had with GPT 5.6 about the paper differences. https://chatgpt.com/share/6a7b64b4-ec0c-83ea-a9d2-ab1f1a1dfe...
If you didn’t allow it, you wouldn’t be able to change models in the same conversation, as key parts of the context would be lost.
Wouldn’t surprise me if the providers just remove that ability and lock the model once the conversation starts.
I believe its a result of chasing the speed and happy path. The intent was to make the data available for reuse on the backend side for the same user session, it kind of makes sense. Making it available across the models is most likely by design.
The flaw is that the data is not strictly tied to user session, making the session data hijacking a lot easier.
1. Its a security issue.
2. Publicly available sessions make it much worse
I'd lean yes; switching between models, even mid conversation, seems to be encouraged by harness vendors. No reasoning makes this much harder.
You can switch models mid conversation so it makes sense that it is allowed. Otherwise you would lose all the reasoning traces.
If I'm reading this right, they literally just ask a LLM to tell them what the traces say, with the key being that the traces are portable across LLM models, so they can switch to a smaller one that's easier to jailbreak.
Correct, yes. It’s delightedly simple. And they validate by asserting the reasoning token length matches
nothing scientific here, they basically just figured out some real issues caused by bad engineering practice.
> For some AIME problems Opus 4.8 sometimes states the answer before deriving it. We find that the API summary does not always preserve this distinction, and can instead make the reasoning appear like a clean derivation.
No surprise here but good to have more confirmation that they just put all that in the training data. And based on the "reasoning", the models have some form of index of those problems (or they are HEAVILY trained on them).
All LLM benchmarks have an expiration date once they're released to the public. They get spread so far and wide across the internet and GitHub that you have to assume they're in the training data for every LLM with a cutoff date after their release.
The real question is whether or not the training was directed to optimize for those benchmarks.
The technique doesn't guarantee that the reasoning is returned verbatim because it relies on the weaker model transcribing it accurately. Looking at the charts, there are a lot of dots that aren't in the 1:1 line that suggests that the output is exactly what was provided.
Not surprised. On many benchmarks (i.e tau), we have seen the same thing. Probably lots of training on every publicly available benchmark
You cannot steal what is not owned.
At least in the EU there is no copyright for LLM outputs, so I guess all they might do is violate the terms of service.
"Stealing" is a strong word to use for looking at the words produced by models built from the collective commons of the world.
And, honestly, being able to see how LLMs make decisions is critical to trust and security. I consider it a valuable feature, somewhat akin to seeing the source of software I use.
Apparently you can do the same by simply running it without reasoning, while giving it a thinking tool...
>guys you do know you can just disable thinking, and instead give it a "deep_think" tool, and it will call it with internal CoT reasoning format right?
>gl fixing that
"Recovery" would be a more apt (although less catchy name). The stealing is on the provider side for not giving you access to tokens you already paid for.
"Stealing" something you already paid for (tokens), but that you can't have access to(!). And trained on the sum of human knowledge.
Training on other model outputs ought to be business as usual, stop using morally charged terms made up by future monopolists: https://thomasdullien.github.io/posts/2026-06-15-rl-economic...
Is this how the eastern labs "distill" SOTA models?
If you can play it right, you don't even need to send suspicious prompts to the frontier models. Just use them for regular tasks, extract the encrypted COT blocks and replay it to a cheaper model to get the plain text COT.
But the real question is: Is it okay to steal from a thief's hoard?
Super cool that this works. I'm surprised these companies re-use the same encryption key across models!
I wonder if you can use these for attacks, like this previous paper showing that if you know how a model reasons, you can "fake its thinking" to control it? https://news.ycombinator.com/item?id=48631888
I did this with Codex's recent encryption of compaction.
Interestingly, I didn't have to drop to a dumber model, just a 2 sentence <developer> prompt auto-injected before and after compaction made all their models output the encrypted compaction data in plaintext.
The result was... interesting. There's nothing unique in there and I still don't understand why they decided to encrypt it in the first place.
Cool find, but can't help myself thinking that registering a domain name and submitting a paper on this to Arxiv is a bit... much. The content here could fit in a tweet or a short blog post as well. Not sure about the scientific novelty here as we're basically poking around the very top layers of someone else's software stack?
Really cool work, you get the actual traces. Looks like the vendors can all reliably fix this one though.
A harder to defend against approach here where they work backwards from the results and ask the model to generate a plausible trace: How to Steal Reasoning Without Reasoning Traces https://arxiv.org/pdf/2603.07267
This proves that OpenAI models reason in grug speak to save tokens! I wonder if open models are going to start doing that too to save on reasoning tokens.
I must say right of the bat this is the best research paper/working paper in regards to its styling. Beautiful
The problem with this kind of excellent work is that the response to it is always to say "Fuck the user".
For example, when there was a paper that came out showing that having model logprobs makes distillation an order of magnitude easier, the closed LLM providers instantly yanked out support for getting the full logprobs at every time step. You get at most top 10 candidates now and I'm sure even that's on the chopping block.
People will use this to argue that a model which has exceeded Opus 4.8 (Kimi K3) somehow got most of its performance through distillation of Opus 4.8.
I still don't buy that distillation was worth more than 3 months of "catch up" time for the chinese labs. Most people who use the word "distillation" to much are revealing their sinophobia.
I used to do a very coarse version of this stealing. I ask a question from ChatGPT pro, once it is done, I ask claude chrome add-in to go through all those thinking from the side bar, extract everything along with all the sources used. Then try to reverse engineer the solution it came up with.
OpenAI and Anthropic will probably now resort to save this server side, instead of relying on encription to be able to keep state on the client.
Last year's reports questioning whether reasoning blocks were actually reflected in the final response were why I stopped using reasoning models altogether. I switched to a separate pipeline and have used that ever since. It's good to see that the concern didn't remain just a suspicion.
I understand it’s cool to have an artistic website, but it’s very noisy and non-accessible.
But very interesting result.
A natural next step is to use the reasoning traces to jailbreak the stronger models (https://arxiv.org/pdf/2603.12277)
Wouldn’t the fix be to encrypt before the api call hits the LLM? You would encrypt/decrypt at a separate layer than the LLM. I am sure i am missing something but would love to be educated.
Can someone tell us how they were able to decrypt the encrypted payload? The article says they inserted the cyphertext into a session with a different model. Ok, but how does that allow you to decrypt it?
The fact that frontier LLM providers pirated all the data that they used for training, then to go on to encrypt all of the reasoning traces that they use to come up with the conclusions it's really disingenuous, and then have the balls to say distillation is some kind of bad behavior. they are the kings of distillation.
The hiding of this data only brings distrust to their frontier models. I think most people want to understand how something comes to a conclusion they don't want have that part left out on purpose...
it's this kind of behavior that forces people move to to open source models in the end, it's the lack of trust. the frontier model providers treat the end user/customer as a threat or adversary. Fable 5 is notorious for this. a lot of the serious questions you ask the model they won't even respond to you because of the woke guardrails. it wasn't only a couple weeks ago that huggingface had to use glm 5.2 to get the right answers about their security incident because Fable 5 didn't want to answer it.
I really wonder how much of safeguarding with SOTA models is actually just "Don't do that" in a prompt?
I'm building agent to control machines at work and have to rely on small models, which are especially bad at remembering rules like this, so it is very apparent to me that soft rules like this are pretty much useless. Might be different for these huge models, but still it seems very shaky to me.
Is this basically a paper on how to distill, in exactly the fashion openai/anthropic don't want/say is copyright theft?
super interesting. So pre-filling Kimi3 reasoning with Opus's reasoning results in thoughts that closely match Opus's. This seems like strong evidence Kimi3 was trained on decrypted Opus chain-of-thought. Meaning the Kimi team likely also broke CoT encryption. Though not exactly a big surprise.
> Prefilling Kimi-K3's reasoning with the first 1% of tokens of Opus 4.8's reasoning moves its visible answer toward Opus's wording, even though the answer itself is never prefilled
is this supportive evidence for the distillation accusations in the news?
These logs containing opaque blobs could accidentally contain secrets, the researchers decoded many of reasoning blocks from public repositories and reported finding PII and credentials.
I was experimenting a bit how I could block these using an ingress path. GitHub /softcane/hamza
A bit shocking. One would think that the encrypted reasoning traces -really, encrypted state cookies- would be bound to the session or user, not just the AI provider. That obviously is the fix.
> The providers did not acknowledge “any security implications arising from side channels or replay attacks.” All model providers acknowledged the receipt of our report and subsequently we were unable to launch the same attacks.
I went straight to the ‘Responsible Disclosure’ section. Not surprising, but still disappointing.
Wow, almost certainly the approach that alternative labs use to distill Claude. I always wondered how far they could get with just the answer missing the reasoning. They probably actually also had the reasoning.
I expect future LLM will refuse to share the reason. "Hey, how did you come up with this idea?" "you have to pay enterprise API to learn this"
Fascinating approach; however, a nightmare to scroll on mobile.
Why they use different models to decode the reasoning content? Can the the model decode it?
curious seeing how anthropic is agressively fighting this stuff how did you get to experiment on this? did you just try and shown them results or did you need approval first? I am interested mostly because I research on distillation
It’s scary the number of security tokens that end up being ingested by these models.
So to make the APIs stateless (the "ideal" where they don't use server side sessions/etc) we ended up with this. I'm sorry but this is kind of hilarious. Given the salaries paid to the workers at these companies and the hype of the models, I can't believe they all fell to the same flaw.
TIL it actually sends the traces. I had assumed this is entirely server side
I really like this website