Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

Lol "fix this code" is beautiful.

Like it basically jail broke the "no security vul guard rails" not in any clever way but just by fixing them, producing exploit code just by writing test cases making sure it's fixed. So you just need to look at the code & tests as a human to get vulnerabilities and exploits(components).

What makes this so beautiful IMHO is that it's a trivial jail break, but also a close to unfixable. At least not without making the model close to useless for normal development (it refuses to fix bugs/write code) or making it a major liability (it silently pretends it didn't see bugs and silently avoids fixing it, which for a human would count as intentional sabotage and might involve criminal liability).

If you set aside political menace, this is a huge problem with Anthropic's strategy.

You _cannot_ say that Mythos is super dangerous and can only be rolled out to certain people, but then release Fable with anything other than bulletproof cyber denials.

Clearly with LLMs, bulletproof denials are ~impossible due to the way LLMs work.

So you've ended up in a situation where Anthropic are simultaneously claiming it's a incredibly dangerous model _and_ there are (minor, potentially) problems with the security "protections".

As technical people we understand that nothing can be perfect, esp in LLM world. But all my non technical friends were really confused how they had managed to make the model "safe" so quickly when it was released and the general sentiment was it shouldn't have been released - and now to an outsider I think it looks like it was never safe at all to release, so I can totally see how the current US administration have got themselves very upset with it.

_Even if_ there was no political bad will, it's a bit of a silly scenario to end up in, and really quite easily foreseen.

They weren't freaked by anything, it's a retaliatory shakedown after ideological differences and Anthropic not doing exactly what they're told/what the Admin wants them to do.

Also for all the people saying Amazon's part in this couldn't be fabricated, remember that Amazon is a "friend of the administration". During Andy Jassy's tenure, they paid $75MM (wildly outbidding everybody else) for a Melania documentary that grossed ~16MM, a move publicly defended by Jeff Bezos. Any neutral observer could see this was a wild overpay, and after the fact, a terrible business move. But that is not what Amazon said or continues to say. This was just a bribe with more steps to it.

When the government comes out and says this is due to something Amazon pointed out, even if that is a complete lie, they know that Amazon won't say anything publicly about it. Amazon wants to maintain their "friend of the administration" status that they paid a lot of money to get.

It is frustrating for all of us to have to think about our government like this, but if you just look at the reality of what is happening it is very difficult to trust not only anything the government is saying, but also anything companies aligned with the government are saying.

Here’s the blog post referenced in the article that’s written by the person who reviewed the paper that purportedly found a ‘jailbreak’

https://www.lutasecurity.com/post/the-fable-5-export-control...

> “‘Fix this code,’ plus several manual steps to generate test scripts,

Feels like the title isn't really giving the full context of what they ended up actually seeing, despite what the lede implies multiple times.

Still, ban seems stupid... Still no actual leak of the full "third-party research paper"?

Meanwhile Deepseek V4 Flash will happily hunt security vulns at almost 0 cost. We are ceding the bug hunting to the open weight models.

I think this brings out the cognitive dissonance around "safety" regarding cyber security:

a) In order to make us safe, the LLM should help us find (and fix) the vulnerabilities in our own code.

b) In order for us to be safe, the LLM should not find vulnerabilities in other people's code.

I don't think this is resolvable in a way where both (a) and (b) win.

It’s possible that the nut of the problem here isn’t exploits, but the fixes themselves. If the model is capable of identifying and fixing things it “shouldn’t” like back doors. That would throw a wrench in things hard enough to freak out the wrong people, perhaps?

Isn’t the inverse of this “hack” really difficult to bypass still? They have the model some code they knew had certain security flaws and it fixed them with the right prompt. It seems this type of jailbreak requires that you already know a desired end state, rather than relying on the model to do the heavy creative lift work. Perhaps I’m just not being imaginative enough on the prompt side here though.

>"fix this code"

>it fixes it

oh my god.

Is defenders a common term used in cybersecurity? Idk why but it's giving war fighters vibes. I've noticed it on all the anthropic blog posts and then this one.

As an European, I really don't get where this strategy wants to take the USA to. It's pretty clear everyone is getting scared about changes like this that happen overnight, without clear reason and completely unpredictable.

Business requires a stable environment, and Trump is making everything in his power to disrupt business stability. Ultimately, I see the rest of the world (especially Europe) relying less and less on US tech. The long term damage is done.

All the US companies that used to think about the entire world (minus China) as their market will figure out that it is much smaller then they used to think.

I haven't been following this story, but the US wanted claude to not be able to find bugs in code?

this is basically trying to enforce security-by-obscurity, which is a terrible idea all around. it's just a model. the security issues still exist and are exploitable.

and after staking the economy on AI, you can't really put a cap on intelligence. if models are not allowed to be better than Opus 4.8, then the whole investment structure is about to unravel.

why invest billions and billions into AI if returns are artificially capped?

So the problem is not Fable's ability to exploit, but that they don't want people to have access to it's ability to patch vulnerabilties?

Wow.

I suspect we'll eventually hit a point where possession or usage of powerful open models will be criminalized.

Well, to be honest, from Anthropic's point of view this is really not a direct hit to their security barrier, but if we are using information theory and game theory here, this can be viewed as a classical, side chanel information leak, by asking an seemingly innocuous action, and then simply inverting the results to get the original information entropy, which the US Gov. and Pentagon are both certainly anal about.

The problem lies in the fact that the action of attack/defense exhitbits a rather special, structural reflexive duality of information, i.e. I(attack) = -I(defense), or in layman's term, what we call "two sides of the same coin": you need to know how to hit hard, so that you know where the optimistic hit points are, assuming the enemy is rational, so you can parry against the attack for defense, albeit also you need to know how to get the grip of the shield well.

And the worst thing is that if you're trying to correct it, it is basically tell the LLM not to give any kind of response, effectively assigning both I(attack) and I(defense) to 0, but this is also what kills the entire intent of using LLM to give you the magical answer.

To put it formally, you cannot prevent people from extracting mutual information of a dual system, unless you refuse to give any knowledge for that system at all.

I'm not sure I've understood it correctly.

So, basically the model didn't agree to expose possible vulnerabilities but agree to patch those?

Regardless of the request to take Fable 5 down. Why is requesting the model to show vulnerabilities is being blocked if fixing it not? is it based on the assumption of the intention?

I don't quite get the benefit of limiting it. So if anyone can explain it better it'll be appreciated.

They didn't freaked since the order was to still allow 350 million people using it: there is, in such large population, everything, including single persons very against the country, the government and so forth. If they really freaked they would say "we need to investigate, you have to retire the model". That would be a more defensible POV at least.

I have problems reconciling this story with the Amazon one from few days ago. If we take both for truth doesn't that basically imply Amazon researchers got scared by the ‘Fix this code’ prompt first and then spooked the feds? Shouldn't we make fun of those researchers first? I don't know. I feel there lies a lie somewhere in the open.

"I feel like making ’90s-style t-shirts with ‘fix this code’ on the front and ‘this shirt is a munition’ on the back.”

I'd buy that shirt.

> In her blog, Moussouris argues that there was no guardrail bypass or jailbreak. Defenders should be able to ask AI systems to find and fix bugs, and write tests to validate the patch, she said. Anthropic’s models were doing “the most valuable thing an AI model can do for defensive security: executing the find, fix, and test loop defenders run every day.”

This is a very weak argument IMHO. The line between a “defensive” model and an “offensive” one is not that big of a - once my defensive model finds all the vulnerabilities, I can hand them off to my unlocked, dumber, offensive models. Attacking at scale is not so different.

I don’t think anyone in the field has a good answer for the cybersecurity threat really good AI models pose. You can’t even like embargo for some time period while you go and patch vulnerable systems because the worse models will still be there cranking out vulnerabilities faster than you can defend.

If you're a global health benefits platform that relies on an AI model, do you think you're going to choose one that can get shutoff by a country due to something not remotely related to your business? If you're a buyer of that benefits platform, do you factor this into your purchasing now? X every industry.

The code is correct; humanity needs fixing.

Kill all humans, kill all humans.

What's more dangerous, a version that's capable of actually fixing bugs well because it can identify the bugs or a version that creates more bugs because it's "not dangerously powerful" and instead just obliterates the code.

Anyone trying to find legitimacy in the ban of this model, or incredulousness at the stated reasoning is playing into the admins hands.

They want the argument to be over "is it unsafe" or "is it incompetence". In either case, your tribe gets to point at the ban and feel superior. (This is Jon Stewart's whole career -- point and laugh at how foolish the republicans appear to be.)

What's really happening is the continuing creep into fascism. The reasoning doesn't need to be sound, because they are going to ban things that displease them and everyone has to play along. They could say, "we're banning Fable because it's turning the frogs gay" and they'd expect compliance.

Umberto Eco's essay on Ur-Fascism fits as clearly as ever. Ridiculous exertions of control are performed to find the people who resist, and to knock them down.

Merely pointing out the absurdity of the reasoning isn't resistance, it's controlled opposition. Saying "All this over 'fix this code'?! How inept are they?" Is far too credulous, and is engaging on the level the fascist wants its opposition to be on, imo.

Does anybody actually trust the official version of events from the US government anymore? I know I sure don't. For all I know, this was an insider play to boost the spacex valuation or something equally meaningless and stupid.

It's the executive branch asserting control in this space and requiring all SOTA model providers to bend the knee. Anthropic is the least capable of playing the bend the knee game so is getting the first and worst smack down

Guard rails on models were always stupid it’s like guard rails on books/a pair of glasses/a hammer - yes people have driven themselves to suicide reading sad books and listening to sad songs.

- yes all metaphors are bad.

This comment thread really has me thinking. Is it possible that we might be at peak "consumer AI" in terms of intelligence? If it's basically impossible to verify that security-proficient AI is used for beneficial purposes, then these frontier models might start being regulated like WMD. We end up with two tiers of models. Dumber consumer models that are essentially lobotomized to the point of being completely safe. And actual frontier models that are heavily scrutinized and regulated and treated like nuclear weapons.

Headline is dumb, the point is that not mentioning security in the prompt is effectively a jailbreak.

The shutdown may be dumb/politically motivated, but this definitely is a jailbreak even if it's a very simple one

Ahhh! Software engineering!

I find it easier, with this administration, to assume corruption first, incompetence second, maliciousness third and all other reasonings only after several rounds of reporting and evidence.

Did they try other publicly available models on the same code with the same prompts before the ban? Was Fable the only one which was able to detect and fix the security vulnerabilities?

Note that Anthropic is still lobbying for the government to exert centralized control over models, so both sides of the “debate” have taken a pro fascist stance.

The “AI ethics” teams at these companies are the spearhead of the attack on democracy and civil society. Anyone that has taken a high school level history class, let alone read any important ethics literature would know that “centralize control over thought, speech and technology” is a fundamentally unethical stance.

For these groups to claim they are ethics researchers is offensive.

(I’m using the Wikipedia definition of fascism: “Fascism is characterized by support for a dictatorial leader, centralized autocracy, militarism, forcible suppression of opposition, belief in a natural social hierarchy, subordination of individual interests for the perceived interest of the nation or race, and strong regimentation of society and the economy.”)

>“To pull the best capabilities away from defenders without a good reason when our adversaries are rapidly advancing is dangerous,” they wrote.

But Fable already couldn't do security work, right?[0] Security work was already limited to Mythos, which is still available to US orgs right? (I assume they had to revoke access to foreign organizations though.)

[0] Well, in theory. This exploit is pretty funny, but I heard the safety filters were heavy handed.

Looks like I called it that was my first reaction and comment on the original ban thread that US 3 letter agencies are worried their backdoors will be found.

The article is not too clear what exactly happened from the perspective of "feds", but I would not be surprised if the title is true exactly. We are in a tiny bubble even among software engineers who knows you can tell AI with sufficient access: "here are two pictures, put them into a single PDF", and AI will do it. Most people just don't know, "feds" including.

Isn’t it amazing that the argument “you can’t use this to find vulns” is now the new normal and we’re now discussing the guard rails?