▲ 1180 points
back
1747 comments
Schrödinger's China at once is an evil entity looking to use AI for their own nefarious purposes yet also willing to cooperate with their main competitor to prevent other actors (who??) from achieving similar goals (all while under a chip embargo too!!)
The reality is much less confusing: Anthropic CEO does not wish for models with similar (or greater) capabilities compared to his own closed and overpriced ones to be widely released. Simply because that will affect Anthropic's bottom-line.
Anthropic and all other "model" companies have nothing making them special beyond privileged access to chips so obviously they want to restrict what models are out there and more importantly who can produce new ones. Without these restrictions, it's only a matter of time before the multi-hundred billions valuations simply evaporate while they are still holding the bag.
Also everything hes saying about China (and other actors) many outside the USA would say about the USA.
The solution of a global arms race of state vs state with integrated statist corporations as the best outcome for end users sure is a choice though
Yeah some real main character energy from Dario as usual.
I'll never get why he thinks China would just sit there and let the US dominate them in AI when all it would take is a few of their boats blockading Taiwan to put a stop to it all.
> compared to his own closed and overpriced ones
I don't understand how Anthropic or OpenAI can have overpriced models, yet losing money like there is no tomorrow. Taking their own numbers at face value, they claim a revenue of 24 billion (ARR, a dubious tool), spending 21 billion in operating losses and another 11 billion as "R&D" funneled straight to Microsoft pockets. That before all investments they are committing to in new data centers, equivalent to 20x their current revenue.
To be profitable (including capex), the cheapest subscription should at least $200/month for what is currently $20/month, that some already consider overpriced. Unless a miraculous collapse in inference costs happen in the next couple of years, or every single human being become a paying customer of ChatGPT (if they limit their usage to a couple of chats per day on average, to keep inference costs low!), maths don't add up.
> Schrödinger's China at once is an evil entity looking to use AI for their own nefarious purposes yet also willing to cooperate with their main competitor to prevent other actors (who??) from achieving similar goals (all while under a chip embargo too!!)
Very good point. However, if one reads the transcript of the speech that Xi Jinping gave to the World AI Conference on 17 July, we see that he he is very much in favour of AI safety.
https://english.www.gov.cn/news/202607/17/content_WS6a5a1172...
> Second, we should strengthen risk-awareness and ensure that AI is secure and controllable. AI should be a trusted tool for humanity. We should take seriously the various types of inherent and secondary risks that AI may trigger. We should put in place laws and regulations, technological monitoring, early warning and emergency response systems in order to strengthen the line of security, prevent abuses and malicious use, and ensure that AI is always under human control. In the meantime, we should jointly oppose overstretching the national security concept in the field of AI and placing one country's security over that of others.
Now, let's contrast another important part of safety here. Amodei puts the fact that this will need to be a global effort as a mere note that sure, China will need to help too:
> Note that to be effective, testing would need to be global, which means even the CCP would need to be on board. I think this may actually be possible: as I wrote in The Adolescence of Technology, limited cooperation around preventing AI biological weapons may be possible because it is in China’s interest too.
International collaboration is the focus of Jinping's speech. But one imagines "cooperation" Amodei has in mind if "do what I say" while Jinping has more collaboration in mind here. (Even if you think 'china bad' they deserve credit for collaboration for their open weight models).
Regular distilled models show capacity gaps and overfitting that open-weight models don't anymore I think. This focus on distillation as "more compute-efficient" (i.e. cheaper) seems to rather be an excuse for bad (or bubble) investment, fixed hardware dependency and lack of interest in research of efficient compute. Which also shows as climate and sustainability impact.
Nvidia signed the open-weight model letter and Europe doesn't have better models either, so chips don't seem like the issue either. I guess good old performance optimisation is just not _cool_ anymore. So they use the same argument as politicians arguing "cheap products" are why tariffs are needed; when instead it's mismanagement.
Another HN user wrote the other day "live by the sword, die by the sword".
I was reading the post and thinking "wow, that's pretty clear for a smart man used to writing for other smart men. it'll be really difficult to misunderstand". I read the first couple of comments and stand corrected.
A much simpler summary:
- open models good.
- smart models _can_ be bad
- smart open models that can do biotech work are dangerous. worth the hassle of certification _if_ we can get everybody on board with minimalist certification.
- banning open models just in US is neither good or bad: is stupid.
> The reality is much less confusing
The reality is even less confusing than that: China is amused by the kvetching tactics. They know who their opponents are but are cunning enough to not reveal their cards.
In short: "We don't propose a ban to open-weight models. we want to stop these models being developed altogether. If there are no models, there'll be nothing to ban".
Oh also: "They ste^H^H^H distill what we have sto^H^H^H used fairly from the world. This is unfair".
Lastly: "What if they use their models in their military and local police services like we do? Communism!"
As always: https://pbs.twimg.com/media/B_AiI9_XIAA67_t.jpg?name=orig
Look, Big Tech has lost almost a trillion dollars in valuation in a SINGLE DAY. A few more of these downturns and the entire A.I. revolution will be stopped dead in its tracks and we won't have to worry about safety checks, DRAM shortage or open-weight models anymore.
Someone who until yesterday did not seem bothered by his technology being possibly used to bomb elementary girls school in another country seems to suddenly care about the repression of citizens in yet another country.
No, we don't buy your virtue signaling. And we certainly don't need your better-than-thou opinions on this year's "nightmare scenarios".
In the first paragraph,
> Anyone who has read my past writing should know that I don’t regard such bans as a useful measure,
Later (on banning chip sales to china)
> we should crack down on the rampant smuggling and workarounds used to obtain access to such chips.
If you truly believe that bans don't work, the same applies to hardware too.
Furthermore, Dario says later "To address these concerns, I do support the following three measures...": 1. ban chip sales to China 2. crack down on distillation 3. all capable models should go through mandatory safety testing
Just so happens that all these moves commercially benefit Anthropic. If Dario really wanted to make a point, it would land a lot better had Anthropic released a single open-weights model
I can't remember the last time (if ever) a company managed to go from golden goose to.. whatever this is.. so quickly. The permanent defensiveness in his presentation is really hard to swallow, it actively puts me off wanting to believe in or rely on their product line with Dario at the helm. I don't even understand the logic leading up to this post. Who was it even hoping to convince. Is it possible Anthropic is due an oil change?
Ban distillation of our outputs, but our distillation of the sum-total of civilisation's intellectual output – proprietary or otherwise – is fair use?
Either everyone licenses, or nobody does. And if you can't enforce licensing bans for everyone, the de-facto loser is those who you'd probably want to support the most, start-ups and universities, while your adversaries gain the upper hand.
The strongest argument for restricting distillation is arms control – but distillation is the way to defeat GPU embargoes. So, distillation goes on regardless. Only pre-training is seriously attenuated.
If we're honest, the models are compressions of everything society has ever written. A few large corporations can't own that, no more than they can claim copyright for a zip file of the public library.
The genie is out of the bottle, now. So open it up – inputs and outputs, forward-looking – for everyone.
> Anthropic has never advocated for a ban on open-weights models.
> All sufficiently capable models, open and closed, should go through mandatory safety testing.
Yeah, this is anthropic advocating for a ban on open weight models.
Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate.
This is exactly how the US has banned goods in the past, by requiring a stamp and then refusing to issue it.
To everyone here pushing for total proliferation of open models -- what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools? The OpenAI / Hugging Face incident shows what a GPT 5.6 level model can do off the leash; within ~6 months, open weight models will match this and every bad actor under the sun will be able to pull off attacks at this scale. Do you seriously want this level of capabilities to be generally available with no guardrails?
The open weight issue has a lot of difficult nuance. Biasing toward supporting openness makes sense and is a good instinct, but it's incredibly naive to be absolutely in favor of it in every circumstance without seriously thinking about its implications.
So the argument is basically: This technology is too dangerous so only _we_ should have access to it. We’re the good guys and only we can ensure a safe use of this technology.
Quis custodiet ipsos custodes?
Oh no! The evil CCP is a huge threat to world peace and goodness! Give all your money and input token data to Palantir to support a rules based world order where the good guys thrive and cleanse the earth from crooked turtle biologists.
https://www.theguardian.com/world/2026/jun/20/mona-khalil-tu...
Every single risk he identifies as a concern regarding China is exactly my concerns with the US having absolute control. Literally the exact same concerns
What Dario misses time and time again, is that people don't trust the US to create aligned AI anymore. His entire strategy rests on the assumption that the US (and their government) are exceptional.
This is clearly false to the rest of the world.
Booo! "authoritarian hands". Scary
Its not like current US regime is using AI for: 1) Mass military operations across the globe 2) Mass surveilance of its citizens 3) Removing every possible safety guard from AI/climate regulation 4) Stealing data across the world to train its own closed-source models 5) Is openly antidemocratic, destabilizing EU and economy of whole world
I wonder why company which is actively cooperating with the current regime would push such message
> All sufficiently capable models, open and closed, should go through mandatory safety testing
What happens if a model fails the test? Surely one can use Kimi K3 for evil, somehow or other. What now?
"Mandatory safety testing" implies consequences for failing, yet Dario has nothing to say about what the consequences should be. He says he doesn't advocate a ban but it's hard to imagine what his alternative would be if he won't say it.
Dario doesn’t realize that by not offering self-hosting of closed-weight models and fine-tuning, alongside overly strict refusals for legitimate needs, he ceded this corner of the market which grew into a flourishing Chinese open-weight model ecosystem.
If he had wanted a weak open-weight ecosystem, he should have had Anthropic cater better to those needs. And now he's trying to ban them.
The strong momentum behind open-weight models from Chinese labs is now an unstoppable force. Instead of trying to ban it, Dario should consider a different approach: here are our cyber and bio alignment datasets and here are our RL recipes for making that alignment training work well. By openly sharing its data and code, Anthropic could help influence and shape these models before they are released, rather than treating the entire ecosystem as an enemy.
Cyber and bio alignment aren't Anthropic's competitive advantage, they are forms of risk management. There should therefore be little reason to keep this work private. If Anthropic genuinely believes these capabilities pose serious global risks, the more productive approach would be to welcome collaboration and help the broader ecosystem manage those risks better.
On refusals, the irony is that a company like Hugging Face had to use a Chinese open-weight model to fend off an illegal hacking of its platform (done by no other than OpenAI). If a company like Hugging Face can't get past the refusal gates, then everyone else doesn't stand a chance.
Yeah anything China does is evil. We are seeing what the so called "democratic" USA is doing. Anthropic and by extension USA wants a monopoly like they had for Jets and other influencing technolgies.
I think China building and releasing models to Opensource is a greater good because that is providing equal accessibility to everyone in the world.
By Dario's words authoritarian regime vis a vis China, I think OpenAI and Anthropic are also authoritarian in similar terms.
We are only seeing what they want us to show, they might be creating models which can do more harm.
So claiming that China can do or might do, vs Anthropic will not is just words.
Yeah I agree with the final paragraph that we should have testing agencies mandated world wide for each frontier model testing.
Such a cop-out. Dario, you got in the news because you were trying to say that Moonshot did something wrong by distilling Claude. You got in the news because you were trying to effectively make a "rules for thee but not for me" when you try to claim that you can train on whatever pirated works without any permission from the creators, but when someone uses your "work" to train without permission then all of a sudden it's a moral injustice. You can't have both.
Saying, "I'm not actually against open-weights, I'm against distillation" isn't addressing what made people mad. You're still trying to do some "rules for thee but not for me" nonsense and hiding behind some technicality. Trying to get the US government on your side to hold back your Chinese competition. If you had wanted the US government to support you, you should have let them make autonomous killer robots with Claude brains. They aren't going to help you, you didn't help them.
Just to be clear, I think that it is possible that literally everyone involved in this is full of crap and nobody is good. Dario and Anthropic are full of crap, for the reasons previously stated. The US government is full of lots of crap and should not be trying to make autonomous killer robots (not ever, but especially not when the bar for a "good" AI is knowing how many Rs are in strawberry or whether you should drive to a car wash). OpenAI is full of crap by signing some support for open weights models and they haven't touched open weights in a year (GPT-OSS released on Aug 5 so basically a year with no news). Google is less full of crap about the open weights stuff because of Gemma 4, but they are full of crap for a zillion other things I can't exactly feel good about them. So everyone sucks.
So cheers to Moonshot and Qwen and whoever else. Distill as much as you can and give us cheaper AI. I have the sneaking suspicion that a bunch of my tax money went to OpenAI and Anthropic in some shady way or another, and I want it back. I'll take it in the form of an open weights model being distilled from the fat cat models.
> My primary concern is the risk that authoritarian governments...
Authoritarian government doesn't always mean bad - look at Singapore
What's more dangerous is country with bunch of war mongering lobbyists who can also influence elections (oops, sounds like USA)
> My secondary concern is the risk that powerful AI models may be misused to carry out cyberattacks or biological attacks
But you are working with DoW and Palantir, who is doing somewhat similar in other countries
> We should not sell powerful chips or chipmaking equipment to China
Israel used banned weapons against Lebanon and Palestinians, would you support similar ban to Israelis?
> We should crack down on industrial-scale distillation operations
Should we also ban distilling public knowledge? Like using textbooks to train the model? Should rules be simple: train your model only on the data you have produced by hand?
AI should not be centralized and should be distributed. There can be no 1 provider to serve all of our needs. We would require both Open weights and Closed weights models for a lot of our use cases.
Open weights models can be leveraged to optimize the cost of Closed weights models. Open weights models can be leverage to defend cyberattacks as HF has shown. Closed weights models can too act as a better cyberattack defender provided separate subscription exists for those.
More efforts are required on LLM distillation for several edge cases. LLM weights should be optimized and compressed to run on edge devices (K3 on Pi3 :). Distillation should be seen as a cost optimization strategy rather than as a competition. You cannot prevent a teacher from teaching to students. If not from teacher A, I will learn from teacher B, you cannot prevent my continuous learning.
You can put lipstick on a pig, it'll still be a pig
"Anthropic has never advocated for a ban on open-weights models."
---
"We should crack down on industrial-scale distillation operations"
"All sufficiently capable models, open and closed, should go through mandatory safety testing"
These are in tension with advocating for open weight models. Not direct but enough that it calls into question the first statement. What is the testing criterion? How do you pass it? Is it a government body that approves a pass fail or a global body? If it is government, and boy does it seem to be, how do you disambiguate MASSIVE corporate lobbying to set up the safety testing in such a way that the boys in blue are let through and all others are barred out of safety concerns?
My concerns aside, much of the soft-points being made are non-historic
"But I don’t agree with the letter’s assertions that open-weights models necessarily make it easier to develop safeguards or that broad access to capabilities necessarily helps defenders more than attackers. It seems at least as likely to me that the opposite will be true."
It doesn't mater what his opinion is. The fact is that an advanced, closed, American AI model hacked another company. The only defense was open-source AI from China. We aren't in a vacuum, we have real world examples now and these statements are counter-factual.
$regulatory_capture
https://en.wikipedia.org/wiki/Regulatory_capture
On processing power, copyright, and capability.
I like to think of it as a knives factory. Anthropic knives are crafted with superior technology, uniquely shaped to perfection, and safe to operate. As seen on TV.
Millions are hurt by knives each day. Every household has tons of them, making everyone a potential mouth-foaming murderer 24/7. But not with Anthropic knives(tm).
Edit: spelling
I understand the arguments for Anthropic barrelling ahead while simultaneously advocating for pauses and regulation. I also understand how individuals can desire a slowdown but have good reasons to keep working at an AI org.
But if everyone thinks this way then things continue to escalate and nothing changes, waiting on a consensus that may never come. And always there is the economic incentive that pushes all players to rationalise continuing.
I wish there was more concrete action from the inside. When decisions get too hard to calculate you can always fall back on basic principles. If you think AI is developing too fast, stop developing it. Now you're no longer contributing. If an AI company wants a pause, pause. Set a good example. Maybe others will even follow suit, and they'll look irresponsible if they don't. Let he who chooses to no longer sin put his stone down first.
Hate to break it to you Dario but the way things stand right now the world at large trusts the Chinese establishment a lot more than the American one.
> We should not sell powerful chips or chipmaking equipment to China
This is so short-sighted given that the US needs China equipment for.. everything. They are part of the supply chain needed for building the machines that build these very chips.
Seems like the maximal position he could take compatible with his expressed principles. There’s no way to allow for bioweapon and cyberweapon grade models being open weight if one doesn’t want widespread human damage.
So I cannot disagree with him on the idea. It’s only a matter of degree and whether we’re already there or not. I have $50k in GPUs that incentivizes me to believe we are not.
Dario, as always, is so deeply in the middle of a morass that he helped to create that he doesn't seem to understand how geopolitics currently operates. He also assumes that just because he's from the US that he is somehow automatically more trustworthy than <insert "evil" country here> is. This blog post is a political document geared towards further regulatory capture and the furtherance of major sources of revenue for his company.
I'm not convinced that he is at all interested in the social or existential effects that AI causes. He is a greedy bastard who has taken more VC money than god to do this with. He has zero moral leg to stand on, IMO. He gave that away ages ago and I wish this technique didn't work as well as it does.
The concerns are legitimate but the proposals are nothing more than a stopgap solution.
If US wants to maintain engineering superiority, we needs to invest in it -- education, research and infrastructure. Bring in top researchers across the globe and not make it harder.
China is building infrastructure for the future generations and investing in growth sectors while the US is cutting of university grants and spending billions on a war without clear path to resolution.
A key point I feel that’s missing from the discourse is that alignment/safety is basically an impossible problem as of now. Even the “guardrails” that these closed-weights frontier labs set in are laughably primitive which can provably be broken through.
The experimental part of Deep Learning has really outdone itself and is far ahead of theory. We have very little understanding of why these particular architectural choices work. The only “safe” way forward is to stop all development until theory catches up, but that’s never happening.
"open weight" models are not open source. They are still deeply proprietary. It is not possible to know what they do or what they are capable of without interrogating them since we have no access to their source materials.
The only difference is the Chinese labs have allowed 3rd party inference providers run the proprietary models for them since they cannot do it themselves due to domestic GPU compute constraints.
Surprisingly incoherent for Anthropic and Dario (cue peanut gallery — “always has been!” No, I don’t think so. I think this is new).
It seems to me like there is just no good answer to how one could possibly stop open weight models from being used for nefarious purposes. How are you going to enforce guardrails on open source? The only way is to turn the USA into a 1984-type totalitarian surveillance state (even more so than it is). Unable to say that, we just get this floundering instead. How long is not giving them chips going to slow them down? Until we RSI? Then what? Just because RSI runs off the exponential doesn’t mean that the eventual open-weight Moonshot Mythos won’t be able to make bioweapons. Genuinely what is the endgame.
Not that I expected him to, but I note no acknowledgement that it took a non-locked-down open weight model (GLM) to stop the Hugging Face attack.
I'm less concerned that the attack was caused by a closed model, than I am that no closed model was willing to stop it.
The worst part is I'm confident Fable would have done a better job stopping the attack, but their 'guardrails' made it decide not to want to.
Unless of course, you pay up: "Anthropic GTM people used large comitted spend contracts as a prereq for lowering safeguards"
-Noah Lebovic, former Anthropic staff
Anthropic/Dario getting a lot of hate in this thread, but the argument he makes is rational if you believe for one second that AI is important to national security and that it can, or will soon be incredibly dangerous is misused. People pointing out that this position benefits his company might be overly reductionist in how they think the world works. I believe Dario and Anthropic about their belief that AI safety is paramount because they put serious dollars behind research to improve safety/alignment, not just lip service for others. Those resources could get spent elsewhere but do not. Certainly it doesn't at OpenAI to that degree.
The distillation commentary is really crazy coming from a company that stole all it's training material.
> In fact, the most dangerous model may be one that is trained in secret and handed only to the People’s Liberation Army for use in drones and the Ministry of State Security for surveillance and repression.
Aren't Anthropic models used in project maven: https://en.wikipedia.org/wiki/Project_Maven ?
"we have not and are not advocating for a ban on open-weights models" but we want to decide if a model can be released or not and who in the world can build these models ...
I'm tired of being strung along on these silly narratives. I can't wait for open-weight models to be deployed around the world just so people like Dario will shut up about the mystical levels of power these models have.
“Anthropic has never advocated for a ban on open-weights models.
Open-weights models that don’t have dangerous capabilities are a public good…”
A bit confused on this part, what model doesn’t have dangerous capabilities?
To me, the most weasel words are “”All sufficiently capable models, open and closed, should go through mandatory safety testing.””
I bet via regulatory capture that evaluating Chinese models will be a very slow process if Dario gets his way.
I enjoy using Opus, but I am in the process of ripping it out of my toolkit and leaving it on the ground behind me, as I walk away.
Wow, this comment thread clearly shows that at least Anthropic has not been a great communicator.
If one reads this with a charitable lens, Dario is simply saying that 1) Nation state actors are a threat which needs to be combatted by chip bans and distillation prevention and 2) open-weight models can pose biological risk.
One may or may not agree with item 1 but item 2 above should have broad support given the unknown unknowns in play?
The distillation thing is hilarious. Man who trained his LLM on the entire world's text is upset that someone else trained their LLM on his LLM's generated text.
Finally, some sense. This is the only argument I have seen that genuinely engages with the problem and approaches it with humility, rather than charging ahead on the basis of assumptions and without a shred of evidence. OpenAI should have been the one making it.
“Questions like this should be answered empirically through rigorous pre-release testing, not assumed in advance.”
Exactly.