Item — HN Reader

Does Newsom believe that an AI model can do this damage autonomously or does he understand it must be wielded and overseen by humans to do so?

In that case, how much of an enabler is an AI to meet the destructive ends, when, if the humans can use AI to conduct the damage, they can surely do it without the AI as well.

The potential for destruction exists either way but is the concern that AI makes this more accessible and effective? What's the boogeyman? I don't think these models have private information regarding infrastructure and systems that could be exploited.

3 comments

“Critical harm” does not include any of the following: (A) Harms caused or materially enabled by information that a covered model or covered model derivative outputs if the information is otherwise reasonably publicly accessible by an ordinary person from sources other than a covered model or covered model derivative.

The bogeyman is not these models, it's future agentic autonomous ones, if and when they can hack major infrastructure or build nukes. The quoted text is very very clear on that.

I am not convinced the text means what you say it means.

All knowledge (publicly available and not) and all tools (AI or not) can be used by people in material ways to commit the aforementioned atrocities, but only the models producing novel knowledge would be at risk. I hope you can see how this law would stifle AI advancement. The boundary between what's acceptable and not would be drawn at generating novel, publicly unavailable information; not at information that could be used to harm - because all information can be used that way.

What if AI solves fusion and countries around the world start building fusion weapons of mass destruction? What if it solves personalized gene therapy and armed forces worldwide develop weapons that selectively kill their wartime foes? Should we not have split the atom just because that power was co-opted for evil means, or should we have not done the contraception research just because the third Reich used it for sterilization in their war crimes? This bill would work towards making AI never invent any of the novel things, simply out of fear that they will be corrupted by people as they have been in history. It would only slow research and whenever the (slower) research makes its discoveries, they would still get corrupted. In other words, there would be no change in human propensity to hurt others with knowledge, simply less knowledge.

Besides, the text is not "very very clear" on AI if and when it hacks major infrastructure or builds nukes. If it was "very very clear" on that, that is what it would say :) - "an AI model is prohibited to be the decision-making agent, solely instigating critical harm to humans". But what the text says is different.

I agree that AI harms to people and humanity need to be minimized but this bill is a miss rather than a hit and the veto is good news. We know AI alignment is needed. Other bills will come.

I'm pretty sure there's a few hundred fusion wmds in silos a few hours north of me, we've had this kind of weapon since 1952.

Nice fact check, thank you. I didn’t know H-bombs used fusion but it makes complete sense. Hydrogen is not exactly the heaviest of elements :)

Well then, for my example, imagine a different future discovery that could be abused. Let’s say biological robots or a some new source of useful energy that is misused. Warring humans find ways to corrupt many scientific achievements for evil.

Sharks with frikkin lasers.

If a model were capable of novel WMD research, I would want it to stay out of reach for the general public! Wouldn't you?

After we split the atom, and saw how dangerous it was, we established some damn strong protections in place to prevent kids from building nukes in their backyards for giggles. If we ever build a physicist AI that is better than Einstein at WMD-relevant research, I sure as hell hope we don't just let the kids ask it how to build nukes or antimatter bombs or whatever. And this is exactly what this regulation is about.

> But what the text says is different.

Since "prohibited" is a word that carries specific consequences, and lawyers are all about consequences, legal texts usually phrase it something like "if a model does critical harm to humans, the human responsible for building and releasing the model will be culpable for the crime", which is a very good TL;DR of the relevant parts of the bill. Isn't it?

Ah, thank you, skipped over that part.

Newsom is the governor who vetoed the bill, not the lawmaker who authored it.

> Does Newsom believe that an AI model can do this damage autonomously or does he understand it must be wielded and overseen by humans to do so?

AI models might not be able to, but an AI system that uses a powerful model might be able to cause damage (including extinction of humanity in the more distant future) unintended and unforeseen by its creators.

The more complex and unpredictable the system the harder it is to properly oversee.