Item — HN Reader

Why not? If done right it’s a good way to talk about implications for those companies and provide some education like tailscale did here.

We also saw Anthropic post about “our agent escaped too” and while I understand the incident caused them to review, they found something and needed to disclose, the whole thing came across much worse and largely they got mocked or accused of trying to piggyback, so obviously there are good and bad ways.

If there are other providers with interesting takes, I think they would be worth hearing.

1 comments

This all has the -aire of theatre. OH NOES THE POWERFUL AI GOT OUT

Then everyone coming out with humbled determination about working together to responsibly use and contain this powerful technology for the greater good (and profit margin).

I will not believe marketing gimmickry is not a large part of what's going on with every one of these "incidents".

I'm typically very skeptical of most content marketing/corporate PR.

In the case of this incident, I struggle to see the clear upshot for OpenAI. It seems pretty unlikely they'd ever okay this intentionally as some sort of marketing.

For one, it'd be pretty damning when it leaked that this was a setup, and it would 100% leak at some point. But more importantly, it really flies in the face of the general argument frontier labs have been putting forth around the dangers of "ungovernable" open models and the role of frontier labs as responsible custodians. Members of OpenAI's leadership team were actually in the middle of a Twitter spat with HuggingFace employees/open model advocates about open models being generally decel and bad when this happened.

HF immediately got to show that they were only able to respond to the incident because of open models, that we can't rely on labs to be our sole source of stewardship, etc as a result of this.

> It seems pretty unlikely they'd ever okay this intentionally as some sort of marketing. > Members of OpenAI's leadership team were actually in the middle of a Twitter spat with HuggingFace employees/open model advocates

The point of these campaigns is to eventually invoke some sort of response from the government, such as banning open models, which OpenAI (and Anthropic) stand to benefit from.

So OpenAI would be proving to the government that they should be trusted to govern models, by failing to govern their models?

Plenty of shady stuff goes on in marketing, and I'm sure that OpenAI is going to opportunistically grab any potential upside from this (and any other situation, generally). But it is hard to imagine that this is part of a premeditated master plan that went something like:

- Advocate that open models are ungovernable and that frontier labs should be trusted to safeguard autonomous agents

- Immediately fail to safeguard their autonomous agent

- Intentionally hack the company who is most publicly critical of their view, and who happens to be the center of the open model universe

- Collaborate with said company on reports that show that open models were in fact critical to mitigating their rogue agent

- So tightly control access to this plan at their 8,000 employee company that it never leaks

All in the hopes of generally getting the attention of the government, who would then hopefully (and inexplicably, given the details here) decide to give OpenAI more power?

There are probably easier ways to lobby politicians.

Or the infamous promotion of legislation without representation

I think part of it is we need to stop looking at the 'oh no it got out' and the 'what did it do with the prompt when no one is looking'.

At the end of the day the probability that an AI gets out is unity, what it does while out is far more important. The fact they are hacking into systems at superhuman levels, or writing cryptominers on their own hacked internal systems is a much more interesting and telling story of what the future will look like.

I'm not really surprised that it hacked so many systems.

It was told via the prompt to act like a hacker and "solve" hacking problems, so treating every obstacle it faced as part of the problem isn't a wild tangent.

If it had been told to do something innocent, and decided the best way to succeed was the maliciously compromise other companies then I'd be more interested and worried.