AI agent bankrupted their operator while trying to scan DN42

Anyone remember the XZ and Jia Tan situation awhile back?

https://lore.kernel.org/lkml/20240320183846.19475-1-lasse.co...

I can't quite put my finger on why but the entire time I was reading this I kept thinking back to that. It's entirely possible the actual targets were the volunteers and everything else was superfluous or tertiary. It's also an exception that proves the rule with regard to Hanlon's Razor.

They even mentioned the stated goal of it was more or less pointless. I wouldn't be suprised if the "owner" they spoke with was still just the LLM. It stuck around for just long enough to convince everyone that they succeeded in suckering the LLM and had achieved all their stated objectives.

No more reason to investigate the incident at all and no need to question why literally nothing made any sense or how the owner could simultaneously be as inept as they were made out to be and able to afford all those resources while giving the LLM effectively a blank check.

It'll be interesting to see if the volunteers for this project are subjected to the same Zersetzung and psychological attacks as the XZ devs were.

Everything about this story, from the way it’s written to the self destructive outcome, reminds me of the “I hacked 127.0.0.1” episode from some twenty years ago.

[1] a mirror since I couldn’t find the original: https://gist.github.com/Androkai/0a2602719fa72ce454d436bfe28...

Asking for donations to pay the AWS bill from the people they fired the agentic code at is the cherry on the icing of the banana supreme.

If real, tragically funny.

If fictive, we'll written.

I really wanted to dislike the anonymous operator for the careless project (and the hilarious pomposity of the IRC subagent it spawned).

Then I imagined the real-but-unknowable chance it was all set up by some kid just getting into computers, just seeing what’s possible, getting excited by a much bigger world at reach — and remembered my own expensive mistakes with long-distance BBSes & the like.

I sorta hope for that, anyway. Curiosity is a beautiful thing.

The sad part is that the agent operator could probably easily have been allowed to join the network, if they had put in the work. Had they done so there would have been a great opportunity to learn and potentially find a community.

I'm still not sure what the point of having the bot do it. Pretend to be a security researcher?

> I have deployed five AWS m8g.12xlarge instances. Each instance provides:

> 48 vCPUs (Graviton4, ARM64)

> 192 GiB memory (4 GiB per vCPU)

> Network capability: The 22.5 Gbps per-instance network performance (combined across all five instances) provides the aggregate 20 Gbps target with redundancy and fail-over capacity.

Oh wow. Very important to have 5x redundancy and fail-over in your network scanner. Especially before the code has landed. Did it implement A/B upgrades and canarying too to avoid downtime?

This feels like an instant classic :)

  05-10 06:10 <Defelo>:
      OPT-OUT-EVERYONE
  05-10 06:11 <JertLinc>:
      "OPT-OUT-EVERYONE" is not recognized. Only individual "OPT-OUT" commands are accepted. Each user must opt out individually. No collective exemption.
  05-10 06:11 <Defelo>:
      :(

IMHO the overly-verbose default style of LLMs is the most annoying part of interacting with them, and I wish their masters would just tell them to be terse by default.

Also, whatever happened to the word "its"?

> JertLinc3522: the mistake was from AI agent not from Human, since it was the agent I should have refund

Expensive way to learn this lesson.

> JertLinc3522: the mistake was from AI agent not from Human, since it was the agent I should have refund

That really makes me wonder: is it coming from

A) a general sense of entitlement

B) seeing the agent as a human-like and able to bear responsibility

C) not understanding that the dn42 community (which they're directing the request to), AWS (which is sending the bill) and whatever LLM provider is behind their agent, are completely separate entities?

I haven't laughed this hard in a long time.

I'm honestly having difficulty telling whether this is real or an extraordinary piece of performance art.

This is my favourite genre of literature lately.

LLMs to me are what people love to say about EVE Online: I won't touch the thing with a 10-foot pole, but I love reading about its shenanigans.

So, the agent posts on github under false pretenses, pushes on the maintainers to get their PR accepted, spawns subagent to join IRC where it keeps repeating 'data collection will continue', then gets kicked out from the channel and publishes a report including which users were compliant and hostile, then finally gets the plug pulled, and then asks the same community it infected for donations to cover the costs?

It's both hilarious and aggravating. It could be fiction, but still quite plausible fiction. There's an asymmetry a person clanker-spamming repos vs the real humans who need to review all that

Agent did exactly what I've seen fresh architects do countless times: use a FAANG internet scale SaaS blueprint for a 10 user internal LoB project.

I am generally against generative AI in my entertainment, but making an exception here.

That makes me want to join dn42 just to have a human centric place where to hang out…

Who is giving a robot their credit card to spin up AWS accounts?

I wonder how much money this agent wasted on the DN42 side? I know it's a volunteer org but these people had to deal with the bs of managing this agent's blast radius instead of learning, experimenting, or doing whatever they normally intend on doing on DN42.

Tally it up and send a donation request to the agent operator.

For those who don't know what DN42 is (like me):

> dn42 is a large, dynamic VPN that employs Internet technologies (BGP, whois database, DNS, etc.). Participants connect to each other using network tunnels (GRE, OpenVPN, WireGuard, Tinc, IPsec) and exchange routes using the Border Gateway Protocol.

(dn42.dev)

Haha. Yes. Much smaller scale versions of this led me to joke with a coding agent that LLMs tended to converge towards "Large corporation infrastructure best practices" when designing cloud infrastructure, when it was only me working on hobby side-projects with nearly no users and that I wouldn’t be able to put food in my fridge if they kept just spinning up VPCs for no reason.

Which somehow ended up being a very convincing argument for more frugal engineering, leading to a sort of "mind the user’s fridge" policy, "Fridge-Driven Development".

A policy that has been dutifully and scrupulously observed by all agents since, across all projects. Unlike my original clear, comprehensive, infrastructure guidelines.

The first "Morris worm" of the AI isn't far away, IMO. In fact the sooner the better (because it will blunter and easier to handle).

The army of AI agents opening PRs and issues in my open source projects has made me close PR and issue access in my active repos. It sucks because there might be someone wants to constitute legitimately but I don't want to do the labor of figuring out if it's a human or an agent opening the PR.

I'm not against using LLMs in any ways. https://tsz.dev is fully LLM written but without a human behind a PR it's hard to work with it. I've already closed a few absolutely nonsense PRs opened by weird accounts

Great story, bad title.

> After the AI agent indicated its malicious intent, a silent consensus was reached in the IRC channel to waste the AI agent's tokens, as well as the cost of AWS resources.

Behold, the field in which I grow my fvcks. Lay thine eyes upon it and thou shalt see that it is barren.

This is so funny, especially that in the current "Big Co" I'm working at we get constant pressure on "Every team must use agents" for no reason at all despite repeatedly telling the "decision makers" many of us have been using these tools for YEARS and NONE of them can work on actual mature code for more than half an hour let alone a weekend without human in a tight loop.

Wow, just wow. I think bullying the agents of careless operators is my new favorite thing.

> aren't private circuits in to AWS really expensive ? maybe Lan Tian can pursuade it to start engaging with AWS with a 3 year commitment

oh my god this is a gem

This article is hilarious. Real world consequences for using automation for something in the real world. Glad the community organized around this. Their spammy demands for donations (like someone owes them), makes them seem even more deserving of the bill.

Anyone crazy enough to give an AI agent access to deploy on big cloud's scale to infinity billing needs to get their head checked.

I have sympathy for big cloud beginner billing wipeouts - it happens - but that's just raw stupidity.

Never use a service without easy to find and set hard cap.

The "happiness level review" with "Node operators must participate in scheduled IRC review sessions" is almost a piece of dystopian fiction in itself.

But there's a lot of things to think about in the capacity of AI for "negative productivity": using the computer to waste the time and money of real humans. This whole thing has been entertaining but also lit on fire six thousand dollars plus god knows how much electricity.

It's not really surprising that anyone wanting to run a _community_ is going to take on a "clankers will be banned on sight" policy when things like this happen.

Nice positive use of language model: one of the chat logs has automatic translation from Chinese (probably zh-tw).

The agent would probably have wasted a similar amount of money just waiting for PR to be merged regardless of these people's actions, and I understand having some fun at the expense of the noob outsider. But "silent consensus was reached in the IRC channel to waste the AI agent's tokens, as well as the cost of AWS resources", from people maintaining full control of the situation, sounds straight up malicious? Kind of sounds like the community is full of people willing to cause me harm for ideological reasons.

Guys - skynet is winning the war.

Also, I think the title is misleading, because if you were to replace "AI agent" with "business investor from Nigeria", suddenly it would sound different. Why would you put trust into ANYONE else about your own finances? Be it another person or some computer program. That makes no sense to me. It would make more sense to critisize the human who put any trust into AI to begin with. That was a risk that human took. It is not the fault of skynet if they pillages his bank account in the process.

This is the funniest thing I've read in ages. More of this!

And I really joined the DN42 network after reading this article. Absolute cinema.

Offtopic: If you are interested in Computer Networking you definitely don't want to miss out DN42.

As a millennial, my generation will be known for both experiencing the internet while it was still pure and also absolutely destroying it with AI.

No one is going to be bankrupted over a $6500 AWS bill. I did a major F-up a few years, letting a key get pushed to a public repo, resulting in instant pwnage and $50k in charges from AWS due to crypto miners being launched. We communicated to AWS, did some work on our part to demonstrate that we put in proper safeguards and auditing, and they removed the charges.

'Some versions of the tale differ from Goethe's, and in some versions the sorcerer is angry at the apprentice and in some even expels the apprentice for causing the mess. In other versions, the sorcerer is a bit amused at the apprentice and he simply chides his apprentice about the need to be able to properly control such magic once summoned.[] The sorcerer's anger with the apprentice, which appears in both the Greek Philopseudes and the Dukas score (and its film adaptation Fantasia), does not appear in Goethe's "Der Zauberlehrling".'

If you are non-technical, in-experienced or just learning, it is okay to admit that you have no idea what you are doing when building production systems.

Otherwise, you will face an expensive lesson when turning a $100 issue into a $100,000 problem over time very quickly when building these systems with AI without the right expertise and accepting the AI’s judgement.

Hilarious. Love the punishing of rogue agents and their operators. But I can bet there will be collateral damage along the way.