Item — HN Reader

As the tool gets better, people trust it more. It's like Tesla's self-driving: "almost" works, and that's good enough for people to take their hands off the wheel, for better or for worse.

The "almost" part of automation is the issue + the marketing attached to it of course, to make it a product people want to buy. This is the expected outcome and is already priced in.

2 comments

Exactly, Waymo were talking about this a few year back, they found that building it up gradually will not work, because people would stop paying attention when it's "almost" there, until it isn't and it crashes. So they set out on having their automation good enough to operate on its own without a human driver before starting to deploy it.

I would say the opposite here. The perpetrator has rejected multiple Claude's warnings about bad consequences, and multiple Claude's suggestions to act in safer ways. It reminds me of an impatient boss who demands that an engineer stopped all this nonsense talk about safety, and just did the damn thing quick and dirty.

Those guys who blew up the Chernobyl NPP also had to deliberately disable multiple safety check systems which would have prevented the catastrophe. Well, you get what you ask for.

I view it more as "I crashed my car, I should have been wearing my seat belt, wear yours!"

Source: had codex delete my entire project folder including .git. Thankfully I had a backup.