Item — HN Reader

> "50k windows-based endpoints or so. All down."

I'm a dev rather than infra guy, but I'm pretty sure everywhere I've worked which has a large server estate has always done rolling patch updates, i.e. over multiple days (if critical) or multiple weekends (if routine), not blast every single machine everywhere all at once.

3 comments

If this comment tree: https://news.ycombinator.com/item?id=41003390 is correct, someone at Crowdstrike looked at their documented update staging process, slammed their beer down, and said: "Fuck it, let's test it in production", and just pushed it to everyone.

Which of course begs the question: How were they able to do that? Was there no internal review? What about automated processes?

For an organization it's always the easiest, most convenient answer to blame a single scapegoat, maybe fire them... but if a single bad decision or error from an employee has this kind of impact, there's always a lack of safety nets.

Even if true, the orgs whose machines they are have the responsibility to validate patches.

This is not a patch per se, it was Crowdstrike updating their virus definition or whatever it's called internal database.

Such things are usually enabled by default to auto-update, because otherwise you lose a big part of the interest (if there's any) of running an antivirus.

Surely their should be at least some staging on update files as well, to avoid the "oops, we accidentally blacklisted explorer.exe" type things (or, indeed, this)?

Companies have staging and test process but CS bypassed it and deployed to prod.

If I understand the thread correctly, CS bypassed the organization's staging system

I'm guessing there's a lesson to be learned here.

This feels like an auto-update functionality. For something that's running in kernel space (presumably, if it can BSOD you?) Which is fucking terrifying.

Crowdstrike auto-updates. Please do not spread misinfo.