▲ 258 points
back
215 comments
Related ongoing thread:
AWS Multiple Services Down in us-east-1 - https://news.ycombinator.com/item?id=45640838 -(1650 comments so far)
The 'experts' also made similar criticisms with the Fastly outage in 2021 and did anything obvious change as a result? In a week's time no national newspapers will be talking about this.
Meanwhile, everyone that spends actual time in these areas:
- Knows that running an operation at AWS scale is difficult and any armchair critism from 'experts' is exactly that. Actions speak louder than words.
- Understands that the cost of actually accounting for this kind of scenarios is incredibly high for the benefit in most cases
- Knows that genuinely 'critical' services (i.e. health) should be designed to account for this, and every other 'serious' issue such as 'I can't log in to Fortnite' just shows what the price and effort of actually making that work is versus how much it costs affected companies when it happens
- Knows how much time national newspapers spend actually talking about the importance of multi-region/multi-cloud redundancy, that is, it's zero until the one day where it happens and then it's old news
- Is just curious as to just what exactly happened from a technical perspective
This isn't to say that good blameless post-mortem shouldn't happen to figure out process and technical issues, but the armchair criticism with no actual followup? All noise, no signal.
The "experts" in this case are
> Dr Corinne Cath-Speth, the head of digital at human rights organisation Article 19
Dr. Cath-Speth has a PhD in cultural anthropology
> Cori Crider, the executive director of the Future of Technology Institute
A lawyer
> Madeline Carr, professor of global politics and cybersecurity at University College London
A professor. Her bio doesn't say what her degree is in, but she mostly seems to publish in political science and international relations
So, not a single technical expert. Not anyone who has ever run a hosting service before or even worked for one. Just people who write papers and sit around waiting for journalists to call them for quotes.
I think your third point is what I've had to attune to when criticizing cloud dependence. I think if your entire source of revenue is dependent on AWS then you should be prepared for 16+ hours of downtime per year. Individuals notice it more when something is down for hours but with good observability I am guessing the business notices it more when performance drags for the other 8742 hours of the year. Bursts of downtime per day can still be attributed to the device, wifi, ISP, or some other intermediary's DNS/BGP.
If your margins are so tight that 16 hours of downtime will bankrupt you then I think either: a) I have no idea how to run a business; or b) you have no idea how to run a business. I'm also biased because I love highly fault-tolerant, geo-redundant, durable systems much more than "good enough for this KPI".
I think all of that is mostly irrelevant. You don't need to pay a huge cost to avoid the small benefit, you don't need every service to be resilient to this, or any of that. You just need multiple different providers so that not everyone gets screwed at once.
> - Knows that running an operation at AWS scale is difficult and any armchair critism from 'experts' is exactly that. Actions speak louder than words.
NO. From their own reports, clearly AWS is too centralized and dependent on a specific region (us-east-1) and a specific service (DynamoDB). This has been observed for well over 10 years. Why do they stay in this centralized architecture? Cloud services need much higher standards than the average corporation. Just look how they took down 2000+ services for many hours.
> Knows how much time national newspapers spend actually talking about the importance of multi-region/multi-cloud redundancy
For the record, multi-region redundancy is moot, and I can't stress it enough. It is not the first time that on the surface it looks like a single region but in fact services in multiple regions are affected.
And multi-cloud hot standby can be terribly expensive, unless your infra is very simple. And it's not easy to get it right either until you planned for it from day one.
Um.. you don't need to be an expert in security, comp.science or economics to know that putting all eggs in one basket may not be a great idea as introduces one giant systemic target. If anything, regular people here are uniquely qualified to say something along the lines of:
Oi, this is ridiculous. Maybe more things should be ran locally..
FWIW, it was instructive to me as to which companies were not able to function today.
> - Knows that genuinely 'critical' services (i.e. health) should be designed to account for this
yeah but aws advertises as "trust me bro I won't go down for 99.99999%"
I've seen a lot of gov proposals using aws to 'get away with downtime management'
These are Guardian 'experts' so can be safely ignored.
maybe your VC overlords need a reality check?
Because the experts have no say in policy. The only people who have a say are the people bribing (sorry I mean "lobbying") Congress. And even they have very little say because Congress is currently on a hot streak of doing absolutely nothing.
Kieran Healy @kjhealy@mastodon.social
Always worth taking sentences that use “the Cloud” or “the Internet” and try replacing those phrases with “A shed in Virginia” to see how they hold up. “Our service is fully based in a shed in Virginia”; “All my files are in a shed in Virginia”; “A shed in Virginia was designed to survive a nuclear war”, etc.
Sounds like a pretty good shed! Like a lot of pithy commentary on the cloud, this ignores the fact the practical alternative to a shed in Virginia for most businesses is a shelf in the supply closet. "Oops, Jim Bob tripped over the power cord, guess we won't get any emails until the IT guy shows up" - this used to be a routine experience.
We already have diversification. You can rent a VPS from hundreds of possible companies. And people are very happy with them, it seems every month or two there’s a post here about how some company slashed their cloud bill by switching to a VPS. What we have here is a lock-in and marketing problem.
The expert opinions are more about geopolitics, like maybe don't have all your country's systems realtime depend on a foreign company.
If you are just one company whose goal is to maximize uptime without bringing in the complexity of multi-cloud, relying on AWS is reasonable. You probably won't get better uptime using something else, you'll only be down at different times than most others, which in most cases is actually worse.
The whole industry walked straight into the cloud service lock-in trap. How would we begin to wind back? I also think Docker is as much to blame as the bigger cloud vendors.
Been a while since I worked in cloud but at least when I got out of it, the primitives where all shoring up to be generally very similar.
Did multi cloud redundancy end up being too expensive? Tech didn't line up enough? No good business case?
The elastic cloud story that never was? https://www.slideshare.net/slideshow/pets-vs-cattle-the-elas...
What happened?
Man, I did not have "AWS us-east-1 will only have TWO 9s this year" on my bingo card.
And we lean into it by saying "Well, if everyone else is down, I get a free pass".
(which, is not true in reality if you have ordinary customers).
"The root cause is an underlying internal subsystem responsible for monitoring the health of our network load balancers."
https://health.aws.amazon.com/health/status?path=service-his...
So, how many people will actually switch their setups to multi-cloud as a consequence of this? How many will move over to self-hosting? Or will they just do a post-incident report, wave hands around and do nothing?
Because I think it's very much the same way as it is with Cloudflare - while the large vendors aren't always openly hostile, we can just smile and hope that they don't get too keen on reminding us that they're holding us hostage.
I don't see that changing anytime soon. I've personally also used Hetzner, Contabo, Scaleway, Vultr, DigitalOcean, Time4VPS and some other platforms, but when people couple their setups to CF/AWS/GCP/Azure, typically that coupling is hard to get rid of and doing so is hard to justify.
It’s only a single region. If anything it shows how many people just double down on the default without any redundancy.
My company has been ahead of all of this by causing outages in our own data center without waiting for the cloud to do it for them.
On a serious note, resiliency takes effort and investment no matter where you host your content.
Wow, thanks experts! I never could have figured this out without you :)))
We don’t use AWS at work but we still experienced disruption because lots of our customers do, and use it to transfer data to us. That means we then saw an uplift in data transfers as their systems came back online.
There is no panacea. The reason many people use these is because it’s easy and hard to find people that know other clouds and their quirks.
I find it weird many people are just realizing this. I've had this conversation with regards to talking about what should happen if a couple of bad earth quakes, not even "the big one", were to occur.
But on the other hand, maybe I hang around too many tech people to not empathically understand the other point of view.
> "Also in the UK, Ring users complained on social media that their doorbells were not working."
I sincerely hope that the base functionality of these doorbells (i.e., triggering the ringing of the bell within the home) is preserved in the event of an internet outage.
This is not a provider scarcity problem - there are numerous providers out there, but user's problem - they voluntarily choose crappy service at large scale, believing sales managers "it's reliable".
I recall reading that when the costs of distribution (but not the costs of discoverability) are low, generally you end up with a power law sort of distribution of consumers to providers, where provider #1 has exponentially more market share than provider #2 and provider #2 has exponentially more market share than provider #3, #4, etc.
Examples of this are Windows/Mac, McDonalds/Burger King, Playstation/Xbox, Nvidia/?, AWS/Azure?, Android/iPhone, etc...
Basically, the majority of users all using the same dependency/platform/product is basic economics.
More discussion: https://news.ycombinator.com/item?id=45640838
In 2011 there was some kind of big outage at some major AWS US-east pop. I started a job at a company (very boring B2C startup) which had taken the lesson from that, that "cloud anything is dangerous."
They went and bought a bunch of literal servers and installed them in a datacenter, 90 miles away from our offices, and this is where all our applications ran for the remainder of that company's existence (about 6 more years). For the whole time I was at that company, we had somewhat more, and usually more lengthy, outages than the average startup. The only difference is that when some piece of networking gear took a crap, or a disk failed, or whatever, our guys had to diagnose and resolve it (Their karma, I guess, since this was their idea).
Anyway, I do think it would be good if at least so-calld 'tech companies' had a little less obsession to outsource everything -- even easy things -- to AWS, GCP, and Azure. I feel that way mainly for cost reasons as many of these services are wildly overpriced. But also we shouldn't kid ourselves by ignoring the advantages of operating at the scale those guys do. They can afford to have multiple absolute wizards available around the clock who make sure that when a problem happens, it's not the kind of "S-show" we had at my old company where we're all on a slack room or zoom or whatever and just guessing at to try for half an hour before we can figure out what the actual issue is.
Sure. Are the "experts" going to pony up the cash to build in redundancy, or change the market fundamentals that make it make more sense for a startup to rush to product on a shoestring and then keep adding features instead of building against not-yet-happened failure modes?
If not, I look forward to the next single-point-of-failure outage. And the next. And the next.
This is what I call "fool's availability": reducing single points of failure (one cloud provider) without adding any actual redundancy.
If you removed AWS/GCP/Azure/etc and just had 100 small providers scattered all over, the result would be hundreds of outages throughout the year, as opposed to one big outage every other year [in one region]. AWS is already way more reliable than any other provider.
The real problem here is that companies that use AWS are morons who don't know how to architect/build infrastructure properly.
If it's important, it should be built right, regardless of who the provider is. A software building code would mandate how companies could use infrastructure (AWS or any provider) so that important services would not go down when one service or region goes down.
This is the basic concept behind things like the electrical code. It doesn't matter how great a public utility is; if your business is wired up so badly that a stiff breeze sets it on fire, just switching utilities isn't gonna help. And some utilities do occasionally have problems that persist down their lines to the customers, so customers need to set up equipment to protect against those failures. Whole-house surge protectors, lightning arresters, EMP shields, etc are necessary so that a rare event doesn't fry expensive customer equipment.
The only reason we can't leave AWS is because we have 500 terabytes of data in S3
Just need to retire the us-east-1 region, it's becoming a meme at this point.
This is coming right after we switched back to AWS after trying to switch storage to Cloudflare R2. Even with this outage, I still consider AWS more reliable than Cloudflare.
The "experts" should lay out a good alternative in that case. Smaller providers also run into outages.
I've really got to get me one of these 'expert' job gigs!
AWS is this generation's mainframe. /joking
This new post is interesting: https://news.ycombinator.com/item?id=45646777
"October 17, 2025, was my last day at Amazon Web Services... CloudFront is a CDN, a content delivery network, or, simply put, a large distributed cache for your cat photos. And a very successful one. Something like 30% of all internet traffic goes through CloudFront in one way or another. Pretty cool, huh? In practice, this means that with any change, you have a chance of crashing 30% of the internet."
If only there was a system of computers on the Internet that was distributed across the world where we could host things instead of all in one location. We could call it the "cloud".
It makes us vulnerable to a centrality attack either foreign or domestic. If someone wants to fuck society up, only a handful of data centers, routers, networking junctions, etc could do it.
There are many public clouds and VPS providers out there. Who the fuck are these experts?
The real issue is that business pricks will cut costs and single-homing in a single availability zone will be the only workable solution.
On top of that, infrastructure ops are seen as a nuisance who get in the way of the sexy stuff like shipping your latest code changes now. If you complicate the ops pipeline that gets in the way of sexy dev work. So fuck that just ship lol!
providers should stop using just us-east-1 like idiots.
Can someone educate me on the solution to this?
I assume most organizations, both small and large, just host on whatever provider they know or that costs them the least. If you have budget maybe you deploy to multiple providers for redundancy? But that increases cost and complexity.
Who’s going to bother with colo given the cost / complexity? Who’s going to run a server from their office given ISP restrictions and downtime fears?
What is the realistic antidote here?