Item — HN Reader

Well, this outage says something about the companies that religiously depend on it.

If your entire service just went down as soon as this happened, Congratulations! You didn't deploy in multiple regions or think about a failsafe/fallback option that redirects from your affected service or instance.

6 comments

Very few companies or systems need near-perfect uptime. Multi-region cloud engineering, especially once data is involved, is incredibly expensive. If you do need the kind of resiliency you usually engineer it for just a very specific component rather than the entire system.

An outage like this happens how often?

Edit: Looks like this is affecting a single AZ... so bit different situation, but I would agree if you're not capable of surviving a single AZ outage in 2019 then your engineering team should be replaced.

> your engineering team should be replaced

My engineers are all React and CSS web developers. They don't know anything about multi tenant data resiliency. But they can make a real pretty "system down" page.

> Very few companies or systems need near-perfect uptime. Multi-region cloud engineering, especially once data is involved, is incredibly expensive.

Without data it doesn't even cost significantly more than running in a single region nowadays, if you are willing to go serverless. As serverless stuff (FaaS, ...) is pay for what you use and the provider handles the scaling automatically behind the curtain you can easily deploy to multiple regions without much additional cost.

With data you have of course the cost of storing the data multiple times in the different regions (or to come up with some kind of sharding) and solving the consistency challenges that come with that, but at least services like DynamoDB and S3 offer cross-region replication out of the box nowadays and you don't have to provision any capacity like you used to (thanks to DynamoDB AutoScaling and so on).

Once you have your application running in multiple regions you can direct users to the closest one, so they enjoy lower latencies.

I believe for a lot of applications running cross-region just makes a lot of sense as it offers various benefits.

Or the PM team that wouldnt let the engineers do it right needs replaced.

Always CYA guys... you will pay for this if you dont.

Most of the connectivity issues we were seeing were with instances in one of the us-east-1 AZs but we were seeing issues in other AZs in us-east-1 as well. Not sure why AWS is acting like this issue in only affecting one AZ.

For many sites hours of downtime every few months is not critical. If the cost of downtime is less than the cost of reducing them, don't bother.

Yeah, but you have to weigh the cost of multi-region deployments and failsafes vs the cost of downtime. For smaller shops downtime may be acceptable.

It’s not an issue with an entire region, it’s an issue with a single AZ in a region. If you did the bare minimum - set up your RDS with a replica in separate AZ’s, run your servers in an autoscaling group (even with just a min/max of 1) configured for separate AZ’s, used services that are multi-AZ by default - almost all of the managed servers - you could still be up.

Nope, our EC2 and RDS instances are multi-AZ, still got affected. Might look into being multi-region though.

Multi AZ would have also sufficed. This appears localized to one zone, not the whole region.

Several people report problems with multizone deployments, so it seems like AWS is downplaying this.