back
52 comments
Down from LA also. Really curious what their revenue loss per minute is in this kind of situation.

Edit: based on product-only revenue for Q1 2020 (since AWS was unaffected), it looks like around $325K per minute if Amazon is down for all of North America. Naively assuming that people wouldn't just wait to order, of course. That's actually lower than I thought.

Sorry, it's a Virginia data center thing. We're working on it.
Can you elaborate on the issue/fix?
Revenue of 280.5 billion USD in 2019 works out to roughly $500k/minute.
Yeah, but how many people come back to make a purchase after the site is back up? I don't think you can just extrapolate that 1 minute down equals $500k lost revenue.
For sure, just napkin math for fun. Also, I believe that number includes revenue from non-amazon.com sources (like AWS).
The revenue loss wouldn't even cover their taxes.
What..taxes?
Is it really all lost or just delayed? Maybe some will still purchase from them when they are back online, but maybe some will not.
AMZN's revenue is about 10k / second.
Wow, down for me too (DNS doesn't resolve, using google dns). Wonder how many $$millions they're losing every minute.
Impossible to say really, since most will just come back later. Especially given that it'll likely be brief.

In any case, must be like a 5 alarm fire at Amazon right now!

Revenue of 280.5 billion USD in 2019 works out to roughly $500k/minute.
The calculation is accurate in one sense, but I don't think it represents actual revenue loss for two reasons. Firstly, spending is non-uniform and peaks a lot on Black Friday, near Christmas, etc. Second, assuming the site is down for only a few minutes (it's back now), a significant fraction of people will probably just try again later. I still imagine this is a significant loss.
guess who just spent 10 minutes debugging pi-hole, this guy
Thank you!
Specifically the revenue that won't come back once the website is back online.
https://smile.amazon.com/ is working for me.

I've got a chrome extension that redirects, so I didn't even notice the issue.

Same here. Smile Always for Chrome.
Aaand it's back.
Yep, back up for me, too. Lucky them - DNS outtages can sometimes take a lot longer to resolve (heh).
I checked and checked again before submitting.

https://downforeveryoneorjustme.com/amazon

All green here -- https://status.aws.amazon.com/ -- although I cannot login to AWS as one of the redirects in the flow is through an amazon.com url.

I know the AWS status page is known for delayed updates but AWS itself is up. Amazon.com (retail) is what is down.
Amazing how many times simple configuration issues can kill you.
% nslookup www.amazon.com

Server: 8.8.8.8

Address: 8.8.8.8#53

server can't find www.amazon.com: SERVFAIL

Colorado, USA

Actually, the www was working. It's a cname to tp.47cf2c8c9-frontier.amazon.com, which should have been a cname to d3ag4hukkh62yn.cloudfront.net, but that record was broken for about 20 minutes.

Update: that frontier cname seems to be bouncing between Cloudfront and Akamai now, so I suspect they're fighting off either an attack or internal issues. I see issues to other route53 servers as well.

When I do `dig amazon.com @8.8.8.8` I get a number of round-robin IP addresses. But `nslookup` returns SERVFAIL. Why are they showing different results?
looks like `amazon.com` is just a redirect to `www.amazon.com`. `dig www.amazon.com @8.8.8.8` returns nada.

% curl -v https://amazon.com

HTTP/1.1 301 Moved Permanently Location: https://www.amazon.com/

Could also be that your default DNS isn't 8.8.8.8?

You're right, thanks. `dig www.amazon.com` for both `@8.8.8.8` and `@8.8.4.4` return no IP addresses. Same for CloudFlare's `1.1.1.1` ... ouch.
Ouch indeed! :)
Cannot log into aws console at the moment.
Down for me as well.
DNS doesn't resolve somehow. works now.
GCPL Do you need help? :troll:
It's down for me too...
seems to be up on mobile
To which URL are you connecting? amazon.ca is up for instance.
amazon.com and aws.amazon.com
aws.amazon.com works here while amazon.com does not, over both my home internet connection and mobile (via my carrier's LTE connection).

Which continent are you on out of curiosity?

No it is not.

Let's not post this generally incorrect type of post.

You waited too long to post.
Seems to work now.

Nairobi, Kenya.

so much for high availability...

edit: I'm don't understand these swarm of downvotes?

If I was a VP/CXO of a $1.2TN company's website being down for even an hour, I would be in a fit of rage and would request a full investigation of how this happened.

Mistakes happen, highly available systems fail, being enraged and demanding heads isn't going to make anyone happy but just scared. A full investigation is probably going to happen and steps will be taken to make sure this doesn't happen again but it's hard to blame it on a particular individual. Ideally you'd have steps such that one person alone can't screw up and it also requires other people to let that happen.
Shit happens, and rage is pointless. Nonetheless, it's worth noting that even the best HA just isn't that good. Certainly I wouldn't care to fly in a plane that critically depended on any web provider, even Amazon.
I feel like that comparison is less compelling after the Boeing 737 MAX debacle.
hmmm, maybe you're right, could have been a gremlin that had chewed on one of the cables.
It's not a gremlin and at the end of the day it's probably a misconfiguration or bug somewhere but it doesn't help anyone to point fingers and blame. A system probably went through a design phase, why didn't the reviewers of the design didn't catch this? The implementation was probably reviewed by peers, why didn't they catch this? It might have gone through QA, why didn't they catch this? And the list can go on and on.

At the end of the day a mistake was made and it should be fixed but being enraged doesn't help anyone. Figure out what went wrong, how to make sure it doesn't happen again and do it.