Item — HN Reader

> With this incident, we failed you, and we are deeply sorry.

I'm always impressed when people actually apologize instead of dancing around an almost apology.

Overall this is one of the best post mortems I've seen - great tone, very well written, super informative, has all of the steps (apology, information on issue, real steps to prevent it happening again) that hurt customers generally want to see. Really impressive timelines too - 43s reconnect after initial issue, 15 minutes to change status.

Gitlab overall seems to have more incidents like this and I really like their custom of working through them in public Google docs. Definitely seems a better idea for incident communication than relying on your own tech (github pages) for incident communication.

3 comments

I think this no-fault mode of incident recovery is a bit overdone. In every HN thread, I find people advocating it as some sort of major revelation.

The mindset that problems are caused by failures in process, and not people is probably right. Even if not, I think it has become popular because it describes a world we would like to live in. It reduces anxiety, and might not negatively effect people's diligence because there tends to be diminishing returns with such negative feelings (c.f.: why the fear of starvation does not promote entrepreneurship).

It might, however, be misapplied when used for companies. Gitlab is the best example here: It seems like they actually came out ahead in the famous incident you're referring to, even though that incident was the most breathtaking series of organisational incompetence imaginable.

So it's completely ok to spare the one guy's head who happened to issue the final 'drop db;'. But let's not congratulate them too often for at least telling us that of their three redundant backup mechanisms, one had never worked, one silently stopped working months earlier, and last was just erased by 20-something who mixed up his terminal windows (paraphrasing from memory).

Definitely checkout Gitlab's post mortem from last year https://about.gitlab.com/2017/02/10/postmortem-of-database-o...

It's very informative and also dives deep into their "Whys" pertaining to the outage.

Thanks for sharing that. We are glad that you appreciate our transparency which is one of the core values at GitLab.

Freakonomics podcast had an episode on apologies recently - http://freakonomics.com/podcast/apologies/ - one of the more interesting things discussed was how the start and end of the apology has a dominating impact. It seems people are incredibly sharp at recognising insincerity too. I'd say github did a great job here.