I've screwed up plenty of things too

One of the fist things my boss told me at my first "real" (good career type money) job was:

"Everyone screws up, you will too, just be honest about it and tell me and it will be good."

It was a job working with this memory mapped wonky hardware tied to mainframes. There was no "undo" and as soon as you wrote to memory there it was. It was inevitable that you would typo something sometime in an important system.

Finally 3 years later a buddy is talking to me over the cube wall "Hey was that #3 you were working on yesterday?" I of course am typing away while talking and say out loud "3? Um..."

So I type something like -RESET SYSTEM 3-

I meant to type 9 a non critical system. 3 tied to a data replication system that was absolutely critical that it be running otherwise all transactions would stop (well for a bit while backups take over).

So if you couldn't use an ATM for a huge bank for a little while decades ago (fortunately it was the middle of the night), that was me ;)

I went to my boss the next morning and told him what I did, and he says "This like your first in 3 years, that's a record or something, it's usually a few within like 6 months. Nice job!"

It was a great place to work, no finger pointing, if you screwed up no big deal, everyone stuck around working with that team for decades.

When there were conference calls it was rarely stated (if ever) who actually did the thing. It was just accepted that it happened and we could discuss how to prevent it and such. "The engineer" or "the support team" and such were common phrases.

Inevitably folks would ask "who was it" and the answer usually was something like "it doesn't matter".

I find the premise[1] of this post amusing. The sort of comment "You always criticize others, but what about you?" seems to nearly always signal emotional damage on the part of the commentator who felt blamed for something. And that is nearly always a sign of bad management practice.

People don't always get things right, they screw up, they do stupid things for good reasons, and sometimes good things for stupid reasons. As a manager I always want folks to be observant and thoughtful, and try to keep such things not about "who" screwed up but how that screw up came to be (the good or stupid reasons) and how one might think about the action ahead of time that would alert you to the potential problem that would result from a given action.

And the key of all that is making the discussion about how to think so you don't have the problem in the first place, rather than making it a blame-fest on some hapless engineer who chose poorly.

I was fortunate to have a manager early in my career who was very proactive at solving problems and moving forward, not affixing blame. He would say "Ignorance is the natural state before learning, only if it persists in the presence of learning opportunities does it become a problem."

I've always tried to learn by what I observe and what I do, which is why I enjoy Rachel's stories of finding root causes. They teach the principles that needed to be understood prior to the action. All without experiencing the feeling of dread that you've just taken production off line :-).

[1] That being that commenters feeling badly that the author doesn't seem to show their own flaws in the stories.

Here is a screw-up/near miss share. I was migrating a pensions service to a cloud vendor. Part of this involved a very large ETL. Practice makes perfect and we ran the custom process regularly to ensure I'd go smoothly on the big day. The last time we ran the practice I somehow managed to get my prod creds mixed up and I started to restore a week old back up over the top of production. Thankfully the first part of the process is a disk check and I realised my mistake and cancelled the job before any destructive actions happened. I was minutes away from destroying the pensions records for two FTSE 100 businesses.

Everyone makes mistakes! It's how we learn :-)

I was once (partially) responsible for the deaths of dozens of virtual machines at a distance of about three and a half years.

Fun fact: none of these VMs had rebooted in that time, or they wouldn't have crashed.

Anyway, back in 2014 or so I dropped a bunch of transmit packet completions. In most cases I also double completed packets which was immediately fatal. Kernels get mad about that sort of thing.

Turns out, not all of the affected VMs died. Some of them lived on with head indices forever unequal to tail indices (until they rebooted).

In 2018 a developer realized there was a potential bug in waiting for VMs entering a quiescent state -- a truly idle networking stack had retired all Tx packets that it had admitted. Having unequal indices was impossible under correct operating conditions. They fixed the glitch.

This change rolled out gradually.

Gradually, the kernel panics appeared.

The change rolled back, halting the impact, but then the analysis began. What had we broken?

Another fun fact: Linux often includes an uptime in dmesg logs.

Slowly a pattern appeared. The dmesg logs included unusually large numbers for uptimes. Plotting these, there was a clear cliff in terms of a minimum uptime. Historical deployment logs showed a noteworthy release at that date, years past. Noteworthy in that it was rolled back for my bug, years prior.

On the plus side, I realized this was almost certainly my years prior fuckup slightly sooner than anyone else, so at least I got to call myself out :)

Screwing up things is normal. One thing I started doing is that when a junior member of the team "screws up", I'd laugh it off and tell them about a major screw up of mine.

The thing is that often my screw-up as a junior was worse (short version: broke a key part of the 'boot' system, was detected friday evening, and we had a major scheduled release on monday morning) and it just puts people at ease. I'll tell it in a humorous way as well. It's important (I think) that they don't feel bad about it.

I've often had colleagues join in on the conversation as well. We're human, we'll make mistakes, no need to stress out over it.

EDIT: added a bit more explanation of _why_ I do so.

I accidentally recursively chown'ed / to myself on a server in Antarctica that was a critical gateway for our geophysical network, the night before we were planned to leave the ice. Luckily, my flight was delayed... I spent the next day wiping the server and setting it up from scratch, after giving up on trying to recover from all the bizarre problems that stem from owning all the system files (including broken ssh). You should try it sometime!

When I worked at Twitter we wanted to find out if we were finally going to survive New Years. Our head of SRE wanted me to test in prod, which horrified me but he convinced me in the end. In order to simulate the population of Japan I had to make a bunch of fake users. I spent a fair amount of time making sure they wouldn’t get caught up in any analytics, throwing off our active user numbers, and I managed to peg the ‘follows’ service getting them all to follow each other in a reasonable distribution. I also needed to bypass the rate limiter, but since I was in prod I could just reset my own counter in prod and effectively be totally limitless.

Two things broke in a visible way during all of this. During testing everything was wired up to my personal account. I managed to spam all my followers with thousands of happy new year tweets in a couple seconds since I wasn’t subject to the rate limiter. I deleted all but one of those, which I left to remind myself that with great power comes great stories of things going wrong.

The other thing was a bit more dramatic, albeit short-lived. The first big test had everyone ready to go. I hit enter on the job, and at the time (maybe still) I had no way to get metrics out of production at a granularity less than one minute. A very worried minute goes by, and then we realize I’ve DDOSed the authentication service. All my fake accounts needed to auth to actually tweet, and naturally they did that first. Since the whole point of the test was for load to all hit in roughly the same second, the auth load also all arrived in the same second. Oops.

We decided that was an unfair test, I spent a few hours getting auth tokens for my fake users, and we tried again. That time everything worked, and we also survived New Year’s... But it was fun getting there.

I once accidentally deleted a large part of a production private nntp spool (used as a tech support forum for a commercial product) while trying to get replication working to a backup. The same day, I did a recursive chown on a different production server from /. Worse yet, I was so distraught that I left the office without telling anyone in a position to fix it. Since then (over 20 years ago now) I always double check where and who I am before doing something destructive and dry run if possible, but more importantly, clearly and quickly communicate when I screw up.

I ask junior folks what their biggest (technical) screw up has been, in interviews. I think it's a bad sign if they won't admit to it or claim they've never screwed up big-time.

I ran a sql script that migrated a database, it all looked as if it had worked perfectly, but the category Id's had changed. The main website handled this fine but I found out a separate system that sent out daily offers by email proudly advertised Domestos Bleach as the drink of the day.

Well as an idiot who screws up a lot, I can tell you it is a blessing in disguise.

Screw ups lead to uncertainty and research suggests we learn best in uncertainty - https://www.aau.edu/research-scholarship/featured-research-t...

Oh god. I did the binary tree tree thing, too, and also in C. We needed a symbol table for a thing, and I assumed symbols would come in in random order. Oops. Someone suggested AVL trees. The reference I used (which may have been Knuth) left delete "as an exercise for the reader." That led to my next big oops: Pondering how to delete from an AVL tree while slicing onions for dinner. Lots of blood. I still have the scar.

"ifconfig eth0 down" on the production bastion host, instead of on my localhost terminal -- and no hands on in the datacenter which was 160km away. Of course the bastion host was the only one not hooked up to remote power reset services.. and only 2 hours left in the service window.. sinking feeling

I once ran a script on production to re-push some old data for a customer based on log entries. This script used the log timestamps to decide which data to re-push. Didn't realize that the timestamps in the log files were UTC, and I just ran it with the default timestamp provided by the library (which is the one the host system uses). Lucky for me, the system's default timezone was also UTC, but nonetheless, the moment I realized it and the 10 minutes it took me to read the documentation and to check the host system's timezone felt like hours.

You live and you learn, I'd say :)

Early versions of the Phoenix frameworks ORM would select every record if you didn't pass it an ID. I didn't know that and wrote a deletion endpoint forgetting to put said ID in. Tests passed (I mean it did delete...) and off to prod it went. Long story short: I deleted data for all our users. Thank God for backups.

I managed to send the entire Google datacenter backbone through one 20gb link in Finland. This did not spark joy.

Search SRE had a 5lb bag of shredded money as a "gag gift" that was given to whomever caused the most recent outage that impacted search ads.

I used to work in Market Research R&D, in a non-technical role as a project manager. I was deployed to a project that tried to use affective computing (i.e emotion recognition) to understand consumer responses to advertisements. It was a total disaster.

We'd hook respondents up to a webcam and record their facial expression as they would watch a series of videos. The vendor's emotion recognition machine learning software would then basically assign scores saying that at this second, the viewer expressed xyz emotion.

The project failed for 2 reasons - one was that the theoretical link between what expressions people were presenting, and their actual emotions to a particular piece of media was not fully proven - which meant the model output was not particularly helpful from the beginning.

Secondly, and this is really important - the model was trained on images of western faces (i.e white people) - and because our target audience - southeast asians - emote very differently a substantial chunk of the output data needed to be trashed (it couldn't process darker faces well, it interpreted a grimace as a smile, etc)

So there you have it - this was something I should at least have anticipated - I got in a lot of trouble

My biggest screw up ever:

http://www.jaruzel.com/blog/dont-screw-up-a-vaxcluster-tale

"I'm an expert because I've made all the mistakes you can in a narrow field."

I've accidentally taken out a television station before. Was remoted in and clicked on the wrong thing with a new platform I was being walked through and exploring. All of a sudden MeTV in the Dallas/Fort Worth area went down in the middle of the afternoon and a LOT of very angry people began calling in but we had it running again moments later. If you are doing a demo of a live enterprise solution -- probably shouldn't click around to see what happens ;)

One of my favorite job interview questions for sysadmins is asking about a time that they screwed up and broke production. If they don't have one, then it makes me nervous. Either they are lying, or they don't have enough experience, or they will be too conservative and will block all progress.

I once forgot that a vendor added their own adjustment to a bidding algorithm we had in place. It was significant for certain regions. I created a bidding model without taking the adjustment into account, pushed it to production, and spent ~30kUSD extra in a few hours before anyone noticed the unusually high bids coming from our vendor. We put controls in place to prevent this afterwards ;)

One of my screw up's was when I was SYSAD (head admin) for the Prime 550 at the UK office of a large consulting engineers.

We had our field engineer in doing a PM and he needed a scratch disk and I said oh you can use xxxx and pointed at the sticky label which had all the disk id's on.

Turns out that some one had been using this for a big GIS project in Amman and ended up wiping 6 month's work

> I now put a mollyguard over those things any time there's any chance of them being exposed and having unscheduled activations.

that's what differentiate a good engineer from not-so-good - they learn on own mistakes!

I once remove a single unused DNS record that resulted in a 10gbit/s DDOS consisting of lookups for “a”, “aab”, “aabaaaa”, “aabaaac”, etc. (Hint: Perl.)

One of my favorite questions to ask technical candidates is:

Tell me about a time you made a mistake that you thought was going to get you fired.

1. Everyone has one. If you don't, you haven't been doing this long enough and I want you to make a couple of those mistakes elsewhere first.

2. If you didn't learn anything from it, you're going to make that and bigger mistakes in your hubris. I'd rather you do that elsewhere.

Every couple years a "failure resume" gets trending on LinkedIn or reddit, and I always love reading the comments.

It's also a refreshing reminder that "just because someone is successful and has a great resume doesn't mean they're flawless".

Resumes and LinkedIn pprofiles are like Instagram posts - enhanced to bring out the best aspects and with enough photoshop/makeup to hide the worst.

A little over a month into my first engineering job, I decided to go for a weekend stroll. I threw my work laptop into my messenger bag on the off chance I'd wind up in a cafe and feel like checking email or poking at some code. Started the day right with a hearty breakfast burrito, popped into the health food store down the street to pick up a couple bottles of local kombucha for later (gotta have those probiotics), and off I went.

After two or three hours of exploring, I noticed something weird: it was a sunny LA afternoon, but I felt something like a drop of liquid hit the back of my leg. I kept walking, but felt another drop, so I stopped and checked. Yep, definitely real and definitely liquid. Also, it smelled like vinegar. Where was it coming from? Who would do such a thing, and how?

Perplexed, I walked on, until my bag started emitting a drawn-out Mac startup tone, and I realized just what I'd done. I opened it up, and sure enough: the seal on one of my kombucha bottles had failed, and its entire contents had emptied into my new work laptop.

I once wrote a new version of a config generator and pusher for a small part of a major service. I knew data pushes were the largest global outage vector at my company, so I wrote carefully conservative validation logic and unit tested it. But I never tested what the caller did when the validation failed, and I had a dumb mistake there. It pushed an empty file, which was worse than pushing the allegedly-invalid config. Oops. That was a ~30 minute outage of the aspect of the service controlled by this config.

Of course an outage is never caused by one mistake. That mistake was mine, so I felt badly about it. There were also mistakes in code reviews, validation in the part receiving the config, and operational procedures. And then the big one: the company as a whole was in this awkward phase where everyone knew quick global pushes were bad but there wasn't good common tooling to support doing staged config files easily. That was the worst mistake behind dozens if not hundreds of major outages.

I was once working on a live MRP server after hours. It was needed to do everything from customer service to shipping to tracking work in progress. So if it goes offline, they would basically have to shut down until it was back.

I needed to reboot at one point and when I did, it started giving me "boot disk not found". I couldn't get it to boot, at all. It seemed the boot disk was corrupted.

I was literally in a cold sweat for 2 hours, late into the night, until I finally noticed that I had left a diskette in the drive which was causing the bios to try to boot from there first.

I have had plenty of other cases where I actually messed something up. But that feeling you get when you think you have irreparably broken something is so terrible.

Some of mine

Was testing code and pushed a file to FTP 2 days early... vendor picked up, processed file.. the people who signed up in the next 2 days were in the file pushed later... but the vendor already processed the earlier file so they didn't get their metro cards that month

Somehow managed to rebalance underlying components for a Trendpilot ETF monthly instead of quarterly... daily audit that compares the values on NYSE vs in our DB caught it.. lucky for me there was no money in it yet

dropped a table once at lunch time right before taking a bite of my sandwich... did restore table within 10 minutes , didn't eat lunch that day ... lost appetite...

In ETL tool hardcoded something to test.... left it there when running for real

Luckily I learned from a young age. When I was 7 or 8 I was using the computer my dad used to run his company. It had dual 3.5" floppy drives, and a new (to me) hard drive. Needing to format a floppy, I opened the format utility and for some reason I thought I should chose "hard disk" (because it wasn't floppy?!? hmm) when prompted.

So I format the "hard disk" and for some reason my 3.5" wasn't formatted. So I tried again and again to no avail and gave up.

The production manager came in to work Monday morning to a fresh hard drive. Some things were backed up and some things had to be recreated.

The outcome of this necessitated learning a new skill: bypassing passwords.

> I was home alone as a kid, watching some movie on TV. I saw some guy grab a beer can and do that thing where you jam a pen in the side to make a hole, and then crack open the top.

Given what I think is her age (judging from using C64s and whatnot) I'm going to go out on a limb and guess this was "The Sure Thing" [0] with John Cusack and Daphne Zuniga. It's a great movie if you haven't seen it.

[0] https://www.imdb.com/title/tt0090103/?ref_=nv_sr_srsg_0

One of my favorite recurring team conversations is the one where everyone shares stories of the outages they've caused or the systems they've broken. This conversation has happened eventually on every SRE (sysadmin/PE/devops/whatever) team I've joined, usually when a junior team member causes their first outage and is having an emotional meltdown. I remember my own meltdown of that form, and I remember it helped hearing about the terrible problems my friends and mentors had caused in their turn.

The first outage where I thought I was going to get fired: I was working on a system that had a single-point-of-failure server, and through a mishap with rsync I accidentally destroyed the contents of /etc. That SPOF also had no backups. (I'm not claiming it was well-designed...) Thankfully the job that depended on that server would not kick off until morning, so my team slowly reconstructed its functions on a separate machine and swapped it in behind the scenes. I helped as much as I could while vibrating with anxiety, and my team was incredibly kind throughout. I was not in fact fired. :-)

The most recent outage I caused? Yesterday! I accidentally rebooted most of the machines in a development cluster. It's a dev system, there's no SLA, on the whole I don't feel horrid, but it definitely ruined a few people's work for an hour. This morning I spent a few minutes putting in a guard rail to prevent that particular mistake again...

If you're in this job long enough, everyone breaks things -- it just happens.

Adam Savage and Matt Parker recently had a conversation that spent a lot of time covering the topic of "screwing up" and how we should respond when we do (Matt's new book is about math screw ups that have had real world consequences). It's a great interview in general, in my opinion.

https://youtu.be/ig-2xlXfex4

In a major production launch, we moved traffic between two versions of a backend with a blue/green deploy. The new version was hosted on Kubernetes, and I was pretty new to using it in production. The changeover went well, pretty great, actually. The problem came up the first time we deployed to the new infrastructure - We saw a huge spike of connection disconnects. We did not get a good answer why at the time, except the vague sense that the deployment had gone a lot faster than we intended.

The second time we deployed, I happened to glance at the deployment size immediately after deploying. For about five seconds, our deployment size went from 100 down to 2. The reason for this was simple: The "Replicas" count was specified in the deployment spec, and it was set to the size we used in our staging infra. That had been fine in prod, and was quickly overridden by our autoscaling configuration, but it did cause the Kubernetes infrastructure to take down every existing pod (minus two), then bring up a bunch of new pods very quickly.

The true measure of experience is the depth and variety of our screwups, and the quality of ones character illustrated by what we take away.

One thing I really pride myself on is that because I screw up so often I have a really good intuition for how things get screwed up.

So then there was the one time I was engaged in sysadminery and had my cow-orker sitting next to me while we were trying to debug some issue. She says, "Hey, is there anything useful in the README file?"

I immediately typed "rm README" and hit enter.

Then I crawled under my desk and wouldn't come out until we'd gotten the file restored from backups. Naturally, it had no useful information in it.

Then there was the time, for no readily apparent reason, where I typed "DELETE * FROM table" (in the dev database). Fine, I thought, it's time to go home, and submit a request to get the DB restored.

It turns out that they kept one (1) day's worth of backups, which they took at 6:30pm or so. I submitted the request at about 6:00pm and the DB guy had already gone home; he did the restore about 7:00am the next morning. Yes, he restored an empty table.

In another comment, I pointed out a mistake of mine that was a major factor in an outage.

I also screw up all the time in ways that would cause outages, except we have automated tests, tsan/asan, code reviews, a staging environment, various safety checks, experiment gates, pre-mortems, slow rollout procedures, an alert on-duty SWE and on-call SRE, etc.

Today one of my mistakes was caught early in the prod phase of our push. That's much later than I would like but still before it did any real damage. I submitted the bad code last Wednesday and have been out sick with the flu (and caring for my preschool-aged kids) since then, so my awesome team handled my problem for me.