We had a cloud deployment system that was also responsible for configuring Nagios instances. The two processes (Java and Python) that were responsible for the Nagios configuration communicated over XMPP. The Java process could only be restarted once a week because we guaranteed a certain uptime to our customers.
One day I decided to refactor the way it worked, and this entailed a change to the api. I began with the Java code, but when I got to the Python parts, more important work came up so I left my code in a branch and forgot all about it. When the next release came up a coworker started to merge all the new features into the new release branch. He asked if he could merge my stuff, and as I had just finished some new features in another branch I said yes. Two days later the new half-refactored, untested and very scary Java code was live in production without my knowledge.
An hour after the release the first alerts of nagios syncing problems came in. We scratched our heads and soon discovered that we had merged too much. Now we had to choose: roll back or roll forward. To roll back we had to take down the application, restore a backup, synchronize a lot of other connected applications. After checking whether the Java code that was in production now actually worked and spit out sensible messages (it did, surprisingly) I decided to try and create the rest of the Python implementation right away.
So with my headphones on and a big NO sign on the door I started programming and created what would normally take one or two days in just over an hour, including the release to production. The good news was that we had had no customer downtime of the Java process and now had a much cleaner Nagios configuration system, the only bad thing was that the alerting system had been down for about 2 hours.
The situation was pretty fucked up ;)