Sleep Deprivation

One of the interesting things in working at a highly-technical high-stakes high-pressure 24/7-uptime mass-market company is you get some pretty big fires to fight. The last 48 hours are a blur, as I spent most of them in the semi-awake state. I’ve had some sleep now, so I can look back at it and laugh.

I had an ordinary busy day on Tuesday and on top of working late, stayed at the office until about 10pm, trying to work out my taxes and other such personal chores. Around midnight I was just getting into my pajamas. I was dog-tired and eagerly looking forward to crawling into my warm bed and dozing off. My phone went off — RED ALERT. Something wonky was going on with our tournaments and while tournaments were starting, and happily collecting people’s registration money, it was failing to sit them down at actual tables and dealing them cards.

Ruh-Roh.

We quickly gathered diagnostics and discovered the cause of the problem. Within a few hours we had a fix for the problem, but had to do extensive testing through the night to make sure it was safe. When you have software that is pushing millions of dollars around and is concurrently serving 50,000 people at any one time, you can’t just hack something together and toss it up on the server. It’s a major task to deploy, and requires careful handling. The other issue of course was cleaning up the huge mess made by the problem. All those players had to get refunds and all those wacky tournaments had to be cleaned up. That was a big task itself. Deploying a new version of the site is always a big job requiring a lot of coordination between many people, and we finally got things deployed after working more than 24 hours straight on the issue. Since this all kicked off right before my head was about to hit the pillow, I ended up being awake for over 40 hours straight.

Towards the end of the stretch, when we were all getting a little loopy, it got kind of fun. I’m not sure if our wise-cracks were getting more clever as we got more sleepy and deranged, or if we just found them funnier. It was a stressful time, but the team came together well. It’s not exactly like getting shelled or nebelwerfered for two days, but our little band of brothers did well under fire.