Rango's Dog House

12 straight hours of debugging a production data issue and I finally had to excuse myself and call it a night.

I started by myself at 6:30 AM. By 9:00 I was hosting a meeting with another developer and a handful of managers. Also a few business people along with their confusing portfolio of queries, all manner of broken. This is a red-alert, all-hands-on-deck problem that is time-sensitive because it’s regarding year-end data.

It’s not right, that data.

So all day I’ve been debugging with people looking over my shoulder as I type and switch tabs and jump around, with managers asking questions and everybody speculating and expressing their concern and exasperation. At the start of the day the assumption was that I broke something spectacularly back in August. By noon we were onto something else, chasing data through linked tables and stored procedures that were created 20 years ago and haven’t changed in 10. By 3:00 we discovered the root cause.

By 6:30 PM we finally had a solution carved out in our test environment. I had made dozens of changes to jobs to simulate rolling back the clock so the year-end process could be run again. We were just about to pull the trigger on the offending jobs when I abruptly had a moment of doubt. I concluded it was unwise to proceed.

We’re not really going to do this in production tonight, are we?

I then explained why I thought it was a bad idea for me, specifically. Mental fatigue is real. 12 hours in the chair, no food, no breaks. But I suspected I was not the only one and I was right. Everybody was struggling to keep up with the details of the diagnosis, much less the workings of the fix. They were looking to me and my engineering partner to push the button and have a month’s worth of problems corrected without overlooking anything or introducing any side-effects. Nice work if you can get it. And what if something does go wrong? We’re potentially looking at another 12 hours.

So I called it. Not that I have the authority to do that. But we had consensus that fatigue was the deal breaker. It’s an emergency, but it’s not <that> kind of emergency. It’s a business emergency. It can wait until morning.

This is the senior part of senior software engineering. I’m old and tired. In my thirties this would not have been an issue, because I was young and foolish and frequently felt immune from fatigue due to my bipolar disorder. And I probably would have pushed to production, sunk the battleship and lost my job. Not today, rabbit. We take our medicine here, and observe doctor’s orders.

As for tomorrow? Thoughts and prayers. Good vibes and juju. Just remember to ask for a raise when this is all said and done.