OH OH OH – Sysadmins and Christmas: What Could Possibly Go Wrong?

“We don’t see things as they are, we see things as we are.”  wrote French diarist Anais Ni. This doesn’t mean that bad things don't happen to you; it simply means that they occur in a random fashion as part of everyday life. In the same breath, however, your assumption is correct: everything that can go wrong, will go wrong.

While we’re sure you have been busy pulling endless hours, answering frustrating questions from your non-IT colleagues and managing people who just can’t seem to remember their passwords for the life of them, we know that heading home for the holidays may not be the complete break from work you are hoping for. You’re a sysadmin and every year you’re dreading the inevitable phone call from your boss, which typically comes right as the festivities are about to begin.

What Could Possibly Go Wrong?

The question “What could possibly go wrong?” should be on the lips of every decent, considerate, sysadmin. A good one is thinking about it all the time:

• when planning systems
• in the heat of an emergency
• in security design

When planning a system, always allow for the fact that mistakes (whether human or technical idiocy) will occur. The system must be able to withstand these challenges. A brief outage shouldn't cause a complete service shutdown. What can be done from the outset to prevent some employees from having their "special moments" and with their thoughtlessness, torpedoing the work of any sysadmin, no matter how circumspect? In the middle of an outage, when you are about to take some remedial action, you can think about whatever you do could make things worse, and decide whether the risk is worth it. Sometimes you can’t possibly make it worse, which might almost feel liberating.

The Call from your Boss this Christmas Holiday

Now let’s take a look at the upcoming Christmas holiday: Is the call from your boss really as inevitable as you might think? Perhaps you have been awoken out of a deep sleep at this time of year before because of unexpected network problems, or you may have been interrupted by the shrill tone of your cellphone during the holidays. The role of a sysadmin is not 9-5 job, but rather 24/7/365. Do yourself a favor and try not to place these events into a job description. If you do, you won’t enjoy your career. This phenomenon is known as “selective perception," and we all have it, whether we realize it or not. It’s the process by which we select, categorize, and analyze incidents from our environment to create meaningful experiences, while blocking out stimuli that rather obviously contradicts our expectations.

Objectivity Takes a Holiday at Christmas

The fact that you can't remember as clearly the countless times when the phone didn't wake you up late at night or interfere when you were on holiday makes these missing negative events less interesting or worth mentioning from today's perspective. This could also be because you've decided to believe something that lacks objectivity: that your job is a string of inappropriate calls that interrupt your sleep, your well-earned vacation and Christmas festivities. Would a statistical assessment make any difference? One thing is certain: In periods of low network load, and especially during the holidays, network problems are a generally unlikely event. Reliable downtime statistics of companies are few because which company would like to publish such numbers? From our many years of experience, however, it can be said quite clearly that there should not be an accumulation of network problems during the Christmas season. A quick look at Google Trends confirms this. What does the helpless user look for in the Christmas and holiday season, if he cannot call his admin directly? "Network down" is certainly a popular search term - and according to Google Trends, there have been some peaks in search behavior in the US during the last 12 months; the typical holiday periods were all unaffected.

But of all things, our fears that there might be network problems during the holiday season, make them not quite so implausible, at least in the context of our subjective perception. Now add the Law of the Worst Case into the mix: The more important the data, the more likely it is to go missing. The older the data, the more likely it is that a copy is corrupt and that one of the tapes is unreadable. Murphy’s Law, of course, stipulates that this will be the tape that holds the necessary information.

Plan for the Worst Before you Celebrate the Best of the Season

If you have internalized the astonishingly simple principle that everything that might go wrong, someday will go wrong, then nothing will surprise you. When preparing to take some time off during the holidays, think about how events during this routine change could surprisingly go wrong, and how you will recover if it does. Having at least a loose idea of how you could back out from any given situation is much better than having to completely make it up during an urgent scenario. Have a plan for even the most likely events and those, that, by all odds, shouldn’t happen. This Christmas use “What could possibly go wrong?” as your personal mantra, and maybe also invest in some good stress relief techniques because you've heeded all eventualities. Keep in mind: it is the job of a good sysadmin to always consider what could go wrong.

Prevent what is Generally Unseasonable

In case you're in danger of being hit by the full force of improbability just when you least expect it or fear it the most, you now have a few days to find a solution that'll let you know what might go wrong before the boss reaches for the phone. If you are not yet using & loving PRTG, we have good news: it isn't impossible to prevent what is generally unseasonable. Happy Holidays!