The 4 Biggest Mistakes in My IT Career
Dec 21, 2018 • 10 minute read
Before I decided to mess up Paessler's online marketing team, I spent almost 20 years as an IT admin. At the end of the previous millennium, I enjoyed a classic apprenticeship as an IT administrator (back then, the profession was still called "data processing manager" in Germany). Following that, I worked for many years in an IT system house and later in the internal IT department of an automotive supplier.
Just in the almost 15 years I spent at the IT system house, I came across countless funny stories worth telling; from funny mistakes by customers to my own misadventures. Probably every admin has experienced that sudden adrenaline rush when the wrong hard disk is formatted, or the critical production server does nothing other than stoically display blue screens.
Since you should never be ashamed of your own mistakes, today I reveal four very personal FUBAR moments of my previous IT career. Maybe you will see yourself in one or the other worst-case scenario. Let's get started...
1. The Bent Processor Pins
It's the year 2000, an era in which servers are still assembled by hand, and the Pentium III processor is the top product in processor architecture. I am in the process of equipping a motherboard of a future small business server with the necessary plug-in cards and cables. To protect the processor board, there is always a blind board plugged in by the manufacturer, which I confuse with an already-installed processor. I hold the actual processor, a Pentium III with an incredible 800 MHz, in my hands and place it on the top of the built-in power supply. After all components have been successfully installed, I try to close the lid of the case. The mechanism is surprisingly stiff. As I exert more pressure, the lid snaps into place with a scraping noise. At that moment I remember the processor that I had placed on the power supply. My fear is confirmed - after removing the lid, the processor is still there, but about 20 of its pins are now bent by about 90°. The processor is a piece of scrap metal, and I feel as if I aged 10 years within 2 minutes!
iIf you want to walk down memory lane here you can find information about the Coppermine technology of the PIII.
2. Mixing Source and Target When Cloning Hard Disks
Well, this story is quickly told, but it can still be one of the biggest mistakes a sysadmin can make. In those days, when you booted Norton Ghost from two floppy disks to make a 1:1 copy of a hard disk, and the menus were two-color and text-based, duplicating media was a challenge. I had a computer with accounting data in front of me, and was supposed to make a data backup by creating a hard disk image. The target disk was an identical model, so both disks were displayed with identical values in the menu of Norton Ghost. Only the jumper setting (remember when you had to put jumpers on the disks?) was different. You can imagine the rest: I happily copied the target hard drive to the source hard drive, and all the data was lost. The hard disk had to be sent in for a professional data recovery.
Lesson Learned: When duplicating source and destination, always be careful not to swap media. It is better to use a hard disk with different specifications to recognize the difference directly. Ideally, save the most important data in the system in advance. On the other hand, this problem hardly exists in modern times, since the backup and image methods work much easier nowadays.
3. 110V May Not Be Equal to 230V
Honestly, I don't exactly remember the year anymore, but it could have been 2006 or 2007. A customer in Germany airlifted a Unix server from a subsidiary in the USA. It contained important ERP data that was to be synchronized with the German branch's server. I was on site as a service provider to put the device into operation. In theory, this means: removing the packaging, plugging in the power cable, network cable and keyboard, switching on, performing a function test and then returning to the office.
Well, since that day I know how big a flash flame of an HP-3000 Series is, if you fire it with 230 Volts instead of 110 Volts!
I had simply forgotten to toggle the dip-switch on the power supply, which provides either 110 or 230 volts to the power supply. The server was already very old at that time, and it was not easy to get a replacement in an electric shop around the corner. A drive of about 1,200 kilometers and a considerable amount of money later, a system house had given me a device of the same construction (with a functioning power supply). I could exchange the defective parts and the customer could keep to the scheduled downtime for the adjustment of the ERP data.
A small cause with a huge effect in the category "senseless errors". Since then I search each power supply for any switches and think three times before I connect a power supply to the current.
4. Gave the Admin Password Over the Phone
What we all know today under the term Social Engineering was already a popular manipulation method in earlier years. I was a fairly new employee in a company, and a customer call was put through to me. Normally the customer was serviced by my colleague, but he wasn't in the office that day. At the other end of the line there was a noticeably angry man who urgently needed the admin password of his company. After my objection that I basically don't give out the admin password to unauthorized persons, especially not on the phone, the man pretended to be the boss of the company and built up pressure. If I didn't give him the password, he would inform my supervisor that there was imminent danger and that I was blocking critical work for the company through my negative actions.
Now the mistake is coming: of course I surrendered and gave the password. I didn't insist on clarifying the matter internally and get back to him in five minutes, but simply gave the administrator password to a stranger on the phone. Was that naive? Yes. Was that stupid? Definitely. Did I get the lecture of my life from my boss? Absolutely! Will that happen to me again? Never!
Remember kids, never, under absolutely any circumstances, ever name sensitive passwords on the phone. No matter who is at the other end of the line.
Sascha Neumeier, blundering system administrator
There are countless better ways to transmit a password than naming it over the phone. Unless you're on the phone with a nuclear power plant manager who needs a password to shut down a reactor just before nuclear fusion, nothing can be so urgent to prevent having five minutes to clarify the caller's authenticity and get clearance to transmit the password.
Just for the fun of it, test your colleagues on how easy it is to give you a password. You'll be surprised how lax the handling of sensitive information outside is in practice! Be sure to inform your colleagues and staff regularly (!) that there is a reason why passwords should be secret!
You Learn From Your Mistakes!
Summarized in a few lines, this somehow reads as if I had no idea about my job! Mind you, these were four situations that I got stuck in my head in 18 years as admin, and in which I really sweated blood. Especially with the story about the server processor, I was afraid for my job. I was in my early 20s, had little professional experience and knew that my company at that time had no insurance against such employee stupidity. The damage was covered by the company and nothing was deducted from my salary. Instead, I was encouraged to say that this would probably not happen a second time in my life. And indeed: Since that day I have been painstakingly careful not to pin things down anywhere. Be it the screwdriver in the PC case (processors are no longer changed by hand), the oil filler cap in the hood of the car or fingers in the door. 😉
Share Your Stories
Now that I have provided such an honest look into the aberrations of my professional life, I can hardly wait to hear your stories. What misadventures have happened to you in your IT career? Write to me in the comments!