Hooray! It’s finally here!

We’ve been celebrating all month long, but today is the official SysAdmin Appreciation Day.

The last challenge of the month was to tell us about your proudest SysAdmin moment, a time when you were the hero. We had some great submissions, but our winning story tugs at the heartstrings and really speaks to what SysAdmin Day is all about.

Winner Patrick Farley is the SysAdmin for Florida Drug Screening Inc., “a national provider of drug testing and alcohol testing and specimen collection services; including DOT, FAA testing, Drug Free Workplace policy development, education, instant on site drug testing devices and At-Home drug testing.” He tells of a week when he was flying SysAdmin solo, and got slammed with a series of unavoidable issues. He overcame all, and in the end got the recognition he deserves.

Happy SysAdmin Day to Patrick and all other SysAdmins. Today is your day, so enjoy it.

-Team OpenDNS

PS. Some of you asked us to give your boss a gentle reminder about SysAdmin Appreciation Day, which we did gladly. 🙂

In Patrick’s own words:

First off, I’m here alone all week. I’m the only person in the Tech Department. So naturally it has been a busy week, but I didn’t mind that so much. It’s just hard to get a lot done when I have to stop constantly to help someone track a problem. Alright, no big deal, that’s what I’m here for. What bothers me is that Monday and Tuesday everyone took the time to come to me for something, but no one had the two seconds afterward to say thanks. That’s ok, anyone in IT has had that “not appreciated”. No big deal. I have my iPod and my servers. They love me.

But then there was Wednesday. I got an email at 4am saying that a particular machine had failed to reboot after an update. Now if it were any other server I’d be dragging myself into work right then and there, but this one isn’t so critical, so it was safe to let it sit until I went into work, which worked out because I wasn’t exactly eager to get out of bed. Little did I know, the reason that server went offline was not because of an update but because of a power failure. And a really nasty one at that. Apparently the transmission lines near our building was host to an epic battle. “small animal” versus Power company. The power had gone out and stuff ran and ran until the batteries died. So I received that call as I was getting ready for work.

I arrive to find everything off. I shuddered. All my Web Services aren’t going anywhere. My websites are all offline, and I’m sure if the phone system worked, I’d be fielding calls about not being able to connect to our SOAP server or our website. Management decides to send the staff to breakfast and have a meeting about current issues. I stick around in case things come back. We get a call from the Power Company saying it will be 2 hours and people are on sight working on it. Alright…nothing I can do. I look at the vice pres and say “I’m going to starbucks…want anything?” He declines and I go. At starbucks I have a triple white mocha and I sit and drink it slowly. In all of the havoc it was rather nice to relax. I get back to the office and the power is still off. I go in and unplug equipment and get it ready. As the power comes back I insisted to my upper management that things must be booted in a certain order and it will be about another 30 minutes before people can log on.

My domain controller didn’t shut down gracefully. I nearly cried and I watched it try to recovered corrupted data. I was sure I’d have to restore from a backup. After about 20 minutes of that, it did recover itself and boot. Ok going good. I booted my SQL machines. Good. Booted webserver. Good. Booted fileserver. Works! Then I went and got all the machines on and people logged in. It was close to noon, but we were back online. Oh and now that the phone system worked, I spent about 30 minutes on it with client explaining to them.

This was my Wednesday. So now for Thursday..

Yesterday at about noon our ISP decided that a service which we pay for to be able to send emails a certain way would no longer be offered. So yes..at noon our email stopped dead. Took me about 5 minutes to figure it out, and another 15 to get on with their tech folks and find out what was up. It should be said that we were never warned in any way that this was going on.

Also I looked at our bills, they may have forgotten to give us notice, but they didn’t forget to bill us for a service they no longer offered!

So…what can I do?… I used the only option I could for an interim situation. I run a mail server at my house and have dedicated net access so I can host for a few customers of mine and of course my own email. I remote to my mail server and configure it as a relay from my work IP address. I then configure each person here to relay off my personal equipment. So now out entire business email is dependant on a server in my closet (in case you think I’m a geek..there’s 5 servers in there, so yes..you are correct).

And of course there were the other things. Our custom software was having issues so I had to figure that out because our programmer was..yes on vacation. I had to pull specific data from a database, and our DBA is..you guessed it, on vacation. I should mention that I’m neither a programmer or a dba. Three cheers for Google.

Throughout all of this, no one seemed to care that I personally solved these problems. No one cared that I’ve been saddled with all these things that are just nuts. Normally everything is quiet here. Even still I felt like no one cared.

But yesterday as I was leaving I stopped in downstairs to check things before I left. Out of nowhere a girl here Nicole (whom is normally very very shy) stood up and said to everyone there that I should be recognized for having such a crazy week, and that I’ve handled myself very well, and remained calm (glad it seemed that way, because in private I was cussing like a sailor). But she voiced that and gave me this nod that seemed to say that she noticed, even if no one else did. It was a rather wonderful feeling. Someone did care. Someone did notice. I come in everyday and do what I love, and someone, even if just one person saw that. I went home and for the first time in a long time, I was just in a great mood.

So here I am today. I wasn’t going to submit anything, but after yesterday, I just felt like I should. Not so much for me though, but more for Nicole.

Everyone at work does a job that’s important somehow. So this is for Nicole and all the others around the world who recognize why your sysadmin is important, but also for the people who realize that everyone is important.

We all work together to get a job done. I may have looked like a hero this week, but when that big account got landed that wasn’t me. When customers have a problem and someone gives them great customer service and get a letter of recognition from a client, that wasn’t me. We’re all part of a larger group. At work, in life, in the world, and we should know that each of us has their place.

Here’s to a calm Friday.

This post is categorized in: