The Data Center: Where the Stuff of Nightmares Lurks

The data center can be a breeding ground for trouble–from new technologies that require power demands IT administrators can’t afford to service providers that cause more harm than good.

I fear upgrades. They always seem to work flawlessly in test environments, but no matter what we do, pushing them out to production environments always seems to come with hiccups. But even though I walk in the valley of the shadow of silicon, I shall fear no evil.

My greatest data center fear is water. We have water lines, condensate lines and plumbing drain lines that run directly above our server and network racks. Water in the data center is not good, trust me. Two jobs ago, there were water chilled HVAC units located directly above the data center. They replaced them. However, they never shut off the water to the old HVAC system (thinking they could use it as a backup in the event of an emergency). Well, one day, the seal on the main valve went bad, and hundreds of gallons of water poured down on top of our Nexus switch rack. Talk about an absolute nightmare. On top of the water hitting that, the power was run under the raised floor. Scary times.

My greatest IT fear (and fear is general) is clowns on backhoes digging all around our facility looking for that undocumented fiber or twisted pair. (Shudder.)

Natural disasters are especially frightening to me as an IT pro, whether it be hurricanes, tornadoes or, particularly, earthquakes. I’m both curious (and not interested) in finding out the technological ramifications an earthquake would have in our data center and the subsequent ripple (no pun intended … OK, pun intended) effect throughout the organization. Not just server racks, but the smaller stuff, too, like all the creative ways an earthquake would kill spinning hard drives. (Though, if racks are just falling over, hard drives would probably be the least of our worries.)

Security breaches—like for so many, I’m sure—are particularly nightmare-inducing for me. Especially if it involves a human on our end. (“Wait, I thought those emails were from the CEO.”) A breach could be going on for months, and it’s possible we wouldn’t even know it.

My worst nightmare is the motherboard on the server blowing up beyond repair and it taking days for the parts to come in. Everything would be down, and everyone would be screaming at me to fix it because they couldn’t get anything done.

My leading fear is our monitoring software either taking an unscheduled dirt nap for some reason or otherwise becoming unavailable. Flying blind in this day and age is a scary proposition. It would indeed be a dark day.

Aside from my technological nightmares, my biggest fear is a server fan eating my beard. There are some devices out there that have massive fans, and I’ve come close to being eaten alive a few times. Aside from the immediate pain, the call to support to get a replacement fan would be quite awkward.

We have our entire VDI environment hosted on one server. My greatest fear is the motherboard failing, sending us into a post-apocalyptic state (OK, maybe that’s a little dramatic). At least I could say, “I told you so,” on the way to the unemployment line.

I fear people and what they’re capable of. The human factor in IT is the most unpredictable and the hardest to account for. Too often, end users know just enough to cause problems. I can build a redundant data center, but if end users cannot or will not set up proper disaster recovery systems … I can give a data center redundant power, but if a truck runs into it … I can build a system that has multiple checks and security lockdowns, but if one careless or overworked or misinformed user with sufficient access clicks on that email link with the brand new virus.

One of my greatest fears (brought about by an actual experience) is another absolutely pitiful on-site help desk tech who completely disregards procedures and starts moving cables around on network devices trying to get one device working again, but takes down six more in the process. (Oh, and it indeed was that one device misbehaving, not the network.)