No Holiday for IT

IT stands among the ranks of vital professionals—healthcare, public safety workers and government—for whom evenings, weekends and holidays are par for the workplace course.

However, without the glamour associated with saving lives, restoring heat to freezing homes or guiding people through the sky so they can be reunited with their loved ones for the holiday, few even realize that IT is keeping the lights on.

For IT professionals, the biggest holidays of the year are rarely a cause for celebration. Systems need to be upgraded when the office is shut down and all too often, servers partake in a Murphys Law, going haywire when there are the fewest people around to restore them.

Resource Library:

Almost every long-time IT professional has at least one horror story to share.

"Back when I lived in Australia—my story is from an Easter weekend—which was the only weekend that the network could be shut down for long enough to restructure things," John Terpstra, an IT professional in Austin, Texas told eWEEK. "Of course, it was stinking hot outside, about 91 degrees. The door handle to the server room was so hot, I couldnt touch it. Inside, it turned out the air conditioning had broken, and all of the servers were still running, but from the moment I walked in, they started dying, one by one."

To read about five steps to better job security, click here.

Of course, Terpstra continued, none of the people who needed to be contacted could be, and while his job had not been to reorganize the hardware, this is what he ended up doing. Out of 24 servers, only five survived the weekend, and each of those died within three months of the incident.

Kevin Behr, the chief technology officer and managing principle at Assemblage Pointe in Lancaster, Penn. could commiserate, having his own story from when he worked for a large retailer a few years back.

"One of the rock star programmers wanted to leave a day early for the Thanksgiving holiday, having gotten all of his work done," Behr said. "Not realizing there had been a change freeze at the end of October, as the company did 60 percent of their business in the last two months of the year, hed put a lot of code changes on their site. Nobody knew that he did this, and when the operations guys who were left rebooted their servers, there were literally no items left for sale on their site."

These IT professionals ended up spending the days before Black Friday trying to figure out why they had nothing for sale on their site. They had to restore the site from the backup, and literally worked three days straight through the holiday weekend to get the site back up, Behr explained.

Of course, the biggest and most famous holiday weekend in which nearly every IT professional in the world was on the clock was the Y2K turnover.

"It was a horrible experience for all of us in any kind of finance," said a former independent consultant and IT specialist in New Jersey who asked not to be identified. "When that COBOL clock would turn over, we all had to be there in case someone didnt code it correctly. When midnight hit and everything was okay, we still had to watch the screens for four or five hours, while the clock turned over in different time zones. "That was how we spent our New Years Eve, while our friends were drinking champagne at parties."

These stories underscore the sadness that can accompany professionals, stuck fixing IT disasters while the rest of the world is out gallivanting.

Bill Light, an applications development and technical expert at the Permanente Medical Group in Oakland, will never forget one Thanksgiving weekend that stood out over two decades he worked at the San Jose water company. His team had a mainframe conversion to do that could only be performed over the holiday. They didnt finish until Monday morning.

Page 2: No Holiday for IT