eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Ghosts and goblins, haunted houses, and smashed pumpkins—sometimes the most harrowing horrors are only steps away from your cubicle or office.
Just ask Steven Calderon.
Calderon was in his second week working as security guard for Frys Electronics when Anaheim, Calif., police walked in and arrested him for crimes including child molestation and rape.
Frys had requested a background check on Calderon, which was done by The Screening Network, a service of ChoicePoint, the $1 billion-a-year data broker based in Alpharetta, Ga. When it came up with criminal warrants and felony charges, nobody—not Frys, not the police—stopped to ask if the data supplied by ChoicePoint was accurate.
Calderon spent a week in jail for crimes he didnt commit because an identity theft report hed filed in Norwalk, Calif., in 1993 wasnt connected with the criminal files that were created in his name.
He went to jail for an IT error.
While, fortunately, not all IT disasters are of these cataclysmic proportions, every weathered IT professional has an all-too-real eerie tale about a day when everything went wrong. Even 10 and 50 years later, these pros retell their horror stories with a startling sharpness, more haunting than ghost stories.
Invasion of the Inept VP
John Mitchell (not his real name), a senior support analyst with an insurance company in Madison, Miss., like many worker bees, had his most chilling IT calamity at the hands of a higher-up who believed he had it all figured out.
“A senior VP, who fashions himself a programmer, decided to install the latest version of Visual Studio 2005 before we, the IT staff, had a chance to test it. Once installed, his code no longer worked. His solution? He uninstalled a previous version of Visual Studio,” said Mitchell.
Peter Coffee looks back at almost 25 years of IT screwups.
Click here
to see the slide show.
Mitchells IT group came to find out that he had not one or two, but three different versions of Visual Studio on his machine.
“Yes, we should be able to control this since were IT, but weve been warned against stifling a users creativity, which loosely translates into Hes a VP. What are you going to do, tell him no?”
Adding insult to injury, the VP put in a Severity 1 trouble ticket, which was supposed to mean that he was incapable of continuing the duties of his job until it was fixed, when it was really more of a Severity 3. Mitchells group, however, had no choice but to drop everything and make this machine their first priority.
“Our programming staff attempts to decipher his code but cant because it seems theyre not trained in early Egyptian hieroglyphics,” said Mitchell. “It then gets transferred to me. I turn his machine inside out trying to make this thing work. I uninstalled and reinstalled every VB application on his computer multiple times, manually cleared the registry, Googled every variation of broken Visual Studio 2005 I could think of, lit a few candles, said a few prayers, but nothing worked.
“Meanwhile, the VP, who took the day off, made sure to phone every 2 hours for updates and to offer suggestions. For two solid days, I monkeyed with the problem, but couldnt fix it. His code continued to crash.”
So what happened? Mitchell came in to work on the morning of the third day to a voice mail from the VP who was driving him up the wall, in which he said that while he had been at the office that weekend, he realized that hed mistyped a character in his data collection string. Once he made the change, the code worked perfectly.
“You can close the ticket, he told me,” Mitchell said. “Fortunately, my office has double-insulated walls; if not, Id be telling you this from the unemployment line.”
Next page: Dawn of the Dismal Data Conversion.
Page 2
Dawn of the Dismal Data Conversion
Stuart Robbins, founder and director of The CIO Collective, a nonprofit association of senior IT executives providing strategic guidance to emerging businesses, and author of “The System is a Mirror” (John Wiley & Sons, 2006), had his worst professional nightmare come true almost 10 years ago, when his group attempted to migrate from a very old to the then-current version of Sybase.
“We were attempting to leap six versions [4.x to 11.x] because our third-party vendor promised it was the only way to maintain the viability of our Scopus data, which depended upon the Sybase upgrade,” said Robbins.
So, just how badly did it go? Cataclysmically, it could be argued.
“There were so many problems that we needed a triage team dedicated to resolve issues that we had caused by the work on the previous day, and after delays of more than a year and many hundreds of thousands of dollars over budget, we backtracked and moved the data to SAP,” said Robbins.
Robbins added that in almost every nightmare project hes heard about from colleagues, custom data conversions, such as his Sybase disaster, were the root cause.
“They never work and should be outlawed,” he said. “No matter what the vendor promises, major vendor changes involving critical corporate data should simply be re-entered by hand using inexpensive Kelly Temp resources.”
Night of the Living Zero-Divide
Jerry Luftman, now an associate dean and distinguished professor at Stevens Institute of Technologys School of Technology Management, was a young IBM systems engineer supporting one of the major television networks during the 1976 Carter-Ford presidential election when disaster struck.
“Like today, even in ancient times, everyone was glued to their televisions (yes there was color, but we did have to get up to change the channel) watching the returns unfold. The networks competed heavily on who could predict the outcomes the fastest and the most accurate,” said Luftman.
Suddenly, at the network where Luftman was supporting, each of the three mainframes—one primary and two backups—within 5 seconds of each other crashed at the peak hour of 9 p.m.
“At first, the team from IBM and the television network looked at each other in fear and puzzled. We then looked at the TV monitor and saw that our [television] star did not skip a beat. They kept looking at their terminal, and without any sign of concern kept relaying the status of the election,” said Luftman.
As the stars were reciting the election results from memory, Luftmans team scrambled to find the cause of the outage, frantically trying to figure out how to get the systems back up quickly.
“Within 5 minutes the core dump was printed and we were analyzing the problem. There was a local election in the Southwest that had a candidate running unopposed. Did you figure it out yet? There was a zero divide, and the systems just could not tolerate it,” said Luftman.
As the clock still ticked and the broadcasters continued to present the results, someone came up with the ingenious idea to add a fictitious candidate to the local race and give them one vote.
“It worked. The three computers came back up, and the broadcaster had current information on their screen. … And we were the first to predict that Carter won, and we were the most accurate.”
Next page: Exchangergeist and The Nightmare on Vendor Street.
Page 3
Exchangergeist
Mike Murphy, an IT administrator at the Deerfield, Ill., construction firm Meridian Design Build, met his most cursed job at the hands of a new Exchange Server.
When his company was bought out by two of its vice presidents who were starting a new company, Murphy received orders to convert and transfer anything and everything in the IT department that would be needed for the changeover… in less than two weeks.
Things were going swimmingly as he began to get all of the workstations up and transfer all of the data over to the new server theyd purchased, but when he set up the new Exchange Server, everything went amok.
“I knew it was going to be a huge problem because of a variety of factors. For one thing, the database had never been defragmented offline before, so it was massive and slow (about 17GB in size). Another factor was the fact that certain mailboxes were over 2 to 3GB and would take a while to transfer over,” said Murphy.
After getting everything set up and all the software installed, the exchange server wouldnt start. Three hours and 10 cups of coffee later, Murphy realized that during the installation, hed typed the distinguished name incorrectly.
“Now the program was looking for a different name than I had given it. … It was a single space between two letters which caused it. I think at that point I banged my head against the wall a few times.”
At 1 a.m., after uninstalling and reinstalling Exchange, everything finally booted as it should, so Murphy gave himself a much-needed nights rest.
But, sure enough, when he returned the next day, he had a whole host of new problems: duplicate copies of every single piece of mail in every single mailbox and servers crashing because the database was too large.
“I was in a Catch 22, so to speak, because I couldnt boot up the server to delete the e-mails, but I needed to boot up the server to get access to the e-mails to delete them. So I had to completely remove the database and restore it from my backup (probably the only thing I did right up to this point), and then start over from scratch,” said Murphy.
After restoring the database, and then the current weeks e-mails, everyones inboxes finally had the correct new company name… a full week after hed started the job.
“After the experience, I thought briefly about actually quitting the IT industry altogether, but later I found out what a learning experience it really was. It was most definitely the most difficult situation I had ever experienced,” said Murphy.
The Nightmare on Vendor Street
Patty Laushman, president of the Lakewood, Colo., IT and VOIP consulting company Uptime Group, found her network demon in probably one of the funniest sagas—that is, as long as youre not the couple involved.
When her company was asked to bid on a network augmentation project, it ended up competing against two other companies.
When the existing “one-man-show” vendor caught wind he was about to be fired a few weeks before the proposal submittals, he did the unthinkable.
“He installed scripts on the customers e-mail server to e-mail him a copy of every e-mail in or out of the president, CFO, IT manager and senior scientists e-mail boxes. In the process, he discovered an affair going on between the senior scientist and his wifes brothers wife,” said Laushman.
The day the proposals were submitted to the company, the vendor zipped nearly 500 e-mails and attached them to an e-mail he sent to the wife of the senior scientist, making it look like the e-mail came from the IT manager who was about to fire him with a “just thought you would like to know” e-mail.
“The fallout was pretty predictable, except that I do not know the scientists wife exact reaction when she opened the e-mail… The day after the proposal submittal, the customers IT manager drove to our office and asked my business partner to do the necessary forensics work to prove the vendor had done what they suspected, which we were able to do successfully,” said Laushman.
One would think this is as low as a embittered vendor could sink, but Laushman offers one last startling bit.
“He was even running an online gambling site off their servers, which we found when we were sleuthing around with the companys permission.”
From haunted servers to vamipiric vendors and unearthly end-users, nothing beats a workplace IT nightmare to send chills down any techies spine.
If you have a story youd like to add, e-mail it to Deb Rothberg, and well publish the most harrowing.
Additional reporting by Allan Alter, Debra DAgostino, Debra Gage and John McCormick