Disaster Recovery Plans Fight Chaos

Putting business continuity plans into action is showing some CIOs the difference between theory and practice.

Hurricane Katrina wreaked havoc on communications and data networks across the Gulf Coast region last week, sending IT managers scurrying to implement their disaster recovery plans and patch together makeshift networks to keep their businesses running as best they could.

For the regions businesses, from metropolitan law firms to the Gulf Coasts plethora of waterfront casinos, meeting the challenges of continuity and disaster recovery plans was predicated on people and resources being available at multiple locations. The situation was exacerbated by mandatory evacuations.

Shortly before Katrina hit, David Erwin, CIO of New Orleans-based Adams and Reese LLP, took his family to stay with relatives in Dallas.

In the midst of executing his disaster recovery plan, Erwin said a hurricanes full-potential might was unimaginable to many.

"We always talked about a Category 5 hurricane coming; it was talk for 20 years. Sure enough, it turned out to be more risk versus [reality], but its happened to us," Erwin said.

"The surprise is the extent of the damage and the things people dont think of. I think a Category 5 was modeled, but I dont know too many people that actually thought it would be real."

/zimages/4/28571.gifRead details here about how communications providers are struggling to restore service in the wake of Hurricane Katrina.

Operating from the tallest building in New Orleans, Erwins law firm prepared its IT systems for the storms onslaught by scheduling backup and moving its tapes to be managed off-site by a third party.

Following the Sept. 11, 2001, terrorist attacks, Adams and Reese decided to help mitigate risk by enlisting the aid of MessageOne Inc.s EMS (Emergency Mail System). Erwin decided to power down his Hewlett-Packard Co. servers and fail over his e-mail systems to MessageOnes hosted service after his offices motion detectors were set off by Katrinas powerful wind gusts shaking the building. He said his e-mail system was down for only 15 minutes.

"The most important thing, if anything, is you have a way to communicate effectively [in a disaster]. It makes things go a lot smoother because we have people all over the place," said Erwin.

"Our people are productive, they can communicate, and our clients that can e-mail us have no problem getting to us, and thats important to continue business as normal as possible."

The law firms employees are dispersed among Baton Rouge, La.; Nashville, Tenn.; Birmingham, Ala.; Houston; Dallas; and New York. Erwin said the EMS service has proved crucial for performing roll call and keeping track of where everyone is at any given time.

/zimages/4/28571.gifRefusing to stop the presses, a New Orleans paper takes to the Web. Click here to read more.

Erwin said he plans to switch his e-mail systems back to the companys Birmingham alternate site by this week. Hes expecting his Mobile, Ala., office to be reopened in about two weeks. However, the New Orleans office could be closed for two months.

"A lot of people who dont work in the technology business dont understand its not just power but its about water. Our [IT environment] needs water to [cool] the servers. So you actually need both," said Erwin.

For the time being, the law firm will operate out of its Baton Rouge office, and all technical equipment will be housed in Birmingham.

It is crucial for customers to update their disaster recovery contracts with their vendors whenever a change is made as any incongruity could affect cost, recovery levels or services rendered, said Bob DiLossi, crisis management manager for SunGard Availability Services, in Wayne, Pa., a division of SunGard Data Systems Inc.

"If you make a change to your technology and how you do your backups and how systems are run and you dont make a [contractual] change with that, youre out of sync," said DiLossi. "Ive seen that happen a couple of times during this event."

Officials at Harrahs Entertainment Inc., which operates several casinos in the hardest-hit areas, said the company is still assessing the situation, although its Grand Casino Biloxi and Grand Casino Gulfport appear to have suffered extensive damage. IT resources from the companys six other properties in Mississippi and Louisiana were being marshaled to assist in the recovery effort, according to a company statement.

Mike Rosenfelt, executive vice president for MessageOne, in Austin, Texas, said his company so far has had 36 clients that have been directly affected or are in the process of recovery from Katrina.

He said people in affected areas have no sense of when they can return to their businesses or when they may become operational again.

"I think thats something we didnt see with [hurricanes] Frances, Charley and Dennis," which ripped through Florida in 2004 and 2005. "I dont think weve even begun to see how devastating this is yet. I think theres a feeling amongst our customers [that] there are second and third parachutes to drop here, and everyone is frightened to know whats still to come."

/zimages/4/28571.gifCheck out eWEEK.coms for the latest news, reviews and analysis on IT management from CIOInsight.com.