eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
For Walter Overby, hurricanes and other severe-weather events are a part of daily life on the Gulf Coast. Having endured Hurricane Ivan last year, Overby, vice president of systems delivery services for Alfa Insurance, in Montgomery, Ala., knew what kind of misery and devastation such storms could bring, and he had no intention of being caught unprepared.
So a month before Hurricane Katrina roared through the Gulf Coast, demolishing New Orleans and causing unprecedented levels of damage to portions of Mississippi and Alabama, a special task force at Alfa chose to invest in a mobile recovery unit from Agility Recovery Solutions, of Charlotte, N.C. The idea was to use the mobile unit to better serve policyholders in Alabama, Mississippi and other areas by providing a safe, self-contained work area with power and voice and data communications that would allow adjusters and other staff to conduct massive amounts of new business.
“A lot of times under these types of disasters, you see adjusters working out of cars or out of hotels with calling centers, or leased office space where they can find it,” Overby said. “Usually [in] an area thats been devastated, [it] has been hard to find adequate [working] conditions. … We just wanted to put our company in a position [that] if people tried to get in touch with us, they could, and get claims calls we could deal with in a timely fashion.”
Once Alfa declared a company emergency during the hurricane, Agility sent a 24-by-68-foot, double-size mobile unit to a location in Mobile, Ala. However, several factors delayed the units arrival, including Federal Emergency Management Agency restrictions on road and highway access to areas slammed by Katrina.
“The magnitude of Katrina posed difficulties nobody could overcome. We got a mobile unit to the border and couldnt get into the state. We had to identify who we were to FEMA,” said Bill Boyd, president and CEO of Agility. “This was a catastrophic event. The United States has never seen something like this, that big a region. An entire city was devastated.”
Alfa was hardly alone in facing challenges. For businesses, universities and organizations throughout the Gulf Coast, even those with well-designed and practical disaster recovery plans, the task of bringing their organizations back online has proved more daunting than they feared. In many cases, technology has turned out to be an invaluable asset in the recovery process, but other companies have learned the hard way that even in an age of unceasing technical innovation, theres only so much that technology can do.
“I think the majority of people [in affected areas] are recognizing the scope of this, and its not something [where] theyre going to be back up and running next week,” said Belinda Wilson, executive director of business continuity and availability services for Hewlett-Packard Co., in Palo Alto, Calif. “Most people have almost accepted the level of devastation and realize this will not be a short-term recovery. Many of these places just need to be completely demolished. Theyre not waiting for any more on the damage assessments. I think its pretty clear.”
In Houston to escape Katrinas destruction, Mike Roppolo, director of health services for the Newman School, in New Orleans, said the task of restoring the popular city is daunting. “Its going to take more than several months to rebuild that city and get it back to some form of what we knew to be New Orleans,” Roppolo said. “Some families will return; some families may not return ever. Things might never be the same way for many of us there ever again.”
Roppolo said Newman officials have decided that his school will not likely reopen until January at the earliest. In fact, an entire calendar year may be lost at the school due to myriad environmental and technical problems still plaguing New Orleans.
Yet, what seemed a relatively minor decision last spring to move all students health and immunization records from paper to an online electronic format stored off-site is now paying huge dividends to hundreds of families scrambling to enroll their children at replacement schools across the country. The Newman School is using Digitech Systems Inc.s ImageSilo service to store students health records. Through the service, Roppolo can grant requesting schools and parents secure access to the archived records.
“Our IT department thought it would be more cost-effective and give us more options to store things off-site. Of course, we could have never imagined we could be in the position we are now or using it for this purpose so soon,” said Roppolo. “We have a lot of families that relocated to Houston, Dallas, Baton Rouge … some are in Colorado, some in Florida, some in Wisconsin. By having all these documents scanned, we can make those immunization and medical records available by getting them off the server and get them in e-mail and PDF format to schools or parents requesting them.”
Although the school itself wasnt damaged and is dry for the most part, Roppolo said, the manpower needed to move records for 1,200 students to Houston in order to fax or mail them would have been impossible to find during the hurricane and its aftermath.
Next Page: The long road to recovery.
TKTK
For some IT users caught in the direct path of Hurricane Katrina, lack of time and lack of power were two big roadblocks on the path to recovery.
“The hurricane crept up on us so quickly,” said Venkata Mahadevan, systems manager for the computer science department at the University of New Orleans. “Its not possible to move giant racks of servers, etc., to other locations without major resources, which we dont have. Even if the servers could be moved, where could they be set up again? This boils down to a lack of resources.”
Like Mahadevan, most system administrators are not trained to effectively deal with such a crisis. He said the best course of action is usually to shut systems down before the storm and set up temporary sites and Web portals.
Although he has all his departments data on backup tapes, with which he evacuated, Mahadevan finds himself lacking servers, a tape drive, and a data center or remote site to begin the rebuilding process. Still, he must formulate a plan to recover when the opportunity eventually comes around.
“I expect that the first thing we will do when we get back to campus, which is not accessible right now but remained mostly dry, would be to come up with a more effective hurricane/ emergency response plan for our department,” Mahadevan said in an e-mail message. “Hindsight is always 20/ 20, but this may include purchasing a generator and installing it on the rooftop of the building and establishing a replicated computing infrastructure at a site outside of New Orleans that mirrors the original data center.”
For many people charged with running IT systems, Katrina proved the ultimate testing ground for disaster recovery and business continuity plans. Tim Babco, senior director of IT for SCP Pool Corp., in Covington, La., said his company designed a core set of disaster recovery documentation several years ago to mitigate hurricane risk. Babco said Katrina will force future revisions.
Three years ago, SCP Pool, a large pool-supplies wholesaler, selected what is now a Vericenter Inc. facility in Dallas as its primary data center for hosting the companys mission-critical, AIX-based ERP (enterprise resource planning) systems. SCP Pools second-tier applications, such as its human resources, benefits and payroll software, typically reside in Covington.
To safeguard its IT systems from Katrina, SCP Pool relocated the staff necessary for shifting its call center, support operations and phone numbers from Covington to Dallas. “My biggest challenge was two things: get my staff and support capabilities to Dallas, and, two, move [secondary] and tertiary systems to Dallas,” said Babco. “Any [disaster recovery] plan has to identify priority. Along with priority, how long can something stay out and unavailable and not impact the business?”
Babco employed what he called a makeshift “Noahs Ark” strategy in redeploying IT staff to Dallas, choosing a handful of experienced staffers—including an AIX administrator, a network administrator, a Microsoft Windows administrator, a help desk agent and a software specialist—to be sent out in a first phase, followed by more staff in later phases.
“By Monday morning, when the hurricane was hitting, I had a core IT staff already positioned in Dallas, with all of our IT help desk calls already forwarding to us. Therefore, all 200 [global SCP Pool] locations needing IT services were uninterrupted,” Babco said.
Although many areas near Covington are still without power and there is limited connectivity at SCP Pools headquarters, Babco said his team will leave the Vericenter facility this week.
Many businesses without the luxury of leaving areas damaged by Katrina must rely on more localized technology offerings for their needs. For instance, HP has shipped two of its mobile trailers equipped with the amenities of the companys data recovery centers.
According to HPs Wilson, a New Orleans bank is using one trailer as a temporary operations location, featuring 100 seats, tellers and even a drive-through window open for business. She said the mobile units most in demand are those with satellite communications capabilities. “Given whats happened in New Orleans and [that] all phone lines are down, it would behoove us to make the investment there and add satellite [phones] to all of them,” Wilson said.
The hurricanes unexpected fury and aftermath have left many IT users—some of whom had left in advance with only a few days worth of clothing or personal items—forced to balance moving ahead with their personal lives with keeping their companies technology and business needs uninterrupted.
“People in New Orleans and affected areas, theyre worried about where theyre going to get their eyeglasses prescription or where their kids are going to go to school. The last thing they want to worry about is [IT],” said Agilitys Boyd.