eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Tom Oreck began shopping for a second manufacturing facility for vacuum cleaner maker Oreck in December. Tulane Universitys Paul Barron now plans for a data center shutdown that lasts weeks, not days. Jan Rideout, of Northrop Grummans Ship Systems Sector, consolidated data centers and made sure they were nowhere near a fault line. These are just some of the lessons technology executives have learned leading up to the one-year anniversary of Hurricane Katrina, which on Aug. 29, 2005, pummeled the Gulf Coast region, left more than 1,300 people dead and caused $96 billion in damages, according to a White House report on Katrinas aftermath.
“Katrina opened our eyes to the fact that our business depended on a single facility. We started looking in December for another facility,” said Oreck, president and CEO of Oreck. “We never thought there would be a storm that would take out both facilities—New Orleans and Long Beach,” Oreck said, referring to his companys New Orleans headquarters and Long Beach, Miss., manufacturing plant 76 miles away.
“Everybody operated on the notion that you would leave for two or three days and then come back. Shutting down the data center, backing up the tapes—everything worked just fine. But nobody contemplated the impact. It was regional,” said Barron, interim provost, vice president for IT and CIO of Tulane, in New Orleans. This year, Tulane is open once again and is about to receive thousands of students for its fall semester; last fall, the university had to send all its students back home.
If Katrina accomplished anything, it forced executives to rethink disaster recovery plans in the wake of the nations costliest weather-related disaster. New Orleans businesses stand ready for this years hurricane season with an array of overhauled defenses. If the Sept. 11, 2001, terrorist attacks made companies create disaster recovery plans, Katrina forced them to rework those initial efforts.
Among the biggest Katrina take-aways:
- Geography matters—a lot. Geography was much on the minds of Oreck staff at a November 2005 corporate disaster strategy session, where company executives decided to build a second manufacturing plant in Cookeville, Tenn., 573 miles from Orecks Long Beach facility. “Its not likely to be hit by the same event,” Oreck said. “We took possession 30 to 40 days ago. Well have our first products in September. That is fast. For us, speed is a strategic advantage in just about everything we do.”
The Cookeville opening was just a start. Orecks New Orleans data center housed an IBM AS/400 server co-hosting corporate data and a Web server for Oreck.com. Now the Web server is hosted in Atlanta, and the AS/400 is mirrored in Boulder, Colo.
Location also was key to Northrop Grummans thinking. Rideout, CIO of Northrop Grummans Ship Systems Sector, in Pascagoula, Miss., was spurred by Katrina to reconsider the location of the companys data centers scattered across the country. A product of a number of corporate mergers over the years, the company has some 200 data center locations and had decided, pre-Katrina, to consolidate them to eight. Katrina prompted the company to boil down that number to four—in Dallas; Rolling Meadows, Ill.; Lafayette, Colo.; and Chesterfield County, Va.—making sure the centers are nowhere near earthquake fault lines or hurricane-prone coastal areas.
Last year, Katrina forced Northrop Grumman to shut down 200 servers in Pascagoula and quickly bring up their replacements at a company data center in Dallas. Previously a minor site, the Dallas data center continues to run the servers for Pascagoula as it did a year ago, and it is now designated as one of the four critical data centers in the companys technology blueprint.
“There are some days when I say, Thank God we did have Katrina,” Rideout said. “That can sound weird, but it did help us to accelerate some technology. There have been good things that have come out of it, as painful as it has been. I do think that it made the business appreciate IT more.”
Next Page: Lesson 2: Budgets may hinder big plans.
Lesson 2
: Budgets May Hinder Big Plans”>
- Budgets may hinder big plans. Not all New Orleans-based organizations have the luxury of rethinking disaster planning on a national scale. Tulanes Barron said the university is taking major strides to improve its preparedness, but due to cost constraints, it must keep its data center in the same leased space in a 14-story building, located across the street from the Superdome, that had its lobby flooded by 18 inches of water during the storm.
Staying put violates the most important lesson of Katrina, Barron said. “Never put a data center in a building you dont own.” When a disaster hits, “you have to stay in the building long enough to do what you have to do,” he said, noting thats not always possible if the landlord says you cant get in. Within its limitations, Tulane is doing what it can, however.
“We are putting in generators that run on diesel and natural gas. Were putting in our own air-cooled chillers for air conditioning, so we dont have to depend on the city water supply. But that project will not be done before this hurricane season is over,” Barron said.
More successful was Tulanes implementation of failover systems, using SunGard Availability Services to mirror its Web site at SunGards Philadelphia facility. Tulanes e-mail system also is mirrored, by MessageOne. “For those two communications mechanisms, weve got a failover system. Thats critically important,” Barron said. The universitys data is stored on tape at an Iron Mountain facility in Baton Rouge, La. In the event of another disaster, the tapes would be flown to SunGards facility in Philadelphia, Barron said.
Katrinas impact was felt far away, even in locations where hurricanes seldom hit. Mike Jones, vice president and CIO of Childrens Hospital and Health System, in Milwaukee—whose greatest concern is tornadoes—purchased satellite phones and deployed a ham radio system thanks to Katrina. In addition, Jones and the hospital are deploying increased amounts of medical supplies, water and diesel fuel for generators. “We can last a week already; we are going beyond that,” he said.
The hospitals disaster plan also must account for the fact that the hospital would likely see a surge in patients in case of an emergency. “Were a 232-bed childrens facility that could handle 300 patients, but we need to go beyond that,” Jones said. He said his future thinking will be in the direction of mirrored data across different locations to create high availability, which would be particularly important as the hospital substitutes electronic forms for paper. “Clinicians have to have electronic records,” he said.
Next Page: Lesson 3: People run your IT systems.
Lesson 3
: People Run Your IT Systems”>
- People run your IT systems. In addition to renewed location awareness, people issues, which might have been far down the to-do list of disaster planning for many IT professionals, were pushed to the top by Katrina, which prevented many people from going to work, deterred others from leaving their families in a crisis and otherwise cut off communications between employees.
“Probably the most important [part of a disaster recovery plan] is the people component. Are your people going to want to leave, after their homes are destroyed, to work at a backup facility?” said Dave Palermo, vice president of marketing for SunGard Availability Services, in Wayne, Pa.
To allay those concerns, executives have responded with measures ranging from the purchase of satellite phones to building databases with up-to-date employee contact information. In addition to purchasing satellite phones for key Northrop Grumman Ship Systems Sector employees, Rideout is deploying Nextels push-to-talk technology to replace a corporate radio system after Katrina heavily damaged transmitters. The shipbuilder is also investing in a mobile home outfitted with networking gear, Rideout said.
To make sure that Northrop Grummans users of Research In Motions BlackBerrys can reach each other, the company has deployed software that lets users download PINs for contacts—and is looking at automating the process via push technology—so users will have the current PINs of fellow employees.
Rideout is addressing another contingency exposed by Katrina—the inability of people familiar with IT systems to get to them and operate them. “Were planning on doing surprise drills, where we would choose a location and call the CIO and the managers in that location and say theres a disaster,” Rideout said. “We want to make sure we can recover with people that are not at that location. Most recovery plans are based on the assumption that the people that are normally there will be there to recover the systems.”
Stuart Suffern, director of IS at Dupré Transport, a trucking company in Lafayette, La., seconded the notion of people first. “The No. 1 lesson was what to do if people cant get to facilities,” Suffern said. “We had to move employees to Baton Rouge or Lafayette from New Orleans. We had to help employees find apartments. We bought vans to carry employees back and forth,” he said.
With its people in place, Dupré could bring its trucks to bear, carrying much-needed fuel and water. “We have many hundreds of trucks that can haul fuel in an emergency,” Suffern said. During Katrina, the company stepped in to help a major coffee roaster, whose coffee is imported from South America by the shipload and roasted at a New Orleans plant. “We brought water to them in tanker trucks,” Suffern said.
“The most important thing is keeping track of your employees, with contact numbers of where the employees will be going in the emergency,” Suffern said. He has purchased both fixed and mobile satellite phones as well as BlackBerrys for managers. “Communication is everything,” Suffern said.
Sometimes keeping in touch is not a matter of high technology. “Weve given everyone a little card with three numbers—one [is] a number to call to tell us where to reach them. And a second is a call-in number for a daily 8 oclock conference call, and a third number that is for a departmental conference call,” Oreck said.
With operations back to normal, Tulanes Barron is now facing another kind of people problem. With so much housing lost due to Katrina, its hard to get people to move to the New Orleans area. “If you could send me some sysadmins or DBAs, I would be forever in your debt,” Barron said. “Its hard to find people. They cant come here because theres no excess housing.”
Whats next?
Most CIOs said their brainstorming has turned to how to prepare for a pandemic such as the bird flu. That kind of disaster would leave IT systems humming but prevent workers from getting to them. Data centers in separate locations that can fail over look like a good answer, several people said. Jones said Katrina has spurred Childrens Hospital to pursue working with other Milwaukee-area medical institutions to develop a plan to deal with a pandemic.
Palermo said SunGard Availability Services is thinking along the same lines. “Companies tend to plan for the last disaster. Were focusing right now on a potential pandemic. No one is exactly sure what might happen. Will the entire country be quarantined? We recommend companies build in flexibility, since you dont know whats going to happen,” Palermo said.
While thinking of a pandemic might seem far-fetched, the most important lesson of Katrina might be that the unexpected—the worst case—not only can, but will happen.