eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Jan Rideout knew it would be tough to get through the next few days.
Hurricane Ivan had drawn a bead on her part of the Gulf Coast, and she knew it would not miss.
Rideout is CIO of of Northrop Grummans Ship Systems Sector in Pascagoula, Mississippi. Her datacenters 350 Unix, Windows 2000, and Windows NT servers and 5.8 terabytes of data are the hub of a collaborative effort with the US Navy and other contractors to build the next generation of warships for the US fleet.
The hurricane potentially could have wiped away hundreds of man-years of engineering work and brought operations at Pascagoula to a hard stop.
Fortunately, Rideout had a plan. In fact, she had a plan of long standing, which is important when your company periodically finds itself in the bulls eye of disaster. And because her organization has two locations, one just outside New Orleans, the chance of being on the spot was doubled.
“Theres a big emphasis on business continuity planning at Northrop Grumman,” Rideout explained. “We have procedures and we follow them.” The company has developed its plans out of long experience, but they also practice the plan and audit it.
But Rideout also noted that sometimes, not everything follows the plan, and then they have to do the best they can.
“Our basic procedure calls for us to shut down,” Rideout explained. She noted that they were paying very close attention to a private weather forecasting service, ImpactWeather, Inc. (www.impactweather.com) for the latest word on exactly where Ivan would come ashore, and just how bad the results would be. “Theyre never wrong,” she said.
After watching Ivans progress for days, on September 14, Rideout knew the time had come to take action. “We made the decision Tuesday morning,” she explained. Her Avondale facility, just outside of New Orleans had to be shut down first because the area was being evacuated. She started backups that had to be completed by noon.
The next step was to back up the massive store of data in Pascagoula – everything from engineering drawings of the ships the company builds to parts lists and material for collaboration. It all had to be saved. “Fortunately, we had just done weekly backups Monday,” Rideout explained, so all that was required was an incremental backup.
Next Page: Deciding which services to keep running.
Decision Making
“We started backups at noon,” she said, arranging for the backups to be carried by courier to a secure location.
Once the plans were underway, the IT staff had to decide which, if any, services to keep running through the hurricane. Eventually, they decided to keep their e-mail and Blackberry servers running.
Then they had to decide what to do with one of their most important projects, the US Navys DDX project, a new type of destroyer already being designed.
Northrop Grumman uses TeamCenter Enterprise, a product lifecycle management (PLM) system from UGS, as a collaboration tool to link the efforts of other companies around the country who are working on the design with them.
“We weighed the risks and decided to make the safest decision,” Rideout said, which was to shut down the DDX Environment, as the company calls the collaboration effort, even though it was clear that the lost time could be expensive.
As Northrop Grummans 350 servers were shut down, each was wrapped in plastic to protect the equipment from any water that might penetrate the data center from leaks or wind damage.
Then, the next challenge: how to provide electrical power for the data center so that the servers that would be kept running would have a means to do so.
While Northrop Grummans shipyard has its own power plant as is the case with many shipyards, the data center received its power from the local company, Mississippi Power. But Rideout knew they would lose that when the storm hit.
“The shipyard leased a generator for us,” Rideout said. One problem: the data center needed a power converter before the generator could be used. A last minute call, and an evening FedEx delivery on the eve of the storm solved that problem.
But there was a new problem.
Shipyard employees couldnt stay. Many were being evacuated; many more had to secure their own homes and their families. All the employees were gone, but someone had to connect the data center to the leased generator, and bring up the servers on generator power. “Some of our employees volunteered to come in Wednesday morning,” Rideout said. The employees worked quickly to bring down the e-mail and Blackberry servers, attach the generator, and bring the server back up.
Finally, the IT staff was ready for Ivan.
Next Page: The hurricane hits.
Hurricane Hits
Ivan struck Pascagoula at 1:00AM.
By then Rideout had evacuated to Dallas, Texas, to wait out the storm. Dallas is the site of one of Northrop Grummans major data centers.
She knew that if it was necessary, e-mail and Blackberry services could be moved there, as could the other functions normally done at the shipyard.
Fortunately, it wasnt necessary.
Ivan struck land east of the shipyard, resulting in less wind than there might have been. Rideout knew this might be the case because of information received ahead of time from ImpactWeather. But she couldnt take a chance.
The storm had passed by midday Thursday.
Rideout headed back from Dallas knowing that commercial power wouldnt be available for several days. Because of this, the IT staff connected the entire data center to the emergency generator, and one by one, brought the servers on line. The data center was alive, and even though it might take some shipyard employees a while to get back to work, at least theyd have IT services available when they did.
Then, at 9:00PM, the generator failed.
Working late into the night, shipyard electricians and IT staffers switched the data center over to the shipyards power plant.
“Its not as clean as Mississippi Power,” Rideout noted, but it was better than no power at all.
Sunday, Mississippi Power restored its service to the shipyard, and everything was back to normal, or as normal as it can be for an area that demonstrates what getting hit by a massive storm is like.
Rideout said that the shipyard actually had very little damage, except for the occasional broken window and subsequent rain and wind damage in isolated areas.
“We absolutely prepared for the worst,” Rideout said, reflecting on the preparations, “We were very lucky.”
Clearly, Rideout, the IT staff, and Northrop Grumman had a big role in making their own luck.