eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Youd think the 9/11 terrorist attacks would have taught IT executives everything they needed to learn about making thorough business continuity and disaster recovery preparations. In the wake of the attacks many companies beefed up their business continuity plans by making sure data assets were mirrored and backed up at secure sites. But after the massive blackout last August that affected 40 million people from New England to Michigan and from New York City to southern Canada, IT executives realized that their plans were far from bulletproof.
For Commerce Bank, the blackout was “a wake-up call that we needed to review our process and plans in the event that something like this happened again,” said Charles DiPietropolo, the banks vice president of data center operations.
The Cherry Hill, N.J.-based banks operations escaped the worst of the blackout because its headquarters, data center and the majority of its branches are concentrated in New Jersey and eastern Pennsylvania, south of the blackout area, DiPietropolo said. “We were one of the lucky ones on that day,” he said, although the banks branches in New York City were affected.
As a result of the blackout, the bank decided to fortify its business continuity plans by adding a third uninterruptible power supply and doubling the size of its backup diesel power generator, DiPietropolo said. “We really needed to have that capacity if and when the time came for another big blackout,” he said.
In addition, the bank now tests its emergency business continuity and disaster recovery procedures twice a year to make sure that the staff is trained and experienced with the procedures, DiPietropolo said. Before the blackout the bank only tested these procedures once a year.
Sungard Availability Services, of Wayne, Pa., has set up a remote data center in Philadelphia that Commerce Bank will staff and operate to provide uninterrupted data access to customers in the event the banks data center in Mt. Laurel, N.J., becomes inaccessible. Twice a year the banks IT staff travels across the Delaware River to Philadelphia to run a 24-hour exercise simulating an emergency transfer of data center operations to Sungards site.
“We split it up into three shifts to give as many people as possible” experience working in the remote site, DiPietropolo said. “That way we arent relying on just one group to know how to do this.”
The bank has also greatly expanded its data backup capability. Before the 9/11 attacks, the bank would do a complete data backup once a day, DiPietropolo said. After working with Sungard to improve its data recovery procedures, data backups are now constant and instantaneous, he said.
“We have made great strides in making our data systems more highly available and more fault-tolerant,” DiPietropolo said.
Next Page: Theres still work to be done.
Firms Not Making the
Grade”>
But not all companies have put so much thought into business continuity and data recovery. In a 2003 Harris Poll commission by Sungard, many senior executives at Fortune 1000 corporations acknowledged that they havent done everything they could since 9/11 to make sure that they would have full and secure access to data and would be able to continue doing business in the midst of another disaster.
Two-thirds of the executives said their companies are more prepared to access critical business data during a disaster than before 9/11. One-third say they are as prepared as before the attacks to provide data access.
However, while 86 percent of the executives contend that their companies are at least somewhat prepared to maintain data availability in a disaster, less than 15 percent say they are completely prepared.
The poll revealed there are serious gaps in corporate data recovery preparedness. Only slightly better than half the executives surveyed said their companies have backup offices for workers in the event of a disaster. The poll showed that 58 percent of the executives reported that their companies disaster recovery training for employees deals with recovery access.
Most revealing was that on average executives gave a grade of C+ to their ability to access business data after a disaster.
“We thought that grade was actually a failure,” said Mike Walsh, Sungards director of marketing communications.
But Walsh said he believes the power outage has made executives take another look at their disaster recovery programs to see how they can make them better. “We see a shift from traditional backup and recovery to examining the problems in terms of business continuity and emergency information accessibility,” Walsh said.
Companies know they face huge financial losses if a disaster shuts down their customer service applications for even a single day, Walsh said. As a result, corporate executives are now looking at the problems in terms of “How do I keep my information up and running no matter what happens?” he said.
“We see people building systems for that,” Walsh said, because for many senior executives disaster recovery “is not just an IT issue anymore. Its become a business issue” that they have to pay close attention to.
Second
-Day Woes”>
One of the major problems that the 2003 blackout revealed was that companies didnt know how to deal with an outage that lasted for more than a day, said Damian Walch, national practice executive with IBMs business resilience and continuity services in Chicago.
Companies “didnt know where their employees were. They didnt know where they were going” to operate emergency business recovery systems, Walch said. “There were companies where employees were walking in the second day and they still didnt have power,” with no idea how soon the problem was going to be fixed.
This has prompted companies to think harder about building higher availability into their systems to keep “the most critical business applications … running,” Walch said.
Its not just a matter of building plenty of redundancy into data processing systems, Walch said: Its also a matter of training people so they will know what to do as soon as an outage hits.
IBM worked with a company, which Walsh declined to name, to simulate an emergency recovery situation so the IT operations staff would understand what they had to do. The exercise simulated a sudden cable cut that the company had to work around to stay connected to its customers and to its data resources.
“Their goal in doing that was to decrease the chaos that naturally occurs immediately following the service disruption,” Walch said. Even in the exercise “there was some anxiety that went along with the discussions” on how the IT operations people would have to respond, he said.
What the exercise revealed is true for a lot of IT departments, Walch said. “Technology operations people dont like being in the spotlight. They dont want to be the ones to make the decisions. But unfortunately in these situations they have to be the ones to make decisions,” he said.
This exercise was particularly valuable for the company because the 2003 power outage hit about a month later. “People understood the steps they had to go through to start the recovery procedures. It decreased the anxiety level they had to deal with,” he said.
Corporations are realizing that they have to address business continuity in the same way they are addressing regulatory information processing requirements, such as the Sarbanes-Oxley accounting standards and the Basel II financial reporting standards. Its not just something that would be good to have. Its something that is a corporate requirement.
“What we are seeing is the companies are really thinking about not just how to build business recovery systems. They are trying to figure out how to build these failover approaches into their day-to-day business practices, “Walch said.
“Rather than have companies focus on disaster recovery, you would rather have companies build an information infrastructure that maintains the availability better,” he said.
Check out eWEEK.coms Infrastructure Center for the latest news, views and analysis on servers, switches and networking protocols for the enterprise and small businesses.