Gaps Remain in Disaster Recovery Plans

Youd think the 9/11 terrorist attacks would have taught IT executives everything they needed to learn about making thorough business continuity and disaster recovery preparations. In the wake of the attacks many companies beefed up their business continuity plans by making sure data assets were mirrored and backed up at secure sites. But after the massive blackout last August that affected 40 million people from New England to Michigan and from New York City to southern Canada, IT executives realized that their plans were far from bulletproof.

For Commerce Bank, the blackout was "a wake-up call that we needed to review our process and plans in the event that something like this happened again," said Charles DiPietropolo, the banks vice president of data center operations.

Resource Library:

The Cherry Hill, N.J.-based banks operations escaped the worst of the blackout because its headquarters, data center and the majority of its branches are concentrated in New Jersey and eastern Pennsylvania, south of the blackout area, DiPietropolo said. "We were one of the lucky ones on that day," he said, although the banks branches in New York City were affected.

As a result of the blackout, the bank decided to fortify its business continuity plans by adding a third uninterruptible power supply and doubling the size of its backup diesel power generator, DiPietropolo said. "We really needed to have that capacity if and when the time came for another big blackout," he said.

In addition, the bank now tests its emergency business continuity and disaster recovery procedures twice a year to make sure that the staff is trained and experienced with the procedures, DiPietropolo said. Before the blackout the bank only tested these procedures once a year.

Sungard Availability Services, of Wayne, Pa., has set up a remote data center in Philadelphia that Commerce Bank will staff and operate to provide uninterrupted data access to customers in the event the banks data center in Mt. Laurel, N.J., becomes inaccessible. Twice a year the banks IT staff travels across the Delaware River to Philadelphia to run a 24-hour exercise simulating an emergency transfer of data center operations to Sungards site.

"We split it up into three shifts to give as many people as possible" experience working in the remote site, DiPietropolo said. "That way we arent relying on just one group to know how to do this."

The bank has also greatly expanded its data backup capability. Before the 9/11 attacks, the bank would do a complete data backup once a day, DiPietropolo said. After working with Sungard to improve its data recovery procedures, data backups are now constant and instantaneous, he said.

"We have made great strides in making our data systems more highly available and more fault-tolerant," DiPietropolo said.

Next Page: Theres still work to be done.