In the Face of Disaster, Ingenuity Saves the Day

Two weeks into his new job as chief technology officer of thriving defense contractor InnovaSystems International in San Diego, Michael McCoy ran headlong into one of the worst situations an IT professional can face: the literal and figurative meltdown of his companys IT infrastructure.

As McCoys family filed into St. Peters Catholic Church in the Northern San Diego town of Fallbrook, Calif., on a Sunday morning in October, he was vaguely aware of a wildfire burning 90 miles away in a sparsely populated area near the Mexican border. By lunchtime on Oct. 21, the blaze had an official name and a death toll.

The Harris fire was burning out of control, fanned by seasonal Santa Ana winds gusting up to 90 miles an hour and spurring the conflagration west toward the south San Diego County cities of National City and Chula Vista.

Resource Library:

Meanwhile, another fire had erupted 40 miles to the north, near the rural area of Witch Creek, threatening the town of Ramona, Calif., directly east of San Diego.

By mid-afternoon, McCoy still felt that his home, and his employer in central San Diego, were well isolated from the growing firestorm, despite the ring of flames to the south and east that had already consumed nearly 30,000 acres. Nevertheless, he began e-mailing colleagues about the need to support employees that might be evacuated from affected areas, and laying plans to work remotely for the first time via InnovaSystems secure VPN.

Read more here about one techies experience fleeing the flames.

For Sony Electronics CIO Drew Martin, the fires had been drawing steadily closer to his Poway, Calif., home and the companys new head office. "It all happened on a Sunday when the fires started. That was a good and a bad thing: We didnt have people in the office, but we started to realize it was headed directly for our corporate headquarters in Rancho Bernardo," Martin said. "It was pretty chaotic."

By Monday morning, the situation had unraveled. McCoy woke to the realization that a third blaze, known as the Rice Fire, had broken out directly east of his Fallbrook home. Smoke and flames soon closed Highway 15, a major north-south artery and his primary route to work. "I realized I might still be able to get to work, but I might not be able to get back home again," McCoy said. McCoy called his new boss, InnovaSystems CIO Jim Halpin, and together they began putting the companys emergency communications plan into action.

At the Martin household to the south, Sonys CIO still felt safe but extremely worried. Situated near the high school that had been designated a safe haven for the now-evacuated residents of Ramona, Martin had been inviting displaced friends to come to his house a few hours earlier. Now the flames were moving unchecked toward his community, and he began preparations to evacuate his own family. County officials had already begun an evacuation program of historic proportions, relying on a new "reverse 911" automated calling system that had been implemented after the fatal Cedar Fire disaster almost exactly four years earlier.

Although Martin never got a call from the countys notification system, he was lucky enough to have three children enrolled in the Poway Unified School District. "In our school district, we had already implemented a reverse 911 system. When the district issued its school closure orders on Monday morning, I instantaneously got an e-mail while my phone got a voice message and my cell phone and my wifes phone both started to ring."

Martin made a mental note to accelerate existing plans to equip Sonys headquarters with it own reverse 911 communications system—that is, if the headquarters remained standing after the fires were finally extinguished.

Page 2: In the Face of Disaster, Ingenuity Saves the Day