eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Two weeks into his new job as chief technology officer of thriving defense contractor InnovaSystems International in San Diego, Michael McCoy ran headlong into one of the worst situations an IT professional can face: the literal and figurative meltdown of his companys IT infrastructure.
As McCoys family filed into St. Peters Catholic Church in the Northern San Diego town of Fallbrook, Calif., on a Sunday morning in October, he was vaguely aware of a wildfire burning 90 miles away in a sparsely populated area near the Mexican border. By lunchtime on Oct. 21, the blaze had an official name and a death toll.
The Harris fire was burning out of control, fanned by seasonal Santa Ana winds gusting up to 90 miles an hour and spurring the conflagration west toward the south San Diego County cities of National City and Chula Vista.
Meanwhile, another fire had erupted 40 miles to the north, near the rural area of Witch Creek, threatening the town of Ramona, Calif., directly east of San Diego.
By mid-afternoon, McCoy still felt that his home, and his employer in central San Diego, were well isolated from the growing firestorm, despite the ring of flames to the south and east that had already consumed nearly 30,000 acres. Nevertheless, he began e-mailing colleagues about the need to support employees that might be evacuated from affected areas, and laying plans to work remotely for the first time via InnovaSystems secure VPN.
Read more here about one techies experience fleeing the flames.
For Sony Electronics CIO Drew Martin, the fires had been drawing steadily closer to his Poway, Calif., home and the companys new head office. “It all happened on a Sunday when the fires started. That was a good and a bad thing: We didnt have people in the office, but we started to realize it was headed directly for our corporate headquarters in Rancho Bernardo,” Martin said. “It was pretty chaotic.”
By Monday morning, the situation had unraveled. McCoy woke to the realization that a third blaze, known as the Rice Fire, had broken out directly east of his Fallbrook home. Smoke and flames soon closed Highway 15, a major north-south artery and his primary route to work. “I realized I might still be able to get to work, but I might not be able to get back home again,” McCoy said. McCoy called his new boss, InnovaSystems CIO Jim Halpin, and together they began putting the companys emergency communications plan into action.
At the Martin household to the south, Sonys CIO still felt safe but extremely worried. Situated near the high school that had been designated a safe haven for the now-evacuated residents of Ramona, Martin had been inviting displaced friends to come to his house a few hours earlier. Now the flames were moving unchecked toward his community, and he began preparations to evacuate his own family. County officials had already begun an evacuation program of historic proportions, relying on a new “reverse 911” automated calling system that had been implemented after the fatal Cedar Fire disaster almost exactly four years earlier.
Although Martin never got a call from the countys notification system, he was lucky enough to have three children enrolled in the Poway Unified School District. “In our school district, we had already implemented a reverse 911 system. When the district issued its school closure orders on Monday morning, I instantaneously got an e-mail while my phone got a voice message and my cell phone and my wifes phone both started to ring.”
Martin made a mental note to accelerate existing plans to equip Sonys headquarters with it own reverse 911 communications system—that is, if the headquarters remained standing after the fires were finally extinguished.
Page 2: In the Face of Disaster, Ingenuity Saves the Day
In the Face of
Disaster, Ingenuity Saves the Day”>
Martins household was among nearly 40,000 that received notifications from the Poway school district that morning, according to PUSD Chief Technology Officer Robert Gravina: “We sent out about 270,000 connected messages that week,” using the districts newly installed ConnectEd Notification System, from The NTI Group.
Amazingly, “of all the messages we sent out, we only got about 15 that were bad e-mail addresses or wrong phone numbers,” Gravina said. He credited not only the flawless operation of the notification system, which is updated nightly, but also the “clean and accurate” databases that the district maintains for its 33,000 students and 5,000 staff members.
“You have to be prepared for the communication,” Gravina said. “If the database isnt accurate, then its worthless.” Unfortunately, “the county did not have good information,” resulting in failed notifications in some cases, and in other cases, families mistakenly being evacuated from areas that were not directly in danger from the fires.
To read more about synching up for disaster recovery, click here.
“Thats not unusual” Gravina added. “People change their addresses and phone numbers all the time.” Constant maintenance of the data, and nightly uploads to a remote server, are key to being prepared when disaster strikes, he said.
By Monday afternoon, the IT professionals interviewed for this story found themselves fleeing from the flames. McCoy received the countys reverse 911 call, and joined the mass exodus with 40,000 of his neighbors as the entire community of Fallbrook came under mandatory evacuation, with orders to head west onto the grounds of the Camp Pendleton Marine Corps base.
Gravinas school district headquarters came under its own evacuation orders. He led a final raid on the district office with a few co-workers to grab irreplaceable backup tapes. “We evacuated, and the Sheriffs Department and Homeland Security took over our office while we were removed to the city offices.”
Soon he had resurrected the districts core IT infrastructure in the Poway City Council chambers, while he coordinated the commandeering of district schools and its transportation fleet by agencies supporting the thousands of firefighters fanning out across the county.
Sonys Martin, a New York native who had already lived through the Sept. 11 attacks in New York City prior to relocating with Sony to Rancho Bernardo in 2004, now was faced with a new and equally volatile situation. Jumping in the car with his wife, kids and dog, he barely had time to grab “photo albums, a change of clothes, some hard drives and some teddy bears,” he said.
Although local authorities were warning residents to stay off their cell phones, to keep communications lines open for emergency crews, Martins mobile phone quickly became a life-line as he joined Sonys hastily assembled “crisis task force.” Finding the corporate campus—in fact, entire city of Rancho Bernardo—suddenly off limits, “we had to make some unilateral battlefield calls,” Martin said.
Realizing that Sonys 2,000-plus local employees would be attending to their physical safety first, and checking messages second, Martin moved to quickly increase the size of each workers e-mail inbox. Next, recognizing that omnipresent BlackBerry devices had suddenly become the primary means of company communications, he issued a dictum asking all workers to refrain from sending large attachments that could choke off mobile communications.
Although power outages were common during the week that the fires raged, Martin and Gravina never suffered a catastrophic loss of corporate communications.
Page 3: In the Face of Disaster, Ingenuity Saves the Day
In the Face of
Disaster, Ingenuity Saves the Day”>
McCoy at InnovaSystems was not so lucky. However, even though the company suffered a temporary loss of its corporate intranet when a remote communications provider fell victim to a fire-related blackout, it quickly recovered its operations within a few hours. McCoy attributes the companys resilience to its core competency in defense readiness systems. The firms Defense Readiness Reporting System is used by multiple branches of the U.S. military to coordinate and track readiness measures for equipment, supplies, personnel and training.
Absence of information
A common complaint from the IT executives and their workers was a general absence of solid information on where the fires were burning—and where they were headed.
Sony workers, unable to find out whether nearby fires had reached their corporate offices, came up with a novel idea to monitor conditions at the Rancho Bernardo campus. Colleagues in Sonys Park Ridge, N.J., office fired up sophisticated videoconferencing equipment located in multiple conference rooms in Rancho Bernardo, training the cameras toward the windows to look for any signs of the approaching flames.
The proximity of the flames wasnt Sonys only worry. “We were considering shutting down our San Diego data center and cutting to our disaster recovery failover site if we thought too much smoke was getting into the data center equipment. So we needed to find out about the air quality,” said Martin.
Monitoring corporate IT infrastructure was also top of mind for Ed L. Goddin, vice president for IT operations and engineering infrastructure at Intuit. While the tax and accounting software makers corporate headquarters is in the Northern California city of Mountain View, it maintains a corporate data center in San Diego, along with a major divisional headquarters that employs about 1,300 people. God-din was among the decision makers who opted to close the companys large Santa Fe Summit facility, the home of popular TurboTax consumer software, even though the active fires seemed far removed at the time of the decision. With employees safe and accounted for, either in their homes or in hotel rooms provided for evacuees at company expense, it was easier to focus on protecting Intuits key infrastructure and data assets, Goddin said.
Although the organizations mentioned in this article all had employees or students who suffered loss or displacement from the fires, none lost their lives, and all were able to return to normal operations within a week of the fires outset. For all of the IT managers, attending to corporate infrastructure needs during the fires was secondary to addressing the physical well-being of their internal IT customers.
“Where we spent the majority of our time on the task force was looking out for the welfare of our employees,” Martin said, “You really need to start there before you can do anything else.”
Check out eWEEK.coms for the latest news, views and analysis on servers, switches and networking protocols for the enterprise and small businesses.