Recovery Is Hard When the Whole World Is Shut Down

By Wayne Rash  |  Posted 2012-07-02 Print this article Print


So the derecho came in the dark of night. The first hint was the flicker of lightening off to the northwest. Then a storm more violent than anything I€™d ever seen before slammed the area. This was worse than the hurricanes I€™d experienced, including one off the West Coast of Africa that was my previous high point when it came to weather-related anxiety. In 45 minutes, it was gone and so was the power, the Internet service, the phone service and the previously reliable cell tower. 

But I got the generators started and began bringing up the lab infrastructure. One by one, the switches and servers came alive, the whir of the fans and the flickering of the lights reassuring me that all was well. Then I started up the HP server that handles the Domain Name System (DNS), the Dynamic Host Configuration Protocol (DHCP) and directory services. The low-voltage alarms started going off one by one. I didn€™t have enough capacity to run the lab, despite my previous tests. 

So I shut down the servers and the other computers, and finished bringing up the infrastructure. I had capacity for that and everything ran, but I was approaching the total capacity of the generators, and that€™s never a good thing. But that€™s when I found that it didn€™t matter. My lab might be operational, but it couldn€™t communicate with the outside world because nothing else was operational. Being able to run when the rest of the world isn€™t really doesn€™t help much€”especially when you realize that you€™re going to have to buy another generator and set up load sharing. 

Actually, I€™ll have to buy two more generators for full N+1 capability. But in the meantime, I€™ll have to also remember that I have to run tests of the entire system more frequently, especially after I add more servers, new switches or network management equipment. I hadn€™t gotten around to that, and it cost me.  

But in this case, all of the planning wouldn€™t have made any difference. As I looked out at the hazy heat that had brought all of this about, the one thing that kept popping into my head (right after the desire for a nice cold beer) was the words of Scottish poet Robert Burns: 

€œThe best-laid schemes o' mice an' men

Gang aft agley,

An' lea'e us nought but grief an' pain,

For promis'd joy!€

Thanks for the reminder, old Rabbie. Tonight, I€™ll have a wee dram of Scotland€™s best in your memory and to remind me that we can€™t plan for everything.

Wayne Rash Wayne Rash is a Senior Analyst for eWEEK Labs and runs the magazine's Washington Bureau. Prior to joining eWEEK as a Senior Writer on wireless technology, he was a Senior Contributing Editor and previously a Senior Analyst in the InfoWorld Test Center. He was also a reviewer for Federal Computer Week and Information Security Magazine. Previously, he ran the reviews and events departments at CMP's InternetWeek.

He is a retired naval officer, a former principal at American Management Systems and a long-time columnist for Byte Magazine. He is a regular contributor to Plane & Pilot Magazine and The Washington Post.

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel