This is the Web site for the California Air Resources Board. Its nothing fancy to look at, but earlier this week it set itself apart for the simple reason that the possibility of its going offline didnt threaten to cut off the agency at the knees.
The same couldnt be said for the many California state agencies that suffered intermittent and random outages of Web pages and e-mail systems following the inadvertent hijacking of the entire ca.gov domain, including all subdomains, on Oct. 2.
California was caught off-guard when the federal GSA (General Services Administration), which manages all “.gov” domains, moved to knock the ca.gov domain offline in an attempt to deal with a hacker having rigged redirects to porn pages onto the site for a Marin County transportation agency.
How the ARB got through the fiasco is as simple as it was effective—as well as cost effective.
Specifically, the ARB has a back-up site hosted on a separate domain. Its a simple static page on a free domain, not a mirrored site, that can be preloaded with instructions for employees or visitors on how to communicate outside of their usual domain. Bill Welty, CIO for the agency—which focuses on air quality and air pollution reduction in California—chose not to specify that domain in an interview with eWEEK. “They probably dont need the attention,” he said. He did note, however, that there are multiple companies—such as Google, Lycos or Yahoo —that offer free site hosting.
On Tuesday, the ARBs site was in fact threatened with outage. The agencys data center had called to inform the ARB about the problem. Federal officials were facing the need to force the domain name through the system again. It was a process that would take some 7.5 hours to complete.
The ARB had only a few hours in which to react. Welty said that an all-hands bulletin went out via the e-mail that hadnt yet gone down, instructing all users that the temporary site was now active.
Welty went into the backup site, posted notifications about the incident, and told users to watch the space as things developed. Fortunately for the agency, their systems came back up and within a few hours there was no measurable impact. If there had been an impact, however, the agency would still have been able to provide information to staff about the ca.gov site and could still have communicated throughout the incident.
Welty said that California getting elbowed offline had a minimal impact on the ARBs operations and that it was in fact a good test of operation recovery plans that both his department and the California EPA have been working on for about a year.
To view an eWEEK slide show on signs that your kid might be a hacker, click here.
“The idea is that governments might look to the Internet [for] places to go when there are these kinds of [circumstances involving outages or disasters] so youre not totally reliant on your own domain to keep things going,” he said.
The technique, which is of course not limited to government organizations, is to avoid keeping all your eggs in one domain basket. “Then if you encounter something like this thats rather sudden, … the public, in the case of government, or your customers or employees, in the case of private companies, are not at a loss if your site goes down.”
The simple solution has added appeal because it involves no intervention from IT staff, Welty said, who are likely scampering to fix an outage in such a situation. “You have almost immediate recovery because youre not relying on IT staff to turn the switch,” he said. “Thats basically what we did.”
This solution doesnt involve a mirrored site, which would to a greater or a lesser degree duplicate live online processes. Rather, its a simple, static page that can be preloaded with, for example, instructions for employees on how to communicate outside of their usual domain. Welty said that the ARBs emergency off-domain page had a simple description of what happened and a request that visitors be patient and keep an eye on the space for updates.
Welty particularly likes the cost. “What I like is its free,” he said. “Thats a big attraction, given that were an open-source shop. Its simple to use [and] to create. Its instantaneous. In a [Hurricane] Katrina situation, without having to rebuild the whole infrastructure, youd still be in business.”
Welty has, in fact, pitched to hosting companies the idea of beefing up these types of pages to include encryption and other features that would make them more enterprise-worthy. But given that his agency only regards the back-up page as a quick fix, meant to be used from a few hours to a few days, its not really worth the bother. “Its only temporary, … so who cares? Its just a way to keep things running if your site takes a hit.”
Check out eWEEK.coms Security Center for the latest security news, reviews and analysis. And for insights on security coverage around the Web, take a look at eWEEKs Security Watch blog.
