It was bad enough that federal officials essentially deleted the ca.gov domain in the process of shutting down a hackers redirect to porn pages on Oct. 2. Whats worse: There are still fully hacked ca.gov sites up and serving redirects to drug purveyors.
California state officials were caught off-guard on Tuesday when the federal GSA (General Services Administration), which manages all ".gov" domains, moved to shut down the states access to the Internet.
The GSA took the drastic action after a hacker managed to insert redirects to porn pages onto the Web site of a transportation agency for the county of Marin.
Jim Hanacek, acting deputy director for the state Department of Technology Services Policy and Planning Division, told eWEEK that he wasnt sure why the GSA decided on the widespread shutdown, but that his department wasnt notified in advance—nor, was the right person notified.
"They sent an e-mail to one of our staff thats kind of the normal contact but not necessarily for a major event like this," Hanacek said. The GSA sent the notice via e-mail around 11 a.m. PDT, but the recipient didnt see it until around noon on Tuesday.
Scrambling ensued. The Informational Technology Department activated its Emergency Operations Center, calling in high-level managers and deputies to get a consortium of expertise in to resolve the situation. It took hours and multiple phone calls from top California officials to resolve the problem, given that nobody quite knew who to call. In addition, it was after normal working hours on the East Coast, Hanacek said.
Read more here about why some foreign and domestic government sites are defenseless to hackers.
The GSA allocates sub domains to the states, and hence allocates the ca.gov domain to California. When the agency became aware that a sub domain allocated to the County of Marin was redirecting to pornographic sites, it basically removed that subdomain—an overreaction that Hanacek likened to using a shotgun to kill a flea.
The effect didnt knock the entire state government offline—at least, not right away. Rather, it was a snowballing effect that cascaded through various networks, with intermittent and random outages for Web pages and e-mail systems.
After the states Informational Technology Department began getting complaints, it moved to use a technique called force propagation to reverse the damage. That technique instructs domains to immediately update their addresses, as opposed to doing it at their normally scheduled time.
By 5:30 p.m. PDT Tuesday, the problem had been mostly cleaned up, and by 7:30 p.m. the Informational Technology Department was mopping up, monitoring to ensure that everything stayed on track, Hanacek said.
Essentially, California averted what could have been "kind of a big one," he said—an error that could have brought all Web and e-mail access down in government agencies. As it turns out, no data was compromised, and the states IT officials have been conferring on an SOA (system outage analysis) to determine what went wrong and how to avoid a repeat of the incident.
The GSA has since apologized, explaining that it was looking to protect children from lewd material. "We apologize for any inconvenience to the citizens of California. ... The potential exposure of pornographic material to the citizens—and tens of thousands of children—in California was a primary motivator for GSA to request immediate corrective action," the agency said in a statement.
As for the knee-jerk way it elbowed California off the Internet, the agency said that theres not much choice nowadays, given the risk from attackers. "In these days of heightened security concerns from hackers, it is important to quickly stop potentially harmful damage to federal, state and local Web sites from those who have no love for our country," the GSA said in the statement.
Still, to ensure it doesnt shoot any more fleas with its shotgun, the agency has also tweaked its policies. "GSA recognizes there must be a balance between protecting citizens while not, at the same time, adversely affecting governments ability to serve citizens via the Internet," the statement continued. "We have therefore revised our policies to now include more internal checks and balances before a site is shut down and to find better ways to more precisely eliminate offending government sites without having to shut down the primary site."