When Following Best Practices Is a Bad Idea
I'm sure that many of you have at least heard of the case of Terry Childs, even if you don't know the name. He's the former network administrator for the City and County of San Francisco who balked at his bosses' demands for the passwords for the networks that Childs maintained for the City, got thrown in jail for a couple of years while awaiting trial, and caught a four year sentence when he was found guilty of computer tampering. Earlier this week, a judge ordered Childs to pay the City almost $1.5 million in restitution, which even for a highly experienced Cisco-certified engineer is an awfully big chunk of change.
City officials claimed that they'd spent $900,000 or so on consultants who'd been called in to survey Childs' work following his arrest. (I have yet to hear that they found anything amiss, and nobody's explained what the extra half-million is supposed to cover, but I have to assume that he's being charged for everything related to his case, including the dry cleaning of Judge Teri Jackson's robes.) After almost a quarter-century living in San Francisco and observing how the city government works, I'm convinced that Childs was railroaded.
During the trial, which ended last August in Childs' conviction, prosecutors made a big fuss out of Childs' refusal to hand over the passwords to the City's networks to anyone short of then-Mayor Gavin Newsom, and of Child's elaborate network setup that included modems connected to the routers and switches of the network. The only problem with this scenario is that Childs was simply following Cisco's recommended best practices, albeit to an extreme degree.
To put it simply, you never, but never, ever give administrator-level access to people who don't need it, and that goes double when the people asking for those credentials don't know enable mode from a sofa cushion. From all accounts, it sounds as if Childs was - to put it mildly - the most competent Cisco engineer on the City's payroll. Given that he was tasked with maintaining the city government's most critical telecommunication infrastructure, I can see why he reached the conclusion that handing over those passwords would have been a dereliction of his duty. I know that I've only had two or three bosses in my career who truly grasped the nature of my work at its deepest levels; I can only imagine how frustrating it would be to work in an environment where one's skills were less important than meeting arbitrary criteria that were designed around political correctness instead of technical competency.
As I noted, the prosecutors depicted Childs as a crazed renegade who might be holding the City hostage, despite the lack of any evidence to the contrary. He didn't demand a raise or any other thing of value in exchange for the passwords. He was, however, adamant in his belief that incompetent people shouldn't be allowed to mismanage the network that was in many respects, his masterpiece.
Then there's the issue of the modems that Childs had rigged up to the networking hardware, which were set up to dial his home in the East Bay suburb of Pittsburg when accessed. That too is a by-the-book application of Cisco's recommended best practices. Even though Childs had grown up in Kansas, far from the unstable faults that criss-cross the Bay Area, he knew very well that in the event of a major catastrophe, it was likely that dial-in access might be the only way to access the City's network gear. Anyone who was here in 1989 can describe how difficult it was to get around the area after the Bay Bridge fell apart during the Loma Prieta quake; despite the fact that the city limits lie just beyond Yerba Buena and Treasure islands in San Francisco Bay, the City's "empire" stretches all the way to Yosemite National Park, where the Hetch Hetchy Reservoir that supplies both water and power to San Francisco is located.
One of the cardinal rules about setting up dial-in access to networking equipment is that those devices must be set up to dial back to known-secure numbers. Childs was, in effect, using his home as a network operations center of last resort; it was, in many respects, a solution to a problem that the City's disaster recovery plans were unlikely to have addressed.
Was Childs insubordinate? Of course he was. The wisest course for him to take would have been to hand over the passwords, stapled to his resignation; once the inevitable happened and less competent technicians made a hash of his work, he then could have billed the City as a consultant far more than he ever would have earned as an employee, and probably more than he now owes the City.
The last time I checked, it's no crime to treat your boss as a yutz; what Childs did was arrogant, but that's not against the law, either. What the man did was to follow best practices to the ultimate degree; in doing so, he landed himself in a lot of trouble. He's out of prison now, thanks in part to being credited with the two years that he spent behind bars, unable to post the millions of dollars that were set as his bail, and at this point he probably wishes he'd taken a job anywhere but in the San Francisco city government.
We wonder why we can't get competent people to work in the public sector, and this story explains why.