Once upon a time in San Francisco, there was a network administrator, one Terry Childs, who had pretty much architected the city’s IT network by himself. Childs is now headed to state prison with a four-year sentence for holding the network hostage, but since he’s spent the last two years in jail, he’s probably going to be a free man sometime in the early months of 2011. His story is a reminder of why securing IT infrastructure is never a job for one person.
To summarize events, Childs had built himself a nice job in the city’s IT department as the resident expert on all things Cisco, and in the summer of 2008 he was about to be transferred to other duties. He refused to turn over his passwords for the FiberWAN network, and was summarily arrested, charged with computer tampering and thrown in jail with bail set at $5 million, a level more commonly seen in the case of kidnappers and murderers.
From all accounts, Childs identified a little too closely with his creation, and was as protective of it as a mama grizzly is of her cubs. This isn’t surprising, considering that the man had been on what amounted to 24/365 call for more a few years, as San Francisco’s Department of Telecommunications and Information Services downsized and reorganized, leaving him effectively the only person on staff who had a clue about network operations.
So it’s understandable how this came to pass: Childs didn’t want incompetents mismanaging the network he had built, and he viewed his bosses as technological dimwits. To be honest, I have to agree with his appraisal. This has less to do with his managers’ knowledge of Cisco IOS, and more to do with common sense, because the bosses violated Rule No. 1 of IT management: Never, but NEVER, put mission-critical systems in the hands of just one employee.
Even 15 or 20 years ago, when I was an up-and-coming single-handed sysadmin, I knew better than to let myself become the single point of failure; Childs seems to have relished the role. Even though my bosses at the time had a fraction of my knowledge of our employer’s critical systems, I had to trust somebody, and I made sure that some simple continuity plans were in place so that one of those somebodies could step into my shoes if necessary.
Most of the time, these took the form of a sealed envelope in my HR file containing the current admin passwords, in case my luck ran out and a Muni driver decided to park a bus on me. If I took so much as an afternoon off, I made sure there was at least one other person (in an organization with a head count of about 50) who could restart production systems without my presence.
Even though Childs is the person who ran afoul of the law, his managers bear much of the blame for allowing things to get to the point they reached. His bosses don’t appear to have been at all effective in establishing their authority or creating a healthy work environment, and in my book, that’s some of the worst management there is. I’ve worked for some real jerks in my day, but the only managers whose names I spit on are the ones who talked big but couldn’t deliver on promises, or who handed me responsibility without authority.
Terry Childs may have become a monster in the process of implementing San Francisco’s FiberWAN, but the city’s IT managers created that monster by giving Childs free rein with the network. They were shocked that the monster had his own ideas about who was calling the shots, though anyone who’s familiar with the plots of “Frankenstein” or “Godzilla” could have told them that this situation was ripe for disaster.