Corrupt File Brought Down FAA's Antiquated IT System

 
 
By Chris Preimesberger  |  Posted 2008-08-27 Email Print this article Print
 
 
 
 
 
 
 

The FAA's (Federal Aviation Administration) flight plan IT network, which went down for about 2.5 hours Aug. 26 and fouled up the takeoff plans of thousands of travelers in more than 40 airports across the country, is back up and running. But for how much longer? The antiquated system consists of two 20-year-old redundant mainframe configurations-one in Georgia, one in Utah-that apparently are hanging on for dear life.

The Federal Aviation Administration's flight plan IT network, which went down for about 2.5 hours Aug. 26 and fouled up the takeoff plans of thousands of travelers in more than 40 airports across the country, was back up and running Aug. 27.

IT staff were still troubleshooting it today in Hampton, Ga., where the agency's primary data center is located.

But for how much longer is it going to be running? The FAA's antiquated system consists of two 20-year-old redundant mainframe configurations-the primary one in Georgia, the backup in Utah-that apparently are hanging on for dear life until reinforcements arrive in the form of a new, state-of-the-art system this winter.

It is intriguing to note that the company that custom-built the mainframes for the FAA has been out of business for 20 years. More on that shortly.

The Crash

"What happened yesterday at 1:25 p.m. [EDT] was that during a normal daily software load something was corrupted in a file, and that brought [the] system down in Atlanta," FAA spokesperson Paul Takemoto told me.

"Basically, all the flight plans that were in the system were kicked out. For aircraft already in the air, or [that] had just been pushed back from the gate, they had no problems. But for all other aircraft, it meant delays."

What made things worse was when operations were shifted to the backup facility in Salt Lake City, which is designed to handle 125 percent of the overall load, Takemoto said.

"It was far more than that [125 percent], because airlines were refiling their flight plans manually. They just kept hitting the 'Enter' button. So the queues immediately became huge," Takemoto said. "On top of that, it happened right during a peak time as traffic was building. Salt Lake City just couldn't keep up."

It was a "perfect storm" combination of all these flight plans being refiled plus a congested time of day and a creaky old IT system that caused the airport backups, Takemoto said.

The FAA then instructed the airlines not to file any flight plans for a specified length of time, and that left many passengers sitting and waiting in terminals. By around 4 p.m., Takemoto said, things started clearing up and the system came back to life.



 
 
 
 
Chris Preimesberger Chris Preimesberger was named Editor-in-Chief of Features & Analysis at eWEEK in November 2011. Previously he served eWEEK as Senior Writer, covering a range of IT sectors that include data center systems, cloud computing, storage, virtualization, green IT, e-discovery and IT governance. His blog, Storage Station, is considered a go-to information source. Chris won a national Folio Award for magazine writing in November 2011 for a cover story on Salesforce.com and CEO-founder Marc Benioff, and he has served as a judge for the SIIA Codie Awards since 2005. In previous IT journalism, Chris was a founding editor of both IT Manager's Journal and DevX.com and was managing editor of Software Development magazine. His diverse resume also includes: sportswriter for the Los Angeles Daily News, covering NCAA and NBA basketball, television critic for the Palo Alto Times Tribune, and Sports Information Director at Stanford University. He has served as a correspondent for The Associated Press, covering Stanford and NCAA tournament basketball, since 1983. He has covered a number of major events, including the 1984 Democratic National Convention, a Presidential press conference at the White House in 1993, the Emmy Awards (three times), two Rose Bowls, the Fiesta Bowl, several NCAA men's and women's basketball tournaments, a Formula One Grand Prix auto race, a heavyweight boxing championship bout (Ali vs. Spinks, 1978), and the 1985 Super Bowl. A 1975 graduate of Pepperdine University in Malibu, Calif., Chris has won more than a dozen regional and national awards for his work. He and his wife, Rebecca, have four children and reside in Redwood City, Calif.Follow on Twitter: editingwhiz
 
 
 
 
 
 
 

Submit a Comment

Loading Comments...
 
Manage your Newsletters: Login   Register My Newsletters























 
 
 
 
 
 
 
 
 
 
 
Rocket Fuel