Corrupt File Brought Down FAA's Antiquated IT System

The Federal Aviation Administration's flight plan IT network, which went down for about 2.5 hours Aug. 26 and fouled up the takeoff plans of thousands of travelers in more than 40 airports across the country, was back up and running Aug. 27.

IT staff were still troubleshooting it today in Hampton, Ga., where the agency's primary data center is located.

But for how much longer is it going to be running? The FAA's antiquated system consists of two 20-year-old redundant mainframe configurations—the primary one in Georgia, the backup in Utah—that apparently are hanging on for dear life until reinforcements arrive in the form of a new, state-of-the-art system this winter.

Resource Library:

It is intriguing to note that the company that custom-built the mainframes for the FAA has been out of business for 20 years. More on that shortly.

The Crash

"What happened yesterday at 1:25 p.m. [EDT] was that during a normal daily software load something was corrupted in a file, and that brought [the] system down in Atlanta," FAA spokesperson Paul Takemoto told me.

"Basically, all the flight plans that were in the system were kicked out. For aircraft already in the air, or [that] had just been pushed back from the gate, they had no problems. But for all other aircraft, it meant delays."

What made things worse was when operations were shifted to the backup facility in Salt Lake City, which is designed to handle 125 percent of the overall load, Takemoto said.

"It was far more than that [125 percent], because airlines were refiling their flight plans manually. They just kept hitting the 'Enter' button. So the queues immediately became huge," Takemoto said. "On top of that, it happened right during a peak time as traffic was building. Salt Lake City just couldn't keep up."

It was a "perfect storm" combination of all these flight plans being refiled plus a congested time of day and a creaky old IT system that caused the airport backups, Takemoto said.

The FAA then instructed the airlines not to file any flight plans for a specified length of time, and that left many passengers sitting and waiting in terminals. By around 4 p.m., Takemoto said, things started clearing up and the system came back to life.