Corrupt File Brought Down FAA's Antiquated IT System (
Page 1 of 2 )
The Federal Aviation Administration's flight plan IT network, which went
down for about 2.5 hours Aug. 26 and fouled up the takeoff plans of thousands
of travelers in more than 40 airports across the country, was back up and
running Aug. 27.
IT staff were still troubleshooting it today in Hampton,
Ga., where the agency's primary data center
is located.
But for how much longer is it going to be running? The FAA's antiquated system
consists of two 20-year-old redundant mainframe configurations—the primary one
in Georgia, the backup in Utah—that apparently are hanging on for dear life
until reinforcements arrive in the form of a new, state-of-the-art system this
winter.
It is intriguing to note that the company that custom-built the mainframes for
the FAA has been out of business for 20 years. More on that shortly.
The Crash
"What happened yesterday at 1:25 p.m.
[EDT] was that during a normal daily
software load something was corrupted in a file, and that brought [the] system down
in Atlanta," FAA spokesperson
Paul Takemoto told me.
"Basically, all the flight plans that were in the system were kicked out.
For aircraft already in the air, or [that] had just been pushed back from the
gate, they had no problems. But for all other aircraft, it meant delays."
What made things worse was when operations were shifted to the backup
facility in Salt Lake City, which is designed to handle 125 percent of the
overall load, Takemoto said.
"It was far more than that [125 percent], because airlines were refiling
their flight plans manually. They just kept hitting the 'Enter' button. So the
queues immediately became huge," Takemoto said. "On top of that, it
happened right during a peak time as traffic was building. Salt
Lake City just couldn't keep up."
It was a "perfect storm" combination of all these flight plans being
refiled plus a congested time of day and a creaky old IT system that caused the
airport backups, Takemoto said.
The FAA then instructed the airlines not to file any flight plans for a
specified length of time, and that left many passengers sitting and waiting in
terminals. By around 4 p.m., Takemoto
said, things started clearing up and the system came back to life.
IT & Network Infrastructure News & Reviews 
