FAA Flight-Plan System Crashes Again, Delays Hundreds of U.S. Flights

By Chris Preimesberger  |  Posted 2009-11-19 Print this article Print

A software configuration problem in one of the routers within the FAA's telecommunications infrastructure system in Salt Lake City shut down its flight-plan and traffic-flow system Nov. 19 for about 4 hours all over the United States, the FAA says. The routing problem also shut down the system's second node in Hampton, Ga.

IT problems are still affecting the Federal Aviation Administration's flight-plan system-even after its multimillion-dollar system upgrade went online earlier in 2009.

A software configuration problem on a card in one of the routers within the FAA's telecommunications infrastructure system in Salt Lake City shut down the flight-plan and traffic-flow system Nov. 19 for about 4 hours all over the United States starting at about 5:15 a.m. EST, the FAA said Nov. 19.

The routing problem also shut down a second node in Hampton, Ga.

Delays at Chicago's O'Hare International Airport averaged 20 minutes, and about 30 minutes at Midway Airport across the city, the Chicago Aviation Department said in a news release. Numerous airports were affected, including airports in Atlanta, New York, Boston and Newark, N.J.

Air traffic control radar and communication with aircraft were not affected during that time and critical safety systems remained up and running, the FAA said, and there was no indication that the outage occurred as a result of a cyber-attack.

When the router defaulted, it wouldn't allow the system to switch to a backup card, an FAA spokesperson said. The glitch forced hundreds of commercial pilots flying that day into the default mode of entering their plans manually via e-mail or by faxing them into the system, causing widespread flight cancellations and delays.

Most flight plans are routine and pre-entered as a template in the system. Pilots normally make only few changes in their altitude, speed and directional plans, depending on weather conditions and the weight of the aircraft. When the templates are not available, pilots are forced to reconstruct the entire flight plan, which is a tedious and time-consuming exercise.

This was the second time in 15 months that a problem with the flight-plan system caused major delays. The FAA, which used a mainframe system to run the flight-plan system from 1988 to 2008, switched to a new server-based system earlier in 2009 in an attempt to put an end to a flurry of problems it had been experiencing in the last five years.

The main new system for NADIN (National Airspace Data Interchange Network) is built on Stratus Technologies servers and handles all the legacy mainframe functions as well as new applications. Other parts of the system are built on open-systems server and storage infrastructure supplied by Sun Microsystems and an IP network provided by Cisco Systems.

The Nov. 19 problem, however, was not with the new data server-and-storage system but with the Federal Telecommunications Infrastructure (FTI) telecom network, which is maintained by Florida-based Harris Corp. under a $2.4 billion federal contract and supplies the pipelines for the new data center system.

The new data center system is built on heavy-duty Stratus FTserver 6400s, which run on Intel Xeon quad-core processors. The system was designed by Lockheed Martin engineers, replacing two 21-year-old Philips DS714 mainframes-located in Atlanta and Salt Lake City-that first went live in 1988 and have been cranking away ever since.

NADIN's original mainframe-based system, an integral part of the overall NAS (National Airspace System) traffic system that processes an average of 1.5 million messages per day, served the agency well for two decades but was obsolete and was beginning to break down due to technical issues.

A team of FAA technical and safety experts is already investigating the outage, an FAA spokesperson said. FAA Administrator Randy Babbitt is meeting with representatives from Harris, a contractor that upgraded the FAA telecom network and maintains that part of the overall system, to discuss system corrections to prevent similar outages in the future.

Harris is a government contractor that makes physical communications devices, such as telephone circuits, broadcast equipment, antennae, routers and radio components. Harris's $2.4 billion project for the FAA involved upgrading circuits that carry telephone calls and e-mail as well as information such as radar and weather data.

Chris Preimesberger Chris Preimesberger was named Editor-in-Chief of Features & Analysis at eWEEK in November 2011. Previously he served eWEEK as Senior Writer, covering a range of IT sectors that include data center systems, cloud computing, storage, virtualization, green IT, e-discovery and IT governance. His blog, Storage Station, is considered a go-to information source. Chris won a national Folio Award for magazine writing in November 2011 for a cover story on Salesforce.com and CEO-founder Marc Benioff, and he has served as a judge for the SIIA Codie Awards since 2005. In previous IT journalism, Chris was a founding editor of both IT Manager's Journal and DevX.com and was managing editor of Software Development magazine. His diverse resume also includes: sportswriter for the Los Angeles Daily News, covering NCAA and NBA basketball, television critic for the Palo Alto Times Tribune, and Sports Information Director at Stanford University. He has served as a correspondent for The Associated Press, covering Stanford and NCAA tournament basketball, since 1983. He has covered a number of major events, including the 1984 Democratic National Convention, a Presidential press conference at the White House in 1993, the Emmy Awards (three times), two Rose Bowls, the Fiesta Bowl, several NCAA men's and women's basketball tournaments, a Formula One Grand Prix auto race, a heavyweight boxing championship bout (Ali vs. Spinks, 1978), and the 1985 Super Bowl. A 1975 graduate of Pepperdine University in Malibu, Calif., Chris has won more than a dozen regional and national awards for his work. He and his wife, Rebecca, have four children and reside in Redwood City, Calif.Follow on Twitter: editingwhiz

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel