eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
IT problems are still affecting the Federal Aviation Administration’s flight-plan system-even after its multimillion-dollar system upgrade went online earlier in 2009.
A software configuration problem on a card in one of the routers within the FAA’s telecommunications infrastructure system in Salt Lake City shut down the flight-plan and traffic-flow system Nov. 19 for about 4 hours all over the United States starting at about 5:15 a.m. EST, the FAA said Nov. 19.
The routing problem also shut down a second node in Hampton, Ga.
Delays at Chicago’s O’Hare International Airport averaged 20 minutes, and about 30 minutes at Midway Airport across the city, the Chicago Aviation Department said in a news release. Numerous airports were affected, including airports in Atlanta, New York, Boston and Newark, N.J.
Air traffic control radar and communication with aircraft were not affected during that time and critical safety systems remained up and running, the FAA said, and there was no indication that the outage occurred as a result of a cyber-attack.
When the router defaulted, it wouldn’t allow the system to switch to a backup card, an FAA spokesperson said. The glitch forced hundreds of commercial pilots flying that day into the default mode of entering their plans manually via e-mail or by faxing them into the system, causing widespread flight cancellations and delays.
Most flight plans are routine and pre-entered as a template in the system. Pilots normally make only few changes in their altitude, speed and directional plans, depending on weather conditions and the weight of the aircraft. When the templates are not available, pilots are forced to reconstruct the entire flight plan, which is a tedious and time-consuming exercise.
This was the second time in 15 months that a problem with the flight-plan system caused major delays. The FAA, which used a mainframe system to run the flight-plan system from 1988 to 2008, switched to a new server-based system earlier in 2009 in an attempt to put an end to a flurry of problems it had been experiencing in the last five years.
The main new system for NADIN (National Airspace Data Interchange Network) is built on Stratus Technologies servers and handles all the legacy mainframe functions as well as new applications. Other parts of the system are built on open-systems server and storage infrastructure supplied by Sun Microsystems and an IP network provided by Cisco Systems.
The Nov. 19 problem, however, was not with the new data server-and-storage system but with the Federal Telecommunications Infrastructure (FTI) telecom network, which is maintained by Florida-based Harris Corp. under a $2.4 billion federal contract and supplies the pipelines for the new data center system.
The new data center system is built on heavy-duty Stratus FTserver 6400s, which run on Intel Xeon quad-core processors. The system was designed by Lockheed Martin engineers, replacing two 21-year-old Philips DS714 mainframes-located in Atlanta and Salt Lake City-that first went live in 1988 and have been cranking away ever since.
NADIN’s original mainframe-based system, an integral part of the overall NAS (National Airspace System) traffic system that processes an average of 1.5 million messages per day, served the agency well for two decades but was obsolete and was beginning to break down due to technical issues.
A team of FAA technical and safety experts is already investigating the outage, an FAA spokesperson said. FAA Administrator Randy Babbitt is meeting with representatives from Harris, a contractor that upgraded the FAA telecom network and maintains that part of the overall system, to discuss system corrections to prevent similar outages in the future.
Harris is a government contractor that makes physical communications devices, such as telephone circuits, broadcast equipment, antennae, routers and radio components. Harris’s $2.4 billion project for the FAA involved upgrading circuits that carry telephone calls and e-mail as well as information such as radar and weather data.