eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
On Feb. 17, 2000, the America West Airlines flight-planning computer system collapsed. The airline was forced to cancel 128 flights that day and following morning, leaving thousands of passengers stranded.
Bob Ewers was headed to his Chicago home from a consulting assignment in Pomona, Calif., and had a ticket for an afternoon flight out of nearby Ontario International Airport. Instead, he found himself standing in lines for hours before being sent to a hotel for the night, then returning the next morning to stand in more lines.
“It put me back, oh, I guess about 30 hours or so,” Ewers recalls now. Yet he had some sympathy for America West employees, having worked in maintenance operations for other airlines. “They had a major failure of the computer system, and theres not too much you can do when that happens,” says Ewers, now an airplane-maintenance instructor at the Aviation Professional Education Center in West Chicago, Ill.
Others, though, swore off flying America West. “For you to cancel a lot of flights and be late all the time—thats the best way in the world to get rid of customers,” says financial analyst Ray Neidl of New York-based Blaylock & Partners.
The February 2000 system meltdown started with a simple hardware failure—a burnt-out network card at the main flight-operations center—and mushroomed. It was an acute example of the technology failures contributing to America Wests poor reputation for reliability at the time. Even when glitches didnt cause cancellations, the airline estimates they contributed to delays at the rate of 100 or more a month. Overall, the airline had one of the worst on-time performance records in the industry, ranking last among the major airlines in 1998 and 1999, according to U.S. Department of Transportation statistics.
While America West had reported record earnings in 1999, things went sour in 2000 on the financial front. Narrowing margins gave way to losses that started in the fourth quarter of 2000 and continued through the first quarter of 2003.
Even though its troubles began well before the trauma of the 9/11 terrorist attacks, America Wests operations started to show signs of improvement in late 2001: It had the best on-time performance between December 2001 and February 2002, and again in May 2002. Today, America Wests on-time performance is consistently above the industry average. One reason is the improved reliability of its supporting information systems, particularly those for the companys Strategic Operations Center—responsible for all flights nationwide—and its main hub in Phoenix.
Technology-related flight delays have dropped to about 25 or 30 per month, including relatively minor problems such as jammed printers, says chief information officer Joe Beery. He takes care to say that technology failures were never the top cause of delays, and that his departments improvements are part of a broader effort to improve operations. Still, Beery suggests America West ought to be able to take control of its systems and get that score for technology-related delays to zero. “Were not there yet, but were getting there,” he says.
So far, most progress has been achieved by eliminating the network infrastructures weaknesses. “We essentially never have system crashes now, where we used to have them frequently,” says Scott Kirby, the airlines executive vice president of sales and marketing.
America Wests improvement in operations, whether due to better systems or more responsive employees, is paying off, says Michael Boyd, an airline-industry analyst with the Boyd Group. Thats why America West is edging back into profitability at a time when most airlines are still suffering losses, he says. “If youre reliable, people will fly you again.”
The results are now evident: America West earned $79.7 million in the second quarter and $32.9 million in the third. And while America West still expects to lose money for the full year, it forecasts a profit for 2004.
Next Page: Raising the confidence level.
Raising the Confidence Level
Raising the Confidence Level
On one side of the room at the Strategic Operations Center (SOC) in Phoenix, dispatchers deal with last-minute changes to flight plans, creating the critical “paperwork” that America West aircraft need to be cleared for departure from airports around the U.S. Across the room, maintenance coordinators track progress on service and repairs, and crew coordinators reassign personnel when a flight is rescheduled or a pilot calls in sick.
In 2002, the SOC was relocated from the Phoenix Sky Harbor International Airport to a new $35 million building nearby that also houses simulators for America Wests flight-training program. Getting it off what Beery describes as the airports “rickety” network infrastructure was part of the effort to improve reliability. If the computer systems supporting this operation fail, planes and passengers sit idle across the country.
One problem with the airport was that it used a token-ring network in which computers are wired together in a circle, a topology thats been declining in popularity for years. One disadvantage is that the network can be disrupted by a break anywhere in the ring. Most organizations now use Ethernet networks in which each computer connects to a hub or switch.
In 2000, when a network card failed in the database server for flight planning, America Wests systems personnel discovered they had no spare on hand and couldnt quickly get a token-ring card that fit the servers proprietary hardware interface.
From there, the problem snowballed. Even after network functions were restored, it took time to restart airline operations, given that many planes werent where they needed to be. That meant the entire operations staff had to scramble to adjust flight, staffing and maintenance plans.
Today, the network is better protected. In May 2002, the SOC applications were switched from the servers at the airport to more-powerful Hewlett-Packard DL-380s in a nearby data center. Those servers are equipped with two network cards, each with an independent network connection, meaning that a server can still function if one card fails. Six weeks later, the operations center itself was relocated from the airport into a wing of the new flight-training center, just outside the airport. Now all workstations are connected to a 100-megabit Ethernet network, which is faster and more dependable.
On a tour of the SOC in September, Beery strolls the perimeter, pointing out the dual flat-panel display terminals at each desk that reveal detailed information at a glance. He spirals into the center of the room where SOC manager John A. Anderson, sitting behind a bank of three screens, monitors the activities of everyone in the room.
Andersons workstation makes it easy for him keep the airline operating smoothly. He toggles between air-traffic control screens, weather maps, aircraft-maintenance status and performance metrics, navigating to each application through a Web framework based on Computer Associates CleverPath portal server. Most important, he can rely on the system.
“If my dispatchers cant get the paperwork out of here, the planes cant go,” Anderson says. Today, work is rarely interrupted by a system outage.
The new network also cuts the time it takes a dispatcher to recalculate a flight plan from 25 or 30 seconds to 5 or 6. “When we push that button to make the calculation, speed is extremely important because we may go through that several times” before producing a flight plan thats agreeable to the pilot and to air-traffic control, Anderson says.
Next Page: Making connections with artificial inteliigence software techniques.
Making Connections
Making Connections
The SOC is one of two critical chokepoints that can either prevent delays (when functioning properly) or cause them (when it goes offline). The other is the hub operations room that still resides within the Phoenix airport. While the SOCs focus is nationwide, hub operations boil down to juggling gate assignments for incoming planes, getting departures off on time and trying to ensure that passengers make their connections. As America Wests main hub, Phoenixs efficiency impacts most of the airlines passengers.
Hub operations stayed within the airport, but America West upgraded the local-area network and moved its servers to the data center. The other big step America West took with hub applications was to switch server vendors.
In 2000, America West implemented sophisticated new software to help hub operations staff optimize gate assignments. The gate-manager software from Ascent Technology of Cambridge, Mass., uses artifical-intelligence techniques to divine the best way to squeeze planes into the airports limited real estate. For example, the software has to be able to account for the fact that the wingspan of larger jets means that two of them cant be parked next to each other at adjacent gates. It also helps hub operations personnel juggle gate assignments when planes arrive early or late.
This replaced a more chaotic system in which a room full of people would watch while one person standing on a platform moved around magnetic markers representing planes on a white board with a map of the airport.
However, gate management was one of several key hub operations applications on a Sun E10000 Starfire server that kept crashing. The problem was a manufacturing flaw related to the particular generation of Suns UltraSparc processors used in the Starfire and other Sun computers, and the associated cache memory subsystem. “It was called the flipped bit problem,” Beery says, spitting out the words with distaste.
It happened when cosmic rays—high-energy radiation from the heavens—called alpha particles zapped electrons off course within the microprocessor. That caused electronic registers that represent bits of data or logic to “flip” 1 to 0 or 0 to 1—and made the operating system “panic.” About once a month, applications like gate management were knocked offline for an hour or more each time.
Sun acknowledged the problem in 2000. In response, it implemented “mirroring” of static random access memory (SRAM) chips so that, even if operating-system instructions are corrupted, the server can recover. Today, Sun believes the issue is irrelevant to its current offerings, having been solved more than two years ago. Sun spokeswoman Cinthia Portugal also emphasizes that “bit flips are not unique to Sun.”
Still, the incident convinced America West to stop paying a premium for Sun servers. Over the past couple of years, the airline has migrated applications to Fujitsus PrimePower servers, which run on Sparc-compatible processors and Suns Solaris operating system.
America West purchased its first Fujitsu server in late 2001 and expanded the relationship to include system-management services for all Unix servers, including Sun products. When Fujitsu suggested its own hardware as an alternative to Suns, Beery was ready to listen. He had joined America West as application development director in 1999 and had just been promoted to chief information officer in 2001. Even in his earlier development role, he had been annoyed by the server problems because “those were my applications that were crashing.”
Beery says his team has learned to lower the cost of operating its information systems, even while improving reliability. He declined to release specific technology-budget numbers, but says the effect can be seen in the airlines overall performance. America West has whittled its cost per available seat mile (casm) from 8.20 cents at the end of 2002 to 7.57 cents in the three months ended Sept. 30. In that same quarter, Southwest and American Airlines reported casms of 7.51 cents and 9.43 cents, respectively.
Beery has no secret recipe, other than persistence and attention to detail. “Its about doing the right things with the tools you have, staying focused and working with some of the suppliers,” he says—and about making sure it all adds up to better performance.
Next Page: Amercia Wests Base Case.
America West Base Case
America West Base Case
Headquarters: 111 W. Rio Salado Parkway, Tempe, AZ 85281
Phone Number: (480) 693-0800
Business: Discount airline with hubs in Phoenix and Las Vegas.
Chief Information Officer: Joe Beery
Financials in 2002: $2 billion, with a net loss of $387.9 million.
Challenge: Improve reliability and cut information-systems costs, as part of an effort to improve the airlines on-time performance.
Baseline Goals:
- Reduce, and eventually eliminate, flight delays attributed to technology, which averaged about 100 per month three years ago.
- Drive down cost per available seat mile (CASM). It went from 8.20 cents at the end of 2002 to 7.57 cents as of September 2003.
- Match Southwest Airlines third-quarter CASM of 7.51 cents.
- Return to profitability, following 10 consecutive quarters of losses since the end of 2000. America West earned $2.02 per share in the second quarter of 2003.