What started as a massive drop in stock prices in China on Feb. 27 eventually showed up problems in the infrastructure of several well-known brokerage firms in the United States.
The day after the Dow Jones industrial average fell 3.3 percent—the largest single-day drop since the Sept. 11, 2001 terrorist attacks—online brokerage firms were left to wonder if their Web applications would bounce back.
By midway through Feb. 28, the Dow Jones industrial average had rebounded and online brokerage companies appeared to have come back as well. However, the initial data compiled by several network and Web-application monitoring services showed serious room for improvement.
Keynote Systems, a leading performance and availability monitoring service for networks and Web applications, based in San Mateo, Calif., found several disturbing problems. According to Keynotes report, several online trading sites took twice the average length of time to complete an online stock transaction.
In addition, Keynote found there was a 25 percent drop in the overall number of successful trades that its own researchers were able to accomplish between 10:30 a.m. and 1:30 p.m. (Pacific Coast Time) on Feb. 27.
“This translates into the possibility that upwards of 25 [percent] of people attempting online trades between 10:30 a.m. and 1:30 p.m. Pacific may not have been successful,” according to Keynotes analysis, which was released on Feb. 28.
The problems for brokerage companies actually began on Tuesday, Feb. 27, when Dow Jones Indexes experienced a “system latency issue” with its computer network that would eventually cause a 70-minute lag in calculating the value of the Dow Jones industrial average.
The main problems, according to Reuters, started within the system responsible for feeding market data into the calculation index. The problem was fixed at 3 p.m. Eastern, but the Dow Jones industrial average would drop about 500 points.
According to Matt Poepsel, vice president of performance strategy at Gomez, based in Lexington, Mass., it was at this time that it started to see problems. Gomez monitors the Web applications of 25 brokerage firms.
As the news of the problems with the Dow Jones Indexes spread, customers began trying to check their accounts and orders but could not get through. The problem, Poepsel said, was that many networks could not handle that volume of traffic at once.
“If you were to check the homepage it would be fine,” Poepsel said. “Once you tried to get into the details, one customer was trying to compete against millions of other customers for very limited resources. Thats when the screens would not load and customers could not see the details of their shares.”
Normally, the brokerage industrys average response time for a market order quote process, which is the time that elapses while downloading each Web page in a multistep business process, is about 13.46 seconds, according to Gomezs benchmarks.
During the Feb. 27 breakdown, traditional brokerage firms that Gomez monitors, such as Wells Fargo and The Vanguard Group, experienced a degradation in performance of about two to eight times the normal speed. For example, Vanguard, which normally responds to an order in 11.5 seconds, took 208 seconds to respond on Tuesday, according to Gomezs measurements.
Less traditional, online firms, Poepsel said, seemed to adjust better to the onslaught from customers Feb. 27. ETrade, for example, kept its response time to about 5 seconds.
While Poepsel could not specifically say why the older firms failed to keep up with demand, he did note that issues could include a companys network capabilities, bandwidth, hardware, the codes that run the various applications, and even the physical distance between a brokerage firm and its customers.
While Tuesdays problems were not an everyday occurrence, the havoc that resulted from these network glitches should prompt brokerage firms to take a serious look at their infrastructures, Poepsel said. “You have to have the right approach to supporting these type of Web applications,” he said.
By Feb. 28, most trading had returned to normal.
“As an IT professional, you didnt know what was going to happen when you walked in today,” Poepsel said. “Things are much calmer today than yesterday but there were some residual effects. Yesterday was a crunch day.”
