As the American public responded to the devastation of Hurricane Katrina with unprecedented generosity, the donations not only overwhelmed survivors and aid workers but also nearly overwhelmed the IT infrastructure of the American Red Cross.
Upgrading online donation capacity in the wake of a disaster is par for the course at the Washington-based charity and relief organization. In the days following the terrorist attacks of Sept. 11, 2001, the organization added transaction servers.
After the massive tsunami in Southeast Asia last December, the Red Cross faced unheard-of (at that point) online donation traffic. The IT staff worked long hours and still offloaded some of the transaction processing to technology partners.
But the donation system wasnt ready for Hurricane Katrina, and that was clear as soon as the New Orleans levees broke.
“As soon as we understood the magnitude of the tragedy, we knew the money would be coming in,” said Dave Clarke, chief technology officer at the Red Cross. “When we began to see the initial transaction volume, we determined that if it continued on that growth curve, we would run out of capacity. And we knew we had to get ready.”
The demand on the servers was unprecedented, Clarke said. Some 50 IT staff members worked around the clock, tripling the capacity of the Internet link and then doubling that capacity again. The group added more servers, but it soon became clear that a complete system overhaul was in order.
“We saw a very large upswing in demand in our online donation systems,” Clarke said. “Although we design our systems to be scalable, and they have been scalable in other relief efforts, the large demand we saw in this case caused us to redesign our public Web-facing infrastructure in about 48 hours. We had built a virtual Web farm designed to host several systems, and we found there were limitations to that, even by adding servers to it. What has amazed me has been the immense talent and dedication of our staff and their ability to do the seemingly impossible almost overnight. There are a lot of heroes in this whole story.”
Two days later, the team had upgraded the network from an initial five servers, shared between the main Web site and the online donation system, to an eventual 12 Web servers and one database server, both dedicated to the donation system, and five servers dedicated to the main Web site. In addition, the team upgraded the Internet links from a single 90MB DS-3 to two 155MB OC-3s. It also added two dedicated DS-3 links to communicate with shelters in areas affected by Katrina.
To handle the traffic, the company offloaded much of it to Akamai Technologies Inc. Then the Red Cross IT staff pulled apart the content management and Web donation applications that had been hosted as a group and gave each its own Web farm instead. “We had brief periods of downtime several times as we were struggling with the load, but in all cases it was episodic,” Clarke said.
One of Clarkes lessons learned from the ordeal was that it might be a good idea to install a collaboration system. Following the system upgrade, the organization deployed file-sharing software from Groove Networks as well as SharePoint software from Microsoft Corp., which recently bought Groove. “Coordination is a major aspect of any relief operation, particularly of this size, so we do intend to develop and deploy a long-term collaboration solution to support our activities,” Clarke said.
Akamai was still hosting static traffic for the Red Cross at the end of last week, but plans for continued partner hosting are unclear.