eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
When holiday shopping hits the frenzy level, Visa International will be well-prepared to handle an expected peak load of 100 million transactions per day—as many as 6,200 per second—thanks to its largest system upgrade ever.
The rollout of the upgrade last month included an unprecedented number of enhancements and applications to Visas payment processing system. It was the successful culmination of an effort that took nine months of intense planning and the efforts of 300 IT professionals, and involved approximately 120,000 code changes, 150,000 worker hours and a revamping of 27 major areas of the system.
In addition to the nine-month planning process, Visa also dedicated several months to internal-systems testing at partner sites and as much as eight weeks of quality assurance testing to ensure the upgrade could be implemented without a hitch.
The upgrade—one of two that occur yearly at the worlds largest credit card brand—will allow the Foster City, Calif., company to provide new services to its merchants and bank partners while ensuring that its network remains flexible and reliable, officials said. (Security measures are added as separate, more frequent updates, they said.)
With research company Forrester Research Inc., of Cambridge, Mass., predicting online holiday sales will increase 20 percent over last year, to reach $13.2 billion this year, new services included in this latest update will be more important than ever.
“We make a promise to cardholders that our global system will work 24 hours a day, even while a great deal of change is being implemented,” said Scott Thompson, executive vice president of Inovant LLC, Visas technology solutions division.
“With the magnitude of the release, things could go wrong, but because we do testing and QA, we have a good sense going into the implementation what our risks are. It really all comes down to planning and execution,” Thompson said.
Visa upgrades its VisaNet systems twice a year, in April and in October. Because upgrades are conducted across all four of the companys data centers (London, Tokyo and two in North America), the major challenge is ensuring the network can continue to process transactions as the upgrade occurs.
This involves a delicate balancing act because two versions of VisaNet—the old and the new—are running simultaneously throughout the 24-hour upgrade period, and the network has to be designed to process transactions originating from either the older or newer systems.
At the same time, Visa has to ensure that its member banks successfully complete the upgrade on their end.
Visa Upgrades Mean Business
“In any other organization, all four system images would be taken down at the same time, the new system would be installed and then each data center would be brought back online one at a time,” Thompson said. “The challenge we face is ensuring our network keeps working 24 hours a day while a great deal of change is being implemented.”
At Visa, the upgrade process begins with the planning stage, which starts nine months before the scheduled upgrade. During this phase, Visas internal IT team receives requests from 21,000 Visa member banks globally for new services, then begins building the corresponding programming code, using mainly Assembler.
Months before the changes are implemented, Visa alerts its partners of the planned enhancements and provides technical specifications so that the banks connected to its network can make changes along with Visa.
“In most organizations, you can go backwards in a project if the release isnt turning out the way you had hoped. We cant do that here. We have to implement to coincide with the changes our partners are making. There is no option for fallback or failure,” Thompson said.
For the system upgrade that went live last month, Visa included enhancements to speed the credit card dispute-resolution process, to simplify currency conversions and to update Visas loyalty-points program.
The new applications will not only allow Visa to process more transactions but will also enable member banks to use technology more efficiently, said Robin Owens, division head for Visas Member Information Clearing and Settlement, in Foster City.
For example, the automation of disputed transactions will significantly reduce the costs for card issuers by resolving cardholder disputes faster. In 2002, the initial implementation of this technology helped reduce chargeback by 21 percent and saved Visas merchants $238 million.
After code is compiled by 300 engineers using Assembler, COBOL, C, C++ and .Net, the company spends six to eight weeks on quality assurance testing. A testbed is also built so IT staffs from member banks can test internal systems against the VisaNet changes. Visa prepares systems to accept previous versions of transactions in the event a member bank has problems with the upgrade.
The global deployment, which began Oct. 1, took 24 hours. Upgrades are managed and monitored from a central command center in Foster City and from a local command facility in each data center. The central command center issues two update calls and circulates multiple written updates to keep the staff in sync. Modules are loaded into the system on a schedule so that everyone knows exactly when the module is added to the system. Visa does not deviate from that schedule, Owens said.
Visa was already planning its April 2005 release during testing of the October update. Owens said the April release will be nowhere as large as the October upgrade, but the same amount of planning and quality assurance testing will still be involved.
“While were not NASA or working on a cure for cancer, we provide technical people with a challenge they can thrive on,” Thompson said. “The key to our success is the fact that were blessed with solving these challenging problems, and were able to attract talent who can come up with a solution.”
Senior Writer Anne Chen can be reached at [email protected].