A foul-up at RBC Financial Group, Canadas largest bank, stifled payroll deposits and left as many as 10 million customers uncertain of their bank account balances in June, 2004. That has prompted an internal review of the banks technology and processes, and will likely result in millions of dollars in damages.
The problem, which took close to two weeks for RBC to correct, began on May 31 when a single worker introduced a “relatively small number” of faulty pieces of code into the banks transaction processing system, which then began issuing error messages to users. In press reports, RBC chief information officer Martin Lippert chalked up the incident to human error.
Nevertheless, it serves as a warning to other banks that they need processes to ensure that proper procedures for updating software are in place and rigidly enforced. They should also expect to compensate customers when errors occur, say analysts. .
RBCs woes also made it a target for hackers. Fraud operators took advantage of the computer glitch to launch a major phishing attack against the Toronto-based banks customers over the Internet.
In the “Dear RBC Royal Bank Customer” e-mail, what looked like an official request asked for names, account numbers and personal identifiers to verify customers standing due to “increased fraudulent activity.” Once a person clicked on the e-mail, went to a spoof site and entered information, hackers could access those accounts.
“Those of us who know how banks operate understand that this type of error shouldnt have happened,” says Paul K. Wing, an industry analyst and former head of the Bank of Nova Scotias information-technology security division. “But the question that needs to be asked is, why werent there controls in place to prevent it?”
RBC controls should have required thorough testing of new code, restricted access to systems and required work be done during off-peak periods. Wing says it is almost unheard of in banking circles for new code to be entered during the work week, when transactions are at a peak.
Check out eWEEK.coms Developer & Web Services Center at http://developer.eweek.com for the latest news, reviews and analysis in programming environments and developer tools.
The banks protocols do call for all new pieces of code to be thoroughly tested, but RBC says the code entered production without being tested “as well as it should have been.” RBC wouldnt reveal whether the worker was disciplined, and Lippert was unavailable. However, the bank did say it did not suspect foul play.
The error was identified and corrected in two hours, but still triggered a series of events that took days to unravel. The new code, which was described as pertaining to bank transit numbers (numbers that identify individual branches), entered into production early in the morning on June 1.
The error caused the banks core account transaction processing software to issue error messages. Bank accounts for some 10 million customers could not be properly updated, and payroll deposits could not be processed. RBC handles payroll deposits for clients at other financial institutions, and those deposits were also delayed.
The bank decided not to restart the software to complete the overnight updating process, to ensure that other banking systems would not be affected. That meant the bank essentially had two days worth of transactions to process in one day on June 2. However, as transactions piled up, some dated May 31 and some June 1, RBCs software had difficulty rectifying the multiple dates, forcing employees to process transactions manually.
The bank admitted that the manual intervention only delayed the recovery. It wasnt until the weekend—four days after the bad code had been entered into the system—that it was finally able to get on top of the problem.
RBC has not yet put a dollar figure on the damage. It was considering waiving service fees for the month, a move that could result in about $165 million Canadian ($123 million U.S.) in lost revenues alone.
The bank also faces a class-action lawsuit filed in Quebec, seeking damages of $500 for each customer impacted by the glitch.
Be sure to add our eWEEK.com developer and Web services news feed to your RSS newsreader or My Yahoo page