Computer Associates’ Top Secret security product for the mainframe blacked out worldwide on June 16, staying dark for 19 hours and bringing down financial institutions such as banks and insurance systems.
CA said in a statement that the bug affected approximately 50 customers worldwide and did not introduce any security issues. “It prevented a subset of CICS users from signing on during a 19-hour period (from 6/16 to 6/17) because of an internal memory representation of the time/date value, which caused the host to deny the sign-on request,” according to the statement.
CA said that there’s no history of this kind of issue with CA Top Secret and CA ACF2. Affected releases for both products are Release 9 SP01+. Fixes are available at CA Support Connect.
A high-level IT professional who requested anonymity said that his organization has had no problems with Top Secret 9 in the nine+ years he’s personally been working with it.
“Top Secret has been very stable other than this,” he said. “My key mainframe manager has 30 years experience on it. … [There have been] no major issues, so it’s just very suspicious, especially with all the issues at CA in the last year.”
One question the IT professional had about the outage: Given CA’s lousy year, could the problem have been man-made by a disgruntled employee? Or was it a latent daylight savings time bug?
CA didn’t respond to such questions, telling users that the outage was due to a date bug in Top Secret Version 9, the current version.
“It was basically a total outage,” the source said in an e-mail exchange. “Business users were completely unable to get into the system. That is why it was so bad. Operations and Admin people could get in but no real users. I’m sure it was the same for everyone else.”
Top Secret runs on z/OS, z/VM, z/VSE, z/OS UNIX, Linux for zSeries, DB2, IMS and CICS environments.
The blackout of CICS (Customer Information Control System), a transaction server that runs primarily on IBM mainframe systems under z/OS or z/VSE, is a particularly serious business. CICS is a transaction processing system designed for both online and batch activity. It’s used in bank teller applications, airline reservation systems, ATM systems and in other financial applications. According to Wikipedia, while CICS is most visible among banks and insurance companies, over 90 percent of Fortune 500 companies also use it for core business functions, as do most state and national governments.
No users could sign into CICS after the bug hit, the source said, although the error logs were completely free of any messages related to the issue. Existing users could continue until they signed off, he said.
CA provided a fix just after 5:30 p.m. ET on Saturday.
What makes the outage particularly serious is that disaster-recovery plans wouldn’t have worked, the source pointed out, given that back-up systems run Version 9.
Uninstalling Version 9 and reinstalling Version 8 wasn’t much of an option when it comes to workarounds, he said, given the risks and the time involved: over 18 hours, he estimated.
This story was updated to incorporate input from CA.