eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Oracle Corp. is sick of it. While Microsoft Corp. and its developers quote chapter and verse of its Security Development Lifecycle blueprint for software creation—a set of procedures that spans code design, development, testing and deployment to build secure software—database king Oracle has been mum about its secure coding initiatives. Customers, however, have been vocal about Oracles poor quality of patches and vulnerabilities left unplugged for too long.
“Part of the reason there are so many [Oracle] patches is directly reflective of the poor quality of the code,” said Dan Downing, vice president of testing services at business applications testing, hosting and managing provider Mentora Group Inc., of Atlanta.
Larger customers tend to be happy that their Oracle databases are safely tucked behind firewalls. “I have been patching Oracle systems for 10 years,” said Howard Fosdick, former president of the Midwest Database Users Group. “My general idea is theyve always done a good job: reasonably timely, reasonably accurate.”
Oracles own customer advisory group, however, has brought up patch quality and, above all else, patch timing, and Oracle is focusing on those two tasks.
Oracle opens up
Oracles recent history includes patches that havent installed correctly, patches to patch patches and then more patches to fix the cumulative security ills.
All the while, Oracle has had a no-comment, protect-our-customers policy on security issues. But thats changing. In an exclusive interview, Oracle executives explained that the company is experimenting with being more open about security flaws. Part of that new openness includes better communication so customers will know when patching and testing are necessary and when they can be avoided.
One member of the team responsible for wrangling these security issues is John Heimann, director of security program management at Oracle.
He reports to Chief Security Officer Mary Ann Davidson and does the front-end work of security: setting standards, training, enforcing security checklists, determining secure configurations, working on secure-by-default initiatives and coordinating with security products marketing.
In a daylong tour of Oracle security given to eWEEK here, Heimann said Oracle is intensifying its code testing and gave a peek at how it is doing it. Last month, Oracle announced it would use static code analysis technology from Fortify Software Inc., of Palo Alto, Calif., to hunt for bugs in C, C++, PL/SQL and Java as part of a program to improve checking for security holes during development, instead of trying to patch holes after the products out the door. Oracle has been working with Fortify for 18 months.
The Fortify tool, installed across all products this month, had to stand up to a brutal load. Oracles database alone contains between 40 million and 50 million lines of code. The tool had to scale to spit out results in a reasonable amount of time and be able to work on parallel machines.
“We want to get an answer in a day, not find out that two or three people have modified the product” while its dragged through testing, said Mark Fallon, Oracles senior manager of software development.
Oracle is also evaluating an automatic black-box test, which checks at boundaries to see if SQL injections can get through. It has identified a possible vendor and is looking at rolling it out across the company, but Heimann declined to state the vendors name or timing specifics.
Bug tracking
This isnt the first time Oracle has tested code in a big way. The company first started security evaluations in 1990 to pass the U.S. Department of Defenses TCSEC (Trusted Computer System Evaluation Criteria), also known as the Orange Book, and Europes ITSEC (Information Technology Security Evaluation Criteria). Those evaluations took place before the Internet and before Web applications blossomed to leak SQL injections and other poisons into back-end databases.
According to Duncan Harris, senior director of security assurance at Oracle, when Oracle7 was first evaluated under the government security codes, Europe found one hole.
Until Dec. 1, 1999, there was only one other reported security vulnerability, and it was handled in a similar way to the first hole: by creating tapes and CDs to ship a patch to affected customers.
In February 2001, Oracle was tracking nine bugs. By September 2001, that number crept up to 17. By December 2002, it leapt to 62. Then came August 2004 and the ill-fated Alert 68, the first security alert that contained more than one fix for more than one vulnerability.
As flaws have blossomed, so has Oracles need to pay attention to code quality. Oracles customers started taking the company to task two years ago on this issue. Oracle responded by signing an agreement with Mercury Interactive Corp. to bring in a volume testing tool and thereby launch an initiative to test better before a software release.
Oracle today is trying to get communications out as quickly as possible and supply risk matrixes so customers can decide whether they should patch.
Nevertheless, Mentoras Downing said that a “high level of skepticism” persists regarding quality when new patches or Family Packs—a group of previously released patches—are released, primarily due to Alert 68, a notoriously poor-quality patch set. “Theres an increasing recognition that at your peril do you put these patches and Family Packs into production without some real testing,” Downing said.
Poor communication around Alert 68 only compounded the technical problems. Aaron Newman, database security expert, chief technology officer and co-founder of Application Security Inc., based in New York, said that when Alert 68 first came out, he had a number of customers call “specifically begging for information” on whether they needed to apply the patches and what exactly the issues were around the vulnerabilities.
Oracles improvements in response time can be seen in the aftermath of the malicious Voyager nonworm code that was tweaked and rereleased earlier this month. (Oracle is touchy about the use of the word “worm,” since the code doesnt automatically replicate and spread.)
Even though the nonworm was a result of insecure configuration on Listener accounts—a server-based process that provides basic network connectivity for clients, application servers and other databases to an Oracle database—and not the result of a code flaw, on the day of eWEEKs visit, Oracle was rushing to get information to customers regarding proper configuration to batten down the hatches.
“Were being more responsive,” Heimann said. “We have a new security response process specifically targeted at that. We saw the response to the original Voyager posting. So were going out today [with an e-mail blast] on the second iteration of Voyager. We [now] have the ability to get this information out quickly.”
The problem: Speed kills quality. Oracle sometimes has to check to ensure that even locking down a given component wont break a 10-year-old version of a supported version.
And Oracles products are getting more complex as the company integrates acquisitions such as PeopleSoft. And then theres Project Fusion, which will either wipe out past sins as the company starts with a brand-new architecture or usher in brand-new sins for a new code set.
Root causes
How will oracle stay on top of all this code, as it buries its hands in the piles of code its acquired and wrestles it into Project Fusion?
One thing it has started to do is root-cause analysis. “When security bugs do occur, [were asking things such as], Why did this bug happen? Were standards unclear? Was training sufficient? Is this bug a single instance of something, or is it more pervasive?” Heimann said.
After all, as he put it, this is software design. “Its basically an art as much as a science,” Heimann said.