Who Tests the Code Testers?

Opinion: Scanning tools are getting better securing code, but there's still nobody telling us which of the code tools are any good or what they're specifically good at.

In order to make software more secure, the industry must get on the bandwagon for code scanning tools, incorporating them into the daily development cycle.

But are the tools currently sturdy enough to stand up to immense code loads?

Scanning tools are in fact getting better.

Theyre scaling better, for one thing.

Theyre able to run on parallel machines, which means they can handle much bigger code loads and get results to developers in a reasonable amount of time—in other words, before the code in question has been revised two or three times as testing drags on.

One remaining problem, though, is theres still nobody telling us which of the code tools are any good or what a given tool is specifically good at.

At this point, the work is hard, and theres just no independent body out there that can point you to the tool that will best fit your needs.

"Its kind of arduous," Oracles Mark Fallon, senior release manager of software development, said in a chat we had before a presentation hes gave on the subject at the RSA Conference on Feb. 15.

"At the moment there are no publicly available benchmarks. There isnt a good body of knowledge [from which] to say Its these guys over these guys, Its these guys for this particular area, so you have to go through and do the evaluation yourself. Thats fine if you have 100 lines of code. We have 50 million lines of code."

Thats a huge body of code with extremely complex paths wending through it. Oracle and companies with comparatively unwieldy code sets at this point have to bring the code in, get the code scanning tool working, make sure it can scan the massive body of code, and then evaluate its results to make sure that theyre real and not false positives.

/zimages/4/28571.gifClick here to read about the free debugger EnterpriseDB shipped with an update to its PostgreSQL-based database.

"With any scanner company weve worked with, weve gone through iterations of where their tool couldnt handle our code, and weve worked with them" to fine-tune the tools ability to churn through the code set, Fallon told me.

Thats why, for example, Oracle, based in Redwood Shores, Calif., worked with Fortify for a year before signing on the bottom line to use its tool. During a year of tweaking, Fortify came in to Oracle repeatedly as the developers put their heads together to optimize results.

The Fortify deal was part of Oracles ongoing effort to knit volume code testing into its development DNA. In December, Oracle announced that it would use static code analysis technology from Fortify to hunt for bugs in C, C++, PL/SQL and Java as part of a program to improve checking for security holes during development, instead of trying to patch holes after the products out the door.

The Fortify tool had to stand up to a brutal load: Oracles database alone contains between 40 million to 50 million lines of code. The tool had to scale to spit out results in a reasonable amount of time and be able to work on parallel machines.

"We want to get an answer in a day, not find out that two or three people have modified the product" while its dragged through testing, Fallon said at the time.

Next Page: How to use a code scanner effectively.