Microsoft System Prioritizes Tests

As software grows larger, ways to test that software need to keep pace. Microsofts Programmer Productivity Research Center has developed a way to use binary comparisons between two versions of a compiled program to infer which software tests should be run on it.

"We built a test prioritization system," said Microsoft Distinguished Engineer Amitabh Srivastava. "The idea is very simple. You may have tens of millions of lines of source code, but you dont change them all," Srivastava said. "When every test is run, we put probes into the program and store [the test coverage map] into a repository. ... If you give me two versions of kernel32.dll, I can tell you whats changed. Now I know which parts of the program [the tests] are covering, and I prioritize the tests."

This mapping between changes in software and the most relevant tests to check those changes is now in use in the Windows, SQL Server and Exchange product groups.

In just a few minutes, developers can run a small number of tests that are most likely to find any bugs the developers might have introduced.

"We are now catching the defects on the developers desktop without them entering the system," said Srivastava. "If we can stop the defect from entering the system, its very cheap."

Srivastava and fellow researcher Jay Thiagarajan presented a paper on their technique in July at the International Symposium on Software Testing and Analysis in Rome.