Faster Testing for Complex Software Systems
Testing applications for large software systems can be a time-consuming and expensive process that ultimately delays the release date. As software systems incorporate more components, one of the biggest challenges is knowing which combinations to test. A University of Nebraska researcher is developing new algorithms that promise to improve this testing process.
In a large software system, each component might perform flawlessly by itself, but interaction faults can emerge when components work with each other. Developers must test each component not only separately but also in each possible combination with different components. Consequently, said Myra Cohen, assistant professor of computer science at the University of Nebraska, the number of tests required can grow exponentially as systems introduce different components and new configurations for them.
Cohen said that no one has quantified the cost of interaction faults specifically. But in 2002, a US National Institute of Standards and Technology study found that computer bugs in general cost the US economy between $22.2 and $59.5 billion in repairs, lost business, and downtime per year. In some cases, the consequences of software failures are grave, said Cohen. For example, software errors have caused medical equipment to overdose patients with radiation and led to the explosion of an Ariane 5 rocket.
Cohen is working on two related projects to improve complex systems' testing. The first one, sponsored by the US National Science Foundation (NSF), is developing algorithms to sample the configurable-components space and test it efficiently because the number of possible combinations is so large. End users, system administrators, and application providers can all manipulate a system's configuration. The NSF project is trying to find ways of ensuring that combinations with a higher likelihood of failure are tested early in the development process.
The second project, Just Enough Testing (JET), is sponsored by the US Air Force Office of Scientific Research and focused on testing software product lines, which are a subset of configurable systems. They're built from a finite component set that the software developer controls. For example, cell phone developers might have different component types for graphics rendering, messaging, and Web browsing, which they could combine in different ways to create a family of cell phones.
Certain faults aren't revealed unless you put the correct configurations together, said Cohen. "We would like to be able to sample that whole configuration space so that when the application ships, we have the confidence that those faults have been discovered."
The JET project is creating algorithms to intelligently reuse tests from prior software component combinations. These algorithms can identify specific component combinations to test 300 times faster than existing techniques. They can also reduce the number of required test cases by an average of 5 percent across 50 different benchmarks. Although other algorithms also do sampling, Cohen said, they don't work well with dependencies across multiple component types.
JET focuses on new feature combinations to improve the entire product family, said Cohen. "The idea is to do more testing in the same amount of time so we can improve the quality of the software."
After testing one combination of components, an organization conducts a series of tests of different combinations and compiles all their results. The organization knows which combination it has tested in different product versions. On a new product, it can focus on testing combinations of features that haven't yet been tested together.
"JET is an approach to test strategies and test-case specifications that uses pragmatism and appropriateness as its central pillars," said Ivan Ericsson, director at the SQS Software Quality Systems consultancy.
Cohen said the research is still in the early phases. One challenge is tuning the algorithms against realistic product lines. Another is demonstrating the reliability of test results in new software component combinations. "We're asking people to reuse this testing information,” Cohen explained, "so we have to be able to show that it is still valid."
Ericsson pointed to another potential problem of managers taking the JET ideas out of context. "There's a real risk that JET is interpreted as 'writing less tests,'" he said, "when it actually means 'write the appropriate level of tests.'"
David R. Luginbuhl, program manager for systems and software at the US Air Force Office of Scientific Research, said, "We think this will help in the development of software and deployment of software in the future. This could eventually be incorporated into a software testing suite."
In the long run, the number of software component combinations will continue to grow with the rise of the Web and more complex systems integration. Cohen said that this type of research will help scale software testing across these larger systems.
George Lawton is a freelance technology writer based in Guerneville, California. Contact him at firstname.lastname@example.org.