How should we test software? Given a range of possible test methods, when one technique preferred to another?

Typically, these kinds of questions are answered with reference to the inherent properties of the assessment mechanism. For example, Lowry et.al. [7] and Menzies & Cukic [8] contrast the costs and defect detection decay rates of formal methods, white box testing, and black box testing. That analysis made assumptions about the completeness of the search and the cost of setting up each run. A drawback with that kind of analysis is it is silent about the model being searched for defects¹.

This paper studies how details of a particular model can effect the efficacy of a search for defects. We find that if the test method is fixed, we can identify classes of software that are more or less testable. Using a combination of model mutators and machine learning, we find that we can isolate topological features that significantly change the effectiveness of a defect detection tool. More specifically, we show that for one defect detection tool (a stochastic search engine) applied to a certain representation (finite state machines), we can increase the average odds of finding a defect from 69% to 91%. The method used to change those odds is quite general and should apply to other defect detection tools being applied to other representations.

These results draw into question the results like those of Lowry et.al. and Menzies & Cukic. If simple changes to a model's topology can increase detect detection to near 100%, then the efficacy of a detect detection tool must be assessed in conjunction with the program being assessed.