In the foreseeable future, software testing will remain one of the best tools we have at our disposal to ensure software dependability. Empirical studies are crucial to software testing research in order to compare and improve software testing techniques and practices. In fact, there is no other way to assess the cost-effectiveness of testing techniques, since all of them are, to various extents, based on heuristics and simplifying assumptions. However, when empirically studying the cost and fault-detection rates of a testing technique, a number of validity issues arise. Further, there are many ways in which empirical studies can be performed, ranging from simulations to controlled experiments with human subjects. What are the strengths and drawbacks of the various approaches? What is the best option under which circumstances?

This paper presents a critical analysis of empirical research in software testing and will attempt to highlight and clarify the issues above in a structured and practical manner.