loading...
 This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
14th International Symposium on Software Reliability Engineering
Reducing wasted development time via continuous testing
Denver, Colorado
November 17-November 21
ISBN: 0-7695-2007-3
David Saff, MIT Computer Science & Artificial Intelligence Lab
Michael D. Ernst, MIT Computer Science & Artificial Intelligence Lab
Testing is often performed frequently during development to ensure software reliability by catching regression errors quickly. However, stopping frequently to test also wastes time by holding up development progress. User studies on real development projects indicate that these two sources of wasted time account for 10-15% of development time. These measurements use a novel technique for computing the wasted extra development time incurred by a delay in discovering a regression error.
We present a model of developer behavior that infers developer beliefs from developer behavior, and that predicts developer behavior in new environments-in particular, when changing testing methodologies or tools to reduce wasted time. Changing test ordering or reporting reduces wasted time by 4-41% in our case study. Changing the frequency with which tests are run can reduce wasted time by 31-82% (but developers cannot know the ideal frequency except after the fact). We introduce and evaluate a new technique, continuous testing, that uses spare CPU resources to continuously run tests in the background, providing rapid feedback about test failures as as source code is edited. Continuous testing reduced wasted time by 92-98%, a substantial improvement over the other approaches.
We have integrated continuous testing into two development environments, and are beginning user studies to evaluate its efficacy. We believe it has the potential to reduce the cost and improve the efficacy of testing and, as a result, to improve the reliability of delivered systems.
Citation:
David Saff, Michael D. Ernst, "Reducing wasted development time via continuous testing," issre, pp.281, 14th International Symposium on Software Reliability Engineering, 2003
Usage of this product signifies your acceptance of the Terms of Use.