The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.10 - Oct. (2013 vol.39)
pp: 1345-1357
Akito Monden , Grad. Sch. of Inf. Sci., Nara Inst. of Sci. & Technol., Ikoma, Japan
Takuma Hayashi , R&D Center, NTT West Corp., Osaka, Japan
Shoji Shinoda , R&D Center, NTT West Corp., Osaka, Japan
Kumiko Shirai , R&D Center, NTT West Corp., Osaka, Japan
Junichi Yoshida , R&D Center, NTT West Corp., Osaka, Japan
Mike Barker , Grad. Sch. of Inf. Sci., Nara Inst. of Sci. & Technol., Ikoma, Japan
Kenichi Matsumoto , Grad. Sch. of Inf. Sci., Nara Inst. of Sci. & Technol., Ikoma, Japan
ABSTRACT
Until now, various techniques for predicting fault-prone modules have been proposed and evaluated in terms of their prediction performance; however, their actual contribution to business objectives such as quality improvement and cost reduction has rarely been assessed. This paper proposes using a simulation model of software testing to assess the cost effectiveness of test effort allocation strategies based on fault prediction results. The simulation model estimates the number of discoverable faults with respect to the given test resources, the resource allocation strategy, a set of modules to be tested, and the fault prediction results. In a case study applying fault prediction of a small system to acceptance testing in the telecommunication industry, results from our simulation model showed that the best strategy was to let the test effort be proportional to "the number of expected faults in a module × log(module size)." By using this strategy with our best fault prediction model, the test effort could be reduced by 25 percent while still detecting as many faults as were normally discovered in testing, although the company required about 6 percent of the test effort for metrics collection, data cleansing, and modeling. The simulation results also indicate that the lower bound of acceptable prediction accuracy is around 0.78 in terms of an effort-aware measure, Norm(Popt). The results indicate that reduction of the test effort can be achieved by fault prediction only if the appropriate test strategy is employed with high enough fault prediction accuracy. Based on these preliminary results, we expect further research to assess their general validity with larger systems.
INDEX TERMS
Testing, Predictive models, Measurement, Software, Resource management, Companies, Accuracy,simulation, Complexity measures, fault prediction, quality assurance, resource allocation
CITATION
Akito Monden, Takuma Hayashi, Shoji Shinoda, Kumiko Shirai, Junichi Yoshida, Mike Barker, Kenichi Matsumoto, "Assessing the Cost Effectiveness of Fault Prediction in Acceptance Testing", IEEE Transactions on Software Engineering, vol.39, no. 10, pp. 1345-1357, Oct. 2013, doi:10.1109/TSE.2013.21
REFERENCES
[1] E. Arisholm, L.C. Briand, and E.B. Johannessen, "A Systematic and Comprehensive Investigation of Methods to Build and Evaluate Fault Prediction Models," J. Systems and Software, vol. 83, no. 1, pp. 2-17, 2010.
[2] M. D'Ambros, M. Lanza, and R. Robbes, "An Extensive Comparison of Bug Prediction Approaches," Proc. Seventh IEEE Working Conf. Mining Software Repositories, pp. 31-41, 2010.
[3] V.R. Basili and B.T. Perricone, "Software Errors and Complexity: An Empirical Investigation," Comm. ACM, vol. 27, pp. 42-52, 1984.
[4] A.S. Foulkes, Applied Statistical Genetics with R. Springer, 2009.
[5] A.L. Goel and K. Okumoto, "Time-Dependent Error-Detection Rate Model for Software Reliability and Other Performance Measures," IEEE Trans. Reliability, vol. 28, no. 3, pp. 206-211, Aug. 1979.
[6] T.L. Graves, A.F. Karr, J.S. Marron, and H. Siy, "Predicting Fault Incidence Using Software Change History," IEEE Trans. Software Eng., vol. 26, no. 7, pp. 653-661, July 2000.
[7] "Information-Technology Promotion Agency, Japan (IPA) Software Engineering Center (SEC) ed.," White Papers on Software Development Projects in Japan, 2010-2011 Ed., 2010.
[8] Y. Kamei, A. Monden, and K. Matsumoto, "Empirical Evaluation of SVM-Based Software Reliability Model," Proc. Fifth ACM/IEEE Int'l Symp. Empirical Software Eng., vol. 2, pp. 39-41, 2006.
[9] Y. Kamei, S. Matsumoto, A. Monden, K. Matsumoto, B. Adams, and A.E. Hassan, "Revisiting Common Bug Prediction Findings Using Effort Aware Models," Proc. IEEE Int'l Conf. Software Maintenance, pp. 1-10, 2010.
[10] T.M. Khoshgoftaar and E.B. Allen, "Modeling Software Quality with Classification Trees," Recent Advances in Reliability and Quality Engineering, pp. 247-270, World Scientific, 1999.
[11] T.M. Khoshgoftaar, A. Pandya, and D. Lanning, "Application of Neural Networks for Predicting Program Fault," Annals of Software Eng., vol. 1, pp. 141-154, 1995.
[12] S. Kim, E.J. WhiteheadJr., and Y. Zhang, "Classifying Software Changes: Clean or Buggy?" IEEE Trans. Software Eng., vol. 34, no. 2, pp. 181-196, Mar./Apr. 2008.
[13] B. Kitchenham and E. Mendes, "Why Comparative Effort Prediction Studies May Be Invalid," Proc. Fifth Int'l Conf. Predictor Models in Software Eng., article 4, 2009.
[14] P. Knab, M. Pinzger, and A. Bernstein, "Predicitng Defect Densities in Source Code Files with Decision Tree Learners," Proc. Third Working Conf. Mining Software Repositories, pp. 119-125, 2006.
[15] A.G. Koru, D. Zhang, K. El Emam, and H. Liu, "An Investigation into the Functional Form of the Size-Defect Relationship for Software Modules," IEEE Trans. Software Eng., vol. 35, no. 2, pp. 293-304, Mar./Apr. 2009.
[16] S. Lessmann, B. Baesens, C. Mues, and S. Pietsch, "Benchmarking Classification Models for Software Defect Prediction: A Proposed Framework and Novel Findings," IEEE Trans. Software Eng., vol. 34, no. 4, pp. 485-496, July/Aug. 2008.
[17] P.L. Li, J. Herbsleb, M. Shaw, and B. Robinson, "Experiences and Results from Initiating Field Defect Prediction and Product Test Prioritization Efforts at ABB Inc.," Proc. 28th Int'l Conf. Software Eng., pp. 413-422, 2006.
[18] T. Mende and R. Koschke, "Revisiting the Evaluation of Defect Prediction Models," Proc. Int'l Conf. Predictor Models Software Eng., pp. 1-10, 2009.
[19] T. Menzies, O. Jalali, J. Hihn, D. Baker, and K. Lum, "Stable Rankings for Different Effort Models," Automated Software Eng., vol. 17, no. 4, pp. 409-437, 2010.
[20] R. Moser, W. Pedrycz, and G. Succi, "A Comparative Analysis of the Efficiency of Change Metrics and Static Code Attributes for Defect Prediction," Proc. 30th Int'l Conf. Software Eng., pp. 181-190, 2008.
[21] J.C. Munson and T.M. Khoshgoftaar, "The Detection of Fault-Prone Programs," IEEE Trans. Software Eng., vol. 18, no. 5, pp. 423-433, May 1992.
[22] N. Nagappan, T. Ball, and A. Zeller, "Mining Metrics to Predict Component Failures," Proc. 28th Int'l Conf. Software Eng., pp. 452-461, 2006.
[23] N. Ohlsson and H. Alberg, "Predicting Fault-Prone Software Modules in Telephone Switches," IEEE Trans. Software Eng., vol. 22, no. 12, pp. 886-894, Dec. 1996.
[24] T.J. Ostrand, E.J. Weyuker, and R.M. Bell, "Predicting the Location and Number of Faults in Large Software Systems," IEEE Trans. Software Eng., vol. 31, no. 4, pp. 340-355, Apr. 2005.
[25] R. "The R Project for Statistical Computing," http:/www. r-project.org/, 2013.
[26] E. Shihab, A. Mockus, Y. Kamei, B. Adams, and A.E. Hassan, "High-Impact Defects: A Study of Breakage and Surprise Defects," Proc. ACM SIGSOFT Symp. Foundations Software Eng., pp. 300-310, 2011.
[27] J. Śliwerski, T. Zimmermann, and A. Zeller, "When Do Changes Induce Fixes?" Proc. Int'l Conf. Mining Software Repositories, pp. 1-5, 2005.
[28] A. Schröter, T. Zimmermann, and A. Zeller, "Predicting Component Failures at Design Time," Proc. ACM/IEEE Int'l Symp. Empirical Software Eng. , pp. 18-27, 2006.
[29] H.B. Tan, Y. Zhao, and H. Zhang, "Conceptual Data Model-Based Software Size Estimation for Information Systems," ACM Trans. Software Eng. Methodologies, vol. 19, no. 2, pp. 1-37, Oct. 2009.
[30] T.M. Therneau, E.J. Atkinson, and M. Foundation, "An Introduction to Recursive Partitioning Using the RPART Routines," http://cran.r-project.org/web/packages/rpart/ vignettes longintro.pdf, 2012.
[31] A. Tosun, B. Turhan, and A. Bener, "Practical Considerations in Deploying AI for Defect Prediction: A Case Study within the Turkish Telecommunication Industry," Proc. Fifth Int'l Conf. Predictor Models in Software Eng., pp. 1-9, 2009.
[32] B. Turhan, T. Menzies, A. Bener, and J. Distefano, "On the Relative Value of Cross-Company and Within-Company Data for Defect Prediction," Empirical Software Eng., vol. 14, no. 5, pp. 540-578, 2009.
[33] S. Yamada and S. Osaki, "Software Reliability Growth Modeling: Models and Applications," IEEE Trans. Software Eng., vol. 11, no. 12, pp. 1431-1437, Dec. 1985.
[34] T. Zimmermann, R. Premraj, and A. Zeller, "Predicting Defects for Eclipse," Proc. Third Int'l Workshop Predictor Models Software Eng., 2007.
99 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool