The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.01 - Jan. (2013 vol.39)
pp: 80-96
Roberto Natella , Federico II University of Naples, Naples
Domenico Cotroneo , Federico II University of Naples, Naples
Joao A. Duraes , Rua Pedro Nunes-Quinta da Nora, Coimbra
Henrique S. Madeira , University of Coimbra, Polo II-Pinhal de Marrocos, Coimbra
ABSTRACT
The injection of software faults in software components to assess the impact of these faults on other components or on the system as a whole, allowing the evaluation of fault tolerance, is relatively new compared to decades of research on hardware fault injection. This paper presents an extensive experimental study (more than 3.8 million individual experiments in three real systems) to evaluate the representativeness of faults injected by a state-of-the-art approach (G-SWFIT). Results show that a significant share (up to 72 percent) of injected faults cannot be considered representative of residual software faults as they are consistently detected by regression tests, and that the representativeness of injected faults is affected by the fault location within the system, resulting in different distributions of representative/nonrepresentative faults across files and functions. Therefore, we propose a new approach to refine the faultload by removing faults that are not representative of residual software faults. This filtering is essential to assure meaningful results and to reduce the cost (in terms of number of faults) of software fault injection campaigns in complex software. The proposed approach is based on classification algorithms, is fully automatic, and can be used for improving fault representativeness of existing software fault injection approaches.
INDEX TERMS
Software, Testing, Fault tolerance, Fault tolerant systems, Hardware, Fault location, Emulation, fault-tolerant systems, Software fault injection, experimental dependability evaluation, software reliability
CITATION
Roberto Natella, Domenico Cotroneo, Joao A. Duraes, Henrique S. Madeira, "On Fault Representativeness of Software Fault Injection", IEEE Transactions on Software Engineering, vol.39, no. 1, pp. 80-96, Jan. 2013, doi:10.1109/TSE.2011.124
REFERENCES
[1] J. Gray, "Why Do Computers Stop and What Can Be Done About It?" Tandem TR85.7, June 1985.
[2] R. Chillarege, I.S. Bhandari, J.K. Chaar, M.J. Halliday, D.S. Moebus, B.K. Ray, and M.-Y. Wong, "Orthogonal Defect Classification—A Concept for In-Process Measurements," IEEE Trans. Software Eng., vol. 18, no. 11, pp. 943-956, Nov. 1992.
[3] J. Musa, Software Reliability Engineering. McGraw-Hill, 1996.
[4] E.J. Weyuker, "Testing Component-Based Software: A Cautionary Tale," IEEE Software, vol. 15, no. 5, pp. 54-59, Sept./Oct. 1998.
[5] J.C. Knight, "Safety Critical Systems: Challenges and Directions" Proc. 24th Int'l Conf. Software Eng., pp. 547-550, 2002.
[6] NASA Software Safety Guidebook, NASA-GB-8719.13, 2004.
[7] Int'l Organization for Standardization, "Product Development: Software Level," ISO 26262-6, 2011.
[8] A. Avizienis, "The N-Version Approach to Fault-Tolerant Software," IEEE Trans. Software Eng., vol. 11, no. 12, pp. 1491-1501, Dec. 1985.
[9] M.R. Lyu, Software Fault Tolerance. John Wiley & Sons, 1995.
[10] F. Cristian, "Exception Handling and Software Fault Tolerance," IEEE Trans. Computers, vol. 31, no. 6, pp. 531-540, June 1982.
[11] J. Arlat, M. Aguera, L. Amat, Y. Crouzet, J.C. Fabre, J.C. Laprie, E. Martin, and D. Powell, "Fault Injection for Dependability Validation: A Methodology and Some Applications," IEEE Trans. Software Eng., vol. 16, no. 2, pp. 166-182, Feb. 1990.
[12] J. Voas, F. Charron, G. McGraw, K. Miller, and M. Friedman, "Predicting How Badly 'Good' Software Can Behave," IEEE Software, vol. 14, no. 4, pp. 73-83, July/Aug. 1997.
[13] J. Christmansson and R. Chillarege, "Generation of an Error Set That Emulates Software Faults Based on Field Data," Proc. 26th IEEE Fault Tolerant Computing Symp., pp. 304-313, 1996.
[14] R.G. Hamlet, "Testing Programs with the Aid of a Compiler," IEEE Trans. Software Eng., vol. 3, no. 4, pp. 279-290, July 1977.
[15] R.A. DeMillo, R.J. Lipton, and F.G. Sayward, "Hints on Test Data Selection: Help for the Practicing Programmer," Computer, vol. 11, no. 4, pp. 34-41, Apr. 1978.
[16] K.N. King and A.J. Offutt, "A Fortran Language System for Mutation-Based Software Testing," Software: Practice and Experience, vol. 21, no. 7, pp. 685-718, July 1991.
[17] J.A. Duraes and H. Madeira, "Emulation of Software Faults: A Field Data Study and a Practical Approach," IEEE Trans. Software Eng., vol. 32, no. 11, pp. 849-867, Nov. 2006.
[18] N.E. Fenton and N. Ohlsson, "Quantitative Analysis of Faults and Failures in a Complex Software System," IEEE Trans. Software Eng., vol. 26, no. 8, pp. 797-814, Aug. 2000.
[19] D. Powell, E. Martins, J. Arlat, and Y. Crouzet, "Estimators for Fault Tolerance Coverage Evaluation," IEEE Trans. Computers, vol. 44, no. 2, pp. 261-274, Feb. 1995.
[20] M. Cukier, D. Powell, and J. Arlat, "Coverage Estimation Methods for Stratified Fault-Injection," IEEE Trans. Computers, vol. 48, no. 7, pp. 707-723, July 1999.
[21] J.H. Andrews, L.C. Briand, and Y. Labiche, "Is Mutation an Appropriate Tool for Testing Experiments?" Proc. 27th Int'l Conf. Software Eng., pp. 402-411, 2005.
[22] H. Do and G. Rothermel, "On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques," IEEE Trans. Software Eng., vol. 32, no. 9, pp. 733-752, Sept. 2006.
[23] R. Geist, A.J. Offutt, and F.C. HarrisJr., "Estimation and Enhancement of Real-Time Software Reliability through Mutation Analysis," IEEE Trans. Computers, vol. 41, no. 5, pp. 550-558, May 1992.
[24] Y. Jia and M. Harman, "An Analysis and Survey of the Development of Mutation Testing," IEEE Trans. Software Eng., vol. 37, no. 5, pp. 649-678, Sept./Oct. 2011.
[25] A.J. Offutt, G. Rothermel, and C. Zapf, "An Experimental Evaluation of Selective Mutation," Proc. 15th Int'l Conf. Software Eng., pp. 100-107, 1993.
[26] A.J. Offutt, A. Lee, G. Rothermel, R.H. Untch, and C. Zapf, "An Experimental Determination of Sufficient Mutant Operators," ACM Trans. Software Eng. and Methodology, vol. 5, no. 2, pp. 99-118, Apr. 1996.
[27] W.E. Wong and A.P. Mathur, "Reducing the Cost of Mutation Testing: An Empirical Study," J. Systems and Software, vol. 31, no. 3, pp. 185-196, Dec. 1995.
[28] M. Sridharan and A.S. Namin, "Prioritizing Mutation Operators Based on Importance Sampling," Proc. 21st Int'l Symp. Software Reliability Eng., pp. 378-387, 2010.
[29] W.T. Ng and P.M. Chen, "The Design and Verification of the Rio File Cache," IEEE Trans. Computers, vol. 50, no. 4, pp. 322-337, Apr. 2001.
[30] S. Chandra and P.M. Chen, "How Fail-Stop Are Faulty Programs?" Proc. IEEE 28th Fault Tolerant Computing Symp., 1998.
[31] J.J. Hudak, B.H. Suh, D.P. Siewiorek, and Z. Segall, "Evaluation and Comparison of Fault-Tolerant Software Techniques," IEEE Trans. Reliability, vol. 42, no. 2, pp. 190-204, June 1993.
[32] M. Vieira and H. Madeira, "A Dependability Benchmark for OLTP Application Environments," Proc. 29th Int'l Conf. Very Large Data Bases, pp. 742-753, 2003.
[33] A. Avizienis, J.-C. Laprie, B. Randell, and C. Landwehr, "Basic Concepts and Taxonomy of Dependable and Secure Computing," IEEE Trans. Dependable and Secure Computing, vol. 1, no. 1, pp. 11-33, Jan.-Mar. 2004.
[34] D. Stott, B. Floering, D. Burke, Z. Kalbarczyk, and R. Iyer, "NFTAPE: A Framework for Assessing Dependability in Distributed Systems with Lightweight Fault Injectors," Proc. Int'l Conf. Computer Performance and Dependability Symp., pp. 91-100, 2000.
[35] J. Aidemark, J. Vinter, P. Folkesson, and J. Karlsson, "GOOFI: Generic Object-Oriented Fault Injection Tool," Proc. Int'l Conf. Dependable Systems and Networks, pp. 83-88, 2001.
[36] Z. Segall, D. Vrsalovic, D. Siewiorek, J. Kownacki, J. Barton, R. Dancey, A. Robinson, and T. Lin, "FIAT—Fault Injection Based Automated Testing Environment" Proc. IEEE 18th Fault Tolerant Computing Symp., pp. 102-107, 1988.
[37] M.-C. Hsueh, T.K. Tsai, and R.K. Iyer, "Fault Injection Techniques and Tools," Computer, vol. 30, no. 4, pp. 75-82, Apr. 1997.
[38] J. Carreira, H. Madeira, and J.G. Silva, "Xception: A Technique for the Experimental Evaluation of Dependability in Modern Computers," IEEE Trans. Software Eng., vol. 24, no. 2, pp. 125-136, Feb. 1998.
[39] R. Moraes, R. Barbosa, J.A. Duraes, N. Mendes, E. Martins, and H. Madeira, "Injection of Faults at Component Interfaces and Inside the Component Code: Are They Equivalent?" Proc. Sixth European Dependable Computing Conf., pp. 53-64, 2006.
[40] P. Koopman and J. DeVale, "The Exception Handling Effectiveness of POSIX Operating Systems," IEEE Trans. Software Eng., vol. 26, no. 9, pp. 837-848, Sept. 2000.
[41] A.K. Ghosh, M. Schmid, and V. Shah, "Testing the Robustness of Windows NT Software," Proc. Ninth IEEE Int'l Symp. Software Reliability Eng., pp. 231-236, 1998.
[42] J. Arlat, J.-C. Fabre, M. Rodriguez, and F. Salles, "Dependability of COTS Microkernel-Based Systems," IEEE Trans. Computers, vol. 51, no. 2, pp. 138-163, Feb. 2002.
[43] E. Martins, C.M.F. Rubira, and N.G.M. Leme, "Jaca: A Reflective Fault Injection Tool Based on Patterns" Proc. IEEE Int'l Conf. Dependable Systems and Networks, pp. 483-487, 2002.
[44] W.-L. Kao, R.K. Iyer, and D. Tang, "FINE: A Fault Injection and Monitoring Environment for Tracing the UNIX System Behavior under Faults," IEEE Trans. Software Eng., vol. 19, no. 11, pp. 1105-1118, Nov. 1993.
[45] H. Madeira, D. Costa, and M. Vieira, "On the Emulation of Software Faults by Software Fault Injection," Proc. IEEE Int'l Conf. Dependable Systems and Networks, pp. 417-426, 2000.
[46] T. Jarboui, J. Arlat, Y. Crouzet, K. Kanoun, and T. Marteau, "Analysis of the Effects of Real and Injected Software Faults: Linux as a Case Study," Proc. Pacific Rim Int'l Symp. Dependable Computing, pp. 51-58, 2002.
[47] M.E. Delamaro and J.C. Maldonado, "Proteum-A Tool for the Assessment of Test Adequacy for C Programs," Proc. Conf. Performability in Computer Systems, pp. 79-95, 1996.
[48] MySQL Market Share, http://www.mysql.com/why-mysql marketshare /, Mar. 2011.
[49] EnterpriseDB's Postgres Plus Users by Application Type, http://www.enterprisedb.com/customer-success customers-by- application-workload, Mar. 2011.
[50] J. Rufino, S. Filipe, M. Coutinho, S. Santos, and J. Windsor, "ARINC 653 Interface in RTEMS," Proc. Data Systems in Aerospace Conf., 2007.
[51] GCC Documentation, http://gcc.gnu.org/onlinedocsgcc/, 2012.
[52] QEMU Virtualization Software, http:/qemu.org, 2012.
[53] SAFE Software Fault Emulation Tool, http://www.mobilab. unina.itSFI.htm, 2012.
[54] TPC-C OLTP Benchmark, http://www.tpc.orgtpcc/, 2012.
[55] D. Sheskin, Handbook of Parametric and Nonparametric Statistical Procedures. Chapman and Hall/CRC, 2004.
[56] N.E. Fenton and S.L. Pfleeger, Software Metrics: A Rigorous and Practical Approach. PWS Publishing Co., 1998.
[57] V.R. Basili and B.T. Perricone, "Software Errors and Complexity: An Empirical Investigation," Comm. ACM, vol. 27, no. 1, pp. 42-52, Jan. 1984.
[58] S. Henry and D. Kafura, "Software Structure Metrics Based on Information Flow," IEEE Trans. Software Eng., vol. 7, no. 5, pp. 510-518, Sept. 1981.
[59] SciTools Understand Tool, http:/www.scitools.com, 2012.
[60] T. Menzies, J. Greenwald, and A. Frank, "Data Mining Static Code Attributes to Learn Defect Predictors," IEEE Trans. Software Eng., vol. 33, no. 1, pp. 2-13, Jan. 2007.
[61] K. Srinivasan and D. Fisher, "Machine Learning Approaches to Estimating Software Development Effort," IEEE Trans. Software Eng., vol. 21, no. 2, pp. 126-137, Feb. 1995.
[62] I. Witten and E. Frank, Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann, 2005.
35 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool