This Article 
 Bibliographic References 
 Add to: 
Emulation of Software Faults: A Field Data Study and a Practical Approach
November 2006 (vol. 32 no. 11)
pp. 849-867
The injection of faults has been widely used to evaluate fault tolerance mechanisms and to assess the impact of faults in computer systems. However, the injection of software faults is not as well understood as other classes of faults (e.g., hardware faults). In this paper, we analyze how software faults can be injected (emulated) in a source-code independent manner. We specifically address important emulation requirements such as fault representativeness and emulation accuracy. We start with the analysis of an extensive collection of real software faults. We observed that a large percentage of faults falls into well-defined classes and can be characterized in a very precise way, allowing accurate emulation of software faults through a small set of emulation operators. A new software fault injection technique (G-SWFIT) based on emulation operators derived from the field study is proposed. This technique consists of finding key programming structures at the machine code-level where high-level software faults can be emulated. The fault-emulation accuracy of this technique is shown. This work also includes a study on the key aspects that may impact the technique accuracy. The portability of the technique is also discussed and it is shown that a high degree of portability can be achieved.

[1] M. Kalyanakrishnam, Z. Kalbarczyk, and R. Iyer, “Failure Data Analysis of a LAN of Windows NT Based Computers,” Proc. Symp. Reliable Distributed Database Systems (SRDS-18), pp. 178-187, 1999.
[2] I. Lee and R.K. Iyer, “Software Dependability in the Tandem GUARDIAN System,” IEEE Trans. Software Eng., vol. 21, no. 5, pp.455-467, May 1995.
[3] M. Sullivan and R. Chillarege, “Comparison of Software Defects in Database Management Systems and Operating Systems,” Proc. 22nd IEEE Fault Tolerant Computing Symp. (FTCS 22), pp. 475-484, July 1992.
[4] J. Gray, “A Census of Tandem Systems Availability between 1985 and 1990,” IEEE Trans. Reliability, vol. 39, no. 4, pp. 409-418, Oct. 1990.
[5] J. Musa, Software Reliability Engineering. McGraw-Hill, 1996.
[6] M.R. Lyu, Handbook of Software Reliability Engineering. IEEE Computer Society Press & McGraw-Hill, 1996.
[7] J. Christmansson and P. Santhanam, “Error Injection Aimed at Fault Removal in Fault Tolerance Mechanisms—Criteria for Error Selection Using Field Data on Software Faults,” Proc. Seventh IEEE Int'l Symp. Software Reliability Eng. (ISSRE '96), Oct./Nov. 1996.
[8] J. Voas, F. Charron, G. McGraw, K. Miller, and M. Friedman, “Predicting How Badly `Good' Software can Behave,” IEEE Software, vol. 14, no. 4, 1997.
[9] W.T. Ng and P.M. Chen, “Systematic Improvement of Fault Tolerance in the RIO File Cache,” Proc. IEEE Fault Tolerant Computing Symp. (FTCS), June 1999.
[10] A. Brown and D. Patterson, “Toward Availability Benchmark: A Case Study of Software RAID Systems,” Proc. USENIX Ann. Technical Conf., pp 263-276, June 2000.
[11] M. Vieira and H Madeira, “A Dependability Benchmark for OLTP Application Environments,” Proc. 29th Int'l Conf. Very Large Databases (VLDB '03), Sept. 2003.
[12] J. Zhu, J. Mauro, and I. Pramanick, “R3—A Framework for Availability Benchmarking,” Proc. Int'l Conf. Dependable Systems and Networks (DSN '03), pp. B86-B87, 2003.
[13] S. Lightstone, J. Hellerstein, W. Tetzlaff, P. Janson, E. Lassettre, C. Norton, B. Rajaraman, and L. Spainhower, “Toward Benchmarking Autonomic Computing Maturity,” Proc. First IEEE Conf. Industrial Automatics (INDIN '03), Aug. 2003.
[14] K. Kanoun, J. Arlat, D. Costa, M.D. Cin, P. Gil, J.C. Laprie, H. Madeira, and N. Suri, “DBench: Dependability Benchmarking,” Proc. Supplement of the IEEE/IFIP Int'l Conf. Dependable Systems and Networks (DSN '01), 2001.
[15] J. Christmansson and R. Chillarege, “Generation of an Error Set that Emulates Software Faults,” Proc. 26th IEEE Fault Tolerant Computing Symp. (FTCS 26), pp. 304-313, June 1996.
[16] J. Durães and H. Madeira, “Characterization of Operating Systems Behavior in the Presence of Faulty Device Drivers through Software Fault Emulation,” Proc. Pacific Rim Int'l Symp. Dependable Computing (PRDC '02), pp. 201-209, 2002.
[17] J. Durães, M. Vieira, and H. Madeira, “Web Server Dependability Benchmaking,” Proc. Conf. Computer Safety (SAFECOMP), 2004.
[18] H. Madeira, M. Vieira, and D. Costa, “On the Emulation of Software Faults by Software Fault Injection,” Proc. Int'l Conf. Dependable Systems and Networks (DSN '00), pp. 417-426, 2000.
[19] R. Chillarege, “Orthogonal Defect Classification,” Handbook of Software Reliability Eng., chapter 9, IEEE CS Press, McGraw-Hill, 1995.
[20] R. DeMillo, D. Guindi, W. McCracken, A. Offut, and K. King, “An Extended Overview of the Mothra Software Testing Environment,” Proc. ACM SIGSOFT/IEEE Second Workshop Software Testing, Verification, and Analysis, pp. 142-151, July 1988.
[21] S. Yamada, M. Ohba, and S. Osaki, “S-Shaped Software Reliability Growth Models and Their Application,” IEEE Trans. Reliability, vol. 33, no. 4, pp. 289-292, Oct. 1984.
[22] J.K. Chaar, M.J. Halliday, I.S. Bhandari, and R. Chillarege, “In-Process Evaluation for Software Inspection and Test,” IEEE Trans. Software Eng., vol. 19, no. 11, pp. 1055-1070, Nov. 1993.
[23] T. Khoshgoftaar et al. “Process Measures for Predicting Software Quality,” Proc. High Assurance Systems Eng. Workshop (HASE '97), 1997.
[24] J.P. Hudepohl et al., “EMERALD: A Case Study in Enhancing Software Reliability,” Proc. IEEE Eighth Int'l Symp. Software Reliability Eng. (ISSRE '98), pp. 85-91, Nov. 1998.
[25] M. Sullivan and R. Chillarege, “Software Defects and Their Impact on Systems Availability—A Study of Field Failures on Operating Systems,” Proc. 21st IEEE Fault Tolerant Computing Symp. (FTCS 21), pp. 2-9, June 1991.
[26] R.K. Iyer, “Experimental Evaluation,” Proc. 25th IEEE Symp. Fault Tolerant Computing (FTCS 25), pp. 115-132, June 1995.
[27] R. Chillarege, I.S. Bhandari, J.K. Chaar, M.J. Halliday, D. Moebus, B. Ray, and M. Wong, “Orthogonal Defect Classification—A Concept for In-Process Measurement,” IEEE Trans. Software Eng., vol. 18, no. 11, pp. 943-956, Nov. 1992.
[28] T. Budd, “Mutation Analysis: Ideas, Examples, Problems, and Prospects,” Computer Program Testing, pp. 129-134, 1981.
[29] M. Daran and P. Thévenod-Fosse, “Software Error Analysis: A Real Case Study Involving Real Faults and Mutations,” Proc. Third Symp. Software Testing and Analysis (ISSTA '03), pp. 158-171, Jan. 1996.
[30] J. Arlat et al., “Fault Injection and Dependability Evaluation of Fault Tolerant Systems,” IEEE Trans. Computers, vol. 42, no. 8, pp.919-923, Aug. 1993.
[31] G. Kanawati, N. Kanawati, and J. Abraham, “FERRARI: A Tool for the Validation of System Dependability Properties,” Proc. 22th IEEE Fault Tolerant Computing Symp. (FTCS 22), pp. 336-344, June 1992.
[32] T. Tsai and R.K. Iyer, “An Approach to Benchmarking of Fault-Tolerant Commercial Systems,” Proc. 26th IEEE Fault Tolerant Computing Symp. (FTCS 26), pp. 314-323, June 1996.
[33] J. Carreira, H. Madeira, and J.G. Silva, “Xception: Software Fault Injection and Monitoring in Processor Functional Units,” IEEE Trans. Software Eng., vol. 24, no. 2, Feb. 1998.
[34] J. Aidemark, J. Vinter, P. Folkesson, and J. Karlsson, ”Goofi: Generic Object-Oriented Fault Injection Tool,” Proc. Int'l Conf. Dependable Systems and Networks (DSN '01), July 2001.
[35] J.-C. Fabre, F. Salles, M. Moreno, and J. Arlat, “Assessment of COTS Microkernels by Fault Injection,” Proc. Seventh IFIP Working Conf. Dependable Computing for Critical Applications (DCCA '99), pp.25-44, 1999.
[36] P. Koopman, J. Sung, C. Dingman, D. Siewiorek, and T. Marz, “Comparing Operating Systems Using Robustness Benchmarks,” Proc. 16th Int'l Symp. Reliable Distributed Systems (SRDS '97), pp.72-79, 1997.
[37] J. Hudak, B. Suth, D. Siewiorek, and Z. Segall, “Evaluation and Comparison of Fault-Tolerant Software Techniques,” IEEE Trans. Reliability, vol. 42, no. 2, pp. 190-204, June 1993.
[38] W. Kao, “Experimental Study of Software Dependability,” PhD thesis, Technical Report CRHC-94-16, Dept. of Computer Science, Univ. of Illinois at Urbana-Champaign, Illi nois, 1994.
[39] J. Bieman, D. Dreilinger, and L. Lin, “Using Fault Injection to Increase Test Coverage,” Proc. Seventh IEEE Int'l Symp. Software Reliability Eng. (ISSRE '96), Oct. /Nov. 1996.
[40] D. Blough and T. Torii, “Fault-Injection-Based Testing of Fault-Tolerant Algorithms in Message-Passing Parallel Computers,” Proc. 27th IEEE Fault Tolerant Computing Symp. (FTCS 27), pp. 258-267, June 1997.
[41] W.T. Ng, C.M. Aycock, and P.M. Chen, “Comparing Disk and Memory's Resistance to Operating System Crashes,” Proc. Seventh IEEE Int'l Symp. Software Reliability Eng. (ISSRE '96), 1996.
[42] D. Costa, T. Rilho, and H. Madeira, “Joint Evaluation of Performance and Robustness of a COTS DBMS through Fault-Injection,” Proc. Dependable Systems and Networks Conf. (DSN '00), June 2000.
[43] D. Costa, T. Rilho, M. Vieira, and H. Madeira, “ESFFI—A Novel Technique for the Emulation of Software Faults in COTS Components,” Proc. Eighth Ann. IEEE Int'l Conf. Eng. of Computer-Based Systems (ECBS '01), Apr. 2001.
[44] J. Durães and H. Madeira, “Emulation of Software Faults by Selective Mutations at Machine-Code Level,” Proc. 13th IEEE Int'l Symp. Software Reliability Eng. (ISSRE '02), Nov. 2002.
[45] J. Durães and H. Madeira, “Definition of Software Fault Emulation Operators: A Field Data Study,” Proc. IEEE/IFIP Int'l Conf. Dependable Systems and Networks, Dependable Computing, and Comm. (DSN-DCC '03), June 2003.
[46] J. Durães and H. Madeira, “Generic Faultloads Based on Software Faults for Dependability Benchmarking,” Proc. IEEE Int'l Symp. Dependable Systems and Networks (DSN '04), 2004.
[47] Intel Architecture Software Developer's Manual, Vol. 2: Instruction Set Reference, Intel, 1997.
[48] J. Voas and G. McGraw, Software Fault Injection: Inoculating Programs against Errors. John Wiley and Sons, 1997.
[49] R.A. DeMillo, R.J. Lipton, and F.G. Sayward, “Hints on Test Data Selection: Help for the Practicing Programmer,” Computer, vol. 11, no. 4, pp. 34-41, Apr. 1978.
[50] R.E. Kirk, Statistics: An Introduction, fourth ed. Assessment Systems Corp., 1999.

Index Terms:
Fault injection, software faults, software reliability.
Jo?o A. Dur?es, Henrique S. Madeira, "Emulation of Software Faults: A Field Data Study and a Practical Approach," IEEE Transactions on Software Engineering, vol. 32, no. 11, pp. 849-867, Nov. 2006, doi:10.1109/TSE.2006.113
Usage of this product signifies your acceptance of the Terms of Use.