This Article 
 Bibliographic References 
 Add to: 
Measuring Software Dependability by Robustness Benchmarking
June 1997 (vol. 23 no. 6)
pp. 366-378

Abstract—Inability to identify weaknesses or to quantify advancements in software system robustness frequently hinders the development of robust software systems. Efforts have been made to develop benchmarks of software robustness to address this problem, but they all suffer from significant shortcomings. This paper presents the various features that are desirable in a benchmark of system robustness, and evaluates some existing benchmarks according to these features. A new hierarchically structured approach to building robustness benchmarks, which overcomes many deficiencies of past efforts, is also presented. This approach has been applied to building a hierarchically structured benchmark that tests part of the Unix file and virtual memory systems. The resultant benchmark has successfully been used to identify new response class structures that were not detected in a similar situation by other less organized techniques.

[1] P. Amaral, C. Jacquemot, and R. Lea, "Implementing a Modular Object Oriented Distributed Operating System on Top of CHORUS," Proc. Open Forum '92,Utrecht, The Netherlands, Nov. 1992.
[2] J.H. Barton, E.W. Czeck, Z.Z. Segall, and D.P. Siewiorek, Fault Injection Experiments Using FIAT IEEE Trans. Computers, vol. 39, no. 4, pp. 575-582, Apr. 1990.
[3] R. Campbell and N. Islam, "Choices: A Parallel Object-Oriented Operating System," Research Directions in Concurrent Object-Oriented Programming. MIT Press, 1993.
[4] F. Cristian, "Understanding Fault-Tolerant Distributed Systems," Comm. ACM, vol. 34, no. 2, Feb. 1991.
[5] H.J. Curnow and B.A. Wichmann, "A Synthetic Benchmark," The Computer J., vol. 19, no. 1, pp. 43-49, 1976.
[6] C.P. Dingman, J. Marshall, and D.P. Siewiorek, "Measuring Robustness of a Fault Tolerant Aerospace System," Proc. 25th Int'l Symp. Fault-Tolerant Computing,Pasadena, Calif., pp. 522-527, June 1995.
[7] J. Gosling and H. McGilton, "The Java(tm) Language Environment: A White Paper," unpublished, whitePaper/java-whitepaper-1.html.
[8] G. Hamilton and P. Kougiouris, "The Spring Nucleus: A Microkernel for Objects," Technical Report TR-93-14, Sun Microsystems Laboratories,Mountain View, Calif..
[9] G.A. Kanawati, N.A. Kanawati, and J.A. Abraham, "FERRARI: A Proc. IEEE Workshop Fault-Tolerant Parallel and Distributed Systems,Amherst, Mass., July 1992.
[10] W. Kao, R. Iyer, and D. Tang, "FINE: A Fault Injection and Monitoring Environment for Tracing the UNIX System Behavior Under Faults," IEEE Trans. Software Eng., vol. 19, no. 11, pp. 1,105-1,118, Nov. 1993.
[11] B.P. Miller, L. Fredrikson, and B. So, "An Empirical Study of the Reliability of Unix Utilities," Comm. ACM, Dec. 1990, pp. 32-44.
[12] M. Rozier, V. Abrossimov, F. Armand, I. Boule, M. Gien, M. Guillermont, F. Herrman, C. Kaiser, S. Langlois, P. Leonard, and W. Neuhauser, "Overview of the Chorus Distributed Operating System," Proc. Usenix Workshop Micro-kernels and Other Kernel Architectures,Seattle, Wash., Apr. 1992.
[13] M.E. Russinovich and Z. Segall, "Open System Fault Management: Fault Tolerant MACH," Research Report CMUCDS-92-8, CMU Research Center for Dependable Systems, Carnegie Mellon Univ., Pittsburgh, Pa.
[14] Standard Performance Evaluation Corporation, SPEC Newsletter, vol. 2, Issue 2, Waterside Assoc., Fremont, Calif., Spring 1990.
[15] B.-H. Suh, J. Hudak, D. Siewiorek, and Z. Segall, "Development of a Benchmark to Measure System Robustness," Fault Tolerant Computing Systems: Proc. 23rd Int'l Symp.,Toulouse, France, June 1993.
[16] M. Sullivan and R. Chillarege, "Software Defects and Their Impact on System Availability—A Study of Field Failures in Operating Systems," Proc. Int'l Symp. Fault-Tolerant Computing, pp. 2-9, 1991.
[17] R.P. Weicker, "Dhrystone: A Synthetic Systems Programming Benchmark," Comm. ACM, vol. 27, no. 10, pp. 1,013-1,030, Oct. 1984.
[18] N. Weiderman, "Hartstone: Synthetic Benchmark Requirements for Hard Real-Time Applications," Ada Letters, vol. 10, no. 3, pp. 126-136, Winter 1990.

Index Terms:
System reliability, software dependability, robustness benchmarking, test suite organization, object-oriented benchmarks, software validation, extensible benchmarks.
Arup Mukherjee, Daniel P. Siewiorek, "Measuring Software Dependability by Robustness Benchmarking," IEEE Transactions on Software Engineering, vol. 23, no. 6, pp. 366-378, June 1997, doi:10.1109/32.601075
Usage of this product signifies your acceptance of the Terms of Use.