This Article 
 Bibliographic References 
 Add to: 
Eliminating Exception Handling Errors with Dependability Cases: A Comparative, Empirical Study
September 2000 (vol. 26 no. 9)
pp. 888-906

Abstract—Programs fail mainly for two reasons: logic errors in the code and exception failures. Exception failures can account for up to two-thirds of system crashes [1], hence, are worthy of serious attention. Traditional approaches to reducing exception failures, such as code reviews, walkthroughs, and formal testing, while very useful, are limited in their ability to address a core problem: The programmer's inadequate coverage of exceptional conditions. The problem of coverage might be rooted in cognitive factors that impede the mental generation (or recollection) of exception cases that would pertain in a particular situation, resulting in insufficient software robustness. This paper describes controlled experiments for testing the hypothesis that robustness for exception failures can be improved through the use of various coverage-enhancing techniques: N-version programming, group collaboration, and dependability cases. N-version programming and collaboration are well known. Dependability cases, derived from safety cases, comprise a new methodology based on structured taxonomies and memory aids for helping software designers think about and improve exception handling coverage. All three methods showed improvements over control conditions in increasing robustness to exception failures but dependability cases proved most efficacious in terms of balancing cost and effectiveness. A controlled experiment conducted with 119 subjects revealed a statistically significant 34 percent increase (p < .01) in exception handling robustness corresponding to use of dependability cases. An ancillary experiment conducted with 53 subjects provided convergent evidence that the effect is authentic and not due to programming expertise alone.

[1] F. Cristian, “Exception Handling and Tolerance of Software Faults,” Software Fault Tolerance, M.R. Lyu, ed., pp. 81–107, Wiley, Chichester, 1995.
[2] J.L. Lions, “ARIANE 5 Flight 501 Failure: Report by the Inquiry Board,” 1996. http://www.esrin.esait/htdocs/tidc/Press/ Press96ariane5rep.html
[3] E. Kit, Software Testing in the Real World: Improving the Process. Harlow, England: Addison-Wesley, 1995.
[4] B. Boehm, Software Engineering Economics, Prentice Hall, Upper Saddle River, N.J., 1981, pp. 533-535.
[5] M. Bush, “Improving Software Quality: The Use of Formal Inspections at the Jet Propulsion Laboratory,” Proc. 12th Int'l Conf. Software Eng., pp. 196–199, 1990.
[6] Computer Emergency Response Team (CERT), CMU Software Engineering Inst., Pittsburgh, Penn., personal communication, Oct. 1997.
[7] J.C. Laprie, “Dependability—Its Attributes, Impairments and Means,” Predictably Dependable Computing Systems, B. Randell, J.C. Laprie, H. Kopetz, and B. Littlewood, eds., pp. 3–24, Springer-Verlag, 1995.
[8] Dictionary of Computing. V. Illingworth, ed., Oxford Univ. Press, forth ed., 1997.
[9] W.N. Toy, “Fault-Tolerant Design of Local ESS Processors,” The Theory and Practice of Reliable System Design, D.P. Siewiorek and R.S. Swarz, eds., pp. 461–496, Bedford, Mass.: Digital Press, 1982.
[10] W. James, The Principles of Psychology. New York: Henry Holt and Company, 1890.
[11] A.D. Swain and H.E. Guttmann, “Handbook of Human Reliability Analysis with Emphasis on Nuclear Power Plant Applications,” Technical Report NUREG/CR-1278, U.S. Nuclear Regulatory Commission, 1983.
[12] J. Reason, Human Error. Cambridge, England: Cambridge Univ. Press, 1990.
[13] W.A. Wickelgren, Learning and Memory. Englewood Cliffs, N.J.: Prentice-Hall, 1977.
[14] G.J. Myers,The Art of Software Testing.New York: Wiley, 1979.
[15] E. Yourdon, Structured Walkthroughs. Englewood Cliffs, N.J.: Prentice-Hall, second ed., 1979.
[16] N.F. Schneidewind and H.M. Hoffmann, “An Experiment in Software Error Data Collection and Analysis,” IEEE Trans. Software Eng., vol. 5, no. 3, pp. 276–286, May 1979.
[17] G.S. Rotter and S.M. Portugal, “Group and Individual Effects in Problem Solving,” J. Applied Psychology, vol. 53, no. 4, pp. 338–341, Aug. 1969.
[18] J.E. McGrath, Groups: Interaction and Performance. Englewood Cliffs, N.J.: Prentice-Hall, 1984.
[19] P.B. Paulus, M.T. Dzindolet, G. Poletes, and L.M. Camacho, “Perception of Performance in Group Brainstorming: The Illusion of Group Productivity,” Personality and Social Psychology Bulletin, vol. 19, no. 1, pp. 78–89, Feb. 1993.
[20] G.W. Hill, “Group versus Individual Performance: Are N+1 Heads Better Than One?,” Psychological Bulletin, vol. 91, no. 3, pp. 517–539, May 1982.
[21] N.K. Clark and G.M. Stephenson, “Group Remembering,” Psychology of Group Influence, P.B. Paulus, ed., pp. 357–391, Hillsdale, N.J.: Lawrence Erlbaum Associates, second ed., 1989.
[22] A. Avizienis and L. Chen, “On the Implementation of N-version Programming for Software Fault Tolerance During Execution,” Proc. COMPSAC-77: First IEEE-CS Int'l Computer Software and Application Conf., pp. 149–155, 1977.
[23] L. Chen and A. Avizienis, “N-Version Programming: A Fault-Tolerant Approach to Reliability of Software Operation,” Proc. Eighth Ann. Int'l Conf. Fault-Tolerant Computing, pp. 3–9, 1978.
[24] A. Avizienis, “The N-Version Approach to Fault-Tolerant Software,” IEEE Trans. Software Eng., vol. 11, no. 12, pp. 1,491–1,501, Dec. 1985.
[25] K.D. Forbus, “The Qualitative Process Engine,” Readings in Qualitative Reasoning about Physical Systems, D. Weld and J. de Kleer, eds, pp. 220-235. San Mateo, Calif.: Morgan Kaufmann, 1990.
[26] P. Bishop, “Software Fault Tolerance by Design Diversity,” Software Fault Tolerance, M.R. Lu, ed., pp. 211–228, Wiley, Chichester, 1995.
[27] J.C. Knight and N.G. Leveson, "An Experimental Evaluation of the Assumption of Independence in Multiversion Programming," IEEE Trans. Software Eng., Vol. 12, No. 1, 1986, pp. 96-109.
[28] L. Hatton, "N-Version Design Versus One Good Version," IEEE Software, Nov./Dec. 1997, pp. 71-76.
[29] N.E. Fenton, S. Lawrence Pfleeger, and R. Glass, “Science and Substance: A Challenge to Software Engineers,” IEEE Software, vol. 11, no. 4, pp. 86–95, July 1994.
[30] Safety and Reliability of Software Based Systems. R. Shaw, ed., Springer-Verlag, 1997.
[31] R. Shaw, “Safety Cases—How Did We Get Here?,” Safety and Reliability of Software Based Systems, R. Shaw, ed., pp. 43–95, Springer-Verlag, 1997.
[32] P.G. Bishop and R.E. Bloomfield, “The SHIP Safety Case Approach,” SafeComp-95: Proc 14th Int'l Conf. Computer Safety, Reliability and Security, G. Rabe, ed., pp. 437–451, 1995.
[33] T. Kletz, Hazop and Hazan: Identifying and Assessing Process Industry Hazards. Inst. Chemical Engineers, Rugby, Warwickshire, England, 1992.
[34] N.G. Leveson, Safeware: System Safety and Computers. Addison-Wesley, 1995.
[35] W.E. Vesely, F.F. Goldberg, N.H. Roberts, and D.F. Haasl, “Fault Tree Handbook,” Technical Report NUREG-0492, U.S. Nuclear Regulatory Commission, 1981.
[36] J. Bertin, Semiology of Graphics. Madison, Wis.: Univ. of Wisconsin Press, 1983.
[37] K. Ishikawa, Guide to Quality Control. Asian Productivity Organization, Tokyo, 1982.
[38] Health and Safety Commission, The Use of Computers in Safety-Critical Applications. Suffolk, UK: HSE Books, 1998.
[39] J. Klayman and Y.W. Ha, “Confirmation, Disconfirmation, and Information in Hypothesis Testing,” Psychological Review, vol. 94, no. 2, pp. 211–228, Apr. 1987.
[40] C. Chatfield, Statistics for Technology: A Course on Applied Statistics. Chapman&Hall, London, 1983.
[41] R.A. Maxion and R.T. Olszewski, “Improving Software Robustness with Dependability Cases,” Proc. 28th Int'l Symp. Fault-Tolerant Computing, pp. 346–355, 1998.
[42] W.S. Humphrey, A Discipline for Software Engineering, Addison-Wesley, Reading, Mass., 1995.
[43] T. Glib, Principles of Software Engineering Management. Workingham, England: Addison-Wesley, 1995.

Index Terms:
Collaboration, dependability, design diversity, empirical methods, exception handling, N-version programming, safety cases, software engineering, software robustness.
Roy A. Maxion, Robert T. Olszewski, "Eliminating Exception Handling Errors with Dependability Cases: A Comparative, Empirical Study," IEEE Transactions on Software Engineering, vol. 26, no. 9, pp. 888-906, Sept. 2000, doi:10.1109/32.877848
Usage of this product signifies your acceptance of the Terms of Use.