This Article 
 Bibliographic References 
 Add to: 
The Exception Handling Effectiveness of POSIX Operating Systems
September 2000 (vol. 26 no. 9)
pp. 837-848

Abstract—Operating systems form a foundation for robust application software, making it important to understand how effective they are at handling exceptional conditions. The Ballista testing system was used to characterize the handling of exceptional input parameter values for up to 233 POSIX functions and system calls on each of 15 widely used operating system (OS) implementations. This identified ways to crash systems with a single call, ways to cause task hangs within OS code, ways to cause abnormal task termination within OS and library code, failures to implement defined POSIX functionality, and failures to report unsuccessful operations. Overall, only 55 percent to 76 percent of the exceptional tests performed generated error codes, depending on the operating system being tested. Approximately 6 percent to 19 percent of tests failed to generate any indication of error despite exceptional inputs. Approximately 1 percent to 3 percent of tests revealed failures to implement defined POSIX functionality for unusual, but specified, situations. Between 18 percent and 33 percent of exceptional tests caused the abnormal termination of an OS system call or library function, and five systems were completely crashed by individual system calls with exceptional parameter values. The most prevalent sources of these robustness failures were illegal pointer values, numeric overflows, and end-of-file overruns. There is significant opportunity for improving exception handling within OS calls and especially within C library functions. However, the role of signals vs. error return codes is both controversial and the source of divergent implementation philosophies, forming a potential barrier to writing portable, robust applications.

[1] A. Avizienis, "The N-Version Approach to Fault-Tolerant Software," IEEE Trans. Software Eng., vol. 11, no. 12, pp. 1,491-1,501, 1985.
[2] J.H. Barton, E.W. Czeck, Z.Z. Segall, and D.P. Siewiorek, Fault Injection Experiments Using FIAT IEEE Trans. Computers, vol. 39, no. 4, pp. 575-582, Apr. 1990.
[3] B. Beizer, Black Box Testing. New York: Wiley 1995.
[4] G. Carrette, "CRASHME: Random Input Testing," , accessed 6 July 1998.
[5] F. Cristian, "Exception Handling and Tolerance of Software Faults," Software Fault Tolerance, Michael R. Lyu, ed., chapter 4, pp. 81-107, Chichester: Wiley, 1995.
[6] E. Czeck, F. Feather, A. Grizzaffi, G. Finelli, Z. Segall, and D. Siewiorek, “Fault-Free Performance Validation of Avionic Multiprocessors,” Proc. IEEE/AIAA Seventh Digital Avionics Systems Conf., pp. 670-677, Oct. 1986.
[7] J. DeVale, P. Koopman, D. Guttendorf, “The Ballista Software Robustness Testing Service,” Proc. 16th Int'l Conf. Testing Computer Software, pp. 33–42, 1999.
[8] C.P. Dingman, J. Marshall, and D.P. Siewiorek, "Measuring Robustness of a Fault Tolerant Aerospace System," Proc. 25th Int'l Symp. Fault-Tolerant Computing,Pasadena, Calif., pp. 522-527, June 1995.
[9] C. Dingman, “Portable Robustness Benchmarks,” Ph.D. thesis, Dept. of Electrical and Computer Eng., Carnegie Mellon Univ., Pittsburgh, Penn., May 1997.
[10] U.S. Department of Defense, “High Level Architecture Run Time Infrastructure Programmer's Guide, RTI 1.3 Version 5,” Dec. 16, 1998.
[11] K. Fernsler and P. Koopman, “Robustness Testing of a Distributed Simulation Backplane,” Proc.10th Int'l Symp. Software Reliability Eng., Nov. 1-4, 1999.
[12] N. Gehani, “Exceptional C or C with Exceptions,” Software–Practice and Experience, vol. 22, no. 10, pp. 827-48, 1992.
[13] J. Goodenough, “Exception Handling: Issues and Proposed Notation,” Comm. ACM, Vol. 18, No. 12, pp. 683–696, 1975.
[14] I. Hill, “Faults in Functions, in ALGOL and FORTRAN,” The Computer J., vol. 14, no. 3, pp. 315–316, Aug. 1971.
[15] IEEE Standard Glossary of Software Engineering Terminology. IEEE Std 610.12–1990, IEEE CS, Dec. 10, 1990.
[16] IEEE Standard for Information Technology—Portable Operating System Interface (POSIX) Part 1: System Application Program Interface (API) Amendment 1: Realtime Extension [C Language]. IEEE Std 1003.1b–1993, IEEE CS, 1994.
[17] E. Jones, ed., The Apollo Lunar Surface J., Apollo 11 Lunar Landing, Entries 102:38:30, 102:42:22, and 102:42:41, National Aeronautics and Space Administration, Washington, DC, 1996.
[18] G. Kanawati, N. Kanawati, and J. Abraham, “FERRARI: A Tool for the Validation of System Dependability Properties,” Proc. IEEE Int'l Symp. Fault-Tolerant Computing, pp. 336–344, 1992.
[19] P. Koopman, J. Sung, C. Dingman, D. Siewiorek, and T. Marz, “Comparing Operating Systems Using Robustness Benchmarks,” Proc. Symp. Reliable and Distributed Systems, pp. 72–79, 1997.
[20] P. Koopman and J. DeVale, Comparing the Robustness of POSIX Operating Systems Proc. 29th Int'l Symp. Fault-Tolerant Computing (FTCS-29), pp. 30-37, 1999.
[21] D. Korn and K.-P. Vo, “SFIO: Safe/Fast String/File IO,” Proc. Summer 1991 USENIX Conf., pp. 235-256, 1991.
[22] N.P. Kropp, P.J. Koopman, and D.P. Siewiorek, Automated Robustness Testing of Off-the-Shelf Software Components Proc. Symp. Fault-Tolerant Computing (FTCS), pp. 230-239, 1998, .
[23] J.L. Lions, chairman,“Ariane 5 Flight 501 Failure: Report by the Inquiry Board,” European Space Agency, July 19, 1996.
[24] B. Marick, The Craft of Software Testing. Prentice Hall, 1995.
[25] B.P. Miller, L. Fredrikson, and B. So, "An Empirical Study of the Reliability of Unix Utilities," Comm. ACM, Dec. 1990, pp. 32-44.
[26] B. Miller, D. Koski, C. Lee, V. Maganty, R. Murthy, A. Natarajan, and J. Steidl, “Fuzz Revisited: A Re-examination of the Reliability of UNIX Utilities and Services” Computer Science Technical Report 1268, Univ. of Wisconsin-Madison, May 1998.
[27] J. Musa, G. Fuoco, N. Irving, D. Kropfl, and B. Julin, “The Operational Profile,” Handbook of Software Reliability Eng., M.R. Lye ed., chapter 5, pp. 167-216, 1996.
[28] Object Management Group, “The Common Object Request Broker: Architecture and Specification, Revision 2.0,” July 1995.
[29] T.J. Ostrand and M.J. Balcer, “The Category-Partition Method for Specifying and Generating Functional Tests,” Comm. ACM, vol. 31, no. 6, June 1988.
[30] M. Schuette, J. Shen, D. Siewiorek, and Y. Zhu, “Experimental Evaluation of Two Concurrent Error Detection Schemes,” Digest of Papers; Proc. 16th Ann. Int'l Symp. Fault-Tolerant Computing Systems, pp. 138-43, 1986.
[31] D.P. Siewiorek, J.J. Hudak, B.-H. Suh, and Z. Segal, “Development of a Benchmark to Measure System Robustness,” Proc. 1993 Int'l Symp. Fault-Tolerant Computing, pp. 88-97, June 1993.
[32] A. Srivastava and A. Eustace, “ATOM: A System for Building Customized Program Analysis Tools,” Research Report WRL–94/2, Digital Western Research Lab, Palo Alto, Calif., 1994.
[33] T. Tsai and R. Iyer, “Measuring Fault Tolerance with the FTAPE Fault Injection Tool,” Proc. Eighth Int'l Conf. Modeling Techniques and Tools for Computer Performance Evaluation, pp. 26-40, 1995.
[34] R. Uhlig, D. Nagle, T. Mudge, S. Sechrest, and J. Emer, “Instruction Fetching: Coping with Code Bloat,” Proc. 22nd Ann. Int'l Symp. Computer Architecture, pp. 345-356, June 1995.
[35] K-P. Vo, Y.-M. Wang, P. Chung, and Y. Huang, “Xept: A Software Instrumentation Method for Exception Handling,” Proc. Eighth Int'l Symp. Software Reliability Eng., pp. 60–69, 1997.

Index Terms:
Exception handling, POSIX, operating systems, robustness, testing, Ballista, multiversion comparison.
Philip Koopman, John DeVale, "The Exception Handling Effectiveness of POSIX Operating Systems," IEEE Transactions on Software Engineering, vol. 26, no. 9, pp. 837-848, Sept. 2000, doi:10.1109/32.877845
Usage of this product signifies your acceptance of the Terms of Use.