This Article 
 Bibliographic References 
 Add to: 
Robustness Testing of Java Server Applications
April 2005 (vol. 31 no. 4)
pp. 292-311
Ana Milanova, IEEE Computer Society
Barbara Gershon Ryder, IEEE Computer Society
David G. Wonnacott, IEEE Computer Society
This paper presents a new compile-time analysis that enables a testing methodology for white-box coverage testing of error recovery code (i.e., exception handlers) of server applications written in Java, using compiler-directed fault injection. The analysis allows compiler-generated instrumentation to guide the fault injection and to record the recovery code exercised. (An injected fault is experienced as a Java exception.) The analysis 1) identifies the exception-flow "def-uses” to be tested in this manner, 2) determines the kind of fault to be requested at a program point, and 3) finds appropriate locations for code instrumentation. The analysis incorporates refinements that establish sufficient context sensitivity to ensure relatively precise def-use links and to eliminate some spurious def-uses due to demonstrably infeasible control flow. A runtime test harness calculates test coverage of these links using an exception def-catch metric. Experiments with the methodology demonstrate the utility of the increased precision in obtaining good test coverage on a set of moderately sized server benchmarks.

[1] J. Arlat, A. Costes, Y. Crouzet, J.-C. Laprie, and D. Powell, “Fault Injection and Dependability Evaluation of Fault-Tolerant Systems,” IEEE Trans. Computers, vol. 42, no. 8, pp. 913-923, Aug. 1993.
[2] M. Cukier, R. Chandra, D. Henke, J. Pistole, and W.H. Sanders, “Fault Injection Based on a Partial View of the Global State of a Distributed System,” Proc. Symp. Reliable Distributed Systems, pp. 168-177, 1999.
[3] S. Dawson, F. Jahanian, and T. Mitton, “ORCHESTRA: A Fault Injection Environment for Distributed Systems,” Proc. 26th Int'l Symp. Fault Tolerant Computing (FTCS-26), pp. 404-414, June 1996.
[4] S. Han, K. Shin, and H. Rosenberg, “DOCTOR: An Integrated Software Fault Injection Environment for Distributed Real-Time Systems,” Proc. Int'l Computer Performance and Dependability Symp. (IPDS '95), pp. 204-213, Apr. 1995.
[5] G.A. Kanawati, N.A. Kanawati, and J.A. Abraham, “FERRARI: A Tool for the Validation of System Dependability Properties,” Proc. 22nd Int'l Symp. Fault Tolerant Computing (FTCS-22), pp. 336-344, 1992.
[6] Z. Segall, D. Vrsalovic, D. Siewiorek, D. Yaskin, J. Kownacki, J. Barton, D. Rancey, A. Robinson, and T. Lin, “FIAT— Fault Injection Based Automated Testing Environment,” Proc. 18th Int'l Symp. Fault-Tolerant Computing (FTCS-18), pp. 102-107, 1988.
[7] R.V. Binder, Testing Object-Oriented Systems. Addison Wesley, 1999.
[8] G.J. Myers, The Art of Software Testing. John Wiley and Sons, 1979.
[9] R.G. Hamlet, “Testing Programs with the Aid of a Compiler,” IEEE Trans. Software Eng., vol. 3, no. 4, pp. 279-290, July 1977.
[10] S. Rapps and E. Weyuker, “Selecting Software Test Data Using Data Flow Information,” IEEE Trans. Software Eng., vol. 11, no. 4, pp. 367-375, Apr. 1985.
[11] C. Fu, R.P. Martin, K. Nagaraja, T.D. Nguyen, B.G. Ryder, and D. Wonnacott, “Compiler-Directed Program-Fault Coverage for Highly Available Java Internet Services,” Proc. Int'l Conf. Dependable Systems and Networks (DSN 2003), June 2003.
[12] C. Fu, R.P. Martin, K. Nagaraja, T.D. Nguyen, B.G. Ryder, and D.G. Wonnacott, “Compiler-Directed Program-Fault Coverage for Highly Available Java Internet Services,” Technical Report DCS-TR-518, Dept. of Computer Science, Rutgers Univ., Jan. 2003.
[13] C. Fu, B.G. Ryder, A. Milanova, and D. Wonnacott, “Testing of Java Web Services For Robustness,” Proc. Int'l Symp. Software Testing and Analysis (ISSTA), pp. 23-33, July 2004.
[14] F. Tip and J. Palsberg, “Scalable Propagation-Based Call Graph Construction Algorithms,” Proc. Conf. Object-Oriented Programming, Languages, Systems and Applications, pp. 281-293, Oct. 2000.
[15] K. Arnold and J. Gosling, The Java Programming Language, second ed. Addison-Wesley, 1997.
[16] X. Li, R.P. Martin, K. Nagaraja, T.D. Nguyen, and B. Zhang, “Mendosus: A SAN-Based Fault-Injection Test-Bed for the Construction of Highly Available Network Services,” Proc. First Workshop Novel Uses of System Area Networks (SAN-1), Jan. 2002.
[17] A. Rountev, A. Milanova, and B.G. Ryder, “Points-To Analysis for Java Using Annotated Constraints,” Proc. Conf. Object-Oriented Programming, Languages, Systems and Applications, pp. 43-55, 2001.
[18] Sable, McGill, “Soot: A Java Optimization Framework,” http://www.sablemcgill.casoot, 2003.
[19] J. Dean, D. Grove, and C. Chambers, “Optimization of Object-Oriented Programs Using Static Class Hierarchy,” Proc. Ninth European Conf. Object-Oriented Programming (ECOOP '95), pp. 77-101, 1995.
[20] D. Bacon and P. Sweeney, “Fast Static Analysis of C++ Virtual Functions Calls,” Proc. ACM SIGPLAN Conf. Object-Oriented Programing Systems, Languages and Applications (OOPSLA '96), pp. 324-341, Oct. 1996.
[21] D. Grove and C. Chambers, “A Framework for Call Graph Construction Algorithms,” ACM Trans. Programming Languages and Systems (TOPLAS), vol. 23, no. 6, 2001.
[22] T. Ogasawara, H. Komatsu, and T. Nakatani, “A Study of Exception Handling and Its Dynamic Optimization in Java,” Proc. ACM SIGPLAN Conf. Object-oriented Programing Systems, Languages and Applications (OOPSLA '01), pp. 83-95, 2001.
[23] A.V. Aho, R. Sethi, and J.D. Ullman, Compilers— Principles, Techniques and Tools. Addison Wesley, 1988.
[24] T.J. Marlowe and B.G. Ryder, “Properties of Data Flow Frameworks: A Unified Model,” Acta Informatica, vol. 28, pp. 121-163, 1990.
[25] A. Milanova, A. Rountev, and B.G. Ryder, “Parameterized Object Sensitivity for Points-To Analysis for Java,” ACM Trans. Software Eng. and Methodology, vol. 14, no. 1, Jan. 2005.
[26] A. Milanova, “Precise and Practical Flow Analsis of Object-Oriented Software,” PhD dissertation, Rutgers Univ., 2003.
[27] M.L. Scott, Programming Language Pragmatics. Morgan Kaufmann, 2000.
[28] M. Sharir and A. Pnueli, “Two Approaches to Interprocedural Data Flow Analysis,” Program Flow Analysis: Theory and Applications, S. Muchnick and N. Jones, eds., Prentice Hall, pp. 189-234, 1981.
[29] O. Shivers, “Control-Flow Analysis of Higher-Order Languages,” PhD dissertation, Carnegie Mellon Univ., 1991.
[30] O. Lhoták and L. Hendren, “Scaling Java Points-To Analysis Using Spark,” Proc. Int'l Conf. Compiler Construction, pp. 153-169, 2003.
[31] B.G. Ryder, “Dimensions of Precision in Reference Analysis of Object-Oriented Programming Languages,” Proc. 12th Int'l Conf. Compiler Construction, pp. 126-137, Apr. 2003.
[32] P. Sortokin, “Ftp Server in Java,” , 2003.
[33] M.J. Radwin, “The Java Network File System,” http://www. /, 2003.
[34] “The Muffin World Wide Web Filtering System,” http:/, 2003.
[35] M. Welsh, D.E. Culler, and E.A. Brewer, “SEDA: An Architecture for Well-Conditioned, Scalable Internet Services,” Proc. Symp. Operating Systems Principles, pp. 230-243, http://www.cs.harvard. edu/~mdw/projseda /, 2001.
[36] Apache Software Foundation, “Apache Jarkarta Project,” http:/, 2004.
[37], “Spec jvm98 Benchmarks,” http://www.spec. orgjvm98/, 1998.
[38] Vola no LLC, “Volanomark,” benchmarks.html, 2004.
[39] D. Tang and R.K. Iyer, “Analysis and Modeling of Correlated Failures in Multicomputer Systems,” ACM Trans. Computer Systems, pp. 567-577, May 1992.
[40] K. Nagaraja, X. Li, B. Zhang, R. Bianchini, R.P. Martin, and T.D. Nguyen, “Using Fault Injection to Evaluate the Performability of Cluster-Based Services,” Proc. Fourth USENIX Symp. Internet Technologies and Systems (USITS 2003), Mar. 2003.
[41] Apache Software Foundation, “Apache Jakarta Tomcat,” http://jakarta.apache.orgtomcat/, 2004.
[42] W.G. Bouricius, W.C. Carter, and P. Schneider, “Reliability Modeling Techniques for Self Repairing Computer Systems,” Proc. 24th Nat'l Conf. the ACM, pp. 295-309, Mar. 1969.
[43] T. Tsai, M. Hsueh, H. Zhao, Z. Kalbarczyk, and R. Iyer, “Stress-Based and Path-Based Fault Injection,” IEEE Trans. Computers, vol. 48, no. 11, pp. 1183-1201, Nov. 1999.
[44] J. Bieman, D. Dreilinger, and L. Lin, “Using Fault Injection to Increase Software Test Coverage,” Proc. Seventh Int'l Symp. Software Reliability Eng. (ISSRE '96), pp. 166-174, 1996.
[45] D. Hamlet, “Foundations of Software Testing: Dependability Theory,” Proc. Second ACM SIGSOFT Symp. Foundations of Software Eng., pp. 128-139, 1994.
[46] D. Tang and H. Hecht, “An Approach to Measuring and Assessing Dependability for Critical Software Systems,” Proc. Eighth Int'l Symp. Software Reliability Eng., pp. 192-202, Nov. 1997.
[47] P. Frankl and E. Weyuker, “An Applicable Family of Data Flow Testing Criteria,” IEEE Trans. Software Eng., vol. 14, no. 10, pp. 1483-1498, Oct. 1988.
[48] S. Sinha and M.J. Harrold, “Criteria for Testing Exception-Handling Constructs in Java Programs,” Proc. Int'l Conf. Software Maintenance, 1999.
[49] M.P. Robillard and G.C. Murphy, “Static Analysis to Support the Evolution of Exception Structure in Object-Oriented Systems,” ACM Trans. Software Eng. and Methodology (TOSEM), vol. 12, no. 2, pp. 191-221, 2003.
[50] J.-W. Jo, B.-M. Chang, K. Yi, and K.-M. Cho, “An Uncaught Exception Analysis for Java,” J. Systems and Software, 2004.
[51] N. Heintze, “Set-Based Analysis of Ml Programs,” Proc. ACM Conf. Lisp and Functional Programmig, pp. 306-317, 1994.
[52] S. Sinha and M.J. Harrold, “Analysis and Testing of Programs With Exception-Handling Constructs,” IEEE Trans. Software Eng., vol. 26, no. 9, pp. 849-871, Sept. 2000.
[53] S. Sinha, A. Orso, and M.J. Harrold, “Automated Support for Development, Maintenance, and Testing in the Presence of Implicit Control Flow,” Proc. 26th Int'l Conf. Software Eng. (ICSE '04), 2004.
[54] J.-D. Choi, D. Grove, M. Hind, and V. Sarkar, “Efficient and Precise Modeling of Exceptions for Analysis of Java Programs,” Proc. ACM SIGPLAN-SIGSOFT Workshop Program Analysis for Software Tools and Eng., pp. 21-31, Sept. 1999.
[55] S. Lee, B.-S. Yang, S. Kim, S. Park, S.-M. Moon, K. Ebcioglu, and E. Altman, “Efficient Java Exception Handling in Just-in-Time Compilation,” Proc. ACM SIGPLAN Java Grande Conf., 2000.
[56] M. Cierniak, G.-Y. Lueh, and J.M. Stichnoth, “Practicing Judo: Java under Dynamic Optimzations,” Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, pp. 13-26, 2000.
[57] J.D.D. Grove, G. DeFouw, and C. Chambers, “Call Graph Construction in Object-Oriented Languages,” Proc. ACM SIGPLAN Conf. Object-Oriented Programing Systems, Languages and Applications (OOPSLA '97), pp. 108-124, Oct. 1997.
[58] R. O'Callahan, “The Generalized Aliasing as a Basis for Software Tools,” PhD dissertation, Carnegie Mellon Univ., 2000.
[59] R. Chatterjee, B.G. Ryder, and W.A. Landi, “Relevant Context Inference,” Proc. ACM SIGACT/SIGPLAN Symp. Principles of Programming Languages, Jan. 1999.
[60] R. Bodik, R. Gupta, and M.L. Soffa, “Refining Data Flow Information Using Infeasible Paths,” Proc. Sixth European Software Eng. Conf. (ESEC/FSE 97), M. Jazayeri and H. Schauer, eds., pp. 361-377, 1997.
[61] A.L. Souter and L.L. Pollock, “Type Infeasible Call Chains,” Proc. IEEE Int'l Workshop Source Code Analysis and Manipulation, 2001.
[62] A.L. Souter and L.L. Pollock, “Characterization and Automatic Identification of Type Infeasible Call Chains,” Information and Software Technology, vol. 44, no. 13, pp. 721-732, Oct. 2002.
[63] A. Rountev, S. Kagan, and M. Gibas, “Static and Dynamic Analysis of Call Chains in Java,” Proc. Int'l Symp. Software Testing and Analysis, pp. 1-11, July 2004.

Index Terms:
Index Terms- Reliability, def-use testing, Java, exceptions, test coverage metrics.
Chen Fu, Ana Milanova, Barbara Gershon Ryder, David G. Wonnacott, "Robustness Testing of Java Server Applications," IEEE Transactions on Software Engineering, vol. 31, no. 4, pp. 292-311, April 2005, doi:10.1109/TSE.2005.51
Usage of this product signifies your acceptance of the Terms of Use.