This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Robustness Testing of Java Server Applications
April 2005 (vol. 31 no. 4)
pp. 292-311
Ana Milanova, IEEE Computer Society
Barbara Gershon Ryder, IEEE Computer Society
David G. Wonnacott, IEEE Computer Society
This paper presents a new compile-time analysis that enables a testing methodology for white-box coverage testing of error recovery code (i.e., exception handlers) of server applications written in Java, using compiler-directed fault injection. The analysis allows compiler-generated instrumentation to guide the fault injection and to record the recovery code exercised. (An injected fault is experienced as a Java exception.) The analysis 1) identifies the exception-flow "def-uses” to be tested in this manner, 2) determines the kind of fault to be requested at a program point, and 3) finds appropriate locations for code instrumentation. The analysis incorporates refinements that establish sufficient context sensitivity to ensure relatively precise def-use links and to eliminate some spurious def-uses due to demonstrably infeasible control flow. A runtime test harness calculates test coverage of these links using an exception def-catch metric. Experiments with the methodology demonstrate the utility of the increased precision in obtaining good test coverage on a set of moderately sized server benchmarks.

[1] J. Arlat, A. Costes, Y. Crouzet, J.-C. Laprie, and D. Powell, “Fault Injection and Dependability Evaluation of Fault-Tolerant Systems,” IEEE Trans. Computers, vol. 42, no. 8, pp. 913-923, Aug. 1993.
[2] M. Cukier, R. Chandra, D. Henke, J. Pistole, and W.H. Sanders, “Fault Injection Based on a Partial View of the Global State of a Distributed System,” Proc. Symp. Reliable Distributed Systems, pp. 168-177, 1999.
[3] S. Dawson, F. Jahanian, and T. Mitton, “ORCHESTRA: A Fault Injection Environment for Distributed Systems,” Proc. 26th Int'l Symp. Fault Tolerant Computing (FTCS-26), pp. 404-414, June 1996.
[4] S. Han, K. Shin, and H. Rosenberg, “DOCTOR: An Integrated Software Fault Injection Environment for Distributed Real-Time Systems,” Proc. Int'l Computer Performance and Dependability Symp. (IPDS '95), pp. 204-213, Apr. 1995.
[5] G.A. Kanawati, N.A. Kanawati, and J.A. Abraham, “FERRARI: A Tool for the Validation of System Dependability Properties,” Proc. 22nd Int'l Symp. Fault Tolerant Computing (FTCS-22), pp. 336-344, 1992.
[6] Z. Segall, D. Vrsalovic, D. Siewiorek, D. Yaskin, J. Kownacki, J. Barton, D. Rancey, A. Robinson, and T. Lin, “FIAT— Fault Injection Based Automated Testing Environment,” Proc. 18th Int'l Symp. Fault-Tolerant Computing (FTCS-18), pp. 102-107, 1988.
[7] R.V. Binder, Testing Object-Oriented Systems. Addison Wesley, 1999.
[8] G.J. Myers, The Art of Software Testing. John Wiley and Sons, 1979.
[9] R.G. Hamlet, “Testing Programs with the Aid of a Compiler,” IEEE Trans. Software Eng., vol. 3, no. 4, pp. 279-290, July 1977.
[10] S. Rapps and E. Weyuker, “Selecting Software Test Data Using Data Flow Information,” IEEE Trans. Software Eng., vol. 11, no. 4, pp. 367-375, Apr. 1985.
[11] C. Fu, R.P. Martin, K. Nagaraja, T.D. Nguyen, B.G. Ryder, and D. Wonnacott, “Compiler-Directed Program-Fault Coverage for Highly Available Java Internet Services,” Proc. Int'l Conf. Dependable Systems and Networks (DSN 2003), June 2003.
[12] C. Fu, R.P. Martin, K. Nagaraja, T.D. Nguyen, B.G. Ryder, and D.G. Wonnacott, “Compiler-Directed Program-Fault Coverage for Highly Available Java Internet Services,” Technical Report DCS-TR-518, Dept. of Computer Science, Rutgers Univ., Jan. 2003.
[13] C. Fu, B.G. Ryder, A. Milanova, and D. Wonnacott, “Testing of Java Web Services For Robustness,” Proc. Int'l Symp. Software Testing and Analysis (ISSTA), pp. 23-33, July 2004.
[14] F. Tip and J. Palsberg, “Scalable Propagation-Based Call Graph Construction Algorithms,” Proc. Conf. Object-Oriented Programming, Languages, Systems and Applications, pp. 281-293, Oct. 2000.
[15] K. Arnold and J. Gosling, The Java Programming Language, second ed. Addison-Wesley, 1997.
[16] X. Li, R.P. Martin, K. Nagaraja, T.D. Nguyen, and B. Zhang, “Mendosus: A SAN-Based Fault-Injection Test-Bed for the Construction of Highly Available Network Services,” Proc. First Workshop Novel Uses of System Area Networks (SAN-1), Jan. 2002.
[17] A. Rountev, A. Milanova, and B.G. Ryder, “Points-To Analysis for Java Using Annotated Constraints,” Proc. Conf. Object-Oriented Programming, Languages, Systems and Applications, pp. 43-55, 2001.
[18] Sable, McGill, “Soot: A Java Optimization Framework,” http://www.sablemcgill.casoot, 2003.
[19] J. Dean, D. Grove, and C. Chambers, “Optimization of Object-Oriented Programs Using Static Class Hierarchy,” Proc. Ninth European Conf. Object-Oriented Programming (ECOOP '95), pp. 77-101, 1995.
[20] D. Bacon and P. Sweeney, “Fast Static Analysis of C++ Virtual Functions Calls,” Proc. ACM SIGPLAN Conf. Object-Oriented Programing Systems, Languages and Applications (OOPSLA '96), pp. 324-341, Oct. 1996.
[21] D. Grove and C. Chambers, “A Framework for Call Graph Construction Algorithms,” ACM Trans. Programming Languages and Systems (TOPLAS), vol. 23, no. 6, 2001.
[22] T. Ogasawara, H. Komatsu, and T. Nakatani, “A Study of Exception Handling and Its Dynamic Optimization in Java,” Proc. ACM SIGPLAN Conf. Object-oriented Programing Systems, Languages and Applications (OOPSLA '01), pp. 83-95, 2001.
[23] A.V. Aho, R. Sethi, and J.D. Ullman, Compilers— Principles, Techniques and Tools. Addison Wesley, 1988.
[24] T.J. Marlowe and B.G. Ryder, “Properties of Data Flow Frameworks: A Unified Model,” Acta Informatica, vol. 28, pp. 121-163, 1990.
[25] A. Milanova, A. Rountev, and B.G. Ryder, “Parameterized Object Sensitivity for Points-To Analysis for Java,” ACM Trans. Software Eng. and Methodology, vol. 14, no. 1, Jan. 2005.
[26] A. Milanova, “Precise and Practical Flow Analsis of Object-Oriented Software,” PhD dissertation, Rutgers Univ., 2003.
[27] M.L. Scott, Programming Language Pragmatics. Morgan Kaufmann, 2000.
[28] M. Sharir and A. Pnueli, “Two Approaches to Interprocedural Data Flow Analysis,” Program Flow Analysis: Theory and Applications, S. Muchnick and N. Jones, eds., Prentice Hall, pp. 189-234, 1981.
[29] O. Shivers, “Control-Flow Analysis of Higher-Order Languages,” PhD dissertation, Carnegie Mellon Univ., 1991.
[30] O. Lhoták and L. Hendren, “Scaling Java Points-To Analysis Using Spark,” Proc. Int'l Conf. Compiler Construction, pp. 153-169, 2003.
[31] B.G. Ryder, “Dimensions of Precision in Reference Analysis of Object-Oriented Programming Languages,” Proc. 12th Int'l Conf. Compiler Construction, pp. 126-137, Apr. 2003.
[32] P. Sortokin, “Ftp Server in Java,” http://peter.sorotokin.com/ftpdftpd.html , 2003.
[33] M.J. Radwin, “The Java Network File System,” http://www. radwin.org/michael/projectsjnfs /, 2003.
[34] “The Muffin World Wide Web Filtering System,” http:/muffin.doit.org/, 2003.
[35] M. Welsh, D.E. Culler, and E.A. Brewer, “SEDA: An Architecture for Well-Conditioned, Scalable Internet Services,” Proc. Symp. Operating Systems Principles, pp. 230-243, http://www.cs.harvard. edu/~mdw/projseda /, 2001.
[36] Apache Software Foundation, “Apache Jarkarta Project,” http:/jakarta.apache.org/, 2004.
[37] Specbench.org, “Spec jvm98 Benchmarks,” http://www.spec. orgjvm98/, 1998.
[38] Vola no LLC, “Volanomark,” http://www.volano.com benchmarks.html, 2004.
[39] D. Tang and R.K. Iyer, “Analysis and Modeling of Correlated Failures in Multicomputer Systems,” ACM Trans. Computer Systems, pp. 567-577, May 1992.
[40] K. Nagaraja, X. Li, B. Zhang, R. Bianchini, R.P. Martin, and T.D. Nguyen, “Using Fault Injection to Evaluate the Performability of Cluster-Based Services,” Proc. Fourth USENIX Symp. Internet Technologies and Systems (USITS 2003), Mar. 2003.
[41] Apache Software Foundation, “Apache Jakarta Tomcat,” http://jakarta.apache.orgtomcat/, 2004.
[42] W.G. Bouricius, W.C. Carter, and P. Schneider, “Reliability Modeling Techniques for Self Repairing Computer Systems,” Proc. 24th Nat'l Conf. the ACM, pp. 295-309, Mar. 1969.
[43] T. Tsai, M. Hsueh, H. Zhao, Z. Kalbarczyk, and R. Iyer, “Stress-Based and Path-Based Fault Injection,” IEEE Trans. Computers, vol. 48, no. 11, pp. 1183-1201, Nov. 1999.
[44] J. Bieman, D. Dreilinger, and L. Lin, “Using Fault Injection to Increase Software Test Coverage,” Proc. Seventh Int'l Symp. Software Reliability Eng. (ISSRE '96), pp. 166-174, 1996.
[45] D. Hamlet, “Foundations of Software Testing: Dependability Theory,” Proc. Second ACM SIGSOFT Symp. Foundations of Software Eng., pp. 128-139, 1994.
[46] D. Tang and H. Hecht, “An Approach to Measuring and Assessing Dependability for Critical Software Systems,” Proc. Eighth Int'l Symp. Software Reliability Eng., pp. 192-202, Nov. 1997.
[47] P. Frankl and E. Weyuker, “An Applicable Family of Data Flow Testing Criteria,” IEEE Trans. Software Eng., vol. 14, no. 10, pp. 1483-1498, Oct. 1988.
[48] S. Sinha and M.J. Harrold, “Criteria for Testing Exception-Handling Constructs in Java Programs,” Proc. Int'l Conf. Software Maintenance, 1999.
[49] M.P. Robillard and G.C. Murphy, “Static Analysis to Support the Evolution of Exception Structure in Object-Oriented Systems,” ACM Trans. Software Eng. and Methodology (TOSEM), vol. 12, no. 2, pp. 191-221, 2003.
[50] J.-W. Jo, B.-M. Chang, K. Yi, and K.-M. Cho, “An Uncaught Exception Analysis for Java,” J. Systems and Software, 2004.
[51] N. Heintze, “Set-Based Analysis of Ml Programs,” Proc. ACM Conf. Lisp and Functional Programmig, pp. 306-317, 1994.
[52] S. Sinha and M.J. Harrold, “Analysis and Testing of Programs With Exception-Handling Constructs,” IEEE Trans. Software Eng., vol. 26, no. 9, pp. 849-871, Sept. 2000.
[53] S. Sinha, A. Orso, and M.J. Harrold, “Automated Support for Development, Maintenance, and Testing in the Presence of Implicit Control Flow,” Proc. 26th Int'l Conf. Software Eng. (ICSE '04), 2004.
[54] J.-D. Choi, D. Grove, M. Hind, and V. Sarkar, “Efficient and Precise Modeling of Exceptions for Analysis of Java Programs,” Proc. ACM SIGPLAN-SIGSOFT Workshop Program Analysis for Software Tools and Eng., pp. 21-31, Sept. 1999.
[55] S. Lee, B.-S. Yang, S. Kim, S. Park, S.-M. Moon, K. Ebcioglu, and E. Altman, “Efficient Java Exception Handling in Just-in-Time Compilation,” Proc. ACM SIGPLAN Java Grande Conf., 2000.
[56] M. Cierniak, G.-Y. Lueh, and J.M. Stichnoth, “Practicing Judo: Java under Dynamic Optimzations,” Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, pp. 13-26, 2000.
[57] J.D.D. Grove, G. DeFouw, and C. Chambers, “Call Graph Construction in Object-Oriented Languages,” Proc. ACM SIGPLAN Conf. Object-Oriented Programing Systems, Languages and Applications (OOPSLA '97), pp. 108-124, Oct. 1997.
[58] R. O'Callahan, “The Generalized Aliasing as a Basis for Software Tools,” PhD dissertation, Carnegie Mellon Univ., 2000.
[59] R. Chatterjee, B.G. Ryder, and W.A. Landi, “Relevant Context Inference,” Proc. ACM SIGACT/SIGPLAN Symp. Principles of Programming Languages, Jan. 1999.
[60] R. Bodik, R. Gupta, and M.L. Soffa, “Refining Data Flow Information Using Infeasible Paths,” Proc. Sixth European Software Eng. Conf. (ESEC/FSE 97), M. Jazayeri and H. Schauer, eds., pp. 361-377, 1997.
[61] A.L. Souter and L.L. Pollock, “Type Infeasible Call Chains,” Proc. IEEE Int'l Workshop Source Code Analysis and Manipulation, 2001.
[62] A.L. Souter and L.L. Pollock, “Characterization and Automatic Identification of Type Infeasible Call Chains,” Information and Software Technology, vol. 44, no. 13, pp. 721-732, Oct. 2002.
[63] A. Rountev, S. Kagan, and M. Gibas, “Static and Dynamic Analysis of Call Chains in Java,” Proc. Int'l Symp. Software Testing and Analysis, pp. 1-11, July 2004.

Index Terms:
Index Terms- Reliability, def-use testing, Java, exceptions, test coverage metrics.
Citation:
Chen Fu, Ana Milanova, Barbara Gershon Ryder, David G. Wonnacott, "Robustness Testing of Java Server Applications," IEEE Transactions on Software Engineering, vol. 31, no. 4, pp. 292-311, April 2005, doi:10.1109/TSE.2005.51
Usage of this product signifies your acceptance of the Terms of Use.