This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Finding Bugs in Web Applications Using Dynamic Test Generation and Explicit-State Model Checking
July/August 2010 (vol. 36 no. 4)
pp. 474-494
Shay Artzi, Thomas J. Watson Research Center, Hawthorne
Adam Kieżun, Women's Hospital/Harvard Medical School, Boston
Julian Dolby, Thomas J. Watson Research Center, Hawthorne
Frank Tip, Thomas J. Watson Research Center, Hawthorne
Danny Dig, University of Illinois at Urbana-Champaign, Urbana
Amit Paradkar, Thomas J. Watson Research Center, Hawthorne
Michael D. Ernst, University of Washington, Seattle
Web script crashes and malformed dynamically generated webpages are common errors, and they seriously impact the usability of Web applications. Current tools for webpage validation cannot handle the dynamically generated pages that are ubiquitous on today's Internet. We present a dynamic test generation technique for the domain of dynamic Web applications. The technique utilizes both combined concrete and symbolic execution and explicit-state model checking. The technique generates tests automatically, runs the tests capturing logical constraints on inputs, and minimizes the conditions on the inputs to failing tests so that the resulting bug reports are small and useful in finding and fixing the underlying faults. Our tool Apollo implements the technique for the PHP programming language. Apollo generates test inputs for a Web application, monitors the application for crashes, and validates that the output conforms to the HTML specification. This paper presents Apollo's algorithms and implementation, and an experimental evaluation that revealed 673 faults in six PHP Web applications.

[1] S. Anand, P. Godefroid, and N. Tillmann, "Demand-Driven Compositional Symbolic Execution," Proc. Int'l Conf. Tools and Algorithms for the Construction and Analysis of Systems, pp. 367-381, 2008.
[2] S. Artzi, A. Kiezun, J. Dolby, F. Tip, D. Dig, A. Paradkar, and M.D. Ernst, "Finding Bugs in Dynamic Web Applications," Proc. Int'l Symp. Software Testing and Analysis, pp. 261-272, 2008.
[3] M. Benedikt, J. Freire, and P. Godefroid, "VeriWeb: Automatically Testing Dynamic Web Sites," Proc. Int'l Conf. World Wide Web, 2002.
[4] D. Brumley, J. Caballero, Z. Liang, J. Newsome, and D. Song, "Towards Automatic Discovery of Deviations in Binary Implementations with Applications to Error Detection and Fingerprint Generation," Proc. 16th USENIX Security Symp., 2007.
[5] C. Cadar, D. Dunbar, and D.R. Engler, "Klee: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs," Proc. USENIX Symp. Operating Systems Design and Implementation, pp. 209-224, 2008.
[6] C. Cadar and D.R. Engler, "Execution Generated Test Cases: How to Make Systems Code Crash Itself," Proc. Int'l SPIN Workshop Model Checking of Software, pp. 2-23, 2005.
[7] C. Cadar, V. Ganesh, P.M. Pawlowski, D.L. Dill, and D.R. Engler, "EXE: Automatically Generating Inputs of Death," Proc. Conf. Computer and Comm. Security, pp. 322-335, 2006.
[8] J. Clause and A. Orso, "Penumbra: Automatically Identifying Failure-Relevant Inputs Using Dynamic Tainting," Proc. Int'l Symp. Software Testing and Analysis, 2009.
[9] H. Cleve and A. Zeller, "Locating Causes of Program Failures," Proc. Int'l Conf. Software Eng., pp. 342-351, 2005.
[10] H. Cleve and A. Zeller, "Locating Causes of Program Failures" Proc. Int'l Conf. Software Eng., pp. 342-351, May 2005.
[11] C. Csallner, N. Tillmann, and Y. Smaragdakis, "DySy: Dynamic Symbolic Execution for Invariant Inference," Proc. Int'l Conf. Software Eng., pp. 281-290, 2008.
[12] D. Dean and D. Wagner, "Intrusion Detection via Static Analysis," Proc. Symp. Research in Security and Privacy, pp. 156-169, May 2001.
[13] C. Demartini, R. Iosif, and R. Sisto, "A Deadlock Detection Tool for Concurrent Java Programs," Software—Practice and Experience, vol. 29, no. 7, pp. 577-603, June 1999.
[14] M. Emmi, R. Majumdar, and K. Sen, "Dynamic Test Input Generation for Database Applications," Proc. Int'l Symp. Software Testing and Analysis, pp. 151-162, 2007.
[15] P. Godefroid, "Compositional Dynamic Test Generation," Proc. Ann. Symp. Principles of Programming Languages, pp. 47-54, 2007.
[16] P. Godefroid, A. Kieżun, and M.Y. Levin, "Grammar-Based Whitebox Fuzzing," Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, pp. 206-215, 2008.
[17] P. Godefroid, N. Klarlund, and K. Sen, "DART: Directed Automated Random Testing," Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, pp. 213-223, 2005.
[18] P. Godefroid, M.Y. Levin, and D. Molnar, "Automated Whitebox Fuzz Testing," Proc. Network Distributed Security Symp., pp. 151-166, 2008.
[19] W.G. Halfond, S. Anand, and A. Orso, "Precise Interface Identification to Improve Testing and Analysis of Web Applications," Proc. Int'l Symp. Software Testing and Analysis, 2009.
[20] W.G.J. Halfond and A. Orso, "Improving Test Case Generation for Web Applications Using Automated Interface Discovery," Proc. Joint Meeting European Software Eng. Conf. and ACM SIGSOFT Symp. Foundations of Software Eng., pp. 145-154, 2007.
[21] K. Havelund and T. Pressburger, "Model Checking Java Programs Using Java PathFinder," Int'l J. Software Tools for Technology Transfer, vol. 2, no. 4, pp. 366-381, 2000.
[22] G.J. Holzmann, "The Model Checker SPIN," Software Eng., vol. 23, no. 5, pp. 279-295, 1997.
[23] Y.-W. Huang, S.-K. Huang, T.-P. Lin, and C.-H. Tsai, "Web Application Security Assessment by Fault Injection and Behavior Monitoring," Proc. 12th Int'l Conf. World Wide Web, pp. 148-159, 2003.
[24] Y.-W. Huang, F. Yu, C. Hang, C.-H. Tsai, D.T. Lee, and S.-Y. Ku, "Verifying Web Applications Using Bounded Model Checking," Proc. Int'l Conf. Dependable Systems and Networks, pp. 199-208, 2004.
[25] K. Inkumsah and T. Xie, "Evacon: A Framework for Integrating Evolutionary and Concolic Testing for Object-Oriented Programs," Proc. IEEE/ACM Int'l Conf. Automated Software Eng., 2007.
[26] M. Johns and C. Beyerlein, "SMask: Preventing Injection Attacks in Web Applications by Approximating Automatic Data/Code Separation," Proc. ACM Symp. Applied Computing, 2007.
[27] N. Jovanovic, C. Kruegel, and E. Kirda, "Pixy: A Static Analysis Tool for Detecting Web Application Vulnerabilities (Short Paper)," Proc. IEEE Symp. Security and Privacy, pp. 258-263, 2006.
[28] A. Kieżun, V. Ganesh, P.J. Guo, P. Hooimeijer, and M.D. Ernst, "HAMPI: A Solver for String Constraints," Proc. Int'l Symp. Software Testing and Analysis, 2009.
[29] A. Kieżun, P. Guo, K. Jayaraman, and M. Ernst, "Automatic Creation of SQL Injection and Cross-Site Scripting Attacks," Proc. Int'l Conf. Software Eng., pp. 199-209, 2009.
[30] B. Livshits, M. Martin, and M.S. Lam, "SecuriFly: Runtime Protection and Recovery from Web Application Vulnerabilities," technical report, Stanford Univ., 2006.
[31] R. Majumdar and K. Sen, "Hybrid Concolic Testing," Proc. Int'l Conf. Software Eng., pp. 416-426, 2007.
[32] R. Majumdar and R.-G. Xu, "Directed Test Generation Using Symbolic Grammars," Proc. IEEE/ACM Int'l Conf. Automated Software Eng., pp. 134-143, 2007.
[33] S. McAllister, E. Kirda, and C. Kruegel, "Leveraging User Interactions for In-Depth Testing of Web Applications," Proc. 11th Int'l Symp. Recent Advances in Intrusion Detection, pp. 191-210, 2008.
[34] Y. Minamide, "Static Approximation of Dynamically Generated Web Pages," Proc. Int'l Conf. World Wide Web 2005.
[35] G. Misherghi and Z. Su, "HDD: Hierarchical Delta Debugging," Proc. Int'l Conf. Software Eng., pp. 142-151, 2006.
[36] A. Nguyen-Tuong, S. Guarnieri, D. Greene, J. Shirley, and D. Evans, "Automatically Hardening Web Applications Using Precise Tainting," Proc. Int'l Conf. Information Security, 2005.
[37] R. O'Callahan, personal communication, 2008.
[38] T. Pietraszek and C.V. Berghe, "Defending against Injection Attacks through Context-Sensitive String Evaluation," Proc. Recent Advances in Intrusion Detection, pp. 124-145, 2005.
[39] K. Sen, D. Marinov, and G. Agha, "CUTE: A Concolic Unit Testing Engine for C," Proc. ACM SIGSOFT Int'l Symp. Foundations of Software Eng., pp. 263-272, 2005.
[40] S. Sinha, H. Shah, C. Görg, S. Jiang, and M. Kim, "Fault Localization and Repair for Java Runtime Exceptions," Proc. Int'l Symp. Software Testing and Analysis, 2009.
[41] Z. Su and G. Wassermann, "The Essence of Command Injection Attacks in Web Applications," Proc. Ann. Symp. Principles of Programming Languages, pp. 372-382, 2006.
[42] W. Visser, C.S. Păsăreanu, and R. Pelánek, "Test Input Generation for Java Containers Using State Matching," Proc. Int'l Symp. Software Testing and Analysis, pp. 37-48, 2006.
[43] G. Wassermann and Z. Su, "Sound and Precise Analysis of Web Applications for Injection Vulnerabilities," Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, pp. 32-41, 2007.
[44] G. Wassermann and Z. Su, "Static Detection of Cross-Site Scripting Vulnerabilities," Proc. Int'l Conf. Software Eng., pp. 171-180, 2008.
[45] G. Wassermann, D. Yu, A. Chander, D. Dhurjati, H. Inamura, and Z. Su, "Dynamic Test Input Generation for Web Applications," Proc. ACM/SIGSOFT Int'l Symp. Software Testing and Analysis, pp. 249-260, 2008.
[46] Y. Xie and A. Aiken, "Static Detection of Security Vulnerabilities in Scripting Languages," Proc. Conf. USENIX Security Symp., pp. 179-192, 2006.
[47] A. Zeller, "Yesterday, My Program Worked. Today, It Does Not. Why?" Proc. ACM SIGSOFT Int'l Symp. Foundations of Software Eng., pp. 253-267, 1999.
[48] F. Zoufaly,"Web Standards and Search Engine Optimization (SEO)—Does Google Care About the Quality of Your Markup?" 2008.

Index Terms:
Software testing, Web applications, dynamic analysis, PHP, reliability, verification.
Citation:
Shay Artzi, Adam Kieżun, Julian Dolby, Frank Tip, Danny Dig, Amit Paradkar, Michael D. Ernst, "Finding Bugs in Web Applications Using Dynamic Test Generation and Explicit-State Model Checking," IEEE Transactions on Software Engineering, vol. 36, no. 4, pp. 474-494, July-Aug. 2010, doi:10.1109/TSE.2010.31
Usage of this product signifies your acceptance of the Terms of Use.