The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.04 - July/August (2011 vol.37)
pp: 486-508
Leonardo Mariani , University of Milano Bicocca, Milan
Fabrizio Pastore , University of Milano Bicocca, Milan
Mauro Pezzè , University of Milan Bicocca, Milano
ABSTRACT
Many software components are provided with incomplete specifications and little access to the source code. Reusing such gray-box components can result in integration faults that can be difficult to diagnose and locate. In this paper, we present Behavior Capture and Test (BCT), a technique that uses dynamic analysis to automatically identify the causes of failures and locate the related faults. BCT augments dynamic analysis techniques with model-based monitoring. In this way, BCT identifies a structured set of interactions and data values that are likely related to failures (failure causes), and indicates the components and the operations that are likely responsible for failures (fault locations). BCT advances scientific knowledge in several ways. It combines classic dynamic analysis with incremental finite state generation techniques to produce dynamic models that capture complementary aspects of component interactions. It uses an effective technique to filter false positives to reduce the effort of the analysis of the produced data. It defines a strategy to extract information about likely causes of failures by automatically ranking and relating the detected anomalies so that developers can focus their attention on the faults. The effectiveness of BCT depends on the quality of the dynamic models extracted from the program. BCT is particularly effective when the test cases sample the execution space well. In this paper, we present a set of case studies that illustrate the adequacy of BCT to analyze both regression testing failures and rare field failures. The results show that BCT automatically filters out most of the false alarms and provides useful information to understand the causes of failures in 69 percent of the case studies.
INDEX TERMS
Dynamic Analysis, diagnosis, fault localization, false positive filters, regression failure analysis, field failure analysis.
CITATION
Leonardo Mariani, Fabrizio Pastore, Mauro Pezzè, "Dynamic Analysis for Diagnosing Integration Faults", IEEE Transactions on Software Engineering, vol.37, no. 4, pp. 486-508, July/August 2011, doi:10.1109/TSE.2010.93
REFERENCES
[1] SAP "Crystal Report," http://www.sap.com/solutions/ sapbusinessobjects/ sme/reportingcrystalreports /, 2009.
[2] JasperForge "Jasper Report," http:/jasperforge.org/, 2009.
[3] Eclipse Community "Eclipse Plugin Central," http:/www. eclipseplugincentral.com, 2009.
[4] M. Morisio, C.B. Seaman, A.T. Parra, V.R. Basili, S.E. Kraft, and S.E. Condon, "Investigating and Improving a COTS-Based Software Development," Proc. 22nd Int'l Conf. Software Eng., pp. 32-41, 2000.
[5] N.G. Leveson, "The Role of Software in Spacecraft Accidents," AIAA J. Spacecraft and Rockets, vol. 41, no. 4, pp. 564-575, 2004.
[6] D. Hovemeyer and W. Pugh, "Finding Bugs Is Easy," Proc. 19th Conf. Object-Oriented Programming Systems, Languages, and Applications, pp. 92-106, 2004.
[7] N. Rutar, C.B. Almazan, and J.S. Foster, "A Comparison of Bug Finding Tools for Java," Proc. IEEE 15th Int'l Symp. Software Reliability Eng., pp. 245-256, 2004.
[8] M. Zitser, R. Lippmann, and T. Leek, "Testing Static Analysis Tools Using Exploitable Buffer Overflows from Open Source Code," Proc. 12th Int'l Symp. Foundations of Software Eng., pp. 97-106, 2004.
[9] S. Hissam and D. Carney, "Isolating Faults in Complex COTS-Based Systems," white paper, Carnegie Mellon Software Eng. Inst., 1998.
[10] A. Zeller, Why Programs Fail: A Guide to Systematic Debugging, Morgan Kaufman. 2005.
[11] M. Renieris and S. Reiss, "Fault Localization with Nearest Neighbor Queries," Proc. IEEE 18th Int'l Conf. Automated Software Eng., pp. 30-39, 2003.
[12] L. Briand, Y. Labiche, and X. Liu, "Using Machine Learning to Support Debugging with Tarantula," Proc. IEEE 18th Int'l Symp. Software Reliability Eng., pp. 137-146, 2007.
[13] W. Masri, A. Podgurski, and D. Leon, "An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows," IEEE Trans. Software Eng., vol. 33, no. 7, pp. 454-477, July 2007.
[14] J. Jones, M. Harrold, and J. Stasko, "Visualization of Test Information to Assist Fault Localization," Proc. 24th Int'l Conf. Software Eng., pp. 467-477, 2002.
[15] A. Zeller, "Isolating Cause-Effect Chains from Computer Programs," Proc. 10th Symp. Foundations of Software Eng., pp. 1-10, 2002.
[16] S. Hangal and M.S. Lam, "Tracking Down Software Bugs Using Automatic Anomaly Detection," Proc. 24th Int'l Conf. Software Eng., pp. 291-301, 2002.
[17] A. Wasylkowski, A. Zeller, and C. Lindig, "Detecting Object Usage Anomalies," Proc. Joint Meeting European Software Eng. Conf. and Symp. Foundations of Software Eng., 2007.
[18] O. Raz, P. Koopman, and M. Shaw, "Semantic Anomaly Detection in Online Data Sources," Proc. 24th Int'l Conf. Software Eng., pp. 302-312, 2002.
[19] L. Mariani and M. Pezzè, "Dynamic Detection of COTS Components Incompatibility," IEEE Software, vol. 24, no. 5, pp. 76-85, Sept./Oct. 2007.
[20] L. Mariani and M. Pezzè, "Behavior Capture and Test: Automated Analysis of Component Integration," Proc. IEEE 10th Int'l Conf. Eng. Complex Computer Systems, pp. 292-301, 2005.
[21] C. Liu and J. Han, "Failure Proximity: A Fault Localization-Based Approach," Proc. 14th Int'l Symp. Foundations of Software Eng., pp. 101-112, 2006.
[22] B. Liblit, M. Naik, A.X. Zheng, A. Aiken, and M.I. Jordan, "Scalable Statistical Bug Isolation," Proc. Conf. Programming Language Design and Implementation, pp. 15-26, 2005.
[23] Eclipse "Bug Reports for Eclipse Project," https://bugs.eclipse. orgbugs/, 2009.
[24] S. Eldh, S. Punnekkat, H. Hansson, and P. Jönsson, "Component Testing Is Not Enough—A Study of Software Faults in Telecom Middleware," Proc. 19th IFIP Int'l Conf. Testing of Communicating Systems, 2007.
[25] M.J.P. van der Meulen, P. Bishop, and R. Villa, "An Exploration of Software Faults and Failure Behaviour in a Large Population of Programs," Proc. IEEE 15th Int'l Symp. Software Reliability Eng., pp. 101-112, 2004.
[26] S. Chandra and P.M. Chen, "Whither Generic Recovery from Application Faults? A Fault Study Using Open-Source Software," Proc. IEEE Int'l Conf. Dependable Systems and Networks/Symp. Fault-Tolerant Computing, pp. 97-106, 2000.
[27] E. Weyuker, "Testing Component-Based Software: A Cautionary Tale," IEEE Software, vol. 15, no. 5, pp. 54-59, Sept./Oct. 1998.
[28] IEEE Standards Board "IEEE Standard Glossary of Software Engineering Terminology," Technical Report Std 610.12-1990, IEEE, 1990.
[29] Aspectwerkz, http:/aspectwerkz.codehaus.org/, 2009.
[30] IBM, "Eclipse Test & Performance Tools Platform," http://www. eclipse.orgtptp/, 2009.
[31] Sun, "Java Reflection," http://java.sun.com/javase/6/docs/ technotes/ guidesreflection/, 2009.
[32] J.H. Perkins and M.D. Ernst, "Efficient Incremental Algorithms for Dynamic Detection of Likely Invariants," Proc. 12th ACM SIGSOFT, pp. 23-32, 2004.
[33] J.A. Clause and A. Orso, "A Technique for Enabling and Supporting Debugging of Field Failures," Proc. IEEE 29th Int'l Conf. Software Eng., pp. 261-270, 2007.
[34] J. Oncina and P. Garcia, "Inferring Regular Languages in Polynomial Update Time," Pattern Recognition and Image Analysis, N. P. de la Blanca, A. Sanfeliu, and E.Vidal, eds., pp. 49-61, World Scientific. 1992.
[35] S. Porat and J. Feldman, "Learning Automata from Ordered Examples," J. Machine Learning, vol. 7, pp. 109-138, 1991.
[36] O. Cicchello and S.C. Kremer, "Inducing Grammars from Sparse Data Sets: A Survey of Algorithms and Results," J. Machine Learning Research, vol. 4, pp. 603-632, 2003.
[37] A. Biermann and J. Feldman, "On the Synthesis of Finite State Machines from Samples of Their Behavior," IEEE Trans. Computers, vol. 21, no. 6, pp. 592-597, June 1972.
[38] J. Cook and A. Wolf, "Discovering Models of Software Processes from Event-Based Data," ACM Trans. Software Eng. and Methodology, vol. 7, no. 3, pp. 215-249, 1998.
[39] S.P. Reiss and M. Renieris, "Encoding Program Executions," Proc. IEEE 23rd Int'l Conf. Software Eng., pp. 221-230, 2001.
[40] M.D. Ernst, J. Cockrell, W.G. Griswold, and D. Notkin, "Dynamically Discovering Likely Program Invariants to Support Program Evolution," IEEE Trans. Software Eng., vol. 27, no. 2, pp. 99-123, Feb. 2001.
[41] D. Lorenzoli, L. Mariani, and M. Pezzè, "Automatic Generation of Software Behavioral Models," Proc. 30th Int'l Conf. Software Eng., 2008.
[42] R. Parekh, C. Nichitiu, and V. Honavar, "A Polynomial Time Incremental Algorithm for Learning DFA," Proc. Fourth Int'l Colloquium Grammatical Inference, 1998.
[43] L. Mariani, F. Pastore, and M. Pezzè, "kBehavior: Algorithms and Complexity," LTA Technical Report lta-2010-01, Univ. of Milano Bicocca, http://www.lta.disco.unimib.it/ltateamPublications. php , 2010.
[44] G. Ammons, T. Ball, and J.R. Larus, "Exploiting Hardware Performance Counters with Flow and Context Sensitive Profiling," Proc. Conf. Programming Language Design and Implementation, pp. 85-96, 1997.
[45] A. Gordon, Classification, second ed. Chapman and Hall/CRC, 1999.
[46] W.-K. Chen, Graph Theory and Its Engineering Applications. World Scientific Publishing Company, 1997.
[47] M.D. Scheemaecker, "Nano XML," http:/nanoxml.cyberelf.be/, 2009.
[48] Eclipse Foundation, "Eclipse," http:/www.eclipse.org/, 2009.
[49] The Apache Software Foundation, "Tomcat," http:/tomcat. apache.org/, 2009.
[50] G. Kiczales, J. Lamping, A. Mendhekar, C. Maeda, C.V. Lopes, J.-M. Loingtier, and J. Irwin, "Aspect-Oriented Programming," Proc. 11th European Conf. Object Oriented Programming, pp. 220-242, 1997.
[51] Galileo Research Group, "Software-Artifact Infrastructure Repository (SIR)," http://esquared.unl.edusir, 2009.
[52] Eclipse Software Foundation, "Tomcat Bug 41939," https://issues.apache.org/bugzillashow_bug.cgi?id=41939 , 2009.
[53] Apache Software Foundation, "Tomcat Bug 40820," https://issues.apache.org/bugzillashow_bug.cgi?id=40820 , 2009.
[54] Eclipse, "Eclipse Bug 181288," https://bugs.eclipse.org/bugsshow_bug.cgi?id=181288 , 2009.
[55] Eclipse, "Eclipse Bug 221738," https://bugs.eclipse.org/bugsshow_bug.cgi?id=221738 , 2009.
[56] Eclipse, "Eclipse Bug 156532," https://bugs.eclipse.org/bugsshow_bug.cgi?id=156532 , 2009.
[57] T.J. Ostrand and M.J. Balcer, "The Category-Partition Method for Specifying and Generating Functional Tests," Comm. ACM, vol. 31, no. 6, pp. 676-686, June 1988.
[58] J. Jones and M.J. Harrold, "Empirical Evaluation of the Tarantula Automatic Fault-Localization Technique," Proc. Int'l Conf. Automated Software Eng., pp. 273-282, 2005.
[59] H. Cleve and A. Zeller, "Locating Causes of Program Failures," Proc. 27th Int'l Conf. Software Eng., pp. 342-351, 2005.
[60] J. Yang, D. Evans, D. Bhardwaj, T. Bhat, and M. Das, "Perracotta: Mining Temporal API Rules from Imperfect Traces," Proc. 28th Int'l Conf. Software Eng., pp. 282-291, 2006.
[61] C. Yilmaz, A. Paradkar, and C. Williams, "Time Will Tell: Fault Localization Using Time Spectra," Proc. 30th Int'l Conf. Software Eng., pp. 81-90, 2008.
[62] J. Henkel, C. Reichenbach, and A. Diwan, "Discovering Documentation for Java Container Classes," IEEE Trans. Software Eng., vol. 33, no. 8, pp. 526-543, Aug. 2007.
[63] C. Csallner, Y. Smaragdakis, and T. Xie, "DSD-Crasher: A Hybrid Analysis Tool for Bug Finding," ACM Trans. Software Eng. and Methodologies, vol. 17, no. 2, pp. 1-37, 2008.
[64] S. McCamant and M.D. Ernst, "Predicting Problems Caused by Component Upgrades," Proc. Ninth European Software Eng. Conf. Held Jointly with 11th Int'l Symp. Foundations of Software Eng., pp. 287-296, 2003.
[65] C. Csallner and Y. Smaragdakis, "Check 'n' Crash: Combining Static Checking and Testing," Proc. 27th Int'l Conf. Software Eng., pp. 422-431, 2005.
[66] T. Xie and D. Notkin, "Tool-Assisted Unit Test Selection Based on Operational Violations," Proc. IEEE 18th Int'l Conf. Automated Software Eng., pp. 40-48, 2003.
[67] M. Harder, J. Mellen, and M.D. Ernst, "Improving Test Suites via Operational Abstraction," Proc. IEEE 25th Int'l Conf. Software Eng., pp. 60-71, 2003.
[68] P. Godefroid, "Compositional Dynamic Test Generation," Proc. 34th Symp. Principles of Programming Languages, pp. 47-54, 2007.
[69] P. Godefroid, N. Klarlund, and K. Sen, "DART: Directed Automated Random Testing," Proc. Conf. Programming Language Design and Implementation, pp. 213-223, 2005.
[70] K. Sen, D. Marinov, and G. Agha, "CUTE: A Concolic unit Testing Engine for C," Proc. 13th Int'l Symp. Foundations of Software Eng., pp. 263-272, 2005.
[71] J. Perkins, S. Kim, S. Larsen, S. Amarasinghe, J. Bachrach, M. Carbin, C. Pacheco, F. Sherwood, S. Sidiroglou, G. Sullivan, W.-F. Wong, Y. Zibin, M. Ernst, and M. Rinard, "Automatically Patching Errors in Deployed Software," Proc. 22nd Symp. Operating Systems Principles, pp. 87-102, 2009.
40 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool