The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.03 - May/June (2011 vol.37)
pp: 341-355
Bas Cornelissen , Software Improvement Group, Amsterdam
Andy Zaidman , Delft University of Technology, Delft
Arie van Deursen , Delft University of Technology, Delft
ABSTRACT
Software maintenance activities require a sufficient level of understanding of the software at hand that unfortunately is not always readily available. Execution trace visualization is a common approach in gaining this understanding, and among our own efforts in this context is Extravis, a tool for the visualization of large traces. While many such tools have been evaluated through case studies, there have been no quantitative evaluations to the present day. This paper reports on the first controlled experiment to quantitatively measure the added value of trace visualization for program comprehension. We designed eight typical tasks aimed at gaining an understanding of a representative subject system, and measured how a control group (using the Eclipse IDE) and an experimental group (using both Eclipse and Extravis) performed these tasks in terms of time spent and solution correctness. The results are statistically significant in both regards, showing a 22 percent decrease in time requirements and a 43 percent increase in correctness for the group using trace visualization.
INDEX TERMS
Program comprehension, dynamic analysis, controlled experiment.
CITATION
Bas Cornelissen, Andy Zaidman, Arie van Deursen, "A Controlled Experiment for Program Comprehension through Trace Visualization", IEEE Transactions on Software Engineering, vol.37, no. 3, pp. 341-355, May/June 2011, doi:10.1109/TSE.2010.47
REFERENCES
[1] T.A. Corbi, "Program Understanding: Challenge for the 1990s," IBM Systems J., vol. 28, no. 2, pp. 294-306, 1989.
[2] V.R. Basili, "Evolving and Packaging Reading Technologies," J. Systems and Software, vol. 38, no. 1, pp. 3-12, 1997.
[3] S.P. Reiss and M. Renieris, "Encoding Program Executions," Proc. 23rd IEEE Int'l Conf. Software Eng., pp. 221-230, 2001.
[4] A. Hamou-Lhadj and T.C. Lethbridge, "Summarizing the Content of Large Traces to Facilitate the Understanding of the Behaviour of a Software System," Proc. 14th IEEE Int'l Conf. Program Comprehension, pp. 181-190, 2006.
[5] W. De Pauw, R. Helm, D. Kimelman, and J.M. Vlissides, "Visualizing the Behavior of Object-Oriented Systems," Proc. Eighth Conf. Object-Oriented Programming Systems, Languages, and Applications, pp. 326-337, 1993.
[6] D.F. Jerding, J.T. Stasko, and T. Ball, "Visualizing Interactions in Program Executions," Proc. 19th IEEE Int'l Conf. Software Eng., pp. 360-370, 1997.
[7] B. Cornelissen, A. Zaidman, D. Holten, L. Moonen, A. van Deursen, and J.J. van Wijk, "Execution Trace Analysis through Massive Sequence and Circular Bundle Views," J. Systems and Software, vol. 81, no. 11, pp. 2252-2268, 2008.
[8] B. Cornelissen, A. Zaidman, B. Van Rompaey, and A. van Deursen, "Trace Visualization for Program Comprehension: A Controlled Experiment," Proc. 17th IEEE Int'l Conf. Program Comprehension, pp. 100-109, 2009.
[9] B. Cornelissen, A. Zaidman, A. van Deursen, L. Moonen, and R. Koschke, "A Systematic Survey of Program Comprehension through Dynamic Analysis," IEEE Trans. Software Eng., vol. 35, no. 5, pp. 684-702, Sept./Oct. 2009.
[10] A. Zaidman and S. Demeyer, "Managing Trace Data Volume through a Heuristical Clustering Process Based on Event Execution Frequency," Proc. Eighth European Conf. Software Maintenance and Reeng., pp. 329-338, 2004.
[11] M.F. Kleyn and P.C. Gingrich, "Graphtrace—Understanding Object-Oriented Systems Using Concurrently Animated Views," Proc. Third Conf. Object-Oriented Programming Systems, Languages, and Applications, pp. 191-205, 1988.
[12] W. De Pauw, D. Lorenz, J. Vlissides, and M. Wegman, "Execution Patterns in Object-Oriented Visualization," Proc. Fourth USENIX Conf. Object-Oriented Technologies and Systems, pp. 219-234, 1998.
[13] W. De Pauw, E. Jensen, N. Mitchell, G. Sevitsky, J.M. Vlissides, and J. Yang, "Visualizing the Execution of Java Programs," Proc. ACM Symp. Software Visualization, pp. 151-162, 2001.
[14] W. De Pauw, S. Krasikov, and J.F. Morar, "Execution Patterns for Visualizing Web Services," Proc. ACM Symp. Software Visualization, pp. 37-45, 2006.
[15] K. Koskimies and H. Mössenböck, "Scene: Using Scenario Diagrams and Active Text for Illustrating Object-Oriented Programs," Proc. 18th IEEE Int'l Conf. Software Eng., pp. 366-375, 1996.
[16] D.F. Jerding and S. Rugaber, "Using Visualization for Architectural Localization and Extraction," Proc. Fourth Working Conf. Reverse Eng., pp. 56-65, 1997.
[17] T. Systä, "On the Relationships between Static and Dynamic Models in Reverse Engineering Java Software," Proc. Sixth Working Conf. Reverse Eng., pp. 304-313, 1999.
[18] T. Systä, K. Koskimies, and H.A. Müller, "Shimba: An Environment for Reverse Engineering Java Software Systems," Software: Practice and Experience, vol. 31, no. 4, pp. 371-394, 2001.
[19] T.S. Souder, S. Mancoridis, and M. Salah, "Form: A Framework for Creating Views of Program Executions," Proc. 17th IEEE Int'l Conf. Software Maintenance, pp. 612-620, 2001.
[20] R. Oechsle and T. Schmitt, "JAVAVIS: Automatic Program Visualization with Object and Sequence Diagrams Using the Java Debug Interface (JDI)," Proc. ACM Symp. Software Visualization, pp. 176-190, 2001.
[21] A. Hamou-Lhadj and T.C. Lethbridge, "Compression Techniques to Simplify the Analysis of Large Execution Traces," Proc. 10th IEEE Int'l Workshop Program Comprehension, pp. 159-168, 2002.
[22] A. Hamou-Lhadj, T.C. Lethbridge, and L. Fu, "Challenges and Requirements for an Effective Trace Exploration Tool," Proc. 12th IEEE Int'l Workshop Program Comprehension, pp. 70-78, 2004.
[23] A. Hamou-Lhadj, E. Braun, D. Amyot, and T.C. Lethbridge, "Recovering Behavioral Design Models from Execution Traces," Proc. Ninth European Conf. Software Maintenance and Reeng., pp. 112-121, 2005.
[24] M. Salah and S. Mancoridis, "Toward an Environment for Comprehending Distributed Systems," Proc. 10th Working Conf. Reverse Eng., pp. 238-247, 2003.
[25] M. Salah and S. Mancoridis, "A Hierarchy of Dynamic Software Views: From Object-Interactions to Feature-Interactions," Proc. 20th IEEE Int'l Conf. Software Maintenance, pp. 72-81, 2004.
[26] M. Salah, T. Denton, S. Mancoridis, A. Shokoufandeh, and F.I. Vokolos, "Scenariographer: A Tool for Reverse Engineering Class Usage Scenarios from Method Invocation Sequences," Proc. 21st IEEE Int'l Conf. Software Maintenance, pp. 155-164, 2005.
[27] M. Salah, S. Mancoridis, G. Antoniol, and M. DiPenta, "Scenario-Driven Dynamic Analysis for Comprehending Large Software Systems," Proc. 10th European Conf. Software Maintenance and Reeng., pp. 71-80, 2006.
[28] L.C. Briand, Y. Labiche, and Y. Miao, "Towards the Reverse Engineering of UML Sequence Diagrams," Proc. 10th Working Conf. Reverse Eng., pp. 57-66, 2003.
[29] L.C. Briand, Y. Labiche, and J. Leduc, "Tracing Distributed Systems Executions Using AspectJ," Proc. 21st IEEE Int'l Conf. Software Maintenance, pp. 81-90, 2005.
[30] L.C. Briand, Y. Labiche, and J. Leduc, "Toward the Reverse Engineering of UML Sequence Diagrams for Distributed Java Software," IEEE Trans. Software Eng., vol. 32, no. 9, pp. 642-663, Sept. 2006.
[31] A. Kuhn and O. Greevy, "Exploiting the Analogy between Traces and Signal Processing," Proc. 22nd IEEE Int'l Conf. Software Maintenance, pp. 320-329, 2006.
[32] O. Greevy, M. Lanza, and C. Wysseier, "Visualizing Live Software Systems in 3D," Proc. ACM Symp. Software Visualization, pp. 47-56, 2006.
[33] J. Koskinen, M. Kettunen, and T. Systä, "Profile-Based Approach to Support Comprehension of Software Behavior," Proc. 14th IEEE Int'l Conf. Program Comprehension, pp. 212-224, 2006.
[34] S. Simmons, D. Edwards, N. Wilde, J. Homan, and M. Groble, "Industrial Tools for the Feature Location Problem: An Exploratory Study," J. Software Maintenance and Evolution: Research and Practice, vol. 18, no. 6, pp. 457-474, 2006.
[35] B. Cornelissen, A. van Deursen, L. Moonen, and A. Zaidman, "Visualizing Testsuites to Aid in Software Understanding," Proc. 11th European Conf. Software Maintenance and Reeng., pp. 213-222, 2007.
[36] R. Voets, "JRET: A Tool for the Reconstruction of Sequence Diagrams from Program Executions," master's thesis, Delft Univ. of Tech nology, 2008.
[37] S.P. Reiss, "Visual Representations of Executing Programs," J. Visual Languages and Computing, vol. 18, no. 2, pp. 126-148, 2007.
[38] J. Jiang, J. Koskinen, A. Ruokonen, and T. Systä, "Constructing Usage Scenarios for API Redocumentation," Proc. 15th IEEE Int'l Conf. Program Comprehension, pp. 259-264, 2007.
[39] B. Cornelissen, D. Holten, A. Zaidman, L. Moonen, J.J. van Wijk, and A. van Deursen, "Understanding Execution Traces Using Massive Sequence and Circular Bundle Views," Proc. 15th IEEE Int'l Conf. Program Comprehension, pp. 49-58, 2007.
[40] C. Bennett, D. Myers, D. Ouellet, M.-A. Storey, M. Salois, D. German, and P. Charland, "A Survey and Evaluation of Tool Features for Understanding Reverse Engineered Sequence Diagrams," J. Software Maintenance and Evolution: Research and Practice, vol. 20, no. 4, pp. 291-315, 2008.
[41] A.R. Dalton and J.O. Hallstrom, "A Toolkit for Visualizing the Runtime Behavior of TinyOS Applications," Proc. 15th IEEE Int'l Conf. Program Comprehension, pp. 43-52, 2008.
[42] M.J. Pacione, M. Roper, and M. Wood, "Comparative Evaluation of Dynamic Visualisation Tools," Proc. 10th Working Conf. Reverse Eng., pp. 80-89, 2003.
[43] A. Hamou-Lhadj and T.C. Lethbridge, "A Survey of Trace Exploration Tools and Techniques," Proc. Conf. Centre for Advanced Studies on Collaborative Research, pp. 42-55, 2004.
[44] N. Wilde and M.C. Scully, "Software Reconnaissance: Mapping Program Features to Code," J. Software Maintenance: Research and Practice, vol. 7, no. 1, pp. 49-62, 1995.
[45] D.F. Jerding and J.T. Stasko, "The Information Mural: A Technique for Displaying and Navigating Large Information Spaces," IEEE Trans. Visualization and Computer Graphics, vol. 4, no. 3, pp. 257-271, July-Sept. 1998.
[46] M.-A. Storey, "Theories, Methods and Tools in Program Comprehension: Past, Present and Future," Proc. 13th IEEE Int'l Workshop Program Comprehension, pp. 181-191, 2005.
[47] C.F.J. Lange and M.R.V. Chaudron, "Interactive Views to Improve the Comprehension of UML Models—An Experimental Validation," Proc. 15th IEEE Int'l Conf. Program Comprehension, pp. 221-230, 2007.
[48] J. Quante, "Do Dynamic Object Process Graphs Support Program Understanding?—A Controlled Experiment," Proc. 16th IEEE Int'l Conf. Program Comprehension, pp. 73-82, 2008.
[49] A. Zaidman, B. Van Rompaey, S. Demeyer, and A. van Deursen, "Mining Software Repositories to Study Co-Evolution of Production & Test Code," Proc. First Int'l Conf. Software Testing, pp. 220-229, 2008.
[50] B. Van Rompaey and S. Demeyer, "Estimation of Test Code Changes Using Historical Release Data," Proc. 15th Working Conf. Reverse Eng., pp. 269-278, 2008.
[51] M.J. Pacione, M. Roper, and M. Wood, "A Novel Software Visualisation Model to Support Software Comprehension," Proc. 11th Working Conf. Reverse Eng., pp. 70-79, 2004.
[52] B. Cornelissen, A. Zaidman, B. Van Rompaey, and A. van Deursen, "Trace Visualization for Program Comprehension: A Controlled Experiment," Technical Report TUD-SERG-2009-001, Delft Univ. of Tech nology, 2009.
[53] M. Di Penta, R.E.K. Stirewalt, and E. Kraemer, "Designing Your Next Empirical Study on Program Comprehension," Proc. 15th IEEE Int'l Conf. Program Comprehension, pp. 281-285, 2007.
[54] C. Wohlin, P. Runeson, M. Höst, M.C. Ohlesson, B. Regnell, and A. Wesslen, Experimentation in Software Engineering—An Introduction. Kluwer Academic Publishers, 2003.
[55] R.K. Yin, Case Study Research: Design and Methods. Sage Publications Inc., 2003.
[56] C. Riva and J.V. Rodriguez, "Combining Static and Dynamic Views for Architecture Reconstruction," Proc. Sixth European Conf. Software Maintenance and Reeng., pp. 47-55, 2002.
13 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool