Issue No.04 - July/August (2010 vol.36)
pp: 575-590
Welf Löwe , Växjö University, Växjö
Joakim Nivre , Växjö University, Växjö
Detection of design pattern occurrences is part of several solutions to software engineering problems, and high accuracy of detection is important to help solve the actual problems. The improvement in accuracy of design pattern occurrence detection requires some way of evaluating various approaches. Currently, there are several different methods used in the community to evaluate accuracy. We show that these differences may greatly influence the accuracy results, which makes it nearly impossible to compare the quality of different techniques. We propose a benchmark suite to improve the situation and a community effort to contribute to, and evolve, the benchmark suite. Also, we propose fine-grained metrics assessing the accuracy of various approaches in the benchmark suite. This allows comparing the detection techniques and helps improve the accuracy of detecting design pattern occurrences.
Patterns, object-oriented design methods, measurement techniques, evaluation, reverse engineering, reengineering, restructuring.
Welf Löwe, Joakim Nivre, "Evaluation of Accuracy in Design Pattern Occurrence Detection", IEEE Transactions on Software Engineering, vol.36, no. 4, pp. 575-590, July/August 2010, doi:10.1109/TSE.2009.92
[1] E. Gamma, R. Helm, R. Johnson, and J. Vlissides, Design Patterns, Elements of Reusable Object-Oriented Software. Addison-Wesley, 1995.
[2] M. Meyer, "Pattern-Based Reengineering of Software Systems," Proc. 13th Working Conf. Reverse Eng., pp. 305-306, Oct. 2006.
[3] M. Vokac, "An Efficient Tool for Recovering Design Patterns from C++ Code," J. Object Technology, vol. 5, no. 1, pp. 139-157, Jan./Feb. 2006.
[4] K. Driesen and U. Hölzle, "The Direct Cost of Virtual Function Calls in C++," Proc. Conf. Object Oriented Programming Systems Languages and Applications, pp. 306-323, Oct. 1996.
[5] U.P. Schultz, J.L. Lawall, and C. Consel, "Specialization Patterns," Proc. 15th Int'l Conf. Automated Software Eng., pp. 197-206, 2000.
[6] S. Buchholz and E. Marsi, "CoNLL-X Shared Task on Multilingual Dependency Parsing," Proc. 10th Conf. Computational Natural Language Learning, , pp. 149-164, June 2006.
[7] S.E. Sim, S. Easterbrook, and R.C. Holt, "Using Benchmarking to Advance Research: A Challenge to Software Engineering," Proc. 25th Int'l Conf. Software Eng., pp. 74-83, 2003.
[8] Y.-G. Guéhéneuc and H. Albin-Amiot, "Recovering Binary Class Relationships: Putting Icing on the UML Cake," Proc. Conf. Object Oriented Programming Systems Languages and Applications, J.M. Vlissides and D.C. Schmidt, eds., pp. 301-314, 2004.
[9] O. Kaczor, Y.-G. Guéhéneuc, and S. Hamel, "Efficient Identification of Design Patterns with Bit-Vector Algorithm," Proc. Conf. Software Maintenance and Reeng., Mar. 2006.
[10] Y.-G. Guéhéneuc, K. Mens, and R. Wuyts, "A Comparative Framework for Design Recovery Tools," Proc. 10th Conf. Software Maintenance and Reeng., pp. 123-134, Mar. 2006.
[11] Y.-G. Guéhéneuc and G. Antoniol, "Demima: A Multi-Layered Approach for Design Pattern Identification," IEEE Trans. Software Eng., vol. 34, no. 5, pp. 667-684, Sept./Oct. 2008.
[12] J. Dong, Y. Zhao, and T. Peng, "Architecture and Design Pattern Discovery Techniques—A Review," Proc. Int'l Workshop System/Software Architectures, June 2007.
[13] T. Peng, J. Dong, and Y. Zhao, "Verifying Behavioral Correctness of Design Pattern Implementation," Proc. Int'l Conf. Software Eng. and Knowledge Eng., pp. 454-459, 2008.
[14] J. Dong, Y. Sun, and Y. Zhao, "Design Pattern Detection by Template Matching," Proc. ACM Symp. Applied Computing, pp. 765-769, 2008.
[15] J. Niere, W. Schafer, J. Wadsack, L. Wendehals, and J. Welsh, "Towards Pattern-Based Design Recovery," Proc. 24th Int'l Conf. Software Eng., pp. 338-348, 2002.
[16] J. Niere, J.P. Wedsack, and L. Wendehals, "Handling Large Search Space in Pattern-Based Reverse Engineering," Proc. 11th IEEE Int'l Workshop Program Comprehension, May 2003.
[17] L. Wendehals, "Improving Design Pattern Instance Recognition by Dynamic Analysis," Proc. Int'l Conf. Software Eng. 2003 Workshop Dynamic Analysis, May 2003.
[18] L. Wendehals, M. Meyer, and A. Elsner, "Selective Tracing of Java Programs," Proc. Second Int'l Fujaba Days, vol. tr-ri-04-253, pp. 7-10, Sept. 2004.
[19] L. Wendehals and A. Orso, "Recognizing Behavioral Patterns at Runtime Using Finite Automata," Proc. Workshop Dynamic Analysis, May 2006.
[20] G. Costagliola, A.D. Lucia, V. Deufemia, C. Gravino, and M. Risi, "Design Pattern Recovery by Visual Language Parsing," Proc. Conf. Software Maintenance and Reeng., Mar. 2005.
[21] G. Costagliola, A.D. Lucia, V. Deufemia, C. Gravino, and M. Risi, "Case Studies of Visual Language Based Design Patterns Recovery," Proc. Conf. Software Maintenance and Reeng., Mar. 2006.
[22] G. Antoniol, R. Fiutem, and L. Christoforetti, "Design Pattern Recovery in Object-Oriented Software," Proc. Sixth Int'l Workshop Program Comprehension, pp. 153-160, June 1998.
[23] D. Heuzeroth, T. Holl, G. Högström, and W. Löwe, "Automatic Design Pattern Detection," Proc. 11th Int'l Workshop Program Comprehension, May 2003.
[24] N. Pettersson and W. Löwe, "A Non-Conservative Approach to Software Pattern Detection," Proc. 15th IEEE Int'l Conf. Program Comprehension, K. Wong, E. Stroulia, and P. Tonella, eds., pp. 189-197, June 2007.
[25] C. Krämer and L. Prechelt, "Design Recovery by Automated Search for Structural Design Patterns in Object-Oriented Software," Proc. Working Conf. Reverse Eng., pp. 208-215, Nov. 1996.
[26] D. Beyer, A. Noack, and C. Lewerentz, "Simple and Efficient Relational Querying of Software Structures," Proc. 10th IEEE Working Conf. Reverse Eng., pp. 216-225, Nov. 2003.
[27] K. Brown, "Design Reverse-Engineering and Automated Design Pattern Detection in Smalltalk," Technical Report TR-96-07, North Carolina State Univ., June 1996.
[28] R.K. Keller, R. Schauer, S. Robitaille, and P. Page, "Pattern-Based Reverse-Engineering of Design Components," Proc. Int'l Conf. Software Eng., pp. 226-235, May 1999.
[29] Z. Balanyi and R. Ferenc, "Mining Design Patterns from C++ Code," Proc. IEEE Int'l Conf. Software Maintenance, Sept. 2003.
[30] I. Philippow, D. Streitferdt, M. Riebisch, and S. Naumann, "An Approach for Reverse Engineering of Design Patterns," Software and Systems Modeling, vol. 4, no. 1, pp. 55-70, Feb. 2005.
[31] N. Pettersson, "Measuring Precision for Static and Dynamic Design Pattern Recognition as a Function of Coverage," Proc. Int'l Conf. Software Eng. Workshop Dynamic Analysis, May 2005.
[32] N. Shi and R.A. Olsson, "Reverse Engineering of Design Patterns from Java Source Code," Proc. Int'l Conf. Automated Software Eng., Sept. 2006.
[33] N. Tsantalis, A. Chatzigeorgiou, G. Stephanides, and S.T. Halkidis, "Design Pattern Detection Using Similarity Scoring," IEEE Trans. Software Eng., vol. 32, no. 11, pp. 896-909, Nov. 2006.
[34] J. Seemann and J.W. von Gudenberg, "Pattern-Based Design Recovery of Java Software," Proc. ACM SIGSOFT, pp. 10-16, Nov. 1998.
[35] J.M. Smith and D. Stotts, "SPQR: Flexible Automated Design Pattern Extraction from Source Code," Proc. IEEE Int'l Conf. Automated Software Eng., Oct. 2003.
[36] Treebanks: Building and Using Parsed Corpora, A. Abeillé, ed. Kluwer Academic Publishers, 2003.
[37] M.P. Marcus, B. Santorini, and M.A. Marcinkiewicz, "Building a Large Annotated Corpus of English: The Penn Treebank," Computational Linguistics, vol. 19, pp. 313-330, 1993.
[38] E. Black, S. Abney, D. Flickinger, C. Gdaniec, R. Grishman, P. Harrison, D. Hindle, R. Ingria, F. Jelinek, J. Klavans, M. Liberman, S. Roukos, B. Santorini, and T. Strzalkowski, "A Procedure for Quantitatively Comparing the Syntactic Coverage of English Grammars," Proc. DARPA Speech and Natural Language Workshop, pp. 306-311, 1991.
[39] R. Grishman, C. Macleod, and J. Sterling, "Evaluating Parsing Strategies Using Standardized Parse Files," Proc. Third ACL Conf. Applied Natural Language Processing, pp. 156-161, 1992.
[40] E.M. Vorhees, "The Philosophy of Information Retrieval Evaluation," Proc. 12th Text Retrieval Conf., pp. 355-370, 2002.
[41] P. Ahlgren and L. Grönqvist, "Measuring Retrieval Effectiveness with Incomplete Relevance Data," Proc. Conf. Current Research in Information Sciences and Technology, 2006.
[42] L. Fülöp, T. Gyovai, and R. Ferenc, "Evaluating C++ Design Pattern Miner Tools," Proc. Sixth IEEE Int'l Workshop Source Code Analysis and Manipulation, Sept. 2006.
[43] N. Pettersson, W. Löwe, and J. Nivre, "On Evaluation of Accuracy in Pattern Detection," Proc. First Int'l Workshop Design Pattern Detection for Reverse Eng., Oct. 2006.
[44] L.J. Fülöp, R. Ferenc, and T. Gyimothy, "Towards a Benchmark for Evaluating Design Pattern Miner Tools," Proc. 12th European Conf. Software Maintenance and Reeng., pp. 143-152, Apr. 2008.
[45] "pinot," pinot/, Jan. 2008.
[46] N. Pettersson, "Towards Pattern Detection in Software," licentiate thesis, School of Math. and Systems Eng., Växjö Univ., May 2006.
[47] N. Pettersson, "Design Pattern Detection Evaluation Suite (DPDES),", May 2009.