The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.02 - Feb. (2013 vol.39)
pp: 147-162
Gustavo Soares , Federal University of Campina Grande, Campina Grande
Rohit Gheyi , Federal University of Campina Grande, Campina Grande
Tiago Massoni , Federal University of Campina Grande, Campina Grande
ABSTRACT
Refactoring is a transformation that preserves the external behavior of a program and improves its internal quality. Usually, compilation errors and behavioral changes are avoided by preconditions determined for each refactoring transformation. However, to formally define these preconditions and transfer them to program checks is a rather complex task. In practice, refactoring engine developers commonly implement refactorings in an ad hoc manner since no guidelines are available for evaluating the correctness of refactoring implementations. As a result, even mainstream refactoring engines contain critical bugs. We present a technique to test Java refactoring engines. It automates test input generation by using a Java program generator that exhaustively generates programs for a given scope of Java declarations. The refactoring under test is applied to each generated program. The technique uses SafeRefactor, a tool for detecting behavioral changes, as an oracle to evaluate the correctness of these transformations. Finally, the technique classifies the failing transformations by the kind of behavioral change or compilation error introduced by them. We have evaluated this technique by testing 29 refactorings in Eclipse JDT, NetBeans, and the JastAdd Refactoring Tools. We analyzed 153,444 transformations, and identified 57 bugs related to compilation errors, and 63 bugs related to behavioral changes.
INDEX TERMS
Java, Metals, Engines, Computer bugs, Testing, Automatic programming, Unified modeling language, program generation, Refactoring, automated testing
CITATION
Gustavo Soares, Rohit Gheyi, Tiago Massoni, "Automated Behavioral Testing of Refactoring Engines", IEEE Transactions on Software Engineering, vol.39, no. 2, pp. 147-162, Feb. 2013, doi:10.1109/TSE.2012.19
REFERENCES
[1] W.F. Opdyke, "Refactoring Object-Oriented Frameworks," PhD dissertation, Univ. of Illinois at Urbana-Champaign, 1992.
[2] M. Fowler, Refactoring: Improving the Design of Existing Code. Addison-Wesley Longman Publishing Co., 1999.
[3] T. Mens and T. Tourwé, "A Survey of Software Refactoring," IEEE Trans. Software Eng., vol. 30, no. 2, pp. 126-139, Feb. 2004.
[4] Eclipse.org, "Eclipse Project," at http:/www.eclipse.org, 2011.
[5] Sun Microsystems, "NetBeans IDE," http:/www.netbeans.org/, 2011.
[6] Jet Brains, "IntelliJ Idea," http://www.intellij.comidea/, 2011.
[7] Embarcadero Tech nologies, "JBuilder," http://www.codegear. com/br/productsjbuilder , 2011.
[8] P. Borba, A. Sampaio, A. Cavalcanti, and M. Cornélio, "Algebraic Reasoning for Object-Oriented Programming," Science of Computer Programming, vol. 52, pp. 53-100, Aug. 2004.
[9] M. Cornélio, "Refactorings as Formal Refinements," PhD dissertation, Fed. Univ. of Pernambuco, 2004.
[10] F. Tip, A. Kieżun, and D. Bäumer, "Refactoring for Generalization Using Type Constraints," Proc. 18th ACM SIGPLAN Conf. Object-Oriented Programming, Systems, Languages, and Applications, pp. 13-26, 2003.
[11] L. Silva, A. Sampaio, and Z. Liu, "Laws of Object-Orientation with Reference Semantics," Proc. Sixth IEEE Int'l Conf. Software Eng. and Formal Methods, pp. 217-226, 2008.
[12] M. Schäfer, T. Ekman, and O. de Moor, "Sound and Extensible Renaming for Java," Proc. 23rd ACM SIGPLAN Conf. Object-Oriented Programming, Systems, Languages, and Applications, pp. 277-294, 2008.
[13] M. Schäfer, M. Verbaere, T. Ekman, and O. Moor, "Stepping Stones over the Refactoring Rubicon," Proc. 23rd European Conf. Object-Oriented Programming, pp. 369-393, 2009.
[14] M. Schäfer and O. de Moor, "Specifying and Implementing Refactorings," Proc. 25th ACM Int'l Conf. Object-Oriented Programming, Systems, Languages, and Applications, pp. 286-301, 2010.
[15] F. Steimann and A. Thies, "From Public to Private to Absent: Refactoring Java Programs under Constrained Accessibility," Proc. 23rd European Conf. Object-Oriented Programming, pp. 419-443, 2009.
[16] M. Schäfer, T. Ekman, and O. de Moor, "Challenge Proposal: Verification of Refactorings," Proc. Third Workshop Programming Languages Meets Program Verification, pp. 67-72, 2008.
[17] B. Daniel, D. Dig, K. Garcia, and D. Marinov, "Automated Testing of Refactoring Engines," Proc. Sixth Joint Meeting European Software Eng. Conf. and ACM SIGSOFT Symp. The Foundations of Software Eng., pp. 185-194, 2007.
[18] G. Soares, R. Gheyi, D. Serey, and T. Massoni, "Making Program Refactoring Safer," IEEE Software, vol. 27, no. 4, pp. 52-57, July/Aug. 2010.
[19] D. Jackson, Software Abstractions: Logic, Language, and Analysis. The MIT Press, 2006.
[20] D. Jackson, I. Schechter, and H. Shlyahter, "Alcoa: The Alloy Constraint Analyzer," Proc. 22nd Int'l Conf. Software Eng., pp. 730-733, 2000.
[21] J. Gosling, B. Joy, G.L. Steele, and G. Bracha, The Java Language Specification, third ed. Addison-Wesley Longman Publishing Co., 2005.
[22] C. Pacheco, S.K. Lahiri, M.D. Ernst, and T. Ball, "Feedback-Directed Random Test Generation," Proc. 29th Int'l Conf. Software Eng., pp. 75-84, 2007.
[23] V. Jagannath, Y.Y. Lee, B. Daniel, and D. Marinov, "Reducing the Costs of Bounded-Exhaustive Testing," Proc. 12th Int'l Conf. Fundamental Approaches to Software Eng.: Held as Part of the Joint European Conf. Theory and Practice of Software, pp. 171-185, 2009.
[24] Eclipse.org, "JDT Core Component," http://www.eclipse.org/jdtcore/, 2011.
[25] G.C. Murphy, M. Kersten, and L. Findlater, "How Are Java Software Developers Using the Eclipse IDE?" IEEE Software, vol. 23, no. 4, pp. 76-83, July/Aug. 2006.
[26] G. Soares, M. Mongiovi, and R. Gheyi, "Identifying Overly Strong Conditions in Refactoring Implementations," Proc. 27th IEEE Int'l Conf. Software Maintenance, pp. 173-182, Sept. 2011.
[27] W. Mckeeman, "Differential Testing for Software," Digital Technical J., vol. 10, no. 1, pp. 100-107, 1998.
[28] M. Schäfer, T. Ekman, R. Ettinger, and M. Verbaere, "Refactoring Bugs," http://code.google .com/p/jrrt/wikiRefactoringBugs , 2011.
[29] L. Baresi and M. Miraz, "Testful: Automatic Unit-Test Generation for Java Classes," Proc. 32nd Int'l Conf. Software Eng., vol. 2, pp. 281-284, 2010.
[30] J.B. Goodenough and S.L. Gerhart, "Toward a Theory of Test Data Selection," SIGPLAN Notes, vol. 10, pp. 493-510, Apr. 1975.
[31] H. Zhu, P.A.V. Hall, and J.H.R. May, "Software Unit Test Coverage and Adequacy," ACM Computing Surveys, vol. 29, pp. 366-427, Dec. 1997.
[32] M. Gligoric, T. Gvero, V. Jagannath, S. Khurshid, V. Kuncak, and D. Marinov, "Test Generation through Programming in UDITA," Proc. 32nd Int'l Conf. Software Eng., vol. 1, pp. 225-234, 2010.
[33] A. Celentano, S.C. Reghizzi, P.D. Vigna, C. Ghezzi, G. Granata, and F. Savoretti, "Compiler Testing Using a Sentence Generator," Software: Practice and Experience, vol. 10, no. 11, pp. 897-918, Nov. 1980.
[34] F. Bazzichi and I. Spadafora, "An Automatic Generator for Compiler Testing," IEEE Trans. Software Eng., vol. 8, no. 4, pp. 343-353, July 1982.
[35] D.M. Hoffman, D. Ly-Gagnon, P. Strooper, and H.-Y. Wang, "Grammar-Based Test Generation with YouGen," Software: Practice and Experience, vol. 41, pp. 427-447, Apr. 2011.
[36] R. Fuhrer, F. Tip, A. Kieżun, J. Dolby, and M. Keller, "Efficiently Refactoring Java Applications to Use Generic Libraries," Proc. 19th European Conf. Object-Oriented Programming, pp. 71-96, 2005.
[37] I. Balaban, F. Tip, and R. Fuhrer, "Refactoring Support for Class Library Migration," Proc. 20th ACM SIGPLAN Conf. Object-Oriented Programming, Systems, Languages, and Applications, pp. 265-279, 2005.
[38] L. Tokuda and D. Batory, "Evolving Object-Oriented Designs with Refactorings," Automated Software Eng., vol. 8, pp. 89-120, Jan. 2001.
[39] T. Mens, S. Demeyer, and D. Janssens, "Formalising Behaviour Preserving Program Transformations," Proc. First Int'l Conf. Graph Transformation, pp. 286-301, 2002.
[40] M. Schäfer, J. Dolby, M. Sridharan, E. Torlak, and F. Tip, "Correct Refactoring of Concurrent Java Code," Proc. 24th European Conf. Object-Oriented Programming, pp. 225-249, 2010.
[41] J.L. Overbey and R.E. Johnson, "Differential Precondition Checking: A Lightweight, Reusable Analysis for Refactoring Tools," Proc. 26th IEEE/ACM Int'l Conf. Automated Software Eng., pp. 303-312, 2011.
[42] D. Dig and R. Johnson, "The Role of Refactorings in API Evolution," Proc. 21st IEEE Int'l Conf. Software Maintenance, pp. 389-398, 2005.
[43] C. Reichenbach, D. Coughlin, and A. Diwan, "Program Metamorphosis," Proc. 23rd European Conf. Object-Oriented Programming, pp. 394-418, 2009.
[44] H. Li and S. Thompson, "Testing Erlang Refactorings with QuickCheck," Proc. 19th Int'l Symp. Implementation and Application of Functional Languages, pp. 19-36, 2008.
[45] B. Korel and A.M. Al-Yami, "Automated Regression Test Generation," Proc. Fourth Int'l Symp. Software Testing and Analysis, pp. 143-152, 1998.
[46] K.V. Hanford, "Automatic Generation of Test Cases," IBM Systems J., vol. 9, pp. 242-257, Dec. 1970.
[47] W. Jin, A. Orso, and T. Xie, "Automated Behavioral Regression Testing," Proc. 23rd Int'l Conf. Software Testing, Verification and Validation, pp. 137-146, 2010.
[48] D. Marinov and S. Khurshid, "TestEra: A Novel Framework for Automated Testing of Java Programs," Proc. IEEE 16th Int'l Conf. Automated Software Eng., pp. 22-34, 2001.
[49] D. Coppit, J. Yang, S. Khurshid, W. Le, and K. Sullivan, "Software Assurance by Bounded Exhaustive Testing," IEEE Trans. Software Eng., vol. 31, no. 4, pp. 328-339, Apr. 2005.
36 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool