| | This Article | |
| |
| |
| | Share | |
| |
| |
| | Bibliographic References | |
| |
| |
| | Add to: | |
| |
 Digg
 Furl
 Spurl
 Blink
 Simpy
 Google
 Del.icio.us
 Y!MyWeb
| |
| | Search | |
| |
| |
| | |
On The Detection of Test Smells: A Metrics-Based Approach for General Fixture and Eager Test
December 2007 (vol. 33 no. 12)
pp. 800-817
As a testing method, white box testing has been demonstrated to be very efficient in early defect detection. However, white box testing introduces test co-evolution as an additional burden to software development. To mitigate the effects of co-evolution, tests should be written in a manner that makes them easy to change. Fortunately, we are able to concretely express what a good test is by exploiting the specific principles underlying white box testing. Analogous to the concept of code smells, violations of these principles are termed test smells. In this paper, we present a formal description of test smells, and propose metrics to support their detection. We validate the feasibility of detecting two test smells, General Fixture and Eager Test, by comparison with human evaluation. We demonstrate the effectiveness of the detection in the case the assessment is agreed upon by evaluators. For the General Fixture, a qualitative investigation showed that an ambiguous test smell definition prohibits the detection by metrics and suggests disentangling its definition. On the bright side, test evolvability can be more concretely expressed than general evolvability due to the exploitation of the specific principles underlying white box testing. In particular, adherence to a rigid setup-stimulate-verify-teardown cycle has been reported an essential characteristic of evolvable tests. In this article, we propose to incorporate structural characteristics of tests in the definition of test smells, thereby providing an objective means to detect test evolution obstacles. We validate the feasibility of detecting test evolution obstacles using such test smells, thereby contributing the first step to the mitigation of the cost of test co-evolution.
[1] K. Beck, Extreme Programming Explained: Embrace Change. Addison-Wesley, 1999.
[2] K. Beck, Test-Driven Development by Example. Addison-Wesley, 2002.
[3] S. Palmer and J. Felsing, A Practical Guide to Feature-Driven Development. Prentice Hall, Feb. 2002.
[4] P. Runeson, “A Survey of Unit Testing Practices,” IEEE Software, vol. 25, no. 4, pp. 22-29, July/Aug. 2006.
[5] A. van Deursen, L. Moonen, A. van den Bergh, and G. Kok, “Refactoring Test Code,” Proc. Second Int'l Conf. Extreme Programming and Flexible Processes in Software Eng., M. Marchesi and G.Succi, eds., 2001.
[6] R. Sangwan and P. Laplante, “Test-Driven Development in Large Projects,” IT Professional, vol. 8, no. 5, pp. 25-29, Sept./Oct. 2006.
[7] M. Harrold, “Testing: A Roadmap,” Proc. Conf. 22nd Int'l Conf. Software Eng., Future of Software Eng. Track, pp. 61-72, 2000.
[8] G. Murphy, P. Townsend, and P. Wong, “Experiences with Cluster and Class Testing,” Comm. ACM, vol. 37, no. 9, pp. 39-47, Sept. 1994.
[9] E. Maximilien and L. Williams, “Assessing Test-Driven Development at IBM,” Proc. 25th Int'l Conf. Software Eng. (ICSE '03), pp.564-569, 2003.
[10] L. Crispin, “Driving Software Quality: How Test-Driven Development Impacts Software Quality,” IEEE Software, vol. 23, no. 6, pp. 70-71, Nov./Dec. 2006.
[11] F. Belli and R. Crisan, “Empirical Performance Analysis of Computer-Supported Code-Reviews,” Proc. Eighth Int'l Symp. Software Reliability Eng., pp. 245-255, 1997.
[12] W. Li and S. Henry, “Object-Oriented Metrics that Predict Maintainability,” J. Systems and Software, vol. 23, no. 2, pp. 111-122, Feb. 1993.
[13] E. Dustin, Effective Software Testing. 50 Specific Way to Improve Your Testing. Addison-Wesley, 2003.
[14] M. Pelgrim, Dive into Python. Apress, 2004.
[15] A. Hunt and D. Thomas, Pragmatic Unit Testing in C# with NUnit. The Pragmatic Programmers, 2004.
[16] R. Patton, “Automated Testing and Test Tools,” Software Testing, chapter 15, Sams, 2005.
[17] M. Feathers, Working Effectively with Legacy Code. Prentice Hall, 2005.
[18] J. Langr, “Assertions and Annotations,” Agile Java Crafting Code with Test-Driven Development. Prentice Hall, 2005.
[19] T. Mackinnon, S. Freeman, and P. Craig, “Endo-Testing: Unit Testing with Mock Objects,” Proc. First Int'l Conf. Extreme Programming and Flexible Processes in Software Eng., 2000.
[20] S. Demeyer, S. Ducasse, and O. Nierstrasz, Object-Oriented Reengineering Patterns. Morgan Kaufmann, 2002.
[21] G. Meszaros, S. Smith, and J. Andrea, “The Test Automation Manifesto,” Proc. Third XP and Second Agile Universe Conf., pp. 73-81, Aug. 2003.
[22] M. Fewster and D. Graham, “Building Maintainable Tests,” Software Test Automation, ACM Press, 1999.
[23] D. Parnas, “On the Criteria to be Used in Decomposing Systems into Modules,” Comm. ACM, vol. 15, no. 12, pp. 1053-1058, Dec. 1972.
[24] M. Gaelli, M. Lanza, and O. Nierstrasz, “Towards a Taxonomy of SUnit Tests,” Proc. 13th Int'l Smalltalk Conf., 2005.
[25] A. Riel, Object-Oriented Design Heuristics. Addison-Wesley, 1996.
[26] V. Basili, L. Briand, and W. Melo, “A Validation of Object-Oriented Design Metrics as Quality Indicators,” IEEE Trans. Software Eng., vol. 22, no. 10, pp. 751-761, Oct. 1996.
[27] G. Meszaros, xUnit Test Patterns: Refactoring Test Code. Addison-Wesley, 2007.
[28] R. Johnson and B. Foote, “Designing Reusable Classes,” J. Object-Oriented Programming, vol. 2, no. 1, pp. 22-35, 1988.
[29] J. Lakos, Large-Scale C++ Software Design. Addison-Wesley, 1996.
[30] R. Marinescu, “Detection Strategies: Metrics-Based Rules for Detecting Design Flaws,” Proc. 20th IEEE Int'l Conf. Software Maintenance, 2004.
[31] M. Salehie, S. Li, and L. Tahvildari, “A Metric-Based Heuristic Framework to Detect Object-Oriented Design Flaws,” Proc. IEEE Int'l Conf. Program Comprehension, pp. 159-168, June 2006.
[32] T. Systä, P. Yu, and H. Müller, “Analyzing Java Software by Combining Metrics and Program Visualization,” Proc. Conf. Software Maintenance and Reeng., p. 199, 2000.
[33] F. Fioravanti and P. Nesi, “A Study on Fault-Proneness Detection of Object-Oriented Systems,” Proc. Fifth European Conf. Software Maintenance and Reengineering, pp. 121-130, 2001.
[34] M. Mäntylä, J. Vanhanen, and C. Lassenius, “A Taxonomy and an Initial Empirical Study of Bad Smells in Code,” Proc. 19th IEEE Int'l Conf. Software Maintenance, pp. 381-384, 2003.
[35] M. Mäntylä, J. Vanhanen, and C. Lassenius, “Bad Smells— Humans as Code Critics,” Proc. 20th IEEE Int'l Conf. Software Maintenance, pp. 399-408, 2004.
[36] L. Briand, J. Daly, and J. Wüst, “A Unified Framework for Coupling Measurement in Object-Oriented Systems,” IEEE Trans. Software Eng., vol. 25, no. 1, pp. 91-121, Jan./Feb. 1999.
[37] K. Beck, “Simple Smalltalk Testing: With Patterns,” The Smalltalk Report, vol. 4, no. 2, pp. 16-18, 1994.
[38] P. Hamill, “The xUnit Family of Unit Test Frameworks,” Unit Test Frameworks, O'Reilly, 2004.
[39] K. Beck and E. Gamma, “Test Infected: Programmers Love Writing Tests,” Java Report, vol. 7, no. 3, pp. 51-56, 1998.
[40] P. Crewson, “Fundamentals of Clinical Research for Radiologists: Reader Agreement Studies,” Am. J. Roentgenology, vol. 184, no. 5, pp. 1391-1397, 2005.
[41] J. Landis and G. Koch, “The Measurement of Observer Agreement for Categorical Data,” J. Biometrics, vol. 33, no. 1, pp. 159-174, Mar. 1972.
[42] H. Coolican, Research Methods and Statistics in Psychology, fourth ed. Hodder Ar nold, 2004.
Index Terms:
Test design, Quality assurance, Maintainability
Citation:
Bart Van Rompaey, Bart Du Bois, Serge Demeyer, Matthias Rieger, "On The Detection of Test Smells: A Metrics-Based Approach for General Fixture and Eager Test," IEEE Transactions on Software Engineering, vol. 33, no. 12, pp. 800-817, Sept. 2007, doi:10.1109/TSE.2007.70745
