The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.06 - November/December (2010 vol.36)
pp: 778-797
Vahid Garousi , University of Calgary, Calgary
ABSTRACT
Genetic algorithms (GAs) have been applied previously to UML-driven stress test requirements generation with the aim of increasing chances of discovering faults relating to network traffic in distributed real-time systems. However, since evolutionary algorithms are heuristic, their performance can vary across multiple executions, which may affect robustness and scalability. To address this, we present the design and technical detail of a UML-driven, GA-based stress test requirements generation tool, together with its empirical analysis. The main goal is to analyze and improve the applicability, efficiency, and effectiveness and also to validate the design choices of the GA used in the tool. Findings of the empirical evaluation reveal that the tool is robust and reasonably scalable when it is executed on large-scale experimental design models. The study also reveals the main bottlenecks and limitations of the tools, e.g., there is a performance bottleneck when the system under test has a large number of sequence diagrams which could be triggered independently from each other. In addition, issues specific to stress testing, e.g., the impact of variations in task arrival pattern types, reveal that the tool generally generates effective test requirements, although the features of those test requirements might be different in different runs (e.g., different stress times from the test start time might be chosen). While the use of evolutionary algorithms to generate software test cases has been widely reported, the extent, depth, and detail of the empirical findings presented in this paper are novel and suggest that the proposed approach is effective and efficient in generating stress test requirements. It is hoped that the findings of this empirical study will help other SBSE researchers with the empirical evaluation of their own techniques and tools.
INDEX TERMS
Index Term—Search-based testing, genetic algorithms, stress testing, test tools, test automation, empirical analysis.
CITATION
Vahid Garousi, "A Genetic Algorithm-Based Stress Test Requirements Generator Tool and Its Empirical Evaluation", IEEE Transactions on Software Engineering, vol.36, no. 6, pp. 778-797, November/December 2010, doi:10.1109/TSE.2010.5
REFERENCES
[1] J.J.P. Tsai, Y. Bi, S.J.H. Yang, and R.A.W. Smith, Distributed Real-Time Systems: Monitoring, Visualization, Debugging, and Analysis. John Wiley & Sons, 1996.
[2] E. Weyuker and F.I. Vokolos, "Experience with Performance Testing of Software Systems: Issues, an Approach and Case Study," IEEE Trans. Software Eng., vol. 26, no. 12, pp. 1147-1156, Dec. 2000.
[3] R. Kuhn, "Sources of Failure in the Public Switched Telephone Network," Computer, vol. 30, no. 4, pp. 31-36, Apr. 1997.
[4] A. Avritzer and E.J. Weyuker, "Generating Test Suites for Software Load Testing," Proc. Int'l Symp. Software Testing and Analysis, pp. 44-57, 1994.
[5] A. Avritzer and E.J. Weyuker, "The Automatic Generation of Load Test Suites and the Assessment of the Resulting Software," IEEE Trans. Software Eng., vol. 21, no. 9, pp. 705-716, Sept. 1995.
[6] J. Brunton, G. Digby, and A. Doherty, "Design and Operational Philosophy for a Metro Power Network SCADA System," Proc. Int'l Conf. Power System Control and Management, pp. 176-180, 1996.
[7] E.-K. Chan and H. Ebenhoh, "The Implementation and Evolution of a SCADA System for a Large Distribution Network," IEEE Trans. Power Systems, vol. 7, no. 1, pp. 320-326, Feb. 1992.
[8] Wikipedia "Northeast Blackout of 2003," http://en.wikipedia. org/wiki2003_North_America_blackout , 2007.
[9] V. Garousi, L. Briand, and Y. Labiche, "Traffic-Aware Stress Testing of Distributed Real-Time Systems Based on UML Models Using Genetic Algorithms," Elsevier J. Systems and Software, special issue on model-based software testing, vol. 81, no. 2, pp. 161-185, 2008.
[10] Object Management Group (OMG), "UML 2.1.2 Superstructure Specification," 2007.
[11] V. Garousi, L. Briand, and Y. Labiche, "Traffic-Aware Stress Testing of Distributed Systems Based on UML Models," Proc. Int'l Conf. Software Eng., pp. 391-400, 2006.
[12] V. Garousi, "GARUS (Genetic Algorithm-Based Test Requirement Tool for Real-Time Distributed Systems)," http://www.softqual. ucalgary.ca/toolsGARUS , Sept. 2009.
[13] L. Briand, "A Critical Analysis of Empirical Research in Software Testing," Proc. Int'l Symp. Empirical Software Eng. and Measurement, Keynote Address, 2007.
[14] M. Harman, "The Current State and Future of Search-Based Software Engineering," Proc. Int'l Conf. Software Eng., Future of Software Eng., pp. 342-357, 2007.
[15] V. Garousi, "Empirical Analysis of a Genetic Algorithm-based Stress Test Technique for Distributed Real-Time Systems," Proc. Genetic and Evolutionary Computation Conf., Search-Based Software Eng. Track, pp. 1743-1750, 2008.
[16] V. Basili, G. Caldeira, and H.D. Rombach, "The Goal Question Metric Approach," Encyclopedia of Software Eng., Wiley, 1994.
[17] Y. Min, X. Jin, X. Su, and B. Peng, "Empirical Analysis of the Spatial Genetic Algorithm on Small-World Networks," Proc. Int'l Conf. Computational Science, pp. 1032-1039, 2006.
[18] G. Rudolph, "Convergence Analysis of Canonical Genetic Algorithms," IEEE Trans. Neural Networks, vol. 5, no. 1, pp. 96-101, Jan. 1994.
[19] K. Sastry, M. Pelikan, and D.E. Goldberg, "Empirical Analysis of Ideal Recombination on Random Decomposable Problems," Proc. Genetic and Evolutionary Computation Conf., pp. 1388-1395, 2007.
[20] M. Harman, Y. Hassoun, K. Lakhotia, P. McMinn, and J. Wegener, "The Impact of Input Domain Reduction on Search-Based Test Data Generation," Proc. Int'l Symp. Foundations of Software Eng., pp. 155-164, 2007.
[21] M. Harman and P. McMinn, "A Theoretical & Empirical Analysis of Evolutionary Testing and Hill Climbing for Structural Test Data Generation," Proc. Int'l Symp. Software Testing and Analysis, pp. 73-83, 2007.
[22] M. Xiao, M.E.M. Reformat, and J. Miller, "Empirical Evaluation of Optimization Algorithms When Used in Goal-Oriented Automated Test Data Generation Techniques," Empirical Software Eng., vol. 12, no. 2, pp. 183-239, 2007.
[23] J.J. Greffenstette, "Optimization of Control Parameters for Genetic Algorithms," IEEE Trans. Systems, Man, and Cybernetics, vol. 16, no. 1, pp. 122-128, Jan. 1986.
[24] Object Management Group (OMG) "UML 2.0 Superstructure Specification," 2005.
[25] V. Garousi, L. Briand, and Y. Labiche, "Control Flow Analysis of UML 2.0 Sequence Diagrams," Proc. European Conf. Model Driven Architecture—Foundations and Applications, pp. 160-174, 2005.
[26] V. Garousi, L. Briand, and Y. Labiche, "Traffic-Aware Stress Testing of Distributed Real-Time Systems Based on UML Models Using Genetic Algorithms," Technical Report SCE-06-09, Carleton Univ., 2006.
[27] V. Garousi, "Traffic-Aware Stress Testing of Distributed Real-Time Systems Based on UML Models Using Genetic Algorithms," PhD thesis, Dept. of Systems and Computer Eng., Carleton Univ., 2006.
[28] V. Garousi, L. Briand, and Y. Labiche, "Control Flow Analysis of UML 2.0 Sequence Diagrams," Proc. European Conf. Model Driven Architecture—Foundations and Applications, pp. 160-174, 2005.
[29] Object Management Group (OMG) "UML Profile for Schedulability, Performance, and Time (v1.1)," 2005.
[30] V. Garousi, "Traffic-Aware Stress Testing of Distributed Real-Time Systems Based on UML Models in the Presence of Time Uncertainty," Proc. IEEE Int'l Conf. Software Testing, Verification, and Validation, pp. 92-101, 2008.
[31] M. Wall, "GAlib: A C++ Library of Genetic Algorithm Components," Documentation Version 2.4, Massachusetts Inst. of Tech nology, 1996.
[32] R.L. Haupt and S.E. Haupt, Practical Genetic Algorithms. Wiley-Interscience, 1998.
5 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool