This Article 
 Bibliographic References 
 Add to: 
An Empirical Study of Testing and Integration Strategies Using Artificial Software Systems
October 1993 (vol. 19 no. 10)
pp. 941-949

There has been much discussion about the merits of various testing and integration strategies. Top-down, bottom-up, big-bang, and sandwich integration strategies are advocated by various authors. Also, some authors insist that modules be unit tested, while others believe that unit testing diverts resources from more effective verification processes. This article addresses the ability of the aforementioned integration strategies to detect defects, and produce reliable systems. It also explores the efficacy of spot unit testing, and compares phased and incremental versions of top-down and bottom-up integration strategies. Relatively large artificial software systems were constructed using a code generator with ten basic module templates. These systems were seeded with known defects and tested using the above testing and integration strategies. A number of experiments were then conducted using a simulator whose validity was established by comparing results against these artificial systems. The defect detection ability and resulting system reliability were measured for each strategy. Results indicated that top-down integration strategies are generally most effective in terms of defect correction. Top-down and big-bang strategies produced the most reliable systems. Results favored neither those strategies that incorporate spot unit testing nor those that do not; also, results favored neither phased nor incremental strategies.

[1] V. R. Basili and R. W. Selby, "Comparing the effectiveness of software testing strategies,"IEEE Trans. Software Eng., vol. SE-13, no. 12, pp. 1278-1296, Dec. 1987.
[2] B. Beizer,Software System Testing and Quality Assurance, Van Nostrand Reinhold, New York, 1984, p. 300.
[3] G. Collins and G. Blay,Structured Systems Development Techniques: Strategic Planning to System Testing. New York: Wiley, 1983, p. 206.
[4] M.S. Deutsch,Software Verification and Validation: Realistic Project Approaches, Prentice-Hall, Englewood Cliffs, N.J., 1982, pp. 51-89.
[5] J. W. Duran and S. C. Ntafos, "An evaluation of random testing,"IEEE Trans. Software Eng., vol. SE-10, no. 4, pp. 438-444, July 1984.
[6] M. R. Girgis and M. R. Woodward, "An experimental comparison of the error exposing ability of program testing criteria," inProc. Workshop Software Testing. Washington, DC: IEEE Computer Society Press, 1986, pp. 64-73.
[7] B. Hetzel,The Complete Guide to Software Testing, 2nd ed. Wellesley, MA: QED Info. Sci., 1988.
[8] W. E. Howden, "Functional program testing,"IEEE Trans. Software Eng., vol. SE-6, pp. 162-169, 1980.
[9] H. D. Mills, "Structured programming: Retrospect and prospect,"IEEE Software, vol. 3, no. 6, pp. 58-66, Nov. 1986.
[10] H. D. Mills, M. Dyer, and R. C. Linger, "Cleanroom software engineering,"IEEE Software, vol. 4, no. 5, pp. 19-25, Sept. 1987.
[11] G. J. Myers,The Art of Software Testing. New York: Wiley, 1979.
[12] G. J. Myers,Software Reliability: Principles and Practices. New York: Wiley, 1976.
[13] S. L. Pfleeger,Software Engineering. New York: Macmillan, 1991.
[14] R. M. Poston and M. W. Bruen, "Counting down to zero software failures,"IEEE Software, vol. 4, no. 5, pp. 54-61, Sept. 1987.
[15] B. P. Miller, C. Macrander, and S. Sechrest, "A distributed programs monitor for Berkeley UNIX,"Software--Practice and Experience, vol. 16, no. 2, pp. 183-200, Feb. 1986.
[16] J. H. Rowland and Y. Zuyuan, "Experimental comparison of three system test strategies: Preliminary report," inProc. ACM SIGSOFT 3rd Symp. Software Testing, Analysis, and Verification. ACM Press, Dec. 1989, pp. 141-145.
[17] H. F. Smith,Data Structures Form and Function, San Diego, CA: Harcourt Brace Jovanovich, 1987.
[18] J. A. Solheim, "Testing and integration strategies of large software systems", Ph.D, dissertation, Univ. Wyoming, Laramie, 1992 (microfiche available through UMI Dissertation Services, Ann Arbor, MI).
[19] I. Sommerville,Software Engineering. London: Addison-Wesley, 1982.
[20] P. Thévenod-Fosse and H. Waeselynck, "An investigation of statistical software testing,"J. Software Testing, Verification and Reliability, vol. 1, no. 2, pp. 5-25, July-Sept. 1991.
[21] R. Walpole and R. Myers,Probability and Statistics for Engineers and Scientists. New York: Macmillan, 1978, pp. 382-383.
[22] E. Yourdon,Managing the Structured Techniques. Englewood Cliffs, NJ: Prentice-Hall, 1989, p. 79.

Index Terms:
artificial software systems; top-down strategies; code generator; sandwich integration strategies; verification processes; reliable systems; spot unit testing; bottom-up integration; system reliability; defect correction; big-bang strategies; program testing; software reliability
J.A. Solheim, J.H. Rowland, "An Empirical Study of Testing and Integration Strategies Using Artificial Software Systems," IEEE Transactions on Software Engineering, vol. 19, no. 10, pp. 941-949, Oct. 1993, doi:10.1109/32.245736
Usage of this product signifies your acceptance of the Terms of Use.