This Article 
 Bibliographic References 
 Add to: 
In-Process Evaluation for Software Inspection and Test
November 1993 (vol. 19 no. 11)
pp. 1055-1070

The goal of software inspection and test is to reduce the expected cost of software failure over the life of a product. The authors extend the use of defect triggers, the events that cause defects to be discovered, to help evaluate the effectiveness of inspections and test scenarios. In the case of inspections, the defect trigger is defined as a set of values that associate the skills of the inspector with the discovered defect. Similarly, for test scenarios, the defect trigger values embody the deferring strategies being used in creating these scenarios. The usefulness of triggers in evaluating the effectiveness of software inspections and tests is demonstrated by evaluating the inspection and test activities of some software products. These evaluations are used to point to deficiencies in inspection and test strategies, and to progress made in improving such strategies. The trigger distribution of the entire inspection or test series may then be used to highlight areas for further investigation, with the aim of improving the design, implementation, and test processes.

[1] R. A. Radice and R. W. Phillips,Software Engineering: An Industrial Approach. Englewood Cliffs, NJ: Prentice Hall, 1988.
[2] W. W. Royce, "Managing the development of large software systems: concepts and techniques," inProc. 9th Int. Conf. Software Eng.(a reprint of 1970 article), Mar. 1987, pp. 328-338.
[3] B. W. Boehm, "A spiral model of software development and enhancement,"ACM SIGSOFT Software Engineering Notes, vol. 11, pp. 14-24, Aug. 1986.
[4] B. Boehm, "Improving software productivity,"Computer, vol. 20, no. 9, pp. 43-57, Sept. 1987.
[5] S. Hekmatpour, "Experience with evolutionary prototyping in a large software system,"ACM SIGSOFT Software Engineering Notes, vol. 12, pp. 38-41, Jan. 1987.
[6] M. E. Fagan, "Design and code inspections to reduce errors in program development,"IBM Syst. J., vol. 15, no. 3, pp. 182-211, 1976.
[7] M. E. Fagan, "Advances in software inspections,"IEEE Trans. Software Eng., vol. SE-12, no. 5, pp. 744-751, July 1986.
[8] G. J. Myers,Software Reliability: Principles and Practices. New York: Wiley, 1976.
[9] G. J. Myers,The Art of Software Testing. New York: Wiley, 1979.
[10] R. Chillarege, I. Bhandari, J. Chaar, M. Halliday, D. Moebus, B. K. Ray, and M. Y. Wong, "Orthogonal defect classification for defect control," Tech. Rep. RC 17320 (Log 76564), IBM Research, 1991.
[11] R. Chillarege, I. Bhandari, J. Chaar, M. Halliday, D. Moebus, B. K. Ray, and M. Y. Wong, "Orthogonal defect classification--a concept for in-process measurements,"IEEE Trans. Software Eng., vol. 18, Nov. 1992.
[12] W. S. Humphrey,Managing the Software Process. Reading, MA: Addison-Wesley, 1989.
[13] W. E. Stephenson, "An analysis of resources used on the SAFEGUARD system software development," Tech. Rep., Bell Labs, Aug. 1976.
[14] E. B. Daly, "Management of software engineering,"IEEE Trans. Software Eng., vol. SE-3, pp. 229-242, May 1977.
[15] B. W. Boehm,Software Engineering Economics. Englewood Cliffs, NJ: Prentice-Hall, 1981.
[16] R. A. Radice, N. K. Roth, A. C. O'Hara, and W. A. Ciarfella, "A programming process architecture,"IBM Syst. J., vol. 24, no. 2, 1985.
[17] W. S. Humphrey, "Characterizing the software process: a maturity framework,"IEEE Software, Mar. 1988.
[18] R. C. Linger, H. D. Mills, and B. I. Witt,Structured Programming Theory and Practice. Reading. MA: Addison-Wesley, 1979.
[19] R. Mays, C. Jones, G. Holloway, and D. Studinski, "Experiences with defect prevention,"IBM Syst. J., vol. 29, no. 1, 1990.
[20] B. Littlewood and J. L. Verrall, "A Bayesian reliability growth model for computer software,"J. Royal Statistical Soc., vol. 22, no. 3, pp. 332-346, 1973.
[21] C. V. Ramamoorthy and F. B. Bastani, "Software reliability--status and perspectives,"IEEE Trans. Software Eng., vol. 8, no. 4, pp. 354-371, 1982.
[22] A. L. Goel, "Software reliability models: assumptions, limitations, and applicability,"IEEE Trans. Software Eng., vol. SE-11, no. 12, pp. 1411-1423, 1985.
[23] J. D. Musaet al., Software Reliability Measurement, Prediction, Application. New York: McGraw-Hill International, 1987.
[24] G. Wenneson, "Quality assurance software inspections at NASA Ames: metrics for feedback and modification," inProc. 10th Ann. Software Eng. Workshop(Goddard Space Flight Center), Dec. 1985.
[25] G. G. Schulmeyer and J. I. McManus,Handbook of Software Quality Assurance. New York: Van Nostrand Reinhold, 1987.
[26] J. C. Kelly, J. S. Sherif, and J. Hops, "An analysis of defect densities found during software inspections,"J. Syst. Software, vol. 17, pp. 111-117, 1992.
[27] R. C. Linger, "Cleamroom software engineering for zero-deficit software," inProc. 15th Int. Conf. Software Eng., pp. 2-13, 1993.
[28] P. A. Currit, M. Dyer, and H. D. Mills, "Certifying the reliability of software,"IEEE Trans. Software Eng., vol. SE-12, no. 1, pp. 3-11, Jan. 1986.
[29] B. Beizer,Software Testing Techniques. New York: Van Nostrand Reinhold, 1983.
[30] H. D. Mills, "On the statistical validation of computer programs," inSoftware Productivity, H. D. Mills, Ed. New York: Dorset House, 1988.
[31] J. C. Knight and P. E. Amman, "An experimental evaluation of error seeding as a program evaluation technique," inProc. 10th Ann. Software Eng. Workshop(Goddard Space Flight Center), Dec. 1985.
[32] D. N. Card, T. L. Clark, and R. A. Berg, "Improving software quality and productivity,"Inform. Software Technol., vol. 29, June 1987.
[33] I. S. Bhandari, M. J. Halliday, J. K. Chaar, and R. Chillaregeet al., "In-process improvement through defect data interpretation," Tech. Rep. (Log 81922), IBM Research, 1993; also submitted toIBM Syst. J.
[34] R. Chillarege, W.-L Kao, and R. G. Condit, "Defect type and its impact on the growth curve," inProc. 13th Int. Conf. Software Engineering, 1991.
[35] M. Sullivan and R. Chillarege, "Software defects and their impact on system availability-A study of field failures in operating systems,"Digest of Papers: The 21st Int. Symp. Fault-Tolerant Computing, pp. 2-9, 1991.
[36] I. Bhandari, "Attribute focusing: open-world data exploration applied to software production process control," Tech. Rep. RC 18320 (Log 80194), IBM Research, 1992.

Index Terms:
in-process evaluation; software inspection; software testing; expected cost; software failure; defect triggers; test scenarios; program testing; software quality; software reliability
J.K. Chaar, M.J. Halliday, I.S. Bhandari, R. Chillarege, "In-Process Evaluation for Software Inspection and Test," IEEE Transactions on Software Engineering, vol. 19, no. 11, pp. 1055-1070, Nov. 1993, doi:10.1109/32.256853
Usage of this product signifies your acceptance of the Terms of Use.