This Article 
 Bibliographic References 
 Add to: 
Software Reliability and Testing Time Allocation: An Architecture-Based Approach
May/June 2010 (vol. 36 no. 3)
pp. 323-337
Roberto Pietrantuono, Federico II University of Naples, Naples
Stefano Russo, Federico II University of Naples, Naples
Kishor S. Trivedi, Duke University, Durham
With software systems increasingly being employed in critical contexts, assuring high reliability levels for large, complex systems can incur huge verification costs. Existing standards usually assign predefined risk levels to components in the design phase, to provide some guidelines for the verification. It is a rough-grained assignment that does not consider the costs and does not provide sufficient modeling basis to let engineers quantitatively optimize resources usage. Software reliability allocation models partially address such issues, but they usually make so many assumptions on the input parameters that their application is difficult in practice. In this paper, we try to reduce this gap, proposing a reliability and testing resources allocation model that is able to provide solutions at various levels of detail, depending upon the information the engineer has about the system. The model aims to quantitatively identify the most critical components of software architecture in order to best assign the testing resources to them. A tool for the solution of the model is also developed. The model is applied to an empirical case study, a program developed for the European Space Agency, to verify model's prediction abilities and evaluate the impact of the parameter estimation errors on the prediction accuracy.

[1] DO-178B/ED12B, "Software Consideration in Airborne Systems and Equipment Certification," RTCA and EUROCAE, Dec. 1992.
[2] SAF.ET1.ST03.1000-MAN-01, "Air Navigation System Safety Assessment Methodology (v2-0)," EUROCONTROL EATMP Safety Management, Apr. 2004.
[3] N. Wattanapongsakorn and S.P. Levitan, "Reliability Optimization Models for Embedded Systems with Multiple Applications," IEEE Trans. Reliability, vol. 53, no. 3, pp. 406-416, Sept. 2004.
[4] J. Onishi, S. Kimura, R.J.W. James, and Y. Nakagawa, "Solving the Redundancy Allocation Problem with a Mix of Components Using the Improved Surrogate Constraint Method," IEEE Trans. Reliability, vol. 56, no. 1, pp. 94-101, Mar. 2007.
[5] F.W. Rice, C.R. Cassady, and R.T. Wise, "Simplifying the Solution of Redundancy Allocation Problems," Proc. Ann. Reliability and Maintainability Symp., pp. 190-194, 1999.
[6] M.R. Lyu, S. Rangarajan, and A.P.A. van Moorsel, "Optimal Allocation of Test Resources for Software Reliability Growth Modeling in Software Development," IEEE Trans. Reliability, vol. 51, no. 2, pp. 183-192, June 2002.
[7] R.H. Hou, S.Y. Kuo, and Y.P. Chang, "Efficient Allocation of Testing Resources for Software Module Testing Based on the Hyper-Geometric Distribution Software Reliability Growth Model," Proc. Seventh Int'l Symp. Software Reliability Eng., pp. 289-298, Oct./Nov. 1996.
[8] Y. Nakagawa and S. Miyazaki, "Surrogate Constraints Algorithm for Reliability Optimization Problems with Two Constraints," IEEE Trans. Reliability, vol. 30, no. 2, pp. 175-181, 1981.
[9] L. Painton and J. Campbell, "Genetic Algorithms in Optimization of System Reliability," IEEE Trans. Reliability, vol. 44, no. 2, pp. 172-178, June 1995.
[10] F.A. Tillman, C.L. Hwang, and W. Kuo, "Determining Component Reliability and Redundancy for Optimum System Reliability," IEEE Trans. Reliability, vol. R-26, pp. 162-165, 1977.
[11] A.O.C. Elegbede, C. Chu, K.H. Adjallah, and F. Yalaoui, "Reliability Allocation through Cost Minimization," IEEE Trans. Reliability, vol. 52, no. 1, pp. 106-111, Mar. 2003.
[12] I. Rani and R.B. Misra, "Economic Allocation of Target Reliability in Modular Software Systems," Proc. Ann. Reliability and Maintainability Symp., pp. 428-432, 2005.
[13] A. Mettas, "Reliability Allocation and Optimization for Complex Systems," Proc. Ann. Reliability and Maintainability Symp., pp. 216-221, 2000.
[14] M.R. Lyu, S. Rangarajan, and A.P.A. van Moorsel, "Optimization of Reliability Allocation and Testing Schedule for Software Systems," Proc. Eighth Int'l Symp. Software Reliability Eng., pp. 336-347, 1997.
[15] M.E. Helander, M. Zhao, and N. Ohisson, "Planning Models for Software Reliability and Cost," IEEE Trans. Software Eng., vol. 24, no. 6, pp. 420-434 , June 1998.
[16] W. Everett, "Software Component Reliability Analysis," Proc. Symp. Application Specific Systems and Software Eng. Technology, pp. 204-211, 1999.
[17] K. Goseva-Popstojanova and K.S. Trivedi, "Architecture-Based Approach to Reliability Assessment of Software Systems," Performance Evaluation, vol. 45, nos. 2/3, pp. 179-204, 2001.
[18] S.S. Gokhale, W.E. Wong, J.R. Horganc, and K.S. Trivedi, "An Analytical Approach to Architecture-Based Software Performance and Reliability Prediction," Performance Evaluation, vol. 58, no. 4, pp. 391-412, 2004.
[19] K. Goseva-Popstojanova, A.P. Mathur, and K.S. Trivedi, "Comparison of Architecture-Based Software Reliability Models," Proc. 12th Int'l Symp. Software Reliability Eng., pp. 22-31, 2001.
[20] S. Gokhale, M.R. Lyu, and K.S. Trivedi, "Incorporating Fault Debugging Activities into Software Reliability Models: A Simulation Approach," IEEE Trans. Reliability, vol. 55, no. 2, pp. 281-292, June 2006.
[21] W. Wang, Y. Wu, and M.H. Chen, "An Architecture-Based Software Reliability Model," Proc. Pacific Rim Dependability Symp., 1999.
[22] S. Gokhale and K.S. Trivedi, "Time/Structure Based Software Reliability Model," Annals of Software Eng., vol. 8, pp. 85-121, 1999.
[23] A. Reibman and K.S. Trivedi, "Numerical Transient Analysis of Markov Models," Computers and Operations Research, vol. 15, no. 1, pp. 19-36, 1988.
[24] S.S. Gokhale and K.S. Trivedi, "Reliability Prediction and Sensitivity Analysis Based on Software Architecture," Proc. 13th Int'l Symp. Software Reliability Eng., pp. 64-75, 2002.
[25] K.S. Trivedi, Probability and Statistics with Reliability, Queuing and Computer Science Applications. John Wiley and Sons, 2001.
[26] R.A. Sahner, K.S. Trivedi, and A. Puliafito, Performance and Reliability Analysis of Computer Systems: An Example-Based Approach Using the SHARPE Software Package. Kluwer Academic Publishers, 1996.
[27] C. Huang, S. Kuo, and M.R. Lyu, "An Assessment of Testing-Effort Dependent Software Reliability Growth Models," IEEE Trans. Reliability, vol. 56, no. 2, pp. 198-211, June 2007.
[28] V.S. Sharma and K.S. Trivedi, "Quantifying Software Performance, Reliability and Security: An Architecture-Based Approach," J. Systems and Software, vol. 80, no. 4, pp. 493-509, Apr. 2007.
[29] S. Yamada, H. Ohtera, and H. Narihisa, "Software Reliability Growth Models with Testing Effort," IEEE Trans. Reliability, vol. 35, no. 1, pp. 19-23, Apr. 1986.
[30] C. Huang and M.R. Lyu, "Optimal Release Time for Software Systems Considering Cost, Testing-Effort, and Test Efficiency," IEEE Trans. Reliability, vol. 54, no. 4, pp. 583-591, Dec. 2005.
[31] A. Pasquini, A.N. Crespo, and P. Matrella, "Sensitivity of Reliability-Growth Models to Operational Profile Errors vs Testing Accuracy," IEEE Trans. Reliability, vol. 45, no. 4, pp. 531-540, 1996.
[32] S. Gokhale and M.R. Lyu, "Regression Tree Modeling for the Prediction of Software Quality," Proc. Third ISSAT Int'l Conf. Reliability, pp. 31-36, Mar. 1997.
[33] M. Lipow, "Number of Faults Per Line of Code," IEEE Trans. Software Eng., vol. 8, no. 4, pp. 437-439, July 1982.
[34] V. Almering, M. Van Genuchten, G. Cloudt, and P.J.M. Sonnemans, "Using Software Reliability Growth Models in Practice," IEEE Software, vol. 24, no. 6, pp. 82-88, Nov./Dec. 2007.
[35] A. Srivastava and A. Eustace, "Atom: A System for Building Customized Program Analysis Tools," Proc. SIGPLAN '94 Conf. Programming Language Design and Implementation, pp. 196-1349, 1994.
[36] S. Yacoub, B. Cukic, and H.H. Ammar, "A Scenario-Based Reliability Analysis Approach for Component-Based Software," IEEE Trans. Reliability, vol. 53, no. 4, pp. 465-480, Dec. 2004.
[37] S.S. Gokhale, T. Philip, P.N. Marinos, and K.S. Trivedi, "Unification of Finite Failure Non-Homogeneous Poisson Process Models through Test Coverage," Proc. Seventh Int'l Symp. Software Reliability Eng., 1996.
[38] C. Stringfellow and A.A. Andrews, "An Empirical Method for Selecting Software Reliability Growth Models," Empirical Software Eng., vol. 7, no. 4, pp. 297-318, 2002.
[39] S. Ramani, S. Gokhale, and K. Trivedi, "SREPT: Software Reliability Estimation and Prediction Tool," Performance Evaluation, Special Issue on Tools for Performance Evaluation, vol. 39, no. 1, pp. 37-60, 2000.
[40] K. Schittkowski, "NLQPL: A FORTRAN-Subroutine Solving Constrained Nonlinear Programming Problems," Annals of Operations Research, vol. 5, pp 485-500, 1985.

Index Terms:
Reliability, software architecture, software testing.
Roberto Pietrantuono, Stefano Russo, Kishor S. Trivedi, "Software Reliability and Testing Time Allocation: An Architecture-Based Approach," IEEE Transactions on Software Engineering, vol. 36, no. 3, pp. 323-337, May-June 2010, doi:10.1109/TSE.2010.6
Usage of this product signifies your acceptance of the Terms of Use.