The Community for Technology Leaders
RSS Icon
Issue No.02 - March/April (2008 vol.34)
pp: 271-286
Jingyue Li , Norwegian University of Science and Technology
Reidar Conradi , Norwegian University of Science and Technology
Odd Petter Slyngstad , Norwegian University of Science and Technology
Marco Torchiano , Politecnico di Torino
Maurizio Morisio , Politecnico di Torino, Italy
Christian Bunse , the International University in Germany
An international survey on risk management in software development with OTS (Off-The-Shelf) components is reported upon and discussed. The survey investigated actual risk-management activities and their correlations with the occurrences of typical risks in OTS component-based development. Data from 133 software projects in Norway, Italy, and Germany were collected using a stratified random sample of IT companies. The results show that OTS components normally do not contribute negatively to the quality of the software system as a whole, as what is commonly expected. However, issues such as the underestimation of integration effort and inefficient debugging remain problematic and require further investigation. The results also illustrate several promising effective risk-reduction activities, e.g. putting more effort into learning relevant OTS components, integrating unfamiliar components first, evaluating the quality of candidate OTS components thoroughly, and regularly monitoring the support capability of OTS providers. Five hypotheses are proposed regarding these risk reduction activities. The results also indicate that several other factors, such as project, cultural, and human-social factors, have to be investigated to deal with the possible risks of OTS-based projects thoroughly.
Software Engineering/Reusable Software, Software Engineering/Management, Software Engineering/Software Engineering Process
Jingyue Li, Reidar Conradi, Odd Petter Slyngstad, Marco Torchiano, Maurizio Morisio, Christian Bunse, "A State-of-the-Practice Survey of Risk Management in Development with Off-the-Shelf Software Components", IEEE Transactions on Software Engineering, vol.34, no. 2, pp. 271-286, March/April 2008, doi:10.1109/TSE.2008.14
[1] C. Szyperski, D. Gruntz, and S. Murer, Component Software— Beyond Object-Oriented Programming. Addison-Wesley, 2002.
[2] L.C. Rose, “Risk Management of COTS Based System Development,” Component-Based Software Quality—Methods and Techniques, pp. 352-373, Springer, 2003.
[3] B.W. Boehm, D. Port, Y. Yang, and J. Bhuta, “Not All CBS Are Created Equally: COTS-Intensive Project Types,” Proc. Second Int'l Conf. COTS-Based Software Systems, pp. 36-50, 2003.
[4] J. Voas, “COTS Software—The Economical Choice?” IEEE Software, vol. 15, no. 2, pp. 16-19, Mar./Apr. 1998.
[5] J. Voas, “The Challenges of Using COTS Software in Component-Based Development,” Computer, vol. 31, no. 6, pp. 44-45, June 1998.
[6] C. Abts, B.W. Boehm, and E.B. Clark, “COCOTS: A COTS Software Integration Lifecycle Cost Model—Model Overview and Preliminary Data Collection Findings,” Technical Report USC-CSE-2000-501, USC Center for Software Eng., 2000/usccse2000-501usccse2000-501.pdf , 2000.
[7] G. Kotonya and A. Rashid, “A Strategy for Managing Risk in Component-Based Software Development,” Proc. 27th EUROMICRO Conf., pp. 12-21, 2001.
[8] COTS Risk Factor, CRMG.htm, 2003.
[9] B. Fitzgerald, “A Critical Look at Open Source,” Computer, vol. 37, no. 7, pp. 92-94, July 2004.
[10] G. Lawton, “Open Source Security: Opportunity or Oxymoron?” Computer, vol. 35, no. 3, pp. 18-21, Mar. 2002.
[11] P. Vitharana, “Risks and Challenges of Component-Based Software Development,” Comm. ACM, vol. 46, no. 8, pp. 67-72, Aug. 2003.
[12] M. Ruffin and C. Ebert, “Using Open Source Software in Product Development: A Primer,” IEEE Software, vol. 21, no. 1, pp. 82-86, Jan./Feb. 2004.
[13] M. Torchiano and M. Morisio, “Overlooked Facts on COTS-Based Development,” IEEE Software, vol. 21, no. 2, pp. 88-93, Mar./Apr. 2004.
[14] J. Li, F.O. Bjørnson, R. Conradi, and V.B. Kampenes, “An Empirical Study of Variations in COTS-Based Software Development Processes in Norwegian IT Industry,” J. Empirical Software Eng., vol. 11, no. 3, pp. 433-461, Sept. 2006.
[15] B.W. Boehm, “Software Risk Management: Principles and Practices,” IEEE Software, vol. 8, no. 1, pp. 32-41, Jan. 1991.
[16] J. Li, M. Torchiano, R. Conradi, O.P.N. Slyngstad, and C. Bunse, “A State-of-the-Practice Survey of the Off-the-Shelf Component-Based Development Processes,” Proc. Ninth Int'l Conf. Software Reuse, pp. 16-28, June 2006.
[17] J. Li, R. Conradi, O.P.N. Slyngstad, C. Bunse, M. Torchiano, and M. Morisio, “An Empirical Study on the Decision Making Process in Off-the-Shelf Component Based Development,” Proc. 28th Int'l Conf. Software Eng., pp. 897-900, May 2006.
[18] J. Li, R. Conradi, C. Bunse, M. Torchiano, O.P.N. Slyngstad, and M. Morisio, “Development with Off-the-Shelf Components: 10Facts,” IEEE Software, to be published.
[19] B.W. Boehm, Software Risk Management, tutorial, IEEE CS Press, 1989.
[20] H. Barki, S. Rivard, and J. Talbot, “Toward an Assessment of Software Development Risk,” J. Management Information Technology, vol. 22, no. 2, pp. 359-371, Dec. 1993.
[21] M. Carr, S. Kondra, I. Monarch, F. Ulrich, and C. Walker, “Taxonomy-Based Risk Identification,” Technical Report SEI-93-TR-006, SEI, 1993.
[22] S.A. Sherer, “The Three Dimensions of Software Risk: Technical, Organizational, and Environmental,” Proc. 28th Hawaii Int'l Conf. System Sciences, pp. 369-378, 1995.
[23] C.G. Chittister and Y.Y. Haimes, “System Integration via Software Risk Management,” IEEE Trans. Systems, Man, and Cybernetics, vol. 26, no. 5, pp. 521-532, Sept. 1996.
[24] J. Ropponen and K. Lyytinen, “Components of Software Development Risk: How to Address Them? A Project Manager Survey,” IEEE Trans. Software Eng., vol. 26, no. 2, pp. 98-112, Feb. 2000.
[25] M. Keil, P.E. Cule, K. Lyytinen, and R.C. Schmidt, “A Framework for Identifying Software Project Risks,” Comm. ACM, vol. 4, no. 11, pp. 76-83, Nov. 1998.
[26] L. Wallace and M. Keil, “Software Project Risks and Their Effect on Outcomes,” Comm. ACM, vol. 47, no. 4, pp. 68-73, Apr. 2004.
[27] B.W. Boehm, “A Spiral Model of Software Development and Enhancement,” Computer, vol. 21, no. 5, pp. 61-72, May 1988.
[28] A. Gemmer, “Risk Management: Moving beyond Process,” Computer, vol. 30, no. 5, pp. 33-41, May 1997.
[29] H. Hecht, Systems Reliability and Failure Prevention. Artech House, 2003.
[30] I. Sommerville, Software Engineering, seventh ed. Addison-Wesley, 2004.
[31] T. Moynihan, “How Experienced Project Managers Assess Risk,” IEEE Software, vol. 14, no. 3, May/June 1997.
[32] R.L. Glass, “Frequently Forgotten Fundamental Facts about Software Engineering,” IEEE Software, vol. 18, no. 3, pp. 110-112, May/June 2001.
[33] I. Crnkovic and M. Larsson, Building Reliable Component-Based Software Systems. Artech House, 2002.
[34] C. Szyperski, Component Software: Beyond Object-Oriented Programming, second ed. Addison-Wesley, 2002.
[35] M. Vigder, M. Gentleman, and J. Dean, “COTS Software Integration: State of the Art,” Technical Report NRC No. 39190, 1996.
[36] V.R. Basili, M. Lindvall, I. Rus, C. Seaman, and B.W. Boehm, “Lessons-Learned Repository for COTS-Based SW Development,” Software Technology Newsletter, vol. 5, no. 3, pp. 4-7,, 2002.
[37] T.R. Madanmohan and R. De', “Open Source Reuse in Commercial Firms,” IEEE Software, vol. 21, no. 1, pp. 62-69, Jan./Feb. 2004.
[38] L. Bass, C. Buhman, S. Comella-Dorda, F. Long, J. Robert, R. Seacord, and K. Wallnau, “Volume I: Market Assessment of Component-Based Software Engineering,” Technical Report CMU/SEI-2001-TN-007, SEI, http:/, 2001.
[39] “Component-Based Software Engineering Network,” CBSEnet, http:/, 2004.
[40] C. Abts, B.W. Boehm, and E.B. Clark, “COCOTS: A COTS Software Integration Cost Model—Model Overview and Preliminary Data Findings,” Proc. 11th European Software Control and Metrics Conf., pp. 325-333, Apr. 2000.
[41] R. Likert, “A Technique for the Measurement of Attitudes,” Archives of Psychology, no. 140, pp. 5-55, 1932.
[42] J.R. Donald, V. Basili, B. Boehm, and B. Clark, “Eight Lessons Learned during COTS-Based Systems Maintenance,” IEEE Software, vol. 20, no. 5, pp. 94-96, Sept./Oct. 2003.
[43] Norwegian Census Bureau (SSB), Oslo, ICT Company Data,, 2002.
[44] E. Arisholm, D.I.K. Sjøberg, G.J. Carelius, and Y. Lindsjørn, “SESE—An Experiment Support Environment for Evaluating Software Engineering Technologies,” Proc. 10th Nordic Workshop Programming and Software Development Tools and Techniques, pp. 81-98, Aug. 2002.
[45] R. Conradi, J. Li, O.P.N. Slyngstad, C. Bunse, M. Torchiano, and M. Morisio, “Reflections on Conducting an International CBSE Survey in ICT Industry,” Proc. Fourth IEEE Int'l Symp. Empirical Software Eng., pp. 214-223, 2005.
[46] P. Spector, “Ratings of Equal and Unequal Response Choice Intervals,” J. Social Psychology, vol. 112, pp. 115-119, 1980.
[47] P.F. Velleman and L. Wilkinson, “Nominal, Ordinal, Interval, and Ratio Typologies Are Misleading,” J. Am. Statistician, vol. 47, no. 1, pp. 65-72, Feb. 1993.
[48] D.J. Hand, “Statistics and Theory of Measurement,” J. Royal Statistical Soc.: Series A (Statistics in Soc.), vol. 159, no. 3, pp. 445-492, 1996.
[49] B.H. Cohen, Explaining Psychological Statistics, second ed. John Wiley & Sons, 2000.
[50] Common Risks and Risk Reduction Actions for a COTS-Based System, filesCommonRisksCOTS.doc, 2005.
[51] V.R. Basili and B.W. Boehm, “COTS-Based Systems Top 10 List,” Computer, vol. 34, no. 5, pp. 91-93, May 2001.
[52] S. Lauesen, “COTS Tenders and Integration Requirements,” J.Requirements Eng., vol. 11, no. 2, pp. 111-122, 2006.
[53] J. Li, R. Conradi, O.P.N. Slyngstad, C. Bunse, U. Khan, M. Torchiano, and M. Morisio, “Validation of New Theses on Off-the-Shelf Component Based Development,” Proc. 11th IEEE Int'l Software Metrics Symp., (abstract), p. 26, Sept. 2005.
[54] J. Cohen, P. Cohen, S.G. West, and L.S. Aiken, Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences, third ed. Lawrence Erlbaum Assoc., 2002.
[55] P.M. Podsakoof and D.W. Organ, “Self-Reports in Organizational Research: Problems and Prospects,” J. Management, vol. 12, no. 4, pp. 531-544, 1986.
[56] S.J. Pocock, M.D. Hughes, and R.J. Lee, “Statistical Problems in Reporting of Clinical Trials: A Survey of Three Medical Journals,” The New England J. Medicine, vol. 317, no. 7, pp. 426-432, 1987.
8 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool