This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Empirical Analysis of Safety-Critical Anomalies During Operations
March 2004 (vol. 30 no. 3)
pp. 172-180
ASCII Text
x 
 
Robyn R. Lutz, In? Carmen Mikulski, "Empirical Analysis of Safety-Critical Anomalies During Operations," IEEE Transactions on Software Engineering, vol. 30, no. 3, pp. 172-180, March, 2004.
 
 
BibTex
x
 
@article{ 10.1109/TSE.2004.1271171,
author = {Robyn R. Lutz and In? Carmen Mikulski},
title = {Empirical Analysis of Safety-Critical Anomalies During Operations},
journal ={IEEE Transactions on Software Engineering},
volume = {30},
number = {3},
issn = {0098-5589},
year = {2004},
pages = {172-180},
doi = {http://doi.ieeecomputersociety.org/10.1109/TSE.2004.1271171},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
}
 
 
RefWorks Procite/RefMan/Endnote
x 
 
TY - JOUR
JO - IEEE Transactions on Software Engineering
TI - Empirical Analysis of Safety-Critical Anomalies During Operations
IS - 3
SN - 0098-5589
SP172
EP180
EPD - 172-180
A1 - Robyn R. Lutz,
A1 - In? Carmen Mikulski,
PY - 2004
KW - Software and system safety
KW - diagnostics
KW - maintenance process
KW - product metrics.
VL - 30
JA - IEEE Transactions on Software Engineering
ER -
 
Analysis of anomalies that occur during operations is an important means of improving the quality of current and future software. Although the benefits of anomaly analysis of operational software are widely recognized, there has been relatively little research on anomaly analysis of safety-critical systems. In particular, patterns of software anomaly data for operational, safety-critical systems are not well understood. This paper presents the results of a pilot study using Orthogonal Defect Classification (ODC) to analyze nearly two hundred such anomalies on seven spacecraft systems. These data show several unexpected classification patterns such as the causal role of difficulties accessing or delivering data, of hardware degradation, and of rare events. The anomalies often revealed latent software requirements that were essential for robust, correct operation of the system. The anomalies also caused changes to documentation and to operational procedures to prevent the same anomalous situations from recurring. Feedback from operational anomaly reports helped measure the accuracy of assumptions about operational profiles, identified unexpected dependencies among embedded software and their systems and environment, and indicated needed improvements to the software, the development process, and the operational procedures. The results indicate that, for long-lived, critical systems, analysis of the most severe anomalies can be a useful mechanism both for maintaining safer, deployed systems and for building safer, similar systems in the future.

[1] V.R. Basili, F. Shull, and F. Lanubile, Building Knowledge through Families of Experiments IEEE Trans. Software Eng., vol. 25, no. 4, pp. 456-473, July/Aug. 1999.
[2] K.H. Bennett and V.T. Rajlich, Software Maintenance and Evolution: A Roadmap The Future of Software Eng., A. Finkelstein, ed., 2000.
[3] P.G. Bishop and R.E. Bloomfield, Worst Case Reliability Prediction Based on a Prior Estimate of Residual Defects Proc. 13th Int'l Symp. Software Reliability Eng., pp. 295-303, 2002.
[4] D.N. Card, Learning from Our Mistakes with Defect Causal Analysis IEEE Software, pp. 56-63, Jan./Feb. 1998.
[5] R. Chillarege et al., "Orthogonal Defect Classification: A Concept for In-Process Measurements," IEEE Trans. Software Eng., Vol. 18, No. 11, Nov. 1992, pp. 943-956.
[6] R. Chillarege and K.A. Bassin, Software Triggers as a Function of Time ODC on Field Faults Proc. Fifth IFIP Working Conf. Dependable Computing for Critical Applications (DCCA-5), Sept. 1995.
[7] R. Chillarege, Orthogonal Defect Classification Handbook of Software Reliability Eng., M. Lyu, ed., pp. 359-400 1995.
[8] R. Chillarege and K.R. Prasad, Test and Development Process Retrospective A Case Study Using ODC Triggers Proc. Int'l Conf. Dependable Systems and Networks, pp. 669-678, 2002.
[9] Committee on Applied and Theoretical Statistics, Statistical Software Engineering, Nat'l Academy of Science, 1996.
[10] S. Dalal, M. Hamada, P. Matthews, and G. Patton, Using Defect Patterns to Uncover Opportunities for Improvement Proc. Int'l Conf. Applications of Software Measurement (ASM), 1999.
[11] K. El Emam and I. Wierczorek, The Repeatability of Code Defect Classifications Proc. Int'l Symp. Software Reliablity Eng. (ISSRE), 1998.
[12] N.O.E. Fenton and M. Neil, “A Critique of Software Defect Prediction Models,” IEEE Trans. Software Eng., vol. 25, no. 5, pp. 675-689, Sept./Oct. 1999.
[13] N.E. Fenton and S.L. Pfleeger, Software Metrics: A Rigorous and Practical Approach. second ed. PWS Publishing, 1997.
[14] R.L. Glass, Pilot Studies: What, Why, and How? J. Systems and Software, vol. 36, pp. 85-97, 1997.
[15] H. Hecht, “Rare Conditions—An Important Cause of Failures,” Proc. Ann. Conf. Computer Assurance, (COMPASS-93), pp. 81-85, 1993.
[16] R.K. Iyer and I. Lee, Measurement-Based Analysis of Software Reliability Handbook of Software Reliability Eng., M.R. Lyu, ed., pp. 303-358, 1995.
[17] Jet Propulsion Laboratory, ICAP Anomaly Process, Glossary Safety and Mission Assurance Information Systems, JPL, Pasadena, Calif., 1997.
[18] B. Kitchenham, Guidelines for Conducting and Evaluating Empirical Studies http://www.cs.utexas.edu/users/software/ 1999/Kitchenhamsld001.htm, 1999.
[19] S. Lauesen and O. Vinter, Preventing Requirements Defects: An Experiment in Process Improvement Requirements Eng. J., vol. 6, no. 1, pp. 37-50, Feb. 2001.
[20] M. Leszak, D.E. Perry, and D. Stoll, Classification and Evaluation of Defects in a Project Retrospective J. Systems and Software, vol. 61, pp. 173-187, Apr. 2002.
[21] N. Leveson, Safeware. Addison-Wesley 1995.
[22] R. Lutz, "Analyzing Software Requirements Errors in Safety-Critical, Embedded Systems," Proc. IEEE Int'l Symp. Requirements Eng., RE '93, pp. 126-133,San Diego, Calif., Jan. 1993.
[23] R. Lutz and I.C. Mikulski, Operational Anomalies as a Cause of Safety-Critical Requirements Evolution The J. Systems and Software, vol. 65, pp. 155-161, 2003.
[24] R. Lutz and I.C. Mikulski, "Requirements Discovery during the Testing of Safety-Critical Software," Proc. 25th Int'l Conf. Software Eng. (ICSE 03), IEEE CS Press, 2003, pp. 578-583.
[25] R. Lutz and I.C. Mikulski, Resolving Requirements Discovery in Testing and Operations Proc. Int'l. Requirements Eng. Conf., pp. 33-41, 2003.
[26] M.G. Mendonça and V.R. Basili, Validation of an Approach for Improving Existing Measurement Frameworks IEEE Trans. Software Eng., vol. 26, no. 6, pp. 484-499, June 2000.
[27] I. Myrtveit, E. Stensrud, and U.H. Olsson, Analyzing Data Sets with Missing Data: An Empirical Evaluation of Imputaton Methods and Likelihood-Based Methods IEEE Trans. Software Eng., vol. 27, no. 11, pp. 999-1013, Nov. 2001.
[28] T.J. Ostrand and E.J. Weyuker, The Distribution of Faults in a Large Industrial Software System Proc. Int'l. Symp. Software Testing and Analysis, pp. 55-64, 2002.
[29] B.C. Seaman, “Qualitative Methods in Empirical Studies of Software Engineering,” IEEE Trans. Software Eng., vol. 25, no. 4, pp. 557-572, July 1999.

Index Terms:
Software and system safety, diagnostics, maintenance process, product metrics.
Citation:
Robyn R. Lutz, In? Carmen Mikulski, "Empirical Analysis of Safety-Critical Anomalies During Operations," IEEE Transactions on Software Engineering, vol. 30, no. 3, pp. 172-180, March 2004, doi:10.1109/TSE.2004.1271171
Usage of this product signifies your acceptance of the Terms of Use.