This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
How Accurate is Scientific Software?
October 1994 (vol. 20 no. 10)
pp. 785-797

This paper describes some results of what, to the authors' knowledge, is the largest N-version programming experiment ever performed. The object of this ongoing four-year study is to attempt to determine just how consistent the results of scientific computation really are, and, from this, to estimate accuracy. The experiment is being carried out in a branch of the earth sciences known as seismic data processing, where 15 or so independently developed large commercial packages that implement mathematical algorithms from the same or similar published specifications in the same programming language (Fortran) have been developed over the last 20 years. The results of processing the same input dataset, using the same user-specified parameters, for nine of these packages is reported in this paper. Finally, feedback of obvious flaws was attempted to reduce the overall disagreement. The results are deeply disturbing. Whereas scientists like to think that their code is accurate to the precision of the arithmetic used, in this study, numerical disagreement grows at around the rate of 1% in average absolute difference per 4000 fines of implemented code, and, even worse, the nature of the disagreement is nonrandom. Furthermore, the seismic data processing industry has better than average quality standards for its software development with both identifiable quality assurance functions and substantial test datasets.

[1] L. Hatton,The Quality and Reliability of Scientific Software in Software Quality Management, Southampton, UK: Computation Mechanics Publications, Elsevier, 1993.
[2] P. G. Neumann, "Risks to the public in computers and related systems,"Commun. ACM Software Eng. Notes, 1993.
[3] J. F. Claerbout,Imaging the Earth's Interior. Oxford, UK: Blackwell Scientific Publications, 1985, p. 398.
[4] L. Hatton, M. H. Worthington, and J. Makin,Seismic Data Processing: Theory and Practice. Oxford, UK: Blackwell Scientific Publications, 1986, p. 177.
[5] O. Yilmaz,Seismic Data Processing, Investigations in Geophysics, Tulsa, OK: Society of Exploration Geophysicists, 1987, p. 526.
[6] L. Hatton,Safer C: Developing in High-Integrity and Safety-Critical Systems. New York: McGraw-Hill, 1994.
[7] L. Hatton, A. Wright, S. Smith, G. Parkes, P. Bennett, and R. Laws, "The seismic kernel system: A large-scale exercise in Fortran 77 portability,"Software Practice and Experience, vol. 18, pp. 301-329, 1988.
[8] T. W. Keller, "Archieving error-free man-rated software," in2nd Int. Software Testing, Anal. Rev. Conf., 1993, Monterey, CA, USA.
[9] J. F. Lindeberg, "The Swedish State Railways' experience withn- version programmed systems," inDirections in Safety-Critical Systems, Bristol, UK: Springer-Verlag, 1993.
[10] A. J. McLeod, "Performance evaluation of normal distribution software,"Statistics and Computing, vol. 2, p. 1-5, 1992.
[11] E. A. Robinson and S. Treitel,Geophysical Signal Analysis. Englewood Cliffs, NJ: Prentice-Hall, 1980, p. 466.
[12] I. D. Hale,Dip Moveout Processing, course notes series, vol. 4, Tulsa, OK: Society of Exploration Geophysicists, 1990.
[13] K. M. Barry, D. A. Cavers, and C. W. Kneale, "Recommended standards for digital tape formats,"Geophys., vol. 40, no. 2, pp. 344-352, 1975.
[14] D. F. Swayne, D. Cook, and A. Buja,User's Manual for XGobi: A Dynamic Graphics Program for Data Analysis Implemented in the X Window System. Piscataway, NJ: Bellcore, 1991.
[15] A. Koenig,C Traps and Pitfalls. Reading, MA: Addison-Wesley, 1988, p. 146.
[16] J. W. Tukey,Exploratory Data Analysis, Behavioural Science: Quantitative Methods. Reading, MA: Addison-Wesley, 1977, p. 688.
[17] L. Hatton and A. Roberts,Analysing the Agreement Between Seismic Software Packages: A Seismic Software Calibration Experiment in 62nd S.E.G.New Orleans, LA: Society of Exploration Geophysicists, 1992.

Index Terms:
programming; seismology; geophysics computing; software packages; software quality; scientific software; N-version programming experiment; scientific computation; seismic data processing; large commercial packages; mathematical algorithms; programming language; Fortran; input dataset; seismic data processing industry; quality standards; software development; quality assurance
Citation:
L. Hatton, A. Roberts, "How Accurate is Scientific Software?," IEEE Transactions on Software Engineering, vol. 20, no. 10, pp. 785-797, Oct. 1994, doi:10.1109/32.328993
Usage of this product signifies your acceptance of the Terms of Use.