This Article 
 Bibliographic References 
 Add to: 
Profiling Deployed Software: Assessing Strategies and Testing Opportunities
April 2005 (vol. 31 no. 4)
pp. 312-327
An understanding of how software is employed in the field can yield many opportunities for quality improvements. Profiling released software can provide such an understanding. However, profiling released software is difficult due to the potentially large number of deployed sites that must be profiled, the transparency requirements at a user's site, and the remote data collection and deployment management process. Researchers have recently proposed various approaches to tap into the opportunities offered by profiling deployed systems and overcome those challenges. Initial studies have illustrated the application of these approaches and have shown their feasibility. Still, the proposed approaches, and the tradeoffs between overhead, accuracy, and potential benefits for the testing activity have been barely quantified. This paper aims to overcome those limitations. Our analysis of 1,200 user sessions on a 155 KLOC deployed system substantiates the ability of field data to support test suite improvements, assesses the efficiency of profiling techniques for released software, and the effectiveness of testing efforts that leverage profiled field data.

[1] M. Arnold and B. Ryder, “A Framework for Reducing the Cost of Instrumented Code,” Proc. Conf. Programming Language Design and Implementation, pp. 168-179, 2001.
[2] T. Ball and J. Larus, “Optimally Profiling and Tracing Programs,” ACM Trans. Programming Languages and Systems, vol. 16, no. 4, pp. 1319-1360, 1994.
[3] T. Ball and J. Laurus, “Optimally Profiling and Tracing Programs,” Proc. Ann. Symp. Principles of Programming, pp. 59-70, Aug. 1992.
[4] B. Calder, P. Feller, and A. Eustace, “Value Profiling,” Proc. Int'l Symp. Microarchitecture, pp. 259-269, Dec. 1997.
[5] W. Dickinson, D. Leon, and A. Podgurski, “Finding Failures by Cluster Analysis of Execution Profiles,” Proc. Int'l Conf. Software Eng., pp. 339-348, May 2001.
[6] S. Elbaum and M. Hardojo, “An Empirical Study of Profiling Strategies for Released Software and Their Impact on Testing Activities,” Proc. Int'l Symp. Software Testing and Analysis, pp. 65-75, July 2004.
[7] S. Elbaum, S. Kanduri, and A. Andrews, “Anomalies as Precursors of Field Failures,” Proc. Int'l Symp. Software Reliability Eng., pp. 108-118, 2003.
[8] S. Elbaum, A.G. Malishevsky, and G. Rothermel, “Test Case Prioritization: A Family of Empirical Studies,” IEEE Trans. Software Eng., vol. 28, no. 2, pp. 159-182, Feb. 2002.
[9] M. Ernst, J. Cockrell, W. Griswold, and D. Notkin, “Dynamically Discovering Likely Program Invariants to Support Program Evolution,” IEEE Trans. Software Eng., vol. 27, no. 2, pp. 99-123, Feb. 2001.
[10] A. Glenn, T. Ball, and J. Larus, “Exploiting Hardware Performance Counters with Flow and Context Sensitive Profiling,” ACM SIGPLAN Notices, vol. 32, no. 5, pp. 85-96, 1997.
[11] S. Graham and M. McKusick, “Gprof: A Call Graph Execution Profiler,” ACM SIGPLAN, vol. 17, no. 6, pp. 120-126, June 1982.
[12] K. Gross, S. McMaster, A. Porter, A. Urmanov, and L. Votta, “Proactive System Maintenance Using Software Telemetry,” Proc. Workshop Remote Analysis and Monitoring Software Systems, pp. 24-26, 2003.
[13] MIT Program Analysis Group, The Daikon Invariant Detector User Manual. doc daikon.html, 2005.
[14] M. Harrold, R. Lipton, and A. Orso, “Gamma: Continuous Evolution of Software After Deployment,” /Research/Projects gamma.html, 2003.
[15] D. Hilbert and D. Redmiles, “An Approach to Large-Scale Collection of Application Usage Data over the Internet,” Proc. Int'l Conf. Software Eng., pp. 136-145, 1998.
[16] D. Hilbert and D. Redmiles, “Separating the Wheat from the Chaff in Internet-Mediated User Feedback,” 1998.
[17] InCert, “Rapid Failure Recovery to Eliminate Application Downtime,”, June 2001.
[18] J. Bowring, A. Orso, and M. Harrold, “Monitoring Deployed Software Using Software Tomography,” Proc. Workshop Program Analysis for Software Tools and Eng., pp. 2-9, 2002.
[19] J. Bowring, J. Rehg, and M.J. Harrold, “Active Learning for Automatic Classification of Software Behavior,” Proc. Int'l Symp. Software Testing and Analysis, pp. 195-205, 2004.
[20] D. Leon, A. Podgurski, and L. White, “Multivariate Visualization in Observation-Based Testing,” Proc. Int'l Conf. Software Eng., pp. 116-125, May 2000.
[21] D. Libes, Exploring Expect: A Tcl-Based Toolkit for Automating Interactive Programs. Sebastopol, Calif.: O'Reilly and Associates, Inc., Nov. 1996.
[22] B. Liblit, A. Aiken, Z. Zheng, and M. Jordan, “Bug Isolation via Remote Program Sampling,” Proc. Conf. Programming Language Design and Implementation, pp. 141-154, June 2003.
[23] A. Memon, A. Porter, C. Yilmaz, A. Nagarajan, D.C. Schmidt, and B. Natarajan, “Skoll: Distributed Continuous Quality Assurance,” Proc. Int'l Conf. Software Eng., pp. 449-458, May 2004.
[24] J. Musa, Software Reliability Engineering. McGraw-Hill, 1999.
[25] Netscape, “Netscape Quality Feedback System,” home.netscape. com/communicator/navigator/ v4.5qfs1.html, 2000.
[26] Nielsen, “Nielsen Net Ratings: Nearly 40 Million Internet Users Connect Via Broadband,”, 2003.
[27] University of Washington, Pine Information Center, http://www.washington.edupine/, 2005.
[28] A. Orso, T. Apiwattanapong, and M.J. Harrold, “Leveraging Field Data for Impact Analysis and Regression Testing,” Foundations of Software Eng., pp. 128-137, Sept. 2003.
[29] A. Orso, D. Liang, M. Harrold, and R. Lipton, “Gamma System: Continuous Evolution of Software after Deployment,” Proc. Int'l Symp. Software Testing and Analysis, pp. 65-69, 2002.
[30] C. Pavlopoulou and M. Young, “Residual Test Coverage Monitoring,” Proc. Int'l Conf. Software Eng., pp. 277-284, May 1999.
[31] S. Reiss and M. Renieris, “Encoding Program Executions,” Proc. Int'l Conf. Software Eng., pp. 221-230, May 2001.
[32] D. Richardson, L. Clarke, L. Osterweil, and M. Young, “Perpetual Testing Project,” html , 1997.
[33] A. van der Hoek, R. Hall, D. Heimbigner, and A. Wolf, “Software Release Management,” Proc. European Software Eng. Conf., M. Jazayeri and H. Schauer, eds., pp. 159-175, 1997.
[34] C. Yilmaz, M.B. Cohen, and A. Porter, “Covering Arrays for Efficient Fault Characterization in Complex Configuration Spaces,” Proc. Int'l Symp. Software Testing and Analysis, pp. 45-54, July 2004.
[35] C. Yilmaz, A. Porter, and A. Schmidt, “Distributed Continuous Quality Assurance: The Skoll Project,” Proc. Workshop Remote Analysis and Monitoring Software Systems, pp. 16-19, 2003.

Index Terms:
Index Terms- Profiling, instrumentation, software deployment, testing, empirical studies.
Sebastian Elbaum, Madeline Diep, "Profiling Deployed Software: Assessing Strategies and Testing Opportunities," IEEE Transactions on Software Engineering, vol. 31, no. 4, pp. 312-327, April 2005, doi:10.1109/TSE.2005.50
Usage of this product signifies your acceptance of the Terms of Use.