This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Finding Atomicity-Violation Bugs through Unserializable Interleaving Testing
July-Aug. 2012 (vol. 38 no. 4)
pp. 844-860
ASCII Text
x 
 
Shan Lu, Soyeon Park, Yuanyuan Zhou, "Finding Atomicity-Violation Bugs through Unserializable Interleaving Testing," IEEE Transactions on Software Engineering, vol. 38, no. 4, pp. 844-860, July-Aug., 2012.
 
 
BibTex
x
 
@article{ 10.1109/TSE.2011.35,
author = {Shan Lu and Soyeon Park and Yuanyuan Zhou},
title = {Finding Atomicity-Violation Bugs through Unserializable Interleaving Testing},
journal ={IEEE Transactions on Software Engineering},
volume = {38},
number = {4},
issn = {0098-5589},
year = {2012},
pages = {844-860},
doi = {http://doi.ieeecomputersociety.org/10.1109/TSE.2011.35},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
}
 
 
RefWorks Procite/RefMan/Endnote
x 
 
TY - JOUR
JO - IEEE Transactions on Software Engineering
TI - Finding Atomicity-Violation Bugs through Unserializable Interleaving Testing
IS - 4
SN - 0098-5589
SP844
EP860
EPD - 844-860
A1 - Shan Lu,
A1 - Soyeon Park,
A1 - Yuanyuan Zhou,
PY - 2012
KW - Testing and debugging
KW - debugging aids
KW - diagnostics
KW - testing strategies
KW - test coverage of code
KW - concurrent programming
KW - bug characteristics
VL - 38
JA - IEEE Transactions on Software Engineering
ER -
 
Shan Lu, University of Wisconsin-Madison, Madison
Soyeon Park, University of California, San Diego, La Jolla
Yuanyuan Zhou, University of California, San Diego, La Jolla
Multicore hardware is making concurrent programs pervasive. Unfortunately, concurrent programs are prone to bugs. Among different types of concurrency bugs, atomicity violations are common and important. How to test the interleaving space and expose atomicity-violation bugs is an open problem. This paper makes three contributions. First, it designs and evaluates a hierarchy of four interleaving coverage criteria using 105 real-world concurrency bugs. This study finds a coverage criterion (Unserializable Interleaving Coverage) that balances the complexity and the capability of exposing atomicity-violation bugs well. Second, it studies stress testing to understand why this common practice cannot effectively expose atomicity-violation bugs from the perspective of unserializable interleaving coverage. Third, it designs CTrigger following the unserializable interleaving coverage criterion. CTrigger uses trace analysis to identify feasible unserializable interleavings, and then exercises low-probability interleavings to expose atomicity-violation bugs. We evaluate CTrigger with real-world atomicity-violation bugs from seven applications. CTrigger efficiently exposes these bugs within 1-235 seconds, two to four orders of magnitude faster than stress testing. Without CTrigger, some of these bugs do not manifest even after seven days of stress testing. Furthermore, once a bug is exposed, CTrigger can reliably reproduce it, usually within 5 seconds, for diagnosis.

[1] C. Artho, K. Havelund, and A. Bierre, "High-Level Data Races," Proc. First Int'l Workshop Verification and Validation of Enterprise Information Systems, 2003.
[2] P. Barford and M. Crovella, "Generating Representative Web Workloads for Network and Server Performance Evaluation," Proc. ACM SIGMETRICS Joint Int'l Conf. Measurement and Modeling of Computer Systems, June 1998.
[3] B. Beizer, Software Testing Techniques, second ed. Van Nostrand Reinhold, 1990.
[4] A. Bron, E. Farchi, Y. Magid, Y. Nir, and S. Ur, "Applications of Synchronization Coverage," Proc. ACM SIGPLAN Symp. Principles and Practice of Parallel Programming, 2005.
[5] G. Dunlap, D. Lucchetti, M. Fetterman, and P. Chen, "Execution Replay of Multiprocessor Virtual Machines," Proc. ACM SIGPLAN/SIGOPS Int'l Conf. Virtual Execution Environment, 2008.
[6] O. Edelstein, E. Farchi, Y. Nir, G. Ratsaby, and S. Ur, "Multi-Threaded Java Program Test Generation," IBM Systems J., vol. 41, pp. 111-125, 2002.
[7] C. Flanagan and S.N. Freund, "Atomizer: A Dynamic Atomicity Checker for Multithreaded Programs," Proc. ACM SIGPLAN-SIGACT Symp. Principles of Programming Languages, 2004.
[8] C. Flanagan, S.N. Freund, and J. Yi, "Velodrome: A Sound and Complete Dynamic Atomicity Checker for Multithreaded Programs," Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, 2008.
[9] P.G. Frankl and E.J. Weyuker, "An Applicable Family of Data Flow Testing Criteria," IEEE Trans. Software Eng., vol. 14, no. 10, pp. 1483-1498, Oct. 1988.
[10] C. Hammer, J. Dolby, M. Vaziri, and F. Tip, "Dynamic Detection of Atomic-Set-Serializability Violations," Proc. Int'l Conf. Software Eng., 2008.
[11] M.J. Harrold and B.A. Malloy, "Data Flow Testing of Parallelized Code," Proc. Int'l Conf. Software Maintenance, 1992.
[12] G.-H. Hwang, K.C. Tai, and T.L. Huang, "Reachability Testing: An Approach to Testing Concurrent Software," Int'l J. Software Eng. and Knowledge Eng., vol. 5, pp. 493-510, 1995.
[13] S.T. King, G.W. Dunlap, and P.M. Chen, "Operating Systems with Time-Traveling Virtual Machines," Proc. USENIX Ann. Technical Conf., 2005.
[14] P.V. Koppol and K.-C. Tai, "An Incremental Approach to Structural Testing of Concurrent Software," Proc. ACM SIGSOFT Int'l Symp. Software Testing and Analysis, 1996.
[15] T.J. LeBlanc and J.M. Mellor-Crummey, "Debugging Parallel Programs with Instant Replay," IEEE Trans. Computers, vol. 36, no. 4, pp. 471-482, Apr. 1987.
[16] Y. Lei and R.H. Carver, "Reachability Testing of Concurrent Programs," IEEE Trans. Software Eng., vol. 32, no. 6, pp. 382-403, June 2006.
[17] Y. Lei, R.H. Carver, R. Kacker, and D. Kung, "A Combinatorial Testing Strategy for Concurrent Programs," Software Testing, Verification and Reliability, vol. 17, no. 4, pp. 207-225, 2007.
[18] Y. Lei and E. Wong, "A Novel Framework for Non-Deterministic Testing of Message-Passing Programs," Proc. Ninth IEEE Int'l Symp. High-Assurance Systems Eng., pp. 66-75, 2005.
[19] S. Lu, W. Jiang, and Y. Zhou, "A Study of Interleaving Coverage Criteria," Proc. Joint Meeting European Software Eng. Conf. and ACM SIGSOFT Symp. Foundations of Software Eng., 2007.
[20] S. Lu, S. Park, C. Hu, X. Ma, W. Jiang, Z. Li, R.A. Popa, and Y. Zhou, "Muvi: Automatically Inferring Multi-Variable Access Correlations and Detecting Related Semantic and Concurrency Bugs," Proc. ACM SIGOPS Symp. Operating Systems Principles, 2007.
[21] S. Lu, S. Park, E. Seo, and Y. Zhou, "Learning from Mistakes—A Comprehensive Study of Real World Concurrency Bug Characteristics," Proc. Int'l Conf. Architectural Support for Programming Languages and Operating Systems, 2008.
[22] S. Lu, J. Tucek, F. Qin, and Y. Zhou, "Avio: Detecting Atomicity Violations via Access Interleaving Invariants," Proc. Int'l Conf. Architectural Support for Programming Languages and Operating Systems, 2006.
[23] B. Lucia, J. Devietti, K. Strauss, and L. Ceze, "Atom-Aid: Detecting and Surviving Atomicity Violations," Proc. Ann. Int'l Symp. Computer Architecture, 2008.
[24] C.-K. Luk, R. Cohn, R. Muth, H. Patil, A. Klauser, G. Lowney, S. Wallace, V.J. Reddi, and K. Hazelwood, "Pin: Building Customized Program Analysis Tools with Dynamic Instrumentation," Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, 2005.
[25] Y. Malaiya, N. Li, R. Karcich, and B. Skbbe, "The Relationship between Test Coverage and Reliability," Proc. Int'l Symp. Software Reliability Eng., 1994.
[26] M. Musuvathi and S. Qadeer, "Iterative Context Bounding for Systematic Testing of Multithreaded Programs," Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, 2007.
[27] M. Musuvathi, S. Qadeer, T. Ball, and G. Basler, "Finding and Reproducing Heisenbugs in Concurrent Programs," Proc. Conf. Operating Systems Design and Implementation, 2008.
[28] S. Narayanasamy, C. Pereira, H. Patil, R. Cohn, and B. Calder, "Automatic Logging of Operating System Effects to Guide Application-Level Architecture Simulation," Proc. Joint Int'l Conf. Measurement and Modeling of Computer Systems, 2006.
[29] S. Narayanasamy, Z. Wang, J. Tigani, A. Edwards, and B. Calder, "Automatically Classifying Benign and Harmful Data Racesallusing Replay Analysis," Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, 2007.
[30] N. Nethercote and J. Seward, "Valgrind: A Framework for Heavyweight Dynamic Binary Instrumentation," Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, 2007.
[31] R.H.B. Netzer and B.P. Miller, "Improving the Accuracy of Data Race Detection," Proc. ACM SIGPLAN Symp. Principles and Practice of Parallel Programming, 1991.
[32] S. Park, S. Lu, and Y. Zhou, "CTrigger: Exposing Atomicity Violation Bugs from Their Hiding Places," Proc. Int'l Conf. Architectural Support for Programming Languages and Operating Systems, 2009.
[33] S. Savage, M. Burrows, G. Nelson, P. Sobalvarro, and T. Anderson, "Eraser: A Dynamic Data Race Detector for Multithreaded Programs," ACM Trans. Computer Systems, vol. 15, pp. 391-411, 1997.
[34] SecurityFocus "Software Bug Contributed to Blackout," http://www.securityfocus.com/news8016, 2011.
[35] K. Sen, "Race Directed Random Testing of Concurrent Programs," Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, 2008.
[36] K. Sen and G. Agha, "Automated Systematic Testing of Open Distributed Programs," Proc. Fundamental Approaches to Software Eng., 2006.
[37] S.D. Stoller, "Testing Concurrent Java Programs Using Randomized Scheduling," Electronic Notes in Theoretical Computer Science, vol. 70, no. 4, pp. 142-157, July 2002.
[38] R.N. Taylor, D.L. Levine, and C.D. Kelly, "Structural Testing of Concurrent Programs," IEEE Trans. Software Eng., vol. 18, no. 3, pp. 206-215, Mar. 1992.
[39] M. Vaziri, F. Tip, and J. Dolby, "Associating Synchronization Constraints with Data in an Object-Oriented Language," Proc. ACM SIGPLAN-SIGACT Symp. Principles of Programming Languages, 2006.
[40] VMware, "(Appendix c) Using the Integrated Virtual Debugger for Visual Studio," http://www.vmware.com/pdfws65_ manual.pdf , 2011.
[41] L. Wang and S.D. Stoller, "Accurate and Efficient Runtime Detection of Atomicity Errors in Concurrent Programs," Proc. ACM SIGPLAN Symp. Principles and Practice of Parallel Programming, 2006.
[42] L. Wang and S.D. Stoller, "Runtime Analysis of Atomicity for Multithreaded Programs," IEEE Trans. Software Eng., vol. 32, no. 2, pp. 93-110, Feb. 2006.
[43] S.N. Weiss, "A Formal Framework for the Study of Concurrent Program Testing," Proc. Second Workshop Software Testing, Verification, and Analysis, 1988.
[44] E.J. Weyuker, "More Experience with Data Flow Testing," IEEE Trans. Software Eng., vol. 19, no. 9, pp. 912-919, Sept. 1993.
[45] S.C. Woo, M. Ohara, E. Torrie, J.P. Singh, and A. Gupta, "The SPLASH-2 Programs: Characterization and Methodological Considerations," Proc. Ann. Int'l Symp. Computer Architecture, 1995.
[46] M. Xu, R. Bodík, and M.D. Hill, "A Serializability Violation Detector for Shared-Memory Server Programs," Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, 2005.
[47] C.-S.D. Yang, A.L. Souter, and L.L. Pollock, "All-Du-Path Coverage for Parallel Programs," Proc. ACM SIGSOFT Int'l Symp. Software Testing and Analysis, 1998.
[48] Y. Yu, T. Rodeheffer, and W. Chen, "RaceTrack: Efficient Detection of Data Race Conditions via Adaptive Tracking," Proc. ACM Symp. Operating Systems Principles, 2005.

Index Terms:
Testing and debugging, debugging aids, diagnostics, testing strategies, test coverage of code, concurrent programming, bug characteristics
Citation:
Shan Lu, Soyeon Park, Yuanyuan Zhou, "Finding Atomicity-Violation Bugs through Unserializable Interleaving Testing," IEEE Transactions on Software Engineering, vol. 38, no. 4, pp. 844-860, July-Aug. 2012, doi:10.1109/TSE.2011.35
Usage of this product signifies your acceptance of the Terms of Use.