Issue No.01 - January/February (2012 vol.9)
pp: 2-15
Casey M. Jeffery , University of Florida, Gainesville
Renato J.O. Figueiredo , University of Florida, Gainesville
There is an increasing need for fault tolerance capabilities in logic devices brought about by the scaling of transistors to ever smaller geometries. This paper presents a hypervisor-based replication approach that can be applied to commodity hardware to allow for virtually lockstepped execution. It offers many of the benefits of hardware-based lockstep while being cheaper and easier to implement and more flexible in the configurations supported. A novel form of processor state fingerprinting is also presented, which can significantly reduce the fault detection latency. This further improves reliability by triggering rollback recovery before errors are recorded to a checkpoint. The mechanisms are validated using a full prototype and the benchmarks considered indicate an average performance overhead of approximately 14 percent with the possibility for significant optimization. Finally, a unique method of using virtual lockstep for fault injection testing is presented and used to show that significant detection latency reduction is achievable by comparing only a small amount of data across replicas.
Virtualization, fault injection, dependable architectures, software reliability, autonomic computing.
Casey M. Jeffery, Renato J.O. Figueiredo, "A Flexible Approach to Improving System Reliability with Virtual Lockstep", IEEE Transactions on Dependable and Secure Computing, vol.9, no. 1, pp. 2-15, January/February 2012, doi:10.1109/TDSC.2010.53
[1] K. Adams and O. Agesen, "A Comparison of Software and Hardware Techniques for x86 Virtualization," Proc. 11th Int'l Conf. Architectural Support for Programming Languages and Operating Systems, Oct. 2006.
[2] N. Aggarwal, P. Ranganathan, N.P. Jouppi, and J.E. Smith, "Configurable Isolation: Building High Availability Systems with Commodity Multi-Core Processors," Proc. 33rd Int'l Symp. Computer Architecture, pp. 470-481, June 2007.
[3] J. Arlat, Y. Crouzet, J. Karlsson, P. Folkesson, E. Fuchs, and G.H. Leber, "Comparison of Physical and Software-Implemented Fault Injection Techniques," IEEE Trans. Computers, vol. 52, no. 9, pp. 1115-1133, Sept. 2003.
[4] J. Arlat, A. Costes, Y. Crouzet, J. Laprie, and D. Powell, "Fault Injection and Dependability Evaluation of Fault-Tolerant Systems," IEEE Trans. Computers, vol. 42, no. 8, pp. 913-923, Aug. 1993.
[5] T.M. Austin, "DIVA: A Reliable Substrate for Deep Submicron Microarchitecture Design," Proc. 32nd Ann. Int'l Symp. Microarchitecture, pp. 196-207, Nov. 1999.
[6] C. Basile, Z. Kalbarczyk, and R.K. Iyer, "Active Replication of Multithreaded Replicas," IEEE Trans. Parallel and Distributed Systems, vol. 17, no. 5, pp. 448-465, May 2006.
[7] R.C. Baumann, "Radiation-Induced Soft Errors in Advanced Semiconductor Technologies," IEEE Trans. Device and Materials Reliability, vol. 5, no. 3, pp. 305-316, Sept. 2005.
[8] D. Bernick, B. Bruckert, P.D. Vigna, D. Garcia, R. Jardine, J. Klecka, and J. Smullen, "NonStop® Advanced Architecture," Proc. Int'l Conf. Dependable Systems and Networks, June 2005.
[9] S. Borkar, "Designing Reliable Systems from Unreliable Components: The Challenges of Transistor Variability Degradation," IEEE Micro, vol. 25, no. 6, pp. 10-16, Dec. 2005.
[10] W.G. Bouricius, W.C. Carter, and P.R. Schneider, "Reliability Modeling Techniques for Self-Repairing Computer Systems," Proc. 24th Nat'l Conf. ACM, pp. 295-309, Aug. 1969.
[11] T.C. Bressoud and F.B. Schneider, "Hypervisor-Based Fault-Tolerance," ACM Trans. Computer Systems, vol. 14, no. 1, pp. 80-107, Feb. 1996.
[12] K. Buchacker and V. Sieh, "Framework for Testing the Fault-Tolerance of Systems Including OS and Network Aspects," Proc. Sixth Int'l Symp. High Assurance Systems Eng., pp. 95-105, Oct. 2001.
[13] J. Carreira, H. Madeira, and J.G. Silva, "Xception: A Technique for the Experimental Evaluation of Dependability in Modern Computers," IEEE Trans. Software Eng., vol. 24, no. 2, pp. 125-136, Feb. 1998.
[14] S. Chandra and P.M. Chen, "The Impact of Recovery Mechanisms on the Likelihood of Saving Corrupted State," Proc. 13th Int'l Symp. Software Reliability Eng., Nov. 2002.
[15] H. Chen, R. Chen, F. Zhang, B. Zang, and P. Yew, "Mercury: Combining Performance with Dependability Using Self-Virtualization," Proc. Int'l Conf. Parallel Processing, p. 9, Sept. 2007.
[16] C. Constantinescu, "Trends and Challenges in VLSI Circuit Reliability," IEEE Micro, vol. 23, no. 4, pp. 14-19, Aug. 2003.
[17] A.L. Cox, K. Mohanram, and S. Rixner, "Dependable ≠ Unaffordable," Proc. First Workshop Architectural and System Support for Improving Software Dependability, pp. 58-62, Oct. 2006.
[18] B. Cully, G. Lefebvre, D. Meyer, M. Feeley, N. Hutchinson, and A. Warfield, "Remus: High Availability via Asynchronous Virtual Machine Replication," Proc. Fifth USENIX Symp. Networked Systems Design and Implementation, Apr. 2008.
[19] D. Dubie, "Virtualization Infiltrates Midsize Companies," The New York Times, http:/, Nov. 2008.
[20] J.B. Dugan and K.S. Trivedi, "Coverage Modeling for Dependability Analysis of Fault-Tolerant Systems," IEEE Trans. Computers, vol. 38, no. 6, pp. 775-787, June 1989.
[21] G.W. Dunlap, D.G. Lucchetti, P.M. Chen, and M.A. Fetterman, "Execution Replay for Multiprocessor Virtual Machines," Proc. Int'l Conf. Virtual Execution Environments, Mar. 2008.
[22] P. Folkesson, S. Svensson, and J. Karlsson, "A Comparison of Simulation Based and Scan Chain Implemented Fault Injection," Proc. 28th Int'l Symp. Fault-Tolerant Computing, pp. 284-293, June 1998.
[23] B.T. Gold, J. Kim, J.C. Smolens, E.S. Chung, V. Liaskovitis, E. Nurvitadhi, B. Falsafi, J.C. Hoe, and A.G. Nowatzyk, "TRUSS: A Reliable, Scalable Server Architecture," IEEE Micro, vol. 25, no. 6, pp. 51-58, Dec. 2005.
[24] W. Gu, Z. Kalbarczyk, and R.K. Iyer, "Error Sensitivity of the Linux Kernel Executing on PowerPC G4 and Pentium 4 Processors," Proc. Int'l Conf. Dependable Systems and Networks, July 2004.
[25] R. Guerraoui and A. Schiper, "Software-Based Replication for Fault Tolerance," Computer, vol. 30, no. 4, pp. 68-74, Apr. 1997.
[26] D.R. Hower and M.D. Hill, "Rerun: Exploiting Episodes for Lightweight Memory Race Recording," Proc. 34th Int'l Symp. Computer Architecture, pp. 265-276, June 2008.
[27] Intel Corporation, Intel 64 and IA-32 Architectures Software Developer's Manual, vol. 3B, Nov. 2008.
[28] International Technology Roadmap for Semiconductors, 2007 ed. Semiconductor Industry Assoc., Int'l SEMATECH, 2007.
[29] C.M. Jeffery and R.J.O. Figueiredo, "Towards Byzantine Fault Tolerance in Many-Core Computing Platforms," Proc. 13th Pacific Rim Int'l Symp. Dependable Computing, Dec. 2007.
[30] C.M. Jeffery and R.J.O. Figueiredo, "Reducing Fault Detection Latencies in Virtually-Lockstepped Systems," IEEE Third Workshop Dependable Architectures, Nov. 2008.
[31] A. Kivity, Y. Kamay, D. Laor, U. Lublin, and A. Liguori, "kvm: The Linux Virtual Machine Monitor," Proc. Ninth Ottawa Linux Symp., June 2007.
[32] M. Le, A. Gallagher, and Y. Tamir, "Challenges and Opportunities with Fault Injection in Virtualized Systems," Proc. First Int'l Workshop Virtualization Performance: Analysis, Characterization, and Tools, Apr. 2008.
[33] D. Lee, B. Wester, K. Veeraraghavan, S. Narayanasamy, P.M. Chen, and J. Flinn, "Respec: Efficient Online Multiprocessor Replay via Speculation and External Determinism," Proc. 15th Int'l Conf. Architectural Support for Programming Languages and Operating Systems, pp. 77-90, Mar. 2010.
[34] M. Li, P. Ramachandran, S.K. Sahoo, S.V. Adve, V.S. Adve, and Y. Zhou, "Understanding the Propagation of Hard Errors to Software and Implications for Resilient System Design," Proc. 13th Int'l Conf. Architectural Support for Programming Languages and Operating Systems, Mar. 2008.
[35] S. Loveland, E.M. Dow, F. LeFevre, D. Beyer, and P.F. Chan, "Leveraging Virtualization to Optimize High-Availability System Configurations," IBM Systems J., vol. 47, no. 4, pp. 591-604, 2008.
[36] D. Lucchetti, S.K. Reinhardt, and P.M. Chen, "ExtraVirt: Detecting and Recovering from Transient Processor Faults," Proc. 20th Symp. Operating Systems Principles, pp. 1-8, Oct. 2005.
[37] Marathon Technologies, "Marathon everRun VM for Citrix XenServer," White Paper, Apr. 2009.
[38] J.-P. Martin and L. Alvisi, "Fast Byzantine Consensus," IEEE Trans. Dependable and Secure Computing, vol. 3, no. 3, pp. 202-215, Sept. 2006.
[39] C. McNairy and R. Bhatia, "Montecito: A Dual-Core, Dual-Threaded Itanium Processor," IEEE Micro, vol. 25, no. 2, pp. 10-20, Apr. 2005.
[40] S.S. Mukherjee, M. Kontz, and S.K. Reinhardt, "Detailed Design and Evaluation of Redundant Multithreading Alternatives," Proc. 29th Int'l Symp. Computer Architecture, May 2002.
[41] M. Portela-García, C. López-Ongil, M. García-Valderas, and L. Entrena, "A Rapid Fault Injection Approach for Measuring SEU Sensitivity in Complex Processors," Proc. 13th Int'l On-Line Testing Symp., pp. 101-106, July 2007.
[42] S. Potyra, V. Sieh, and M.D. Cin, "Evaluating Fault-Tolerant Systems Designs Using FAUmachine," Proc. Second Workshop Eng. Fault Tolerant Systems, Sept. 2007.
[43] M. Prvulovic, Z. Zhang, and J. Torrellas, "ReVive: Cost-effective Architectural Support for Rollback Recovery in Shared-Memory Multiprocessors," Proc. 29th Int'l Symp. Computer Architecture, pp. 111-122, May 2002.
[44] H.V. Ramasamy and M. Schunter, "Architecting Dependable Systems Using Virtualization," Proc. IEEE First Workshop Architecting Dependable Systems, June 2007.
[45] G.A. Reis, J. Chang, N. Vachharajani, R. Rangan, and D.I. August, "SWIFT: Software Implemented Fault Tolerance," Proc. Int'l Symp. Code Generation and Optimization, pp. 243-254, Mar. 2005.
[46] H.P. Reiser and R. Kapitza, "Hypervisor-Based Efficient Proactive Recovery," Proc. 26th Symp. Reliable Distributed Systems, Oct. 2007.
[47] G.P. Saggese, A. Vetteth, Z. Kalbarczyk, and R. Iyer, "Microprocessor Sensitivity to Failures: Control vs. Execution and Combinational vs. Sequential Logic," Proc. Int'l Conf. Dependable Systems and Networks, June 2005.
[48] D. Scales, "Fault Tolerant VMs in VMware Infrastructure: Operation and Best Practices," Proc. VMworld Conf., Sept. 2008.
[49] F.B. Schneider, "Implementing Fault-Tolerant Services Using the State Machine Approach: A Tutorial," ACM Computing Surveys, vol. 22, no. 4, pp. 299-319, Dec. 1990.
[50] R. Sedgewick, Algorithms in C. Addison-Wesley, 1997.
[51] P. Shivakumar, M. Kistler, S.W. Keckler, D. Burger, and L. Alvisi, "Modeling the Effect of Technology Trends on the Soft Error Rate of Combinational Logic," Proc. Int'l Conf. Dependable Systems and Networks, May 2002.
[52] A. Shye, T. Moseley, V.J. Reddi, J. Blomstedt, and D.A. Connors, "Using Process-Level Redundancy to Exploit Multiple Cores for Transient Fault Tolerance," Proc. Int'l Conf. Dependable Systems and Networks, June 2007.
[53] J.C. Smolens, B.T. Gold, J. Kim, B. Falsafi, J.C. Hoe, and A.G. Nowatzyk, "Fingerprinting: Bounding Soft-Error-Detection Latency and Bandwidth," IEEE Micro, vol. 24, no. 6, pp. 22-29, Nov. 2004.
[54] D.J. Sorin, M.K. Martin, M.D. Hill, and D.A. Wood, "SafetyNet: Improving the Availability of Shared Memory Multiprocessors with Global Checkpoint/Recovery," Proc. 29th Int'l Symp. Computer Architecture, pp. 123-134, May 2002.
[55] J. Srinivasan, S.V. Adve, P. Bose, and J.A. Rivers, "The Impact of Technology Scaling on Lifetime Reliability," Proc. Int'l Conf. Dependable Systems and Networks, July 2004.
[56] VMware "A Performance Study of Hypervisors," White Paper, Feb. 2007.
[57] VMware, "Protecting Mission-Critical Workloads with VMware Fault Tolerance," White Paper, May 2009.
[58] N.J. Wang and S.J. Patel, "ReStore: Symptom-Based Soft Error Detection in Microprocessors," IEEE Trans. Dependable and Secure Computing, vol. 3, no. 3, pp. 188-201, Sept. 2006.
[59] S. Webber and J. Beirne, "The Stratus Architecture," Proc. 21st Int'l Symp. Fault-Tolerant Computing, June 1991.
[60] M. Xu, V. Malyugin, J. Sheldon, G. Venkitachalam, and B. Weissman, "ReTrace: Collecting Execution Trace with Virtual Machine Deterministic Replay," Proc. Workshop Modeling, Benchmarking and Simulation, June 2007.