The Community for Technology Leaders
RSS Icon
Issue No.12 - December (2011 vol.22)
pp: 1986-1999
Chen Yu , Sch. of Comput. Sci. & Technol., Huazhong Univ. of Sci. & Technol., Wuhan, China
Cheng-Zhong Xu , Dept. of Electr. & Comput. Eng., Wayne State Univ., Detroit, MI, USA
Xiaofei Liao , Sch. of Comput. Sci. & Technol., Huazhong Univ. of Sci. & Technol., Wuhan, China
Hai Jin , Sch. of Comput. Sci. & Technol., Huazhong Univ. of Sci. & Technol., Wuhan, China
Haikun Liu , Sch. of Comput. Sci. & Technol., Huazhong Univ. of Sci. & Technol., Wuhan, China
Live migration of virtual machines (VM) across physical hosts provides a significant new benefit for administrators of data centers and clusters. Previous memory-to-memory approaches demonstrate the effectiveness of live VM migration in local area networks (LAN), but they would cause a long period of downtime in a wide area network (WAN) environment. This paper describes the design and implementation of a novel approach, namely, CR/TR-Motion, which adopts checkpointing/recovery and trace/replay technologies to provide fast, transparent VM migration for both LAN and WAN environments. With execution trace logged on the source host, a synchronization algorithm is performed to orchestrate the running source and target VMs until they reach a consistent state. CR/TR-Motion can greatly reduce the migration downtime and network bandwidth consumption. Experimental results show that the approach can drastically reduce migration overheads compared with memory-to-memory approach in a LAN: up to 72.4 percent on application observed downtime, up to 31.5 percent on total migration time, and up to 95.9 percent on the data to synchronize the VM state. The application performance overhead due to migration is kept within 8.54 percent on average. The results also show that for a variety of workloads migrated across WANs, the migration downtime is less than 300 milliseconds.
wide area networks, checkpointing, computer centres, local area networks, virtual machines, network bandwidth consumption reduction, live virtual machine migration, asynchronous replication, state synchronization, data centers, data clusters, local area networks, wide area network, CR/TR-Motion, checkpointing technology, recovery technology, trace technology, replay technology, migration downtime reduction, Synchronization, Algorithm design and analysis, Wide area networks, Local area networks, Virtual machining, Checkpointing, virtual machine., Copy-on-write checkpointing, deterministic replay, live migration, trace
Chen Yu, Cheng-Zhong Xu, Xiaofei Liao, Hai Jin, Haikun Liu, "Live Virtual Machine Migration via Asynchronous Replication and State Synchronization", IEEE Transactions on Parallel & Distributed Systems, vol.22, no. 12, pp. 1986-1999, December 2011, doi:10.1109/TPDS.2011.86
[1] , 2011.
[2] unixbench, 2011.
[3] sheet.pdf , 2011.
[4] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield, "Xen and the Art of Virtualization," Proc. 19th ACM Symp. Operating Systems Principles (SOSP '03), pp. 164-177, Oct. 2003.
[5] G. Bronevetsky, R. Fernandes, D. Marques, K. Pingali, and P. Stodghill, "Recent Advances in Checkpoint / Recovery Systems," Proc. 20th Int'l Parallel and Distributed Processing Symp. (IPDPS '06), Apr. 2006.
[6] K. Buchacker and V. Sieh, "Framework for Testing the Fault-Tolerance of Systems Including OS and Network Aspects," Proc. Sixth IEEE Int'l Symp. High Assurance Systems Eng. (HASE '01), pp. 95-105, Oct. 2001.
[7] R. Bradford, E. Kotsovinos, A. Feldmann, and H. Schioeberg, "Live Wide-Area Migration of Virtual Machines Including Local Persistent State," Proc. Third Int'l Conf. Virtual Execution Environments (VEE '07), pp. 169-179, June 2007.
[8] C. Clark, K. Fraser, S. Hand, J.G. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield, "Live Migration of Virtual Machines," Proc. Second Conf. Symp. Networked Systems Design and Implementation (NSDI '05), pp. 273-286, May 2005.
[9] B. Cully, G. Lefebvre, D.T. Meyer, A. Karollil, M.J. Feeley, N.C. Hutchinson, and A. Warfield, "Remus: High Availability via Asynchronous Virtual Machine Replication," Proc. Fifth Symp. Networked Systems Design and Implementation (NSDI '08), Apr. 2008.
[10] G.W. Dunlap, S.T. King, S. Cinar, M. Basrai, and P.M. Chen, "ReVirt: Enabling Intrusion Analysis through Virtual-Machine Logging and Replay," Proc. Fifth Symp. Operating Systems Design and Implementation (OSDI '02), pp. 211-224, Dec. 2002.
[11] G.W. Dunlap, D. Lucchetti, M. Fetterman, and P. Chen, "Execution Replay on Multiprocessor Virtual Machines," Proc. ACM Int'l Conf. Virtual Execution Environments (VEE '08), pp. 121-130, Mar. 2008.
[12] R. Davoli, "VDE: Virtual Distributed Ethernet," Proc. First Int'l Conf. Testbeds and Research Infrastructures for the Development of Networks and Communities (Tridentcom '05), pp. 213-220, Feb. 2005.
[13] S. Fu and C.-Z. Xu, "Stochastic Modeling and Analysis of Hybrid Mobility in Reconfigurable Distributed Virtual Machines," J. Parallel and Distributed Computing, vol. 66, no. 11, pp. 1442-1454, Nov. 2006.
[14] W. Huang, Q. Gao, J. Liu, and D.K. Panda, "High Performance Virtual Machine Migration with RDMA over Modern Interconnects," Proc. IEEE Int'l Conf. Cluster Computing (Cluster '07), Sept. 2007.
[15] W. Huang, J. Liu, M. Koop, B. Abali, and D. Panda, "Nomad: Migrating OS-Bypass Networks in Virtual Machines," Proc. Third ACM/USENIX Conf. Virtual Execution Environment (VEE '07), June 2007.
[16] M.R. Hines and K. Gopalan, "Post-Copy Based Live Virtual Machine Migration Using Adaptive Pre-Paging and Dynamic Self-Ballooning," Proc. Fifth Int'l Conf. Virtual Execution Environments (VEE '09), pp. 51-60, Mar. 2009.
[17] S.T. King, G.W. Dunlap, and P.M. Chen, "Debugging Operating Systems with Time-Traveling Virtual Machines," Proc. USENIX Ann. Technical Conf., Apr. 2005.
[18] M. Kozuch and M. Satyanarayanan, "Internet Suspend/Resume," Proc. IEEE Workshop Mobile Computing Systems and Applications, June 2002.
[19] H.A. Lagar-Cavilla, N. Tolia, R. Balan, E. de Lara, M. Satyanarayanan, and D. O'Hallaron, "Dimorphic Computing," Technical Report CMU-CS-06-123, Carnegie Mellon Univ., Apr. 2006.
[20] Y. Luo, B. Zhang, X. Wang, Z. Wang, Y. Sun, and H. Chen, "Live and Incremental Whole-System Migration of Virtual Machines Using Block-Bitmap," Proc. IEEE Int'l Conf. Cluster Computing (Cluster '08), Sept./Oct. 2008.
[21] H. Liu, H. Jin, X. Liao, L. Hu, and C. Yu, "Live Migration of Virtual Machine Based on Full System Trace and Replay," Proc. 18th Int'l Symp. High Performance Distributed Computing (HPDC '09), pp. 101-110, June 2009.
[22] K.Z. Meth and J. Satran, "Design of the iSCSI Protocol," Proc. 20th IEEE/11th NASA Goddard Conf. Mass Storage Systems and Technologies (MSST '03), pp. 116-122, Apr. 2003.
[23] A.B. Nagarajan, F. Mueller, C. Engelmann, and S.L. Scott, "Proactive Fault Tolerance for HPC with Xen Virtualization," Proc. ACM Int'l Conf. Supercomputing (ICS '07), pp. 23-32, June 2007.
[24] M. Nelson, B.H. Lim, and G. Hutchins, "Fast Transparent Migration for Virtual Machines," Proc. USENIX Ann. Technical Conf., pp. 391-394, Apr. 2005.
[25] R. Nathuji and K. Schwan, "Virtual Power: Coordinated Power Management in Virtualized Enterprise Systems," Proc. ACM Symp. Operating Systems Principles (SOSP '07), Oct. 2007.
[26] D.A.S. de Oliveira, J.R. Crandall, G. Wassermann, S.F. Wu, Z. Su, and F.T. Chong, "ExecRecorder: VM-Based Full-System Replay for Attack Analysis and System Recovery," Proc. Workshop Architectural and System Support for Improving Software Dependability (ASID '06), pp. 66-71, Oct. 2006.
[27] C. Perkins, "IP Encapsulation within IP," RFC 2003, 1996.
[28] K.K. Ramakrishnan, P. Shenoy, and J. van der Merwe, "Live Data Center Migration across WANs: A Robust Cooperative Context Aware Approach," Proc. Second ACM Workshop Internet Network Management (INM '07), Aug. 2007.
[29] P. Ruth, J. Rhee, D. Xu, R. Kennell, and S. Goasguen, "Autonomic Live Adaptation of Virtual Computational Environments in a Multi-Domain Infrastructure," Proc. IEEE Int'l Conf. Autonomic Computing (ICAC '06), pp. 5-14, June 2006.
[30] C.P. Sapuntzakis, R. Chandra, B. Pfaff, J. Chow, M.S. Lam, and M. Rosenblum, "Optimizing the Migration of Virtual Computers," Proc. Fifth Symp. Operating Systems Design and Implementation (OSDI '02), Dec. 2002.
[31] A. Surie, H.A. Lagar-Cavilla, E. de Lara, and M. Satyanarayanan, "Low-Bandwidth VM Migration via Opportunistic Replay," Proc. Ninth Workshop Mobile Computing Systems and Applications (HotMobile '08), Feb. 2008.
[32] A. Sundararaj and P. Dinda, "Towards Virtual Networks for Virtual Machine Grid Computing," Proc. Third Conf. Virtual Machine Research and Technology Symp. (VM '04), pp. 177-190, May 2004.
[33] D.P. Scarpazza, P. Mullaney, O. Villa, F. Petrini, V. Tipparaju, D.M.L. Brown, and J. Nieplocha, "Transparent System-Level Migration of PGAS Applications Using Xen on InfiniBand," Proc. IEEE Int'l Conf. Cluster Computing (Cluster '07), pp. 74-83, Sept. 2007.
[34] F. Travostino, P. Daspit, L. Gommans, C. Jog, C. de Laat, J. Mambretti, I. Monga, B. van Oudenaarde, S. Raghunath, and P. Wang, "Seamless Live Migration of Virtual Machines over the MAN/WAN," Future Generation Computer Systems, vol. 22, no. 8, pp. 901-907, Oct. 2006.
[35] A. Whitaker, R.S. Cox, M. Shaw, and S.D. Gribble, "Constructing Services with Interposable Virtual Hardware," Proc. First Symp. Networked Systems Design and Implementation (NSDI '04), pp. 169-182, Mar. 2004.
[36] B. Wellington, "Secure DNS Dynamic Update," RFC 3007, 2000.
[37] M. Xu, V. Malyugin, J. Sheldon, G. Venkitachalam, and B. Weissman, "ReTrace: Collecting Execution Trace with Virtual Machine Deterministic Replay," Proc. Third Ann. Workshop Modeling, Benchmarking and Simulation, June 2007.
6 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool