The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.11 - November (2011 vol.22)
pp: 1788-1796
Yi-Hsuan Feng , National Tsing Hua University, Hsinchu
Nen-Fu Huang , National Tsing Hua University, Hsinchu
Yen-Min Wu , IBM, Taipei
ABSTRACT
Stateful stream process engines in high availability clusters (HACs) track a large number of concurrent flow states and replicate them to backups to provide reliable functionality. Under high traffic loads, existing solutions in such HACs are expensive owing to precise stateful replication. This work presents two novel methods to address this issue: randomization on replication representation and a replication scheme designed for when system becomes overloaded. A hashing structure called Multilevel Counting Bloom Filter (MLCBF) is proposed as a low resource-consuming solution of stateful replication. Its performance and tradeoffs are then evaluated based on theoretic analysis and extensive trace-based tests. Trace-based simulation reveals that MLCBF reduces network and memory requirements of replication typically by over 90 percent for URL categorization. Most importantly, MLCBF is quite as simple and practical for implementation and maintenance. Moreover, an adaptive scheme called dynamic lazy insertion is designed to prevent replication from overloading system continuously and optimize the throughput of HAC. Testbed evaluation demonstrates its feasibility and effectiveness in an overloaded HAC.
INDEX TERMS
Multiple hash functions, bloom filters, adaptive method, high availability, replication.
CITATION
Yi-Hsuan Feng, Nen-Fu Huang, Yen-Min Wu, "Efficient and Adaptive Stateful Replication for Stream Processing Engines in High-Availability Cluster", IEEE Transactions on Parallel & Distributed Systems, vol.22, no. 11, pp. 1788-1796, November 2011, doi:10.1109/TPDS.2011.83
REFERENCES
[1] M. Balazinska, H. Balakrishnan, S.R. Madden, and M. Stonebraker, "Fault-Tolerance in the Borealis Distributed Stream Processing System," ACM Trans. Database Systems, vol. 33, no. 1, pp. 1-44, 2008.
[2] L. Fan, P. Cao, J. Almeida, and A.Z. Broder, "Summary Cache: A Scalable Wide-Area Web Cache Sharing Protocol," IEEE/ACM Trans. Networking, vol. 8, no. 3, pp. 281-293, June 2000.
[3] A. Broder and M. Mitzenmacher, "Network Applications of Bloom Filter: A Survey," Allerton, vol. 1, no. 4, pp. 485-509, 2002.
[4] Y.-H. Feng, N.-F. Huang, and Y.-M. Wu, "Evaluation of TCP State Replication Methods in Cluster-Based Firewall," Proc. IEEE Globecom, 2008.
[5] A. Broder and M. Mitzenmacher, "Using Multiple Hash Functions to Improve IP Lookups," Proc. IEEE INFOCOM, 2001.
[6] F. Bonomi, M. Mitzenmacher, R. Panigrahy, S. Singh, and G. Varghese, "An Improved Construction for Counting Bloom Filters," Proc. 14th Ann. European Symp. Algorithms, pp. 684-695, 2006.
[7] F. Bonomi, M. Mitzenmacher, R. Panigraphy, S. Singh, and G. Varghese, "Beyond Bloom Filters: From Approximate Membership Checks to Approximate State Machines," Proc. ACM SIGCOMM, Sept. 2006.
[8] D. Ficara, S. Giordano, G. Procissi, and F. Vitucci, "Multilayer Compressed Counting Bloom Filters," Proc. IEEE INFOCOM, 2008.
[9] W. Shi, M.H. MacGregor, and P. Gburzynski, "Load Balancing for Parallel Forwarding," IEEE/ACM Trans. Networking, vol. 13, no. 4, pp. 790-801, Aug. 2005.
[10] N. Brownlee and K.C. Claffy, "Understanding Internet Traffic Stream: Dragonflies and Tortoises," IEEE Comm., vol. 40, no. 10, pp. 110-117, Oct. 2002.
[11] A. Shaikh, J. Rexford, and K.G. Shin, "Load-Sensitive Routing of Long-Lived IP Flows," Proc. ACM SIGCOMM, Sept. 1999.
[12] H. Kim, J.-H. Kim, I. Kang, and S. Bahk, "Preventing Session Table Explosion in Packet Inspection Computers," IEEE Trans. Computers, vol. 54, no. 2, pp. 238-240 , Feb. 2005.
[13] NLANR PMA Trace, http:/pma.nlanr.net/, 2011.
[14] D. Lee and N. Brownlee, "Passive Measurement of One-Way and Two-Way Flow Lifetimes," Proc. ACM SIGCOMM, 2007.
[15] P. Felber and P. Narasimhan, "Experiences, Strategies, and Challenges in Building Fault-Tolerant CORBA Systems," IEEE Trans. Computers, vol. 53, no. 5, pp. 497-511, May 2004.
[16] R. Zhang, T.F. Abdelzaher, and J.A. Stankovic, "Efficient TCP Connection Failover in Web Server Clusters," Proc. IEEE INFOCOM, 2004.
[17] A. Kirsch and M. Mitzenmacher, "Simple Summaries for Hashing with Choices," IEEE/ACM Trans. Networking, vol. 16, no. 1, pp. 218-231, Feb. 2008.
[18] A.Z. Broder and A.R. Karlin, "Multilevel Adaptive Hashing," Proc. ACM-SIAM SODA, pp. 43-53, 1990.
[19] S. Kumar, J. Turner, and P. Crowley, "Peacock Hashing: Deterministic and Updatable Hashing for High Performance Networking," Proc. IEEE INFOCOM, pp. 556-564, 2008.
[20] Y. Kanizo, D. Hay, and I. Keslassy, "Optimal Fast Hashing," Proc. IEEE INFOCOM, 2009.
[21] Y. Zhu, H. Jiang, J. Wang, and F. Xian, "HBA: Distributed Metadata Management for Large Cluster-Based Storage Systems," IEEE Trans. Parallel and Distributed Systems, vol. 19, no. 6, pp. 750-763, June 2008.
[22] A. Kumar, J. Xu, and E. Zegura, "Efficient and Scalable Query Routing for Unstructured Peer-to-Peer Networks," Proc. IEEE INFOCOM, 2005.
84 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool