Issue No.07 - July (2009 vol.20)
pp: 968-982
Thomas Repantis , University of California, Riverside, Riverside
Xiaohui Gu , North Carolina State University, Raleigh
Vana Kalogeraki , University of California, Riverside, Riverside
Many emerging online data analysis applications require applying continuous query operations such as correlation, aggregation, and filtering to data streams in real time. Distributed stream processing systems allow in-network stream processing to achieve better scalability and quality-of-service (QoS) provision. In this paper, we present Synergy, a novel distributed stream processing middleware that provides automatic sharing-aware component composition capability. Synergy enables efficient reuse of both result streams and processing components, while composing distributed stream processing applications with QoS demands. It provides a set of fully distributed algorithms to discover and evaluate the reusability of available result streams and processing components when instantiating new stream applications. Specifically, Synergy performs QoS impact projection to examine whether the shared processing can cause QoS violations on currently running applications. The QoS impact projection algorithm can handle different types of streams including both regular traffic and bursty traffic. If no existing processing components can be reused, Synergy dynamically deploys new components at strategic locations to satisfy new application requests. We have implemented a prototype of the Synergy middleware and evaluated its performance on both PlanetLab and simulation testbeds. The experimental results show that Synergy can achieve much better resource utilization and QoS provisioning than previously proposed schemes, by judiciously sharing streams and components during application composition.
Distributed stream processing, component composition, shared processing, QoS, resource management.
Thomas Repantis, Xiaohui Gu, Vana Kalogeraki, "QoS-Aware Shared Component Composition for Distributed Stream Processing Systems", IEEE Transactions on Parallel & Distributed Systems, vol.20, no. 7, pp. 968-982, July 2009, doi:10.1109/TPDS.2008.165
[1] S. Chandrasekaran et al., “TelegraphCQ: Continuous Dataflow Processing for an Uncertain World,” Proc. First Biennial Conf. Innovative Data Systems Research (CIDR '03), Jan. 2003.
[2] R. Motwani et al., “Query Processing, Resource Management, and Approximation in a Data Stream Management System,” Proc. First Biennial Conf. Innovative Data Systems Research (CIDR '03), Jan. 2003.
[3] D. Abadi et al., “The Design of the Borealis Stream Processing Engine,” Proc. Second Biennial Conf. Innovative Data Systems Research (CIDR '05), Jan. 2005.
[4] L. Chen, K. Reddy, and G. Agrawal, “GATES: A Grid-Based Middleware for Distributed Processing of Data Streams,” Proc. 13th Int'l Symp. High-Performance Distributed Computing (HPDC -13 '04), June 2004.
[5] X. Gu, P. Yu, and K. Nahrstedt, “Optimal Component Composition for Scalable Stream Processing,” Proc. 25th Int'l Conf. Distributed Computing Systems (ICDCS '05), June 2005.
[6] V. Kumar, B. Cooper, Z. Cai, G. Eisenhauer, and K. Schwan, “Resource-Aware Distributed Stream Management Using DynamicOverlays,” Proc. 25th Int'l Conf. Distributed Computing Systems (ICDCS '05), June 2005.
[7] P. Pietzuch, J. Ledlie, J. Shneidman, M. Roussopoulos, M. Welsh, and M. Seltzer, “Network-Aware Operator Placement for Stream-Processing Systems,” Proc. 22nd Int'l Conf. Data Eng. (ICDE '06), Apr. 2006.
[8] L. Amini, N. Jain, A. Sehgal, J. Silber, and O. Verscheure, “Adaptive Control of Extreme-Scale Stream Processing Systems,” Proc. 26th Int'l Conf. Distributed Computing Systems (ICDCS '06), July 2006.
[9] K.L. Wu et al., “Challenges and Experience in Prototyping a Multi-Modal Stream Analytic and Monitoring Application on System S,” Proc. 33rd Int'l Conf. Very Large Data Bases (VLDB '07), Sept. 2007.
[10] X. Gu and K. Nahrstedt, “On Composing Stream Applications in Peer-to-Peer Environments,” IEEE Trans. Parallel and Distributed Systems, vol. 17, no. 8, pp. 824-837, Aug. 2006.
[11] M. Hammad, M. Franklin, W. Aref, and A. Elmagarmid, “Scheduling for Shared Window Joins over Data Streams,” Proc. 29th Int'l Conf. Very Large Data Bases (VLDB '03), Sept. 2003.
[12] A. Bavier et al., “Operating System Support for Planetary-Scale Network Services,” Proc. First Symp. Networked Systems Design and Implementation (NSDI '04), Mar. 2004.
[13] T. Abdelzaher, “An Automated Profiling Subsystem for QoS-Aware Services,” Proc. Sixth IEEE Real Time Technology and Applications Symp. (RTAS '00), June 2000.
[14] X. Gu and K. Nahrstedt, “Distributed Multimedia Service Composition with Statistical QoS Assurances,” IEEE Trans. Multimedia, vol. 8, no. 1, pp. 141-151, Feb. 2006.
[15] A. Rowstron and P. Druschel, “Pastry: Scalable, Distributed Object Location and Routing for Large-Scale Peer-to-Peer Systems,” Proc. IFIP/ACM Int'l Middleware Conf. (Middleware '01), Nov. 2001.
[16] F. Chen, T. Repantis, and V. Kalogeraki, “Coordinated Media Streaming and Transcoding in Peer-to-Peer Systems,” Proc. 19th Int'l Parallel and Distributed Processing Symp. (IPDPS '05), Apr. 2005.
[17] J. Ledlie, P. Gardner, and M. Seltzer, “Network Coordinates in the Wild,” Proc. Fourth Symp. Networked Systems Design and Implementation (NSDI '07), Apr. 2007.
[18] Y. Ahmad and U. Çetintemel, “Network-Aware Query Processing for Stream-Based Applications,” Proc. 30th Int'l Conf. Very Large Data Bases (VLDB '04), Aug. 2004.
[19] Y. Wei, V. Prasad, S. Son, and J. Stankovic, “Prediction-Based QoS Management for Real-Time Data Streams,” Proc. 27th IEEE Real-Time Systems Symp. (RTSS '06), Dec. 2006.
[20] L. Kleinrock, Queueing Systems. Volume 1: Theory. John Wiley & Sons, 1975.
[21] N. Antunes, C. Fricker, F. Guillemin, and P. Robert, “Integration of Streaming Services and TCP Data Transmission in the Internet,” Elsevier Performance Evaluation, vol. 62, nos. 1-4, pp.263-277, Oct. 2005.
[22] A. Markopoulou, F. Tobagi, and M. Karam, “Assessing the Quality of Voice Communications over Internet Backbones,” IEEE/ACM Trans. Networking, vol. 11, no. 5, pp. 747-760, Oct. 2003.
[23] X. Gu, Z. Wen, and P. Yu, “BridgeNet: An Adaptive Multi-Source Stream Dissemination Service Overlay,” Proc. IEEE INFOCOM '07, May 2007.
[24] L. Cherkasova and M. Gupta, “Analysis of Enterprise Media Server Workloads: Access Patterns, Locality, Content Evolution, and Rates of Change,” IEEE/ACM Trans. Networking, vol. 12, no. 5, pp. 781-794, Oct. 2004.
[25] E. Zegura, K. Calvert, and S. Bhattacharjee, “How to Model an Internetwork,” Proc. IEEE INFOCOM '96, Mar. 1996.
[26] T. Repantis, X. Gu, and V. Kalogeraki, “Synergy: Sharing-Aware Component Composition for Distributed Stream Processing Systems,” Proc. Seventh ACM/IFIP/USENIX Int'l Middleware Conf. (Middleware '06), Nov. 2006.
[27] Stream Query Repository: Network Traffic Management, , 2008.
[28] The Internet Traffic Archive,, 2008.
[29] UC Berkeley Sonoma Dust, , 2008.
[30] B. Gedik and L. Liu, “PeerCQ: A Decentralized and Self-Configuring Peer-to-Peer Information Monitoring System,” Proc.23rd Int'l Conf. Distributed Computing Systems (ICDCS '03), May 2003.
[31] D. Menasce, “Composing Web Services: A QoS View,” IEEE Internet Computing, vol. 8, no. 6, pp. 88-90, Nov./Dec. 2004.
[32] T. Yu, Y. Zhang, and K. Lin, “Efficient Algorithms for Web Services Selection with End-to-End QoS Constraints,” ACM Trans. Web, vol. 1, no. 1, pp. 1-26, May 2007.
[33] L. Zeng, B. Benatallah, A. Ngu, M. Dumas, J. Kalagnanam, and H. Chang, “QoS-Aware Middleware for Web Services Composition,” IEEE Trans. Software Eng., vol. 30, no. 5, pp. 311-327, May 2004.
[34] T. Repantis and V. Kalogeraki, “Hot-Spot Prediction and Alleviation in Distributed Stream Processing Applications,” Proc. 38th Int'l Conf. Dependable Systems and Networks (DSN '08), June 2008.
[35] X. Gu, S. Papadimitriou, P.S. Yu, and S.P. Chang, “Toward Predictive Failure Management for Distributed Stream Processing Systems,” Proc. 28th Int'l Conf. Distributed Computing Systems (ICDCS '08), June 2008.
[36] T. Repantis and V. Kalogeraki, “Replica Placement for High Availability in Distributed Stream Processing Systems,” Proc. Second Int'l Conf. Distributed Event-Based Systems (DEBS '08), July 2008.
[37] M. Balazinska, H. Balakrishnan, S. Madden, and M. Stonebraker, “Fault-Tolerance in the Borealis Distributed Stream Processing System,” Proc. ACM SIGMOD '05, June 2005.
[38] J. Hwang, Y. Xing, U. Çetintemel, and S. Zdonik, “A Cooperative, Self-Configuring High-Availability Solution for Stream Processing,” Proc. 23rd Int'l Conf. Data Eng. (ICDE '07), Apr. 2007.