loading...
 This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
26th IEEE International Conference on Distributed Computing Systems (ICDCS'06)
Adaptive Control of Extreme-scale Stream Processing Systems
Lisboa, Portugal
July 04-July 07
ISBN: 0-7695-2540-7
Lisa Amini, IBM T. J. Watson Research Center, NY
Navendu Jain, IBM T. J. Watson Research Center, NY
Anshul Sehgal, IBM T. J. Watson Research Center, NY
Jeremy Silber, IBM T. J. Watson Research Center, NY
Olivier Verscheure, IBM T. J. Watson Research Center, NY
Distributed stream processing systems offer a highly scalable and dynamically configurable platform for time-critical applications ranging from real-time, exploratory data mining to high performance transaction processing. Resource management for distributed stream processing systems is complicated by a number of factors processing elements are constrained by their producer-consumer relationships, data and processing rates can be highly bursty, and traditional measures of effectiveness, such as utilization, can be misleading. In this paper, we propose a novel distributed, adaptive control algorithm that maximizes weighted throughput while ensuring stable operation in the face of highly bursty workloads. Our algorithm is designed to meet the challenges of extreme-scale stream processing systems, where overprovisioning is not an option, by making the best use of resources even when the proffered load is greater than available resources. We have implemented our algorithm in a real-world distributed stream processing system and a simulation environment. Our results show that our algorithm is not only self-stabilizing and robust to errors, but also outperforms traditional approaches over a broad range of buffer sizes, processing graphs, and burstiness types and levels.
Citation:
Lisa Amini, Navendu Jain, Anshul Sehgal, Jeremy Silber, Olivier Verscheure, "Adaptive Control of Extreme-scale Stream Processing Systems," icdcs, pp.71, 26th IEEE International Conference on Distributed Computing Systems (ICDCS'06), 2006
Usage of this product signifies your acceptance of the Terms of Use.