25th IEEE International Conference on Distributed Computing Systems (ICDCS'05)
Resource-Aware Distributed Stream Management Using Dynamic Overlays
Columbus, Ohio, USA
June 06-June 10
ISBN: 0-7695-2331-5
We consider distributed applications that continuously stream data across the network, where data needs to be aggregated and processed to produce a 'useful' stream of updates. Centralized approaches to performing data aggregation suffer from high communication overheads, lack of scalability, and unpredictably high processing workloads at central servers. This paper describes a scalable and efficient solution to distributed stream management based on (1) resource-awareness, which is middleware-level knowledge of underlying network and processing resources, (2) overlay-based in-network data aggregation, and (3) high-level programming constructs to describe data-flow graphs for composing useful streams. Technical contributions include a novel algorithm based on resource-aware network partitioning to support dynamic deployment of dataflow graph components across the network, where efficiency of the deployed overlay is maintained by making use of partition-level resource-awareness. Contributions also include efficient middleware-based support for component deployment, utilizing runtime code generation rather than interpretation techniques, thereby addressing both high performance and resource-constrained applications. Finally, simulation experiments and benchmarks attained with actual operational data corroborate this paper's claims.
Citation:
Vibhore Kumar, Brian F. Cooper, Zhongtang Cai, Greg Eisenhauer, Karsten Schwan, "Resource-Aware Distributed Stream Management Using Dynamic Overlays," icdcs, pp.783-792, 25th IEEE International Conference on Distributed Computing Systems (ICDCS'05), 2005