loading...
 This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
19th International Conference on Data Engineering (ICDE'03)
Flux: An Adaptive Partitioning Operator for Continuous Query Systems
Bangalore, India
March 05-March 08
ISBN: 0-7803-7665-X
Mehul A. Shah, U.C. Berkeley
Joseph M. Hellerstein, U.C. Berkeley
Sirish Chandrasekaran, U.C. Berkeley
Michael J. Franklin, U.C. Berkeley
The long-running nature of continuous queries poses new scalability challenges for dataflow processing. CQ systems execute pipelined dataflows that may be shared across multiple queries. The scalability of these dataflows is limited by their constituent, stateful operators - e.g. windowed joins or grouping operators. To scale such operators, a natural solution is to partition them across a shared-nothing platform. But in the CQ context, traditional, static techniques for partitioned parallelism can exhibit detrimental imbalances as workload and runtime conditions evolve. Long-running CQ dataflows must continue to function robustly in the face of these imbalances.
To address this challenge, we introduce a dataflow operator called Flux that encapsulates adaptive state partitioning and dataflow routing. Flux is placed between producer-consumer stages in a dataflow pipeline to repartition stateful operators while the pipeline is still executing. We present the Flux architecture, along with repartitioning policies that can be used for CQ operators under shifting processing and memory loads. We show that the Flux mechanism and these policies can provide several factors improvement in throughput and orders of magnitude improvement in average latency over the static case.
Citation:
Mehul A. Shah, Joseph M. Hellerstein, Sirish Chandrasekaran, Michael J. Franklin, "Flux: An Adaptive Partitioning Operator for Continuous Query Systems," icde, pp.25, 19th International Conference on Data Engineering (ICDE'03), 2003
Usage of this product signifies your acceptance of the Terms of Use.