21st International Conference on Data Engineering (ICDE'05) Dynamic Load Distribution in the Borealis Stream Processor Tokyo, Japan April 05-April 08 ISBN: 0-7695-2285-8
DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/ICDE.2005.53
Distributed and parallel computing environments are becoming cheap and commonplace. The availability of large numbers of CPU?s makes it possible to process more data at higher speeds. Stream-processing systems are also becoming more important, as broad classes of applications require results in real-time. Since load can vary in unpredictable ways, exploiting the abundant processor cycles requires effective dynamic load distribution techniques. Although load distribution has been extensively studied for the traditional pull-based systems, it has not yet been fully studied in the context of push-based continuous query processing. In this paper, we present a correlation based load distribution algorithm that aims at avoiding overload and minimizing end-to-end latency by minimizing load variance and maximizing load correlation. While finding the optimal solution for such a problem is NP-hard, our greedy algorithm can find reasonable solutions in polynomial time. We present both a global algorithm for initial load distribution and a pair-wise algorithm for dynamic load migration.
Citation:
Ying Xing, Stan Zdonik, Jeong-Hyon Hwang, "Dynamic Load Distribution in the Borealis Stream Processor," icde, pp.791-802, 21st International Conference on Data Engineering (ICDE'05), 2005 Usage of this product signifies your acceptance of the Terms of Use. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||