The Community for Technology Leaders
2017 IEEE 23rd International Conference on Parallel and Distributed Systems (ICPADS) (2017)
Shenzhen, Guangdong, China
Dec 15, 2017 to Dec 17, 2017
ISSN: 1521-9097
ISBN: 978-1-5386-2129-5
pp: 485-492
ABSTRACT
Scheduling streaming applications in Data Stream Management Systems (DSMS) has been investigated for years. However, there lacks an intelligent system that is capable of monitoring application execution, modelling its resource usages, and then adjusting the scheduling plan under different sizes of inputs without requiring users' intervention. In this paper, we model the scheduling problem as a bin-packing variant and propose a heuristic-based algorithm to solve it with minimised inter-node communication. We also implement the D-Storm prototype to validate the efficacy and efficiency of our scheduling algorithm, by extending the Apache Storm framework into a self-adaptive MAPE (Monitoring, Analysis, Planning, Execution) architecture. The evaluation carried out on both synthetic and realistic applications proves that D-Storm outperforms the existing resource-aware scheduler and the default Storm scheduler by at least 16.25% in terms of the inter-node traffic reduction and yields a significant amount of resource savings through consolidation.
INDEX TERMS
telecommunication network planning, telecommunication scheduling
CITATION

X. Liu and R. Buyya, "D-Storm: Dynamic Resource-Efficient Scheduling of Stream Processing Applications," 2017 IEEE 23rd International Conference on Parallel and Distributed Systems (ICPADS), Shenzhen, Guangdong, China, 2018, pp. 485-492.
doi:10.1109/ICPADS.2017.00070
182 ms
(Ver 3.3 (11022016))