The Community for Technology Leaders
Green Image
Issue No. 03 - March (2018 vol. 29)
ISSN: 1045-9219
pp: 572-585
Federico Lombardi , Department of Computer, Control, and Management Engineering Antonio Ruberti, Research Center of Cyber Intelligence and Information Security, Sapienza University of Rome, Roma, Italy
Leonardo Aniello , Department of Computer, Control, and Management Engineering Antonio Ruberti, Research Center of Cyber Intelligence and Information Security, Sapienza University of Rome, Roma, Italy
Silvia Bonomi , Department of Computer, Control, and Management Engineering Antonio Ruberti, Research Center of Cyber Intelligence and Information Security, Sapienza University of Rome, Roma, Italy
Leonardo Querzoni , Department of Computer, Control, and Management Engineering Antonio Ruberti, Research Center of Cyber Intelligence and Information Security, Sapienza University of Rome, Roma, Italy
ABSTRACT
Distributed stream processing frameworks are designed to perform continuous computation on possibly unbounded data streams whose rates can change over time. Devising solutions to make such systems elastically scale is a fundamental goal to achieve desired performance and cut costs caused by resource over-provisioning. These systems can be scaled along two dimensions: the operator parallelism and the number of resources. In this paper, we show how these two dimensions, as two symbiotic entities, are independent but must mutually interact for the global benefit of the system. On the basis of this observation, we propose a fine-grained model for estimating the resource utilization of a stream processing application that enables the independent scaling of operators and resources. A simple, yet effective, combined management of the two dimensions allows us to propose ELYSIUM, a novel elastic scaling approach that provides efficient resource utilization. We implemented the proposed approach within Apache Storm and tested it by running two real-world applications with different input load curves. The outcomes backup our claims showing that the proposed symbiotic management outperforms elastic scaling strategies where operators and resources are jointly scaled.
INDEX TERMS
Symbiosis, Parallel processing, Runtime, Computational modeling, Stress, Storms, Process control
CITATION

F. Lombardi, L. Aniello, S. Bonomi and L. Querzoni, "Elastic Symbiotic Scaling of Operators and Resources in Stream Processing Systems," in IEEE Transactions on Parallel & Distributed Systems, vol. 29, no. 3, pp. 572-585, 2018.
doi:10.1109/TPDS.2017.2762683
284 ms
(Ver 3.3 (11022016))