The Community for Technology Leaders
Green Image
Issue No. 01 - March (2018 vol. 4)
ISSN: 2332-7790
pp: 130-137
Yuxuan Jiang , Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong
Zhe Huang , Department of Electrical Engineering, Princeton University, Princeton, NJ
Danny H.K. Tsang , Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong
ABSTRACT
Distributed stream big data analytics platforms have emerged to tackle the continuously generated data streams. In stream big data analytics, the data processing workflow is abstracted as a directed graph referred to as a topology. Data are read from the storage and processed tuple by tuple, and these processing results are updated dynamically. The performance of a topology is evaluated by its throughput. This paper proposes an efficient resource allocation scheme for a heterogeneous stream big data analytics cluster shared by multiple topologies, in order to achieve max-min fairness in the utilities of the throughput for all the topologies. We first formulate a novel resource allocation problem, which is a mixed 0-1 integer program. The NP-hardness of the problem is rigorously proven. To tackle this problem, we transform the non-convex constraint to several linear constraints using linearization and reformulation techniques. Based on the analysis of the problem-specific structure and characteristics, we propose an approach that iteratively solves the continuous problem with a fixed set of discrete variables optimally, and updates the discrete variables heuristically. Simulations show that our proposed resource allocation scheme remarkably improves the max-min fairness in utilities of the topology throughput, and is low in computational complexity.
INDEX TERMS
Topology, Throughput, Resource management, Big data, Computational modeling, Storms, Fasteners
CITATION

Y. Jiang, Z. Huang and D. H. Tsang, "Towards Max-Min Fair Resource Allocation for Stream Big Data Analytics in Shared Clouds," in IEEE Transactions on Big Data, vol. 4, no. 1, pp. 130-137, 2018.
doi:10.1109/TBDATA.2016.2638860
373 ms
(Ver 3.3 (11022016))