The Community for Technology Leaders
2012 21st International Conference on Parallel Architectures and Compilation Techniques (PACT) (2012)
Minneapolis, MN, USA
Sept. 19, 2012 to Sept. 23, 2012
ISBN: 978-1-5090-6609-4
pp: 53-64
Scott Schneider , IBM Thomas J. Watson Research Center, IBM Research, Hawthorne, New York, 10532, USA
Martin Hirzel , IBM Thomas J. Watson Research Center, IBM Research, Hawthorne, New York, 10532, USA
Bugra Gedik , Department of Computer Engineering, Bilkent University, Ankara, 06800, Turkey
Kun-Lung Wu , IBM Thomas J. Watson Research Center, IBM Research, Hawthorne, New York, 10532, USA
ABSTRACT
Streaming applications transform possibly infinite streams of data and often have both high throughput and low latency requirements. They are comprised of operator graphs that produce and consume data tuples. The streaming programming model naturally exposes task and pipeline parallelism, enabling it to exploit parallel systems of all kinds, including large clusters. However, it does not naturally expose data parallelism, which must instead be extracted from streaming applications. This paper presents a compiler and runtime system that automatically extract data parallelism for distributed stream processing. Our approach guarantees safety, even in the presence of stateful, selective, and user-defined operators. When constructing parallel regions, the compiler ensures safety by considering an operator's selectivity, state, partitioning, and dependencies on other operators in the graph. The distributed runtime system ensures that tuples always exit parallel regions in the same order they would without data parallelism, using the most efficient strategy as identified by the compiler. Our experiments using 100 cores across 14 machines show linear scalability for standard parallel regions, and near linear scalability when tuples are shuffled across parallel regions.
INDEX TERMS
Parallel processing, Runtime, Aggregates, Pipelines, Servers, Data mining, Safety,automatic parallelization, distributed stream processing
CITATION
Scott Schneider, Martin Hirzel, Bugra Gedik, Kun-Lung Wu, "Auto-parallelizing stateful distributed streaming applications", 2012 21st International Conference on Parallel Architectures and Compilation Techniques (PACT), vol. 00, no. , pp. 53-64, 2012, doi:
105 ms
(Ver 3.3 (11022016))