The Community for Technology Leaders
2014 International Symposium on Computer Architecture and High Performance Computing Workshop (SBAC-PADW) (2014)
Paris, France
Oct. 22, 2014 to Oct. 24, 2014
ISBN: 978-1-4799-7014-8
pp: 120-125
ABSTRACT
Most high-performance data processing (aka big-data) systems allow users to express their computation using abstractions (like map-reduce) that simplify the extraction of parallelism from applications. Most frameworks, however, do not allow users to specify how communication must take place: that element is deeply embedded into the run-time system (RTS), making changes hard to implement. In this work we describe our reengineering of the Watershed system, a framework based on the filter-stream paradigm and focused on continuous stream processing. Like other big-data environments, watershed provided object-oriented abstractions to express computation (filters), but the implementation of streams was an RTS element. By isolating stream functionality into appropriate classes, combination of communication patterns and reuse of common message handling functions (like compression and blocking) become possible. The new architecture even allow the design of new communication patterns, for example, allowing users to choose MPI, TCP or shared memory implementations of communication channels as their problem demand. Applications designed for the new interface showed reductions in code size on the order of 50%and above in some cases, with no significant performance penalty.
INDEX TERMS
Ports (Computers), Decoding, Computers, Computational modeling, Training, Distributed databases, Libraries,parallel programming, big data, high-performance computing, stream processing system, programming model
CITATION
Rodrigo Caetano Rocha, Renato Ferreira, Wagner Meira, Dorgival Guedes, "Watershed reengineering: Making Streams Programmable", 2014 International Symposium on Computer Architecture and High Performance Computing Workshop (SBAC-PADW), vol. 00, no. , pp. 120-125, 2014, doi:10.1109/SBAC-PADW.2014.31
91 ms
(Ver 3.3 (11022016))