The Community for Technology Leaders
2006 International Conference on Parallel Architectures and Compilation Techniques (PACT) (2006)
Seattle, WA, USA
Sept. 16, 2006 to Sept. 20, 2006
ISBN: 978-1-5090-3022-4
pp: 1
Jeffrey Dean , Google, Inc., Mountain View, CA, USA
ABSTRACT
MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a Map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a Reduce function that merges all intermediate values associated with the same intermediate key. Many real world tasks are expressible in this model. Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines. The MapReduce run-time system takes care of the details of partitioning the input data, scheduling the program's execution across a set of machines, handling machine failures, and managing the required inter-machine communication. This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system. Our implementation of MapReduce runs on a large cluster of commodity machines and is highly scalable: a typical MapReduce computation processes many terabytes of data on thousands of machines. Programmers find the system easy to use: thousands of MapReduce programs have been implemented and several thousand thousand MapReduce jobs are executed on Google's clusters every day. In this talk I'll describe the basic programming model, discuss our experience using it in a variety of domains, and talk about the implications of programming models like MapReduce as one paradigm to simplify development of parallel software for multi-core microprocessors.
INDEX TERMS
Reliability, Algorithms, Design, Performance
CITATION
Jeffrey Dean, "Keynote talk: Experiences with MapReduce, an abstraction for large-scale computation", 2006 International Conference on Parallel Architectures and Compilation Techniques (PACT), vol. 00, no. , pp. 1, 2006, doi:
709 ms
(Ver 3.3 (11022016))