skip to main content
10.1145/1152154.1152155acmconferencesArticle/Chapter ViewAbstractPublication PagespactConference Proceedingsconference-collections
Article

Experiences with MapReduce, an abstraction for large-scale computation

Published: 16 September 2006 Publication History

Abstract

MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a Map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a Reduce function that merges all intermediate values associated with the same intermediate key. Many real world tasks are expressible in this model. Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines.The MapReduce run-time system takes care of the details of partitioning the input data, scheduling the program's execution across a set of machines, handling machine failures, and managing the required intermachine communication. This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system.Our implementation of MapReduce runs on a large cluster of commodity machines and is highly scalable: a typical MapReduce computation processes many terabytes of data on thousands of machines. Programmers find the system easy to use: thousands of MapReduce programs have been implemented and several thousand thousand MapReduce jobs are executed on Google's clusters every day.In this talk I'll describe the basic programming model, discuss our experience using it in a variety of domains, and talk about the implications of programming models like MapReduce as one paradigm to simplify development of parallel software for multi-core microprocessors.

Cited By

View all
  • (2023)GHDC: a dual-centric data center network architecture by using multi-port servers with greater incremental scalabilityThe Journal of Supercomputing10.1007/s11227-023-05046-079:9(9932-9963)Online publication date: 23-Jan-2023
  • (2021)Distributed Computing for Internet of Things (IoT)Research Anthology on Architectures, Frameworks, and Integration Strategies for Distributed and Cloud Computing10.4018/978-1-7998-5339-8.ch091(1874-1894)Online publication date: 2021
  • (2021)Slow Replica and Shared Protection: Energy-Efficient and Reliable Task Assignment in Cloud Data CentersIEEE Transactions on Reliability10.1109/TR.2019.292377070:3(931-943)Online publication date: Sep-2021
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
PACT '06: Proceedings of the 15th international conference on Parallel architectures and compilation techniques
September 2006
308 pages
ISBN:159593264X
DOI:10.1145/1152154
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 September 2006

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. algorithms
  2. design
  3. performance
  4. reliability

Qualifiers

  • Article

Conference

PACT06
Sponsor:

Acceptance Rates

Overall Acceptance Rate 121 of 471 submissions, 26%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)7
  • Downloads (Last 6 weeks)0
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2023)GHDC: a dual-centric data center network architecture by using multi-port servers with greater incremental scalabilityThe Journal of Supercomputing10.1007/s11227-023-05046-079:9(9932-9963)Online publication date: 23-Jan-2023
  • (2021)Distributed Computing for Internet of Things (IoT)Research Anthology on Architectures, Frameworks, and Integration Strategies for Distributed and Cloud Computing10.4018/978-1-7998-5339-8.ch091(1874-1894)Online publication date: 2021
  • (2021)Slow Replica and Shared Protection: Energy-Efficient and Reliable Task Assignment in Cloud Data CentersIEEE Transactions on Reliability10.1109/TR.2019.292377070:3(931-943)Online publication date: Sep-2021
  • (2020)Task failure resilience technique for improving the performance of MapReduce in HadoopETRI Journal10.4218/etrij.2018-0265Online publication date: 18-Aug-2020
  • (2020)Big Data Solutions Proposed for Cluster Computing Systems ChallengesProceedings of the 3rd International Conference on Networking, Information Systems & Security10.1145/3386723.3387826(1-7)Online publication date: 31-Mar-2020
  • (2020)A Distributed Low-Complexity Coding Solution for Large-Scale Distributed FFTIEEE Transactions on Communications10.1109/TCOMM.2020.301664868:11(6617-6628)Online publication date: Nov-2020
  • (2019)Network Intrusion Detection with a Hashing Based Apriori Algorithm Using Hadoop MapReduceComputers10.3390/computers80400868:4(86)Online publication date: 2-Dec-2019
  • (2019)Adaptive and Dynamic Adjustment of Fault Detection Cycles in Cloud ComputingIEEE Transactions on Industrial Informatics10.1109/TII.2019.2922681(1-1)Online publication date: 2019
  • (2019)Time Estimation and Resource Minimization Scheme for Apache Spark and Hadoop Big Data Systems With FailuresIEEE Access10.1109/ACCESS.2019.28910017(9658-9666)Online publication date: 2019
  • (2018)Utilizing MapReduce to Improve Probe-Car Track Data MiningISPRS International Journal of Geo-Information10.3390/ijgi70702877:7(287)Online publication date: 23-Jul-2018
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media