Hadoop MapReduce Applications With Combiners

Apache Hadoop    |    Intermediate
  • 13 videos | 1h 23m 5s
  • Includes Assessment
  • Earns a Badge
Rating 4.5 of 4 users Rating 4.5 of 4 users (4)
In this Skillsoft Aspire course, explore the use of Combiners to make MapReduce applications more efficient by minimizing data transfers. Start by learning about the need for Combiners to optimize the execution of a MapReduce application by minimizing data transfers within a cluster. Recall the steps to process data in a MapReduce application, and look at using a Combiner to perform partial reduction of data output from the Mapper. Then create a new project to calculate average automobile prices using Maven for a MapReduce application. Next, develop the Mapper and Reducer to calculate the average price for automobile makes in the input data set. Create a driver program for the MapReduce application, run it, and check output to get the average price per automobile. Learn how to code up a Combiner for a MapReduce application, fix the bug in the application so it can be used to correctly calculate the average price, then run the fixed application to verify that the prices are being calculated correctly. The concluding exercise concerns optimizing MapReduce with Combiners.

WHAT YOU WILL LEARN

  • Recognize the need for combiners to optimize the execution of a mapreduce application by minimizing data transfers within a cluster
    Recall the steps involved in processing data in a mapreduce application
    Describe the working of a combiner in performing a partial reduction of the data that is output from the mapper
    Configure a combiner to optimize a mapreduce application that calculates an average value
    Use maven to create a new project for a mapreduce application and plan out the map and reduce phases by examining the auto prices dataset
    Develop the mapper and reducer for the application that will calculate the average price for each make of automobile in the input dataset
  • Create the driver program for the mapreduce application
    Run the mapreduce application and check the output to get the average price for each automobile make
    Code up a combiner for the mapreduce application and configure the driver to use it for a partial reduction on the mapper nodes of the cluster
    Fix the bug in the previous application by defining a type that represents both the aggregate price and count of automobiles that can be used to correctly calculate the average price
    Compare the output of the modified application with the previous buggy version and verify that the average prices for the vehicles are being calculated correctly
    Identify the shortcomings of regular mapreduce operations which are addressed by combiners, and how combiners differ from reducers

IN THIS COURSE

  • 2m 32s
  • 5m 19s
    Upon completion of this video, you will be able to recognize the need for combiners to optimize the execution of a MapReduce application by minimizing data transfers within a cluster. FREE ACCESS
  • Locked
    3.  Revisiting MapReduce
    5m 2s
    Upon completion of this video, you will be able to recall the steps involved in processing data in a MapReduce application. FREE ACCESS
  • Locked
    4.  Working with Combiners
    5m 47s
    After completing this video, you will be able to describe how a Combiner works in performing a partial reduction of the data that is output from the Mapper. FREE ACCESS
  • Locked
    5.  Using Combiners for Calculating Averages
    8m 22s
    In this video, find out how to configure a Combiner to optimize a MapReduce application that calculates an average value. FREE ACCESS
  • Locked
    6.  Creating a Project to Calculate Averages
    7m 1s
    In this video, you will use Maven to create a new project for a MapReduce application. You will also plan out the Map and Reduce phases by examining the auto prices dataset. FREE ACCESS
  • Locked
    7.  Coding the Map and Reduce Phases
    8m 3s
    Find out how to develop the Mapper and Reducer for the application that will calculate the average price for each make of automobile in the input dataset. FREE ACCESS
  • Locked
    8.  Configure the Application in the Driver
    3m 13s
    In this video, you will create the driver program for the MapReduce application. FREE ACCESS
  • Locked
    9.  Executing the Application and Examining the Output
    7m 54s
    Learn how to run the MapReduce application and check the output to get the average price for each automobile make. FREE ACCESS
  • Locked
    10.  Adding a Combiner to a MapReduce Application
    9m 31s
    In this video, you will code up a Combiner for the MapReduce application and configure the Driver to use it for a partial reduction on the Mapper nodes of the cluster. FREE ACCESS
  • Locked
    11.  Conveying a Pair of Numbers from the Mapper
    9m 21s
    In this video, learn how to fix the bug in the previous application by defining a type that represents both the aggregate price and count of automobiles. This will allow you to correctly calculate the average price. FREE ACCESS
  • Locked
    12.  Running the Fixed Application
    5m 6s
    Learn how to compare the output of the modified application with the previous buggy version and verify that the average prices for the vehicles are being calculated correctly. FREE ACCESS
  • Locked
    13.  Exercise: Optimizing MapReduce With Combiners
    5m 54s
    In this video, you will identify the shortcomings of regular MapReduce operations which are addressed by Combiners, and how Combiners differ from Reducers. FREE ACCESS

EARN A DIGITAL BADGE WHEN YOU COMPLETE THIS COURSE

Skillsoft is providing you the opportunity to earn a digital badge upon successful completion on some of our courses, which can be shared on any social network or business platform.

Digital badges are yours to keep, forever.

PEOPLE WHO VIEWED THIS ALSO VIEWED THESE

Rating 4.7 of 7 users Rating 4.7 of 7 users (7)
Rating 4.4 of 54 users Rating 4.4 of 54 users (54)
Rating 4.1 of 38 users Rating 4.1 of 38 users (38)