Counters in Hadoop MapReduce help in getting statistics about the MapReduce job. With counters in Hadoop you can get general information about the executed job like launched map and reduce tasks, map input records, use the information to diagnose if there is any problem with data, use information provided by…
In a MapReduce job when Map tasks start producing output, the output is sorted by keys and the map outputs are also transferred to the nodes where reducers are running. This whole process is known as shuffle phase in the Hadoop MapReduce. Though shuffle phase is internal to Hadoop framework…
In the post WordCount MapReduce program we have seen how to write a MapReduce program in Java, create a jar and run it. There are a lot of things that you do to create a MapReduce job and Hadoop framework also do a lot of processing internally. In this post…
Once you have installed Hadoop on your system and initial verification is done you would be looking to write your first MapReduce program. Before digging deeper into the intricacies of MapReduce programming first step is the word count MapReduce program in Hadoop which is also known as the “Hello World”…
In this post we’ll see some of the ways to improve performance of the Map-Reduce job in Hadoop. The tips given here for improving the performance of MapReduce job are more from the MapReduce code and configuration perspective rather than cluster and hardware perspective. 1- Enabling uber mode– Like Hadoop…
In Hadoop framework distributed processing happens where map and reduce tasks are spawned on different nodes and process part of the data. In this type of distributed processing it is important to ensure that framework knows when a particular task finishes or there is a need to abort the task…
With in Hadoop framework there are some predefined Mapper and Reducer classes which can be used as is in the required scenarios. That way you are not required to write mapper or reducer for those scenarios, you can use ready made classes instead. Let’s see some of the predefined Mapper…
In many scenarios you would like to create a sequence of MapReduce jobs to completely transform and process the data. This is better than putting every thing in a single MapReduce job job and making it very complex. In fact you can get your data through various sources and use…
In this post we’ll see what Distributed cache in Hadoop is. Table of contents What is a distributed cache Methods for adding the files in Distributed Cache How to use distributed cache Distributed cache example MapReduce code What is a distributed cache As the name suggests distributed cache in Hadoop…
This post shows what is combiner in Hadoop MapReduce and how combiner function can be used to reduce the overall memory, I/O and network requirement of the overall MapReduce execution. Table of contents Why is combiner needed in MapReduce Combiner function in MapReduce How to specify a combiner in MapReduce…
