With in Hadoop framework there are some predefined Mapper and Reducer classes which can be used as is in the required scenarios. That way you are not required to write mapper or reducer for those scenarios, you can use ready made classes instead. Let’s see some of the predefined Mapper…
In this post we’ll see what Distributed cache in Hadoop is. Table of contents What is a distributed cache Methods for adding the files in Distributed Cache How to use distributed cache Distributed cache example MapReduce code What is a distributed cache As the name suggests distributed cache in Hadoop…
This post shows what is combiner in Hadoop MapReduce and how combiner function can be used to reduce the overall memory, I/O and network requirement of the overall MapReduce execution. Table of contents Why is combiner needed in MapReduce Combiner function in MapReduce How to specify a combiner in MapReduce…
Generally when we think of MapReduce job in Hadoop we think of both mappers and reducers doing their share of processing. That is true for most of the cases but you can have scenarios where you want to have a mapper only job in Hadoop. When do you need map…
This post shows an Avro MapReduce example program using the Avro MapReduce API. As an example word count MapReduce program is used where the output will be an Avro data file. Required jars avro-mapred-1.8.2.jar Avro word count MapReduce example Since output is Avro file so an Avro schema has to…
While writing a program, in order to debug we do put some logs or system.out to display messages. In your MapReduce program also you can use logger or sysouts for debugging purposes. In this post we’ll see how you can access those logs or system.out.print messages in Hadoop MR2. How…
Apache Parquet is a columnar storage format used in the Apache Hadoop eco system. Table of contents What is a column oriented format Benefits of using Columnar Storage format Parquet file format Parquet file format Structure Types in Parquet format Logical Types in Parquet format What is a column oriented…
In this post we’ll see how to read and write Parquet file in Hadoop using the Java API. We’ll also see how you can use MapReduce to write Parquet files in Hadoop. Rather than using the ParquetWriter and ParquetReader directly AvroParquetWriter and AvroParquetReader are used to write and read parquet…
This post shows ways to fix missing or corrupt blocks in HDFS and under replicated blocks in HDFS. How to find out corrupt blocks To list the corrupt blocks in the HDFS you can use the following command. $ hdfs fsck -list-corruptfileblocks This command will give you the list of…
When a MapReduce job is started to process a file stored in HDFS, one of the thing Hadoop does is to divide the input into logical splits, these splits are known as input splits in Hadoop. InputSplit represents the data to be processed by an individual map task which means…
