This post shows ways to fix missing or corrupt blocks in HDFS and under replicated blocks in HDFS. How to find out corrupt blocks To list the corrupt blocks in the HDFS you can use the following command. $ hdfs fsck -list-corruptfileblocks This command will give you the list of…
When a MapReduce job is started to process a file stored in HDFS, one of the thing Hadoop does is to divide the input into logical splits, these splits are known as input splits in Hadoop. InputSplit represents the data to be processed by an individual map task which means…
In this Hadoop tutorial we’ll talk about data locality in Hadoop, how data locality helps in running the job faster and saves cluster bandwidth. Data locality in Hadoop When a file is stored in HDFS it is divided into blocks of 128 MB (Default block size) and these blocks are…
This post shows how to install and use LZO compression in Hadoop. The topics covered are- Installing LZO packages. Downloading and packaging hadoop-lzo. Using hadoop-lzo makes LZO compressed files splittable when used as input to MapReduce job. Configuring LZO packages so that you can use LZO compression in Hadoop. Java…
This post shows how to read and write SequenceFile in Hadoop using Java API, using Hadoop MapReduce and how can you provide compression options for a SequenceFile. Table of contents Writing a sequence file Java program Reading a sequence file Java program Writing SequenceFile using MapReduce Job Reading SequenceFile using…
This post shows how to install Ubuntu along with Windows for a dual boot Ubuntu-Windows system. Here assumption is that you already have a Windows system and you want to install Ubuntu too on the same machine for dual boot. For this post Ubuntu 16.04.3 is installed on Windows 10.…
If you want to compress output of the MapReduce job in Hadoop that can be done per-job basis by setting properties in your job configuration or at a whole cluster level by setting the properties in mapred-site.xml. Properties for compressing MapReduce job output mapreduce.output.fileoutputformat.compress-Set to true if job outputs should…
In a Hadoop MapReduce job you can opt to compress output of the Map phase. Since the output of Map task is stored on local disk and data is also transferred across the network to reducer nodes, compressing map phase output should help your MapReduce job to run faster. You…
If you want to check for which compressors native libraries are present in Hadoop, you can do that using checknative -a command. $ hadoop checknative -a 18/03/16 19:47:24 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native 18/03/16 19:47:24 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library Native library checking:…
In this post we’ll see a Java program that shows how to compress file using gzip format in Hadoop. Compression format gzip does not support splitting so MapReduce job won’t be able to create input splits though compressed file can still be stored as separate HDFS blocks (Size 128 MB by…
