This post shows how to install and use LZO compression in Hadoop. The topics covered are- Installing LZO packages. Downloading and packaging hadoop-lzo. Using hadoop-lzo makes LZO compressed files splittable when used as input to MapReduce job. Configuring LZO packages so that you can use LZO compression in Hadoop. Java…
This post shows how to read and write SequenceFile in Hadoop using Java API, using Hadoop MapReduce and how can you provide compression options for a SequenceFile. Table of contents Writing a sequence file Java program Reading a sequence file Java program Writing SequenceFile using MapReduce Job Reading SequenceFile using…
Object class in Java, residing in java.lang package is the root of the Java class hierarchy. Every class in Java inherits either directly or indirectly from the Object class. If a Java class doesn’t extend any other class then also it implicitly extends Object class making it a direct descendant…
If you want to compress output of the MapReduce job in Hadoop that can be done per-job basis by setting properties in your job configuration or at a whole cluster level by setting the properties in mapred-site.xml. Properties for compressing MapReduce job output mapreduce.output.fileoutputformat.compress-Set to true if job outputs should…
In a Hadoop MapReduce job you can opt to compress output of the Map phase. Since the output of Map task is stored on local disk and data is also transferred across the network to reducer nodes, compressing map phase output should help your MapReduce job to run faster. You…
If you want to check for which compressors native libraries are present in Hadoop, you can do that using checknative -a command. As you can see using checknative command you can check which native libraries are present in hadoop. For zlib, snappy, lz4 and bzip2 it is true, for zstd…
In this post we’ll see a Java program that shows how to compress file using gzip format in Hadoop. Compression format gzip does not support splitting so MapReduce job won’t be able to create input splits though compressed file can still be stored as separate HDFS blocks (Size 128 MB by…
This post shows how to write a Java program to compress a file in HDFS using bzip2 compression. The program takes input file from local file system and write a BZip2 compressed file as output in HDFS. Java program to compress file in bzip2 format Hadoop compression codec that has…
Speculative execution in Hadoop framework is an optimization technique to ensure that the submitted job finishes in a time-bound manner. Need for speculative execution in Hadoop When a MapReduce job is submitted there will be several map tasks running in parallel working on the portion of the data (input splits).…
In this post we’ll see what all happens internally with in the Hadoop framework to execute a job when a MapReduce job is submitted to YARN. The three main components when running a MapReduce job in YARN are – The client which submits a job. YARN daemons that manage the…