This post shows an Avro MapReduce example program using the Avro MapReduce API. As an example word count MapReduce program is used where the output will be an Avro data file. Required jars avro-mapred-1.8.2.jar Avro word count MapReduce example Since output is Avro file so an Avro schema has to…
Apache Parquet is a columnar storage format used in the Apache Hadoop eco system. Table of contents What is a column oriented format Benefits of using Columnar Storage format Parquet file format Parquet file format Structure Types in Parquet format Logical Types in Parquet format What is a column oriented…
In this post we’ll see how to read and write Parquet file in Hadoop using the Java API. We’ll also see how you can use MapReduce to write Parquet files in Hadoop. Rather than using the ParquetWriter and ParquetReader directly AvroParquetWriter and AvroParquetReader are used to write and read parquet…
This post shows how to read and write SequenceFile in Hadoop using Java API, using Hadoop MapReduce and how can you provide compression options for a SequenceFile. Table of contents Writing a sequence file Java program Reading a sequence file Java program Writing SequenceFile using MapReduce Job Reading SequenceFile using…
If you want to compress output of the MapReduce job in Hadoop that can be done per-job basis by setting properties in your job configuration or at a whole cluster level by setting the properties in mapred-site.xml. Properties for compressing MapReduce job output mapreduce.output.fileoutputformat.compress-Set to true if job outputs should…
In a Hadoop MapReduce job you can opt to compress output of the Map phase. Since the output of Map task is stored on local disk and data is also transferred across the network to reducer nodes, compressing map phase output should help your MapReduce job to run faster. You…
If you want to check for which compressors native libraries are present in Hadoop, you can do that using checknative -a command. $ hadoop checknative -a 18/03/16 19:47:24 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native 18/03/16 19:47:24 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library Native library checking:…
In this post we’ll see a Java program that shows how to compress file using gzip format in Hadoop. Compression format gzip does not support splitting so MapReduce job won’t be able to create input splits though compressed file can still be stored as separate HDFS blocks (Size 128 MB by…
This post shows how to install and use LZO compression in Hadoop. The topics covered are- Installing LZO packages. Downloading and packaging hadoop-lzo. Using hadoop-lzo makes LZO compressed files splittable when used as input to MapReduce job. Configuring LZO packages so that you can use LZO compression in Hadoop. Java…
This post shows how to write a Java program to compress a file in HDFS using bzip2 compression. The program takes input file from local file system and write a BZip2 compressed file as output in HDFS. Java program to compress file in bzip2 format Hadoop compression codec that has…
