In this post we’ll see how to read and write Avro files in Hadoop using the Java API. Required Jars To write Java programs to read and write Avro files you will need to put following jars in classpath. You can add them as Maven dependency or copy the jars.…
Sequence files in Hadoop are flat files that store data in the form of serialized key/value pairs. Sequence file format is one of the binary file format supported by Hadoop and it integrates very well with MapReduce (also Hive and PIG). Some of the features of the Sequence files in…
Apache Avro is a data serialization system native to Hadoop which is also language independent. Apache Avro project was created by Doug Cutting, creator of Hadoop to increase data interoperability in Hadoop. Avro implementations for C, C++, C#, Java, PHP, Python, and Ruby are available making it easier to interchange…
In Hadoop framework, where large data sets are stored and processed, you will need storage for large files. These files are divided into blocks and those blocks are stored in different nodes across the cluster so lots of I/O and network data transfer is also involved. In order to reduce…
Apache Parquet is a columnar storage format used in the Apache Hadoop eco system. Table of contents What is a column oriented format Benefits of using Columnar Storage format Parquet file format Parquet file format Structure Types in Parquet format Logical Types in Parquet format What is a column…
This post shows an Avro MapReduce example program using the Avro MapReduce API. As an example word count MapReduce program is used where the output will be an Avro data file. Required jars avro-mapred-1.8.2.jar Avro word count MapReduce example Since output is Avro file so an Avro schema has to…
In this post we’ll see how to read and write Parquet file in Hadoop using the Java API. We’ll also see how you can use MapReduce to write Parquet files in Hadoop. Rather than using the ParquetWriter and ParquetReader directly AvroParquetWriter and AvroParquetReader are used to write and read parquet…
This post shows how to install and use LZO compression in Hadoop. The topics covered are- Installing LZO packages. Downloading and packaging hadoop-lzo. Using hadoop-lzo makes LZO compressed files splittable when used as input to MapReduce job. Configuring LZO packages so that you can use LZO compression in Hadoop. Java…
This post shows how to read and write SequenceFile in Hadoop using Java API, using Hadoop MapReduce and how can you provide compression options for a SequenceFile. Table of contents Writing a sequence file Java program Reading a sequence file Java program Writing SequenceFile using MapReduce Job Reading SequenceFile using…
If you want to compress output of the MapReduce job in Hadoop that can be done per-job basis by setting properties in your job configuration or at a whole cluster level by setting the properties in mapred-site.xml. Properties for compressing MapReduce job output mapreduce.output.fileoutputformat.compress-Set to true if job outputs should…