Avro MapReduce Example

This post shows an Avro MapReduce example program using the Avro MapReduce API.
As an example word count MapReduce program is used where the output will be an Avro data file.

Required jars

avro-mapred-1.8.2.jar

Avro word count MapReduce example

Since output is Avro file so an avro schema has to be defined, we’ll have two fields in the schema “word” and “count”.

In the code you can see the use of AvroKey and AvroValue for the key and value pairs. Also for output AvroKeyOutputFormat class is used.
To define the map output and the output of a MaReduce job AvroJob class is used for job configuration.

Avro MapReduce

After creating jar you can run this Avro MapReduce program using the following command.

This program is executed on a simple text file with only two lines.

The output file can be checked using the avrotools.jar.

That’s all for the topic Avro MapReduce Example. If something is missing or you have something to share about the topic please write a comment.


You may also like

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.