In a Hadoop MapReduce job you can opt to compress output of the map phase. Since the output of Map task is stored on local disk and data is also transferred across the network to reducer nodes, compressing intermediate map output should help your MapReduce job to run faster.
You can use a fast compressor like snappy or LZ4 for compressing map output as compressor is splittable or not doesn’t matter in case of intermediate output.
Here configuration steps for compressing Map output are given using Snappy codec.
In case you don’t have native snappy compressor library you can install it using the following command in Ubuntu. Using native libraries for compression makes it faster and helps in improving performance of MapReduce job.
|
1 |
$ sudo apt-get install libsnappy-dev |
- Refer How to Check For Which Compressors Native Libraries Are Present to know how to check native libraries for the compressors are present or not.
Required config changes
If you want to compress output of the map phase using Snappy compression at the whole cluster level, set the following properties in mapred-site.xml:
|
1 2 3 4 5 6 7 8 |
<property> <name>mapreduce.map.output.compress</name> <value>true</value> </property> <property> <name>mapreduce.map.output.compress.codec</name> <value>org.apache.hadoop.io.compress.SnappyCodec</value> </property> |
Description for the properties is as follows-
- mapreduce.map.output.compress– Should the outputs of the maps be compressed before being sent across the network. Default is false.
- mapreduce.map.output.compress.codec– If the map outputs are compressed, then what codec should be used. Default is org.apache.hadoop.io.compress.DefaultCodec
If you want to set the property as per-job-basis for compressing the map output then you need to add following lines in your job.
|
1 2 3 |
Configuration conf = new Configuration(); conf.setBoolean("mapreduce.map.output.compress", true); conf.set("mapreduce.map.output.compress.codec", "org.apache.hadoop.io.compress.SnappyCodec"); |
That’s all for the topic How to Compress Intermediate Output of The Map Phase. If something is missing or you have something to share about the topic please write a comment.
You may also like