How to See Logs And Sysouts in Hadoop MapReduce

While writing a program, in order to debug we do put some logs or system.out to display messages. In your MapReduce program also you can use logger or sysouts for debugging purposes. In this post we’ll see how you can access those logs or system.out.print messages in Hadoop MR2.

How to see log messages in MapReduce2

First thing of course is to put logs in your code. Then at the time of running your MapReduce job you can note the application_id of the job from the console. Once you run your MapReduce job you will get a line as following displayed on the console showing the application id.

With the same application_id a folder will be created in the location HADOOP_INSTALLATION_DIR/logs/userlogs/ there you will find folders having logs for your mappers and reducers. In those folders you can check stdout file for any system.out.print and syslog for log messages.

Example MapReduce showing how to put logs

You can use apache commons logging which comes with the Hadoop bundle for logging purposes. Here is a simple word count MapReduce program with some and sysout messages put in.

Once you run it and you know the application id just go to the location and check the stdout and syslog files.

As example after running this code I can access the stdout at the path-HADOOP_INSTALLATION_DIR/logs/userlogs/application_1528883210739_0001/container_1528883210739_0001_01_000002/stdout and see my sysouts there-

Or I can access syslog at the path- HADOOP_INSTALLATION_DIR/logs/userlogs/application_1528883210739_0001/container_1528883210739_0001_01_000003/syslog and see the loggers for the reducer.

That’s all for the topic How to See Logs And Sysouts in Hadoop MapReduce. If something is missing or you have something to share about the topic please write a comment.

You may also like

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.