How to See Logs And Sysouts in Hadoop MapReduce

While writing a program, in order to debug we do put some logs or system.out to display messages. In your MapReduce program also you can use logger or sysouts for debugging purposes. In this post we’ll see how you can access those logs or system.out.print messages in Hadoop MR2.

How to see log messages in MapReduce2

First thing of course is to put logs in your code. Then at the time of running your MapReduce job you can note the application_id of the job from the console. Once you run your MapReduce job you will get a line as following displayed on the console showing the application id.

18/06/13 15:20:59 INFO impl.YarnClientImpl: Submitted application application_1528883210739_0001

With the same application_id a folder will be created in the location HADOOP_INSTALLATION_DIR/logs/userlogs/ there you will find folders having logs for your mappers and reducers. In those folders you can check stdout file for any system.out.print and syslog for log messages.

Example MapReduce showing how to put logs

You can use Apache commons logging which comes with the Hadoop bundle for logging purposes. Here is a simple word count MapReduce program with some log.info and sysout messages put in.

import java.io.IOException;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount{
  public static final Log log = LogFactory.getLog(WordCount.class);
  // Map function
  public static class WordMapper extends Mapper<LongWritable, Text, Text, IntWritable>{
				
    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();
    public void map(LongWritable key, Text value, Context context) 
        throws IOException, InterruptedException {
      log.info("in map method");
      // Splitting the line on spaces
      String[] stringArr = value.toString().split("\\s+");
      for (String str : stringArr) {
        word.set(str);
        System.out.println("word -- " + word.toString());
        context.write(word, one);
      }	 
    }
  }
	
  // Reduce function
  public static class CountReducer extends Reducer<Text, IntWritable, Text, IntWritable>{		   
    private IntWritable result = new IntWritable();

    public void reduce(Text key, Iterable values, Context context) 
        throws IOException, InterruptedException {
      log.info("in reducer ");
      int sum = 0;
      for (IntWritable val : values) {
        sum += val.get();
      }
      log.info(key + " -- Sum is --- " + sum);
      result.set(sum);
      context.write(key, result);
    }
  }
	
  public static void main(String[] args) throws Exception{
    Configuration conf = new Configuration();
    Job job = Job.getInstance(conf, "word count");
    job.setJarByClass(WordCount.class);
    job.setMapperClass(WordMapper.class);  
    //job.setNumReduceTasks(0);
    job.setReducerClass(CountReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);
    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));
  
    System.exit(job.waitForCompletion(true) ? 0 : 1);
  }
}

Once you run it and you know the application id just go to the location and check the stdout and syslog files.

As example after running this code I can access the stdout at the path-HADOOP_INSTALLATION_DIR/logs/userlogs/application_1528883210739_0001/container_1528883210739_0001_01_000002/stdout and see my sysouts there-

word -- This
word -- is
word -- a
word -- test
word -- file.
word -- This
word -- is
word -- a
word -- Hadoop
word -- MapReduce
word -- program
word – file.

Or I can access syslog at the path- HADOOP_INSTALLATION_DIR/logs/userlogs/application_1528883210739_0001/container_1528883210739_0001_01_000003/syslog and see the loggers for the reducer.

2018-06-13 15:21:15,321 INFO [main] org.knpcode.WordCount$WordMapper: in reducer 
2018-06-13 15:21:15,321 INFO [main] org.knpcode.WordCount$WordMapper: Hadoop -- Sum is --- 1
2018-06-13 15:21:15,322 INFO [main] org.knpcode.WordCount$WordMapper: in reducer 
2018-06-13 15:21:15,322 INFO [main] org.knpcode.WordCount$WordMapper: MapReduce -- Sum is --- 1
2018-06-13 15:21:15,322 INFO [main] org.knpcode.WordCount$WordMapper: in reducer 
2018-06-13 15:21:15,322 INFO [main] org.knpcode.WordCount$WordMapper: This -- Sum is --- 2
2018-06-13 15:21:15,322 INFO [main] org.knpcode.WordCount$WordMapper: in reducer 
2018-06-13 15:21:15,322 INFO [main] org.knpcode.WordCount$WordMapper: a -- Sum is --- 2
2018-06-13 15:21:15,322 INFO [main] org.knpcode.WordCount$WordMapper: in reducer 
2018-06-13 15:21:15,322 INFO [main] org.knpcode.WordCount$WordMapper: file. -- Sum is --- 2
2018-06-13 15:21:15,322 INFO [main] org.knpcode.WordCount$WordMapper: in reducer 
2018-06-13 15:21:15,322 INFO [main] org.knpcode.WordCount$WordMapper: is -- Sum is --- 2
2018-06-13 15:21:15,322 INFO [main] org.knpcode.WordCount$WordMapper: in reducer 
2018-06-13 15:21:15,322 INFO [main] org.knpcode.WordCount$WordMapper: program -- Sum is --- 1
2018-06-13 15:21:15,323 INFO [main] org.knpcode.WordCount$WordMapper: in reducer 
2018-06-13 15:21:15,323 INFO [main] org.knpcode.WordCount$WordMapper: test -- Sum is --- 1

That's all for the topic How to See Logs And Sysouts in Hadoop MapReduce. If something is missing or you have something to share about the topic please write a comment.

You may also like

KnpCode

December 22, 2023