In this post working of the HDFS components Namenode, Datanode and Secondary Namenode are explained in detail. Namenode in Hadoop HDFS works on a master/slave architecture. In HDFS cluster Namenode is the master and the centerpiece of the HDFS file system. Namenode manages the file system namespace. It keeps the…
In this post we’ll see how to install Hadoop in Pseudo-distributed mode (single node cluster). With the steps given here for Hadoop installation you will have Hadoop common, HDFS, MapReduce and YARN installed. Hadoop release used for installation is Hadoop 2.9.0 and it is installed on Ubuntu 16.04. Modes in…
In this post we’ll see the HDFS data flow in Hadoop. What happens internally when file is read in HDFS and what happens internally when file is written in HDFS. Table of contents Consulting the configuration Getting instance of DFSClient Writing file to HDFS – Internal steps Reading File from…
Big Data means a very large volume of data. Term big data is used to describe data so huge and ever growing that has gone beyond the storage and processing capabilities of traditional data management and processing tools. Some Examples Facebook which stores data about your posts, notification clicks, post…
In this post we’ll see what is HDFS high availability, high availability architecture and the configuration needed for HDFS high availability in Hadoop cluster. Some background on HDFS high availability Prior to Hadoop 2, the NameNode was a single point of failure (SPOF) in an HDFS cluster. In a HDFS cluster there’s…
This post shows what is HDFS federation in Hadoop framework and what configuration changes are required for setting up HDFS federation. Problem with HDFS architecture In a Hadoop cluster namespace management and block management both are done by Namenode. So, essentially the Namenode has to perform the following tasks- 1-…
In this post we’ll see some of the ways to improve performance of the Map-Reduce job in Hadoop. The tips given here for improving the performance of MapReduce job are more from the MapReduce code and configuration perspective rather than cluster and hardware perspective. 1- Enabling uber mode– Like Hadoop…
In Hadoop framework distributed processing happens where map and reduce tasks are spawned on different nodes and process part of the data. In this type of distributed processing it is important to ensure that framework knows when a particular task finishes or there is a need to abort the task…
When you run MapReduce program from command line you provide the jar name, the class that has the code, input and output paths in HDFS. That’s the bare minimum you have to provide to run a MapReduce job. There may be other configurations that you can set with in your…
In many scenarios you would like to create a sequence of MapReduce jobs to completely transform and process the data. This is better than putting every thing in a single MapReduce job job and making it very complex. In fact you can get your data through various sources and use…
