This post shows ways to fix missing or corrupt blocks in HDFS and under replicated blocks in HDFS. How to find out corrupt blocks To list the corrupt blocks in the HDFS you can use the following command. This command will give you the list of missing blocks and the…
This post shows a Java program to write a file in HDFS using the Hadoop FileSystem API. Steps for writing a file in HDFS using Java are as follows – FileSystem is an abstraction of file system of which HDFS is one implementation. So you will have to get an instance…
This post shows a Java program to read a file from HDFS using the Hadoop FileSystem API. Steps for reading the file in HDFS using Java are as follows – FileSystem is an abstraction of file system of which HDFS is one implementation. So you will have to get an instance…
In this post we’ll see the HDFS data flow in Hadoop. What happens internally when file is read in HDFS and what happens internally when file is written in HDFS. Consulting the configuration While reading or writing a file in HDFS first thing Hadoop framework does is to consult the…
In this post we’ll see what is HDFS high availability, high availability architecture and the configuration needed for HDFS high availability in Hadoop cluster. Some background on HDFS high availability Prior to Hadoop 2, the NameNode was a single point of failure (SPOF) in an HDFS cluster. In a HDFS cluster there’s…
This post shows what is Safemode in Namenode and what are the configurations for the safemode. You will also see the commands available to enter and leave the safemode explicitly. When the Namenode is started it loads the file system state into memory initially from the fsimage and then apply…
This post shows what is HDFS federation in Hadoop framework and what configuration changes are required for setting up HDFS federation. Problem with HDFS architecture In a Hadoop cluster namespace management and block management both are done by Namenode. So, essentially the Namenode has to perform the following tasks- 1-…
In this post there is a compilation of some of the frequently used HDFS commands with examples which can be used as reference. All HDFS commands are invoked by the bin/hdfs script. Running the hdfs script without any arguments prints the description for all commands. 1- HDFS command to create a…
In this post working of the HDFS components Namenode, Datanode and Secondary Namenode are explained in detail. Namenode in Hadoop HDFS works on a master/slave architecture. In HDFS cluster Namenode is the master and the centerpiece of the HDFS file system. Namenode manages the file system namespace. It keeps the…
As per the replica placement policy in Hadoop each HDFS block is replicated across different nodes. Default replication factor is 3 which means by default each HDFS block is replicated on three different nodes in order to make HDFS reliable and fault tolerant. Considerations for HDFS replica placement policy When…