In this post there is a compilation of some of the frequently used HDFS commands with examples which can be used as reference. All HDFS commands are invoked by the bin/hdfs script. Running the hdfs script without any arguments prints the description for all commands. 1- HDFS command to create a…
This post shows what is Safemode in Namenode and what are the configurations for the safemode in Hadoop. You will also see the commands available to enter and leave the safemode explicitly. When the Namenode is started it loads the file system state into memory initially from the fsimage and…
HDFS (Hadoop Distributed File System) is a distributed file system, that is part of Hadoop framework. HDFS in Hadoop framework is designed to store and manage very large files. In HDFS large file is divided into blocks and then those blocks are distributed across the nodes of the cluster. When…
In this post working of the HDFS components Namenode, Datanode and Secondary Namenode are explained in detail. Namenode in Hadoop HDFS works on a master/slave architecture. In HDFS cluster Namenode is the master and the centerpiece of the HDFS file system. Namenode manages the file system namespace. It keeps the…
In this post we’ll see the HDFS data flow in Hadoop. What happens internally when file is read in HDFS and what happens internally when file is written in HDFS. Table of contents Consulting the configuration Getting instance of DFSClient Writing file to HDFS – Internal steps Reading File from…
In this post we’ll see what is HDFS high availability, high availability architecture and the configuration needed for HDFS high availability in Hadoop cluster. Some background on HDFS high availability Prior to Hadoop 2, the NameNode was a single point of failure (SPOF) in an HDFS cluster. In a HDFS cluster there’s…
This post shows what is HDFS federation in Hadoop framework and what configuration changes are required for setting up HDFS federation. Problem with HDFS architecture In a Hadoop cluster namespace management and block management both are done by Namenode. So, essentially the Namenode has to perform the following tasks- 1-…
This post shows ways to fix missing or corrupt blocks in HDFS and under replicated blocks in HDFS. How to find out corrupt blocks To list the corrupt blocks in the HDFS you can use the following command. $ hdfs fsck -list-corruptfileblocks This command will give you the list of…
This post shows a Java program to read a file from HDFS using the Hadoop FileSystem API. Steps for reading the file in HDFS using Java are as follows- FileSystem is an abstraction of file system of which HDFS is one implementation. So you will have to get an instance of…
This post shows a Java program to write a file in HDFS using the Hadoop FileSystem API. Steps for writing a file in HDFS using Java are as follows- FileSystem is an abstraction of file system of which HDFS is one implementation. So you will have to get an instance of…
