In this post there is a compilation of some of the frequently used HDFS commands with examples which can be used as reference.
All HDFS commands are invoked by the bin/hdfs script. Running the hdfs script without any arguments prints the description for all commands.
1- HDFS command to create a directory
hdfs dfs -mkdir
Example– To create a new directory input inside the /user directory
hdfs dfs -mkdir /user/input
2- HDFS command to list all the files
hdfs dfs -ls
Example– To list content of the root directory in HDFS
hdfs dfs -ls /
HDFS command to recursively list all the sub directories
hdfs dfs -ls -R /
3- HDFS command To copy file from local to HDFS
hfds dfs -copyFromLocal
Example– Copy file test.txt from local directory /usr/test to /user/input directory in HDFS
hdfs dfs -copyFromLocal /usr/test/aa.txt /user/input/
If you want overwrite existing file use -f option
hdfs dfs -copyFromLocal -f /usr/test/aa.txt /user/input/
4- put command is also used to copy file from local to HDFS
hfds dfs -put
Example– HDFS command to copy the entire directory /usr/test to HDFS directory /user/input
hdfs dfs -put /usr/test /user/input
5- To copy files with in HDFS
hdfs dfs -cp
Example– Copy file /user/input/test/aa.txt in HDFS to /user/output in HDFS
hdfs dfs -cp /user/input/test/aa.txt /user/output/
6- HDFS command to display free space
hdfs dfs -df
Example– With -h option to show output in human readable format
hdfs dfs -df -h
7- HDFS command to copy file to local file system from HDFS
hdfs dfs -copyToLocal
Example– Copy file part-r-00000 in HDFS location /user/output/ to /usr/Test
hdfs dfs -copyToLocal /user/output/part-r-00000 /usr/test
8- You can also use get command to copy file to local file system from HDFS
hdfs dfs -get
Example– Copy file part-r-00000 in HDFS location /user/output/ to /usr/Test with -f option to overwrite if exists.
hdfs dfs -get -f /user/output/part-r-00000 /home/knpcode/Documents/test
9- HDFS command to delete a file in HDFS
hdfs dfs -rm
Example– Deleting all the .txt files in /user/input/test directory
hdfs dfs -rm /user/input/test/*.txt
HDFS command to recursively delete directory /user/input/test using -R (recursive) option
hdfs dfs -rm -R /user/input/test
10- HDFS command to delete a directory.
hdfs dfs -rmdir
It will delete a directory only if it is empty.
11- To view content of a file in HDFS.
Hdfs dfs -cat
Example– HDFS command to display content of aa.txt file in directory /user/input
hdfs dfs -cat /user/input/test/aa.txt
12- To change group association of files.
hdfs dfs -chgrp
Example– Change group of /user/input/test/aa.txt file to acp.
hdfs dfs -chgrp acp /user/input/test/aa.txt
13- To Change the permissions of files in HDFS.
hdfs dfs -chmod
Permissions are same as in Linux. Read, write and execute permissions for user, group and others. if you want to provide read, write and execute permissions for all then you can use 777 as argument.
hdfs dfs -chmod 777 /user/input/test/aa.txt
You can also do the same thing using the following command.
hdfs dfs -chmod a+rwx /user/input/test/aa.txt
14- To Change the owner of files.
hadoop fs -chown
15- HDFS command to permanently delete files.
hdfs dfs -expunge
16- To display the access control lists for file and directory.
hdfs dfs -getfacl
Example– Getting owner, group and permission information for /user/input/test/aa.txt file.
hdfs dfs -getfacl /user/input/test/aa.txt
17– HDFS command to change replication factor of a file.
hdfs dfs -setrep
Example– Change the replication factor to two.
hdfs dfs -setrep 2 /user/input/test/aa.txt
18- Getting help about commands
hdfs dfs -help
19– Command to run the HDFS filesystem checking utility.
hdfs fsck
Example– Running fsck for path /user/input
hdfs fsck /user/input
20- HDFS command to print the version.
hdfs version
Related Posts
- What is Hadoop
- HDFS Replica Placement Policy
- Namenode in Safemode
- HDFS Data Flow – File Read And Write in HDFS
- How to Fix Corrupt Blocks And Under Replicated Blocks in HDFS
- How to See Logs And Sysouts in Hadoop MapReduce
- How to Check For Which Compressors Native Libraries Are Present
- Uber Task in YARN
That’s all for the topic Frequently Used HDFS Commands With Examples. If something is missing or you have something to share about the topic please write a comment.
You may also like
