In this post there is a compilation of some of the frequently used HDFS commands with examples which can be used as reference.
All HDFS commands are invoked by the bin/hdfs script. Running the hdfs script without any arguments prints the description for all commands.
1- HDFS command to create a directory
hdfs dfs -mkdir
Example – To create a new directory input inside the /user directory
hdfs dfs -mkdir /user/input
2- HDFS command to list all the files
hdfs dfs -ls
Example – To list content of the root directory in HDFS
hdfs dfs -ls /
To recursively list all the subdirectories
hdfs dfs -ls -R /
3- Command To copy file from local to HDFS
hfds dfs -copyFromLocal
Example – Copy file test.txt from local directory /usr/test to /user/input directory in HDFS
hdfs dfs -copyFromLocal /usr/test/aa.txt /user/input/
If you want overwrite existing file use -f option
hdfs dfs -copyFromLocal -f /usr/test/aa.txt /user/input/
4- put command is also used to copy file from local to HDFS
hfds dfs -put
Example – Copy the entire directory /usr/test to HDFS directory /user/input
hdfs dfs -put /usr/test /user/input
5- To copy files with in HDFS
hdfs dfs -cp
Example – Copy file /user/input/test/aa.txt in HDFS to /user/output in HDFS
hdfs dfs -cp /user/input/test/aa.txt /user/output/
6- HDFS command to display free space
hdfs dfs -df
Example – With -h option to show output in human readable format
hdfs dfs -df -h
7- HDFS command to copy file to local file system from HDFS
hdfs dfs -copyToLocal
Example – Copy file part-r-00000 in HDFS location /user/output/ to /usr/Test
hdfs dfs -copyToLocal /user/output/part-r-00000 /usr/test
8- You can also use get command to copy file to local file system from HDFS
hdfs dfs -get
Example – Copy file part-r-00000 in HDFS location /user/output/ to /usr/Test with -f option to overwrite if exists.
hdfs dfs -get -f /user/output/part-r-00000 /home/knpcode/Documents/test
9- HDFS command to delete a file in HDFS
hdfs dfs -rm
Example – Deleting all the .txt files in /user/input/test directory
hdfs dfs -rm /user/input/test/*.txt
Deleting directory /user/input/test with any content in it recursively with -R (recursive) option
hdfs dfs -rm -R /user/input/test
10 – HDFS command to delete a directory.
hdfs dfs -rmdir
It will delete a directory only if it is empty.
11- To view content of a file in HDFS.
Hdfs dfs -cat
Example – Display content of aa.txt file in directory /user/input
hdfs dfs -cat /user/input/test/aa.txt
12- To change group association of files.
hdfs dfs -chgrp
Example – Change group of /user/input/test/aa.txt file to acp.
hdfs dfs -chgrp acp /user/input/test/aa.txt
13- To Change the permissions of files in HDFS.
hdfs dfs -chmod
Permissions are same as in Linux. Read, write and execute permissions for user, group and others. if you want to provide read, write and execute permissions for all then you can use 777 as argument.
hdfs dfs -chmod 777 /user/input/test/aa.txt
You can also o the same thing using the following command.
hdfs dfs -chmod a+rwx /user/input/test/aa.txt
14- To Change the owner of files.
hadoop fs -chown
15- To Permanently delete files.
hdfs dfs -expunge
16- To display the access control lists for file and directory.
hdfs dfs -getfacl
Example – Getting owner, group and permission information for /user/input/test/aa.txt file.
hdfs dfs -getfacl /user/input/test/aa.txt
17 – HDFS command to change replication factor of a file.
hdfs dfs -setrep
Example – Change the replication factor to two.
hdfs dfs -setrep 2 /user/input/test/aa.txt
18- Getting help about commands
hdfs dfs -help
19 – To run the HDFS filesystem checking utility.
hdfs fsck
Example – Running fsck for path /user/input
hdfs fsck /user/input
20 – Print the version.
hdfs version
That’s all for the topic Frequently Used HDFS Commands. If something is missing or you have something to share about the topic please write a comment.
You may also like
Pingback: Java Program to Write a File in HDFS - KnpCode
Pingback: Java Program to Compress File in gzip Format in Hadoop - KnpCode