Java Program to Write a File in HDFS

This post shows a Java program to write a file in HDFS using the Hadoop FileSystem API.

Steps for writing a file in HDFS using Java are as follows –

  1. FileSystem is an abstraction of file system of which HDFS is one implementation. So you will have to get an instance of FileSystem (HDFS in this case) using the get method.
  2. In the program you can see get() method takes Configuration as an argument. Configuration object has all the configuration related information read from the configuration files (i.e. core-site.xml from where it gets the file system).
  3. In HDFS, Path object represents the Full file path.
  4. Using create() method of FileSystem you can create a file, method returns FSDataOutputStream.

Java Program to write to a file in HDFS

In the above program both input and output files are in HDFS if your input file is in local file system then you can use BufferedInputStream to create an input stream as given here-

Executing program in Hadoop environment

To execute above Java program in Hadoop environment, you will need to add the directory containing the .class file for the Java program in Hadoop’s classpath.

I have my HDFSFileWrite.class file in location /huser/eclipse-workspace/knpcode/bin so I have exported that path.

Then you can run the program by providing the path of the input file from which data is read and the path of the output file to which content is written.

By using the ls HDFS command you can verify that the file is created or not.

hdfs dfs -ls /user/input/test/

-rw-r–r– 1 knpcode supergroup 10 2018-01-18 14:55 /user/input/test/write.txt

Writing HDFS file using IOUtils class

Hadoop framework provides IOUtils class that has many convenient methods related to I/O. You can use that to copy bytes from the input stream to output stream.

Java program to write HDFS file

That’s all for the topic Java Program to Write a File in HDFS. If something is missing or you have something to share about the topic please write a comment.


You may also like

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.