public class

HdfsConnector

extends Object
java.lang.Object
   ↳ org.mule.modules.hdfs.HdfsConnector
Known Direct Subclasses
Known Indirect Subclasses

Class Overview

Hadoop Distributed File System (HDFS) Connector.

{@sample.config INCLUDE_ERROR} {@sample.config INCLUDE_ERROR} {@sample.config INCLUDE_ERROR}

Summary

Constants
String HDFS
String HDFS_CONTENT_SUMMARY
String HDFS_FILE_CHECKSUM
String HDFS_FILE_STATUS
String HDFS_PATH_EXISTS
Fields
private static final Logger LOGGER
private Map<String, String> configurationEntries A Map of configuration entries to be used by the HDFS client.
private List<String> configurationResources A List of configuration resource files to be loaded by the HDFS client.
private String defaultFileSystemName The name of the file system to connect to.
private FileSystem fileSystem A readily configured FileSystem to use to connect to HDFS.
Public Constructors
HdfsConnector()
Public Methods
void appendToPath(String path, int bufferSize, InputStream payload)
Append the current payload to a file located at the designated path.
void connect(String connectionKey)
Establish the connection to the Hadoop Distributed File System.
void deleteDirectory(String path)
Delete the file or directory located at the designated path.
void deleteFile(String path)
Delete the file or directory located at the designated path.
void disconnect()
Disconnect from the Hadoop Distributed File System.
Map<String, String> getConfigurationEntries()
List<String> getConfigurationResources()
String getDefaultFileSystemName()
FileSystem getFileSystem()
String getFileSystemUri()
void getPathMetaData(String path, MuleEvent muleEvent)
Get the metadata of a path, as described in readFromPath(String, int, SourceCallback), and store it in flow variables.
boolean isConnected()
Are we connected?
void makeDirectories(String path, String permission)
Make the given file and all non-existent parents into directories.
Object readFromPath(String path, int bufferSize, SourceCallback sourceCallback)
Read the content of a file designated by its path and streams it to the rest of the flow, while adding the path metadata in the following inbound properties:
void setConfigurationEntries(Map<String, String> configurationEntries)
void setConfigurationResources(List<String> configurationResources)
void setDefaultFileSystemName(String defaultFileSystemName)
void setFileSystem(FileSystem fileSystem)
void writeToPath(String path, String permission, boolean overwrite, int bufferSize, int replication, long blockSize, String ownerUserName, String ownerGroupName, InputStream payload)
Write the current payload to the designated path, either creating a new file or appending to an existing one.
[Expand]
Inherited Methods
From class java.lang.Object

Constants

public static final String HDFS

Constant Value: "hdfs"

public static final String HDFS_CONTENT_SUMMARY

Constant Value: "hdfs.content.summary"

public static final String HDFS_FILE_CHECKSUM

Constant Value: "hdfs.file.checksum"

public static final String HDFS_FILE_STATUS

Constant Value: "hdfs.file.status"

public static final String HDFS_PATH_EXISTS

Constant Value: "hdfs.path.exists"

Fields

private static final Logger LOGGER

private Map<String, String> configurationEntries

A Map of configuration entries to be used by the HDFS client.

private List<String> configurationResources

A List of configuration resource files to be loaded by the HDFS client.

private String defaultFileSystemName

The name of the file system to connect to. It is passed to HDFS client as the {@value FileSystem#FS_DEFAULT_NAME_KEY} configuration entry. It can be overriden by values in configurationResources and configurationEntries.

private FileSystem fileSystem

A readily configured FileSystem to use to connect to HDFS.

Public Constructors

public HdfsConnector ()

Public Methods

public void appendToPath (String path, int bufferSize, InputStream payload)

Append the current payload to a file located at the designated path. Note: by default the Hadoop server has the append option disabled. In order to be able append any data to an existing file refer to dfs.support.append configuration parameter

Parameters
path The path of the file to write to.
bufferSize The buffer size to use when appending to the file.
payload The payload to append to the file.
Throws
Exception if any issue occurs during the execution.

public void connect (String connectionKey)

Establish the connection to the Hadoop Distributed File System.

Parameters
connectionKey A connection key.
Throws
ConnectionException Holding one of the possible values in ConnectionExceptionCode.

public void deleteDirectory (String path)

Delete the file or directory located at the designated path.

Parameters
path The path of the directory to delete.
Throws
Exception if any issue occurs during the execution.

public void deleteFile (String path)

Delete the file or directory located at the designated path.

Parameters
path The path of the file to delete.
Throws
Exception if any issue occurs during the execution.

public void disconnect ()

Disconnect from the Hadoop Distributed File System.

Throws
IOException if there is an issue connecting with the file system.

public Map<String, String> getConfigurationEntries ()

public List<String> getConfigurationResources ()

public String getDefaultFileSystemName ()

public FileSystem getFileSystem ()

public String getFileSystemUri ()

public void getPathMetaData (String path, MuleEvent muleEvent)

Get the metadata of a path, as described in readFromPath(String, int, SourceCallback), and store it in flow variables.

This flow variables are:

  • hdfs.path.exists - Indicates if the path exists (true or false)
  • hdfs.content.summary - A resume of the path info
  • hdfs.file.checksum - MD5 digest of the file (if it is a file and exists)
  • hdfs.file.status - A Hadoop object that contains info about the status of the file (org.apache.hadoop.fs.FileStatus

Parameters
path The path whose existence must be checked.
muleEvent The MuleEvent currently being processed.
Returns
  • the result of executing the next message processors if the path exists, otherwise null.
Throws
Exception if any issue occurs during the execution.

public boolean isConnected ()

Are we connected?

Returns
  • boolean true if the connection is still valid or false otherwise.

public void makeDirectories (String path, String permission)

Make the given file and all non-existent parents into directories. Has the semantics of Unix 'mkdir -p'. Existence of the directory hierarchy is not an error.

Parameters
path The path to create directories for.
permission The file system permission to use when creating the directories, either in octal or symbolic format (umask).
Throws
Exception if any issue occurs during the execution.

public Object readFromPath (String path, int bufferSize, SourceCallback sourceCallback)

Read the content of a file designated by its path and streams it to the rest of the flow, while adding the path metadata in the following inbound properties:

Parameters
path The path of the file to read.
bufferSize The buffer size to use when reading the file.
sourceCallback The SourceCallback used to propagate the event to the rest of the flow.
Returns
  • the result from executing the rest of the flow.
Throws
Exception if any issue occurs during the execution.

public void setConfigurationEntries (Map<String, String> configurationEntries)

Parameters
configurationEntries

public void setConfigurationResources (List<String> configurationResources)

Parameters
configurationResources

public void setDefaultFileSystemName (String defaultFileSystemName)

Parameters
defaultFileSystemName

public void setFileSystem (FileSystem fileSystem)

Parameters
fileSystem

public void writeToPath (String path, String permission, boolean overwrite, int bufferSize, int replication, long blockSize, String ownerUserName, String ownerGroupName, InputStream payload)

Write the current payload to the designated path, either creating a new file or appending to an existing one.

Parameters
path The path of the file to write to.
permission The file system permission to use if a new file is created, either in octal or symbolic format (umask).
overwrite If a pre-existing file should be overwritten with the new content.
bufferSize The buffer size to use when appending to the file.
replication Block replication for the file.
blockSize The buffer size to use when appending to the file.
ownerUserName The username owner of the file.
ownerGroupName The group owner of the file.
payload The payload to write to the file.
Throws
Exception if any issue occurs during the execution.