org.mule.modules

mule-module-hdfs

config

Namespace	http://www.mulesoft.org/schema/mule/hdfs
Schema Location	http://www.mulesoft.org/schema/mule/hdfs/current/mule-hdfs.xsd (View Schema)
Schema Version	3.4
Minimum Mule Version	3.3.0

Module Overview

Hadoop Distributed File System (HDFS) Connector.

Summary

Configuration
<hdfs:config> Configure an instance of this module

Message Processors
<hdfs:append> Append the current payload to a file located at the designated path.
<hdfs:delete-directory> Delete the file or directory located at the designated path.
<hdfs:delete-file> Delete the file or directory located at the designated path.
<hdfs:get-metadata> Get the metadata of a path, as described in `readFromPath(String, int, SourceCallback)`, and store it in flow variables.
<hdfs:make-directories> Make the given file and all non-existent parents into directories.
<hdfs:read> Read the content of a file designated by its path and streams it to the rest of the flow, while adding the path metadata in the following inbound properties: `HDFS_PATH_EXISTS`: a boolean set to true if the path exists `HDFS_CONTENT_SUMMARY`: an instance of ContentSummary if the path exists.
<hdfs:write> Write the current payload to the designated path, either creating a new file or appending to an existing one.

Configuration

To use the this module within a flow the namespace to the module must be included. The resulting flow will look similar to the following:

<mule xmlns="http://www.mulesoft.org/schema/mule/core"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xmlns:hdfs="http://www.mulesoft.org/schema/mule/hdfs"
      xsi:schemaLocation="
               http://www.mulesoft.org/schema/mule/core
               http://www.mulesoft.org/schema/mule/core/current/mule.xsd
               http://www.mulesoft.org/schema/mule/hdfs
               http://www.mulesoft.org/schema/mule/hdfs/current/mule-hdfs.xsd">

      <!-- here goes your flows and configuration elements -->

</mule>

This module is configured using the config element. This element must be placed outside of your flows and at the root of your Mule application. You can create as many configurations as you deem necessary as long as each carries its own name.

Each message processor, message source or transformer carries a config-ref attribute that allows the invoker to specify which configuration to use.

Attributes
Type	Name	Default Value	Description	Java Type	MIME Type	Encoding
xs:string	name		Optional. Give a name to this configuration so it can be later referenced.
	configurationEntries		Optional. A Map of configuration entries to be used by the HDFS client.
	configurationResources		Optional. A List of configuration resource files to be loaded by the HDFS client.
xs:string	defaultFileSystemName		Optional. The name of the file system to connect to.
	fileSystem		Optional. A readily configured FileSystem to use to connect to HDFS.

Example

INCLUDE_ERROR

Example

INCLUDE_ERROR

Example

INCLUDE_ERROR

Connection Pool

This connector offers automatic connection management via the use of a connection pool. The pool will act a storage mechanism for all the connections that are in-use by the user of this connector.

Prior to execution of a processor, the connector will attempt to lookup an already established connection and if one doesn't exists it will create one. That lookup mechanism is done in the connection pool via the use of connection variables declared as keys.

The user of the connector can configure the pool by adding a connection-pooling-profile to the connector configuration like this:

    <hdfs:connection-pooling-profile maxActive="10" maxIdle="10"
                             exhaustedAction="WHEN_EXHAUSTED_GROW" maxWait="120" minEvictionMillis="60000" evictionCheckIntervalMillis="30000"/>

The following is a list of connection attributes, each connection attribute can be configured at the config element level or they can also be added to each processor. If they are used at the processor level they get the benefit of full expression resolution.

Connection Attributes
Name	Default Value	Description	Java Type	MIME Type	Encoding
config-ref		Optional. Specify which configuration to use.
connectionKey	DEFAULT	A connection key.	String	/	UTF-8

Reconnection Strategies

Reconnection Strategies specify how a connector behaves when its connection fails. You can control how Mule attempts to reconnect by specifying a number of criteria:

The type of exception
Number and frequency of reconnection attempts
The notifications generated, and more

With a reconnection strategy, you can better control the behavior of a failed connection, by configuring it, for example, to re-attempt the connection only once every 15 minutes, and to give up after 30 attempts. You can also send an automatic notification to your IT administrator whenever this reconnection strategy goes into effect. You can even define a strategy that attempts to reconnect only during business hours. Such a setting can prove useful if your server is frequently shut down for nightly maintenance.

Configuration

A reconnection strategy that allows the user to configure how many times a reconnection should be attempted and how long to wait between attempts.

    <hdfs:config>
         <reconnect count="5" frequency="1000"/>
    </hdfs:config>

Reconnect Attributes
Name	Default Value	Description
frequency	2000	Optional. How often (in ms) to reconnect
count	2	Optional. How many reconnection attempts to make

For more information about reconnection strategies in Mule, or even how to write your own custom reconnection strategy please check this section.

Message Processors

<hdfs:append>

Append the current payload to a file located at the designated path. Note: by default the Hadoop server has the append option disabled. In order to be able append any data to an existing file refer to dfs.support.append configuration parameter

XML Sample

INCLUDE_ERROR

XML Sample

INCLUDE_ERROR

Attributes

Name	Default Value	Description	Java Type	MIME Type	Encoding
config-ref		Optional. Specify which configuration to use.
path		The path of the file to write to.	String	/	UTF-8
bufferSize	4096	Optional. The buffer size to use when appending to the file.	int	/
payload		The payload to append to the file.	InputStream	/
Connection Parameters This are only required if you didn't specified them at the configuration element. They are also useful for overriding the values of the configurations or even if you need to extract them from the Mule message since they support expression evaluation.
connectionKey	DEFAULT	Optional. A connection key.	String	/	UTF-8

Exception Payloads

Payload Class	Description
Exception	if any issue occurs during the execution.

<hdfs:delete-directory>

Delete the file or directory located at the designated path.

XML Sample

INCLUDE_ERROR

Attributes

Name	Default Value	Description	Java Type	MIME Type	Encoding
config-ref		Optional. Specify which configuration to use.
path		The path of the directory to delete.	String	/	UTF-8
Connection Parameters This are only required if you didn't specified them at the configuration element. They are also useful for overriding the values of the configurations or even if you need to extract them from the Mule message since they support expression evaluation.
connectionKey	DEFAULT	Optional. A connection key.	String	/	UTF-8

Exception Payloads

Payload Class	Description
Exception	if any issue occurs during the execution.

<hdfs:delete-file>

Delete the file or directory located at the designated path.

XML Sample

INCLUDE_ERROR

Attributes

Name	Default Value	Description	Java Type	MIME Type	Encoding
config-ref		Optional. Specify which configuration to use.
path		The path of the file to delete.	String	/	UTF-8
Connection Parameters This are only required if you didn't specified them at the configuration element. They are also useful for overriding the values of the configurations or even if you need to extract them from the Mule message since they support expression evaluation.
connectionKey	DEFAULT	Optional. A connection key.	String	/	UTF-8

Exception Payloads

Payload Class	Description
Exception	if any issue occurs during the execution.

<hdfs:get-metadata>

Get the metadata of a path, as described in readFromPath(String, int, SourceCallback), and store it in flow variables.

This flow variables are:

hdfs.path.exists - Indicates if the path exists (true or false)
hdfs.content.summary - A resume of the path info
hdfs.file.checksum - MD5 digest of the file (if it is a file and exists)
hdfs.file.status - A Hadoop object that contains info about the status of the file (org.apache.hadoop.fs.FileStatus

XML Sample

INCLUDE_ERROR

Attributes

Name	Default Value	Description	Java Type	MIME Type	Encoding
config-ref		Optional. Specify which configuration to use.
path		The path whose existence must be checked.	String	/	UTF-8
muleEvent		The MuleEvent currently being processed.	MuleEvent	/
Connection Parameters This are only required if you didn't specified them at the configuration element. They are also useful for overriding the values of the configurations or even if you need to extract them from the Mule message since they support expression evaluation.
connectionKey	DEFAULT	Optional. A connection key.	String	/	UTF-8

Returns

Return Type	Description
void	the result of executing the next message processors if the path exists, otherwise null.

Exception Payloads

Payload Class	Description
Exception	if any issue occurs during the execution.

<hdfs:make-directories>

Make the given file and all non-existent parents into directories. Has the semantics of Unix 'mkdir -p'. Existence of the directory hierarchy is not an error.

XML Sample

INCLUDE_ERROR

XML Sample

INCLUDE_ERROR

Attributes

Name	Default Value	Description	Java Type	MIME Type	Encoding
config-ref		Optional. Specify which configuration to use.
path		The path to create directories for.	String	/	UTF-8
permission		Optional. The file system permission to use when creating the directories, either in octal or symbolic format (umask).	String	/	UTF-8
Connection Parameters This are only required if you didn't specified them at the configuration element. They are also useful for overriding the values of the configurations or even if you need to extract them from the Mule message since they support expression evaluation.
connectionKey	DEFAULT	Optional. A connection key.	String	/	UTF-8

Exception Payloads

Payload Class	Description
Exception	if any issue occurs during the execution.

<hdfs:read>

Read the content of a file designated by its path and streams it to the rest of the flow, while adding the path metadata in the following inbound properties:

HDFS_PATH_EXISTS: a boolean set to true if the path exists
HDFS_CONTENT_SUMMARY: an instance of ContentSummary if the path exists.
HDFS_FILE_STATUS: an instance of FileStatus if the path exists.
HDFS_FILE_CHECKSUM: an instance of FileChecksum if the path exists, is a file and has a checksum.

XML Sample

INCLUDE_ERROR

XML Sample

INCLUDE_ERROR

Attributes

Name	Default Value	Description	Java Type	MIME Type	Encoding
config-ref		Optional. Specify which configuration to use.
path		The path of the file to read.	String	/	UTF-8
bufferSize	4096	Optional. The buffer size to use when reading the file.	int	/
sourceCallback		The SourceCallback used to propagate the event to the rest of the flow.	SourceCallback	/
Connection Parameters This are only required if you didn't specified them at the configuration element. They are also useful for overriding the values of the configurations or even if you need to extract them from the Mule message since they support expression evaluation.
connectionKey	DEFAULT	Optional. A connection key.	String	/	UTF-8

Returns

Return Type	Description
Object	the result from executing the rest of the flow.

Exception Payloads

Payload Class	Description
Exception	if any issue occurs during the execution.

<hdfs:write>

Write the current payload to the designated path, either creating a new file or appending to an existing one.

XML Sample

INCLUDE_ERROR

XML Sample

INCLUDE_ERROR

Attributes

Name	Default Value	Description	Java Type	MIME Type	Encoding
config-ref		Optional. Specify which configuration to use.
path		The path of the file to write to.	String	/	UTF-8
permission		Optional. The file system permission to use if a new file is created, either in octal or symbolic format (umask).	String	/	UTF-8
overwrite	true	Optional. If a pre-existing file should be overwritten with the new content.	boolean	/
bufferSize	4096	Optional. The buffer size to use when appending to the file.	int	/
replication	1	Optional. Block replication for the file.	int	/
blockSize	4096	Optional. The buffer size to use when appending to the file.	long	/
ownerUserName		Optional. The username owner of the file.	String	/	UTF-8
ownerGroupName		Optional. The group owner of the file.	String	/	UTF-8
payload		The payload to write to the file.	InputStream	/
Connection Parameters This are only required if you didn't specified them at the configuration element. They are also useful for overriding the values of the configurations or even if you need to extract them from the Mule message since they support expression evaluation.
connectionKey	DEFAULT	Optional. A connection key.	String	/	UTF-8

Exception Payloads

Payload Class	Description
Exception	if any issue occurs during the execution.

mule-module-hdfs

Module Overview

Summary

Configuration

Example

Example

Example

Connection Pool

Reconnection Strategies

Configuration

Message Processors

<hdfs:append>

XML Sample

XML Sample

Attributes

Exception Payloads

<hdfs:delete-directory>

XML Sample

Attributes

Exception Payloads

<hdfs:delete-file>

XML Sample

Attributes

Exception Payloads

<hdfs:get-metadata>

XML Sample

Attributes

Returns

Exception Payloads

<hdfs:make-directories>

XML Sample

XML Sample

Attributes

Exception Payloads

<hdfs:read>

XML Sample

XML Sample

Attributes

Returns

Exception Payloads

<hdfs:write>

XML Sample

XML Sample

Attributes

Exception Payloads

Message Sources

Transformers