org.apache.flink.formats.hadoop.bulk.committer

Class HadoopRenameFileCommitter

java.lang.Object

org.apache.flink.formats.hadoop.bulk.committer.HadoopRenameFileCommitter

All Implemented Interfaces:

HadoopFileCommitter

public class HadoopRenameFileCommitter
extends Object
implements HadoopFileCommitter

The Hadoop file committer that directly rename the in-progress file to the target file. For FileSystem like S3, renaming may lead to additional copies.

Constructor Summary

Constructors

Constructor and Description
HadoopRenameFileCommitter(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path targetFilePath)
HadoopRenameFileCommitter(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path targetFilePath, org.apache.hadoop.fs.Path inProgressPath)

Method Summary

Modifier and Type	Method and Description
void	commit() Commits the in-progress file to the target path.
void	commitAfterRecovery() Re-commits the in-progress file to the target path after fail-over.
org.apache.hadoop.fs.Path	getTargetFilePath() Gets the target path to commit to.
org.apache.hadoop.fs.Path	getTempFilePath() Gets the path of the intermediate file to commit.
void	preCommit() Prepares the intermediates file for committing.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopRenameFileCommitter

public HadoopRenameFileCommitter(org.apache.hadoop.conf.Configuration configuration,
                                 org.apache.hadoop.fs.Path targetFilePath)
                          throws IOException

Throws:

IOException

HadoopRenameFileCommitter

public HadoopRenameFileCommitter(org.apache.hadoop.conf.Configuration configuration,
                                 org.apache.hadoop.fs.Path targetFilePath,
                                 org.apache.hadoop.fs.Path inProgressPath)
                          throws IOException

Throws:

IOException

Method Detail

getTargetFilePath

public org.apache.hadoop.fs.Path getTargetFilePath()

Description copied from interface: HadoopFileCommitter

Gets the target path to commit to.

Specified by:

getTargetFilePath in interface HadoopFileCommitter

Returns:

The target path to commit to.

getTempFilePath

public org.apache.hadoop.fs.Path getTempFilePath()

Description copied from interface: HadoopFileCommitter

Gets the path of the intermediate file to commit.

Specified by:

getTempFilePath in interface HadoopFileCommitter

Returns:

The path of the intermediate file to commit.

preCommit

public void preCommit()

Description copied from interface: HadoopFileCommitter

Prepares the intermediates file for committing.

Specified by:

preCommit in interface HadoopFileCommitter

commit

public void commit()
            throws IOException

Description copied from interface: HadoopFileCommitter

Commits the in-progress file to the target path.

Specified by:

commit in interface HadoopFileCommitter

Throws:

IOException

commitAfterRecovery

public void commitAfterRecovery()
                         throws IOException

Description copied from interface: HadoopFileCommitter

Re-commits the in-progress file to the target path after fail-over.

Specified by:

commitAfterRecovery in interface HadoopFileCommitter

Throws:

IOException