org.apache.flink.streaming.api.functions.sink.filesystem

Class BulkBucketWriter<IN,BucketID>

java.lang.Object

org.apache.flink.streaming.api.functions.sink.filesystem.BulkBucketWriter<IN,BucketID>

Type Parameters:

IN - The type of input elements.

BucketID - The type of ids for the buckets, as returned by the BucketAssigner.

All Implemented Interfaces:

BucketWriter<IN,BucketID>

@Internal
public class BulkBucketWriter<IN,BucketID>
extends Object

A factory that creates BulkPartWriters.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter

BucketWriter.PendingFile

Constructor Summary

Constructors

Constructor and Description
BulkBucketWriter(org.apache.flink.core.fs.RecoverableWriter recoverableWriter, org.apache.flink.api.common.serialization.BulkWriter.Factory<IN> writerFactory)

Method Summary

Modifier and Type	Method and Description
boolean	cleanupInProgressFileRecoverable(InProgressFileWriter.InProgressFileRecoverable inProgressFileRecoverable) Frees up any resources that were previously occupied in order to be able to recover from a (potential) failure.
WriterProperties	getProperties()
InProgressFileWriter<IN,BucketID>	openNew(BucketID bucketId, org.apache.flink.core.fs.RecoverableFsDataOutputStream stream, org.apache.flink.core.fs.Path path, long creationTime)
InProgressFileWriter<IN,BucketID>	openNewInProgressFile(BucketID bucketID, org.apache.flink.core.fs.Path path, long creationTime) Used to create a new InProgressFileWriter.
BucketWriter.PendingFile	recoverPendingFile(InProgressFileWriter.PendingFileRecoverable pendingFileRecoverable) Recovers a pending file for finalizing and committing.
InProgressFileWriter<IN,BucketID>	resumeFrom(BucketID bucketId, org.apache.flink.core.fs.RecoverableFsDataOutputStream stream, org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable resumable, long creationTime)
InProgressFileWriter<IN,BucketID>	resumeInProgressFileFrom(BucketID bucketID, InProgressFileWriter.InProgressFileRecoverable inProgressFileRecoverable, long creationTime) Used to resume a InProgressFileWriter from a InProgressFileWriter.InProgressFileRecoverable.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BulkBucketWriter

public BulkBucketWriter(org.apache.flink.core.fs.RecoverableWriter recoverableWriter,
                        org.apache.flink.api.common.serialization.BulkWriter.Factory<IN> writerFactory)
                 throws IOException

Throws:

IOException

Method Detail

resumeFrom

public InProgressFileWriter<IN,BucketID> resumeFrom(BucketID bucketId,
                                                    org.apache.flink.core.fs.RecoverableFsDataOutputStream stream,
                                                    org.apache.flink.core.fs.RecoverableWriter.ResumeRecoverable resumable,
                                                    long creationTime)
                                             throws IOException

Throws:

IOException

openNew

public InProgressFileWriter<IN,BucketID> openNew(BucketID bucketId,
                                                 org.apache.flink.core.fs.RecoverableFsDataOutputStream stream,
                                                 org.apache.flink.core.fs.Path path,
                                                 long creationTime)
                                          throws IOException

Throws:

IOException

openNewInProgressFile

public InProgressFileWriter<IN,BucketID> openNewInProgressFile(BucketID bucketID,
                                                               org.apache.flink.core.fs.Path path,
                                                               long creationTime)
                                                        throws IOException

Description copied from interface: BucketWriter

Used to create a new InProgressFileWriter.

Specified by:

openNewInProgressFile in interface BucketWriter<IN,BucketID>

Parameters:

bucketID - the id of the bucket this writer is writing to.

path - the path this writer will write to.

creationTime - the creation time of the file.

Returns:

the new InProgressFileWriter

Throws:

IOException - Thrown if creating a writer fails.

resumeInProgressFileFrom

public InProgressFileWriter<IN,BucketID> resumeInProgressFileFrom(BucketID bucketID,
                                                                  InProgressFileWriter.InProgressFileRecoverable inProgressFileRecoverable,
                                                                  long creationTime)
                                                           throws IOException

Description copied from interface: BucketWriter

Used to resume a InProgressFileWriter from a InProgressFileWriter.InProgressFileRecoverable.

Specified by:

resumeInProgressFileFrom in interface BucketWriter<IN,BucketID>

Parameters:

bucketID - the id of the bucket this writer is writing to.

inProgressFileRecoverable - the state of the part file.

creationTime - the creation time of the file.

Returns:

the resumed InProgressFileWriter

Throws:

IOException - Thrown if resuming a writer fails.

recoverPendingFile

public BucketWriter.PendingFile recoverPendingFile(InProgressFileWriter.PendingFileRecoverable pendingFileRecoverable)
                                            throws IOException

Description copied from interface: BucketWriter

Recovers a pending file for finalizing and committing.

Specified by:

recoverPendingFile in interface BucketWriter<IN,BucketID>

Parameters:

pendingFileRecoverable - The handle with the recovery information.

Returns:

A pending file

Throws:

IOException - Thrown if recovering a pending file fails.

cleanupInProgressFileRecoverable

public boolean cleanupInProgressFileRecoverable(InProgressFileWriter.InProgressFileRecoverable inProgressFileRecoverable)
                                         throws IOException

Description copied from interface: BucketWriter

Frees up any resources that were previously occupied in order to be able to recover from a (potential) failure.

NOTE: This operation should not throw an exception, but return false if the cleanup did not happen for any reason.

Specified by:

cleanupInProgressFileRecoverable in interface BucketWriter<IN,BucketID>

Parameters:

inProgressFileRecoverable - the InProgressFileWriter.InProgressFileRecoverable whose state we want to clean-up.

Returns:

true if the resources were successfully freed, false otherwise (e.g. the file to be deleted was not there for any reason - already deleted or never created).

Throws:

IOException - if an I/O error occurs

getProperties

public WriterProperties getProperties()

Specified by:

getProperties in interface BucketWriter<IN,BucketID>

Returns:

the property of the BucketWriter