Package org.apache.flink.streaming.api.functions.sink.filesystem

Class OutputStreamBasedPartFileWriter<IN,BucketID>

java.lang.Object

org.apache.flink.streaming.api.functions.sink.filesystem.AbstractPartFileWriter<IN,BucketID>

org.apache.flink.streaming.api.functions.sink.filesystem.OutputStreamBasedPartFileWriter<IN,BucketID>

Type Parameters:

IN - the element type

BucketID - the bucket type

All Implemented Interfaces:

CompactingFileWriter, InProgressFileWriter<IN,BucketID>, OutputStreamBasedCompactingFileWriter, PartFileInfo<BucketID>, RecordWiseCompactingFileWriter<IN>

Direct Known Subclasses:

RowWisePartWriter

public abstract class OutputStreamBasedPartFileWriter<IN,BucketID>
extends AbstractPartFileWriter<IN,BucketID>
implements OutputStreamBasedCompactingFileWriter

The base class for all the part file writer that use RecoverableFsDataOutputStream.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	OutputStreamBasedPartFileWriter.OutputStreamBasedInProgressFileRecoverable	The InProgressFileWriter.InProgressFileRecoverable implementation for OutputStreamBasedPartFileWriter.OutputStreamBasedBucketWriter.
static class	OutputStreamBasedPartFileWriter.OutputStreamBasedInProgressFileRecoverableSerializer	The serializer for OutputStreamBasedPartFileWriter.OutputStreamBasedInProgressFileRecoverable.
static class	OutputStreamBasedPartFileWriter.OutputStreamBasedPendingFileRecoverable	The InProgressFileWriter.PendingFileRecoverable implementation for OutputStreamBasedPartFileWriter.OutputStreamBasedBucketWriter.
static class	OutputStreamBasedPartFileWriter.OutputStreamBasedPendingFileRecoverableSerializer	The serializer for OutputStreamBasedPartFileWriter.OutputStreamBasedPendingFileRecoverable.

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.sink.filesystem.CompactingFileWriter

CompactingFileWriter.Type

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter

InProgressFileWriter.InProgressFileRecoverable, InProgressFileWriter.PendingFileRecoverable

Method Summary

Modifier and Type	Method	Description
OutputStream	asOutputStream()	Gets the output stream underlying the writer.
InProgressFileWriter.PendingFileRecoverable	closeForCommit()	Closes the writer and gets the InProgressFileWriter.PendingFileRecoverable of the written compacting file.
void	dispose()	Dispose the part file.
protected void	ensureWriteType(CompactingFileWriter.Type type)
long	getSize()
InProgressFileWriter.InProgressFileRecoverable	persist()

Methods inherited from class org.apache.flink.streaming.api.functions.sink.filesystem.AbstractPartFileWriter

getBucketId, getCreationTime, getLastUpdateTime, markWrite

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter

write, write

Method Detail

persist

public InProgressFileWriter.InProgressFileRecoverable persist()
                                                       throws IOException

Specified by:

persist in interface InProgressFileWriter<IN,BucketID>

Returns:

The state of the current part file.

Throws:

IOException - Thrown if persisting the part file fails.

closeForCommit

public InProgressFileWriter.PendingFileRecoverable closeForCommit()
                                                           throws IOException

Description copied from interface: CompactingFileWriter

Closes the writer and gets the InProgressFileWriter.PendingFileRecoverable of the written compacting file.

Specified by:

closeForCommit in interface CompactingFileWriter

Specified by:

closeForCommit in interface InProgressFileWriter<IN,BucketID>

Returns:

The state of the pending part file. Bucket uses this to commit the pending file.

Throws:

IOException - Thrown if an I/O error occurs.

dispose

public void dispose()

Description copied from interface: InProgressFileWriter

Dispose the part file.

Specified by:

dispose in interface InProgressFileWriter<IN,BucketID>

getSize

public long getSize()
             throws IOException

Specified by:

getSize in interface PartFileInfo<IN>

Returns:

The size of the currently open part file.

Throws:

IOException

asOutputStream

public OutputStream asOutputStream()
                            throws IOException

Description copied from interface: OutputStreamBasedCompactingFileWriter

Gets the output stream underlying the writer. The close method of the returned stream should never be called.

Specified by:

asOutputStream in interface OutputStreamBasedCompactingFileWriter

Returns:

The output stream to write the compacting file.

Throws:

IOException - Thrown if acquiring the stream fails.

ensureWriteType

protected void ensureWriteType(CompactingFileWriter.Type type)