org.apache.flink.runtime.io.network.partition.hybrid.tiered.file

Class SegmentPartitionFileWriter

java.lang.Object

org.apache.flink.runtime.io.network.partition.hybrid.tiered.file.SegmentPartitionFileWriter

All Implemented Interfaces:

PartitionFileWriter

public class SegmentPartitionFileWriter
extends Object
implements PartitionFileWriter

The implementation of PartitionFileWriter with segment file mode. In this mode, each segment of one subpartition is written to an independent file.

After finishing writing a segment, a segment-finish file is written to ensure the downstream reads only when the entire segment file is written, avoiding partial data reads. The downstream can determine if the current segment is complete by checking for the existence of the segment-finish file.

To minimize the number of files, each subpartition keeps only a single segment-finish file. For instance, if segment-finish file 5 exists, it indicates that segments 1 to 5 have all been finished.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.runtime.io.network.partition.hybrid.tiered.file.PartitionFileWriter

PartitionFileWriter.SegmentBufferContext, PartitionFileWriter.SubpartitionBufferContext

Method Summary

Modifier and Type	Method and Description
void	release() Release all the resources of the PartitionFileWriter.
CompletableFuture<Void>	write(TieredStoragePartitionId partitionId, List<PartitionFileWriter.SubpartitionBufferContext> buffersToWrite) Write the buffers to the partition file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

write

public CompletableFuture<Void> write(TieredStoragePartitionId partitionId,
                                     List<PartitionFileWriter.SubpartitionBufferContext> buffersToWrite)

Description copied from interface: PartitionFileWriter

Write the buffers to the partition file. The written buffers may belong to multiple subpartitions, but these buffers will be consecutive in the file.

Specified by:

write in interface PartitionFileWriter

Parameters:

partitionId - the partition id

buffersToWrite - the buffers to be written to the partition file

Returns:

the completable future indicating whether the writing file process has finished. If the CompletableFuture is completed, the written process is completed.

release

public void release()

Description copied from interface: PartitionFileWriter

Release all the resources of the PartitionFileWriter.

Specified by:

release in interface PartitionFileWriter