org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage

Class SortBufferAccumulator

java.lang.Object

org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage.SortBufferAccumulator

All Implemented Interfaces:

AutoCloseable, BufferAccumulator

public class SortBufferAccumulator
extends Object
implements BufferAccumulator

The sort-based implementation of the BufferAccumulator. The BufferAccumulator receives the records from TieredStorageProducerClient and the records will accumulate and transform to finished buffers. The accumulated buffers will be transferred to the corresponding tier dynamically.

The SortBufferAccumulator can help use less buffers to accumulate data, which decouples the buffer usage with the number of parallelism. The number of buffers used by the SortBufferAccumulator will be numBuffers at most. Once the DataBuffer is full, or switching from broadcast to non-broadcast(or vice versa), the buffer in the sort buffer will be flushed to the tiers.

Note that this class need not be thread-safe, because it should only be accessed from the main thread.

Constructor Summary

Constructors

Constructor and Description
SortBufferAccumulator(int numSubpartitions, int numBuffers, int bufferSizeBytes, TieredStorageMemoryManager memoryManager, boolean isPartialRecordAllowed)

Method Summary

Modifier and Type	Method and Description
void	close() Close the accumulator.
void	receive(ByteBuffer record, TieredStorageSubpartitionId subpartitionId, Buffer.DataType dataType, boolean isBroadcast) Receives the records from tiered store producer, these records will be accumulated and transformed into finished buffers.
void	setup(org.apache.flink.util.function.TriConsumer<TieredStorageSubpartitionId,Buffer,Integer> bufferFlusher) Setup the accumulator.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SortBufferAccumulator

public SortBufferAccumulator(int numSubpartitions,
                             int numBuffers,
                             int bufferSizeBytes,
                             TieredStorageMemoryManager memoryManager,
                             boolean isPartialRecordAllowed)

Method Detail

setup

public void setup(org.apache.flink.util.function.TriConsumer<TieredStorageSubpartitionId,Buffer,Integer> bufferFlusher)

Description copied from interface: BufferAccumulator

Setup the accumulator.

Specified by:

setup in interface BufferAccumulator

Parameters:

bufferFlusher - accepts the accumulated buffers. The first field is the subpartition id, the second is the accumulated buffer to flush, and the third is the number of remaining buffers to be written consecutively to the same segment.

receive

public void receive(ByteBuffer record,
                    TieredStorageSubpartitionId subpartitionId,
                    Buffer.DataType dataType,
                    boolean isBroadcast)
             throws IOException

Description copied from interface: BufferAccumulator

Receives the records from tiered store producer, these records will be accumulated and transformed into finished buffers.

Note that when isBroadcast is true, for a broadcast-only partition, the subpartitionId value will always be 0. Conversely, for a non-broadcast-only partition, the subpartitionId value will range from 0 to the number of subpartitions.

Specified by:

receive in interface BufferAccumulator

Parameters:

record - the received record

subpartitionId - the subpartition id of the record

dataType - the data type of the record

isBroadcast - whether the record is a broadcast record

Throws:

IOException

close

public void close()

Description copied from interface: BufferAccumulator

Close the accumulator. This will flush all the remaining data and release all the resources.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface BufferAccumulator