org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage

Class HashBufferAccumulator

java.lang.Object

org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage.HashBufferAccumulator

All Implemented Interfaces:

AutoCloseable, BufferAccumulator, HashSubpartitionBufferAccumulatorContext

public class HashBufferAccumulator
extends Object
implements BufferAccumulator, HashSubpartitionBufferAccumulatorContext

The hash implementation of the BufferAccumulator. The BufferAccumulator receives the records from TieredStorageProducerClient and the records will accumulate and transform to finished buffers. The accumulated buffers will be transferred to the corresponding tier dynamically.

To avoid the buffer waiting deadlock between the subpartitions, the HashBufferAccumulator requires at least n buffers (n is the number of subpartitions) to make sure that each subpartition has at least one buffer to accumulate the receiving data. Once an accumulated buffer is finished, the buffer will be flushed immediately.

Note that this class need not be thread-safe, because it should only be accessed from the main thread.

Constructor Summary

Constructors

Constructor and Description
HashBufferAccumulator(int numSubpartitions, int bufferSize, TieredStorageMemoryManager memoryManager, boolean isPartialRecordAllowed)

Method Summary

Modifier and Type	Method and Description
void	close() Close the accumulator.
void	flushAccumulatedBuffers(TieredStorageSubpartitionId subpartitionId, Buffer accumulatedBuffer, int numRemainingConsecutiveBuffers) Flush the accumulated Buffers of the subpartition.
void	receive(ByteBuffer record, TieredStorageSubpartitionId subpartitionId, Buffer.DataType dataType, boolean isBroadcast) Receives the records from tiered store producer, these records will be accumulated and transformed into finished buffers.
BufferBuilder	requestBufferBlocking() Request BufferBuilder from the BufferPool.
void	setup(org.apache.flink.util.function.TriConsumer<TieredStorageSubpartitionId,Buffer,Integer> accumulatedBufferFlusher) Setup the accumulator.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HashBufferAccumulator

public HashBufferAccumulator(int numSubpartitions,
                             int bufferSize,
                             TieredStorageMemoryManager memoryManager,
                             boolean isPartialRecordAllowed)

Method Detail

setup

public void setup(org.apache.flink.util.function.TriConsumer<TieredStorageSubpartitionId,Buffer,Integer> accumulatedBufferFlusher)

Description copied from interface: BufferAccumulator

Setup the accumulator.

Specified by:

setup in interface BufferAccumulator

Parameters:

accumulatedBufferFlusher - accepts the accumulated buffers. The first field is the subpartition id, the second is the accumulated buffer to flush, and the third is the number of remaining buffers to be written consecutively to the same segment.

receive

public void receive(ByteBuffer record,
                    TieredStorageSubpartitionId subpartitionId,
                    Buffer.DataType dataType,
                    boolean isBroadcast)
             throws IOException

Description copied from interface: BufferAccumulator

Receives the records from tiered store producer, these records will be accumulated and transformed into finished buffers.

Note that when isBroadcast is true, for a broadcast-only partition, the subpartitionId value will always be 0. Conversely, for a non-broadcast-only partition, the subpartitionId value will range from 0 to the number of subpartitions.

Specified by:

receive in interface BufferAccumulator

Parameters:

record - the received record

subpartitionId - the subpartition id of the record

dataType - the data type of the record

isBroadcast - whether the record is a broadcast record

Throws:

IOException

close

public void close()

Description copied from interface: BufferAccumulator

Close the accumulator. This will flush all the remaining data and release all the resources.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface BufferAccumulator

requestBufferBlocking

public BufferBuilder requestBufferBlocking()

Description copied from interface: HashSubpartitionBufferAccumulatorContext

Request BufferBuilder from the BufferPool.

Specified by:

requestBufferBlocking in interface HashSubpartitionBufferAccumulatorContext

Returns:

the requested buffer

flushAccumulatedBuffers

public void flushAccumulatedBuffers(TieredStorageSubpartitionId subpartitionId,
                                    Buffer accumulatedBuffer,
                                    int numRemainingConsecutiveBuffers)

Description copied from interface: HashSubpartitionBufferAccumulatorContext

Flush the accumulated Buffers of the subpartition.

Specified by:

flushAccumulatedBuffers in interface HashSubpartitionBufferAccumulatorContext

Parameters:

subpartitionId - the subpartition id

accumulatedBuffer - the accumulated buffer

numRemainingConsecutiveBuffers - number of buffers that would be passed in the following invocations and should be written to the same segment as this one