org.apache.flink.runtime.io.network.partition.hybrid

Class HsMemoryDataManager

java.lang.Object

org.apache.flink.runtime.io.network.partition.hybrid.HsMemoryDataManager

All Implemented Interfaces:

HsMemoryDataManagerOperation, HsSpillingInfoProvider

public class HsMemoryDataManager
extends Object
implements HsSpillingInfoProvider, HsMemoryDataManagerOperation

This class is responsible for managing data in memory.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.runtime.io.network.partition.hybrid.HsSpillingInfoProvider

HsSpillingInfoProvider.ConsumeStatus, HsSpillingInfoProvider.ConsumeStatusWithId, HsSpillingInfoProvider.SpillStatus

Constructor Summary

Constructors

Constructor and Description
HsMemoryDataManager(int numSubpartitions, int bufferSize, BufferPool bufferPool, HsSpillingStrategy spillStrategy, HsFileDataIndex fileDataIndex, Path dataFilePath, BufferCompressor bufferCompressor, long poolSizeCheckInterval)

Method Summary

Modifier and Type	Method and Description
void	append(ByteBuffer record, int targetChannel, Buffer.DataType dataType) Append record to HsMemoryDataManager, It will be managed by HsSubpartitionMemoryDataManager witch it belongs to.
void	close() Close this HsMemoryDataManager, it means no data can append to memory and all buffer taken by this class will recycle.
Deque<BufferIndexAndChannel>	getBuffersInOrder(int subpartitionId, HsSpillingInfoProvider.SpillStatus spillStatus, HsSpillingInfoProvider.ConsumeStatusWithId consumeStatusWithId) Get all buffers with the expected status from the subpartition.
List<Integer>	getNextBufferIndexToConsume(HsConsumerId consumerId) Get all subpartition's next buffer index to consume of specific consumer.
int	getNumSubpartitions() Get the number of downstream consumers.
int	getNumTotalRequestedBuffers() Get total number of buffers requested from buffer pool.
int	getNumTotalUnSpillBuffers() Get total number of not decided to spill buffers.
int	getPoolSize() Get the current size of buffer pool.
void	markBufferReleasedFromFile(int subpartitionId, int bufferIndex) This method is called when buffer should mark as released in HsFileDataIndex.
void	onBufferConsumed(BufferIndexAndChannel consumedBuffer) This method is called when buffer is consumed.
void	onBufferFinished() This method is called when buffer is finished.
void	onConsumerReleased(int subpartitionId, HsConsumerId consumerId) This method is called when consumer is decided to released.
void	onDataAvailable(int subpartitionId, Collection<HsConsumerId> consumerIds) This method is called when subpartition data become available.
HsDataView	registerNewConsumer(int subpartitionId, HsConsumerId consumerId, HsSubpartitionConsumerInternalOperations viewOperations) Register HsSubpartitionConsumerInternalOperations to subpartitionViewOperationsMap.
BufferBuilder	requestBufferFromPool() Request buffer from buffer pool.
void	setOutputMetrics(HsOutputMetrics metrics)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HsMemoryDataManager

public HsMemoryDataManager(int numSubpartitions,
                           int bufferSize,
                           BufferPool bufferPool,
                           HsSpillingStrategy spillStrategy,
                           HsFileDataIndex fileDataIndex,
                           Path dataFilePath,
                           BufferCompressor bufferCompressor,
                           long poolSizeCheckInterval)
                    throws IOException

Throws:

IOException

Method Detail

append

public void append(ByteBuffer record,
                   int targetChannel,
                   Buffer.DataType dataType)
            throws IOException

Append record to HsMemoryDataManager, It will be managed by HsSubpartitionMemoryDataManager witch it belongs to.

Parameters:

record - to be managed by this class.

targetChannel - target subpartition of this record.

dataType - the type of this record. In other words, is it data or event.

Throws:

IOException

registerNewConsumer

public HsDataView registerNewConsumer(int subpartitionId,
                                      HsConsumerId consumerId,
                                      HsSubpartitionConsumerInternalOperations viewOperations)

Register HsSubpartitionConsumerInternalOperations to subpartitionViewOperationsMap. It is used to obtain the consumption progress of the subpartition.

close

public void close()

Close this HsMemoryDataManager, it means no data can append to memory and all buffer taken by this class will recycle.

setOutputMetrics

public void setOutputMetrics(HsOutputMetrics metrics)

getPoolSize

public int getPoolSize()

Description copied from interface: HsSpillingInfoProvider

Get the current size of buffer pool.

Specified by:

getPoolSize in interface HsSpillingInfoProvider

getNumSubpartitions

public int getNumSubpartitions()

Description copied from interface: HsSpillingInfoProvider

Get the number of downstream consumers.

Specified by:

getNumSubpartitions in interface HsSpillingInfoProvider

Returns:

Number of subpartitions.

getNumTotalRequestedBuffers

public int getNumTotalRequestedBuffers()

Description copied from interface: HsSpillingInfoProvider

Get total number of buffers requested from buffer pool.

Specified by:

getNumTotalRequestedBuffers in interface HsSpillingInfoProvider

getNumTotalUnSpillBuffers

public int getNumTotalUnSpillBuffers()

Description copied from interface: HsSpillingInfoProvider

Get total number of not decided to spill buffers.

Specified by:

getNumTotalUnSpillBuffers in interface HsSpillingInfoProvider

getBuffersInOrder

public Deque<BufferIndexAndChannel> getBuffersInOrder(int subpartitionId,
                                                      HsSpillingInfoProvider.SpillStatus spillStatus,
                                                      HsSpillingInfoProvider.ConsumeStatusWithId consumeStatusWithId)

Description copied from interface: HsSpillingInfoProvider

Get all buffers with the expected status from the subpartition.

Specified by:

getBuffersInOrder in interface HsSpillingInfoProvider

Parameters:

subpartitionId - target buffers belong to.

spillStatus - expected buffer spill status.

consumeStatusWithId - expected buffer consume status and consumer id.

Returns:

all buffers satisfy specific status of this subpartition, This queue must be sorted according to bufferIndex from small to large, in other words, head is the buffer with the minimum bufferIndex in the current subpartition.

getNextBufferIndexToConsume

public List<Integer> getNextBufferIndexToConsume(HsConsumerId consumerId)

Description copied from interface: HsSpillingInfoProvider

Get all subpartition's next buffer index to consume of specific consumer.

Specified by:

getNextBufferIndexToConsume in interface HsSpillingInfoProvider

Parameters:

consumerId - of the target downstream consumer.

Returns:

A list containing all subpartition's next buffer index to consume of specific consumer, if the downstream subpartition view has not been registered, the corresponding return value is -1.

markBufferReleasedFromFile

public void markBufferReleasedFromFile(int subpartitionId,
                                       int bufferIndex)

Description copied from interface: HsMemoryDataManagerOperation

This method is called when buffer should mark as released in HsFileDataIndex.

Specified by:

markBufferReleasedFromFile in interface HsMemoryDataManagerOperation

Parameters:

subpartitionId - the subpartition that target buffer belong to.

bufferIndex - index of buffer to mark as released.

requestBufferFromPool

public BufferBuilder requestBufferFromPool()
                                    throws InterruptedException

Description copied from interface: HsMemoryDataManagerOperation

Request buffer from buffer pool.

Specified by:

requestBufferFromPool in interface HsMemoryDataManagerOperation

Returns:

requested buffer.

Throws:

InterruptedException

onBufferConsumed

public void onBufferConsumed(BufferIndexAndChannel consumedBuffer)

Description copied from interface: HsMemoryDataManagerOperation

This method is called when buffer is consumed.

Specified by:

onBufferConsumed in interface HsMemoryDataManagerOperation

Parameters:

consumedBuffer - target buffer to mark as consumed.

onBufferFinished

public void onBufferFinished()

Description copied from interface: HsMemoryDataManagerOperation

This method is called when buffer is finished.

Specified by:

onBufferFinished in interface HsMemoryDataManagerOperation

onDataAvailable

public void onDataAvailable(int subpartitionId,
                            Collection<HsConsumerId> consumerIds)

Description copied from interface: HsMemoryDataManagerOperation

This method is called when subpartition data become available.

Specified by:

onDataAvailable in interface HsMemoryDataManagerOperation

Parameters:

subpartitionId - the subpartition's identifier that this consumer belongs to.

consumerIds - the consumer's identifier which need notify data available.

onConsumerReleased

public void onConsumerReleased(int subpartitionId,
                               HsConsumerId consumerId)

Description copied from interface: HsMemoryDataManagerOperation

This method is called when consumer is decided to released.

Specified by:

onConsumerReleased in interface HsMemoryDataManagerOperation

Parameters:

subpartitionId - the subpartition's identifier that this consumer belongs to.

consumerId - the consumer's identifier which decided to be released.