Package org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage

Class TieredStorageMemoryManagerImpl

java.lang.Object

org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage.TieredStorageMemoryManagerImpl

All Implemented Interfaces:

TieredStorageMemoryManager

public class TieredStorageMemoryManagerImpl
extends Object
implements TieredStorageMemoryManager

The implementation for TieredStorageMemoryManager. This is to request or recycle buffers from LocalBufferPool for different memory owners, for example, the tiers, the buffer accumulator, etc.

Note that the TieredStorageMemorySpecs of the tiered storages should be ready when setting up the memory manager. Only after the setup process is finished, the tiered storage can request buffers from this manager.

Constructor Summary

Constructors

Constructor	Description
TieredStorageMemoryManagerImpl(float numTriggerReclaimBuffersRatio, boolean mayReclaimBuffer)	The constructor of the TieredStorageMemoryManagerImpl.

Method Summary

Modifier and Type	Method	Description
boolean	ensureCapacity(int numAdditionalBuffers)	Try best to reserve enough buffers that are guaranteed reclaimable along with the additional ones.
BufferPool	getBufferPool()	Expose and get the internal BufferPool.
int	getMaxNonReclaimableBuffers(Object owner)	Return the number of the non-reclaimable buffers for the owner.
void	listenBufferReclaimRequest(Runnable onBufferReclaimRequest)	Register a listener to listen the buffer reclaim request from the TieredStorageMemoryManager.
int	numOwnerRequestedBuffer(Object owner)	Return the number of requested buffers belonging to a specific owner.
void	release()	Release all the resources(if exists) and check the state of the TieredStorageMemoryManager.
BufferBuilder	requestBufferBlocking(Object owner)	Request a BufferBuilder instance for a specific owner.
void	setMetricGroup(TaskIOMetricGroup metricGroup)	Set the TaskIOMetricGroup for this memory manager.
void	setup(BufferPool bufferPool, List<TieredStorageMemorySpec> storageMemorySpecs)	Setup the TieredStorageMemoryManager.
void	transferBufferOwnership(Object oldOwner, Object newOwner, Buffer buffer)	Notify the memory manager that transferring one buffer's ownership from the old owner to the new owner.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TieredStorageMemoryManagerImpl

public TieredStorageMemoryManagerImpl(float numTriggerReclaimBuffersRatio,
                                      boolean mayReclaimBuffer)

The constructor of the TieredStorageMemoryManagerImpl.

Parameters:

numTriggerReclaimBuffersRatio - the buffer pool usage ratio of requesting each tiered storage to reclaim buffers

mayReclaimBuffer - indicate whether buffer reclaiming is supported

Method Detail

setup

public void setup(BufferPool bufferPool,
                  List<TieredStorageMemorySpec> storageMemorySpecs)

Description copied from interface: TieredStorageMemoryManager

Setup the TieredStorageMemoryManager. When setting up the manager, the TieredStorageMemorySpecs for different tiered storages should be ready to indicate each tiered storage's memory requirement specs.

Specified by:

setup in interface TieredStorageMemoryManager

Parameters:

bufferPool - the local buffer pool

storageMemorySpecs - the memory specs for different tiered storages

setMetricGroup

public void setMetricGroup(TaskIOMetricGroup metricGroup)

Description copied from interface: TieredStorageMemoryManager

Set the TaskIOMetricGroup for this memory manager.

Specified by:

setMetricGroup in interface TieredStorageMemoryManager

Parameters:

metricGroup - the metric group to set

listenBufferReclaimRequest

public void listenBufferReclaimRequest(Runnable onBufferReclaimRequest)

Description copied from interface: TieredStorageMemoryManager

Register a listener to listen the buffer reclaim request from the TieredStorageMemoryManager.

When the left buffers in the BufferPool are not enough, TieredStorageMemoryManager will try to reclaim the buffers from the memory owners.

Specified by:

listenBufferReclaimRequest in interface TieredStorageMemoryManager

Parameters:

onBufferReclaimRequest - a Runnable to process the buffer reclaim request

getBufferPool

public BufferPool getBufferPool()

Description copied from interface: TieredStorageMemoryManager

Expose and get the internal BufferPool. Please note that this method is a temporary workaround for the remote tier plugin and may be removed at any time in the future. We strongly advise that users do not rely on this method.

Specified by:

getBufferPool in interface TieredStorageMemoryManager

requestBufferBlocking

public BufferBuilder requestBufferBlocking(Object owner)

Description copied from interface: TieredStorageMemoryManager

Request a BufferBuilder instance for a specific owner. The TieredStorageMemoryManagerImpl will not check whether a buffer can be requested. The manager only records the number of requested buffers. If the buffers is not enough to meet the request, the manager will request each tiered storage to reclaim their requested buffers as much as possible.

This is not thread safe and is expected to be called only from the task thread.

Specified by:

requestBufferBlocking in interface TieredStorageMemoryManager

Parameters:

owner - the owner to request buffer

Returns:

the requested buffer

getMaxNonReclaimableBuffers

public int getMaxNonReclaimableBuffers(Object owner)

Description copied from interface: TieredStorageMemoryManager

Return the number of the non-reclaimable buffers for the owner.

Note that the available buffers are calculated dynamically based on some conditions, for example, the state of the BufferPool, the TieredStorageMemorySpec of the owner, etc. So the caller should always check before requesting non-reclaimable buffers.

When invoking this method, the caller should be aware that the return value may occasionally be negative. This is due to the possibility of the buffer pool size shrinking to a point where it is smaller than the buffers owned by other users. In such cases, the maximum non-reclaimable buffer value returned may be negative.

Specified by:

getMaxNonReclaimableBuffers in interface TieredStorageMemoryManager

ensureCapacity

public boolean ensureCapacity(int numAdditionalBuffers)

Description copied from interface: TieredStorageMemoryManager

Try best to reserve enough buffers that are guaranteed reclaimable along with the additional ones.

Note that the available buffers are calculated dynamically based on some conditions, for example, the state of the BufferPool, the TieredStorageMemorySpec of the owner, etc. So the caller should always ensure capacity before requesting non-reclaimable buffers.

Specified by:

ensureCapacity in interface TieredStorageMemoryManager

Parameters:

numAdditionalBuffers - the number of buffers that need to also be reserved in addition to guaranteed reclaimable buffers.

Returns:

True if the capacity meets the requirements, false otherwise.

numOwnerRequestedBuffer

public int numOwnerRequestedBuffer(Object owner)

Description copied from interface: TieredStorageMemoryManager

Return the number of requested buffers belonging to a specific owner.

Specified by:

numOwnerRequestedBuffer in interface TieredStorageMemoryManager

Parameters:

owner - the owner of requesting buffers

Returns:

the number of requested buffers belonging to the owner.

transferBufferOwnership

public void transferBufferOwnership(Object oldOwner,
                                    Object newOwner,
                                    Buffer buffer)

Description copied from interface: TieredStorageMemoryManager

Notify the memory manager that transferring one buffer's ownership from the old owner to the new owner.

Specified by:

transferBufferOwnership in interface TieredStorageMemoryManager

Parameters:

oldOwner - the old owner of one buffer

newOwner - the new owner of one buffer

buffer - the buffer to transfer the ownership

release

public void release()

Description copied from interface: TieredStorageMemoryManager

Release all the resources(if exists) and check the state of the TieredStorageMemoryManager.

Specified by:

release in interface TieredStorageMemoryManager