Package org.apache.flink.runtime.io.network.buffer

Class NetworkBufferPool

  • All Implemented Interfaces:
    org.apache.flink.core.memory.MemorySegmentProvider, AvailabilityProvider, BufferPoolFactory
    public class NetworkBufferPool
extends Object
implements BufferPoolFactory, org.apache.flink.core.memory.MemorySegmentProvider, AvailabilityProvider
    The NetworkBufferPool is a fixed size pool of MemorySegment instances for the network stack.

    The NetworkBufferPool creates LocalBufferPools from which the individual tasks draw the buffers for the network data transfer. When new local buffer pools are created, the NetworkBufferPool dynamically redistributes the buffers between the pools.

    • Constructor Detail

      • NetworkBufferPool

        @VisibleForTesting
public NetworkBufferPool​(int numberOfSegmentsToAllocate,
                         int segmentSize)

      • NetworkBufferPool

        public NetworkBufferPool​(int numberOfSegmentsToAllocate,
                         int segmentSize,
                         Duration requestSegmentsTimeout)
        Allocates all MemorySegment instances managed by this pool.

    • Method Detail

      • requestPooledMemorySegment

        @Nullable
public org.apache.flink.core.memory.MemorySegment requestPooledMemorySegment()
        Different from requestUnpooledMemorySegments(int) for unpooled segments allocation. This method and the below requestPooledMemorySegmentsBlocking(int) method are designed to be used from LocalBufferPool for pooled memory segments allocation. Note that these methods for pooled memory segments requesting and recycling are prohibited from acquiring the factoryLock to avoid deadlock.

      • requestPooledMemorySegmentsBlocking

        public List<org.apache.flink.core.memory.MemorySegment> requestPooledMemorySegmentsBlocking​(int numberOfSegmentsToRequest)
                                                                                     throws IOException
        Throws:
        IOException

      • requestUnpooledMemorySegments

        public List<org.apache.flink.core.memory.MemorySegment> requestUnpooledMemorySegments​(int numberOfSegmentsToRequest)
                                                                               throws IOException
        Unpooled memory segments are requested directly from NetworkBufferPool, as opposed to pooled segments, that are requested through BufferPool that was created from this NetworkBufferPool (see createBufferPool(int, int)). They are used for example for exclusive RemoteInputChannel credits, that are permanently assigned to that channel, and never returned to any BufferPool. As opposed to pooled segments, when requested, unpooled segments needs to be accounted against numTotalRequiredBuffers, which might require redistribution of the segments.
        Specified by:
        requestUnpooledMemorySegments in interface org.apache.flink.core.memory.MemorySegmentProvider
        Throws:
        IOException

      • recycleUnpooledMemorySegments

        public void recycleUnpooledMemorySegments​(Collection<org.apache.flink.core.memory.MemorySegment> segments)
        Corresponding to requestUnpooledMemorySegments(int), this method is for unpooled memory segments recycling.
        Specified by:
        recycleUnpooledMemorySegments in interface org.apache.flink.core.memory.MemorySegmentProvider

      • destroy

        public void destroy()

      • isDestroyed

        public boolean isDestroyed()

      • getTotalNumberOfMemorySegments

        public int getTotalNumberOfMemorySegments()

      • getTotalMemory

        public long getTotalMemory()

      • getNumberOfAvailableMemorySegments

        public int getNumberOfAvailableMemorySegments()

      • getAvailableMemory

        public long getAvailableMemory()

      • getNumberOfUsedMemorySegments

        public int getNumberOfUsedMemorySegments()

      • getUsedMemory

        public long getUsedMemory()

      • getNumberOfRegisteredBufferPools

        public int getNumberOfRegisteredBufferPools()

      • getEstimatedNumberOfRequestedMemorySegments

        public long getEstimatedNumberOfRequestedMemorySegments()

      • getEstimatedRequestedMemory

        public long getEstimatedRequestedMemory()

      • getEstimatedRequestedSegmentsUsage

        public int getEstimatedRequestedSegmentsUsage()

      • maybeLogUsageWarning

        public void maybeLogUsageWarning()

      • countBuffers

        public int countBuffers()

      • getAvailableFuture

        public CompletableFuture<?> getAvailableFuture()
        Returns a future that is completed when there are free segments in this pool.
        Specified by:
        getAvailableFuture in interface AvailabilityProvider
        Returns:
        a future that is completed if the respective provider is available.

      • createBufferPool

        public BufferPool createBufferPool​(int numRequiredBuffers,
                                   int maxUsedBuffers)
                            throws IOException
        Description copied from interface: BufferPoolFactory
        Tries to create a buffer pool, which is guaranteed to provide at least the number of required buffers.

        The buffer pool is of dynamic size with at least numRequiredBuffers buffers.

        Specified by:
        createBufferPool in interface BufferPoolFactory
        Parameters:
        numRequiredBuffers - minimum number of network buffers in this pool
        maxUsedBuffers - maximum number of network buffers this pool offers
        Throws:
        IOException

      • createBufferPool

        public BufferPool createBufferPool​(int numRequiredBuffers,
                                   int maxUsedBuffers,
                                   int numSubpartitions,
                                   int maxBuffersPerChannel,
                                   int maxOverdraftBuffersPerGate)
                            throws IOException
        Description copied from interface: BufferPoolFactory
        Tries to create a buffer pool with an owner, which is guaranteed to provide at least the number of required buffers.

        The buffer pool is of dynamic size with at least numRequiredBuffers buffers.

        Specified by:
        createBufferPool in interface BufferPoolFactory
        Parameters:
        numRequiredBuffers - minimum number of network buffers in this pool
        maxUsedBuffers - maximum number of network buffers this pool offers
        numSubpartitions - number of subpartitions in this pool
        maxBuffersPerChannel - maximum number of buffers to use for each channel
        maxOverdraftBuffersPerGate - maximum number of overdraft buffers to use for each gate
        Throws:
        IOException

      • destroyAllBufferPools

        public void destroyAllBufferPools()
        Destroys all buffer pools that allocate their buffers from this buffer pool (created via createBufferPool(int, int)).