Package org.apache.flink.runtime.io.network.partition.hybrid.tiered.common

Class TieredStorageUtils

  • public class TieredStorageUtils
extends Object
    Utils for reading from or writing to tiered storage.

    • Constructor Detail

      • TieredStorageUtils

        public TieredStorageUtils()

    • Method Detail

      • getNumBuffersTriggerFlushRatio

        public static float getNumBuffersTriggerFlushRatio()
        When the number of buffers that have been requested exceeds this threshold, trigger the flushing operation in each TierProducerAgent.
        Returns:
        flush ratio.

      • getAccumulatorExclusiveBuffers

        public static int getAccumulatorExclusiveBuffers()
        Get exclusive buffer number of accumulator.

        The buffer number is used to compare with the subpartition number to determine the type of BufferAccumulator.

        If the exclusive buffer number is larger than (subpartitionNum + 1), the accumulator will use HashBufferAccumulator. If the exclusive buffer number is equal to or smaller than (subpartitionNum + 1), the accumulator will use SortBufferAccumulator

        Returns:
        the buffer number.

      • getPoolSizeCheckInterval

        public static long getPoolSizeCheckInterval()
        Get the pool size check interval.

      • getMinBuffersPerGate

        public static int getMinBuffersPerGate()
        Get the number of minimum buffers per input gate. It is only used when taskmanager.network.hybrid-shuffle.memory-decoupling.enabled is set to true.
        Returns:
        the buffer number.

      • getMinBuffersPerResultPartition

        public static int getMinBuffersPerResultPartition()
        *

        Get the number of minimum buffers per result partition.

        Returns:
        the buffer number.

      • getMemoryTierName

        public static String getMemoryTierName()

      • getDiskTierName

        public static String getDiskTierName()

      • getRemoteTierName

        public static String getRemoteTierName()

      • generateBufferWithHeaders

        public static ByteBuffer[] generateBufferWithHeaders​(List<org.apache.flink.api.java.tuple.Tuple2<Buffer,​Integer>> bufferWithIndexes)

      • compressBufferIfPossible

        public static Buffer compressBufferIfPossible​(Buffer buffer,
                                              BufferCompressor bufferCompressor)
        Try compress buffer if possible.

      • canBeCompressed

        public static boolean canBeCompressed​(Buffer buffer,
                                      BufferCompressor bufferCompressor)
        Whether the buffer can be compressed or not. Note that event is not compressed because it is usually small and the size can become even larger after compression.