org.apache.flink.runtime.io.network.partition.hybrid.tiered.common

Class TieredStorageUtils

java.lang.Object

org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageUtils

public class TieredStorageUtils
extends Object

Utils for reading from or writing to tiered storage.

Constructor Summary

Constructors

Constructor and Description
TieredStorageUtils()

Method Summary

Modifier and Type	Method and Description
static BufferCompressor	buildBufferCompressor(int bufferSizeBytes, org.apache.flink.configuration.Configuration configuration) Construct the BufferCompressor from configuration.
static boolean	canBeCompressed(Buffer buffer, BufferCompressor bufferCompressor) Whether the buffer can be compressed or not.
static Buffer	compressBufferIfPossible(Buffer buffer, BufferCompressor bufferCompressor) Try compress buffer if possible.
static ByteBuffer[]	generateBufferWithHeaders(List<org.apache.flink.api.java.tuple.Tuple2<Buffer,Integer>> bufferWithIndexes)
static int	getAccumulatorExclusiveBuffers() Get exclusive buffer number of accumulator.
static String	getDiskTierName()
static String	getMemoryTierName()
static int	getMinBuffersPerGate() Get the number of minimum buffers per input gate.
static int	getMinBuffersPerResultPartition() * Get the number of minimum buffers per result partition.
static float	getNumBuffersTriggerFlushRatio() When the number of buffers that have been requested exceeds this threshold, trigger the flushing operation in each TierProducerAgent.
static long	getPoolSizeCheckInterval() Get the pool size check interval.
static String	getRemoteTierName()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TieredStorageUtils

public TieredStorageUtils()

Method Detail

getNumBuffersTriggerFlushRatio

public static float getNumBuffersTriggerFlushRatio()

When the number of buffers that have been requested exceeds this threshold, trigger the flushing operation in each TierProducerAgent.

Returns:

flush ratio.

getAccumulatorExclusiveBuffers

public static int getAccumulatorExclusiveBuffers()

Get exclusive buffer number of accumulator.

The buffer number is used to compare with the subpartition number to determine the type of BufferAccumulator.

If the exclusive buffer number is larger than (subpartitionNum + 1), the accumulator will use HashBufferAccumulator. If the exclusive buffer number is equal to or smaller than (subpartitionNum + 1), the accumulator will use SortBufferAccumulator

Returns:

the buffer number.

getPoolSizeCheckInterval

public static long getPoolSizeCheckInterval()

Get the pool size check interval.

getMinBuffersPerGate

public static int getMinBuffersPerGate()

Get the number of minimum buffers per input gate. It is only used when taskmanager.network.hybrid-shuffle.memory-decoupling.enabled is set to true.

Returns:

the buffer number.

getMinBuffersPerResultPartition

public static int getMinBuffersPerResultPartition()

*

Get the number of minimum buffers per result partition.

Returns:

the buffer number.

getMemoryTierName

public static String getMemoryTierName()

getDiskTierName

public static String getDiskTierName()

getRemoteTierName

public static String getRemoteTierName()

generateBufferWithHeaders

public static ByteBuffer[] generateBufferWithHeaders(List<org.apache.flink.api.java.tuple.Tuple2<Buffer,Integer>> bufferWithIndexes)

compressBufferIfPossible

public static Buffer compressBufferIfPossible(Buffer buffer,
                                              BufferCompressor bufferCompressor)

Try compress buffer if possible.

canBeCompressed

public static boolean canBeCompressed(Buffer buffer,
                                      BufferCompressor bufferCompressor)

Whether the buffer can be compressed or not. Note that event is not compressed because it is usually small and the size can become even larger after compression.

buildBufferCompressor

public static BufferCompressor buildBufferCompressor(int bufferSizeBytes,
                                                     org.apache.flink.configuration.Configuration configuration)

Construct the BufferCompressor from configuration.

Note: This is just a workaround for released version as we can not change the interface of TierFactory.