org.apache.flink.contrib.streaming.state

Class RocksDBMemoryControllerUtils

java.lang.Object

org.apache.flink.contrib.streaming.state.RocksDBMemoryControllerUtils

public class RocksDBMemoryControllerUtils
extends Object

Utils to create Cache and WriteBufferManager which are used to control total memory usage of RocksDB.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	RocksDBMemoryControllerUtils.RocksDBMemoryFactory Factory for Write Buffer Manager and Bock Cache.

Constructor Summary

Constructors

Constructor and Description
RocksDBMemoryControllerUtils()

Method Summary

Modifier and Type	Method and Description
static org.apache.flink.contrib.streaming.state.RocksDBSharedResources	allocateRocksDBSharedResources(long totalMemorySize, double writeBufferRatio, double highPriorityPoolRatio, boolean usingPartitionedIndexFilters, RocksDBMemoryControllerUtils.RocksDBMemoryFactory factory) Allocate memory controllable RocksDB shared resources.
static long	calculateActualCacheCapacity(long totalMemorySize, double writeBufferRatio) Calculate the actual memory capacity of cache, which would be shared among rocksDB instance(s).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RocksDBMemoryControllerUtils

public RocksDBMemoryControllerUtils()

Method Detail

allocateRocksDBSharedResources

public static org.apache.flink.contrib.streaming.state.RocksDBSharedResources allocateRocksDBSharedResources(long totalMemorySize,
                                                                                                             double writeBufferRatio,
                                                                                                             double highPriorityPoolRatio,
                                                                                                             boolean usingPartitionedIndexFilters,
                                                                                                             RocksDBMemoryControllerUtils.RocksDBMemoryFactory factory)

Allocate memory controllable RocksDB shared resources.

Parameters:

totalMemorySize - The total memory limit size.

writeBufferRatio - The ratio of total memory which is occupied by write buffer manager.

highPriorityPoolRatio - The high priority pool ratio of cache.

factory - creates Write Buffer Manager and Bock Cache

Returns:

memory controllable RocksDB shared resources.

calculateActualCacheCapacity

@VisibleForTesting
public static long calculateActualCacheCapacity(long totalMemorySize,
                                                                   double writeBufferRatio)

Calculate the actual memory capacity of cache, which would be shared among rocksDB instance(s). We introduce this method because: a) We cannot create a strict capacity limit cache util FLINK-15532 resolved. b) Regardless of the memory usage of blocks pinned by RocksDB iterators, which is difficult to calculate and only happened when we iterator entries in RocksDBMapState, the overuse of memory is mainly occupied by at most half of the write buffer usage. (see the flush implementation of write buffer manager). Thus, we have four equations below: write_buffer_manager_memory = 1.5 * write_buffer_manager_capacity write_buffer_manager_memory = total_memory_size * write_buffer_ratio write_buffer_manager_memory + other_part = total_memory_size write_buffer_manager_capacity + other_part = cache_capacity And we would deduce the formula: cache_capacity = (3 - write_buffer_ratio) * total_memory_size / 3 write_buffer_manager_capacity = 2 * total_memory_size * write_buffer_ratio / 3

Parameters:

totalMemorySize - Total off-heap memory size reserved for RocksDB instance(s).

writeBufferRatio - The ratio of total memory size which would be reserved for write buffer manager and its over-capacity part.

Returns:

The actual calculated cache capacity.