Package org.apache.flink.state.rocksdb
Class RocksDBMemoryControllerUtils
- java.lang.Object
-
- org.apache.flink.state.rocksdb.RocksDBMemoryControllerUtils
-
public class RocksDBMemoryControllerUtils extends Object
Utils to createCache
andWriteBufferManager
which are used to control total memory usage of RocksDB.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static interface
RocksDBMemoryControllerUtils.RocksDBMemoryFactory
Factory for Write Buffer Manager and Bock Cache.
-
Constructor Summary
Constructors Constructor Description RocksDBMemoryControllerUtils()
-
Method Summary
All Methods Static Methods Concrete Methods Modifier and Type Method Description static org.apache.flink.state.rocksdb.RocksDBSharedResources
allocateRocksDBSharedResources(long totalMemorySize, double writeBufferRatio, double highPriorityPoolRatio, boolean usingPartitionedIndexFilters, RocksDBMemoryControllerUtils.RocksDBMemoryFactory factory)
Allocate memory controllable RocksDB shared resources.static long
calculateActualCacheCapacity(long totalMemorySize, double writeBufferRatio)
Calculate the actual memory capacity of cache, which would be shared among rocksDB instance(s).
-
-
-
Method Detail
-
allocateRocksDBSharedResources
public static org.apache.flink.state.rocksdb.RocksDBSharedResources allocateRocksDBSharedResources(long totalMemorySize, double writeBufferRatio, double highPriorityPoolRatio, boolean usingPartitionedIndexFilters, RocksDBMemoryControllerUtils.RocksDBMemoryFactory factory)
Allocate memory controllable RocksDB shared resources.- Parameters:
totalMemorySize
- The total memory limit size.writeBufferRatio
- The ratio of total memory which is occupied by write buffer manager.highPriorityPoolRatio
- The high priority pool ratio of cache.factory
- creates Write Buffer Manager and Bock Cache- Returns:
- memory controllable RocksDB shared resources.
-
calculateActualCacheCapacity
@VisibleForTesting public static long calculateActualCacheCapacity(long totalMemorySize, double writeBufferRatio)
Calculate the actual memory capacity of cache, which would be shared among rocksDB instance(s). We introduce this method because: a) We cannot create a strict capacity limit cache util FLINK-15532 resolved. b) Regardless of the memory usage of blocks pinned by RocksDB iterators, which is difficult to calculate and only happened when we iterator entries in RocksDBMapState, the overuse of memory is mainly occupied by at most half of the write buffer usage. (see the flush implementation of write buffer manager). Thus, we have four equations below: write_buffer_manager_memory = 1.5 * write_buffer_manager_capacity write_buffer_manager_memory = total_memory_size * write_buffer_ratio write_buffer_manager_memory + other_part = total_memory_size write_buffer_manager_capacity + other_part = cache_capacity And we would deduce the formula: cache_capacity = (3 - write_buffer_ratio) * total_memory_size / 3 write_buffer_manager_capacity = 2 * total_memory_size * write_buffer_ratio / 3- Parameters:
totalMemorySize
- Total off-heap memory size reserved for RocksDB instance(s).writeBufferRatio
- The ratio of total memory size which would be reserved for write buffer manager and its over-capacity part.- Returns:
- The actual calculated cache capacity.
-
-