Package | Description |
---|---|
org.apache.flink.contrib.streaming.state | |
org.apache.flink.contrib.streaming.state.restore | |
org.apache.flink.contrib.streaming.state.snapshot | |
org.apache.flink.runtime.checkpoint | |
org.apache.flink.runtime.jobmaster | |
org.apache.flink.runtime.query | |
org.apache.flink.runtime.scheduler | |
org.apache.flink.runtime.state | |
org.apache.flink.runtime.state.filesystem | |
org.apache.flink.runtime.state.heap |
This package contains the classes for key/value state backends that store the state
on the JVM heap as objects.
|
org.apache.flink.runtime.state.memory | |
org.apache.flink.runtime.taskexecutor.rpc |
Modifier and Type | Method and Description |
---|---|
static KeyedStateHandle |
RocksDBIncrementalCheckpointUtils.chooseTheBestStateHandleForInitial(Collection<KeyedStateHandle> restoreStateHandles,
KeyGroupRange targetKeyGroupRange)
Choose the best state handle according to the
RocksDBIncrementalCheckpointUtils.STATE_HANDLE_EVALUATOR
to init the initial db. |
static void |
RocksDBIncrementalCheckpointUtils.clipDBWithKeyGroupRange(org.rocksdb.RocksDB db,
List<org.rocksdb.ColumnFamilyHandle> columnFamilyHandles,
KeyGroupRange targetKeyGroupRange,
KeyGroupRange currentKeyGroupRange,
int keyGroupPrefixBytes,
long writeBatchSize)
The method to clip the db instance according to the target key group range using
the
RocksDB.delete(ColumnFamilyHandle, byte[]) . |
<K> AbstractKeyedStateBackend<K> |
RocksDBStateBackend.createKeyedStateBackend(Environment env,
JobID jobID,
String operatorIdentifier,
TypeSerializer<K> keySerializer,
int numberOfKeyGroups,
KeyGroupRange keyGroupRange,
TaskKvStateRegistry kvStateRegistry,
TtlTimeProvider ttlTimeProvider,
MetricGroup metricGroup,
Collection<KeyedStateHandle> stateHandles,
CloseableRegistry cancelStreamRegistry) |
Constructor and Description |
---|
RocksDBKeyedStateBackendBuilder(String operatorIdentifier,
ClassLoader userCodeClassLoader,
File instanceBasePath,
RocksDBResourceContainer optionsContainer,
java.util.function.Function<String,org.rocksdb.ColumnFamilyOptions> columnFamilyOptionsFactory,
TaskKvStateRegistry kvStateRegistry,
TypeSerializer<K> keySerializer,
int numberOfKeyGroups,
KeyGroupRange keyGroupRange,
ExecutionConfig executionConfig,
LocalRecoveryConfig localRecoveryConfig,
RocksDBStateBackend.PriorityQueueStateType priorityQueueStateType,
TtlTimeProvider ttlTimeProvider,
MetricGroup metricGroup,
Collection<KeyedStateHandle> stateHandles,
StreamCompressionDecorator keyGroupCompressionDecorator,
CloseableRegistry cancelStreamRegistry) |
Modifier and Type | Field and Description |
---|---|
protected KeyGroupRange |
AbstractRocksDBRestoreOperation.keyGroupRange |
Constructor and Description |
---|
AbstractRocksDBRestoreOperation(KeyGroupRange keyGroupRange,
int keyGroupPrefixBytes,
int numberOfTransferringThreads,
CloseableRegistry cancelStreamRegistry,
ClassLoader userCodeClassLoader,
Map<String,RocksDBKeyedStateBackend.RocksDbKvStateInfo> kvStateInformation,
StateSerializerProvider<K> keySerializerProvider,
File instanceBasePath,
File instanceRocksDBPath,
org.rocksdb.DBOptions dbOptions,
java.util.function.Function<String,org.rocksdb.ColumnFamilyOptions> columnFamilyOptionsFactory,
RocksDBNativeMetricOptions nativeMetricOptions,
MetricGroup metricGroup,
Collection<KeyedStateHandle> stateHandles,
RocksDbTtlCompactFiltersManager ttlCompactFiltersManager) |
RocksDBFullRestoreOperation(KeyGroupRange keyGroupRange,
int keyGroupPrefixBytes,
int numberOfTransferringThreads,
CloseableRegistry cancelStreamRegistry,
ClassLoader userCodeClassLoader,
Map<String,RocksDBKeyedStateBackend.RocksDbKvStateInfo> kvStateInformation,
StateSerializerProvider<K> keySerializerProvider,
File instanceBasePath,
File instanceRocksDBPath,
org.rocksdb.DBOptions dbOptions,
java.util.function.Function<String,org.rocksdb.ColumnFamilyOptions> columnFamilyOptionsFactory,
RocksDBNativeMetricOptions nativeMetricOptions,
MetricGroup metricGroup,
Collection<KeyedStateHandle> restoreStateHandles,
RocksDbTtlCompactFiltersManager ttlCompactFiltersManager,
long writeBatchSize) |
RocksDBIncrementalRestoreOperation(String operatorIdentifier,
KeyGroupRange keyGroupRange,
int keyGroupPrefixBytes,
int numberOfTransferringThreads,
CloseableRegistry cancelStreamRegistry,
ClassLoader userCodeClassLoader,
Map<String,RocksDBKeyedStateBackend.RocksDbKvStateInfo> kvStateInformation,
StateSerializerProvider<K> keySerializerProvider,
File instanceBasePath,
File instanceRocksDBPath,
org.rocksdb.DBOptions dbOptions,
java.util.function.Function<String,org.rocksdb.ColumnFamilyOptions> columnFamilyOptionsFactory,
RocksDBNativeMetricOptions nativeMetricOptions,
MetricGroup metricGroup,
Collection<KeyedStateHandle> restoreStateHandles,
RocksDbTtlCompactFiltersManager ttlCompactFiltersManager,
long writeBatchSize) |
RocksDBNoneRestoreOperation(KeyGroupRange keyGroupRange,
int keyGroupPrefixBytes,
int numberOfTransferringThreads,
CloseableRegistry cancelStreamRegistry,
ClassLoader userCodeClassLoader,
Map<String,RocksDBKeyedStateBackend.RocksDbKvStateInfo> kvStateInformation,
StateSerializerProvider<K> keySerializerProvider,
File instanceBasePath,
File instanceRocksDBPath,
org.rocksdb.DBOptions dbOptions,
java.util.function.Function<String,org.rocksdb.ColumnFamilyOptions> columnFamilyOptionsFactory,
RocksDBNativeMetricOptions nativeMetricOptions,
MetricGroup metricGroup,
Collection<KeyedStateHandle> restoreStateHandles,
RocksDbTtlCompactFiltersManager ttlCompactFiltersManager) |
Modifier and Type | Field and Description |
---|---|
protected KeyGroupRange |
RocksDBSnapshotStrategyBase.keyGroupRange
The key-group range for the task.
|
Constructor and Description |
---|
RocksDBSnapshotStrategyBase(String description,
org.rocksdb.RocksDB db,
ResourceGuard rocksDBResourceGuard,
TypeSerializer<K> keySerializer,
LinkedHashMap<String,RocksDBKeyedStateBackend.RocksDbKvStateInfo> kvStateInformation,
KeyGroupRange keyGroupRange,
int keyGroupPrefixBytes,
LocalRecoveryConfig localRecoveryConfig,
CloseableRegistry cancelStreamRegistry) |
RocksFullSnapshotStrategy(org.rocksdb.RocksDB db,
ResourceGuard rocksDBResourceGuard,
TypeSerializer<K> keySerializer,
LinkedHashMap<String,RocksDBKeyedStateBackend.RocksDbKvStateInfo> kvStateInformation,
KeyGroupRange keyGroupRange,
int keyGroupPrefixBytes,
LocalRecoveryConfig localRecoveryConfig,
CloseableRegistry cancelStreamRegistry,
StreamCompressionDecorator keyGroupCompressionDecorator) |
RocksIncrementalSnapshotStrategy(org.rocksdb.RocksDB db,
ResourceGuard rocksDBResourceGuard,
TypeSerializer<K> keySerializer,
LinkedHashMap<String,RocksDBKeyedStateBackend.RocksDbKvStateInfo> kvStateInformation,
KeyGroupRange keyGroupRange,
int keyGroupPrefixBytes,
LocalRecoveryConfig localRecoveryConfig,
CloseableRegistry cancelStreamRegistry,
File instanceBasePath,
UUID backendUID,
SortedMap<Long,Set<StateHandleID>> materializedSstFiles,
long lastCompletedCheckpointId,
int numberOfTransferingThreads) |
Modifier and Type | Method and Description |
---|---|
static List<KeyGroupRange> |
StateAssignmentOperation.createKeyGroupPartitions(int numberKeyGroups,
int parallelism)
Groups the available set of key groups into key group partitions.
|
Modifier and Type | Method and Description |
---|---|
static void |
StateAssignmentOperation.extractIntersectingState(Collection<? extends KeyedStateHandle> originalSubtaskStateHandles,
KeyGroupRange rangeToExtract,
List<KeyedStateHandle> extractedStateCollector)
Extracts certain key group ranges from the given state handles and adds them to the collector.
|
static List<KeyedStateHandle> |
StateAssignmentOperation.getManagedKeyedStateHandles(OperatorState operatorState,
KeyGroupRange subtaskKeyGroupRange)
Collect
managedKeyedStateHandles which have intersection with given
KeyGroupRange from operatorState |
static List<KeyedStateHandle> |
StateAssignmentOperation.getRawKeyedStateHandles(OperatorState operatorState,
KeyGroupRange subtaskKeyGroupRange)
|
Modifier and Type | Method and Description |
---|---|
CompletableFuture<Acknowledge> |
KvStateRegistryGateway.notifyKvStateRegistered(JobID jobId,
JobVertexID jobVertexId,
KeyGroupRange keyGroupRange,
String registrationName,
KvStateID kvStateId,
InetSocketAddress kvStateServerAddress)
Notifies that queryable state has been registered.
|
CompletableFuture<Acknowledge> |
JobMaster.notifyKvStateRegistered(JobID jobId,
JobVertexID jobVertexId,
KeyGroupRange keyGroupRange,
String registrationName,
KvStateID kvStateId,
InetSocketAddress kvStateServerAddress) |
CompletableFuture<Acknowledge> |
KvStateRegistryGateway.notifyKvStateUnregistered(JobID jobId,
JobVertexID jobVertexId,
KeyGroupRange keyGroupRange,
String registrationName)
Notifies that queryable state has been unregistered.
|
CompletableFuture<Acknowledge> |
JobMaster.notifyKvStateUnregistered(JobID jobId,
JobVertexID jobVertexId,
KeyGroupRange keyGroupRange,
String registrationName) |
Modifier and Type | Method and Description |
---|---|
void |
KvStateRegistryListener.notifyKvStateRegistered(JobID jobId,
JobVertexID jobVertexId,
KeyGroupRange keyGroupRange,
String registrationName,
KvStateID kvStateId)
Notifies the listener about a registered KvState instance.
|
void |
KvStateLocationRegistry.notifyKvStateRegistered(JobVertexID jobVertexId,
KeyGroupRange keyGroupRange,
String registrationName,
KvStateID kvStateId,
InetSocketAddress kvStateServerAddress)
Notifies the registry about a registered KvState instance.
|
void |
KvStateRegistryListener.notifyKvStateUnregistered(JobID jobId,
JobVertexID jobVertexId,
KeyGroupRange keyGroupRange,
String registrationName)
Notifies the listener about an unregistered KvState instance.
|
void |
KvStateLocationRegistry.notifyKvStateUnregistered(JobVertexID jobVertexId,
KeyGroupRange keyGroupRange,
String registrationName)
Notifies the registry about an unregistered KvState instance.
|
KvStateID |
KvStateRegistry.registerKvState(JobID jobId,
JobVertexID jobVertexId,
KeyGroupRange keyGroupRange,
String registrationName,
InternalKvState<?,?,?> kvState)
Registers the KvState instance and returns the assigned ID.
|
void |
KvStateLocation.registerKvState(KeyGroupRange keyGroupRange,
KvStateID kvStateId,
InetSocketAddress kvStateAddress)
Registers a KvState instance for the given key group index.
|
void |
TaskKvStateRegistry.registerKvState(KeyGroupRange keyGroupRange,
String registrationName,
InternalKvState<?,?,?> kvState)
Registers the KvState instance at the KvStateRegistry.
|
void |
KvStateRegistry.unregisterKvState(JobID jobId,
JobVertexID jobVertexId,
KeyGroupRange keyGroupRange,
String registrationName,
KvStateID kvStateId)
Unregisters the KvState instance identified by the given KvStateID.
|
Modifier and Type | Method and Description |
---|---|
void |
SchedulerBase.notifyKvStateRegistered(JobID jobId,
JobVertexID jobVertexId,
KeyGroupRange keyGroupRange,
String registrationName,
KvStateID kvStateId,
InetSocketAddress kvStateServerAddress) |
void |
SchedulerNG.notifyKvStateRegistered(JobID jobId,
JobVertexID jobVertexId,
KeyGroupRange keyGroupRange,
String registrationName,
KvStateID kvStateId,
InetSocketAddress kvStateServerAddress) |
void |
SchedulerBase.notifyKvStateUnregistered(JobID jobId,
JobVertexID jobVertexId,
KeyGroupRange keyGroupRange,
String registrationName) |
void |
SchedulerNG.notifyKvStateUnregistered(JobID jobId,
JobVertexID jobVertexId,
KeyGroupRange keyGroupRange,
String registrationName) |
Modifier and Type | Field and Description |
---|---|
static KeyGroupRange |
KeyGroupRange.EMPTY_KEY_GROUP_RANGE
The empty key-group
|
protected KeyGroupRange |
KeyGroupPartitioner.keyGroupRange
The key-group range for the input data, covered in this partitioning.
|
protected KeyGroupRange |
AbstractKeyedStateBackendBuilder.keyGroupRange |
protected KeyGroupRange |
AbstractKeyedStateBackend.keyGroupRange
Range of key-groups for which this backend is responsible.
|
Modifier and Type | Method and Description |
---|---|
static KeyGroupRange |
KeyGroupRangeAssignment.computeKeyGroupRangeForOperatorIndex(int maxParallelism,
int parallelism,
int operatorIndex)
Computes the range of key-groups that are assigned to a given operator under the given parallelism and maximum
parallelism.
|
KeyGroupRange |
KeyGroupRange.getIntersection(KeyGroupRange other)
Create a range that represent the intersection between this range and the given range.
|
KeyGroupRange |
KeyGroupRangeOffsets.getKeyGroupRange() |
KeyGroupRange |
KeyGroupsStateHandle.getKeyGroupRange() |
KeyGroupRange |
IncrementalRemoteKeyedStateHandle.getKeyGroupRange() |
KeyGroupRange |
KeyedStateHandle.getKeyGroupRange()
Returns the range of the key groups contained in the state.
|
KeyGroupRange |
DirectoryKeyedStateHandle.getKeyGroupRange() |
KeyGroupRange |
AbstractKeyedStateBackend.getKeyGroupRange() |
static KeyGroupRange |
KeyGroupRange.of(int startKeyGroup,
int endKeyGroup)
Factory method that also handles creation of empty key-groups.
|
Modifier and Type | Method and Description |
---|---|
<K> AbstractKeyedStateBackend<K> |
StateBackend.createKeyedStateBackend(Environment env,
JobID jobID,
String operatorIdentifier,
TypeSerializer<K> keySerializer,
int numberOfKeyGroups,
KeyGroupRange keyGroupRange,
TaskKvStateRegistry kvStateRegistry,
TtlTimeProvider ttlTimeProvider,
MetricGroup metricGroup,
Collection<KeyedStateHandle> stateHandles,
CloseableRegistry cancelStreamRegistry)
Creates a new
AbstractKeyedStateBackend that is responsible for holding keyed state
and checkpointing it. |
abstract <K> AbstractKeyedStateBackend<K> |
AbstractStateBackend.createKeyedStateBackend(Environment env,
JobID jobID,
String operatorIdentifier,
TypeSerializer<K> keySerializer,
int numberOfKeyGroups,
KeyGroupRange keyGroupRange,
TaskKvStateRegistry kvStateRegistry,
TtlTimeProvider ttlTimeProvider,
MetricGroup metricGroup,
Collection<KeyedStateHandle> stateHandles,
CloseableRegistry cancelStreamRegistry) |
KeyGroupRangeOffsets |
KeyGroupRangeOffsets.getIntersection(KeyGroupRange keyGroupRange)
Returns a key-group range with offsets which is the intersection of the internal key-group range with the given
key-group range.
|
KeyGroupsStateHandle |
KeyGroupsStateHandle.getIntersection(KeyGroupRange keyGroupRange) |
KeyedStateHandle |
IncrementalRemoteKeyedStateHandle.getIntersection(KeyGroupRange keyGroupRange) |
KeyedStateHandle |
KeyedStateHandle.getIntersection(KeyGroupRange keyGroupRange)
Returns a state over a range that is the intersection between this
handle's key-group range and the provided key-group range.
|
KeyedStateHandle |
DirectoryKeyedStateHandle.getIntersection(KeyGroupRange otherKeyGroupRange) |
KeyGroupRange |
KeyGroupRange.getIntersection(KeyGroupRange other)
Create a range that represent the intersection between this range and the given range.
|
Constructor and Description |
---|
AbstractKeyedStateBackendBuilder(TaskKvStateRegistry kvStateRegistry,
TypeSerializer<K> keySerializer,
ClassLoader userCodeClassLoader,
int numberOfKeyGroups,
KeyGroupRange keyGroupRange,
ExecutionConfig executionConfig,
TtlTimeProvider ttlTimeProvider,
Collection<KeyedStateHandle> stateHandles,
StreamCompressionDecorator keyGroupCompressionDecorator,
CloseableRegistry cancelStreamRegistry) |
DirectoryKeyedStateHandle(DirectoryStateHandle directoryStateHandle,
KeyGroupRange keyGroupRange) |
IncrementalLocalKeyedStateHandle(UUID backendIdentifier,
long checkpointId,
DirectoryStateHandle directoryStateHandle,
KeyGroupRange keyGroupRange,
StreamStateHandle metaDataState,
Set<StateHandleID> sharedStateHandleIDs) |
IncrementalRemoteKeyedStateHandle(UUID backendIdentifier,
KeyGroupRange keyGroupRange,
long checkpointId,
Map<StateHandleID,StreamStateHandle> sharedState,
Map<StateHandleID,StreamStateHandle> privateState,
StreamStateHandle metaStateHandle) |
KeyedStateCheckpointOutputStream(CheckpointStreamFactory.CheckpointStateOutputStream delegate,
KeyGroupRange keyGroupRange) |
KeyGroupPartitioner(T[] partitioningSource,
int numberOfElements,
T[] partitioningDestination,
KeyGroupRange keyGroupRange,
int totalKeyGroups,
KeyExtractorFunction<T> keyExtractorFunction,
KeyGroupPartitioner.ElementWriterFunction<T> elementWriterFunction)
Creates a new
KeyGroupPartitioner . |
KeyGroupRangeOffsets(KeyGroupRange keyGroupRange)
Creates key-group range with offsets for the given key-group range, where all offsets are initially zero.
|
KeyGroupRangeOffsets(KeyGroupRange keyGroupRange,
long[] offsets)
Creates key-group range with offsets from the given key-group range.
|
StateSnapshotContextSynchronousImpl(long checkpointId,
long checkpointTimestamp,
CheckpointStreamFactory streamFactory,
KeyGroupRange keyGroupRange,
CloseableRegistry closableRegistry) |
Modifier and Type | Method and Description |
---|---|
<K> AbstractKeyedStateBackend<K> |
FsStateBackend.createKeyedStateBackend(Environment env,
JobID jobID,
String operatorIdentifier,
TypeSerializer<K> keySerializer,
int numberOfKeyGroups,
KeyGroupRange keyGroupRange,
TaskKvStateRegistry kvStateRegistry,
TtlTimeProvider ttlTimeProvider,
MetricGroup metricGroup,
Collection<KeyedStateHandle> stateHandles,
CloseableRegistry cancelStreamRegistry) |
Modifier and Type | Method and Description |
---|---|
KeyGroupRange |
InternalKeyContextImpl.getKeyGroupRange() |
KeyGroupRange |
InternalKeyContext.getKeyGroupRange()
Returns the key groups for this backend.
|
Constructor and Description |
---|
HeapKeyedStateBackendBuilder(TaskKvStateRegistry kvStateRegistry,
TypeSerializer<K> keySerializer,
ClassLoader userCodeClassLoader,
int numberOfKeyGroups,
KeyGroupRange keyGroupRange,
ExecutionConfig executionConfig,
TtlTimeProvider ttlTimeProvider,
Collection<KeyedStateHandle> stateHandles,
StreamCompressionDecorator keyGroupCompressionDecorator,
LocalRecoveryConfig localRecoveryConfig,
HeapPriorityQueueSetFactory priorityQueueSetFactory,
boolean asynchronousSnapshots,
CloseableRegistry cancelStreamRegistry) |
HeapPriorityQueueSet(PriorityComparator<T> elementPriorityComparator,
KeyExtractorFunction<T> keyExtractor,
int minimumCapacity,
KeyGroupRange keyGroupRange,
int totalNumberOfKeyGroups)
Creates an empty
HeapPriorityQueueSet with the requested initial capacity. |
HeapPriorityQueueSetFactory(KeyGroupRange keyGroupRange,
int totalKeyGroups,
int minimumCapacity) |
HeapPriorityQueueSnapshotRestoreWrapper(HeapPriorityQueueSet<T> priorityQueue,
RegisteredPriorityQueueStateBackendMetaInfo<T> metaInfo,
KeyExtractorFunction<T> keyExtractorFunction,
KeyGroupRange localKeyGroupRange,
int totalKeyGroups) |
InternalKeyContextImpl(KeyGroupRange keyGroupRange,
int numberOfKeyGroups) |
KeyGroupPartitionedPriorityQueue(KeyExtractorFunction<T> keyExtractor,
PriorityComparator<T> elementPriorityComparator,
KeyGroupPartitionedPriorityQueue.PartitionQueueSetFactory<T,PQ> orderedCacheFactory,
KeyGroupRange keyGroupRange,
int totalKeyGroups) |
Modifier and Type | Method and Description |
---|---|
<K> AbstractKeyedStateBackend<K> |
MemoryStateBackend.createKeyedStateBackend(Environment env,
JobID jobID,
String operatorIdentifier,
TypeSerializer<K> keySerializer,
int numberOfKeyGroups,
KeyGroupRange keyGroupRange,
TaskKvStateRegistry kvStateRegistry,
TtlTimeProvider ttlTimeProvider,
MetricGroup metricGroup,
Collection<KeyedStateHandle> stateHandles,
CloseableRegistry cancelStreamRegistry) |
Modifier and Type | Method and Description |
---|---|
void |
RpcKvStateRegistryListener.notifyKvStateRegistered(JobID jobId,
JobVertexID jobVertexId,
KeyGroupRange keyGroupRange,
String registrationName,
KvStateID kvStateId) |
void |
RpcKvStateRegistryListener.notifyKvStateUnregistered(JobID jobId,
JobVertexID jobVertexId,
KeyGroupRange keyGroupRange,
String registrationName) |
Copyright © 2014–2020 The Apache Software Foundation. All rights reserved.