org.apache.flink.contrib.streaming.state.snapshot

Class RocksDBSnapshotStrategyBase<K,R extends SnapshotResources>

java.lang.Object

org.apache.flink.contrib.streaming.state.snapshot.RocksDBSnapshotStrategyBase<K,R>

Type Parameters:

K - type of the backend keys.

All Implemented Interfaces:

AutoCloseable, CheckpointListener, SnapshotStrategy<KeyedStateHandle,RocksDBSnapshotStrategyBase.NativeRocksDBSnapshotResources>

Direct Known Subclasses:

RocksIncrementalSnapshotStrategy, RocksNativeFullSnapshotStrategy

public abstract class RocksDBSnapshotStrategyBase<K,R extends SnapshotResources>
extends Object
implements CheckpointListener, SnapshotStrategy<KeyedStateHandle,RocksDBSnapshotStrategyBase.NativeRocksDBSnapshotResources>, AutoCloseable

Abstract base class for SnapshotStrategy implementations for RocksDB state backend.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	RocksDBSnapshotStrategyBase.NativeRocksDBSnapshotResources A SnapshotResources for native rocksdb snapshot.
protected static class	RocksDBSnapshotStrategyBase.PreviousSnapshot Previous snapshot with uploaded sst files.
protected class	RocksDBSnapshotStrategyBase.RocksDBSnapshotOperation Common operation in native rocksdb snapshot result supplier.

Nested classes/interfaces inherited from interface org.apache.flink.runtime.state.SnapshotStrategy

SnapshotStrategy.SnapshotResultSupplier<S extends StateObject>

Field Summary

Fields

Modifier and Type	Field and Description
protected UUID	backendUID The state handle ids of all sst files materialized in snapshots for previous checkpoints.
protected org.rocksdb.RocksDB	db RocksDB instance from the backend.
protected static RocksDBSnapshotStrategyBase.PreviousSnapshot	EMPTY_PREVIOUS_SNAPSHOT
protected File	instanceBasePath Base path of the RocksDB instance.
protected int	keyGroupPrefixBytes Number of bytes in the key-group prefix.
protected KeyGroupRange	keyGroupRange The key-group range for the task.
protected TypeSerializer<K>	keySerializer The key serializer of the backend.
protected LinkedHashMap<String,RocksDBKeyedStateBackend.RocksDbKvStateInfo>	kvStateInformation Key/Value state meta info from the backend.
protected String	localDirectoryName The local directory name of the current snapshot strategy.
protected LocalRecoveryConfig	localRecoveryConfig The configuration for local recovery.
protected ResourceGuard	rocksDBResourceGuard Resource guard for the RocksDB instance.

Constructor Summary

Constructors

Constructor and Description
RocksDBSnapshotStrategyBase(String description, org.rocksdb.RocksDB db, ResourceGuard rocksDBResourceGuard, TypeSerializer<K> keySerializer, LinkedHashMap<String,RocksDBKeyedStateBackend.RocksDbKvStateInfo> kvStateInformation, KeyGroupRange keyGroupRange, int keyGroupPrefixBytes, LocalRecoveryConfig localRecoveryConfig, File instanceBasePath, UUID backendUID)

Method Summary

Modifier and Type	Method and Description
protected void	cleanupIncompleteSnapshot(CloseableRegistry tmpResourcesRegistry, SnapshotDirectory localBackupDirectory)
abstract void	close()
String	getDescription()
protected SnapshotResult<StreamStateHandle>	materializeMetaData(CloseableRegistry snapshotCloseableRegistry, CloseableRegistry tmpResourcesRegistry, List<StateMetaInfoSnapshot> stateMetaInfoSnapshots, long checkpointId, CheckpointStreamFactory checkpointStreamFactory)
protected SnapshotDirectory	prepareLocalSnapshotDirectory(long checkpointId)
protected abstract RocksDBSnapshotStrategyBase.PreviousSnapshot	snapshotMetaData(long checkpointId, List<StateMetaInfoSnapshot> stateMetaInfoSnapshots)
RocksDBSnapshotStrategyBase.NativeRocksDBSnapshotResources	syncPrepareResources(long checkpointId) Performs the synchronous part of the snapshot.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted, notifyCheckpointComplete

Methods inherited from interface org.apache.flink.runtime.state.SnapshotStrategy

asyncSnapshot

Field Detail

db

@Nonnull
protected org.rocksdb.RocksDB db

RocksDB instance from the backend.

rocksDBResourceGuard

@Nonnull
protected final ResourceGuard rocksDBResourceGuard

Resource guard for the RocksDB instance.

keySerializer

@Nonnull
protected final TypeSerializer<K> keySerializer

The key serializer of the backend.

kvStateInformation

@Nonnull
protected final LinkedHashMap<String,RocksDBKeyedStateBackend.RocksDbKvStateInfo> kvStateInformation

Key/Value state meta info from the backend.

keyGroupRange

@Nonnull
protected final KeyGroupRange keyGroupRange

The key-group range for the task.

keyGroupPrefixBytes

@Nonnegative
protected final int keyGroupPrefixBytes

Number of bytes in the key-group prefix.

localRecoveryConfig

@Nonnull
protected final LocalRecoveryConfig localRecoveryConfig

The configuration for local recovery.

instanceBasePath

@Nonnull
protected final File instanceBasePath

Base path of the RocksDB instance.

localDirectoryName

protected final String localDirectoryName

The local directory name of the current snapshot strategy.

backendUID

@Nonnull
protected final UUID backendUID

The state handle ids of all sst files materialized in snapshots for previous checkpoints.

EMPTY_PREVIOUS_SNAPSHOT

protected static final RocksDBSnapshotStrategyBase.PreviousSnapshot EMPTY_PREVIOUS_SNAPSHOT

Constructor Detail

RocksDBSnapshotStrategyBase

public RocksDBSnapshotStrategyBase(@Nonnull
                                   String description,
                                   @Nonnull
                                   org.rocksdb.RocksDB db,
                                   @Nonnull
                                   ResourceGuard rocksDBResourceGuard,
                                   @Nonnull
                                   TypeSerializer<K> keySerializer,
                                   @Nonnull
                                   LinkedHashMap<String,RocksDBKeyedStateBackend.RocksDbKvStateInfo> kvStateInformation,
                                   @Nonnull
                                   KeyGroupRange keyGroupRange,
                                   @Nonnegative
                                   int keyGroupPrefixBytes,
                                   @Nonnull
                                   LocalRecoveryConfig localRecoveryConfig,
                                   @Nonnull
                                   File instanceBasePath,
                                   @Nonnull
                                   UUID backendUID)

Method Detail

getDescription

@Nonnull
public String getDescription()

syncPrepareResources

public RocksDBSnapshotStrategyBase.NativeRocksDBSnapshotResources syncPrepareResources(long checkpointId)
                                                                                throws Exception

Description copied from interface: SnapshotStrategy

Performs the synchronous part of the snapshot. It returns resources which can be later on used in the asynchronous part.

Specified by:

syncPrepareResources in interface SnapshotStrategy<KeyedStateHandle,RocksDBSnapshotStrategyBase.NativeRocksDBSnapshotResources>

Parameters:

checkpointId - The ID of the checkpoint.

Returns:

Resources needed to finish the snapshot.

Throws:

Exception

snapshotMetaData

protected abstract RocksDBSnapshotStrategyBase.PreviousSnapshot snapshotMetaData(long checkpointId,
                                                                                 @Nonnull
                                                                                 List<StateMetaInfoSnapshot> stateMetaInfoSnapshots)

prepareLocalSnapshotDirectory

@Nonnull
protected SnapshotDirectory prepareLocalSnapshotDirectory(long checkpointId)
                                                            throws IOException

Throws:

IOException

cleanupIncompleteSnapshot

protected void cleanupIncompleteSnapshot(@Nonnull
                                         CloseableRegistry tmpResourcesRegistry,
                                         @Nonnull
                                         SnapshotDirectory localBackupDirectory)

materializeMetaData

@Nonnull
protected SnapshotResult<StreamStateHandle> materializeMetaData(@Nonnull
                                                                         CloseableRegistry snapshotCloseableRegistry,
                                                                         @Nonnull
                                                                         CloseableRegistry tmpResourcesRegistry,
                                                                         @Nonnull
                                                                         List<StateMetaInfoSnapshot> stateMetaInfoSnapshots,
                                                                         long checkpointId,
                                                                         @Nonnull
                                                                         CheckpointStreamFactory checkpointStreamFactory)
                                                                  throws Exception

Throws:

Exception

close

public abstract void close()

Specified by:

close in interface AutoCloseable