org.apache.flink.api.common.state

Class StateTtlConfig.Builder

java.lang.Object

org.apache.flink.api.common.state.StateTtlConfig.Builder

Enclosing class:

StateTtlConfig

public static class StateTtlConfig.Builder
extends Object

Builder for the StateTtlConfig.

Constructor Summary

Constructors

Constructor and Description
Builder(Time ttl)

Method Summary

Modifier and Type	Method and Description
StateTtlConfig	build()
StateTtlConfig.Builder	cleanupFullSnapshot() Cleanup expired state in full snapshot on checkpoint.
StateTtlConfig.Builder	cleanupInBackground() Deprecated. enabled by default, no need to enable it manually
StateTtlConfig.Builder	cleanupIncrementally(int cleanupSize, boolean runCleanupForEveryRecord) Cleanup expired state incrementally cleanup local state.
StateTtlConfig.Builder	cleanupInRocksdbCompactFilter() Deprecated. Use more general configuration method cleanupInBackground() instead
StateTtlConfig.Builder	cleanupInRocksdbCompactFilter(long queryTimeAfterNumEntries) Cleanup expired state while Rocksdb compaction is running.
StateTtlConfig.Builder	disableCleanupInBackground() Disable default cleanup of expired state in background (enabled by default).
StateTtlConfig.Builder	neverReturnExpired()
StateTtlConfig.Builder	returnExpiredIfNotCleanedUp()
StateTtlConfig.Builder	setStateVisibility(StateTtlConfig.StateVisibility stateVisibility) Sets the state visibility.
StateTtlConfig.Builder	setTtl(Time ttl) Sets the ttl time.
StateTtlConfig.Builder	setTtlTimeCharacteristic(StateTtlConfig.TtlTimeCharacteristic ttlTimeCharacteristic) Sets the time characteristic.
StateTtlConfig.Builder	setUpdateType(StateTtlConfig.UpdateType updateType) Sets the ttl update type.
StateTtlConfig.Builder	updateTtlOnCreateAndWrite()
StateTtlConfig.Builder	updateTtlOnReadAndWrite()
StateTtlConfig.Builder	useProcessingTime()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Builder

public Builder(@Nonnull
               Time ttl)

Method Detail

setUpdateType

@Nonnull
public StateTtlConfig.Builder setUpdateType(StateTtlConfig.UpdateType updateType)

Sets the ttl update type.

Parameters:

updateType - The ttl update type configures when to update last access timestamp which prolongs state TTL.

updateTtlOnCreateAndWrite

@Nonnull
public StateTtlConfig.Builder updateTtlOnCreateAndWrite()

updateTtlOnReadAndWrite

@Nonnull
public StateTtlConfig.Builder updateTtlOnReadAndWrite()

setStateVisibility

@Nonnull
public StateTtlConfig.Builder setStateVisibility(@Nonnull
                                                          StateTtlConfig.StateVisibility stateVisibility)

Sets the state visibility.

Parameters:

stateVisibility - The state visibility configures whether expired user value can be returned or not.

returnExpiredIfNotCleanedUp

@Nonnull
public StateTtlConfig.Builder returnExpiredIfNotCleanedUp()

neverReturnExpired

@Nonnull
public StateTtlConfig.Builder neverReturnExpired()

setTtlTimeCharacteristic

@Nonnull
public StateTtlConfig.Builder setTtlTimeCharacteristic(@Nonnull
                                                                StateTtlConfig.TtlTimeCharacteristic ttlTimeCharacteristic)

Sets the time characteristic.

Parameters:

ttlTimeCharacteristic - The time characteristic configures time scale to use for ttl.

useProcessingTime

@Nonnull
public StateTtlConfig.Builder useProcessingTime()

cleanupFullSnapshot

@Nonnull
public StateTtlConfig.Builder cleanupFullSnapshot()

Cleanup expired state in full snapshot on checkpoint.

cleanupIncrementally

@Nonnull
public StateTtlConfig.Builder cleanupIncrementally(@Nonnegative
                                                            int cleanupSize,
                                                            boolean runCleanupForEveryRecord)

Cleanup expired state incrementally cleanup local state.

Upon every state access this cleanup strategy checks a bunch of state keys for expiration and cleans up expired ones. It keeps a lazy iterator through all keys with relaxed consistency if backend supports it. This way all keys should be regularly checked and cleaned eventually over time if any state is constantly being accessed.

Additionally to the incremental cleanup upon state access, it can also run per every record. Caution: if there are a lot of registered states using this option, they all will be iterated for every record to check if there is something to cleanup.

Note: if no access happens to this state or no records are processed in case of runCleanupForEveryRecord, expired state will persist.

Note: Time spent for the incremental cleanup increases record processing latency.

Note: At the moment incremental cleanup is implemented only for Heap state backend. Setting it for RocksDB will have no effect.

Note: If heap state backend is used with synchronous snapshotting, the global iterator keeps a copy of all keys while iterating because of its specific implementation which does not support concurrent modifications. Enabling of this feature will increase memory consumption then. Asynchronous snapshotting does not have this problem.

Parameters:

cleanupSize - max number of keys pulled from queue for clean up upon state touch for any key

runCleanupForEveryRecord - run incremental cleanup per each processed record

cleanupInRocksdbCompactFilter

@Nonnull
 @Deprecated
public StateTtlConfig.Builder cleanupInRocksdbCompactFilter()

Deprecated. Use more general configuration method cleanupInBackground() instead

Cleanup expired state while Rocksdb compaction is running.

RocksDB runs periodic compaction of state updates and merges them to free storage. During this process, the TTL filter checks timestamp of state entries and drops expired ones. The feature has to be activated in RocksDb backend firstly using the following Flink configuration option: state.backend.rocksdb.ttl.compaction.filter.enabled.

Due to specifics of RocksDB compaction filter, cleanup is not properly guaranteed if put and merge operations are used at the same time: https://github.com/facebook/rocksdb/blob/master/include/rocksdb/compaction_filter.h#L69 It means that the TTL filter should be tested for List state taking into account this caveat.

cleanupInRocksdbCompactFilter

@Nonnull
public StateTtlConfig.Builder cleanupInRocksdbCompactFilter(long queryTimeAfterNumEntries)

Cleanup expired state while Rocksdb compaction is running.

RocksDB compaction filter will query current timestamp, used to check expiration, from Flink every time after processing queryTimeAfterNumEntries number of state entries. Updating the timestamp more often can improve cleanup speed but it decreases compaction performance because it uses JNI call from native code.

Parameters:

queryTimeAfterNumEntries - number of state entries to process by compaction filter before updating current timestamp

cleanupInBackground

@Nonnull
 @Deprecated
public StateTtlConfig.Builder cleanupInBackground()

Deprecated. enabled by default, no need to enable it manually

Enable default cleanup of expired state in background (enabled by default).

Depending on actually used backend, the corresponding default cleanup will kick in if supported. If some specific cleanup is also configured, e.g. cleanupIncrementally(int, boolean) or cleanupInRocksdbCompactFilter(), then the specific one will kick in instead of default.

disableCleanupInBackground

@Nonnull
public StateTtlConfig.Builder disableCleanupInBackground()

Disable default cleanup of expired state in background (enabled by default).

If some specific cleanup is configured, e.g. cleanupIncrementally(int, boolean) or cleanupInRocksdbCompactFilter(), this setting does not disable it.

setTtl

@Nonnull
public StateTtlConfig.Builder setTtl(@Nonnull
                                              Time ttl)

Sets the ttl time.

Parameters:

ttl - The ttl time.

build

@Nonnull
public StateTtlConfig build()