org.apache.flink.runtime.state

Class AbstractKeyedStateBackend<K>

java.lang.Object

org.apache.flink.runtime.state.AbstractKeyedStateBackend<K>

Type Parameters:

K - Type of the key by which state is keyed.

All Implemented Interfaces:

Closeable, AutoCloseable, CheckpointListener, InternalKeyContext<K>, KeyedStateBackend<K>, Snapshotable<KeyedStateHandle>

Direct Known Subclasses:

HeapKeyedStateBackend, RocksDBKeyedStateBackend

public abstract class AbstractKeyedStateBackend<K>
extends Object
implements KeyedStateBackend<K>, Snapshotable<KeyedStateHandle>, Closeable, CheckpointListener

Base implementation of KeyedStateBackend. The state can be checkpointed to streams using Snapshotable.snapshot(long, long, CheckpointStreamFactory, CheckpointOptions).

Field Summary

Fields

Modifier and Type	Field and Description
protected CloseableRegistry	cancelStreamRegistry Registry for all opened streams, so they can be closed if the task using this backend is closed
protected K	currentKey The currently active key.
protected StreamCompressionDecorator	keyGroupCompressionDecorator Decoratores the input and output streams to write key-groups compressed.
protected KeyGroupRange	keyGroupRange Range of key-groups for which this backend is responsible
protected TypeSerializer<K>	keySerializer TypeSerializer for our key.
protected HashMap<String,InternalKvState<?>>	keyValueStatesByName So that we can give out state when the user uses the same key.
protected TaskKvStateRegistry	kvStateRegistry KvStateRegistry helper for this task
protected int	numberOfKeyGroups The number of key-groups aka max parallelism
protected ClassLoader	userCodeClassLoader

Constructor Summary

Constructors

Constructor and Description
AbstractKeyedStateBackend(TaskKvStateRegistry kvStateRegistry, TypeSerializer<K> keySerializer, ClassLoader userCodeClassLoader, int numberOfKeyGroups, KeyGroupRange keyGroupRange, ExecutionConfig executionConfig)

Method Summary

Modifier and Type	Method and Description
void	close()
protected abstract <N,T,ACC,R> InternalAggregatingState<N,T,R>	createAggregatingState(TypeSerializer<N> namespaceSerializer, AggregatingStateDescriptor<T,ACC,R> stateDesc) Creates and returns a new AggregatingState.
protected abstract <N,T,ACC> InternalFoldingState<N,T,ACC>	createFoldingState(TypeSerializer<N> namespaceSerializer, FoldingStateDescriptor<T,ACC> stateDesc) Deprecated. will be removed in a future version
protected abstract <N,T> InternalListState<N,T>	createListState(TypeSerializer<N> namespaceSerializer, ListStateDescriptor<T> stateDesc) Creates and returns a new ListState.
protected abstract <N,UK,UV> InternalMapState<N,UK,UV>	createMapState(TypeSerializer<N> namespaceSerializer, MapStateDescriptor<UK,UV> stateDesc) Creates and returns a new MapState.
protected abstract <N,T> InternalReducingState<N,T>	createReducingState(TypeSerializer<N> namespaceSerializer, ReducingStateDescriptor<T> stateDesc) Creates and returns a new ReducingState.
protected abstract <N,T> InternalValueState<N,T>	createValueState(TypeSerializer<N> namespaceSerializer, ValueStateDescriptor<T> stateDesc) Creates and returns a new ValueState.
void	dispose() Closes the state backend, releasing all internal resources, but does not delete any persistent checkpoint data.
K	getCurrentKey() Used by states to access the current key.
int	getCurrentKeyGroupIndex() Returns the key-group to which the current key belongs.
StreamCompressionDecorator	getKeyGroupCompressionDecorator()
KeyGroupRange	getKeyGroupRange() Returns the key groups for this backend.
TypeSerializer<K>	getKeySerializer() TypeSerializer for the state backend key type.
int	getNumberOfKeyGroups() Returns the number of key-groups aka max parallelism.
<N,S extends State,V> S	getOrCreateKeyedState(TypeSerializer<N> namespaceSerializer, StateDescriptor<S,V> stateDescriptor) Creates or retrieves a keyed state backed by this state backend.
<N,S extends State> S	getPartitionedState(N namespace, TypeSerializer<N> namespaceSerializer, StateDescriptor<S,?> stateDescriptor) TODO: NOTE: This method does a lot of work caching / retrieving states just to update the namespace.
abstract int	numStateEntries() Returns the total number of state entries across all keys/namespaces.
void	setCurrentKey(K newKey) Sets the current key that is used for partitioned state.
boolean	supportsAsynchronousSnapshots()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.state.KeyedStateBackend

getKeys

Methods inherited from interface org.apache.flink.runtime.state.Snapshotable

restore, snapshot

Methods inherited from interface org.apache.flink.runtime.state.CheckpointListener

notifyCheckpointComplete

Field Detail

keySerializer

protected final TypeSerializer<K> keySerializer

TypeSerializer for our key.

currentKey

protected K currentKey

The currently active key.

keyValueStatesByName

protected final HashMap<String,InternalKvState<?>> keyValueStatesByName

So that we can give out state when the user uses the same key.

numberOfKeyGroups

protected final int numberOfKeyGroups

The number of key-groups aka max parallelism

keyGroupRange

protected final KeyGroupRange keyGroupRange

Range of key-groups for which this backend is responsible

kvStateRegistry

protected final TaskKvStateRegistry kvStateRegistry

KvStateRegistry helper for this task

cancelStreamRegistry

protected CloseableRegistry cancelStreamRegistry

Registry for all opened streams, so they can be closed if the task using this backend is closed

userCodeClassLoader

protected final ClassLoader userCodeClassLoader

keyGroupCompressionDecorator

protected final StreamCompressionDecorator keyGroupCompressionDecorator

Decoratores the input and output streams to write key-groups compressed.

Constructor Detail

AbstractKeyedStateBackend

public AbstractKeyedStateBackend(TaskKvStateRegistry kvStateRegistry,
                                 TypeSerializer<K> keySerializer,
                                 ClassLoader userCodeClassLoader,
                                 int numberOfKeyGroups,
                                 KeyGroupRange keyGroupRange,
                                 ExecutionConfig executionConfig)

Method Detail

dispose

public void dispose()

Closes the state backend, releasing all internal resources, but does not delete any persistent checkpoint data.

Specified by:

dispose in interface KeyedStateBackend<K>

createValueState

protected abstract <N,T> InternalValueState<N,T> createValueState(TypeSerializer<N> namespaceSerializer,
                                                                  ValueStateDescriptor<T> stateDesc)
                                                           throws Exception

Creates and returns a new ValueState.

Type Parameters:

N - The type of the namespace.

T - The type of the value that the ValueState can store.

Parameters:

namespaceSerializer - TypeSerializer for the state namespace.

stateDesc - The StateDescriptor that contains the name of the state.

Throws:

Exception

createListState

protected abstract <N,T> InternalListState<N,T> createListState(TypeSerializer<N> namespaceSerializer,
                                                                ListStateDescriptor<T> stateDesc)
                                                         throws Exception

Creates and returns a new ListState.

Type Parameters:

N - The type of the namespace.

T - The type of the values that the ListState can store.

Parameters:

namespaceSerializer - TypeSerializer for the state namespace.

stateDesc - The StateDescriptor that contains the name of the state.

Throws:

Exception

createReducingState

protected abstract <N,T> InternalReducingState<N,T> createReducingState(TypeSerializer<N> namespaceSerializer,
                                                                        ReducingStateDescriptor<T> stateDesc)
                                                                 throws Exception

Creates and returns a new ReducingState.

Type Parameters:

N - The type of the namespace.

T - The type of the values that the ListState can store.

Parameters:

namespaceSerializer - TypeSerializer for the state namespace.

stateDesc - The StateDescriptor that contains the name of the state.

Throws:

Exception

createAggregatingState

protected abstract <N,T,ACC,R> InternalAggregatingState<N,T,R> createAggregatingState(TypeSerializer<N> namespaceSerializer,
                                                                                      AggregatingStateDescriptor<T,ACC,R> stateDesc)
                                                                               throws Exception

Creates and returns a new AggregatingState.

Type Parameters:

N - The type of the namespace.

T - The type of the values that the ListState can store.

Parameters:

namespaceSerializer - TypeSerializer for the state namespace.

stateDesc - The StateDescriptor that contains the name of the state.

Throws:

Exception

createFoldingState

@Deprecated
protected abstract <N,T,ACC> InternalFoldingState<N,T,ACC> createFoldingState(TypeSerializer<N> namespaceSerializer,
                                                                                          FoldingStateDescriptor<T,ACC> stateDesc)
                                                                                   throws Exception

Deprecated. will be removed in a future version

Creates and returns a new FoldingState.

Type Parameters:

N - The type of the namespace.

T - Type of the values folded into the state

ACC - Type of the value in the state

Parameters:

namespaceSerializer - TypeSerializer for the state namespace.

stateDesc - The StateDescriptor that contains the name of the state.

Throws:

Exception

createMapState

protected abstract <N,UK,UV> InternalMapState<N,UK,UV> createMapState(TypeSerializer<N> namespaceSerializer,
                                                                      MapStateDescriptor<UK,UV> stateDesc)
                                                               throws Exception

Creates and returns a new MapState.

Type Parameters:

N - The type of the namespace.

UK - Type of the keys in the state

UV - Type of the values in the state *

Parameters:

namespaceSerializer - TypeSerializer for the state namespace.

stateDesc - The StateDescriptor that contains the name of the state.

Throws:

Exception

setCurrentKey

public void setCurrentKey(K newKey)

Description copied from interface: KeyedStateBackend

Sets the current key that is used for partitioned state.

Specified by:

setCurrentKey in interface KeyedStateBackend<K>

Parameters:

newKey - The new current key.

See Also:

KeyedStateBackend

getKeySerializer

public TypeSerializer<K> getKeySerializer()

Description copied from interface: InternalKeyContext

TypeSerializer for the state backend key type.

Specified by:

getKeySerializer in interface InternalKeyContext<K>

See Also:

KeyedStateBackend

getCurrentKey

public K getCurrentKey()

Description copied from interface: InternalKeyContext

Used by states to access the current key.

Specified by:

getCurrentKey in interface InternalKeyContext<K>

See Also:

KeyedStateBackend

getCurrentKeyGroupIndex

public int getCurrentKeyGroupIndex()

Description copied from interface: InternalKeyContext

Returns the key-group to which the current key belongs.

Specified by:

getCurrentKeyGroupIndex in interface InternalKeyContext<K>

See Also:

KeyedStateBackend

getNumberOfKeyGroups

public int getNumberOfKeyGroups()

Description copied from interface: InternalKeyContext

Returns the number of key-groups aka max parallelism.

Specified by:

getNumberOfKeyGroups in interface InternalKeyContext<K>

See Also:

KeyedStateBackend

getKeyGroupRange

public KeyGroupRange getKeyGroupRange()

Description copied from interface: InternalKeyContext

Returns the key groups for this backend.

Specified by:

getKeyGroupRange in interface InternalKeyContext<K>

See Also:

KeyedStateBackend

getOrCreateKeyedState

public <N,S extends State,V> S getOrCreateKeyedState(TypeSerializer<N> namespaceSerializer,
                                                     StateDescriptor<S,V> stateDescriptor)
                                              throws Exception

Description copied from interface: KeyedStateBackend

Creates or retrieves a keyed state backed by this state backend.

Specified by:

getOrCreateKeyedState in interface KeyedStateBackend<K>

Type Parameters:

N - The type of the namespace.

S - The type of the state.

Parameters:

namespaceSerializer - The serializer used for the namespace type of the state

stateDescriptor - The identifier for the state. This contains name and can create a default state value.

Returns:

A new key/value state backed by this backend.

Throws:

Exception - Exceptions may occur during initialization of the state and should be forwarded.

See Also:

KeyedStateBackend

getPartitionedState

public <N,S extends State> S getPartitionedState(N namespace,
                                                 TypeSerializer<N> namespaceSerializer,
                                                 StateDescriptor<S,?> stateDescriptor)
                                          throws Exception

TODO: NOTE: This method does a lot of work caching / retrieving states just to update the namespace. This method should be removed for the sake of namespaces being lazily fetched from the keyed state backend, or being set on the state directly.

Specified by:

getPartitionedState in interface KeyedStateBackend<K>

Type Parameters:

N - The type of the namespace.

S - The type of the state.

stateDescriptor - The identifier for the state. This contains name and can create a default state value.

Returns:

A new key/value state backed by this backend.

Throws:

Exception - Exceptions may occur during initialization of the state and should be forwarded.

See Also:

KeyedStateBackend

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

supportsAsynchronousSnapshots

@VisibleForTesting
public boolean supportsAsynchronousSnapshots()

getKeyGroupCompressionDecorator

@VisibleForTesting
public StreamCompressionDecorator getKeyGroupCompressionDecorator()

numStateEntries

@VisibleForTesting
public abstract int numStateEntries()

Returns the total number of state entries across all keys/namespaces.