org.apache.flink.runtime.checkpoint

Class StateAssignmentOperation

java.lang.Object

org.apache.flink.runtime.checkpoint.StateAssignmentOperation

@Internal
public class StateAssignmentOperation
extends Object

This class encapsulates the operation of assigning restored state when restoring from a checkpoint.

Constructor Summary

Constructors

Constructor and Description
StateAssignmentOperation(long restoreCheckpointId, Set<ExecutionJobVertex> tasks, Map<OperatorID,OperatorState> operatorStates, boolean allowNonRestoredState)

Method Summary

Modifier and Type	Method and Description
static <T extends StateObject> Map<OperatorInstanceID,List<T>>	applyRepartitioner(OperatorID operatorID, OperatorStateRepartitioner<T> opStateRepartitioner, List<List<T>> chainOpParallelStates, int oldParallelism, int newParallelism)
static <T> List<List<T>>	applyRepartitioner(OperatorStateRepartitioner<T> opStateRepartitioner, List<List<T>> chainOpParallelStates, int oldParallelism, int newParallelism) Repartitions the given operator state using the given OperatorStateRepartitioner with respect to the new parallelism.
void	assignStates()
void	checkParallelismPreconditions(List<OperatorState> operatorStates, ExecutionJobVertex executionJobVertex)
static List<KeyGroupRange>	createKeyGroupPartitions(int numberKeyGroups, int parallelism) Groups the available set of key groups into key group partitions.
static void	extractIntersectingState(Collection<? extends KeyedStateHandle> originalSubtaskStateHandles, KeyGroupRange rangeToExtract, List<KeyedStateHandle> extractedStateCollector) Extracts certain key group ranges from the given state handles and adds them to the collector.
static List<KeyedStateHandle>	getManagedKeyedStateHandles(OperatorState operatorState, KeyGroupRange subtaskKeyGroupRange) Collect managedKeyedStateHandles which have intersection with given KeyGroupRange from operatorState.
static List<KeyedStateHandle>	getRawKeyedStateHandles(OperatorState operatorState, KeyGroupRange subtaskKeyGroupRange) Collect rawKeyedStateHandles which have intersection with given KeyGroupRange from operatorState.
static OperatorSubtaskState	operatorSubtaskStateFrom(OperatorInstanceID instanceID, Map<OperatorInstanceID,List<OperatorStateHandle>> subManagedOperatorState, Map<OperatorInstanceID,List<OperatorStateHandle>> subRawOperatorState, Map<OperatorInstanceID,List<InputChannelStateHandle>> inputChannelStates, Map<OperatorInstanceID,List<ResultSubpartitionStateHandle>> resultSubpartitionStates, Map<OperatorInstanceID,List<KeyedStateHandle>> subManagedKeyedState, Map<OperatorInstanceID,List<KeyedStateHandle>> subRawKeyedState)
static <T extends StateObject> Map<OperatorInstanceID,List<T>>	reDistributePartitionableStates(List<OperatorState> oldOperatorStates, int newParallelism, List<OperatorIDPair> newOperatorIDs, java.util.function.Function<OperatorSubtaskState,StateObjectCollection<T>> extractHandle, OperatorStateRepartitioner<T> stateRepartitioner)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StateAssignmentOperation

public StateAssignmentOperation(long restoreCheckpointId,
                                Set<ExecutionJobVertex> tasks,
                                Map<OperatorID,OperatorState> operatorStates,
                                boolean allowNonRestoredState)

Method Detail

assignStates

public void assignStates()

operatorSubtaskStateFrom

public static OperatorSubtaskState operatorSubtaskStateFrom(OperatorInstanceID instanceID,
                                                            Map<OperatorInstanceID,List<OperatorStateHandle>> subManagedOperatorState,
                                                            Map<OperatorInstanceID,List<OperatorStateHandle>> subRawOperatorState,
                                                            Map<OperatorInstanceID,List<InputChannelStateHandle>> inputChannelStates,
                                                            Map<OperatorInstanceID,List<ResultSubpartitionStateHandle>> resultSubpartitionStates,
                                                            Map<OperatorInstanceID,List<KeyedStateHandle>> subManagedKeyedState,
                                                            Map<OperatorInstanceID,List<KeyedStateHandle>> subRawKeyedState)

checkParallelismPreconditions

public void checkParallelismPreconditions(List<OperatorState> operatorStates,
                                          ExecutionJobVertex executionJobVertex)

reDistributePartitionableStates

public static <T extends StateObject> Map<OperatorInstanceID,List<T>> reDistributePartitionableStates(List<OperatorState> oldOperatorStates,
                                                                                                      int newParallelism,
                                                                                                      List<OperatorIDPair> newOperatorIDs,
                                                                                                      java.util.function.Function<OperatorSubtaskState,StateObjectCollection<T>> extractHandle,
                                                                                                      OperatorStateRepartitioner<T> stateRepartitioner)

getManagedKeyedStateHandles

public static List<KeyedStateHandle> getManagedKeyedStateHandles(OperatorState operatorState,
                                                                 KeyGroupRange subtaskKeyGroupRange)

Collect managedKeyedStateHandles which have intersection with given KeyGroupRange from operatorState.

Parameters:

operatorState - all state handles of a operator

subtaskKeyGroupRange - the KeyGroupRange of a subtask

Returns:

all managedKeyedStateHandles which have intersection with given KeyGroupRange

getRawKeyedStateHandles

public static List<KeyedStateHandle> getRawKeyedStateHandles(OperatorState operatorState,
                                                             KeyGroupRange subtaskKeyGroupRange)

Collect rawKeyedStateHandles which have intersection with given KeyGroupRange from operatorState.

Parameters:

operatorState - all state handles of a operator

subtaskKeyGroupRange - the KeyGroupRange of a subtask

Returns:

all rawKeyedStateHandles which have intersection with given KeyGroupRange

extractIntersectingState

@VisibleForTesting
public static void extractIntersectingState(Collection<? extends KeyedStateHandle> originalSubtaskStateHandles,
                                                               KeyGroupRange rangeToExtract,
                                                               List<KeyedStateHandle> extractedStateCollector)

Extracts certain key group ranges from the given state handles and adds them to the collector.

createKeyGroupPartitions

public static List<KeyGroupRange> createKeyGroupPartitions(int numberKeyGroups,
                                                           int parallelism)

Groups the available set of key groups into key group partitions. A key group partition is the set of key groups which is assigned to the same task. Each set of the returned list constitutes a key group partition.

IMPORTANT: The assignment of key groups to partitions has to be in sync with the KeyGroupStreamPartitioner.

Parameters:

numberKeyGroups - Number of available key groups (indexed from 0 to numberKeyGroups - 1)

parallelism - Parallelism to generate the key group partitioning for

Returns:

List of key group partitions

applyRepartitioner

public static <T extends StateObject> Map<OperatorInstanceID,List<T>> applyRepartitioner(OperatorID operatorID,
                                                                                         OperatorStateRepartitioner<T> opStateRepartitioner,
                                                                                         List<List<T>> chainOpParallelStates,
                                                                                         int oldParallelism,
                                                                                         int newParallelism)

applyRepartitioner

public static <T> List<List<T>> applyRepartitioner(OperatorStateRepartitioner<T> opStateRepartitioner,
                                                   List<List<T>> chainOpParallelStates,
                                                   int oldParallelism,
                                                   int newParallelism)

Repartitions the given operator state using the given OperatorStateRepartitioner with respect to the new parallelism.

Parameters:

opStateRepartitioner - partitioner to use

chainOpParallelStates - state to repartition

oldParallelism - parallelism with which the state is currently partitioned

newParallelism - parallelism with which the state should be partitioned

Returns:

repartitioned state