org.apache.flink.table.planner.plan.nodes.exec.stream

Class StreamExecDeduplicate

java.lang.Object

org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase<RowData>

org.apache.flink.table.planner.plan.nodes.exec.stream.StreamExecDeduplicate

All Implemented Interfaces:

ExecNode<RowData>, ExecNodeTranslator<RowData>, FusionCodegenExecNode, SingleTransformationTranslator<RowData>, StreamExecNode<RowData>

@ExecNodeMetadata(name="stream-exec-deduplicate",
                  version=1,
                  consumedOptions={"table.exec.mini-batch.enabled","table.exec.mini-batch.size","table.exec.deduplicate.insert-update-after-sensitive-enabled","table.exec.deduplicate.mini-batch.compact-changes-enabled"},
                  producedTransformations="deduplicate",
                  minPlanVersion=v1_15,
                  minStateVersion=v1_15)
public class StreamExecDeduplicate
extends ExecNodeBase<RowData>
implements StreamExecNode<RowData>, SingleTransformationTranslator<RowData>

Stream ExecNode which deduplicate on keys and keeps only first row or last row. This node is an optimization of StreamExecRank for some special cases. Compared to StreamExecRank, this node could use mini-batch and access less state.

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEDUPLICATE_TRANSFORMATION
static String	FIELD_NAME_GENERATE_UPDATE_BEFORE
static String	FIELD_NAME_IS_ROWTIME
static String	FIELD_NAME_KEEP_LAST_ROW
static String	FIELD_NAME_UNIQUE_KEYS
static String	STATE_NAME

Fields inherited from interface org.apache.flink.table.planner.plan.nodes.exec.ExecNode

FIELD_NAME_CONFIGURATION, FIELD_NAME_DESCRIPTION, FIELD_NAME_ID, FIELD_NAME_INPUT_PROPERTIES, FIELD_NAME_OUTPUT_TYPE, FIELD_NAME_STATE, FIELD_NAME_TYPE

Constructor Summary

Constructors

Constructor and Description
StreamExecDeduplicate(int id, ExecNodeContext context, ReadableConfig persistedConfig, int[] uniqueKeys, boolean isRowtime, boolean keepLastRow, boolean generateUpdateBefore, List<StateMetadata> stateMetadataList, List<InputProperty> inputProperties, RowType outputType, String description)
StreamExecDeduplicate(ReadableConfig tableConfig, int[] uniqueKeys, boolean isRowtime, boolean keepLastRow, boolean generateUpdateBefore, InputProperty inputProperty, RowType outputType, String description)

Method Summary

Modifier and Type	Method and Description
protected Transformation<RowData>	translateToPlanInternal(org.apache.flink.table.planner.delegation.PlannerBase planner, ExecNodeConfig config) Internal method, translates this node into a Flink operator.

Methods inherited from class org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase

accept, createFormattedTransformationDescription, createFormattedTransformationName, createTransformationDescription, createTransformationMeta, createTransformationMeta, createTransformationName, createTransformationUid, getContextFromAnnotation, getDescription, getId, getInputEdges, getInputProperties, getOutputType, getPersistedConfig, getSimplifiedName, getTransformation, inputsContainSingleton, replaceInputEdge, setCompiled, setInputEdges, supportFusionCodegen, translateToFusionCodegenSpec, translateToFusionCodegenSpecInternal, translateToPlan

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.table.planner.plan.nodes.exec.ExecNode

accept, getDescription, getId, getInputEdges, getInputProperties, getOutputType, replaceInputEdge, setCompiled, setInputEdges

Methods inherited from interface org.apache.flink.table.planner.plan.nodes.exec.ExecNodeTranslator

translateToPlan

Methods inherited from interface org.apache.flink.table.planner.plan.nodes.exec.FusionCodegenExecNode

supportFusionCodegen, translateToFusionCodegenSpec

Field Detail

DEDUPLICATE_TRANSFORMATION

public static final String DEDUPLICATE_TRANSFORMATION

See Also:

Constant Field Values

FIELD_NAME_UNIQUE_KEYS

public static final String FIELD_NAME_UNIQUE_KEYS

See Also:

Constant Field Values

FIELD_NAME_IS_ROWTIME

public static final String FIELD_NAME_IS_ROWTIME

See Also:

Constant Field Values

FIELD_NAME_KEEP_LAST_ROW

public static final String FIELD_NAME_KEEP_LAST_ROW

See Also:

Constant Field Values

FIELD_NAME_GENERATE_UPDATE_BEFORE

public static final String FIELD_NAME_GENERATE_UPDATE_BEFORE

See Also:

Constant Field Values

STATE_NAME

public static final String STATE_NAME

See Also:

Constant Field Values

Constructor Detail

StreamExecDeduplicate

public StreamExecDeduplicate(ReadableConfig tableConfig,
                             int[] uniqueKeys,
                             boolean isRowtime,
                             boolean keepLastRow,
                             boolean generateUpdateBefore,
                             InputProperty inputProperty,
                             RowType outputType,
                             String description)

StreamExecDeduplicate

public StreamExecDeduplicate(int id,
                             ExecNodeContext context,
                             ReadableConfig persistedConfig,
                             int[] uniqueKeys,
                             boolean isRowtime,
                             boolean keepLastRow,
                             boolean generateUpdateBefore,
                             @Nullable
                             List<StateMetadata> stateMetadataList,
                             List<InputProperty> inputProperties,
                             RowType outputType,
                             String description)

Method Detail

translateToPlanInternal

protected Transformation<RowData> translateToPlanInternal(org.apache.flink.table.planner.delegation.PlannerBase planner,
                                                          ExecNodeConfig config)

Description copied from class: ExecNodeBase

Internal method, translates this node into a Flink operator.

Specified by:

translateToPlanInternal in class ExecNodeBase<RowData>

Parameters:

planner - The planner.

config - per-ExecNode configuration that contains the merged configuration from various layers which all the nodes implementing this method should use, instead of retrieving configuration from the planner. For more details check ExecNodeConfig.