org.apache.flink.streaming.connectors.kinesis.table

Class KinesisDynamicSource

java.lang.Object

org.apache.flink.streaming.connectors.kinesis.table.KinesisDynamicSource

All Implemented Interfaces:

SupportsReadingMetadata, DynamicTableSource, ScanTableSource

@Internal
public class KinesisDynamicSource
extends Object
implements ScanTableSource, SupportsReadingMetadata

Kinesis-backed ScanTableSource.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.table.connector.source.ScanTableSource

ScanTableSource.ScanContext, ScanTableSource.ScanRuntimeProvider

Nested classes/interfaces inherited from interface org.apache.flink.table.connector.source.DynamicTableSource

DynamicTableSource.Context, DynamicTableSource.DataStructureConverter

Constructor Summary

Constructors

Constructor and Description
KinesisDynamicSource(DataType physicalDataType, String stream, Properties consumerProperties, DecodingFormat<DeserializationSchema<RowData>> decodingFormat)
KinesisDynamicSource(DataType physicalDataType, String stream, Properties consumerProperties, DecodingFormat<DeserializationSchema<RowData>> decodingFormat, DataType producedDataType, List<RowDataKinesisDeserializationSchema.Metadata> requestedMetadataFields)

Method Summary

Modifier and Type	Method and Description
void	applyReadableMetadata(List<String> metadataKeys, DataType producedDataType) Provides a list of metadata keys that the produced RowData must contain as appended metadata columns.
String	asSummaryString() Returns a string that summarizes this source for printing to a console or log.
DynamicTableSource	copy() Creates a copy of this instance during planning.
boolean	equals(Object o)
ChangelogMode	getChangelogMode() Returns the set of changes that the planner can expect during runtime.
ScanTableSource.ScanRuntimeProvider	getScanRuntimeProvider(ScanTableSource.ScanContext runtimeProviderContext) Returns a provider of runtime implementation for reading the data.
int	hashCode()
Map<String,DataType>	listReadableMetadata() Returns the map of metadata keys and their corresponding data types that can be produced by this table source for reading.

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.table.connector.source.abilities.SupportsReadingMetadata

supportsMetadataProjection

Constructor Detail

KinesisDynamicSource

public KinesisDynamicSource(DataType physicalDataType,
                            String stream,
                            Properties consumerProperties,
                            DecodingFormat<DeserializationSchema<RowData>> decodingFormat)

KinesisDynamicSource

public KinesisDynamicSource(DataType physicalDataType,
                            String stream,
                            Properties consumerProperties,
                            DecodingFormat<DeserializationSchema<RowData>> decodingFormat,
                            DataType producedDataType,
                            List<RowDataKinesisDeserializationSchema.Metadata> requestedMetadataFields)

Method Detail

getChangelogMode

public ChangelogMode getChangelogMode()

Description copied from interface: ScanTableSource

Returns the set of changes that the planner can expect during runtime.

Specified by:

getChangelogMode in interface ScanTableSource

See Also:

RowKind

getScanRuntimeProvider

public ScanTableSource.ScanRuntimeProvider getScanRuntimeProvider(ScanTableSource.ScanContext runtimeProviderContext)

Description copied from interface: ScanTableSource

Returns a provider of runtime implementation for reading the data.

There might exist different interfaces for runtime implementation which is why ScanTableSource.ScanRuntimeProvider serves as the base interface. Concrete ScanTableSource.ScanRuntimeProvider interfaces might be located in other Flink modules.

Independent of the provider interface, the table runtime expects that a source implementation emits internal data structures (see RowData for more information).

The given ScanTableSource.ScanContext offers utilities by the planner for creating runtime implementation with minimal dependencies to internal data structures.

SourceProvider is the recommended core interface. SourceFunctionProvider in flink-table-api-java-bridge and InputFormatProvider are available for backwards compatibility.

Specified by:

getScanRuntimeProvider in interface ScanTableSource

See Also:

SourceProvider

copy

public DynamicTableSource copy()

Description copied from interface: DynamicTableSource

Creates a copy of this instance during planning. The copy should be a deep copy of all mutable members.

Specified by:

copy in interface DynamicTableSource

asSummaryString

public String asSummaryString()

Description copied from interface: DynamicTableSource

Returns a string that summarizes this source for printing to a console or log.

Specified by:

asSummaryString in interface DynamicTableSource

listReadableMetadata

public Map<String,DataType> listReadableMetadata()

Description copied from interface: SupportsReadingMetadata

Returns the map of metadata keys and their corresponding data types that can be produced by this table source for reading.

The returned map will be used by the planner for validation and insertion of explicit casts (see LogicalTypeCasts.supportsExplicitCast(LogicalType, LogicalType)) if necessary.

The iteration order of the returned map determines the order of metadata keys in the list passed in SupportsReadingMetadata.applyReadableMetadata(List, DataType). Therefore, it might be beneficial to return a LinkedHashMap if a strict metadata column order is required.

If a source forwards metadata from one or more formats, we recommend the following column order for consistency:

 KEY FORMAT METADATA COLUMNS + VALUE FORMAT METADATA COLUMNS + SOURCE METADATA COLUMNS
 

Metadata key names follow the same pattern as mentioned in Factory. In case of duplicate names in format and source keys, format keys shall have higher precedence.

Regardless of the returned DataTypes, a metadata column is always represented using internal data structures (see RowData).

Specified by:

listReadableMetadata in interface SupportsReadingMetadata

See Also:

DecodingFormat.listReadableMetadata()

applyReadableMetadata

public void applyReadableMetadata(List<String> metadataKeys,
                                  DataType producedDataType)

Description copied from interface: SupportsReadingMetadata

Provides a list of metadata keys that the produced RowData must contain as appended metadata columns.

Implementations of this method must be idempotent. The planner might call this method multiple times.

Note: Use the passed data type instead of ResolvedSchema.toPhysicalRowDataType() for describing the final output data type when creating TypeInformation. If the source implements SupportsProjectionPushDown, the projection is already considered in the given output data type, use the producedDataType provided by this method instead of the producedDataType provided by SupportsProjectionPushDown.applyProjection(int[][], DataType).

Specified by:

applyReadableMetadata in interface SupportsReadingMetadata

Parameters:

metadataKeys - a subset of the keys returned by SupportsReadingMetadata.listReadableMetadata(), ordered by the iteration order of returned map

producedDataType - the final output type of the source, it is intended to be only forwarded and the planner will decide on the field names to avoid collisions

See Also:

DecodingFormat.applyReadableMetadata(List)

equals

public boolean equals(Object o)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object