org.apache.flink.connectors.hive

Class HiveLookupTableSource

java.lang.Object

org.apache.flink.connectors.hive.HiveTableSource

org.apache.flink.connectors.hive.HiveLookupTableSource

All Implemented Interfaces:

SupportsDynamicFiltering, SupportsLimitPushDown, SupportsPartitionPushDown, SupportsProjectionPushDown, SupportsStatisticReport, DynamicTableSource, LookupTableSource, ScanTableSource

public class HiveLookupTableSource
extends HiveTableSource
implements LookupTableSource

Hive Table Source that has lookup ability.

Hive Table source has both lookup and continuous read ability, when it acts as continuous read source it does not have the lookup ability but can be a temporal table just like other stream sources. When it acts as bounded table, it has the lookup ability.

A common user case is use hive table as dimension table and always lookup the latest partition data, in this case, hive table source is a continuous read source but currently we implements it by LookupFunction. Because currently TableSource can not tell the downstream when the latest partition has been read finished. This is a temporarily workaround and will re-implement in the future.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.connectors.hive.HiveTableSource

HiveTableSource.HiveContinuousPartitionFetcherContext<T extends Comparable<T>>

Nested classes/interfaces inherited from interface org.apache.flink.table.connector.source.LookupTableSource

LookupTableSource.LookupContext, LookupTableSource.LookupRuntimeProvider

Nested classes/interfaces inherited from interface org.apache.flink.table.connector.source.DynamicTableSource

DynamicTableSource.Context, DynamicTableSource.DataStructureConverter

Nested classes/interfaces inherited from interface org.apache.flink.table.connector.source.ScanTableSource

ScanTableSource.ScanContext, ScanTableSource.ScanRuntimeProvider

Field Summary

Fields inherited from class org.apache.flink.connectors.hive.HiveTableSource

catalogTable, dynamicFilterPartitionKeys, flinkConf, hiveShim, hiveVersion, jobConf, limit, producedDataType, projectedFields, remainingPartitions, tablePath

Constructor Summary

Constructors

Constructor and Description
HiveLookupTableSource(org.apache.hadoop.mapred.JobConf jobConf, ReadableConfig flinkConf, ObjectPath tablePath, ResolvedCatalogTable catalogTable)

Method Summary

Modifier and Type	Method and Description
DynamicTableSource	copy() Creates a copy of this instance during planning.
LookupTableSource.LookupRuntimeProvider	getLookupRuntimeProvider(LookupTableSource.LookupContext context) Returns a provider of runtime implementation for reading the data.

Methods inherited from class org.apache.flink.connectors.hive.HiveTableSource

applyDynamicFiltering, applyLimit, applyPartitions, applyProjection, asSummaryString, getChangelogMode, getDataStream, getJobConf, getScanRuntimeProvider, isStreamingSource, listAcceptedFilterFields, listPartitions, reportStatistics, supportsNestedProjection

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.table.connector.source.DynamicTableSource

asSummaryString

Methods inherited from interface org.apache.flink.table.connector.source.abilities.SupportsProjectionPushDown

applyProjection

Constructor Detail

HiveLookupTableSource

public HiveLookupTableSource(org.apache.hadoop.mapred.JobConf jobConf,
                             ReadableConfig flinkConf,
                             ObjectPath tablePath,
                             ResolvedCatalogTable catalogTable)

Method Detail

getLookupRuntimeProvider

public LookupTableSource.LookupRuntimeProvider getLookupRuntimeProvider(LookupTableSource.LookupContext context)

Description copied from interface: LookupTableSource

Returns a provider of runtime implementation for reading the data.

There exist different interfaces for runtime implementation which is why LookupTableSource.LookupRuntimeProvider serves as the base interface.

Independent of the provider interface, a source implementation can work on either arbitrary objects or internal data structures (see org.apache.flink.table.data for more information).

The given LookupTableSource.LookupContext offers utilities by the planner for creating runtime implementation with minimal dependencies to internal data structures.

Specified by:

getLookupRuntimeProvider in interface LookupTableSource

See Also:

LookupFunctionProvider, AsyncLookupFunctionProvider

copy

public DynamicTableSource copy()

Description copied from interface: DynamicTableSource

Creates a copy of this instance during planning. The copy should be a deep copy of all mutable members.

Specified by:

copy in interface DynamicTableSource

Overrides:

copy in class HiveTableSource