org.apache.flink.connectors.hive

Class HiveSourceDynamicFileEnumerator

java.lang.Object

org.apache.flink.connectors.hive.HiveSourceDynamicFileEnumerator

All Implemented Interfaces:

DynamicFileEnumerator, FileEnumerator

public class HiveSourceDynamicFileEnumerator
extends Object
implements DynamicFileEnumerator

A DynamicFileEnumerator implementation for hive source. It uses HiveSourceFileEnumerator.createInputSplits(int, java.util.List<org.apache.flink.connectors.hive.HiveTablePartition>, org.apache.hadoop.mapred.JobConf, boolean) to generate splits like HiveSourceFileEnumerator, but only enumerates HiveTablePartitions that exist in the DynamicFilteringData if a DynamicFilteringData is provided.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HiveSourceDynamicFileEnumerator.Provider A factory to create HiveSourceDynamicFileEnumerator.

Field Summary

Fields

Modifier and Type	Field and Description
static Set<LogicalTypeRoot>	SUPPORTED_TYPES

Constructor Summary

Constructors

Constructor and Description
HiveSourceDynamicFileEnumerator(String table, List<String> dynamicFilterPartitionKeys, List<HiveTablePartition> allPartitions, String hiveVersion, org.apache.hadoop.mapred.JobConf jobConf)

Method Summary

Modifier and Type	Method and Description
Collection<FileSourceSplit>	enumerateSplits(Path[] paths, int minDesiredSplits) Generates all file splits for the relevant files under the given paths.
List<HiveTablePartition>	getFinalPartitions()
void	setDynamicFilteringData(DynamicFilteringData data) Provides a DynamicFilteringData for filtering while the enumerator is enumerating splits.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SUPPORTED_TYPES

public static final Set<LogicalTypeRoot> SUPPORTED_TYPES

Constructor Detail

HiveSourceDynamicFileEnumerator

public HiveSourceDynamicFileEnumerator(String table,
                                       List<String> dynamicFilterPartitionKeys,
                                       List<HiveTablePartition> allPartitions,
                                       String hiveVersion,
                                       org.apache.hadoop.mapred.JobConf jobConf)

Method Detail

setDynamicFilteringData

public void setDynamicFilteringData(DynamicFilteringData data)

Description copied from interface: DynamicFileEnumerator

Provides a DynamicFilteringData for filtering while the enumerator is enumerating splits.

The DynamicFilteringData is typically collected by a collector operator, and transferred here by a coordinating event. The method should never be called directly by users.

Specified by:

setDynamicFilteringData in interface DynamicFileEnumerator

enumerateSplits

public Collection<FileSourceSplit> enumerateSplits(Path[] paths,
                                                   int minDesiredSplits)
                                            throws IOException

Description copied from interface: FileEnumerator

Generates all file splits for the relevant files under the given paths. The minDesiredSplits is an optional hint indicating how many splits would be necessary to exploit parallelism properly.

Specified by:

enumerateSplits in interface FileEnumerator

Throws:

IOException

getFinalPartitions

public List<HiveTablePartition> getFinalPartitions()