org.apache.flink.connectors.hive

Class HiveSourceFileEnumerator

java.lang.Object

org.apache.flink.connectors.hive.HiveSourceFileEnumerator

All Implemented Interfaces:

FileEnumerator

public class HiveSourceFileEnumerator
extends Object
implements FileEnumerator

A FileEnumerator implementation for hive source, which generates splits based on HiveTablePartitions.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HiveSourceFileEnumerator.Provider A factory to create HiveSourceFileEnumerator.

Constructor Summary

Constructors

Constructor and Description
HiveSourceFileEnumerator(List<HiveTablePartition> partitions, org.apache.hadoop.mapred.JobConf jobConf)

Method Summary

Modifier and Type	Method and Description
static List<HiveSourceSplit>	createInputSplits(int minNumSplits, List<HiveTablePartition> partitions, org.apache.hadoop.mapred.JobConf jobConf)
static org.apache.hadoop.mapred.InputSplit[]	createMRSplits(int minNumSplits, org.apache.hadoop.hive.metastore.api.StorageDescriptor sd, org.apache.hadoop.mapred.JobConf jobConf)
Collection<FileSourceSplit>	enumerateSplits(Path[] paths, int minDesiredSplits) Generates all file splits for the relevant files under the given paths.
static int	getNumFiles(List<HiveTablePartition> partitions, org.apache.hadoop.mapred.JobConf jobConf)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HiveSourceFileEnumerator

public HiveSourceFileEnumerator(List<HiveTablePartition> partitions,
                                org.apache.hadoop.mapred.JobConf jobConf)

Method Detail

enumerateSplits

public Collection<FileSourceSplit> enumerateSplits(Path[] paths,
                                                   int minDesiredSplits)
                                            throws IOException

Description copied from interface: FileEnumerator

Generates all file splits for the relevant files under the given paths. The minDesiredSplits is an optional hint indicating how many splits would be necessary to exploit parallelism properly.

Specified by:

enumerateSplits in interface FileEnumerator

Throws:

IOException

createInputSplits

public static List<HiveSourceSplit> createInputSplits(int minNumSplits,
                                                      List<HiveTablePartition> partitions,
                                                      org.apache.hadoop.mapred.JobConf jobConf)
                                               throws IOException

Throws:

IOException

createMRSplits

public static org.apache.hadoop.mapred.InputSplit[] createMRSplits(int minNumSplits,
                                                                   org.apache.hadoop.hive.metastore.api.StorageDescriptor sd,
                                                                   org.apache.hadoop.mapred.JobConf jobConf)
                                                            throws IOException

Throws:

IOException

getNumFiles

public static int getNumFiles(List<HiveTablePartition> partitions,
                              org.apache.hadoop.mapred.JobConf jobConf)
                       throws IOException

Throws:

IOException