org.apache.flink.connectors.hive.read

Class HiveTableInputFormat

java.lang.Object

org.apache.flink.api.common.io.RichInputFormat<T,SPITTYPE>

org.apache.flink.api.java.hadoop.common.HadoopInputFormatCommonBase<RowData,HiveTableInputSplit>

org.apache.flink.connectors.hive.read.HiveTableInputFormat

All Implemented Interfaces:

Serializable, CheckpointableInputFormat<HiveTableInputSplit,Long>, InputFormat<RowData,HiveTableInputSplit>, InputSplitSource<HiveTableInputSplit>

public class HiveTableInputFormat
extends HadoopInputFormatCommonBase<RowData,HiveTableInputSplit>
implements CheckpointableInputFormat<HiveTableInputSplit,Long>

The HiveTableInputFormat are inspired by the HCatInputFormat and HadoopInputFormatBase. It's used to read from hive partition/non-partition table.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected SplitReader	reader

Fields inherited from class org.apache.flink.api.java.hadoop.common.HadoopInputFormatCommonBase

credentials

Constructor Summary

Constructors

Constructor and Description
HiveTableInputFormat(int threadNum, org.apache.hadoop.mapred.JobConf jobConf, List<String> partitionKeys, DataType[] fieldTypes, String[] fieldNames, int[] projectedFields, Long limit, String hiveVersion, boolean useMapRedReader, List<HiveTablePartition> partitions)

Method Summary

Modifier and Type	Method and Description
void	close() Method that marks the end of the life-cycle of an input split.
void	configure(Configuration parameters) Configures this input format.
HiveTableInputSplit[]	createInputSplits(int minNumSplits) Computes the input splits.
static HiveTableInputSplit[]	createInputSplits(int minNumSplits, List<HiveTablePartition> partitions, int threadNum, org.apache.hadoop.mapred.JobConf jobConf)
Long	getCurrentState() Returns the split currently being read, along with its current state.
InputSplitAssigner	getInputSplitAssigner(HiveTableInputSplit[] inputSplits) Returns the assigner for the input splits.
org.apache.hadoop.mapred.JobConf	getJobConf()
BaseStatistics	getStatistics(BaseStatistics cachedStats) Gets the basic statistics from the input described by this format.
RowData	nextRecord(RowData reuse) Reads the next record from the input.
void	open(HiveTableInputSplit split) Opens a parallel instance of the input format to work on a split.
boolean	reachedEnd() Method used to check if the end of the input is reached.
void	reopen(HiveTableInputSplit split, Long state) Restores the state of a parallel instance reading from an InputFormat.

Methods inherited from class org.apache.flink.api.java.hadoop.common.HadoopInputFormatCommonBase

getCredentialsFromUGI, read, write

Methods inherited from class org.apache.flink.api.common.io.RichInputFormat

closeInputFormat, getRuntimeContext, openInputFormat, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

reader

@VisibleForTesting
protected transient SplitReader reader

Constructor Detail

HiveTableInputFormat

public HiveTableInputFormat(int threadNum,
                            org.apache.hadoop.mapred.JobConf jobConf,
                            List<String> partitionKeys,
                            DataType[] fieldTypes,
                            String[] fieldNames,
                            int[] projectedFields,
                            Long limit,
                            String hiveVersion,
                            boolean useMapRedReader,
                            List<HiveTablePartition> partitions)

Method Detail

getJobConf

public org.apache.hadoop.mapred.JobConf getJobConf()

configure

public void configure(Configuration parameters)

Description copied from interface: InputFormat

Configures this input format. Since input formats are instantiated generically and hence parameterless, this method is the place where the input formats set their basic fields based on configuration values.

This method is always called first on a newly instantiated input format.

Specified by:

configure in interface InputFormat<RowData,HiveTableInputSplit>

Parameters:

parameters - The configuration with all parameters (note: not the Flink config but the TaskConfig).

open

public void open(HiveTableInputSplit split)
          throws IOException

Description copied from interface: InputFormat

Opens a parallel instance of the input format to work on a split.

When this method is called, the input format it guaranteed to be configured.

Specified by:

open in interface InputFormat<RowData,HiveTableInputSplit>

Parameters:

split - The split to be opened.

Throws:

IOException - Thrown, if the spit could not be opened due to an I/O problem.

reopen

public void reopen(HiveTableInputSplit split,
                   Long state)
            throws IOException

Description copied from interface: CheckpointableInputFormat

Restores the state of a parallel instance reading from an InputFormat. This is necessary when recovering from a task failure. When this method is called, the input format it guaranteed to be configured.

NOTE: The caller has to make sure that the provided split is the one to whom the state belongs.

Specified by:

reopen in interface CheckpointableInputFormat<HiveTableInputSplit,Long>

Parameters:

split - The split to be opened.

state - The state from which to start from. This can contain the offset, but also other data, depending on the input format.

Throws:

IOException

getCurrentState

public Long getCurrentState()

Description copied from interface: CheckpointableInputFormat

Returns the split currently being read, along with its current state. This will be used to restore the state of the reading channel when recovering from a task failure. In the case of a simple text file, the state can correspond to the last read offset in the split.

Specified by:

getCurrentState in interface CheckpointableInputFormat<HiveTableInputSplit,Long>

Returns:

The state of the channel.

reachedEnd

public boolean reachedEnd()
                   throws IOException

Description copied from interface: InputFormat

Method used to check if the end of the input is reached.

When this method is called, the input format it guaranteed to be opened.

Specified by:

reachedEnd in interface InputFormat<RowData,HiveTableInputSplit>

Returns:

True if the end is reached, otherwise false.

Throws:

IOException - Thrown, if an I/O error occurred.

nextRecord

public RowData nextRecord(RowData reuse)
                   throws IOException

Description copied from interface: InputFormat

Reads the next record from the input.

When this method is called, the input format it guaranteed to be opened.

Specified by:

nextRecord in interface InputFormat<RowData,HiveTableInputSplit>

Parameters:

reuse - Object that may be reused.

Returns:

Read record.

Throws:

IOException - Thrown, if an I/O error occurred.

close

public void close()
           throws IOException

Description copied from interface: InputFormat

Method that marks the end of the life-cycle of an input split. Should be used to close channels and streams and release resources. After this method returns without an error, the input is assumed to be correctly read.

When this method is called, the input format it guaranteed to be opened.

Specified by:

close in interface InputFormat<RowData,HiveTableInputSplit>

Throws:

IOException - Thrown, if the input could not be closed properly.

createInputSplits

public HiveTableInputSplit[] createInputSplits(int minNumSplits)
                                        throws IOException

Description copied from interface: InputSplitSource

Computes the input splits. The given minimum number of splits is a hint as to how many splits are desired.

Specified by:

createInputSplits in interface InputFormat<RowData,HiveTableInputSplit>

Specified by:

createInputSplits in interface InputSplitSource<HiveTableInputSplit>

Parameters:

minNumSplits - Number of minimal input splits, as a hint.

Returns:

An array of input splits.

Throws:

IOException

createInputSplits

public static HiveTableInputSplit[] createInputSplits(int minNumSplits,
                                                      List<HiveTablePartition> partitions,
                                                      int threadNum,
                                                      org.apache.hadoop.mapred.JobConf jobConf)
                                               throws IOException

Throws:

IOException

getStatistics

public BaseStatistics getStatistics(BaseStatistics cachedStats)

Description copied from interface: InputFormat

Gets the basic statistics from the input described by this format. If the input format does not know how to create those statistics, it may return null. This method optionally gets a cached version of the statistics. The input format may examine them and decide whether it directly returns them without spending effort to re-gather the statistics.

When this method is called, the input format is guaranteed to be configured.

Specified by:

getStatistics in interface InputFormat<RowData,HiveTableInputSplit>

Parameters:

cachedStats - The statistics that were cached. May be null.

Returns:

The base statistics for the input, or null, if not available.

getInputSplitAssigner

public InputSplitAssigner getInputSplitAssigner(HiveTableInputSplit[] inputSplits)

Description copied from interface: InputSplitSource

Returns the assigner for the input splits. Assigner determines which parallel instance of the input format gets which input split.

Specified by:

getInputSplitAssigner in interface InputFormat<RowData,HiveTableInputSplit>

Specified by:

getInputSplitAssigner in interface InputSplitSource<HiveTableInputSplit>

Returns:

The input split assigner.