org.apache.flink.hcatalog

Class HCatInputFormatBase<T>

java.lang.Object

org.apache.flink.api.common.io.RichInputFormat<T,HadoopInputSplit>

org.apache.flink.hcatalog.HCatInputFormatBase<T>

Type Parameters:

T -

All Implemented Interfaces:

Serializable, InputFormat<T,HadoopInputSplit>, ResultTypeQueryable<T>, InputSplitSource<HadoopInputSplit>

Direct Known Subclasses:

HCatInputFormat

public abstract class HCatInputFormatBase<T>
extends RichInputFormat<T,HadoopInputSplit>
implements ResultTypeQueryable<T>

A InputFormat to read from HCatalog tables. The InputFormat supports projection (selection and order of fields) and partition filters.

Data can be returned as HCatRecord or Flink-native tuple.

Note: Flink tuples might only support a limited number of fields (depending on the API).

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected String[]	fieldNames
protected org.apache.hive.hcatalog.data.schema.HCatSchema	outputSchema

Constructor Summary

Constructors

Constructor and Description
HCatInputFormatBase()
HCatInputFormatBase(String database, String table) Creates a HCatInputFormat for the given database and table.
HCatInputFormatBase(String database, String table, Configuration config) Creates a HCatInputFormat for the given database, table, and Configuration.

Method Summary

Modifier and Type	Method and Description
HCatInputFormatBase<T>	asFlinkTuples() Specifies that the InputFormat returns Flink tuples instead of HCatRecord.
protected abstract T	buildFlinkTuple(T t, org.apache.hive.hcatalog.data.HCatRecord record)
void	close() Method that marks the end of the life-cycle of an input split.
void	configure(Configuration parameters) Configures this input format.
HadoopInputSplit[]	createInputSplits(int minNumSplits) Computes the input splits.
Configuration	getConfiguration() Returns the Configuration of the HCatInputFormat.
HCatInputFormatBase<T>	getFields(String... fields) Specifies the fields which are returned by the InputFormat and their order.
InputSplitAssigner	getInputSplitAssigner(HadoopInputSplit[] inputSplits) Returns the assigner for the input splits.
protected abstract int	getMaxFlinkTupleSize()
org.apache.hive.hcatalog.data.schema.HCatSchema	getOutputSchema() Returns the HCatSchema of the HCatRecord returned by this InputFormat.
TypeInformation<T>	getProducedType() Gets the data type (as a TypeInformation) produced by this function or input format.
BaseStatistics	getStatistics(BaseStatistics cachedStats) Gets the basic statistics from the input described by this format.
T	nextRecord(T record) Reads the next record from the input.
void	open(HadoopInputSplit split) Opens a parallel instance of the input format to work on a split.
boolean	reachedEnd() Method used to check if the end of the input is reached.
HCatInputFormatBase<T>	withFilter(String filter) Specifies a SQL-like filter condition on the table's partition columns.

Methods inherited from class org.apache.flink.api.common.io.RichInputFormat

closeInputFormat, getRuntimeContext, openInputFormat, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

fieldNames

protected String[] fieldNames

outputSchema

protected org.apache.hive.hcatalog.data.schema.HCatSchema outputSchema

Constructor Detail

HCatInputFormatBase

public HCatInputFormatBase()

HCatInputFormatBase

public HCatInputFormatBase(String database,
                           String table)
                    throws IOException

Creates a HCatInputFormat for the given database and table. By default, the InputFormat returns HCatRecord. The return type of the InputFormat can be changed to Flink-native tuples by calling asFlinkTuples().

Parameters:

database - The name of the database to read from.

table - The name of the table to read.

Throws:

IOException

HCatInputFormatBase

public HCatInputFormatBase(String database,
                           String table,
                           Configuration config)
                    throws IOException

Creates a HCatInputFormat for the given database, table, and Configuration. By default, the InputFormat returns HCatRecord. The return type of the InputFormat can be changed to Flink-native tuples by calling asFlinkTuples().

Parameters:

database - The name of the database to read from.

table - The name of the table to read.

config - The Configuration for the InputFormat.

Throws:

IOException

Method Detail

getFields

public HCatInputFormatBase<T> getFields(String... fields)
                                 throws IOException

Specifies the fields which are returned by the InputFormat and their order.

Parameters:

fields - The fields and their order which are returned by the InputFormat.

Returns:

This InputFormat with specified return fields.

Throws:

IOException

withFilter

public HCatInputFormatBase<T> withFilter(String filter)
                                  throws IOException

Specifies a SQL-like filter condition on the table's partition columns. Filter conditions on non-partition columns are invalid. A partition filter can significantly reduce the amount of data to be read.

Parameters:

filter - A SQL-like filter condition on the table's partition columns.

Returns:

This InputFormat with specified partition filter.

Throws:

IOException

asFlinkTuples

public HCatInputFormatBase<T> asFlinkTuples()
                                     throws org.apache.hive.hcatalog.common.HCatException

Specifies that the InputFormat returns Flink tuples instead of HCatRecord.

Note: Flink tuples might only support a limited number of fields (depending on the API).

Returns:

This InputFormat.

Throws:

org.apache.hive.hcatalog.common.HCatException

getMaxFlinkTupleSize

protected abstract int getMaxFlinkTupleSize()

getConfiguration

public Configuration getConfiguration()

Returns the Configuration of the HCatInputFormat.

Returns:

The Configuration of the HCatInputFormat.

getOutputSchema

public org.apache.hive.hcatalog.data.schema.HCatSchema getOutputSchema()

Returns the HCatSchema of the HCatRecord returned by this InputFormat.

Returns:

The HCatSchema of the HCatRecords returned by this InputFormat.

configure

public void configure(Configuration parameters)

Description copied from interface: InputFormat

Configures this input format. Since input formats are instantiated generically and hence parameterless, this method is the place where the input formats set their basic fields based on configuration values.

This method is always called first on a newly instantiated input format.

Specified by:

configure in interface InputFormat<T,HadoopInputSplit>

Parameters:

parameters - The configuration with all parameters (note: not the Flink config but the TaskConfig).

getStatistics

public BaseStatistics getStatistics(BaseStatistics cachedStats)
                             throws IOException

Description copied from interface: InputFormat

Gets the basic statistics from the input described by this format. If the input format does not know how to create those statistics, it may return null. This method optionally gets a cached version of the statistics. The input format may examine them and decide whether it directly returns them without spending effort to re-gather the statistics.

When this method is called, the input format is guaranteed to be configured.

Specified by:

getStatistics in interface InputFormat<T,HadoopInputSplit>

Parameters:

cachedStats - The statistics that were cached. May be null.

Returns:

The base statistics for the input, or null, if not available.

Throws:

IOException

createInputSplits

public HadoopInputSplit[] createInputSplits(int minNumSplits)
                                     throws IOException

Description copied from interface: InputSplitSource

Computes the input splits. The given minimum number of splits is a hint as to how many splits are desired.

Specified by:

createInputSplits in interface InputFormat<T,HadoopInputSplit>

Specified by:

createInputSplits in interface InputSplitSource<HadoopInputSplit>

Parameters:

minNumSplits - Number of minimal input splits, as a hint.

Returns:

An array of input splits.

Throws:

IOException

getInputSplitAssigner

public InputSplitAssigner getInputSplitAssigner(HadoopInputSplit[] inputSplits)

Description copied from interface: InputSplitSource

Returns the assigner for the input splits. Assigner determines which parallel instance of the input format gets which input split.

Specified by:

getInputSplitAssigner in interface InputFormat<T,HadoopInputSplit>

Specified by:

getInputSplitAssigner in interface InputSplitSource<HadoopInputSplit>

Returns:

The input split assigner.

open

public void open(HadoopInputSplit split)
          throws IOException

Description copied from interface: InputFormat

Opens a parallel instance of the input format to work on a split.

When this method is called, the input format it guaranteed to be configured.

Specified by:

open in interface InputFormat<T,HadoopInputSplit>

Parameters:

split - The split to be opened.

Throws:

IOException - Thrown, if the spit could not be opened due to an I/O problem.

reachedEnd

public boolean reachedEnd()
                   throws IOException

Description copied from interface: InputFormat

Method used to check if the end of the input is reached.

When this method is called, the input format it guaranteed to be opened.

Specified by:

reachedEnd in interface InputFormat<T,HadoopInputSplit>

Returns:

True if the end is reached, otherwise false.

Throws:

IOException - Thrown, if an I/O error occurred.

nextRecord

public T nextRecord(T record)
             throws IOException

Description copied from interface: InputFormat

Reads the next record from the input.

When this method is called, the input format it guaranteed to be opened.

Specified by:

nextRecord in interface InputFormat<T,HadoopInputSplit>

Parameters:

record - Object that may be reused.

Returns:

Read record.

Throws:

IOException - Thrown, if an I/O error occurred.

buildFlinkTuple

protected abstract T buildFlinkTuple(T t,
                                     org.apache.hive.hcatalog.data.HCatRecord record)
                              throws org.apache.hive.hcatalog.common.HCatException

Throws:

org.apache.hive.hcatalog.common.HCatException

close

public void close()
           throws IOException

Description copied from interface: InputFormat

Method that marks the end of the life-cycle of an input split. Should be used to close channels and streams and release resources. After this method returns without an error, the input is assumed to be correctly read.

When this method is called, the input format it guaranteed to be opened.

Specified by:

close in interface InputFormat<T,HadoopInputSplit>

Throws:

IOException - Thrown, if the input could not be closed properly.

getProducedType

public TypeInformation<T> getProducedType()

Description copied from interface: ResultTypeQueryable

Gets the data type (as a TypeInformation) produced by this function or input format.

Specified by:

getProducedType in interface ResultTypeQueryable<T>

Returns:

The data type produced by this function or input format.