org.apache.flink.connector.hbase.source

Class HBaseRowDataInputFormat

java.lang.Object

org.apache.flink.api.common.io.RichInputFormat<T,org.apache.flink.connector.hbase.source.TableInputSplit>

org.apache.flink.connector.hbase.source.HBaseRowDataInputFormat

All Implemented Interfaces:

Serializable, InputFormat<RowData,org.apache.flink.connector.hbase.source.TableInputSplit>, InputSplitSource<org.apache.flink.connector.hbase.source.TableInputSplit>

public class HBaseRowDataInputFormat
extends RichInputFormat<T,org.apache.flink.connector.hbase.source.TableInputSplit>

InputFormat subclass that wraps the access for HTables. Returns the result as RowData

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected byte[]	currentRow
protected boolean	endReached
protected org.apache.hadoop.hbase.client.ResultScanner	resultScanner HBase iterator wrapper.
protected org.apache.hadoop.hbase.client.Scan	scan
protected long	scannedRows
protected byte[]	serializedConfig
protected org.apache.hadoop.hbase.client.HTable	table

Constructor Summary

Constructors

Constructor and Description
HBaseRowDataInputFormat(Configuration conf, String tableName, HBaseTableSchema schema, String nullStringLiteral)

Method Summary

Modifier and Type	Method and Description
void	close() Method that marks the end of the life-cycle of an input split.
void	closeInputFormat() Closes this InputFormat instance.
void	configure(Configuration parameters) Creates a Scan object and opens the HTable connection.
org.apache.flink.connector.hbase.source.TableInputSplit[]	createInputSplits(int minNumSplits) Computes the input splits.
protected Configuration	getHadoopConfiguration()
InputSplitAssigner	getInputSplitAssigner(org.apache.flink.connector.hbase.source.TableInputSplit[] inputSplits) Returns the assigner for the input splits.
protected org.apache.hadoop.hbase.client.Scan	getScanner() Returns an instance of Scan that retrieves the required subset of records from the HBase table.
BaseStatistics	getStatistics(BaseStatistics cachedStatistics) Gets the basic statistics from the input described by this format.
String	getTableName() What table is to be read.
protected boolean	includeRegionInScan(byte[] startKey, byte[] endKey) Test if the given region is to be included in the scan while splitting the regions of a table.
protected RowData	mapResultToOutType(org.apache.hadoop.hbase.client.Result res) HBase returns an instance of Result.
T	nextRecord(T reuse) Reads the next record from the input.
void	open(org.apache.flink.connector.hbase.source.TableInputSplit split) Opens a parallel instance of the input format to work on a split.
boolean	reachedEnd() Method used to check if the end of the input is reached.

Methods inherited from class org.apache.flink.api.common.io.RichInputFormat

getRuntimeContext, openInputFormat, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

endReached

protected boolean endReached

table

protected transient org.apache.hadoop.hbase.client.HTable table

scan

protected transient org.apache.hadoop.hbase.client.Scan scan

resultScanner

protected org.apache.hadoop.hbase.client.ResultScanner resultScanner

HBase iterator wrapper.

currentRow

protected byte[] currentRow

scannedRows

protected long scannedRows

serializedConfig

protected byte[] serializedConfig

Constructor Detail

HBaseRowDataInputFormat

public HBaseRowDataInputFormat(Configuration conf,
                               String tableName,
                               HBaseTableSchema schema,
                               String nullStringLiteral)

Method Detail

configure

public void configure(Configuration parameters)

Creates a Scan object and opens the HTable connection.

These are opened here because they are needed in the createInputSplits which is called before the openInputFormat method.

The connection is opened in this method and closed in closeInputFormat().

Specified by:

configure in interface InputFormat<RowData,org.apache.flink.connector.hbase.source.TableInputSplit>

Parameters:

parameters - The configuration that is to be used

See Also:

Configuration

getScanner

protected org.apache.hadoop.hbase.client.Scan getScanner()

Returns an instance of Scan that retrieves the required subset of records from the HBase table.

Returns:

The appropriate instance of Scan for this use case.

getTableName

public String getTableName()

What table is to be read.

Per instance of a TableInputFormat derivative only a single table name is possible.

Returns:

The name of the table

mapResultToOutType

protected RowData mapResultToOutType(org.apache.hadoop.hbase.client.Result res)

HBase returns an instance of Result.

This method maps the returned Result instance into the output type T.

Parameters:

res - The Result instance from HBase that needs to be converted

Returns:

The appropriate instance of T that contains the data of Result.

getHadoopConfiguration

protected Configuration getHadoopConfiguration()

open

public void open(org.apache.flink.connector.hbase.source.TableInputSplit split)
          throws IOException

Description copied from interface: InputFormat

Opens a parallel instance of the input format to work on a split.

When this method is called, the input format it guaranteed to be configured.

Parameters:

split - The split to be opened.

Throws:

IOException - Thrown, if the spit could not be opened due to an I/O problem.

nextRecord

public T nextRecord(T reuse)
             throws IOException

Description copied from interface: InputFormat

Reads the next record from the input.

When this method is called, the input format it guaranteed to be opened.

Parameters:

reuse - Object that may be reused.

Returns:

Read record.

Throws:

IOException - Thrown, if an I/O error occurred.

reachedEnd

public boolean reachedEnd()
                   throws IOException

Description copied from interface: InputFormat

Method used to check if the end of the input is reached.

When this method is called, the input format it guaranteed to be opened.

Returns:

True if the end is reached, otherwise false.

Throws:

IOException - Thrown, if an I/O error occurred.

close

public void close()
           throws IOException

Description copied from interface: InputFormat

Method that marks the end of the life-cycle of an input split. Should be used to close channels and streams and release resources. After this method returns without an error, the input is assumed to be correctly read.

When this method is called, the input format it guaranteed to be opened.

Throws:

IOException - Thrown, if the input could not be closed properly.

closeInputFormat

public void closeInputFormat()
                      throws IOException

Description copied from class: RichInputFormat

Closes this InputFormat instance. This method is called once per parallel instance. Resources allocated during RichInputFormat.openInputFormat() should be closed in this method.

Overrides:

closeInputFormat in class RichInputFormat<T,org.apache.flink.connector.hbase.source.TableInputSplit>

Throws:

IOException - in case closing the resources failed

See Also:

InputFormat

createInputSplits

public org.apache.flink.connector.hbase.source.TableInputSplit[] createInputSplits(int minNumSplits)
                                                                            throws IOException

Description copied from interface: InputSplitSource

Computes the input splits. The given minimum number of splits is a hint as to how many splits are desired.

Parameters:

minNumSplits - Number of minimal input splits, as a hint.

Returns:

An array of input splits.

Throws:

IOException

includeRegionInScan

protected boolean includeRegionInScan(byte[] startKey,
                                      byte[] endKey)

Test if the given region is to be included in the scan while splitting the regions of a table.

Parameters:

startKey - Start key of the region

endKey - End key of the region

Returns:

true, if this region needs to be included as part of the input (default).

getInputSplitAssigner

public InputSplitAssigner getInputSplitAssigner(org.apache.flink.connector.hbase.source.TableInputSplit[] inputSplits)

Description copied from interface: InputSplitSource

Returns the assigner for the input splits. Assigner determines which parallel instance of the input format gets which input split.

Returns:

The input split assigner.

getStatistics

public BaseStatistics getStatistics(BaseStatistics cachedStatistics)

Description copied from interface: InputFormat

Gets the basic statistics from the input described by this format. If the input format does not know how to create those statistics, it may return null. This method optionally gets a cached version of the statistics. The input format may examine them and decide whether it directly returns them without spending effort to re-gather the statistics.

When this method is called, the input format is guaranteed to be configured.

Parameters:

cachedStatistics - The statistics that were cached. May be null.

Returns:

The base statistics for the input, or null, if not available.