org.apache.flink.connector.hbase2.source

Class AbstractTableInputFormat<T>

java.lang.Object

org.apache.flink.api.common.io.RichInputFormat<T,TableInputSplit>

org.apache.flink.connector.hbase2.source.AbstractTableInputFormat<T>

All Implemented Interfaces:

Serializable, InputFormat<T,TableInputSplit>, InputSplitSource<TableInputSplit>

Direct Known Subclasses:

HBaseInputFormat, HBaseRowDataInputFormat, HBaseRowInputFormat

@Internal
public abstract class AbstractTableInputFormat<T>
extends RichInputFormat<T,TableInputSplit>

Abstract InputFormat to read data from HBase tables.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.hbase.client.Connection	connection
protected byte[]	currentRow
protected boolean	endReached
protected static org.slf4j.Logger	LOG
protected org.apache.hadoop.hbase.client.RegionLocator	regionLocator
protected org.apache.hadoop.hbase.client.ResultScanner	resultScanner HBase iterator wrapper.
protected org.apache.hadoop.hbase.client.Scan	scan
protected long	scannedRows
protected byte[]	serializedConfig
protected org.apache.hadoop.hbase.client.Table	table

Constructor Summary

Constructors

Constructor and Description
AbstractTableInputFormat(Configuration hConf)

Method Summary

Modifier and Type	Method and Description
void	close() Method that marks the end of the life-cycle of an input split.
void	closeTable()
void	configure(Configuration parameters) Configures this input format.
TableInputSplit[]	createInputSplits(int minNumSplits) Computes the input splits.
org.apache.hadoop.hbase.client.Connection	getConnection()
protected Configuration	getHadoopConfiguration()
InputSplitAssigner	getInputSplitAssigner(TableInputSplit[] inputSplits) Returns the assigner for the input splits.
protected abstract org.apache.hadoop.hbase.client.Scan	getScanner() Returns an instance of Scan that retrieves the required subset of records from the HBase table.
BaseStatistics	getStatistics(BaseStatistics cachedStatistics) Gets the basic statistics from the input described by this format.
protected abstract String	getTableName() What table is to be read.
protected boolean	includeRegionInScan(byte[] startKey, byte[] endKey) Test if the given region is to be included in the scan while splitting the regions of a table.
protected abstract void	initTable() Creates a Scan object and opens the HTable connection to initialize the HBase table.
protected abstract T	mapResultToOutType(org.apache.hadoop.hbase.client.Result r) HBase returns an instance of Result.
T	nextRecord(T reuse) Reads the next record from the input.
void	open(TableInputSplit split) Creates a Scan object and opens the HTable connection.
boolean	reachedEnd() Method used to check if the end of the input is reached.

Methods inherited from class org.apache.flink.api.common.io.RichInputFormat

closeInputFormat, getRuntimeContext, openInputFormat, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

protected static final org.slf4j.Logger LOG

endReached

protected boolean endReached

connection

protected transient org.apache.hadoop.hbase.client.Connection connection

table

protected transient org.apache.hadoop.hbase.client.Table table

regionLocator

protected transient org.apache.hadoop.hbase.client.RegionLocator regionLocator

scan

protected transient org.apache.hadoop.hbase.client.Scan scan

resultScanner

protected org.apache.hadoop.hbase.client.ResultScanner resultScanner

HBase iterator wrapper.

currentRow

protected byte[] currentRow

scannedRows

protected long scannedRows

serializedConfig

protected byte[] serializedConfig

Constructor Detail

AbstractTableInputFormat

public AbstractTableInputFormat(Configuration hConf)

Method Detail

initTable

protected abstract void initTable()
                           throws IOException

Creates a Scan object and opens the HTable connection to initialize the HBase table.

Throws:

IOException - Thrown, if the connection could not be opened due to an I/O problem.

getScanner

protected abstract org.apache.hadoop.hbase.client.Scan getScanner()

Returns an instance of Scan that retrieves the required subset of records from the HBase table.

Returns:

The appropriate instance of Scan for this use case.

getTableName

protected abstract String getTableName()

What table is to be read.

Per instance of a TableInputFormat derivative only a single table name is possible.

Returns:

The name of the table

mapResultToOutType

protected abstract T mapResultToOutType(org.apache.hadoop.hbase.client.Result r)

HBase returns an instance of Result.

This method maps the returned Result instance into the output type T.

Parameters:

r - The Result instance from HBase that needs to be converted

Returns:

The appropriate instance of T that contains the data of Result.

configure

public void configure(Configuration parameters)

Description copied from interface: InputFormat

Configures this input format. Since input formats are instantiated generically and hence parameterless, this method is the place where the input formats set their basic fields based on configuration values.

This method is always called first on a newly instantiated input format.

Parameters:

parameters - The configuration with all parameters (note: not the Flink config but the TaskConfig).

getHadoopConfiguration

protected Configuration getHadoopConfiguration()

open

public void open(TableInputSplit split)
          throws IOException

Creates a Scan object and opens the HTable connection. The connection is opened in this method and closed in close().

Parameters:

split - The split to be opened.

Throws:

IOException - Thrown, if the spit could not be opened due to an I/O problem.

nextRecord

public T nextRecord(T reuse)
             throws IOException

Description copied from interface: InputFormat

Reads the next record from the input.

When this method is called, the input format it guaranteed to be opened.

Parameters:

reuse - Object that may be reused.

Returns:

Read record.

Throws:

IOException - Thrown, if an I/O error occurred.

reachedEnd

public boolean reachedEnd()
                   throws IOException

Description copied from interface: InputFormat

Method used to check if the end of the input is reached.

When this method is called, the input format it guaranteed to be opened.

Returns:

True if the end is reached, otherwise false.

Throws:

IOException - Thrown, if an I/O error occurred.

close

public void close()
           throws IOException

Description copied from interface: InputFormat

Method that marks the end of the life-cycle of an input split. Should be used to close channels and streams and release resources. After this method returns without an error, the input is assumed to be correctly read.

When this method is called, the input format it guaranteed to be opened.

Throws:

IOException - Thrown, if the input could not be closed properly.

closeTable

public void closeTable()

createInputSplits

public TableInputSplit[] createInputSplits(int minNumSplits)
                                    throws IOException

Description copied from interface: InputSplitSource

Computes the input splits. The given minimum number of splits is a hint as to how many splits are desired.

Parameters:

minNumSplits - Number of minimal input splits, as a hint.

Returns:

An array of input splits.

Throws:

IOException

includeRegionInScan

protected boolean includeRegionInScan(byte[] startKey,
                                      byte[] endKey)

Test if the given region is to be included in the scan while splitting the regions of a table.

Parameters:

startKey - Start key of the region

endKey - End key of the region

Returns:

true, if this region needs to be included as part of the input (default).

getInputSplitAssigner

public InputSplitAssigner getInputSplitAssigner(TableInputSplit[] inputSplits)

Description copied from interface: InputSplitSource

Returns the assigner for the input splits. Assigner determines which parallel instance of the input format gets which input split.

Returns:

The input split assigner.

getStatistics

public BaseStatistics getStatistics(BaseStatistics cachedStatistics)

Description copied from interface: InputFormat

Gets the basic statistics from the input described by this format. If the input format does not know how to create those statistics, it may return null. This method optionally gets a cached version of the statistics. The input format may examine them and decide whether it directly returns them without spending effort to re-gather the statistics.

When this method is called, the input format is guaranteed to be configured.

Parameters:

cachedStatistics - The statistics that were cached. May be null.

Returns:

The base statistics for the input, or null, if not available.

getConnection

@VisibleForTesting
public org.apache.hadoop.hbase.client.Connection getConnection()