org.apache.flink.connector.jdbc

Class JdbcInputFormat

java.lang.Object

org.apache.flink.api.common.io.RichInputFormat<Row,InputSplit>

org.apache.flink.connector.jdbc.JdbcInputFormat

All Implemented Interfaces:

Serializable, InputFormat<Row,InputSplit>, ResultTypeQueryable<Row>, InputSplitSource<InputSplit>

Direct Known Subclasses:

JDBCInputFormat

@Experimental
public class JdbcInputFormat
extends RichInputFormat<Row,InputSplit>
implements ResultTypeQueryable<Row>

InputFormat to read data from a database and generate Rows. The InputFormat has to be configured using the supplied InputFormatBuilder. A valid RowTypeInfo must be properly configured in the builder, e.g.:

 TypeInformation[] fieldTypes = new TypeInformation[] {
        BasicTypeInfo.INT_TYPE_INFO,
        BasicTypeInfo.STRING_TYPE_INFO,
        BasicTypeInfo.STRING_TYPE_INFO,
        BasicTypeInfo.DOUBLE_TYPE_INFO,
        BasicTypeInfo.INT_TYPE_INFO
 };

 RowTypeInfo rowTypeInfo = new RowTypeInfo(fieldTypes);

 JdbcInputFormat jdbcInputFormat = JdbcInputFormat.buildJdbcInputFormat()
                        .setDrivername("org.apache.derby.jdbc.EmbeddedDriver")
                        .setDBUrl("jdbc:derby:memory:ebookshop")
                        .setQuery("select * from books")
                        .setRowTypeInfo(rowTypeInfo)
                        .finish();
 

In order to query the JDBC source in parallel, you need to provide a parameterized query template (i.e. a valid PreparedStatement) and a JdbcParameterValuesProvider which provides binding values for the query parameters. E.g.:

 Serializable[][] queryParameters = new String[2][1];
 queryParameters[0] = new String[]{"Kumar"};
 queryParameters[1] = new String[]{"Tan Ah Teck"};

 JdbcInputFormat jdbcInputFormat = JdbcInputFormat.buildJdbcInputFormat()
                        .setDrivername("org.apache.derby.jdbc.EmbeddedDriver")
                        .setDBUrl("jdbc:derby:memory:ebookshop")
                        .setQuery("select * from books WHERE author = ?")
                        .setRowTypeInfo(rowTypeInfo)
                        .setParametersProvider(new JdbcGenericParameterValuesProvider(queryParameters))
                        .finish();
 

See Also:

Row, JdbcParameterValuesProvider, PreparedStatement, DriverManager, Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	JdbcInputFormat.JdbcInputFormatBuilder Builder for JdbcInputFormat.

Field Summary

Fields

Modifier and Type	Field and Description
protected Boolean	autoCommit
protected Connection	dbConn
protected String	dbURL
protected String	drivername
protected int	fetchSize
protected boolean	hasNext
protected static org.slf4j.Logger	LOG
protected Object[][]	parameterValues
protected String	password
protected String	queryTemplate
protected ResultSet	resultSet
protected int	resultSetConcurrency
protected int	resultSetType
protected RowTypeInfo	rowTypeInfo
protected static long	serialVersionUID
protected PreparedStatement	statement
protected String	username

Constructor Summary

Constructors

Constructor and Description
JdbcInputFormat()

Method Summary

Modifier and Type	Method and Description
static JdbcInputFormat.JdbcInputFormatBuilder	buildJdbcInputFormat() A builder used to set parameters to the output format's configuration in a fluent way.
void	close() Closes all resources used.
void	closeInputFormat() Closes this InputFormat instance.
void	configure(Configuration parameters) Configures this input format.
InputSplit[]	createInputSplits(int minNumSplits) Computes the input splits.
protected Connection	getDbConn()
InputSplitAssigner	getInputSplitAssigner(InputSplit[] inputSplits) Returns the assigner for the input splits.
RowTypeInfo	getProducedType() Gets the data type (as a TypeInformation) produced by this function or input format.
protected PreparedStatement	getStatement()
BaseStatistics	getStatistics(BaseStatistics cachedStatistics) Gets the basic statistics from the input described by this format.
Row	nextRecord(Row reuse) Stores the next resultSet row in a tuple.
void	open(InputSplit inputSplit) Connects to the source database and executes the query in a parallel fashion if this InputFormat is built using a parameterized query (i.e.
void	openInputFormat() Opens this InputFormat instance.
boolean	reachedEnd() Checks whether all data has been read.

Methods inherited from class org.apache.flink.api.common.io.RichInputFormat

getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

serialVersionUID

protected static final long serialVersionUID

See Also:

Constant Field Values

LOG

protected static final org.slf4j.Logger LOG

username

protected String username

password

protected String password

drivername

protected String drivername

dbURL

protected String dbURL

queryTemplate

protected String queryTemplate

resultSetType

protected int resultSetType

resultSetConcurrency

protected int resultSetConcurrency

rowTypeInfo

protected RowTypeInfo rowTypeInfo

dbConn

protected transient Connection dbConn

statement

protected transient PreparedStatement statement

resultSet

protected transient ResultSet resultSet

fetchSize

protected int fetchSize

autoCommit

protected Boolean autoCommit

hasNext

protected boolean hasNext

parameterValues

protected Object[][] parameterValues

Constructor Detail

JdbcInputFormat

public JdbcInputFormat()

Method Detail

getProducedType

public RowTypeInfo getProducedType()

Description copied from interface: ResultTypeQueryable

Gets the data type (as a TypeInformation) produced by this function or input format.

Specified by:

getProducedType in interface ResultTypeQueryable<Row>

Returns:

The data type produced by this function or input format.

configure

public void configure(Configuration parameters)

Description copied from interface: InputFormat

Configures this input format. Since input formats are instantiated generically and hence parameterless, this method is the place where the input formats set their basic fields based on configuration values.

This method is always called first on a newly instantiated input format.

Specified by:

configure in interface InputFormat<Row,InputSplit>

Parameters:

parameters - The configuration with all parameters (note: not the Flink config but the TaskConfig).

openInputFormat

public void openInputFormat()

Description copied from class: RichInputFormat

Opens this InputFormat instance. This method is called once per parallel instance. Resources should be allocated in this method. (e.g. database connections, cache, etc.)

Overrides:

openInputFormat in class RichInputFormat<Row,InputSplit>

See Also:

InputFormat

closeInputFormat

public void closeInputFormat()

Description copied from class: RichInputFormat

Closes this InputFormat instance. This method is called once per parallel instance. Resources allocated during RichInputFormat.openInputFormat() should be closed in this method.

Overrides:

closeInputFormat in class RichInputFormat<Row,InputSplit>

See Also:

InputFormat

open

public void open(InputSplit inputSplit)
          throws IOException

Connects to the source database and executes the query in a parallel fashion if this InputFormat is built using a parameterized query (i.e. using a PreparedStatement) and a proper JdbcParameterValuesProvider, in a non-parallel fashion otherwise.

Specified by:

open in interface InputFormat<Row,InputSplit>

Parameters:

inputSplit - which is ignored if this InputFormat is executed as a non-parallel source, a "hook" to the query parameters otherwise (using its splitNumber)

Throws:

IOException - if there's an error during the execution of the query

close

public void close()
           throws IOException

Closes all resources used.

Specified by:

close in interface InputFormat<Row,InputSplit>

Throws:

IOException - Indicates that a resource could not be closed.

reachedEnd

public boolean reachedEnd()
                   throws IOException

Checks whether all data has been read.

Specified by:

reachedEnd in interface InputFormat<Row,InputSplit>

Returns:

boolean value indication whether all data has been read.

Throws:

IOException

nextRecord

public Row nextRecord(Row reuse)
               throws IOException

Stores the next resultSet row in a tuple.

Specified by:

nextRecord in interface InputFormat<Row,InputSplit>

Parameters:

reuse - row to be reused.

Returns:

row containing next Row

Throws:

IOException

getStatistics

public BaseStatistics getStatistics(BaseStatistics cachedStatistics)
                             throws IOException

Description copied from interface: InputFormat

Gets the basic statistics from the input described by this format. If the input format does not know how to create those statistics, it may return null. This method optionally gets a cached version of the statistics. The input format may examine them and decide whether it directly returns them without spending effort to re-gather the statistics.

When this method is called, the input format is guaranteed to be configured.

Specified by:

getStatistics in interface InputFormat<Row,InputSplit>

Parameters:

cachedStatistics - The statistics that were cached. May be null.

Returns:

The base statistics for the input, or null, if not available.

Throws:

IOException

createInputSplits

public InputSplit[] createInputSplits(int minNumSplits)
                               throws IOException

Description copied from interface: InputSplitSource

Computes the input splits. The given minimum number of splits is a hint as to how many splits are desired.

Specified by:

createInputSplits in interface InputFormat<Row,InputSplit>

Specified by:

createInputSplits in interface InputSplitSource<InputSplit>

Parameters:

minNumSplits - Number of minimal input splits, as a hint.

Returns:

An array of input splits.

Throws:

IOException

getInputSplitAssigner

public InputSplitAssigner getInputSplitAssigner(InputSplit[] inputSplits)

Description copied from interface: InputSplitSource

Returns the assigner for the input splits. Assigner determines which parallel instance of the input format gets which input split.

Specified by:

getInputSplitAssigner in interface InputFormat<Row,InputSplit>

Specified by:

getInputSplitAssigner in interface InputSplitSource<InputSplit>

Returns:

The input split assigner.

getStatement

@VisibleForTesting
protected PreparedStatement getStatement()

getDbConn

@VisibleForTesting
protected Connection getDbConn()

buildJdbcInputFormat

public static JdbcInputFormat.JdbcInputFormatBuilder buildJdbcInputFormat()

A builder used to set parameters to the output format's configuration in a fluent way.

Returns:

builder