org.apache.flink.table.sources

Class CsvTableSource

java.lang.Object

org.apache.flink.table.sources.CsvTableSource

All Implemented Interfaces:

BatchTableSource<Row>, LookupableTableSource<Row>, ProjectableTableSource<Row>, StreamTableSource<Row>, TableSource<Row>

public class CsvTableSource
extends Object
implements StreamTableSource<Row>, BatchTableSource<Row>, LookupableTableSource<Row>, ProjectableTableSource<Row>

A StreamTableSource and BatchTableSource for simple CSV files with a (logically) unlimited number of fields.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CsvTableSource.Builder A builder for creating CsvTableSource instances.
static class	CsvTableSource.CsvLookupFunction LookupFunction to support lookup in CsvTableSource.

Constructor Summary

Constructors

Constructor and Description
CsvTableSource(String path, String[] fieldNames, TypeInformation<?>[] fieldTypes) A InputFormatTableSource and LookupableTableSource for simple CSV files with a (logically) unlimited number of fields.
CsvTableSource(String path, String[] fieldNames, TypeInformation<?>[] fieldTypes, int[] selectedFields, String fieldDelim, String lineDelim, Character quoteCharacter, boolean ignoreFirstLine, String ignoreComments, boolean lenient) A InputFormatTableSource and LookupableTableSource for simple CSV files with a (logically) unlimited number of fields.
CsvTableSource(String path, String[] fieldNames, TypeInformation<?>[] fieldTypes, String fieldDelim, String lineDelim, Character quoteCharacter, boolean ignoreFirstLine, String ignoreComments, boolean lenient) A InputFormatTableSource and LookupableTableSource for simple CSV files with a (logically) unlimited number of fields.

Method Summary

Modifier and Type	Method and Description
static CsvTableSource.Builder	builder() Return a new builder that builds a CsvTableSource.
boolean	equals(Object o)
String	explainSource() Describes the table source.
AsyncTableFunction<Row>	getAsyncLookupFunction(String[] lookupKeys) Gets the AsyncTableFunction which supports async lookup one key at a time.
DataSet<Row>	getDataSet(ExecutionEnvironment execEnv) Returns the data of the table as a DataSet.
DataStream<Row>	getDataStream(StreamExecutionEnvironment execEnv) Returns the data of the table as a DataStream.
TableFunction<Row>	getLookupFunction(String[] lookupKeys) Gets the TableFunction which supports lookup one key at a time.
TypeInformation<Row>	getReturnType()
TableSchema	getTableSchema() Returns the schema of the produced table.
int	hashCode()
boolean	isAsyncEnabled() Returns true if async lookup is enabled.
boolean	isBounded() Returns true if this is a bounded source, false if this is an unbounded source.
CsvTableSource	projectFields(int[] fields) Creates a copy of the TableSource that projects its output to the given field indexes.

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.table.sources.TableSource

getProducedDataType

Constructor Detail

CsvTableSource

public CsvTableSource(String path,
                      String[] fieldNames,
                      TypeInformation<?>[] fieldTypes)

A InputFormatTableSource and LookupableTableSource for simple CSV files with a (logically) unlimited number of fields.

Parameters:

path - The path to the CSV file.

fieldNames - The names of the table fields.

fieldTypes - The types of the table fields.

CsvTableSource

public CsvTableSource(String path,
                      String[] fieldNames,
                      TypeInformation<?>[] fieldTypes,
                      String fieldDelim,
                      String lineDelim,
                      Character quoteCharacter,
                      boolean ignoreFirstLine,
                      String ignoreComments,
                      boolean lenient)

A InputFormatTableSource and LookupableTableSource for simple CSV files with a (logically) unlimited number of fields.

Parameters:

path - The path to the CSV file.

fieldNames - The names of the table fields.

fieldTypes - The types of the table fields.

fieldDelim - The field delimiter, "," by default.

lineDelim - The row delimiter, "\n" by default.

quoteCharacter - An optional quote character for String values, null by default.

ignoreFirstLine - Flag to ignore the first line, false by default.

ignoreComments - An optional prefix to indicate comments, null by default.

lenient - Flag to skip records with parse error instead to fail, false by default.

CsvTableSource

public CsvTableSource(String path,
                      String[] fieldNames,
                      TypeInformation<?>[] fieldTypes,
                      int[] selectedFields,
                      String fieldDelim,
                      String lineDelim,
                      Character quoteCharacter,
                      boolean ignoreFirstLine,
                      String ignoreComments,
                      boolean lenient)

A InputFormatTableSource and LookupableTableSource for simple CSV files with a (logically) unlimited number of fields.

Parameters:

path - The path to the CSV file.

fieldNames - The names of the table fields.

fieldTypes - The types of the table fields.

selectedFields - The fields which will be read and returned by the table source. If None, all fields are returned.

fieldDelim - The field delimiter, "," by default.

lineDelim - The row delimiter, "\n" by default.

quoteCharacter - An optional quote character for String values, null by default.

ignoreFirstLine - Flag to ignore the first line, false by default.

ignoreComments - An optional prefix to indicate comments, null by default.

lenient - Flag to skip records with parse error instead to fail, false by default.

Method Detail

builder

public static CsvTableSource.Builder builder()

Return a new builder that builds a CsvTableSource. For example:

 CsvTableSource source = new CsvTableSource.builder()
     .path("/path/to/your/file.csv")
     .field("myfield", Types.STRING)
     .field("myfield2", Types.INT)
     .build();
 

Returns:

a new builder to build a CsvTableSource

getReturnType

public TypeInformation<Row> getReturnType()

Specified by:

getReturnType in interface TableSource<Row>

getTableSchema

public TableSchema getTableSchema()

Description copied from interface: TableSource

Returns the schema of the produced table.

Specified by:

getTableSchema in interface TableSource<Row>

Returns:

The TableSchema of the produced table.

projectFields

public CsvTableSource projectFields(int[] fields)

Description copied from interface: ProjectableTableSource

Creates a copy of the TableSource that projects its output to the given field indexes. The field indexes relate to the physical poduced data type (TableSource.getProducedDataType()) and not to the table schema (TableSource.getTableSchema() of the TableSource.

The table schema (TableSource.getTableSchema() of the TableSource copy must not be modified by this method, but only the produced data type (TableSource.getProducedDataType()) and the produced DataSet (BatchTableSource#getDataSet() or DataStream (StreamTableSource#getDataStream).

If the TableSource implements the DefinedFieldMapping interface, it might be necessary to adjust the mapping as well.

IMPORTANT: This method must return a true copy and must not modify the original table source object.

Specified by:

projectFields in interface ProjectableTableSource<Row>

Parameters:

fields - The indexes of the fields to return.

Returns:

A copy of the TableSource that projects its output.

isBounded

public boolean isBounded()

Description copied from interface: StreamTableSource

Returns true if this is a bounded source, false if this is an unbounded source. Default is unbounded for compatibility.

Specified by:

isBounded in interface StreamTableSource<Row>

getDataStream

public DataStream<Row> getDataStream(StreamExecutionEnvironment execEnv)

Description copied from interface: StreamTableSource

Returns the data of the table as a DataStream.

NOTE: This method is for internal use only for defining a TableSource. Do not use it in Table API programs.

Specified by:

getDataStream in interface StreamTableSource<Row>

getDataSet

public DataSet<Row> getDataSet(ExecutionEnvironment execEnv)

Description copied from interface: BatchTableSource

Returns the data of the table as a DataSet.

NOTE: This method is for internal use only for defining a TableSource. Do not use it in Table API programs.

Specified by:

getDataSet in interface BatchTableSource<Row>

getLookupFunction

public TableFunction<Row> getLookupFunction(String[] lookupKeys)

Description copied from interface: LookupableTableSource

Gets the TableFunction which supports lookup one key at a time.

Specified by:

getLookupFunction in interface LookupableTableSource<Row>

Parameters:

lookupKeys - the chosen field names as lookup keys, it is in the defined order

getAsyncLookupFunction

public AsyncTableFunction<Row> getAsyncLookupFunction(String[] lookupKeys)

Description copied from interface: LookupableTableSource

Gets the AsyncTableFunction which supports async lookup one key at a time.

Specified by:

getAsyncLookupFunction in interface LookupableTableSource<Row>

Parameters:

lookupKeys - the chosen field names as lookup keys, it is in the defined order

isAsyncEnabled

public boolean isAsyncEnabled()

Description copied from interface: LookupableTableSource

Returns true if async lookup is enabled.

The lookup function returned by LookupableTableSource.getAsyncLookupFunction(String[]) will be used if returns true. Otherwise, the lookup function returned by LookupableTableSource.getLookupFunction(String[]) will be used.

Specified by:

isAsyncEnabled in interface LookupableTableSource<Row>

explainSource

public String explainSource()

Description copied from interface: TableSource

Describes the table source.

Specified by:

explainSource in interface TableSource<Row>

Returns:

A String explaining the TableSource.

equals

public boolean equals(Object o)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object