org.apache.flink.table.filesystem

Class FileSystemTableSource

java.lang.Object

org.apache.flink.table.filesystem.FileSystemTableSource

All Implemented Interfaces:

FilterableTableSource<RowData>, LimitableTableSource<RowData>, PartitionableTableSource, ProjectableTableSource<RowData>, StreamTableSource<RowData>, TableSource<RowData>

public class FileSystemTableSource
extends Object
implements StreamTableSource<RowData>, PartitionableTableSource, ProjectableTableSource<RowData>, LimitableTableSource<RowData>, FilterableTableSource<RowData>

File system table source.

Constructor Summary

Constructors

Constructor and Description
FileSystemTableSource(TableSchema schema, Path path, List<String> partitionKeys, String defaultPartName, Map<String,String> properties) Construct a file system table source.

Method Summary

Modifier and Type	Method and Description
FileSystemTableSource	applyLimit(long limit) Check and push down the limit to the table source.
FileSystemTableSource	applyPartitionPruning(List<Map<String,String>> remainingPartitions) Applies the remaining partitions to the table source.
FileSystemTableSource	applyPredicate(List<Expression> predicates) Check and pick all predicates this table source can support.
String	explainSource() Describes the table source.
DataStream<RowData>	getDataStream(StreamExecutionEnvironment execEnv) Returns the data of the table as a DataStream.
List<Map<String,String>>	getPartitions() Returns all the partitions of this PartitionableTableSource.
DataType	getProducedDataType() Returns the DataType for the produced data of the TableSource.
TableSchema	getTableSchema() Returns the schema of the produced table.
boolean	isBounded() Returns true if this is a bounded source, false if this is an unbounded source.
boolean	isFilterPushedDown() Return the flag to indicate whether filter push down has been tried.
boolean	isLimitPushedDown() Return the flag to indicate whether limit push down has been tried.
FileSystemTableSource	projectFields(int[] fields) Creates a copy of the TableSource that projects its output to the given field indexes.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.table.sources.TableSource

getReturnType

Constructor Detail

FileSystemTableSource

public FileSystemTableSource(TableSchema schema,
                             Path path,
                             List<String> partitionKeys,
                             String defaultPartName,
                             Map<String,String> properties)

Construct a file system table source.

Parameters:

schema - schema of the table.

path - directory path of the file system table.

partitionKeys - partition keys of the table.

defaultPartName - The default partition name in case the dynamic partition column value is null/empty string.

properties - table properties.

Method Detail

getDataStream

public DataStream<RowData> getDataStream(StreamExecutionEnvironment execEnv)

Description copied from interface: StreamTableSource

Returns the data of the table as a DataStream.

NOTE: This method is for internal use only for defining a TableSource. Do not use it in Table API programs.

Specified by:

getDataStream in interface StreamTableSource<RowData>

isBounded

public boolean isBounded()

Description copied from interface: StreamTableSource

Returns true if this is a bounded source, false if this is an unbounded source. Default is unbounded for compatibility.

Specified by:

isBounded in interface StreamTableSource<RowData>

getPartitions

public List<Map<String,String>> getPartitions()

Description copied from interface: PartitionableTableSource

Returns all the partitions of this PartitionableTableSource.

Specified by:

getPartitions in interface PartitionableTableSource

applyPartitionPruning

public FileSystemTableSource applyPartitionPruning(List<Map<String,String>> remainingPartitions)

Description copied from interface: PartitionableTableSource

Applies the remaining partitions to the table source. The remainingPartitions is the remaining partitions of PartitionableTableSource.getPartitions() after partition pruning applied.

After trying to apply partition pruning, we should return a new TableSource instance which holds all pruned-partitions.

Specified by:

applyPartitionPruning in interface PartitionableTableSource

Parameters:

remainingPartitions - Remaining partitions after partition pruning applied.

Returns:

A new cloned instance of TableSource holds all pruned-partitions.

projectFields

public FileSystemTableSource projectFields(int[] fields)

Description copied from interface: ProjectableTableSource

Creates a copy of the TableSource that projects its output to the given field indexes. The field indexes relate to the physical poduced data type (TableSource.getProducedDataType()) and not to the table schema (TableSource.getTableSchema() of the TableSource.

The table schema (TableSource.getTableSchema() of the TableSource copy must not be modified by this method, but only the produced data type (TableSource.getProducedDataType()) and the produced DataSet (BatchTableSource#getDataSet() or DataStream (StreamTableSource#getDataStream).

If the TableSource implements the DefinedFieldMapping interface, it might be necessary to adjust the mapping as well.

IMPORTANT: This method must return a true copy and must not modify the original table source object.

Specified by:

projectFields in interface ProjectableTableSource<RowData>

Parameters:

fields - The indexes of the fields to return.

Returns:

A copy of the TableSource that projects its output.

applyLimit

public FileSystemTableSource applyLimit(long limit)

Description copied from interface: LimitableTableSource

Check and push down the limit to the table source.

Specified by:

applyLimit in interface LimitableTableSource<RowData>

Parameters:

limit - the value which limit the number of records.

Returns:

A new cloned instance of TableSource.

isLimitPushedDown

public boolean isLimitPushedDown()

Description copied from interface: LimitableTableSource

Return the flag to indicate whether limit push down has been tried. Must return true on the returned instance of LimitableTableSource.applyLimit(long).

Specified by:

isLimitPushedDown in interface LimitableTableSource<RowData>

applyPredicate

public FileSystemTableSource applyPredicate(List<Expression> predicates)

Description copied from interface: FilterableTableSource

Check and pick all predicates this table source can support. The passed in predicates have been translated in conjunctive form, and table source can only pick those predicates that it supports.

WARNING: Flink planner will push down PlannerExpressions (which are defined in flink-table-planner module), while Blink planner will push down Expressions. So the implementation for Flink planner and Blink planner should be different and incompatible. PlannerExpression will be removed in the future.

After trying to push predicates down, we should return a new TableSource instance which holds all pushed down predicates. Even if we actually pushed nothing down, it is recommended that we still return a new TableSource instance since we will mark the returned instance as filter push down has been tried.

We also should note to not changing the form of the predicates passed in. It has been organized in CNF conjunctive form, and we should only take or leave each element from the list. Don't try to reorganize the predicates if you are absolutely confident with that.

Specified by:

applyPredicate in interface FilterableTableSource<RowData>

Parameters:

predicates - A list contains conjunctive predicates, you should pick and remove all expressions that can be pushed down. The remaining elements of this list will further evaluated by framework.

Returns:

A new cloned instance of TableSource with or without any filters been pushed into it.

isFilterPushedDown

public boolean isFilterPushedDown()

Description copied from interface: FilterableTableSource

Return the flag to indicate whether filter push down has been tried. Must return true on the returned instance of FilterableTableSource.applyPredicate(java.util.List<org.apache.flink.table.expressions.Expression>).

Specified by:

isFilterPushedDown in interface FilterableTableSource<RowData>

getProducedDataType

public DataType getProducedDataType()

Description copied from interface: TableSource

Returns the DataType for the produced data of the TableSource.

Specified by:

getProducedDataType in interface TableSource<RowData>

Returns:

The data type of the returned DataSet or DataStream.

getTableSchema

public TableSchema getTableSchema()

Description copied from interface: TableSource

Returns the schema of the produced table.

Specified by:

getTableSchema in interface TableSource<RowData>

Returns:

The TableSchema of the produced table.

explainSource

public String explainSource()

Description copied from interface: TableSource

Describes the table source.

Specified by:

explainSource in interface TableSource<RowData>

Returns:

A String explaining the TableSource.