public class ParquetTableSource extends Object implements BatchTableSource<Row>, FilterableTableSource<Row>, ProjectableTableSource<Row>
ParquetTableSource supports projection and filter push-down.
ParquetTableSource is used as shown in the example below.
ParquetTableSource parquetSrc = ParquetTableSource.builder() .path("file:///my/data/file.parquet") .schema(messageType) .build(); tEnv.registerTableSourceInternal("parquetTable", parquetSrc); Table res = tableEnv.sqlQuery("SELECT * FROM parquetTable");
|Modifier and Type||Class and Description|
|Modifier and Type||Method and Description|
Check and pick all predicates this table source can support.
Describes the table source.
Returns the data of the table as a
Returns the schema of the produced table.
Return the flag to indicate whether filter push down has been tried.
Creates a copy of the
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
public TableSource<Row> projectFields(int fields)
TableSourcethat projects its output to the given field indexes. The field indexes relate to the physical poduced data type (
TableSource.getProducedDataType()) and not to the table schema (
The table schema (
TableSource.getTableSchema() of the
TableSource copy must
not be modified by this method, but only the produced data type (
TableSource.getProducedDataType()) and the produced
TableSource implements the
DefinedFieldMapping interface, it might
be necessary to adjust the mapping as well.
IMPORTANT: This method must return a true copy and must not modify the original table source object.
fields- The indexes of the fields to return.
TableSourcethat projects its output.
public DataSet<Row> getDataSet(ExecutionEnvironment executionEnvironment)
NOTE: This method is for internal use only for defining a
TableSource. Do not use
it in Table API programs.
public TableSource<Row> applyPredicate(List<Expression> predicates)
WARNING: Flink planner will push down PlannerExpressions (which are
defined in flink-table-planner module), while Blink planner will push down
Expressions. So the implementation for Flink planner and Blink planner should be different
and incompatible. PlannerExpression will be removed in the future.
After trying to push predicates down, we should return a new
which holds all pushed down predicates. Even if we actually pushed nothing down, it is
recommended that we still return a new
TableSource instance since we will mark the
returned instance as filter push down has been tried.
We also should note to not changing the form of the predicates passed in. It has been organized in CNF conjunctive form, and we should only take or leave each element from the list. Don't try to reorganize the predicates if you are absolutely confident with that.
predicates- A list contains conjunctive predicates, you should pick and remove all expressions that can be pushed down. The remaining elements of this list will further evaluated by framework.
TableSourcewith or without any filters been pushed into it.
public boolean isFilterPushedDown()
public TypeInformation<Row> getReturnType()
public TableSchema getTableSchema()
TableSchemaof the produced table.
public String explainSource()
public static ParquetTableSource.Builder builder()
Copyright © 2014–2022 The Apache Software Foundation. All rights reserved.