org.apache.flink.orc

Class OrcTableSource

java.lang.Object

org.apache.flink.orc.OrcTableSource

All Implemented Interfaces:

BatchTableSource<Row>, FilterableTableSource<Row>, ProjectableTableSource<Row>, TableSource<Row>

public class OrcTableSource
extends Object
implements BatchTableSource<Row>, ProjectableTableSource<Row>, FilterableTableSource<Row>

A TableSource to read ORC files.

The OrcTableSource supports projection and filter push-down.

An OrcTableSource is used as shown in the example below.

 
 OrcTableSource orcSrc = OrcTableSource.builder()
   .path("file:///my/data/file.orc")
   .forOrcSchema("struct<col1:boolean,col2:tinyint,col3:smallint,col4:int>")
   .build();

 tEnv.registerTableSource("orcTable", orcSrc);
 Table res = tableEnv.sqlQuery("SELECT * FROM orcTable");
 
 

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	OrcTableSource.Builder Constructs an OrcTableSource.

Method Summary

Modifier and Type	Method and Description
TableSource<Row>	applyPredicate(List<Expression> predicates) Check and pick all predicates this table source can support.
static OrcTableSource.Builder	builder()
protected OrcRowInputFormat	buildOrcInputFormat()
String	explainSource() Describes the table source.
DataSet<Row>	getDataSet(ExecutionEnvironment execEnv) Returns the data of the table as a DataSet.
TypeInformation<Row>	getReturnType()
TableSchema	getTableSchema() Returns the schema of the produced table.
boolean	isFilterPushedDown() Return the flag to indicate whether filter push down has been tried.
TableSource<Row>	projectFields(int[] selectedFields) Creates a copy of the TableSource that projects its output to the given field indexes.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.table.sources.TableSource

getProducedDataType

Method Detail

getDataSet

public DataSet<Row> getDataSet(ExecutionEnvironment execEnv)

Description copied from interface: BatchTableSource

Returns the data of the table as a DataSet.

NOTE: This method is for internal use only for defining a TableSource. Do not use it in Table API programs.

Specified by:

getDataSet in interface BatchTableSource<Row>

buildOrcInputFormat

@VisibleForTesting
protected OrcRowInputFormat buildOrcInputFormat()

getReturnType

public TypeInformation<Row> getReturnType()

Specified by:

getReturnType in interface TableSource<Row>

getTableSchema

public TableSchema getTableSchema()

Description copied from interface: TableSource

Returns the schema of the produced table.

Specified by:

getTableSchema in interface TableSource<Row>

Returns:

The TableSchema of the produced table.

projectFields

public TableSource<Row> projectFields(int[] selectedFields)

Description copied from interface: ProjectableTableSource

Creates a copy of the TableSource that projects its output to the given field indexes. The field indexes relate to the physical poduced data type (TableSource.getProducedDataType()) and not to the table schema (TableSource.getTableSchema() of the TableSource.

The table schema (TableSource.getTableSchema() of the TableSource copy must not be modified by this method, but only the produced data type (TableSource.getProducedDataType()) and the produced DataSet (BatchTableSource#getDataSet() or DataStream (StreamTableSource#getDataStream).

If the TableSource implements the DefinedFieldMapping interface, it might be necessary to adjust the mapping as well.

IMPORTANT: This method must return a true copy and must not modify the original table source object.

Specified by:

projectFields in interface ProjectableTableSource<Row>

Parameters:

selectedFields - The indexes of the fields to return.

Returns:

A copy of the TableSource that projects its output.

applyPredicate

public TableSource<Row> applyPredicate(List<Expression> predicates)

Description copied from interface: FilterableTableSource

Check and pick all predicates this table source can support. The passed in predicates have been translated in conjunctive form, and table source can only pick those predicates that it supports.

WARNING: Flink planner will push down PlannerExpressions (which are defined in flink-table-planner module), while Blink planner will push down Expressions. So the implementation for Flink planner and Blink planner should be different and incompatible. PlannerExpression will be removed in the future.

After trying to push predicates down, we should return a new TableSource instance which holds all pushed down predicates. Even if we actually pushed nothing down, it is recommended that we still return a new TableSource instance since we will mark the returned instance as filter push down has been tried.

We also should note to not changing the form of the predicates passed in. It has been organized in CNF conjunctive form, and we should only take or leave each element from the list. Don't try to reorganize the predicates if you are absolutely confident with that.

Specified by:

applyPredicate in interface FilterableTableSource<Row>

Parameters:

predicates - A list contains conjunctive predicates, you should pick and remove all expressions that can be pushed down. The remaining elements of this list will further evaluated by framework.

Returns:

A new cloned instance of TableSource with or without any filters been pushed into it.

isFilterPushedDown

public boolean isFilterPushedDown()

Description copied from interface: FilterableTableSource

Return the flag to indicate whether filter push down has been tried. Must return true on the returned instance of FilterableTableSource.applyPredicate(java.util.List<org.apache.flink.table.expressions.Expression>).

Specified by:

isFilterPushedDown in interface FilterableTableSource<Row>

explainSource

public String explainSource()

Description copied from interface: TableSource

Describes the table source.

Specified by:

explainSource in interface TableSource<Row>

Returns:

A String explaining the TableSource.

builder

public static OrcTableSource.Builder builder()