@Internal public final class StreamTableEnvironmentImpl extends TableEnvironmentImpl implements StreamTableEnvironment
StreamTableEnvironment
. This enables conversions from/to
DataStream
.
It binds to a given StreamExecutionEnvironment
.
execEnv, functionCatalog, planner, tableConfig
Constructor and Description |
---|
StreamTableEnvironmentImpl(CatalogManager catalogManager,
FunctionCatalog functionCatalog,
TableConfig tableConfig,
StreamExecutionEnvironment executionEnvironment,
Planner planner,
Executor executor,
boolean isStreamingMode) |
Modifier and Type | Method and Description |
---|---|
StreamTableDescriptor |
connect(ConnectorDescriptor connectorDescriptor)
Creates a table source and/or table sink from a descriptor.
|
static StreamTableEnvironment |
create(StreamExecutionEnvironment executionEnvironment,
EnvironmentSettings settings,
TableConfig tableConfig) |
StreamExecutionEnvironment |
execEnv()
This is a temporary workaround for Python API.
|
String |
explain(boolean extended)
Returns the AST of the specified Table API and SQL queries and the execution plan to compute
the result of multiple-sinks plan.
|
<T> Table |
fromDataStream(DataStream<T> dataStream)
Converts the given
DataStream into a Table . |
<T> Table |
fromDataStream(DataStream<T> dataStream,
String fields)
Converts the given
DataStream into a Table with specified field names. |
void |
insertInto(Table table,
StreamQueryConfig queryConfig,
String sinkPath,
String... sinkPathContinued)
|
protected boolean |
isEagerOperationTranslation()
Defines the behavior of this
TableEnvironment . |
<T> void |
registerDataStream(String name,
DataStream<T> dataStream)
Registers the given
DataStream as table in the TableEnvironment 's catalog. |
<T> void |
registerDataStream(String name,
DataStream<T> dataStream,
String fields)
Registers the given
DataStream as table with specified field names in the
TableEnvironment 's catalog. |
<T,ACC> void |
registerFunction(String name,
AggregateFunction<T,ACC> aggregateFunction)
Registers an
AggregateFunction under a unique name in the TableEnvironment's catalog. |
<T,ACC> void |
registerFunction(String name,
TableAggregateFunction<T,ACC> tableAggregateFunction)
Registers an
TableAggregateFunction under a unique name in the TableEnvironment's
catalog. |
<T> void |
registerFunction(String name,
TableFunction<T> tableFunction)
Registers a
TableFunction under a unique name in the TableEnvironment's catalog. |
void |
sqlUpdate(String stmt,
StreamQueryConfig config)
Evaluates a SQL statement such as INSERT, UPDATE or DELETE; or a DDL statement;
NOTE: Currently only SQL INSERT statements are supported.
|
<T> DataStream<T> |
toAppendStream(Table table,
Class<T> clazz)
Converts the given
Table into an append DataStream of a specified type. |
<T> DataStream<T> |
toAppendStream(Table table,
Class<T> clazz,
StreamQueryConfig queryConfig)
Converts the given
Table into an append DataStream of a specified type. |
<T> DataStream<T> |
toAppendStream(Table table,
TypeInformation<T> typeInfo)
Converts the given
Table into an append DataStream of a specified type. |
<T> DataStream<T> |
toAppendStream(Table table,
TypeInformation<T> typeInfo,
StreamQueryConfig queryConfig)
Converts the given
Table into an append DataStream of a specified type. |
<T> DataStream<Tuple2<Boolean,T>> |
toRetractStream(Table table,
Class<T> clazz)
Converts the given
Table into a DataStream of add and retract messages. |
<T> DataStream<Tuple2<Boolean,T>> |
toRetractStream(Table table,
Class<T> clazz,
StreamQueryConfig queryConfig)
Converts the given
Table into a DataStream of add and retract messages. |
<T> DataStream<Tuple2<Boolean,T>> |
toRetractStream(Table table,
TypeInformation<T> typeInfo)
Converts the given
Table into a DataStream of add and retract messages. |
<T> DataStream<Tuple2<Boolean,T>> |
toRetractStream(Table table,
TypeInformation<T> typeInfo,
StreamQueryConfig queryConfig)
Converts the given
Table into a DataStream of add and retract messages. |
protected void |
validateTableSource(TableSource<?> tableSource)
Subclasses can override this method to add additional checks.
|
create, createTable, execute, explain, explain, fromTableSource, getCatalog, getCompletionHints, getConfig, getCurrentCatalog, getCurrentDatabase, getPlanner, getRegisteredExternalCatalog, insertInto, listCatalogs, listDatabases, listTables, listUserDefinedFunctions, registerCatalog, registerExternalCatalog, registerFunction, registerTable, registerTableInternal, registerTableSink, registerTableSink, registerTableSource, scan, sqlQuery, sqlUpdate, useCatalog, useDatabase
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
create, create, create, execute
create, explain, explain, fromTableSource, getCatalog, getCompletionHints, getConfig, getCurrentCatalog, getCurrentDatabase, getRegisteredExternalCatalog, insertInto, listCatalogs, listDatabases, listTables, listUserDefinedFunctions, registerCatalog, registerExternalCatalog, registerFunction, registerTable, registerTableSink, registerTableSink, registerTableSource, scan, sqlQuery, sqlUpdate, useCatalog, useDatabase
public StreamTableEnvironmentImpl(CatalogManager catalogManager, FunctionCatalog functionCatalog, TableConfig tableConfig, StreamExecutionEnvironment executionEnvironment, Planner planner, Executor executor, boolean isStreamingMode)
public static StreamTableEnvironment create(StreamExecutionEnvironment executionEnvironment, EnvironmentSettings settings, TableConfig tableConfig)
public <T> void registerFunction(String name, TableFunction<T> tableFunction)
StreamTableEnvironment
TableFunction
under a unique name in the TableEnvironment's catalog.
Registered functions can be referenced in Table API and SQL queries.registerFunction
in interface StreamTableEnvironment
T
- The type of the output row.name
- The name under which the function is registered.tableFunction
- The TableFunction to register.public <T,ACC> void registerFunction(String name, AggregateFunction<T,ACC> aggregateFunction)
StreamTableEnvironment
AggregateFunction
under a unique name in the TableEnvironment's catalog.
Registered functions can be referenced in Table API and SQL queries.registerFunction
in interface StreamTableEnvironment
T
- The type of the output value.name
- The name under which the function is registered.aggregateFunction
- The AggregateFunction to register.public <T,ACC> void registerFunction(String name, TableAggregateFunction<T,ACC> tableAggregateFunction)
StreamTableEnvironment
TableAggregateFunction
under a unique name in the TableEnvironment's
catalog. Registered functions can only be referenced in Table API.registerFunction
in interface StreamTableEnvironment
T
- The type of the output value.name
- The name under which the function is registered.tableAggregateFunction
- The TableAggregateFunction to register.public <T> Table fromDataStream(DataStream<T> dataStream)
StreamTableEnvironment
DataStream
into a Table
.
The field names of the Table
are automatically derived from the type of the
DataStream
.fromDataStream
in interface StreamTableEnvironment
T
- The type of the DataStream
.dataStream
- The DataStream
to be converted.Table
.public <T> Table fromDataStream(DataStream<T> dataStream, String fields)
StreamTableEnvironment
DataStream
into a Table
with specified field names.
Example:
DataStream<Tuple2<String, Long>> stream = ...
Table tab = tableEnv.fromDataStream(stream, "a, b");
fromDataStream
in interface StreamTableEnvironment
T
- The type of the DataStream
.dataStream
- The DataStream
to be converted.fields
- The field names of the resulting Table
.Table
.public <T> void registerDataStream(String name, DataStream<T> dataStream)
StreamTableEnvironment
DataStream
as table in the TableEnvironment
's catalog.
Registered tables can be referenced in SQL queries.
The field names of the Table
are automatically derived
from the type of the DataStream
.registerDataStream
in interface StreamTableEnvironment
T
- The type of the DataStream
to register.name
- The name under which the DataStream
is registered in the catalog.dataStream
- The DataStream
to register.public <T> void registerDataStream(String name, DataStream<T> dataStream, String fields)
StreamTableEnvironment
DataStream
as table with specified field names in the
TableEnvironment
's catalog.
Registered tables can be referenced in SQL queries.
Example:
DataStream<Tuple2<String, Long>> set = ...
tableEnv.registerDataStream("myTable", set, "a, b")
registerDataStream
in interface StreamTableEnvironment
T
- The type of the DataStream
to register.name
- The name under which the DataStream
is registered in the catalog.dataStream
- The DataStream
to register.fields
- The field names of the registered table.public <T> DataStream<T> toAppendStream(Table table, Class<T> clazz)
StreamTableEnvironment
Table
into an append DataStream
of a specified type.
The Table
must only have insert (append) changes. If the Table
is also modified
by update or delete changes, the conversion will fail.
The fields of the Table
are mapped to DataStream
fields as follows:
Row
and Tuple
types: Fields are mapped by position, field types must match.DataStream
types: Fields are mapped by field name, field types must match.toAppendStream
in interface StreamTableEnvironment
T
- The type of the resulting DataStream
.table
- The Table
to convert.clazz
- The class of the type of the resulting DataStream
.DataStream
.public <T> DataStream<T> toAppendStream(Table table, TypeInformation<T> typeInfo)
StreamTableEnvironment
Table
into an append DataStream
of a specified type.
The Table
must only have insert (append) changes. If the Table
is also modified
by update or delete changes, the conversion will fail.
The fields of the Table
are mapped to DataStream
fields as follows:
Row
and Tuple
types: Fields are mapped by position, field types must match.DataStream
types: Fields are mapped by field name, field types must match.toAppendStream
in interface StreamTableEnvironment
T
- The type of the resulting DataStream
.table
- The Table
to convert.typeInfo
- The TypeInformation
that specifies the type of the DataStream
.DataStream
.public <T> DataStream<T> toAppendStream(Table table, Class<T> clazz, StreamQueryConfig queryConfig)
StreamTableEnvironment
Table
into an append DataStream
of a specified type.
The Table
must only have insert (append) changes. If the Table
is also modified
by update or delete changes, the conversion will fail.
The fields of the Table
are mapped to DataStream
fields as follows:
Row
and Tuple
types: Fields are mapped by position, field types must match.DataStream
types: Fields are mapped by field name, field types must match.toAppendStream
in interface StreamTableEnvironment
T
- The type of the resulting DataStream
.table
- The Table
to convert.clazz
- The class of the type of the resulting DataStream
.queryConfig
- The configuration of the query to generate.DataStream
.public <T> DataStream<T> toAppendStream(Table table, TypeInformation<T> typeInfo, StreamQueryConfig queryConfig)
StreamTableEnvironment
Table
into an append DataStream
of a specified type.
The Table
must only have insert (append) changes. If the Table
is also modified
by update or delete changes, the conversion will fail.
The fields of the Table
are mapped to DataStream
fields as follows:
Row
and Tuple
types: Fields are mapped by position, field types must match.DataStream
types: Fields are mapped by field name, field types must match.toAppendStream
in interface StreamTableEnvironment
T
- The type of the resulting DataStream
.table
- The Table
to convert.typeInfo
- The TypeInformation
that specifies the type of the DataStream
.queryConfig
- The configuration of the query to generate.DataStream
.public <T> DataStream<Tuple2<Boolean,T>> toRetractStream(Table table, Class<T> clazz)
StreamTableEnvironment
Table
into a DataStream
of add and retract messages.
The message will be encoded as Tuple2
. The first field is a Boolean
flag,
the second field holds the record of the specified type T
.
A true Boolean
flag indicates an add message, a false flag indicates a retract message.
The fields of the Table
are mapped to DataStream
fields as follows:
Row
and Tuple
types: Fields are mapped by position, field types must match.DataStream
types: Fields are mapped by field name, field types must match.toRetractStream
in interface StreamTableEnvironment
T
- The type of the requested record type.table
- The Table
to convert.clazz
- The class of the requested record type.DataStream
.public <T> DataStream<Tuple2<Boolean,T>> toRetractStream(Table table, TypeInformation<T> typeInfo)
StreamTableEnvironment
Table
into a DataStream
of add and retract messages.
The message will be encoded as Tuple2
. The first field is a Boolean
flag,
the second field holds the record of the specified type T
.
A true Boolean
flag indicates an add message, a false flag indicates a retract message.
The fields of the Table
are mapped to DataStream
fields as follows:
Row
and Tuple
types: Fields are mapped by position, field types must match.DataStream
types: Fields are mapped by field name, field types must match.toRetractStream
in interface StreamTableEnvironment
T
- The type of the requested record type.table
- The Table
to convert.typeInfo
- The TypeInformation
of the requested record type.DataStream
.public <T> DataStream<Tuple2<Boolean,T>> toRetractStream(Table table, Class<T> clazz, StreamQueryConfig queryConfig)
StreamTableEnvironment
Table
into a DataStream
of add and retract messages.
The message will be encoded as Tuple2
. The first field is a Boolean
flag,
the second field holds the record of the specified type T
.
A true Boolean
flag indicates an add message, a false flag indicates a retract message.
The fields of the Table
are mapped to DataStream
fields as follows:
Row
and Tuple
types: Fields are mapped by position, field types must match.DataStream
types: Fields are mapped by field name, field types must match.toRetractStream
in interface StreamTableEnvironment
T
- The type of the requested record type.table
- The Table
to convert.clazz
- The class of the requested record type.queryConfig
- The configuration of the query to generate.DataStream
.public <T> DataStream<Tuple2<Boolean,T>> toRetractStream(Table table, TypeInformation<T> typeInfo, StreamQueryConfig queryConfig)
StreamTableEnvironment
Table
into a DataStream
of add and retract messages.
The message will be encoded as Tuple2
. The first field is a Boolean
flag,
the second field holds the record of the specified type T
.
A true Boolean
flag indicates an add message, a false flag indicates a retract message.
The fields of the Table
are mapped to DataStream
fields as follows:
Row
and Tuple
types: Fields are mapped by position, field types must match.DataStream
types: Fields are mapped by field name, field types must match.toRetractStream
in interface StreamTableEnvironment
T
- The type of the requested record type.table
- The Table
to convert.typeInfo
- The TypeInformation
of the requested record type.queryConfig
- The configuration of the query to generate.DataStream
.public StreamTableDescriptor connect(ConnectorDescriptor connectorDescriptor)
TableEnvironment
Descriptors allow for declaring the communication to external systems in an implementation-agnostic way. The classpath is scanned for suitable table factories that match the desired configuration.
The following example shows how to read from a connector using a JSON format and register a table source as "MyTable":
tableEnv
.connect(
new ExternalSystemXYZ()
.version("0.11"))
.withFormat(
new Json()
.jsonSchema("{...}")
.failOnMissingField(false))
.withSchema(
new Schema()
.field("user-name", "VARCHAR").from("u_name")
.field("count", "DECIMAL")
.registerSource("MyTable");
connect
in interface StreamTableEnvironment
connect
in interface TableEnvironment
connect
in class TableEnvironmentImpl
connectorDescriptor
- connector descriptor describing the external systempublic void sqlUpdate(String stmt, StreamQueryConfig config)
StreamTableEnvironment
All tables referenced by the query must be registered in the TableEnvironment.
A Table
is automatically registered when its Table#toString()
method is
called, for example when it is embedded into a String.
Hence, SQL queries can directly reference a Table
as follows:
// register the configured table sink into which the result is inserted.
tEnv.registerTableSink("sinkTable", configuredSink);
Table sourceTable = ...
String tableName = sourceTable.toString();
// sourceTable is not registered to the table environment
tEnv.sqlUpdate(s"INSERT INTO sinkTable SELECT * FROM tableName", config);
sqlUpdate
in interface StreamTableEnvironment
stmt
- The SQL statement to evaluate.config
- The QueryConfig
to use.public void insertInto(Table table, StreamQueryConfig queryConfig, String sinkPath, String... sinkPathContinued)
StreamTableEnvironment
Table
to a TableSink
that was registered under the specified name.
See the documentation of TableEnvironment.useDatabase(String)
or
TableEnvironment.useCatalog(String)
for the rules on the path resolution.
insertInto
in interface StreamTableEnvironment
table
- The Table to write to the sink.queryConfig
- The StreamQueryConfig
to use.sinkPath
- The first part of the path of the registered TableSink
to which the Table
is
written. This is to ensure at least the name of the TableSink
is provided.sinkPathContinued
- The remaining part of the path of the registered TableSink
to which the
Table
is written.@Internal public StreamExecutionEnvironment execEnv()
protected void validateTableSource(TableSource<?> tableSource)
TableEnvironmentImpl
validateTableSource
in class TableEnvironmentImpl
tableSource
- tableSource to validateprotected boolean isEagerOperationTranslation()
TableEnvironmentImpl
TableEnvironment
. If true the queries will
be translated immediately. If false the ModifyOperation
s will be buffered
and translated only when TableEnvironmentImpl.execute(String)
is called.
If the TableEnvironment
works in a lazy manner it is undefined what
configurations values will be used. It depends on the characteristic of the particular
parameter. Some might used values current to the time of query construction (e.g. the currentCatalog)
and some use values from the time when TableEnvironmentImpl.execute(String)
is called (e.g. timeZone).
isEagerOperationTranslation
in class TableEnvironmentImpl
public String explain(boolean extended)
TableEnvironment
explain
in interface TableEnvironment
explain
in class TableEnvironmentImpl
extended
- if the plan should contain additional properties such as
e.g. estimated cost, traitsCopyright © 2014–2020 The Apache Software Foundation. All rights reserved.