Package org.apache.flink.table.api

Interface Table

  • All Superinterfaces:
    Executable, Explainable<Table>
    All Known Implementing Classes:
    TableImpl
    @PublicEvolving
public interface Table
extends Explainable<Table>, Executable
    The Table object is the core abstraction of the Table API. Similar to how the DataStream API has DataStreams, the Table API is built around Tables.

    A Table object describes a pipeline of data transformations. It does not contain the data itself in any way. Instead, it describes how to read data from a DynamicTableSource and how to eventually write data to a DynamicTableSink. The declared pipeline can be printed, optimized, and eventually executed in a cluster. The pipeline can work with bounded or unbounded streams which enables both streaming and batch scenarios.

    By the definition above, a Table object can actually be considered as a view in SQL terms.

    The initial Table object is constructed by a TableEnvironment. For example, TableEnvironment.from(String)) obtains a table from a catalog. Every Table object has a schema that is available through getResolvedSchema(). A Table object is always associated with its original table environment during programming.

    Every transformation (i.e. select(Expression...) or filter(Expression)) on a Table object leads to a new Table object.

    Use Executable.execute() to execute the pipeline and retrieve the transformed data locally during development. Otherwise, use executeInsert(String) to write the data into a table sink.

    Many methods of this class take one or more Expressions as parameters. For fluent definition of expressions and easier readability, we recommend to add a star import: 

      import static org.apache.flink.table.api.Expressions.*;

    Check the documentation for more programming language specific APIs, for example, by using Scala implicits.

    The following example shows how to work with a Table object.

    Java Example (with static import for expressions): 

    
 TableEnvironment tableEnv = TableEnvironment.create(...);

 Table table = tableEnv.from("MyTable").select($("colA").trim(), $("colB").plus(12));

 table.execute().print();

    Scala Example (with implicits for expressions): 

    
 val tableEnv = TableEnvironment.create(...)

 val table = tableEnv.from("MyTable").select($"colA".trim(), $"colB" + 12)

 table.execute().print()

    • Method Detail

      • getResolvedSchema

        ResolvedSchema getResolvedSchema()
        Returns the resolved schema of this table.

      • printSchema

        void printSchema()
        Prints the schema of this table to the console in a summary format.

      • getQueryOperation

        QueryOperation getQueryOperation()
        Returns underlying logical representation of this table.

      • select

        Table select​(Expression... fields)
        Performs a selection operation. Similar to a SQL SELECT statement. The field expressions can contain complex expressions and aggregations.

        Java Example: 

        
 tab.select($("key"), $("value").avg().plus(" The average").as("average"));

        Scala Example: 

        
 tab.select($"key", $"value".avg + " The average" as "average")

      • createTemporalTableFunction

        TemporalTableFunction createTemporalTableFunction​(Expression timeAttribute,
                                                  Expression primaryKey)
        Creates TemporalTableFunction backed up by this table as a history table. Temporal Tables represent a concept of a table that changes over time and for which Flink keeps track of those changes. TemporalTableFunction provides a way how to access those data.

        For more information please check Flink's documentation on Temporal Tables.

        Currently TemporalTableFunctions are only supported in streaming.

        Parameters:
        timeAttribute - Must points to a time indicator. Provides a way to compare which records are a newer or older version.
        primaryKey - Defines the primary key. With primary key it is possible to update a row or to delete it.
        Returns:
        TemporalTableFunction which is an instance of TableFunction. It takes one single argument, the timeAttribute, for which it returns matching version of the Table, from which TemporalTableFunction was created.

      • as

        Table as​(String field,
         String... fields)
        Renames the fields of the expression result. Use this to disambiguate fields before joining to operations.

        Example: 

        
 tab.as("a", "b")

      • as

        @Deprecated
Table as​(Expression... fields)
        Deprecated.
        use as(String, String...)
        Renames the fields of the expression result. Use this to disambiguate fields before joining to operations.

        Java Example: 

        
 tab.as($("a"), $("b"))

        Scala Example: 

        
 tab.as($"a", $"b")

      • filter

        Table filter​(Expression predicate)
        Filters out elements that don't pass the filter predicate. Similar to a SQL WHERE clause.

        Java Example: 

        
 tab.filter($("name").isEqual("Fred"));

        Scala Example: 

        
 tab.filter($"name" === "Fred")

      • where

        Table where​(Expression predicate)
        Filters out elements that don't pass the filter predicate. Similar to a SQL WHERE clause.

        Java Example: 

        
 tab.where($("name").isEqual("Fred"));

        Scala Example: 

        
 tab.where($"name" === "Fred")

      • groupBy

        GroupedTable groupBy​(Expression... fields)
        Groups the elements on some grouping keys. Use this before a selection with aggregations to perform the aggregation on a per-group basis. Similar to a SQL GROUP BY statement.

        Java Example: 

        
 tab.groupBy($("key")).select($("key"), $("value").avg());

        Scala Example: 

        
 tab.groupBy($"key").select($"key", $"value".avg)

      • distinct

        Table distinct()
        Removes duplicate values and returns only distinct (different) values.

        Example: 

        
 tab.select($("key"), $("value")).distinct();

      • join

        Table join​(Table right)
        Joins two Tables. Similar to a SQL join. The fields of the two joined operations must not overlap, use as to rename fields if necessary. You can use where and select clauses after a join to further specify the behaviour of the join.

        Note: Both tables must be bound to the same TableEnvironment .

        Example: 

        
 left.join(right)
     .where($("a").isEqual($("b")).and($("c").isGreater(3))
     .select($("a"), $("b"), $("d"));

      • join

        Table join​(Table right,
           Expression joinPredicate)
        Joins two Tables. Similar to a SQL join. The fields of the two joined operations must not overlap, use as to rename fields if necessary.

        Note: Both tables must be bound to the same TableEnvironment .

        Java Example: 

        
 left.join(right, $("a").isEqual($("b")))
     .select($("a"), $("b"), $("d"));

        Scala Example: 

        
 left.join(right, $"a" === $"b")
     .select($"a", $"b", $"d")

      • leftOuterJoin

        Table leftOuterJoin​(Table right)
        Joins two Tables. Similar to a SQL left outer join. The fields of the two joined operations must not overlap, use as to rename fields if necessary.

        Note: Both tables must be bound to the same TableEnvironment and its TableConfig must have null check enabled (default).

        Example: 

        
 left.leftOuterJoin(right)
     .select($("a"), $("b"), $("d"));

      • leftOuterJoin

        Table leftOuterJoin​(Table right,
                    Expression joinPredicate)
        Joins two Tables. Similar to a SQL left outer join. The fields of the two joined operations must not overlap, use as to rename fields if necessary.

        Note: Both tables must be bound to the same TableEnvironment and its TableConfig must have null check enabled (default).

        Java Example: 

        
 left.leftOuterJoin(right, $("a").isEqual($("b")))
     .select($("a"), $("b"), $("d"));

        Scala Example: 

        
 left.leftOuterJoin(right, $"a" === $"b")
     .select($"a", $"b", $"d")

      • rightOuterJoin

        Table rightOuterJoin​(Table right,
                     Expression joinPredicate)
        Joins two Tables. Similar to a SQL right outer join. The fields of the two joined operations must not overlap, use as to rename fields if necessary.

        Note: Both tables must be bound to the same TableEnvironment and its TableConfig must have null check enabled (default).

        Java Example: 

        
 left.rightOuterJoin(right, $("a").isEqual($("b")))
     .select($("a"), $("b"), $("d"));

        Scala Example: 

        
 left.rightOuterJoin(right, $"a" === $"b")
     .select($"a", $"b", $"d")

      • fullOuterJoin

        Table fullOuterJoin​(Table right,
                    Expression joinPredicate)
        Joins two Tables. Similar to a SQL full outer join. The fields of the two joined operations must not overlap, use as to rename fields if necessary.

        Note: Both tables must be bound to the same TableEnvironment and its TableConfig must have null check enabled (default).

        Java Example: 

        
 left.fullOuterJoin(right, $("a").isEqual($("b")))
     .select($("a"), $("b"), $("d"));

        Scala Example: 

        
 left.fullOuterJoin(right, $"a" === $"b")
     .select($"a", $"b", $"d")

      • joinLateral

        Table joinLateral​(Expression tableFunctionCall)
        Joins this Table with an user-defined TableFunction. This join is similar to a SQL inner join with ON TRUE predicate but works with a table function. Each row of the table is joined with all rows produced by the table function.

        Java Example: 

        
 class MySplitUDTF extends TableFunction<String> {
   public void eval(String str) {
     str.split("#").forEach(this::collect);
   }
 }

 table.joinLateral(call(MySplitUDTF.class, $("c")).as("s"))
      .select($("a"), $("b"), $("c"), $("s"));

        Scala Example: 

        
 class MySplitUDTF extends TableFunction[String] {
   def eval(str: String): Unit = {
     str.split("#").foreach(collect)
   }
 }

 val split = new MySplitUDTF()
 table.joinLateral(split($"c") as "s")
      .select($"a", $"b", $"c", $"s")

      • joinLateral

        Table joinLateral​(Expression tableFunctionCall,
                  Expression joinPredicate)
        Joins this Table with an user-defined TableFunction. This join is similar to a SQL inner join but works with a table function. Each row of the table is joined with all rows produced by the table function.

        Java Example: 

        
 class MySplitUDTF extends TableFunction<String> {
   public void eval(String str) {
     str.split("#").forEach(this::collect);
   }
 }

 table.joinLateral(call(MySplitUDTF.class, $("c")).as("s"), $("a").isEqual($("s")))
      .select($("a"), $("b"), $("c"), $("s"));

        Scala Example: 

        
 class MySplitUDTF extends TableFunction[String] {
   def eval(str: String): Unit = {
     str.split("#").foreach(collect)
   }
 }

 val split = new MySplitUDTF()
 table.joinLateral(split($"c") as "s", $"a" === $"s")
      .select($"a", $"b", $"c", $"s")

      • leftOuterJoinLateral

        Table leftOuterJoinLateral​(Expression tableFunctionCall)
        Joins this Table with an user-defined TableFunction. This join is similar to a SQL left outer join with ON TRUE predicate but works with a table function. Each row of the table is joined with all rows produced by the table function. If the table function does not produce any row, the outer row is padded with nulls.

        Java Example: 

        
 class MySplitUDTF extends TableFunction<String> {
   public void eval(String str) {
     str.split("#").forEach(this::collect);
   }
 }

 table.leftOuterJoinLateral(call(MySplitUDTF.class, $("c")).as("s"))
      .select($("a"), $("b"), $("c"), $("s"));

        Scala Example: 

        
 class MySplitUDTF extends TableFunction[String] {
   def eval(str: String): Unit = {
     str.split("#").foreach(collect)
   }
 }

 val split = new MySplitUDTF()
 table.leftOuterJoinLateral(split($"c") as "s")
      .select($"a", $"b", $"c", $"s")

      • leftOuterJoinLateral

        Table leftOuterJoinLateral​(Expression tableFunctionCall,
                           Expression joinPredicate)
        Joins this Table with an user-defined TableFunction. This join is similar to a SQL left outer join with ON TRUE predicate but works with a table function. Each row of the table is joined with all rows produced by the table function. If the table function does not produce any row, the outer row is padded with nulls.

        Java Example: 

        
 class MySplitUDTF extends TableFunction<String> {
   public void eval(String str) {
     str.split("#").forEach(this::collect);
   }
 }

 table.leftOuterJoinLateral(call(MySplitUDTF.class, $("c")).as("s"), $("a").isEqual($("s")))
      .select($("a"), $("b"), $("c"), $("s"));

        Scala Example: 

        
 class MySplitUDTF extends TableFunction[String] {
   def eval(str: String): Unit = {
     str.split("#").foreach(collect)
   }
 }

 val split = new MySplitUDTF()
 table.leftOuterJoinLateral(split($"c") as "s", $"a" === $"s")
      .select($"a", $"b", $"c", $"s")

      • minus

        Table minus​(Table right)
        Minus of two Tables with duplicate records removed. Similar to a SQL EXCEPT clause. Minus returns records from the left table that do not exist in the right table. Duplicate records in the left table are returned exactly once, i.e., duplicates are removed. Both tables must have identical field types.

        Note: Both tables must be bound to the same TableEnvironment.

        Example: 

        
 left.minus(right);

      • minusAll

        Table minusAll​(Table right)
        Minus of two Tables. Similar to a SQL EXCEPT ALL. Similar to a SQL EXCEPT ALL clause. MinusAll returns the records that do not exist in the right table. A record that is present n times in the left table and m times in the right table is returned (n - m) times, i.e., as many duplicates as are present in the right table are removed. Both tables must have identical field types.

        Note: Both tables must be bound to the same TableEnvironment.

        Example: 

        
 left.minusAll(right);

      • union

        Table union​(Table right)
        Unions two Tables with duplicate records removed. Similar to a SQL UNION. The fields of the two union operations must fully overlap.

        Note: Both tables must be bound to the same TableEnvironment.

        Example: 

        
 left.union(right);

      • unionAll

        Table unionAll​(Table right)
        Unions two Tables. Similar to a SQL UNION ALL. The fields of the two union operations must fully overlap.

        Note: Both tables must be bound to the same TableEnvironment.

        Example: 

        
 left.unionAll(right);

      • intersect

        Table intersect​(Table right)
        Intersects two Tables with duplicate records removed. Intersect returns records that exist in both tables. If a record is present in one or both tables more than once, it is returned just once, i.e., the resulting table has no duplicate records. Similar to a SQL INTERSECT. The fields of the two intersect operations must fully overlap.

        Note: Both tables must be bound to the same TableEnvironment.

        Example: 

        
 left.intersect(right);

      • intersectAll

        Table intersectAll​(Table right)
        Intersects two Tables. IntersectAll returns records that exist in both tables. If a record is present in both tables more than once, it is returned as many times as it is present in both tables, i.e., the resulting table might have duplicate records. Similar to an SQL INTERSECT ALL. The fields of the two intersect operations must fully overlap.

        Note: Both tables must be bound to the same TableEnvironment.

        Example: 

        
 left.intersectAll(right);

      • orderBy

        Table orderBy​(Expression... fields)
        Sorts the given Table. Similar to SQL ORDER BY.

        The resulting Table is globally sorted across all parallel partitions.

        Java Example: 

        
 tab.orderBy($("name").desc());

        Scala Example: 

        
 tab.orderBy($"name".desc)

        For unbounded tables, this operation requires a sorting on a time attribute or a subsequent fetch operation.

      • offset

        Table offset​(int offset)
        Limits a (possibly sorted) result from an offset position.

        This method can be combined with a preceding orderBy(Expression...) call for a deterministic order and a subsequent fetch(int) call to return n rows after skipping the first o rows. 

        
 // skips the first 3 rows and returns all following rows.
 tab.orderBy($("name").desc()).offset(3);
 // skips the first 10 rows and returns the next 5 rows.
 tab.orderBy($("name").desc()).offset(10).fetch(5);

        For unbounded tables, this operation requires a subsequent fetch operation.

        Parameters:
        offset - number of records to skip

      • fetch

        Table fetch​(int fetch)
        Limits a (possibly sorted) result to the first n rows.

        This method can be combined with a preceding orderBy(Expression...) call for a deterministic order and offset(int) call to return n rows after skipping the first o rows. 

        
 // returns the first 3 records.
 tab.orderBy($("name").desc()).fetch(3);
 // skips the first 10 rows and returns the next 5 rows.
 tab.orderBy($("name").desc()).offset(10).fetch(5);
        Parameters:
        fetch - the number of records to return. Fetch must be >= 0.

      • limit

        default Table limit​(int fetch)
        Limits a (possibly sorted) result to the first n rows.

        This method is a synonym for fetch(int).

      • limit

        default Table limit​(int offset,
                    int fetch)
        Limits a (possibly sorted) result to the first n rows from an offset position.

        This method is a synonym for offset(int) followed by fetch(int).

      • window

        GroupWindowedTable window​(GroupWindow groupWindow)
        Groups the records of a table by assigning them to windows defined by a time or row interval.

        For streaming tables of infinite size, grouping into windows is required to define finite groups on which group-based aggregates can be computed.

        For batch tables of finite size, windowing essentially provides shortcuts for time-based groupBy.

        Note: Computing windowed aggregates on a streaming table is only a parallel operation if additional grouping attributes are added to the groupBy(...) clause. If the groupBy(...) only references a GroupWindow alias, the streamed table will be processed by a single task, i.e., with parallelism 1.

        Parameters:
        groupWindow - groupWindow that specifies how elements are grouped.
        Returns:
        A windowed table.

      • window

        OverWindowedTable window​(OverWindow... overWindows)
        Defines over-windows on the records of a table.

        An over-window defines for each record an interval of records over which aggregation functions can be computed.

        Java Example: 

        
 table
   .window(Over.partitionBy($("c")).orderBy($("rowTime")).preceding(lit(10).seconds()).as("ow")
   .select($("c"), $("b").count().over($("ow")), $("e").sum().over($("ow")));

        Scala Example: 

        
 table
   .window(Over partitionBy $"c" orderBy $"rowTime" preceding 10.seconds as "ow")
   .select($"c", $"b".count over $"ow", $"e".sum over $"ow")

        Note: Computing over window aggregates on a streaming table is only a parallel operation if the window is partitioned. Otherwise, the whole stream will be processed by a single task, i.e., with parallelism 1.

        Note: Over-windows for batch tables are currently not supported.

        Parameters:
        overWindows - windows that specify the record interval over which aggregations are computed.
        Returns:
        An OverWindowedTable to specify the aggregations.

      • addColumns

        Table addColumns​(Expression... fields)
        Adds additional columns. Similar to a SQL SELECT statement. The field expressions can contain complex expressions, but can not contain aggregations. It will throw an exception if the added fields already exist.

        Java Example: 

        
 tab.addColumns(
    $("a").plus(1).as("a1"),
    concat($("b"), "sunny").as("b1")
 );

        Scala Example: 

        
 tab.addColumns(
    $"a" + 1 as "a1",
    concat($"b", "sunny") as "b1"
 )

      • addOrReplaceColumns

        Table addOrReplaceColumns​(Expression... fields)
        Adds additional columns. Similar to a SQL SELECT statement. The field expressions can contain complex expressions, but can not contain aggregations. Existing fields will be replaced. If the added fields have duplicate field name, then the last one is used.

        Java Example: 

        
 tab.addOrReplaceColumns(
    $("a").plus(1).as("a1"),
    concat($("b"), "sunny").as("b1")
 );

        Scala Example: 

        
 tab.addOrReplaceColumns(
    $"a" + 1 as "a1",
    concat($"b", "sunny") as "b1"
 )

      • renameColumns

        Table renameColumns​(Expression... fields)
        Renames existing columns. Similar to a field alias statement. The field expressions should be alias expressions, and only the existing fields can be renamed.

        Java Example: 

        
 tab.renameColumns(
    $("a").as("a1"),
    $("b").as("b1")
 );

        Scala Example: 

        
 tab.renameColumns(
    $"a" as "a1",
    $"b" as "b1"
 )

      • dropColumns

        Table dropColumns​(Expression... fields)
        Drops existing columns. The field expressions should be field reference expressions.

        Java Example: 

        
 tab.dropColumns($("a"), $("b"));

        Scala Example: 

        
 tab.dropColumns($"a", $"b")

      • map

        Table map​(Expression mapFunction)
        Performs a map operation with an user-defined scalar function or built-in scalar function. The output will be flattened if the output type is a composite type.

        Java Example: 

        
 tab.map(call(MyMapFunction.class, $("c")))

        Scala Example: 

        
 val func = new MyMapFunction()
 tab.map(func($"c"))

      • flatMap

        Table flatMap​(Expression tableFunction)
        Performs a flatMap operation with an user-defined table function or built-in table function. The output will be flattened if the output type is a composite type.

        Java Example: 

        
 tab.flatMap(call(MyFlatMapFunction.class, $("c")))

        Scala Example: 

        
 val func = new MyFlatMapFunction()
 tab.flatMap(func($"c"))

      • aggregate

        AggregatedTable aggregate​(Expression aggregateFunction)
        Performs a global aggregate operation with an aggregate function. You have to close the aggregate(Expression) with a select statement. The output will be flattened if the output type is a composite type.

        Java Example: 

        
 tab.aggregate(call(MyAggregateFunction.class, $("a"), $("b")).as("f0", "f1", "f2"))
   .select($("f0"), $("f1"));

        Scala Example: 

        
 val aggFunc = new MyAggregateFunction
 table.aggregate(aggFunc($"a", $"b") as ("f0", "f1", "f2"))
   .select($"f0", $"f1")

      • flatAggregate

        FlatAggregateTable flatAggregate​(Expression tableAggregateFunction)
        Perform a global flatAggregate without groupBy. FlatAggregate takes a TableAggregateFunction which returns multiple rows. Use a selection after the flatAggregate.

        Java Example: 

        
 tab.flatAggregate(call(MyTableAggregateFunction.class, $("a"), $("b")).as("x", "y", "z"))
   .select($("x"), $("y"), $("z"));

        Scala Example: 

        
 val tableAggFunc: TableAggregateFunction = new MyTableAggregateFunction
 tab.flatAggregate(tableAggFunc($"a", $"b") as ("x", "y", "z"))
   .select($"x", $"y", $"z")

      • insertInto

        TablePipeline insertInto​(TableDescriptor descriptor)
        Declares that the pipeline defined by the given Table object should be written to a table (backed by a DynamicTableSink) expressed via the given TableDescriptor.

        The descriptor won't be registered in the catalog, but it will be propagated directly in the operation tree. Note that calling this method multiple times, even with the same descriptor, results in multiple sink tables instances.

        This method allows to declare a Schema for the sink descriptor. The declaration is similar to a CREATE TABLE DDL in SQL and allows to:

        • overwrite automatically derived columns with a custom DataType
        • add metadata columns next to the physical columns
        • declare a primary key

        It is possible to declare a schema without physical/regular columns. In this case, those columns will be automatically derived and implicitly put at the beginning of the schema declaration.

        Examples: 

        
 Schema schema = Schema.newBuilder()
   .column("f0", DataTypes.STRING())
   .build();

 Table table = tableEnv.from(TableDescriptor.forConnector("datagen")
   .schema(schema)
   .build());

 table.insertInto(TableDescriptor.forConnector("blackhole")
   .schema(schema)
   .build());

        One can execute the returned TablePipeline using Executable.execute(), or compile it to a CompiledPlan using Compilable.compilePlan().

        If multiple pipelines should insert data into one or more sink tables as part of a single execution, use a StatementSet (see TableEnvironment.createStatementSet()).

        Parameters:
        descriptor - Descriptor describing the sink table into which data should be inserted.
        Returns:
        The complete pipeline from one or more source tables to a sink table.

      • insertInto

        TablePipeline insertInto​(TableDescriptor descriptor,
                         boolean overwrite)
        Declares that the pipeline defined by the given Table object should be written to a table (backed by a DynamicTableSink) expressed via the given TableDescriptor.

        The descriptor won't be registered in the catalog, but it will be propagated directly in the operation tree. Note that calling this method multiple times, even with the same descriptor, results in multiple sink tables being registered.

        This method allows to declare a Schema for the sink descriptor. The declaration is similar to a CREATE TABLE DDL in SQL and allows to:

        • overwrite automatically derived columns with a custom DataType
        • add metadata columns next to the physical columns
        • declare a primary key

        It is possible to declare a schema without physical/regular columns. In this case, those columns will be automatically derived and implicitly put at the beginning of the schema declaration.

        Examples: 

        
 Schema schema = Schema.newBuilder()
   .column("f0", DataTypes.STRING())
   .build();

 Table table = tableEnv.from(TableDescriptor.forConnector("datagen")
   .schema(schema)
   .build());

 table.insertInto(TableDescriptor.forConnector("blackhole")
   .schema(schema)
   .build(), true);

        One can execute the returned TablePipeline using Executable.execute(), or compile it to a CompiledPlan using Compilable.compilePlan().

        If multiple pipelines should insert data into one or more sink tables as part of a single execution, use a StatementSet (see TableEnvironment.createStatementSet()).

        Parameters:
        descriptor - Descriptor describing the sink table into which data should be inserted.
        overwrite - Indicates whether existing data should be overwritten.
        Returns:
        The complete pipeline from one or more source tables to a sink table.