@Experimental public interface PartitionableTableSink
TableSink. A partitionable sink can writes query results to partitions.
For example, a partitioned table named
my_table with a table schema
[a INT, b VARCHAR, c DOUBLE, dt VARCHAR, country VARCHAR] is partitioned on columns
dt, country. Then
dt is the first partition column, and
country is the secondary partition column.
We can insert data into table partitions using INSERT INTO PARTITION syntax, for example:
When all the partition columns are set a value in PARTITION clause, it is inserting into a static partition. It will writes the query result into a static partition, i.e.
INSERT INTO my_table PARTITION (dt='2019-06-20', country='bar') select a, b, c from my_view
dt='2019-06-20', country='bar'. The user specified static partitions will be told to the sink via
The INSERT INTO PARTITION syntax also supports dynamic partition inserts.
When partial partition columns (prefix part of all partition columns) are set a value in PARTITION clause, it is writing the query result into a dynamic partition. In the above example, the static partition part is
INSERT INTO my_table PARTITION (dt='2019-06-20') select a, b, c, country from another_view
dt='2019-06-20'which will be told to the sink via
setStaticPartition(Map). And the
countryis the dynamic partition which will be get from each record.
|Modifier and Type||Method and Description|
If returns true, sink can trust all records will definitely be grouped by partition fields before consumed by the
Gets the partition field names of the table.
Sets the static partition into the
All the partition fields should exist in the
TableSink. The static partition may be partial of all partition columns. See the class Javadoc for more details.
The static partition is represented as a
Map<String, String> which maps from
partition field name to partition value. The partition values are all encoded as strings,
i.e. encoded using String.valueOf(...). For example, if we have a static partition
f0=1024, f1="foo", f2="bar". f0 is an integer type, f1 and f2 are string types.
They will all be encoded as strings: "1024", "foo", "bar". And can be decoded to original
literals based on the field types.
partitions- user specified static partition
default boolean configurePartitionGrouping(boolean supportsGrouping)
TableSink, i.e. the sink will receive all elements of one partition and then all elements of another partition, elements of different partitions will not be mixed. For some sinks, this can be used to reduce number of the partition writers to improve writing performance.
This method is used to configure the behavior of input whether to be grouped by partition, if true, at the same time the sink should also configure itself, i.e. set an internal field that changes the writing behavior (writing one partition at a time).
supportsGrouping- whether the execution mode supports grouping, e.g. grouping (usually use sort to implement) is only supported in batch mode, not supported in streaming mode.
supportsGroupingis false, it should never return true (requires grouping), otherwise it will fail.
Copyright © 2014–2020 The Apache Software Foundation. All rights reserved.