@Experimental public interface PartitionableTableSink
TableSink. A partitionable sink can writes query results to partitions.
Partition columns are defined via catalog table.
For example, a partitioned table named
my_table with a table schema
[a INT, b
VARCHAR, c DOUBLE, dt VARCHAR, country VARCHAR] is partitioned on columns
dt is the first partition column, and
country is the secondary partition
We can insert data into table partitions using INSERT INTO PARTITION syntax, for example:
INSERT INTO my_table PARTITION (dt='2019-06-20', country='bar') select a, b, c from my_view
When all the partition columns are set a value in PARTITION clause, it is inserting into a
static partition. It will writes the query result into a static partition, i.e.
dt='2019-06-20', country='bar'. The user specified static partitions will be told to the sink
The INSERT INTO PARTITION syntax also supports dynamic partition inserts.
INSERT INTO my_table PARTITION (dt='2019-06-20') select a, b, c, country from another_view
When partial partition columns (prefix part of all partition columns) are set a value in
PARTITION clause, it is writing the query result into a dynamic partition. In the above example,
the static partition part is
dt='2019-06-20' which will be told to the sink via
setStaticPartition(Map). And the
country is the dynamic partition which will be get
from each record.
|Modifier and Type||Method and Description|
If returns true, sink can trust all records will definitely be grouped by partition fields before consumed by the
Sets the static partition into the
TableSink. The static partition may be partial of all partition columns. See the class Javadoc for more details.
The static partition is represented as a
Map<String, String> which maps from
partition field name to partition value. The partition values are all encoded as strings,
i.e. encoded using String.valueOf(...). For example, if we have a static partition
f0=1024, f1="foo", f2="bar". f0 is an integer type, f1 and f2 are string types. They will
all be encoded as strings: "1024", "foo", "bar". And can be decoded to original literals
based on the field types.
partitions- user specified static partition
default boolean configurePartitionGrouping(boolean supportsGrouping)
TableSink, i.e. the sink will receive all elements of one partition and then all elements of another partition, elements of different partitions will not be mixed. For some sinks, this can be used to reduce number of the partition writers to improve writing performance.
This method is used to configure the behavior of input whether to be grouped by partition, if true, at the same time the sink should also configure itself, i.e. set an internal field that changes the writing behavior (writing one partition at a time).
supportsGrouping- whether the execution mode supports grouping, e.g. grouping (usually use sort to implement) is only supported in batch mode, not supported in streaming mode.
supportsGroupingis false, it should never return true (requires grouping), otherwise it will fail.
Copyright © 2014–2021 The Apache Software Foundation. All rights reserved.