@Experimental public interface PartitionableTableSink
TableSink
. A partitionable sink can writes
query results to partitions.
Partition columns are defined via catalog table.
For example, a partitioned table named my_table
with a table schema
[a INT, b VARCHAR, c DOUBLE, dt VARCHAR, country VARCHAR]
is partitioned on columns
dt, country
. Then dt
is the first partition column, and
country
is the secondary partition column.
We can insert data into table partitions using INSERT INTO PARTITION syntax, for example:
INSERT INTO my_table PARTITION (dt='2019-06-20', country='bar') select a, b, c from my_view
When all the partition columns are set a value in PARTITION clause, it is inserting into a
static partition. It will writes the query result into a static partition,
i.e. dt='2019-06-20', country='bar'
. The user specified static partitions will be told
to the sink via setStaticPartition(Map)
.
The INSERT INTO PARTITION syntax also supports dynamic partition inserts.
INSERT INTO my_table PARTITION (dt='2019-06-20') select a, b, c, country from another_view
When partial partition columns (prefix part of all partition columns) are set a value in
PARTITION clause, it is writing the query result into a dynamic partition. In the above example,
the static partition part is dt='2019-06-20'
which will be told to the sink via
setStaticPartition(Map)
. And the country
is the dynamic partition which will be
get from each record.Modifier and Type | Method and Description |
---|---|
default boolean |
configurePartitionGrouping(boolean supportsGrouping)
If returns true, sink can trust all records will definitely be grouped by partition fields
before consumed by the
TableSink , i.e. |
void |
setStaticPartition(Map<String,String> partitions)
Sets the static partition into the
TableSink . |
void setStaticPartition(Map<String,String> partitions)
TableSink
. The static partition may be partial
of all partition columns. See the class Javadoc for more details.
The static partition is represented as a Map<String, String>
which maps from
partition field name to partition value. The partition values are all encoded as strings,
i.e. encoded using String.valueOf(...). For example, if we have a static partition
f0=1024, f1="foo", f2="bar"
. f0 is an integer type, f1 and f2 are string types.
They will all be encoded as strings: "1024", "foo", "bar". And can be decoded to original
literals based on the field types.
partitions
- user specified static partitiondefault boolean configurePartitionGrouping(boolean supportsGrouping)
TableSink
, i.e. the sink will receive all elements of one
partition and then all elements of another partition, elements of different partitions
will not be mixed. For some sinks, this can be used to reduce number of the partition
writers to improve writing performance.
This method is used to configure the behavior of input whether to be grouped by partition, if true, at the same time the sink should also configure itself, i.e. set an internal field that changes the writing behavior (writing one partition at a time).
supportsGrouping
- whether the execution mode supports grouping,
e.g. grouping (usually use sort to implement) is only supported
in batch mode, not supported in streaming mode.supportsGrouping
is false, it should never return true (requires grouping), otherwise it will fail.Copyright © 2014–2020 The Apache Software Foundation. All rights reserved.