Append Only Table
This documentation is for an unreleased version of Apache Flink Table Store. We recommend you use the latest stable version.

Append Only Table #

By specifying 'write-mode' = 'append-only' when creating the table, user creates an append-only table.

You can only insert a complete record into the table. No delete or update is supported and you cannot define primary keys. This type of table is suitable for use cases that do not require updates (such as log data synchronization).

Bucketing #

You can also define bucket number for Append-only table, see Bucket.

It is recommended that you set the bucket-key field. Otherwise, the data will be hashed according to the whole row, and the performance will be poor.

Streaming Read Order #

For streaming reads, records are produced in the following order:

  • For any two records from two different partitions
    • If scan.plan-sort-partition is set to true, the record with a smaller partition value will be produced first.
    • Otherwise, the record with an earlier partition creation time will be produced first.
  • For any two records from the same partition and the same bucket, the first written record will be produced first.
  • For any two records from the same partition but two different buckets, different buckets are processed by different tasks, there is no order guarantee between them.

Compaction #

By default, the sink node will automatically perform compaction to control the number of files. The following options control the strategy of compaction:

Key Default Type Description
write-only
 false Boolean If set to true, compactions and snapshot expiration will be skipped. This option is used along with dedicated compact jobs.
compaction.min.file-num
 5 Integer For file set [f_0,...,f_N], the minimum file number which satisfies sum(size(f_i)) >= targetFileSize to trigger a compaction for append-only table. This value avoids almost-full-file to be compacted, which is not cost-effective.
compaction.early-max.file-num
 50 Integer For file set [f_0,...,f_N], the maximum file number to trigger a compaction for append-only table, even if sum(size(f_i)) < targetFileSize. This value avoids pending too much small files, which slows down the performance.

Example #

The following is an example of creating the Append-Only table and specifying the bucket key.

CREATE TABLE MyTable (
    product_id BIGINT,
    price DOUBLE,
    sales BIGINT
) WITH (
    'write-mode' = 'append-only',
    'bucket' = '8',
    'bucket-key' = 'product_id'
);