This documentation is for an unreleased version of Apache Flink CDC. We recommend you use the latest stable version.
Paimon
Paimon Pipeline Connector #
The Paimon Pipeline connector can be used as the Data Sink of the pipeline, and write data to Paimon. This document describes how to set up the Paimon Pipeline connector.
What can the connector do? #
- Create table automatically if not exist
- Schema change synchronization
- Data synchronization
How to create Pipeline #
The pipeline for reading data from MySQL and sink to Paimon can be defined as follows:
source:
type: mysql
name: MySQL Source
hostname: 127.0.0.1
port: 3306
username: admin
password: pass
tables: adb.\.*, bdb.user_table_[0-9]+, [app|web].order_\.*
server-id: 5401-5404
sink:
type: paimon
name: Paimon Sink
catalog.properties.metastore: filesystem
catalog.properties.warehouse: /path/warehouse
pipeline:
name: MySQL to Paimon Pipeline
parallelism: 2
Pipeline Connector Options #
Option | Required | Default | Type | Description |
---|---|---|---|---|
type | required | (none) | String | Specify what connector to use, here should be 'paimon' . |
name | optional | (none) | String | The name of the sink. |
catalog.properties.metastore | required | (none) | String | Metastore of paimon catalog, supports filesystem and hive. |
catalog.properties.warehouse | optional | (none) | String | The warehouse root path of catalog. |
catalog.properties.uri | optional | (none) | String | Uri of metastore server. |
commit.user | optional | "admin" |
String | User name for committing data files. |
partition.key | optional | (none) | String | Partition keys for each partitioned table, allow setting multiple primary keys for multiTables. Each table are separated by ';', and each partition key are separated by ','. For example, we can set partition.key of two tables by 'testdb.table1:id1,id2;testdb.table2:name'. |
catalog.properties.* | optional | (none) | String | Pass options of Paimon catalog to pipeline,See Paimon catalog options。 |
table.properties.* | optional | (none) | String | Pass options of Paimon table to pipeline,See Paimon table options。 |
Usage Notes #
-
Only support Paimon primary key table, so the source table must have primary keys.
-
Not support exactly-once. The connector uses at-least-once + primary key table for idempotent writing.
Data Type Mapping #
CDC type | Paimon type | NOTE |
---|---|---|
TINYINT | TINYINT | |
SMALLINT | SMALLINT | |
INT | INT | |
BIGINT | BIGINT | |
FLOAT | FLOAT | |
DOUBLE | DOUBLE | |
DECIMAL(p, s) | DECIMAL(p, s) | |
BOOLEAN | BOOLEAN | |
DATE | DATE | |
TIMESTAMP | TIMESTAMP | |
TIMESTAMP_LTZ | TIMESTAMP_LTZ | |
CHAR(n) | CHAR(n) | |
VARCHAR(n) | VARCHAR(n) |