Doris Connector #

This article introduces of Doris Connector

Example #

source:
   type: values
   name: ValuesSource

sink:
   type: doris
   name: Doris Sink
   fenodes: 127.0.0.1:8030
   username: root
   password: ""
   table.create.properties.replication_num: 1

pipeline:
   parallelism: 1

Connector Options #

Option	Required	Default	Type	Description
type	required	(none)	String	Specify the Sink to use, here is `'doris'`.
name	optional	(none)	String	Name of PipeLine
fenodes	required	(none)	String	Http address of Doris cluster FE, such as 127.0.0.1:8030
benodes	optional	(none)	String	Http address of Doris cluster BE, such as 127.0.0.1:8040
jdbc-url	optional	(none)	String	JDBC address of Doris cluster, for example: jdbc:mysql://127.0.0.1:9030/db
username	required	(none)	String	Username of Doris cluster
password	optional	(none)	String	Password for Doris cluster
auto-redirect	optional	false	String	Whether to write through FE redirection and directly connect to BE to write
sink.enable.batch-mode	optional	true	Boolean	Whether to use the batch method to write to Doris
sink.flush.queue-size	optional	2	Integer	Queue size for batch writing
sink.buffer-flush.max-rows	optional	50000	Integer	Maximum number of Flush records in a single batch
sink.buffer-flush.max-bytes	optional	10485760(10MB)	Integer	Maximum number of bytes flushed in a single batch
sink.buffer-flush.interval	optional	10s	String	Flush interval duration. If this time is exceeded, the data will be flushed asynchronously
sink.properties.	optional	(none)	String	Parameters of StreamLoad. For example: `sink.properties.strict_mode: true`. See more about StreamLoad Properties properties
table.create.properties.*	optional	(none)	String	Create the Properties configuration of the table. For example: `table.create.properties.replication_num: 1`. See more about Doris Table Properties properties

Data Type Mapping #

Flink CDC Type	Doris Type	Note
TINYINT	TINYINT
SMALLINT	SMALLINT
INT	INT
BIGINT	BIGINT
DECIMAL	DECIMAL
FLOAT	FLOAT
DOUBLE	DOUBLE
BOOLEAN	BOOLEAN
DATE	DATE
TIMESTAMP [(p)]	DATETIME [(p)]
TIMESTAMP_LTZ [(p)]	DATETIME [(p)]
CHAR(n)	CHAR(n*3)	In Doris, strings are stored in UTF-8 encoding, so English characters occupy 1 byte and Chinese characters occupy 3 bytes. The length here is multiplied by 3. The maximum length of CHAR is 255. Once exceeded, it will automatically be converted to VARCHAR type.
VARCHAR(n)	VARCHAR(n*3)	Same as above. The length here is multiplied by 3. The maximum length of VARCHAR is 65533. Once exceeded, it will automatically be converted to STRING type.
BINARY(n)	STRING
VARBINARY(N)	STRING
STRING	STRING