Route

Definition #

Route specifies the rule of matching a list of source-table and mapping to sink-table. The most typical scenario is the merge of sub-databases and sub-tables, routing multiple upstream source tables to the same sink table.

Parameters #

To describe a route, the follows are required:

parameter meaning optional/required
source-table Source table id, supports regular expressions required
sink-table Sink table id, supports symbol replacement required
replace-symbol Special symbol in sink-table for pattern replacing, will be replaced by original table name optional
description Routing rule description(a default value provided) optional

A route module can contain a list of source-table/sink-table rules.

Example #

Route one Data Source table to one Data Sink table #

if synchronize the table web_order in the database mydb to a Doris table ods_web_order, we can use this yaml file to define this route：

route:
  - source-table: mydb.web_order
    sink-table: mydb.ods_web_order
    description: sync table to one destination table with given prefix ods_

Route multiple Data Source tables to one Data Sink table #

What’s more, if you want to synchronize the sharding tables in the database mydb to a Doris table ods_web_order, we can use this yaml file to define this route：

route:
  - source-table: mydb\.*
    sink-table: mydb.ods_web_order
    description: sync sharding tables to one destination table

Complex Route via combining route rules #

What’s more, if you want to specify many different mapping rules, we can use this yaml file to define this route：

route:
  - source-table: mydb.orders
    sink-table: ods_db.ods_orders
    description: sync orders table to orders
  - source-table: mydb.shipments
    sink-table: ods_db.ods_shipments
    description: sync shipments table to ods_shipments
  - source-table: mydb.products
    sink-table: ods_db.ods_products
    description: sync products table to ods_products

Pattern Replacement in routing rules #

If you’d like to route source tables and rename them to sink tables with specific patterns, replace-symbol could be used to resemble source table names like this:

route:
  - source-table: source_db.\.*
    sink-table: sink_db.<>
    replace-symbol: <>
    description: route all tables in source_db to sink_db

Then, all tables including source_db.XXX will be routed to sink_db.XXX without hassle.