org.apache.flink.table.connector.format

Interface DecodingFormat<I>

Type Parameters:

I - runtime interface needed by the table source

All Superinterfaces:

Format

All Known Subinterfaces:

BulkDecodingFormat<T>, ProjectableDecodingFormat<I>

All Known Implementing Classes:

CanalJsonDecodingFormat, ChangelogCsvFormat, CsvFileFormatFactory.CsvBulkDecodingFormat, DebeziumJsonDecodingFormat, MaxwellJsonDecodingFormat, OggJsonDecodingFormat, OrcFileFormatFactory.OrcBulkDecodingFormat, ParquetFileFormatFactory.ParquetBulkDecodingFormat, PbDecodingFormat, UpsertKafkaDynamicTableFactory.DecodingFormatWrapper

@PublicEvolving
public interface DecodingFormat<I>
extends Format

A Format for a DynamicTableSource for reading rows.

Implementing a DecodingFormat

createRuntimeDecoder(DynamicTableSource.Context, DataType) takes a physicalDataType. This DataType has usually been derived from a table's ResolvedSchema and excludes partition, metadata, and other auxiliary columns. The physicalDataType should describe exactly the full serialized record. In other words: for every field in the serialized record there is a corresponding field at the same position in the physicalDataType. Some implementations may decide to be more lenient and allow users to omit fields but this depends on the format characteristics. For example, a CSV format implementation might allow the user to define the schema only for the first 5 of the 10 total columns available in each row.

If the format supports projections, that is it can exclude certain fields from being parsed independently of the fields defined in the schema and can reorder fields in the produced RowData, then it should implement ProjectableDecodingFormat. ProjectableDecodingFormat.createRuntimeDecoder(DynamicTableSource.Context, DataType, int[][]) provides the physicalDataType as described above and provides projections to compute the type to produce using Projection.of(projections).project(physicalDataType). For example, a JSON format implementation may match the fields based on the JSON object keys, hence it can easily produce RowData excluding unused object values and set values inside the RowData using the index provided by the projections array.

Whenever possible, it's highly recommended implementing ProjectableDecodingFormat, as it might help to reduce the data volume when users are reading large records but are using only a small subset of fields.

Using a DecodingFormat

DynamicTableSource that doesn't implement SupportsProjectionPushDown should invoke createRuntimeDecoder(DynamicTableSource.Context, DataType). Usually, DynamicTableFactory.Context#getPhysicalRowDataType() can provide the physicalDataType (stripped of any fields not available in the serialized record).

DynamicTableSource implementing SupportsProjectionPushDown should check whether the DecodingFormat is an instance of ProjectableDecodingFormat:

• If yes, then the connector can invoke ProjectableDecodingFormat.createRuntimeDecoder(DynamicTableSource.Context, DataType, int[][]) providing a non null projections array excluding auxiliary fields. The built runtime implementation will take care of projections, producing records of type Projection.of(projections).project(physicalDataType).
• If no, then the connector must take care of performing the projection, for example using ProjectedRowData to project physical RowData emitted from the decoder runtime implementation.

Method Summary

Modifier and Type	Method and Description
default void	applyReadableMetadata(List<String> metadataKeys) Provides a list of metadata keys that the produced row must contain as appended metadata columns.
I	createRuntimeDecoder(DynamicTableSource.Context context, DataType physicalDataType) Creates runtime decoder implementation that is configured to produce data of the given data type.
default Map<String,DataType>	listReadableMetadata() Returns the map of metadata keys and their corresponding data types that can be produced by this format for reading.

Methods inherited from interface org.apache.flink.table.connector.format.Format

getChangelogMode

Method Detail

createRuntimeDecoder

I createRuntimeDecoder(DynamicTableSource.Context context,
                       DataType physicalDataType)

Creates runtime decoder implementation that is configured to produce data of the given data type.

Parameters:

context - the context provides several utilities required to instantiate the runtime decoder implementation of the format

physicalDataType - For more details check the documentation of DecodingFormat.

listReadableMetadata

default Map<String,DataType> listReadableMetadata()

Returns the map of metadata keys and their corresponding data types that can be produced by this format for reading. By default, this method returns an empty map.

Metadata columns add additional columns to the table's schema. A decoding format is responsible to add requested metadata columns at the end of produced rows.

See SupportsReadingMetadata for more information.

Note: This method is only used if the outer DynamicTableSource implements SupportsReadingMetadata and calls this method in SupportsReadingMetadata.listReadableMetadata().

applyReadableMetadata

default void applyReadableMetadata(List<String> metadataKeys)

Provides a list of metadata keys that the produced row must contain as appended metadata columns. By default, this method throws an exception if metadata keys are defined.

See SupportsReadingMetadata for more information.

Note: This method is only used if the outer DynamicTableSource implements SupportsReadingMetadata and calls this method in SupportsReadingMetadata.applyReadableMetadata(List, DataType).