@Internal public class PbFileFormatFactory extends Object implements BulkReaderFormatFactory, BulkWriterFormatFactory
ValidationException
when using Protobuf format factory for file system.
In practice, there is
no standard for storing bulk protobuf messages. This factory is present to prevent falling
back to the DeserializationSchemaAdapter
, a
line-based format which could silently succeed but write unrecoverable data to disk.
If your use case requires storing bulk protobuf messages on disk, the parquet file format might be the appropriate container and has an API for mapping records to protobuf messages.
Constructor and Description |
---|
PbFileFormatFactory() |
Modifier and Type | Method and Description |
---|---|
BulkDecodingFormat<RowData> |
createDecodingFormat(DynamicTableFactory.Context context,
ReadableConfig formatOptions)
Creates a
BulkDecodingFormat from the given context and format options. |
EncodingFormat<BulkWriter.Factory<RowData>> |
createEncodingFormat(DynamicTableFactory.Context context,
ReadableConfig formatOptions)
Creates a format from the given context and format options.
|
String |
factoryIdentifier()
Returns a unique identifier among same factory interfaces.
|
Set<ConfigOption<?>> |
forwardOptions()
Returns a set of
ConfigOption that are directly forwarded to the runtime
implementation but don't affect the final execution topology. |
Set<ConfigOption<?>> |
optionalOptions()
Returns a set of
ConfigOption that an implementation of this factory consumes in
addition to Factory.requiredOptions() . |
Set<ConfigOption<?>> |
requiredOptions()
Returns a set of
ConfigOption that an implementation of this factory requires in
addition to Factory.optionalOptions() . |
public String factoryIdentifier()
Factory
For consistency, an identifier should be declared as one lower case word (e.g. kafka
). If multiple factories exist for different versions, a version should be appended
using "-" (e.g. elasticsearch-7
).
factoryIdentifier
in interface Factory
public Set<ConfigOption<?>> requiredOptions()
Factory
ConfigOption
that an implementation of this factory requires in
addition to Factory.optionalOptions()
.
See the documentation of Factory
for more information.
requiredOptions
in interface Factory
public Set<ConfigOption<?>> optionalOptions()
Factory
ConfigOption
that an implementation of this factory consumes in
addition to Factory.requiredOptions()
.
See the documentation of Factory
for more information.
optionalOptions
in interface Factory
public Set<ConfigOption<?>> forwardOptions()
FormatFactory
ConfigOption
that are directly forwarded to the runtime
implementation but don't affect the final execution topology.
Options declared here can override options of the persisted plan during an enrichment
phase. Since a restored topology is static, an implementer has to ensure that the declared
options don't affect fundamental abilities such as ChangelogMode
.
For example, given a JSON format, if an option defines how to parse timestamps, changing
the parsing behavior does not affect the pipeline topology and can be allowed. However, an
option that defines whether the format results in a ProjectableDecodingFormat
or not
is not allowed. The wrapping connector and planner might not react to the changed abilities
anymore.
forwardOptions
in interface FormatFactory
DynamicTableFactory.Context.getEnrichmentOptions()
public BulkDecodingFormat<RowData> createDecodingFormat(DynamicTableFactory.Context context, ReadableConfig formatOptions)
BulkReaderFormatFactory
BulkDecodingFormat
from the given context and format options.createDecodingFormat
in interface BulkReaderFormatFactory
createDecodingFormat
in interface DecodingFormatFactory<BulkFormat<RowData,FileSourceSplit>>
public EncodingFormat<BulkWriter.Factory<RowData>> createEncodingFormat(DynamicTableFactory.Context context, ReadableConfig formatOptions)
EncodingFormatFactory
The format options have been projected to top-level options (e.g. from format.ignore-errors
to ignore-errors
).
createEncodingFormat
in interface EncodingFormatFactory<BulkWriter.Factory<RowData>>
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.