org.apache.flink.connector.file.table

Class FileSystemTableSource

java.lang.Object

org.apache.flink.connector.file.table.FileSystemTableSource

All Implemented Interfaces:

SupportsFilterPushDown, SupportsLimitPushDown, SupportsPartitionPushDown, SupportsProjectionPushDown, SupportsReadingMetadata, SupportsStatisticReport, DynamicTableSource, ScanTableSource

@Internal
public class FileSystemTableSource
extends Object
implements ScanTableSource, SupportsProjectionPushDown, SupportsLimitPushDown, SupportsPartitionPushDown, SupportsFilterPushDown, SupportsReadingMetadata, SupportsStatisticReport

File system table source.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.table.connector.source.ScanTableSource

ScanTableSource.ScanContext, ScanTableSource.ScanRuntimeProvider

Nested classes/interfaces inherited from interface org.apache.flink.table.connector.source.DynamicTableSource

DynamicTableSource.Context, DynamicTableSource.DataStructureConverter

Nested classes/interfaces inherited from interface org.apache.flink.table.connector.source.abilities.SupportsFilterPushDown

SupportsFilterPushDown.Result

Constructor Summary

Constructors

Constructor and Description
FileSystemTableSource(ObjectIdentifier tableIdentifier, DataType physicalRowDataType, List<String> partitionKeys, ReadableConfig tableOptions, DecodingFormat<BulkFormat<RowData,FileSourceSplit>> bulkReaderFormat, DecodingFormat<DeserializationSchema<RowData>> deserializationFormat)

Method Summary

Modifier and Type	Method and Description
SupportsFilterPushDown.Result	applyFilters(List<ResolvedExpression> filters) Provides a list of filters in conjunctive form.
void	applyLimit(long limit) Provides the expected maximum number of produced records for limiting on a best-effort basis.
void	applyPartitions(List<Map<String,String>> remainingPartitions) Provides a list of remaining partitions.
void	applyProjection(int[][] projectedFields, DataType producedDataType) Provides the field index paths that should be used for a projection.
void	applyReadableMetadata(List<String> metadataKeys, DataType producedDataType) Provides a list of metadata keys that the produced RowData must contain as appended metadata columns.
String	asSummaryString() Returns a string that summarizes this source for printing to a console or log.
FileSystemTableSource	copy() Creates a copy of this instance during planning.
ChangelogMode	getChangelogMode() Returns the set of changes that the planner can expect during runtime.
ScanTableSource.ScanRuntimeProvider	getScanRuntimeProvider(ScanTableSource.ScanContext scanContext) Returns a provider of runtime implementation for reading the data.
Optional<List<Map<String,String>>>	listPartitions() Returns a list of all partitions that a source can read if available.
Map<String,DataType>	listReadableMetadata() Returns the map of metadata keys and their corresponding data types that can be produced by this table source for reading.
TableStats	reportStatistics() Returns the estimated statistics of this DynamicTableSource, else TableStats.UNKNOWN if some situations are not supported or cannot be handled.
boolean	supportsNestedProjection() Returns whether this source supports nested projection.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.table.connector.source.abilities.SupportsProjectionPushDown

applyProjection

Methods inherited from interface org.apache.flink.table.connector.source.abilities.SupportsReadingMetadata

supportsMetadataProjection

Constructor Detail

FileSystemTableSource

public FileSystemTableSource(ObjectIdentifier tableIdentifier,
                             DataType physicalRowDataType,
                             List<String> partitionKeys,
                             ReadableConfig tableOptions,
                             @Nullable
                             DecodingFormat<BulkFormat<RowData,FileSourceSplit>> bulkReaderFormat,
                             @Nullable
                             DecodingFormat<DeserializationSchema<RowData>> deserializationFormat)

Method Detail

getScanRuntimeProvider

public ScanTableSource.ScanRuntimeProvider getScanRuntimeProvider(ScanTableSource.ScanContext scanContext)

Description copied from interface: ScanTableSource

Returns a provider of runtime implementation for reading the data.

There might exist different interfaces for runtime implementation which is why ScanTableSource.ScanRuntimeProvider serves as the base interface. Concrete ScanTableSource.ScanRuntimeProvider interfaces might be located in other Flink modules.

Independent of the provider interface, the table runtime expects that a source implementation emits internal data structures (see RowData for more information).

The given ScanTableSource.ScanContext offers utilities by the planner for creating runtime implementation with minimal dependencies to internal data structures.

SourceProvider is the recommended core interface. SourceFunctionProvider in flink-table-api-java-bridge and InputFormatProvider are available for backwards compatibility.

Specified by:

getScanRuntimeProvider in interface ScanTableSource

See Also:

SourceProvider

getChangelogMode

public ChangelogMode getChangelogMode()

Description copied from interface: ScanTableSource

Returns the set of changes that the planner can expect during runtime.

Specified by:

getChangelogMode in interface ScanTableSource

See Also:

RowKind

applyFilters

public SupportsFilterPushDown.Result applyFilters(List<ResolvedExpression> filters)

Description copied from interface: SupportsFilterPushDown

Provides a list of filters in conjunctive form. A source can pick filters and return the accepted and remaining filters.

See the documentation of SupportsFilterPushDown for more information.

Specified by:

applyFilters in interface SupportsFilterPushDown

applyLimit

public void applyLimit(long limit)

Description copied from interface: SupportsLimitPushDown

Provides the expected maximum number of produced records for limiting on a best-effort basis.

Specified by:

applyLimit in interface SupportsLimitPushDown

listPartitions

public Optional<List<Map<String,String>>> listPartitions()

Description copied from interface: SupportsPartitionPushDown

Returns a list of all partitions that a source can read if available.

A single partition maps each partition key to a partition value.

If Optional.empty() is returned, the list of partitions is queried from the catalog.

Specified by:

listPartitions in interface SupportsPartitionPushDown

applyPartitions

public void applyPartitions(List<Map<String,String>> remainingPartitions)

Description copied from interface: SupportsPartitionPushDown

Provides a list of remaining partitions. After those partitions are applied, a source must not read the data of other partitions during runtime.

See the documentation of SupportsPartitionPushDown for more information.

Specified by:

applyPartitions in interface SupportsPartitionPushDown

supportsNestedProjection

public boolean supportsNestedProjection()

Description copied from interface: SupportsProjectionPushDown

Returns whether this source supports nested projection.

Specified by:

supportsNestedProjection in interface SupportsProjectionPushDown

reportStatistics

public TableStats reportStatistics()

Description copied from interface: SupportsStatisticReport

Returns the estimated statistics of this DynamicTableSource, else TableStats.UNKNOWN if some situations are not supported or cannot be handled.

Specified by:

reportStatistics in interface SupportsStatisticReport

copy

public FileSystemTableSource copy()

Description copied from interface: DynamicTableSource

Creates a copy of this instance during planning. The copy should be a deep copy of all mutable members.

Specified by:

copy in interface DynamicTableSource

asSummaryString

public String asSummaryString()

Description copied from interface: DynamicTableSource

Returns a string that summarizes this source for printing to a console or log.

Specified by:

asSummaryString in interface DynamicTableSource

applyProjection

public void applyProjection(int[][] projectedFields,
                            DataType producedDataType)

Description copied from interface: SupportsProjectionPushDown

Provides the field index paths that should be used for a projection. The indices are 0-based and support fields within (possibly nested) structures if this is enabled via SupportsProjectionPushDown.supportsNestedProjection().

In the example mentioned in SupportsProjectionPushDown, this method would receive:

• [[2], [1]] which is equivalent to [["s"], ["r"]] if SupportsProjectionPushDown.supportsNestedProjection() returns false.
• [[2], [1, 0]] which is equivalent to [["s"], ["r", "d"]]] if SupportsProjectionPushDown.supportsNestedProjection() returns true.

Note: Use the passed data type instead of ResolvedSchema.toPhysicalRowDataType() for describing the final output data type when creating TypeInformation.

Specified by:

applyProjection in interface SupportsProjectionPushDown

Parameters:

projectedFields - field index paths of all fields that must be present in the physically produced data

producedDataType - the final output type of the source, with the projection applied

applyReadableMetadata

public void applyReadableMetadata(List<String> metadataKeys,
                                  DataType producedDataType)

Description copied from interface: SupportsReadingMetadata

Provides a list of metadata keys that the produced RowData must contain as appended metadata columns.

Implementations of this method must be idempotent. The planner might call this method multiple times.

Note: Use the passed data type instead of ResolvedSchema.toPhysicalRowDataType() for describing the final output data type when creating TypeInformation. If the source implements SupportsProjectionPushDown, the projection is already considered in the given output data type, use the producedDataType provided by this method instead of the producedDataType provided by SupportsProjectionPushDown.applyProjection(int[][], DataType).

Specified by:

applyReadableMetadata in interface SupportsReadingMetadata

Parameters:

metadataKeys - a subset of the keys returned by SupportsReadingMetadata.listReadableMetadata(), ordered by the iteration order of returned map

producedDataType - the final output type of the source, it is intended to be only forwarded and the planner will decide on the field names to avoid collisions

See Also:

DecodingFormat.applyReadableMetadata(List)

listReadableMetadata

public Map<String,DataType> listReadableMetadata()

Description copied from interface: SupportsReadingMetadata

Returns the map of metadata keys and their corresponding data types that can be produced by this table source for reading.

The returned map will be used by the planner for validation and insertion of explicit casts (see LogicalTypeCasts.supportsExplicitCast(LogicalType, LogicalType)) if necessary.

The iteration order of the returned map determines the order of metadata keys in the list passed in SupportsReadingMetadata.applyReadableMetadata(List, DataType). Therefore, it might be beneficial to return a LinkedHashMap if a strict metadata column order is required.

If a source forwards metadata from one or more formats, we recommend the following column order for consistency:

 KEY FORMAT METADATA COLUMNS + VALUE FORMAT METADATA COLUMNS + SOURCE METADATA COLUMNS
 

Metadata key names follow the same pattern as mentioned in Factory. In case of duplicate names in format and source keys, format keys shall have higher precedence.

Regardless of the returned DataTypes, a metadata column is always represented using internal data structures (see RowData).

Specified by:

listReadableMetadata in interface SupportsReadingMetadata

See Also:

DecodingFormat.listReadableMetadata()