org.apache.flink.connector.file.src.enumerate

Interface FileEnumerator

All Known Implementing Classes:

BlockSplittingRecursiveEnumerator, HiveSourceFileEnumerator, NonSplittingRecursiveEnumerator

@PublicEvolving
public interface FileEnumerator

The FileEnumerator's task is to discover all files to be read and to split them into a set of FileSourceSplit.

This includes possibly, path traversals, file filtering (by name or other patterns) and deciding whether to split files into multiple splits, and how to split them.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static interface	FileEnumerator.Provider Factory for the FileEnumerator, to allow the FileEnumerator to be eagerly initialized and to not be serializable.

Method Summary

Modifier and Type	Method and Description
Collection<FileSourceSplit>	enumerateSplits(Path[] paths, int minDesiredSplits) Generates all file splits for the relevant files under the given paths.

Method Detail

enumerateSplits

Collection<FileSourceSplit> enumerateSplits(Path[] paths,
                                            int minDesiredSplits)
                                     throws IOException

Generates all file splits for the relevant files under the given paths. The minDesiredSplits is an optional hint indicating how many splits would be necessary to exploit parallelism properly.

Throws:

IOException