@PublicEvolving public class BlockSplittingRecursiveEnumerator extends NonSplittingRecursiveEnumerator
FileEnumeratorenumerates all files under the given paths recursively, and creates a separate split for each file block.
Please note that file blocks are only exposed by some file systems, such as HDFS. File systems that do not expose block information will not create multiple file splits per file, but keep the files as one source split.
Files with suffixes corresponding to known compression formats (for example '.gzip', '.bz2',
...) will not be split. See
StandardDeCompressors for a list of known formats and
The default instantiation of this enumerator filters files with the common hidden file prefixes '.' and '_'. A custom file filter can be specified.
|Constructor and Description|
Creates a new enumerator that enumerates all files except hidden files.
Creates a new enumerator that uses the given predicate as a filter for file paths, and avoids splitting files with the given extension (typically to avoid splitting compressed files).
|Modifier and Type||Method and Description|
The enumerator does not split files that have a suffix corresponding to a known
compression format (for example '.gzip', '.bz2', '.xy', '.zip', ...). See
StandardDeCompressors for details.
public BlockSplittingRecursiveEnumerator(java.util.function.Predicate<Path> fileFilter, String nonSplittableFileSuffixes)
protected void convertToSourceSplits(FileStatus file, FileSystem fs, List<FileSourceSplit> target) throws IOException
protected boolean isFileSplittable(Path filePath)
Copyright © 2014–2023 The Apache Software Foundation. All rights reserved.