@Internal public class NonSplittingRecursiveAllDirEnumerator extends NonSplittingRecursiveEnumerator
FileEnumerator
enumerates all files under the given paths recursively except the
hidden directories. Each file matched the given regex pattern becomes one split; this enumerator
does not split files into smaller "block" units.
The default instantiation of this enumerator filters files with the common hidden file prefixes '.' and '_'. A custom file filter can be specified.
Compared to NonSplittingRecursiveEnumerator
, this enumerator will enumerate all files
even through its parent directory is filtered out by the file filter.
FileEnumerator.Provider
fileFilter
Constructor and Description |
---|
NonSplittingRecursiveAllDirEnumerator(java.util.function.Predicate<Path> fileFilter)
Creates a NonSplittingRegexEnumerator that enumerates all files whose file path matches the
regex.
|
NonSplittingRecursiveAllDirEnumerator(String pathRegexPattern)
Creates a NonSplittingRegexEnumerator that enumerates all files whose file path matches the
regex except hidden files.
|
Modifier and Type | Method and Description |
---|---|
protected void |
addSplitsForPath(FileStatus fileStatus,
FileSystem fs,
ArrayList<FileSourceSplit> target) |
convertToSourceSplits, enumerateSplits, getNextId
public NonSplittingRecursiveAllDirEnumerator(String pathRegexPattern)
public NonSplittingRecursiveAllDirEnumerator(java.util.function.Predicate<Path> fileFilter)
protected void addSplitsForPath(FileStatus fileStatus, FileSystem fs, ArrayList<FileSourceSplit> target) throws IOException
addSplitsForPath
in class NonSplittingRecursiveEnumerator
IOException
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.