org.apache.flink.connector.file.src.enumerate

Class NonSplittingRecursiveAllDirEnumerator

java.lang.Object

org.apache.flink.connector.file.src.enumerate.NonSplittingRecursiveEnumerator

org.apache.flink.connector.file.src.enumerate.NonSplittingRecursiveAllDirEnumerator

All Implemented Interfaces:

FileEnumerator

@Internal
public class NonSplittingRecursiveAllDirEnumerator
extends NonSplittingRecursiveEnumerator

This FileEnumerator enumerates all files under the given paths recursively except the hidden directories. Each file matched the given regex pattern becomes one split; this enumerator does not split files into smaller "block" units.

The default instantiation of this enumerator filters files with the common hidden file prefixes '.' and '_'. A custom file filter can be specified.

Compared to NonSplittingRecursiveEnumerator, this enumerator will enumerate all files even through its parent directory is filtered out by the file filter.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.connector.file.src.enumerate.FileEnumerator

FileEnumerator.Provider

Field Summary

Fields inherited from class org.apache.flink.connector.file.src.enumerate.NonSplittingRecursiveEnumerator

fileFilter

Constructor Summary

Constructors

Constructor and Description
NonSplittingRecursiveAllDirEnumerator(Predicate<org.apache.flink.core.fs.Path> fileFilter) Creates a NonSplittingRegexEnumerator that enumerates all files whose file path matches the regex.
NonSplittingRecursiveAllDirEnumerator(String pathRegexPattern) Creates a NonSplittingRegexEnumerator that enumerates all files whose file path matches the regex except hidden files.

Method Summary

Modifier and Type	Method and Description
protected void	addSplitsForPath(org.apache.flink.core.fs.FileStatus fileStatus, org.apache.flink.core.fs.FileSystem fs, ArrayList<FileSourceSplit> target)

Methods inherited from class org.apache.flink.connector.file.src.enumerate.NonSplittingRecursiveEnumerator

convertToSourceSplits, enumerateSplits, getNextId

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

NonSplittingRecursiveAllDirEnumerator

public NonSplittingRecursiveAllDirEnumerator(String pathRegexPattern)

Creates a NonSplittingRegexEnumerator that enumerates all files whose file path matches the regex except hidden files. Hidden files are considered files where the filename starts with '.' or with '_'.

NonSplittingRecursiveAllDirEnumerator

public NonSplittingRecursiveAllDirEnumerator(Predicate<org.apache.flink.core.fs.Path> fileFilter)

Creates a NonSplittingRegexEnumerator that enumerates all files whose file path matches the regex. Support to use given custom predicate as a filter for file paths.

Method Detail

addSplitsForPath

protected void addSplitsForPath(org.apache.flink.core.fs.FileStatus fileStatus,
                                org.apache.flink.core.fs.FileSystem fs,
                                ArrayList<FileSourceSplit> target)
                         throws IOException

Overrides:

addSplitsForPath in class NonSplittingRecursiveEnumerator

Throws:

IOException