org.apache.flink.connector.file.src.enumerate

Class BlockSplittingRecursiveAllDirEnumerator

java.lang.Object

org.apache.flink.connector.file.src.enumerate.NonSplittingRecursiveEnumerator

org.apache.flink.connector.file.src.enumerate.BlockSplittingRecursiveEnumerator

org.apache.flink.connector.file.src.enumerate.BlockSplittingRecursiveAllDirEnumerator

All Implemented Interfaces:

FileEnumerator

@Internal
public class BlockSplittingRecursiveAllDirEnumerator
extends BlockSplittingRecursiveEnumerator

This FileEnumerator enumerates all files under the given paths recursively except the hidden directories, and creates a separate split for each file block.

Please note that file blocks are only exposed by some file systems, such as HDFS. File systems that do not expose block information will not create multiple file splits per file, but keep the files as one source split.

Files with suffixes corresponding to known compression formats (for example '.gzip', '.bz2', ...) will not be split. See StandardDeCompressors for a list of known formats and suffixes.

Compared to BlockSplittingRecursiveEnumerator, this enumerator will enumerate all files even through its parent directory is filtered out by the file filter.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.connector.file.src.enumerate.FileEnumerator

FileEnumerator.Provider

Field Summary

Fields inherited from class org.apache.flink.connector.file.src.enumerate.NonSplittingRecursiveEnumerator

fileFilter

Constructor Summary

Constructors

Constructor and Description
BlockSplittingRecursiveAllDirEnumerator(Predicate<org.apache.flink.core.fs.Path> fileFilter, String[] nonSplittableFileSuffixes) Creates a new enumerator that uses the given predicate as a filter for file paths, and avoids splitting files with the given extension (typically to avoid splitting compressed files).
BlockSplittingRecursiveAllDirEnumerator(String pathPattern) Creates a new enumerator that enumerates all files whose file path matches the regex except hidden files.

Method Summary

Modifier and Type	Method and Description
protected void	addSplitsForPath(org.apache.flink.core.fs.FileStatus fileStatus, org.apache.flink.core.fs.FileSystem fs, ArrayList<FileSourceSplit> target)

Methods inherited from class org.apache.flink.connector.file.src.enumerate.BlockSplittingRecursiveEnumerator

convertToSourceSplits, isFileSplittable

Methods inherited from class org.apache.flink.connector.file.src.enumerate.NonSplittingRecursiveEnumerator

enumerateSplits, getNextId

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BlockSplittingRecursiveAllDirEnumerator

public BlockSplittingRecursiveAllDirEnumerator(String pathPattern)

Creates a new enumerator that enumerates all files whose file path matches the regex except hidden files. Hidden files are considered files where the filename starts with '.' or with '_'.

The enumerator does not split files that have a suffix corresponding to a known compression format (for example '.gzip', '.bz2', '.xy', '.zip', ...). See StandardDeCompressors for details.

BlockSplittingRecursiveAllDirEnumerator

public BlockSplittingRecursiveAllDirEnumerator(Predicate<org.apache.flink.core.fs.Path> fileFilter,
                                               String[] nonSplittableFileSuffixes)

Creates a new enumerator that uses the given predicate as a filter for file paths, and avoids splitting files with the given extension (typically to avoid splitting compressed files).

Method Detail

addSplitsForPath

protected void addSplitsForPath(org.apache.flink.core.fs.FileStatus fileStatus,
                                org.apache.flink.core.fs.FileSystem fs,
                                ArrayList<FileSourceSplit> target)
                         throws IOException

Overrides:

addSplitsForPath in class NonSplittingRecursiveEnumerator

Throws:

IOException