org.apache.flink.connector.file.src.enumerate

类 BlockSplittingRecursiveEnumerator

java.lang.Object

org.apache.flink.connector.file.src.enumerate.NonSplittingRecursiveEnumerator

org.apache.flink.connector.file.src.enumerate.BlockSplittingRecursiveEnumerator

所有已实现的接口:

FileEnumerator

@PublicEvolving
public class BlockSplittingRecursiveEnumerator
extends NonSplittingRecursiveEnumerator

This FileEnumerator enumerates all files under the given paths recursively, and creates a separate split for each file block.

Please note that file blocks are only exposed by some file systems, such as HDFS. File systems that do not expose block information will not create multiple file splits per file, but keep the files as one source split.

Files with suffixes corresponding to known compression formats (for example '.gzip', '.bz2', ...) will not be split. See StandardDeCompressors for a list of known formats and suffixes.

The default instantiation of this enumerator filters files with the common hidden file prefixes '.' and '_'. A custom file filter can be specified.

嵌套类概要

从接口继承的嵌套类/接口 org.apache.flink.connector.file.src.enumerate.FileEnumerator

FileEnumerator.Provider

构造器概要

构造器

构造器和说明
BlockSplittingRecursiveEnumerator() Creates a new enumerator that enumerates all files except hidden files.
BlockSplittingRecursiveEnumerator(java.util.function.Predicate<org.apache.flink.core.fs.Path> fileFilter, String[] nonSplittableFileSuffixes) Creates a new enumerator that uses the given predicate as a filter for file paths, and avoids splitting files with the given extension (typically to avoid splitting compressed files).

方法概要

限定符和类型	方法和说明
protected void	convertToSourceSplits(org.apache.flink.core.fs.FileStatus file, org.apache.flink.core.fs.FileSystem fs, List<FileSourceSplit> target)
protected boolean	isFileSplittable(org.apache.flink.core.fs.Path filePath)

从类继承的方法 org.apache.flink.connector.file.src.enumerate.NonSplittingRecursiveEnumerator

enumerateSplits, getNextId

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

BlockSplittingRecursiveEnumerator

public BlockSplittingRecursiveEnumerator()

Creates a new enumerator that enumerates all files except hidden files. Hidden files are considered files where the filename starts with '.' or with '_'.

The enumerator does not split files that have a suffix corresponding to a known compression format (for example '.gzip', '.bz2', '.xy', '.zip', ...). See StandardDeCompressors for details.

BlockSplittingRecursiveEnumerator

public BlockSplittingRecursiveEnumerator(java.util.function.Predicate<org.apache.flink.core.fs.Path> fileFilter,
                                         String[] nonSplittableFileSuffixes)

Creates a new enumerator that uses the given predicate as a filter for file paths, and avoids splitting files with the given extension (typically to avoid splitting compressed files).

方法详细资料

convertToSourceSplits

protected void convertToSourceSplits(org.apache.flink.core.fs.FileStatus file,
                                     org.apache.flink.core.fs.FileSystem fs,
                                     List<FileSourceSplit> target)
                              throws IOException

覆盖:

convertToSourceSplits 在类中 NonSplittingRecursiveEnumerator

抛出:

IOException

isFileSplittable

protected boolean isFileSplittable(org.apache.flink.core.fs.Path filePath)