org.apache.flink.connector.file.src.impl

Class DynamicFileSplitEnumerator<SplitT extends FileSourceSplit>

java.lang.Object

org.apache.flink.connector.file.src.impl.DynamicFileSplitEnumerator<SplitT>

All Implemented Interfaces:

AutoCloseable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.api.connector.source.SplitEnumerator<SplitT,PendingSplitsCheckpoint<SplitT>>, org.apache.flink.api.connector.source.SupportsHandleExecutionAttemptSourceEvent

@Internal
public class DynamicFileSplitEnumerator<SplitT extends FileSourceSplit>
extends Object
implements org.apache.flink.api.connector.source.SplitEnumerator<SplitT,PendingSplitsCheckpoint<SplitT>>, org.apache.flink.api.connector.source.SupportsHandleExecutionAttemptSourceEvent

A SplitEnumerator implementation that supports dynamic filtering.

This enumerator handles DynamicFilteringEvent and filter out the desired input splits with the support of the DynamicFileEnumerator.

If the enumerator receives the first split request before any dynamic filtering data is received, it will enumerate all splits. If a DynamicFilterEvent is received during the fully enumerating, the remaining splits will be filtered accordingly.

Constructor Summary

Constructors

Constructor and Description
DynamicFileSplitEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<SplitT> context, DynamicFileEnumerator.Provider fileEnumeratorFactory, FileSplitAssigner.Provider splitAssignerFactory)

Method Summary

Modifier and Type	Method and Description
void	addReader(int subtaskId)
void	addSplitsBack(List<SplitT> splits, int subtaskId)
void	close()
void	handleSourceEvent(int subtaskId, int attemptNumber, org.apache.flink.api.connector.source.SourceEvent sourceEvent)
void	handleSourceEvent(int subtaskId, org.apache.flink.api.connector.source.SourceEvent sourceEvent)
void	handleSplitRequest(int subtask, String hostname)
PendingSplitsCheckpoint<SplitT>	snapshotState(long checkpointId)
void	start()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.connector.source.SplitEnumerator

notifyCheckpointComplete

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted

Constructor Detail

DynamicFileSplitEnumerator

public DynamicFileSplitEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<SplitT> context,
                                  DynamicFileEnumerator.Provider fileEnumeratorFactory,
                                  FileSplitAssigner.Provider splitAssignerFactory)

Method Detail

start

public void start()

Specified by:

start in interface org.apache.flink.api.connector.source.SplitEnumerator<SplitT extends FileSourceSplit,PendingSplitsCheckpoint<SplitT extends FileSourceSplit>>

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface org.apache.flink.api.connector.source.SplitEnumerator<SplitT extends FileSourceSplit,PendingSplitsCheckpoint<SplitT extends FileSourceSplit>>

Throws:

IOException

addReader

public void addReader(int subtaskId)

Specified by:

addReader in interface org.apache.flink.api.connector.source.SplitEnumerator<SplitT extends FileSourceSplit,PendingSplitsCheckpoint<SplitT extends FileSourceSplit>>

handleSplitRequest

public void handleSplitRequest(int subtask,
                               @Nullable
                               String hostname)

Specified by:

handleSplitRequest in interface org.apache.flink.api.connector.source.SplitEnumerator<SplitT extends FileSourceSplit,PendingSplitsCheckpoint<SplitT extends FileSourceSplit>>

handleSourceEvent

public void handleSourceEvent(int subtaskId,
                              org.apache.flink.api.connector.source.SourceEvent sourceEvent)

Specified by:

handleSourceEvent in interface org.apache.flink.api.connector.source.SplitEnumerator<SplitT extends FileSourceSplit,PendingSplitsCheckpoint<SplitT extends FileSourceSplit>>

addSplitsBack

public void addSplitsBack(List<SplitT> splits,
                          int subtaskId)

Specified by:

addSplitsBack in interface org.apache.flink.api.connector.source.SplitEnumerator<SplitT extends FileSourceSplit,PendingSplitsCheckpoint<SplitT extends FileSourceSplit>>

snapshotState

public PendingSplitsCheckpoint<SplitT> snapshotState(long checkpointId)

Specified by:

snapshotState in interface org.apache.flink.api.connector.source.SplitEnumerator<SplitT extends FileSourceSplit,PendingSplitsCheckpoint<SplitT extends FileSourceSplit>>

handleSourceEvent

public void handleSourceEvent(int subtaskId,
                              int attemptNumber,
                              org.apache.flink.api.connector.source.SourceEvent sourceEvent)

Specified by:

handleSourceEvent in interface org.apache.flink.api.connector.source.SupportsHandleExecutionAttemptSourceEvent