org.apache.flink.connector.base.source.reader.fetcher

Class SplitFetcherManager<E,SplitT extends org.apache.flink.api.connector.source.SourceSplit>

java.lang.Object

org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager<E,SplitT>

Direct Known Subclasses:

SingleThreadFetcherManager

@PublicEvolving
public abstract class SplitFetcherManager<E,SplitT extends org.apache.flink.api.connector.source.SourceSplit>
extends Object

A class responsible for starting the SplitFetcher and manage the life cycles of them. This class works with the SourceReaderBase.

The split fetcher manager could be used to support different threading models by implementing the addSplits(List) method differently. For example, a single thread split fetcher manager would only start a single fetcher and assign all the splits to it. A one-thread-per-split fetcher may spawn a new thread every time a new split is assigned.

Field Summary

Fields

Modifier and Type	Field and Description
protected Map<Integer,SplitFetcher<E,SplitT>>	fetchers A map keeping track of all the split fetchers.

Constructor Summary

Constructors

Constructor and Description
SplitFetcherManager(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue, Supplier<SplitReader<E,SplitT>> splitReaderFactory, org.apache.flink.configuration.Configuration configuration) Deprecated. Please use SplitFetcherManager(Supplier, Configuration) instead.
SplitFetcherManager(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue, Supplier<SplitReader<E,SplitT>> splitReaderFactory, org.apache.flink.configuration.Configuration configuration, Consumer<Collection<String>> splitFinishedHook) Deprecated. Please use SplitFetcherManager(Supplier, Configuration, Consumer) instead.
SplitFetcherManager(Supplier<SplitReader<E,SplitT>> splitReaderFactory, org.apache.flink.configuration.Configuration configuration) Create a split fetcher manager.
SplitFetcherManager(Supplier<SplitReader<E,SplitT>> splitReaderFactory, org.apache.flink.configuration.Configuration configuration, Consumer<Collection<String>> splitFinishedHook) Create a split fetcher manager.

Method Summary

Modifier and Type	Method and Description
abstract void	addSplits(List<SplitT> splitsToAdd)
void	checkErrors()
void	close(long timeoutMs) Close the split fetcher manager.
protected SplitFetcher<E,SplitT>	createSplitFetcher() Synchronize method to ensure no fetcher is created after the split fetcher manager has closed.
int	getNumAliveFetchers()
FutureCompletingBlockingQueue<RecordsWithSplitIds<E>>	getQueue() Return the queue contains data produced by split fetchers.This method is Internal and only used in SourceReaderBase.
boolean	maybeShutdownFinishedFetchers() Check and shutdown the fetchers that have completed their work.
void	pauseOrResumeSplits(Collection<String> splitIdsToPause, Collection<String> splitIdsToResume)
abstract void	removeSplits(List<SplitT> splitsToRemove)
protected void	startFetcher(SplitFetcher<E,SplitT> fetcher)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

fetchers

protected final Map<Integer,SplitFetcher<E,SplitT extends org.apache.flink.api.connector.source.SourceSplit>> fetchers

A map keeping track of all the split fetchers.

Constructor Detail

SplitFetcherManager

@Deprecated
public SplitFetcherManager(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue,
                                       Supplier<SplitReader<E,SplitT>> splitReaderFactory,
                                       org.apache.flink.configuration.Configuration configuration)

Deprecated. Please use SplitFetcherManager(Supplier, Configuration) instead.

Create a split fetcher manager.

Parameters:

elementsQueue - the queue that split readers will put elements into.

splitReaderFactory - a supplier that could be used to create split readers.

configuration - the configuration of this fetcher manager.

SplitFetcherManager

@Deprecated
 @VisibleForTesting
public SplitFetcherManager(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue,
                                                           Supplier<SplitReader<E,SplitT>> splitReaderFactory,
                                                           org.apache.flink.configuration.Configuration configuration,
                                                           Consumer<Collection<String>> splitFinishedHook)

Deprecated. Please use SplitFetcherManager(Supplier, Configuration, Consumer) instead.

Create a split fetcher manager.

Parameters:

elementsQueue - the queue that split readers will put elements into.

splitReaderFactory - a supplier that could be used to create split readers.

configuration - the configuration of this fetcher manager.

splitFinishedHook - Hook for handling finished splits in split fetchers.

SplitFetcherManager

public SplitFetcherManager(Supplier<SplitReader<E,SplitT>> splitReaderFactory,
                           org.apache.flink.configuration.Configuration configuration)

Create a split fetcher manager.

Parameters:

splitReaderFactory - a supplier that could be used to create split readers.

configuration - the configuration of this fetcher manager.

SplitFetcherManager

public SplitFetcherManager(Supplier<SplitReader<E,SplitT>> splitReaderFactory,
                           org.apache.flink.configuration.Configuration configuration,
                           Consumer<Collection<String>> splitFinishedHook)

Create a split fetcher manager.

Parameters:

splitReaderFactory - a supplier that could be used to create split readers.

configuration - the configuration of this fetcher manager.

splitFinishedHook - Hook for handling finished splits in split fetchers.

Method Detail

addSplits

public abstract void addSplits(List<SplitT> splitsToAdd)

removeSplits

public abstract void removeSplits(List<SplitT> splitsToRemove)

pauseOrResumeSplits

public void pauseOrResumeSplits(Collection<String> splitIdsToPause,
                                Collection<String> splitIdsToResume)

startFetcher

protected void startFetcher(SplitFetcher<E,SplitT> fetcher)

createSplitFetcher

protected SplitFetcher<E,SplitT> createSplitFetcher()

Synchronize method to ensure no fetcher is created after the split fetcher manager has closed.

Returns:

the created split fetcher.

Throws:

IllegalStateException - if the split fetcher manager has closed.

maybeShutdownFinishedFetchers

public boolean maybeShutdownFinishedFetchers()

Check and shutdown the fetchers that have completed their work.

Returns:

true if all the fetchers have completed the work, false otherwise.

getQueue

@Internal
public FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> getQueue()

Return the queue contains data produced by split fetchers.This method is Internal and only used in SourceReaderBase.

close

public void close(long timeoutMs)
           throws Exception

Close the split fetcher manager.

Parameters:

timeoutMs - the max time in milliseconds to wait.

Throws:

Exception - when failed to close the split fetcher manager.

checkErrors

public void checkErrors()

getNumAliveFetchers

@VisibleForTesting
public int getNumAliveFetchers()