org.apache.flink.connector.base.source.reader.fetcher

Class SingleThreadFetcherManager<E,SplitT extends org.apache.flink.api.connector.source.SourceSplit>

java.lang.Object

org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager<E,SplitT>

org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager<E,SplitT>

@PublicEvolving
public class SingleThreadFetcherManager<E,SplitT extends org.apache.flink.api.connector.source.SourceSplit>
extends SplitFetcherManager<E,SplitT>

A Fetcher Manager with a single fetching thread (I/O thread) that handles all splits concurrently.

This pattern is, for example, useful for connectors like File Readers, Apache Kafka Readers, etc. In the example of Kafka, there is a single thread that reads all splits (topic partitions) via the same client. In the example of the file source, there is a single thread that reads the files after another.

Field Summary

Fields inherited from class org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager

fetchers

Constructor Summary

Constructors

Constructor and Description
SingleThreadFetcherManager(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue, Supplier<SplitReader<E,SplitT>> splitReaderSupplier) Deprecated. Please use SingleThreadFetcherManager(Supplier, Configuration) instead.
SingleThreadFetcherManager(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue, Supplier<SplitReader<E,SplitT>> splitReaderSupplier, org.apache.flink.configuration.Configuration configuration) Deprecated. Please use SingleThreadFetcherManager(Supplier, Configuration) instead.
SingleThreadFetcherManager(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue, Supplier<SplitReader<E,SplitT>> splitReaderSupplier, org.apache.flink.configuration.Configuration configuration, Consumer<Collection<String>> splitFinishedHook) Deprecated. Please use SingleThreadFetcherManager(Supplier, Configuration, Consumer) instead.
SingleThreadFetcherManager(Supplier<SplitReader<E,SplitT>> splitReaderSupplier) Creates a new SplitFetcherManager with a single I/O threads.
SingleThreadFetcherManager(Supplier<SplitReader<E,SplitT>> splitReaderSupplier, org.apache.flink.configuration.Configuration configuration) Creates a new SplitFetcherManager with a single I/O threads.
SingleThreadFetcherManager(Supplier<SplitReader<E,SplitT>> splitReaderSupplier, org.apache.flink.configuration.Configuration configuration, Consumer<Collection<String>> splitFinishedHook) Creates a new SplitFetcherManager with a single I/O threads.

Method Summary

Modifier and Type	Method and Description
void	addSplits(List<SplitT> splitsToAdd)
protected SplitFetcher<E,SplitT>	getRunningFetcher()
void	removeSplits(List<SplitT> splitsToRemove)

Methods inherited from class org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager

checkErrors, close, createSplitFetcher, getNumAliveFetchers, getQueue, maybeShutdownFinishedFetchers, pauseOrResumeSplits, startFetcher

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SingleThreadFetcherManager

@Deprecated
public SingleThreadFetcherManager(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue,
                                              Supplier<SplitReader<E,SplitT>> splitReaderSupplier)

Deprecated. Please use SingleThreadFetcherManager(Supplier, Configuration) instead.

Creates a new SplitFetcherManager with a single I/O threads.

Parameters:

elementsQueue - The queue that is used to hand over data from the I/O thread (the fetchers) to the reader (which emits the records and book-keeps the state. This must be the same queue instance that is also passed to the SourceReaderBase.

splitReaderSupplier - The factory for the split reader that connects to the source system.

SingleThreadFetcherManager

@Deprecated
public SingleThreadFetcherManager(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue,
                                              Supplier<SplitReader<E,SplitT>> splitReaderSupplier,
                                              org.apache.flink.configuration.Configuration configuration)

Deprecated. Please use SingleThreadFetcherManager(Supplier, Configuration) instead.

Creates a new SplitFetcherManager with a single I/O threads.

Parameters:

elementsQueue - The queue that is used to hand over data from the I/O thread (the fetchers) to the reader (which emits the records and book-keeps the state. This must be the same queue instance that is also passed to the SourceReaderBase.

splitReaderSupplier - The factory for the split reader that connects to the source system.

configuration - The configuration to create the fetcher manager.

SingleThreadFetcherManager

@VisibleForTesting
 @Deprecated
public SingleThreadFetcherManager(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue,
                                                                  Supplier<SplitReader<E,SplitT>> splitReaderSupplier,
                                                                  org.apache.flink.configuration.Configuration configuration,
                                                                  Consumer<Collection<String>> splitFinishedHook)

Deprecated. Please use SingleThreadFetcherManager(Supplier, Configuration, Consumer) instead.

Creates a new SplitFetcherManager with a single I/O threads.

Parameters:

elementsQueue - The queue that is used to hand over data from the I/O thread (the fetchers) to the reader (which emits the records and book-keeps the state. This must be the same queue instance that is also passed to the SourceReaderBase.

splitReaderSupplier - The factory for the split reader that connects to the source system.

configuration - The configuration to create the fetcher manager.

splitFinishedHook - Hook for handling finished splits in split fetchers

SingleThreadFetcherManager

public SingleThreadFetcherManager(Supplier<SplitReader<E,SplitT>> splitReaderSupplier)

Creates a new SplitFetcherManager with a single I/O threads.

Parameters:

splitReaderSupplier - The factory for the split reader that connects to the source system.

SingleThreadFetcherManager

public SingleThreadFetcherManager(Supplier<SplitReader<E,SplitT>> splitReaderSupplier,
                                  org.apache.flink.configuration.Configuration configuration)

Creates a new SplitFetcherManager with a single I/O threads.

Parameters:

splitReaderSupplier - The factory for the split reader that connects to the source system.

configuration - The configuration to create the fetcher manager.

SingleThreadFetcherManager

public SingleThreadFetcherManager(Supplier<SplitReader<E,SplitT>> splitReaderSupplier,
                                  org.apache.flink.configuration.Configuration configuration,
                                  Consumer<Collection<String>> splitFinishedHook)

Creates a new SplitFetcherManager with a single I/O threads.

Parameters:

splitReaderSupplier - The factory for the split reader that connects to the source system.

configuration - The configuration to create the fetcher manager.

splitFinishedHook - Hook for handling finished splits in split fetchers

Method Detail

addSplits

public void addSplits(List<SplitT> splitsToAdd)

Specified by:

addSplits in class SplitFetcherManager<E,SplitT extends org.apache.flink.api.connector.source.SourceSplit>

removeSplits

public void removeSplits(List<SplitT> splitsToRemove)

Specified by:

removeSplits in class SplitFetcherManager<E,SplitT extends org.apache.flink.api.connector.source.SourceSplit>

getRunningFetcher

protected SplitFetcher<E,SplitT> getRunningFetcher()