org.apache.flink.connector.base.source.reader

Class SingleThreadMultiplexSourceReaderBase<E,T,SplitT extends org.apache.flink.api.connector.source.SourceSplit,SplitStateT>

java.lang.Object

org.apache.flink.connector.base.source.reader.SourceReaderBase<E,T,SplitT,SplitStateT>

org.apache.flink.connector.base.source.reader.SingleThreadMultiplexSourceReaderBase<E,T,SplitT,SplitStateT>

Type Parameters:

E - The type of the records (the raw type that typically contains checkpointing information).

T - The final type of the records emitted by the source.

SplitT - The type of the splits processed by the source.

SplitStateT - The type of the mutable state per split.

All Implemented Interfaces:

AutoCloseable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.api.connector.source.SourceReader<T,SplitT>

@PublicEvolving
public abstract class SingleThreadMultiplexSourceReaderBase<E,T,SplitT extends org.apache.flink.api.connector.source.SourceSplit,SplitStateT>
extends SourceReaderBase<E,T,SplitT,SplitStateT>

A base for SourceReaders that read splits with one thread using one SplitReader. The splits can be read either one after the other (like in a file source) or concurrently by changing the subscription in the split reader (like in the Kafka Source).

To implement a source reader based on this class, implementors need to supply the following:

• A SplitReader, which connects to the source and reads/polls data. The split reader gets notified whenever there is a new split. The split reader would read files, contain a Kafka or other source client, etc.
• A RecordEmitter that takes a record from the Split Reader and updates the checkpointing state and converts it into the final form. For example for Kafka, the Record Emitter takes a ConsumerRecord, puts the offset information into state, transforms the records with the deserializers into the final type, and emits the record.
• The class must override the methods to convert back and forth between the immutable splits (SplitT) and the mutable split state representation (SplitStateT).
• Finally, the reader must decide what to do when it starts (SourceReaderBase.start()) or when a split is finished (SourceReaderBase.onSplitFinished(java.util.Map)).

Field Summary

Fields inherited from class org.apache.flink.connector.base.source.reader.SourceReaderBase

config, context, eofRecordEvaluator, options, recordEmitter, splitFetcherManager

Constructor Summary

Constructors

Constructor and Description
SingleThreadMultiplexSourceReaderBase(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue, SingleThreadFetcherManager<E,SplitT> splitFetcherManager, RecordEmitter<E,T,SplitStateT> recordEmitter, org.apache.flink.configuration.Configuration config, org.apache.flink.api.connector.source.SourceReaderContext context) Deprecated. Please use SingleThreadMultiplexSourceReaderBase(SingleThreadFetcherManager, RecordEmitter, Configuration, SourceReaderContext) instead.
SingleThreadMultiplexSourceReaderBase(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue, SingleThreadFetcherManager<E,SplitT> splitFetcherManager, RecordEmitter<E,T,SplitStateT> recordEmitter, RecordEvaluator<T> eofRecordEvaluator, org.apache.flink.configuration.Configuration config, org.apache.flink.api.connector.source.SourceReaderContext context) Deprecated. Please use SingleThreadMultiplexSourceReaderBase(SingleThreadFetcherManager, RecordEmitter, RecordEvaluator, Configuration, SourceReaderContext) instead.
SingleThreadMultiplexSourceReaderBase(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue, Supplier<SplitReader<E,SplitT>> splitReaderSupplier, RecordEmitter<E,T,SplitStateT> recordEmitter, org.apache.flink.configuration.Configuration config, org.apache.flink.api.connector.source.SourceReaderContext context) Deprecated. Please use SingleThreadMultiplexSourceReaderBase(Supplier, RecordEmitter, Configuration, SourceReaderContext) instead.
SingleThreadMultiplexSourceReaderBase(SingleThreadFetcherManager<E,SplitT> splitFetcherManager, RecordEmitter<E,T,SplitStateT> recordEmitter, org.apache.flink.configuration.Configuration config, org.apache.flink.api.connector.source.SourceReaderContext context) This constructor behaves like SingleThreadMultiplexSourceReaderBase(Supplier, RecordEmitter, Configuration, SourceReaderContext), but accepts a specific FutureCompletingBlockingQueue and SingleThreadFetcherManager.
SingleThreadMultiplexSourceReaderBase(SingleThreadFetcherManager<E,SplitT> splitFetcherManager, RecordEmitter<E,T,SplitStateT> recordEmitter, RecordEvaluator<T> eofRecordEvaluator, org.apache.flink.configuration.Configuration config, org.apache.flink.api.connector.source.SourceReaderContext context) This constructor behaves like SingleThreadMultiplexSourceReaderBase(Supplier, RecordEmitter, Configuration, SourceReaderContext), but accepts a specific FutureCompletingBlockingQueue, SingleThreadFetcherManager and RecordEvaluator.
SingleThreadMultiplexSourceReaderBase(Supplier<SplitReader<E,SplitT>> splitReaderSupplier, RecordEmitter<E,T,SplitStateT> recordEmitter, org.apache.flink.configuration.Configuration config, org.apache.flink.api.connector.source.SourceReaderContext context) The primary constructor for the source reader.

Method Summary

Methods inherited from class org.apache.flink.connector.base.source.reader.SourceReaderBase

addSplits, close, getNumberOfCurrentlyAssignedSplits, handleSourceEvents, initializedState, isAvailable, notifyNoMoreSplits, onSplitFinished, pauseOrResumeSplits, pollNext, snapshotState, start, toSplitType

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.connector.source.SourceReader

notifyCheckpointComplete

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted

Constructor Detail

SingleThreadMultiplexSourceReaderBase

public SingleThreadMultiplexSourceReaderBase(Supplier<SplitReader<E,SplitT>> splitReaderSupplier,
                                             RecordEmitter<E,T,SplitStateT> recordEmitter,
                                             org.apache.flink.configuration.Configuration config,
                                             org.apache.flink.api.connector.source.SourceReaderContext context)

The primary constructor for the source reader.

The reader will use a handover queue sized as configured via SourceReaderOptions.ELEMENT_QUEUE_CAPACITY.

SingleThreadMultiplexSourceReaderBase

@Deprecated
public SingleThreadMultiplexSourceReaderBase(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue,
                                                         Supplier<SplitReader<E,SplitT>> splitReaderSupplier,
                                                         RecordEmitter<E,T,SplitStateT> recordEmitter,
                                                         org.apache.flink.configuration.Configuration config,
                                                         org.apache.flink.api.connector.source.SourceReaderContext context)

Deprecated. Please use SingleThreadMultiplexSourceReaderBase(Supplier, RecordEmitter, Configuration, SourceReaderContext) instead.

This constructor behaves like SingleThreadMultiplexSourceReaderBase(Supplier, RecordEmitter, Configuration, SourceReaderContext), but accepts a specific FutureCompletingBlockingQueue.

SingleThreadMultiplexSourceReaderBase

@Deprecated
public SingleThreadMultiplexSourceReaderBase(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue,
                                                         SingleThreadFetcherManager<E,SplitT> splitFetcherManager,
                                                         RecordEmitter<E,T,SplitStateT> recordEmitter,
                                                         org.apache.flink.configuration.Configuration config,
                                                         org.apache.flink.api.connector.source.SourceReaderContext context)

Deprecated. Please use SingleThreadMultiplexSourceReaderBase(SingleThreadFetcherManager, RecordEmitter, Configuration, SourceReaderContext) instead.

This constructor behaves like SingleThreadMultiplexSourceReaderBase(Supplier, RecordEmitter, Configuration, SourceReaderContext), but accepts a specific FutureCompletingBlockingQueue and SingleThreadFetcherManager.

SingleThreadMultiplexSourceReaderBase

@Deprecated
public SingleThreadMultiplexSourceReaderBase(FutureCompletingBlockingQueue<RecordsWithSplitIds<E>> elementsQueue,
                                                         SingleThreadFetcherManager<E,SplitT> splitFetcherManager,
                                                         RecordEmitter<E,T,SplitStateT> recordEmitter,
                                                         @Nullable
                                                         RecordEvaluator<T> eofRecordEvaluator,
                                                         org.apache.flink.configuration.Configuration config,
                                                         org.apache.flink.api.connector.source.SourceReaderContext context)

Deprecated. Please use SingleThreadMultiplexSourceReaderBase(SingleThreadFetcherManager, RecordEmitter, RecordEvaluator, Configuration, SourceReaderContext) instead.

This constructor behaves like SingleThreadMultiplexSourceReaderBase(Supplier, RecordEmitter, Configuration, SourceReaderContext), but accepts a specific FutureCompletingBlockingQueue, SingleThreadFetcherManager and RecordEvaluator.

SingleThreadMultiplexSourceReaderBase

public SingleThreadMultiplexSourceReaderBase(SingleThreadFetcherManager<E,SplitT> splitFetcherManager,
                                             RecordEmitter<E,T,SplitStateT> recordEmitter,
                                             org.apache.flink.configuration.Configuration config,
                                             org.apache.flink.api.connector.source.SourceReaderContext context)

This constructor behaves like SingleThreadMultiplexSourceReaderBase(Supplier, RecordEmitter, Configuration, SourceReaderContext), but accepts a specific FutureCompletingBlockingQueue and SingleThreadFetcherManager.

SingleThreadMultiplexSourceReaderBase

public SingleThreadMultiplexSourceReaderBase(SingleThreadFetcherManager<E,SplitT> splitFetcherManager,
                                             RecordEmitter<E,T,SplitStateT> recordEmitter,
                                             @Nullable
                                             RecordEvaluator<T> eofRecordEvaluator,
                                             org.apache.flink.configuration.Configuration config,
                                             org.apache.flink.api.connector.source.SourceReaderContext context)

This constructor behaves like SingleThreadMultiplexSourceReaderBase(Supplier, RecordEmitter, Configuration, SourceReaderContext), but accepts a specific FutureCompletingBlockingQueue, SingleThreadFetcherManager and RecordEvaluator.