org.apache.flink.streaming.api.operators

Class SourceOperator<OUT,SplitT extends org.apache.flink.api.connector.source.SourceSplit>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

org.apache.flink.streaming.api.operators.SourceOperator<OUT,SplitT>

Type Parameters:

OUT - The output type of the operator.

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.runtime.io.AvailabilityProvider, org.apache.flink.runtime.operators.coordination.OperatorEventHandler, KeyContext, KeyContextHandler, SetupableStreamOperator<OUT>, TimestampsAndWatermarks.WatermarkUpdateListener, StreamOperator<OUT>, StreamOperatorStateHandler.CheckpointedStreamOperator, YieldingOperator<OUT>, PushingAsyncDataInput<OUT>

@Internal
public class SourceOperator<OUT,SplitT extends org.apache.flink.api.connector.source.SourceSplit>
extends AbstractStreamOperator<OUT>
implements org.apache.flink.runtime.operators.coordination.OperatorEventHandler, PushingAsyncDataInput<OUT>, TimestampsAndWatermarks.WatermarkUpdateListener

Base source operator only used for integrating the source reader which is proposed by FLIP-27. It implements the interface of PushingAsyncDataInput which is naturally compatible with one input processing in runtime stack.

Important Note on Serialization: The SourceOperator inherits the Serializable interface from the StreamOperator, but is in fact NOT serializable. The operator must only be instantiated in the StreamTask from its factory.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.runtime.io.PushingAsyncDataInput

PushingAsyncDataInput.DataOutput<T>

Nested classes/interfaces inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

org.apache.flink.runtime.io.AvailabilityProvider.AvailabilityHelper

Field Summary

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, config, lastRecordAttributes1, lastRecordAttributes2, latencyStats, LOG, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager

Fields inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

AVAILABLE

Constructor Summary

Constructors

Constructor and Description

SourceOperator(org.apache.flink.util.function.FunctionWithException<org.apache.flink.api.connector.source.SourceReaderContext,org.apache.flink.api.connector.source.SourceReader<OUT,SplitT>,Exception> readerFactory, org.apache.flink.runtime.operators.coordination.OperatorEventGateway operatorEventGateway, org.apache.flink.core.io.SimpleVersionedSerializer<SplitT> splitSerializer, org.apache.flink.api.common.eventtime.WatermarkStrategy<OUT> watermarkStrategy, ProcessingTimeService timeService, org.apache.flink.configuration.Configuration configuration, String localHostname, boolean emitProgressiveWatermarks, StreamTask.CanEmitBatchOfRecordsChecker canEmitBatchOfRecords)

Method Summary

Modifier and Type	Method and Description
void	close() This method is called at the very end of the operator's life, both in the case of a successful completion of the operation, and in the case of a failure and canceling.
DataInputStatus	emitNext(PushingAsyncDataInput.DataOutput<OUT> output) Pushes elements to the output from current data input, and returns the input status to indicate whether there are more available data in current input.
void	finish() This method is called at the end of data processing.
CompletableFuture<?>	getAvailableFuture()
org.apache.flink.runtime.metrics.groups.InternalSourceReaderMetricGroup	getSourceMetricGroup()
org.apache.flink.api.connector.source.SourceReader<OUT,SplitT>	getSourceReader()
void	handleOperatorEvent(org.apache.flink.runtime.operators.coordination.OperatorEvent event)
void	initializeState(org.apache.flink.runtime.state.StateInitializationContext context) Stream operators with state which can be restored need to override this hook method.
void	initReader() Initializes the reader.
protected void	initSourceMetricGroup()
void	notifyCheckpointAborted(long checkpointId)
void	notifyCheckpointComplete(long checkpointId)
void	open() This method is called immediately before any elements are processed, it should contain the operator's initialization logic, e.g. state initialization.
void	setup(StreamTask<?,?> containingTask, StreamConfig config, Output<StreamRecord<OUT>> output) Initializes the operator.
void	snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context) Stream operators with state, which want to participate in a snapshot need to override this hook method.
void	splitFinished(String splitId) Notifies that split has finished.
CompletableFuture<Void>	stop(org.apache.flink.runtime.io.network.api.StopMode mode)
void	updateCurrentEffectiveWatermark(long watermark) Update the effective watermark.
void	updateCurrentSplitWatermark(String splitId, long watermark) Notifies about changes to per split watermarks.
void	updateIdle(boolean isIdle) It should be called once the idle is changed.

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, isUsingCustomRawKeyedState, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, snapshotState, useSplittableTimers

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

and, isApproximatelyAvailable, isAvailable, or

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

getOperatorAttributes

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

hasKeyContext

Constructor Detail

SourceOperator

public SourceOperator(org.apache.flink.util.function.FunctionWithException<org.apache.flink.api.connector.source.SourceReaderContext,org.apache.flink.api.connector.source.SourceReader<OUT,SplitT>,Exception> readerFactory,
                      org.apache.flink.runtime.operators.coordination.OperatorEventGateway operatorEventGateway,
                      org.apache.flink.core.io.SimpleVersionedSerializer<SplitT> splitSerializer,
                      org.apache.flink.api.common.eventtime.WatermarkStrategy<OUT> watermarkStrategy,
                      ProcessingTimeService timeService,
                      org.apache.flink.configuration.Configuration configuration,
                      String localHostname,
                      boolean emitProgressiveWatermarks,
                      StreamTask.CanEmitBatchOfRecordsChecker canEmitBatchOfRecords)

Method Detail

setup

public void setup(StreamTask<?,?> containingTask,
                  StreamConfig config,
                  Output<StreamRecord<OUT>> output)

Description copied from interface: SetupableStreamOperator

Initializes the operator. Sets access to the context and the output.

Specified by:

setup in interface SetupableStreamOperator<OUT>

Overrides:

setup in class AbstractStreamOperator<OUT>

initSourceMetricGroup

@VisibleForTesting
protected void initSourceMetricGroup()

initReader

public void initReader()
                throws Exception

Initializes the reader. The code from this method should ideally happen in the constructor or in the operator factory even. It has to happen here at a slightly later stage, because of the lazy metric initialization.

Calling this method explicitly is an optional way to have the reader initialization a bit earlier than in open(), as needed by the SourceOperatorStreamTask

This code should move to the constructor once the metric groups are available at task setup time.

Throws:

Exception

getSourceMetricGroup

public org.apache.flink.runtime.metrics.groups.InternalSourceReaderMetricGroup getSourceMetricGroup()

open

public void open()
          throws Exception

Description copied from class: AbstractStreamOperator

This method is called immediately before any elements are processed, it should contain the operator's initialization logic, e.g. state initialization.

The default implementation does nothing.

Specified by:

open in interface StreamOperator<OUT>

Overrides:

open in class AbstractStreamOperator<OUT>

Throws:

Exception - An exception in this method causes the operator to fail.

finish

public void finish()
            throws Exception

Description copied from interface: StreamOperator

This method is called at the end of data processing.

The method is expected to flush all remaining buffered data. Exceptions during this flushing of buffered data should be propagated, in order to cause the operation to be recognized as failed, because the last data items are not processed properly.

After this method is called, no more records can be produced for the downstream operators.

WARNING: It is not safe to use this method to commit any transactions or other side effects! You can use this method to flush any buffered data that can later on be committed e.g. in a CheckpointListener.notifyCheckpointComplete(long).

NOTE:This method does not need to close any resources. You should release external resources in the StreamOperator.close() method.

Specified by:

finish in interface StreamOperator<OUT>

Overrides:

finish in class AbstractStreamOperator<OUT>

Throws:

Exception - An exception in this method causes the operator to fail.

stop

public CompletableFuture<Void> stop(org.apache.flink.runtime.io.network.api.StopMode mode)

close

public void close()
           throws Exception

Description copied from interface: StreamOperator

This method is called at the very end of the operator's life, both in the case of a successful completion of the operation, and in the case of a failure and canceling.

This method is expected to make a thorough effort to release all resources that the operator has acquired.

NOTE:It can not emit any records! If you need to emit records at the end of processing, do so in the StreamOperator.finish() method.

Specified by:

close in interface StreamOperator<OUT>

Overrides:

close in class AbstractStreamOperator<OUT>

Throws:

Exception

emitNext

public DataInputStatus emitNext(PushingAsyncDataInput.DataOutput<OUT> output)
                         throws Exception

Description copied from interface: PushingAsyncDataInput

Pushes elements to the output from current data input, and returns the input status to indicate whether there are more available data in current input.

This method should be non blocking.

Specified by:

emitNext in interface PushingAsyncDataInput<OUT>

Throws:

Exception

snapshotState

public void snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context)
                   throws Exception

Description copied from class: AbstractStreamOperator

Stream operators with state, which want to participate in a snapshot need to override this hook method.

Specified by:

snapshotState in interface StreamOperatorStateHandler.CheckpointedStreamOperator

Overrides:

snapshotState in class AbstractStreamOperator<OUT>

Parameters:

context - context that provides information and means required for taking a snapshot

Throws:

Exception

getAvailableFuture

public CompletableFuture<?> getAvailableFuture()

Specified by:

getAvailableFuture in interface org.apache.flink.runtime.io.AvailabilityProvider

initializeState

public void initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
                     throws Exception

Description copied from class: AbstractStreamOperator

Stream operators with state which can be restored need to override this hook method.

Specified by:

initializeState in interface StreamOperatorStateHandler.CheckpointedStreamOperator

Overrides:

initializeState in class AbstractStreamOperator<OUT>

Parameters:

context - context that allows to register different states.

Throws:

Exception

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)
                              throws Exception

Specified by:

notifyCheckpointComplete in interface org.apache.flink.api.common.state.CheckpointListener

Overrides:

notifyCheckpointComplete in class AbstractStreamOperator<OUT>

Throws:

Exception

notifyCheckpointAborted

public void notifyCheckpointAborted(long checkpointId)
                             throws Exception

Specified by:

notifyCheckpointAborted in interface org.apache.flink.api.common.state.CheckpointListener

Overrides:

notifyCheckpointAborted in class AbstractStreamOperator<OUT>

Throws:

Exception

handleOperatorEvent

public void handleOperatorEvent(org.apache.flink.runtime.operators.coordination.OperatorEvent event)

Specified by:

handleOperatorEvent in interface org.apache.flink.runtime.operators.coordination.OperatorEventHandler

updateIdle

public void updateIdle(boolean isIdle)

Description copied from interface: TimestampsAndWatermarks.WatermarkUpdateListener

It should be called once the idle is changed.

Specified by:

updateIdle in interface TimestampsAndWatermarks.WatermarkUpdateListener

updateCurrentEffectiveWatermark

public void updateCurrentEffectiveWatermark(long watermark)

Description copied from interface: TimestampsAndWatermarks.WatermarkUpdateListener

Update the effective watermark. If an output becomes idle, please call this#updateIdle instead of update the watermark to Long.MAX_VALUE. Because the output needs to distinguish between idle and real watermark.

Specified by:

updateCurrentEffectiveWatermark in interface TimestampsAndWatermarks.WatermarkUpdateListener

updateCurrentSplitWatermark

public void updateCurrentSplitWatermark(String splitId,
                                        long watermark)

Description copied from interface: TimestampsAndWatermarks.WatermarkUpdateListener

Notifies about changes to per split watermarks.

Specified by:

updateCurrentSplitWatermark in interface TimestampsAndWatermarks.WatermarkUpdateListener

splitFinished

public void splitFinished(String splitId)

Description copied from interface: TimestampsAndWatermarks.WatermarkUpdateListener

Notifies that split has finished.

Specified by:

splitFinished in interface TimestampsAndWatermarks.WatermarkUpdateListener

getSourceReader

@VisibleForTesting
public org.apache.flink.api.connector.source.SourceReader<OUT,SplitT> getSourceReader()