Package org.apache.flink.runtime.asyncprocessing.operators

Class AbstractAsyncKeyOrderedStreamOperator<OUT>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

org.apache.flink.runtime.asyncprocessing.operators.AbstractAsyncKeyOrderedStreamOperator<OUT>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.state.CheckpointListener, KeyContext, KeyContextHandler, StreamOperator<OUT>, StreamOperatorStateHandler.CheckpointedStreamOperator, YieldingOperator<OUT>, AsyncKeyOrderedProcessing, AsyncKeyOrderedProcessingOperator

Direct Known Subclasses:

AbstractAsyncRunnableStreamOperator, AbstractAsyncStateStreamOperator

@Internal
public abstract class AbstractAsyncKeyOrderedStreamOperator<OUT>
extends AbstractStreamOperator<OUT>
implements AsyncKeyOrderedProcessingOperator

This operator is an abstract class that give the AbstractStreamOperator the ability to perform AsyncKeyOrderedProcessing. The aim is to make any subclass of AbstractStreamOperator could manipulate async state with only a change of base class.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field	Description
protected AsyncExecutionController	asyncExecutionController
protected RecordContext	currentProcessingContext	Act as a cache for setAsyncKeyedContextElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<T>, org.apache.flink.api.java.functions.KeySelector<T, ?>) and postProcessElement().
protected DeclarationManager	declarationManager
protected Environment	environment

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

combinedWatermark, config, lastRecordAttributes1, lastRecordAttributes2, latencyStats, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager

Constructor Summary

Constructors

Constructor	Description
AbstractAsyncKeyOrderedStreamOperator()

Method Summary

Modifier and Type	Method	Description
<K> void	asyncProcessWithKey(K key, org.apache.flink.util.function.ThrowingRunnable<Exception> processing)	Asynchronously process a code with a key provided.
void	beforeInitializeStateHandler()	Initialize necessary state components for AbstractStreamOperator.
void	close()	This method is called at the very end of the operator's life, both in the case of a successful completion of the operation, and in the case of a failure and canceling.
protected abstract AsyncExecutionController	createAsyncExecutionController()	Create a new AsyncExecutionController for the operator.
void	drainStateRequests()
void	finish()	This method is called at the end of data processing.
<K> AsyncKeyedStateBackend<K>	getAsyncKeyedStateBackend()
Object	getCurrentKey()
DeclarationManager	getDeclarationManager()	Get the declaration manager for user-logic declaring.
ElementOrder	getElementOrder()	Get the ElementOrder of this operator.
<K,N> InternalTimerService<N>	getInternalTimerService(String name, org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer, Triggerable<K,N> triggerable)	Returns a InternalTimerService that can be used to query current processing time and event time and to set timers.
protected abstract org.apache.flink.api.java.functions.KeySelector	getKeySelectorForAsyncKeyedContext(int index)	Get the KeySelector for the given input index.
<N,S extends org.apache.flink.api.common.state.v2.State,T> S	getOrCreateKeyedState(N defaultNamespace, org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer, org.apache.flink.api.common.state.v2.StateDescriptor<T> stateDescriptor)	Create new state (v2) based on new state descriptor.
<T> org.apache.flink.util.function.ThrowingConsumer<StreamRecord<T>,Exception>	getRecordProcessor(int inputId)	Get the record processor that could process record from input, which is the only entry for async processing.
protected void	handleAsyncException(String message, Throwable exception)
boolean	isAsyncKeyOrderedProcessingEnabled()	Indicates whether this operator is enabling the async state.
void	newKeySelected(Object newKey)	A hook that will be invoked after a new key is selected.
void	postProcessElement()	A callback that will be triggered after an element finishes processElement.
Watermark	postProcessWatermark(Watermark watermark)	A hook that will be invoked after finishing advancing the watermark and right before the watermark being emitting downstream.
void	prepareSnapshotPreBarrier(long checkpointId)	This method is called when the operator should do a snapshot, before it emits its own checkpoint barrier.
Watermark	preProcessWatermark(Watermark watermark)	A hook that will be triggered when receiving a watermark.
void	preserveRecordOrderAndProcess(org.apache.flink.util.function.ThrowingRunnable<Exception> processing)	Check the order of same-key record, and then process the record.
protected void	processNonRecord(org.apache.flink.util.function.ThrowingRunnable<? extends Exception> triggerAction, org.apache.flink.util.function.ThrowingRunnable<? extends Exception> finalAction)	Process a non-record event.
void	processRecordAttributes(RecordAttributes recordAttributes)
void	processRecordAttributes1(RecordAttributes recordAttributes)
void	processRecordAttributes2(RecordAttributes recordAttributes)
void	processWatermark(WatermarkEvent watermark)
void	processWatermark(Watermark mark)	Process a watermark when receiving it.
void	processWatermark1(WatermarkEvent watermark)
void	processWatermark1Internal(WatermarkEvent watermark)
void	processWatermark2(WatermarkEvent watermark)
void	processWatermark2Internal(WatermarkEvent watermark)
void	processWatermarkInternal(WatermarkEvent watermark)
void	processWatermarkStatus(WatermarkStatus watermarkStatus)
protected void	processWatermarkStatus(WatermarkStatus watermarkStatus, int index)
protected void	reportOrForwardLatencyMarker(LatencyMarker marker)
<T> void	setAsyncKeyedContextElement(StreamRecord<T> record, org.apache.flink.api.java.functions.KeySelector<T,?> keySelector)	Set key context for async state processing.
void	setKeyContextElement1(StreamRecord record)
void	setKeyContextElement2(StreamRecord record)

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

getContainingTask, getExecutionConfig, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, open, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2, processWatermarkStatus1, processWatermarkStatus2, setCurrentKey, setMailboxExecutor, setProcessingTimeService, setup, snapshotState, snapshotState, useSplittableTimers

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

hasKeyContext

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

getOperatorAttributes

Field Detail

asyncExecutionController

protected AsyncExecutionController asyncExecutionController

currentProcessingContext

protected RecordContext currentProcessingContext

Act as a cache for setAsyncKeyedContextElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<T>, org.apache.flink.api.java.functions.KeySelector<T, ?>) and postProcessElement().

environment

protected Environment environment

declarationManager

protected DeclarationManager declarationManager

Constructor Detail

AbstractAsyncKeyOrderedStreamOperator

public AbstractAsyncKeyOrderedStreamOperator()

Method Detail

beforeInitializeStateHandler

public final void beforeInitializeStateHandler()

Initialize necessary state components for AbstractStreamOperator.

Overrides:

beforeInitializeStateHandler in class AbstractStreamOperator<OUT>

getKeySelectorForAsyncKeyedContext

protected abstract org.apache.flink.api.java.functions.KeySelector getKeySelectorForAsyncKeyedContext(int index)

Get the KeySelector for the given input index. The KeySelector is used to extract the key from the element being processed, used in key ordered processing.

Parameters:

index - the input index, start from 1.

createAsyncExecutionController

protected abstract AsyncExecutionController createAsyncExecutionController()

Create a new AsyncExecutionController for the operator.

handleAsyncException

protected void handleAsyncException(String message,
                                    Throwable exception)

isAsyncKeyOrderedProcessingEnabled

public boolean isAsyncKeyOrderedProcessingEnabled()

Description copied from class: AbstractStreamOperator

Indicates whether this operator is enabling the async state. Can be overridden by subclasses.

Specified by:

isAsyncKeyOrderedProcessingEnabled in interface AsyncKeyOrderedProcessing

Overrides:

isAsyncKeyOrderedProcessingEnabled in class AbstractStreamOperator<OUT>

Returns:

ture if async state processing is enabled.

getElementOrder

public ElementOrder getElementOrder()

Description copied from interface: AsyncKeyOrderedProcessingOperator

Get the ElementOrder of this operator.

Specified by:

getElementOrder in interface AsyncKeyOrderedProcessingOperator

setAsyncKeyedContextElement

public final <T> void setAsyncKeyedContextElement(StreamRecord<T> record,
                                                  org.apache.flink.api.java.functions.KeySelector<T,?> keySelector)
                                           throws Exception

Description copied from interface: AsyncKeyOrderedProcessingOperator

Set key context for async state processing.

Specified by:

setAsyncKeyedContextElement in interface AsyncKeyOrderedProcessingOperator

Type Parameters:

T - the type of the record.

Parameters:

record - the record.

keySelector - the key selector to select a key from record.

Throws:

Exception

newKeySelected

public void newKeySelected(Object newKey)

A hook that will be invoked after a new key is selected. It is not recommended to perform async state here. Only some synchronous logic is suggested.

Parameters:

newKey - the new key selected.

postProcessElement

public final void postProcessElement()

Description copied from interface: AsyncKeyOrderedProcessingOperator

A callback that will be triggered after an element finishes processElement.

Specified by:

postProcessElement in interface AsyncKeyOrderedProcessingOperator

preserveRecordOrderAndProcess

public final void preserveRecordOrderAndProcess(org.apache.flink.util.function.ThrowingRunnable<Exception> processing)

Description copied from interface: AsyncKeyOrderedProcessingOperator

Check the order of same-key record, and then process the record. Mainly used when the AsyncKeyOrderedProcessingOperator.getElementOrder() returns ElementOrder.RECORD_ORDER.

Specified by:

preserveRecordOrderAndProcess in interface AsyncKeyOrderedProcessingOperator

Parameters:

processing - the record processing logic.

asyncProcessWithKey

public <K> void asyncProcessWithKey(K key,
                                    org.apache.flink.util.function.ThrowingRunnable<Exception> processing)

Description copied from interface: AsyncKeyOrderedProcessingOperator

Asynchronously process a code with a key provided.

Specified by:

asyncProcessWithKey in interface AsyncKeyOrderedProcessingOperator

Type Parameters:

K - the type of key.

Parameters:

key - the specified key.

processing - the process logic.

getDeclarationManager

public final DeclarationManager getDeclarationManager()

Description copied from interface: AsyncKeyOrderedProcessingOperator

Get the declaration manager for user-logic declaring.

Specified by:

getDeclarationManager in interface AsyncKeyOrderedProcessingOperator

getRecordProcessor

public final <T> org.apache.flink.util.function.ThrowingConsumer<StreamRecord<T>,Exception> getRecordProcessor(int inputId)

Description copied from interface: AsyncKeyOrderedProcessing

Get the record processor that could process record from input, which is the only entry for async processing.

Specified by:

getRecordProcessor in interface AsyncKeyOrderedProcessing

Parameters:

inputId - the input identifier, start from 1. Borrow the design from org.apache.flink.streaming.api.operators.AbstractInput#inputId. This is only relevant if there is multiple inputs for the instance.

processNonRecord

protected void processNonRecord(@Nullable
                                org.apache.flink.util.function.ThrowingRunnable<? extends Exception> triggerAction,
                                @Nullable
                                org.apache.flink.util.function.ThrowingRunnable<? extends Exception> finalAction)

Process a non-record event. This method is used to process events that are not related to records, such as watermarks or latency markers. It is used to ensure that the async state processing is performed in the correct order. Subclasses could override this method to inject some async state processing logic.

Parameters:

triggerAction - the action that will be performed when the event is triggered.

finalAction - the action that will be performed when the event is finished considering the epoch control.

getOrCreateKeyedState

public <N,S extends org.apache.flink.api.common.state.v2.State,T> S getOrCreateKeyedState(@Nonnull
                                                                                                      N defaultNamespace,
                                                                                                      @Nonnull
                                                                                                      org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer,
                                                                                                      @Nonnull
                                                                                                      org.apache.flink.api.common.state.v2.StateDescriptor<T> stateDescriptor)
                                                                                               throws Exception

Create new state (v2) based on new state descriptor.

Throws:

Exception

prepareSnapshotPreBarrier

public void prepareSnapshotPreBarrier(long checkpointId)
                               throws Exception

Description copied from interface: StreamOperator

This method is called when the operator should do a snapshot, before it emits its own checkpoint barrier.

This method is intended not for any actual state persistence, but only for emitting some data before emitting the checkpoint barrier. Operators that maintain some small transient state that is inefficient to checkpoint (especially when it would need to be checkpointed in a re-scalable way) but can simply be sent downstream before the checkpoint. An example are opportunistic pre-aggregation operators, which have small the pre-aggregation state that is frequently flushed downstream.

Important: This method should not be used for any actual state snapshot logic, because it will inherently be within the synchronous part of the operator's checkpoint. If heavy work is done within this method, it will affect latency and downstream checkpoint alignments.

Specified by:

prepareSnapshotPreBarrier in interface StreamOperator<OUT>

Overrides:

prepareSnapshotPreBarrier in class AbstractStreamOperator<OUT>

Parameters:

checkpointId - The ID of the checkpoint.

Throws:

Exception - Throwing an exception here causes the operator to fail and go into recovery.

getInternalTimerService

public <K,N> InternalTimerService<N> getInternalTimerService(String name,
                                                                   org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer,
                                                                   Triggerable<K,N> triggerable)

Returns a InternalTimerService that can be used to query current processing time and event time and to set timers. An operator can have several timer services, where each has its own namespace serializer. Timer services are differentiated by the string key that is given when requesting them, if you call this method with the same key multiple times you will get the same timer service instance in subsequent requests.

Timers are always scoped to a key, the currently active key of a keyed stream operation. When a timer fires, this key will also be set as the currently active key.

Each timer has attached metadata, the namespace. Different timer services can have a different namespace type. If you don't need namespace differentiation you can use VoidNamespaceSerializer as the namespace serializer.

Overrides:

getInternalTimerService in class AbstractStreamOperator<OUT>

Type Parameters:

N - The type of the timer namespace.

Parameters:

name - The name of the requested timer service. If no service exists under the given name a new one will be created and returned.

namespaceSerializer - TypeSerializer for the timer namespace.

triggerable - The Triggerable that should be invoked when timers fire

setKeyContextElement1

public void setKeyContextElement1(StreamRecord record)
                           throws Exception

Specified by:

setKeyContextElement1 in interface StreamOperator<OUT>

Overrides:

setKeyContextElement1 in class AbstractStreamOperator<OUT>

Throws:

Exception

setKeyContextElement2

public void setKeyContextElement2(StreamRecord record)
                           throws Exception

Specified by:

setKeyContextElement2 in interface StreamOperator<OUT>

Overrides:

setKeyContextElement2 in class AbstractStreamOperator<OUT>

Throws:

Exception

getCurrentKey

public Object getCurrentKey()

Specified by:

getCurrentKey in interface KeyContext

Overrides:

getCurrentKey in class AbstractStreamOperator<OUT>

reportOrForwardLatencyMarker

protected void reportOrForwardLatencyMarker(LatencyMarker marker)

Overrides:

reportOrForwardLatencyMarker in class AbstractStreamOperator<OUT>

preProcessWatermark

public Watermark preProcessWatermark(Watermark watermark)
                              throws Exception

A hook that will be triggered when receiving a watermark. Some async state can safely go within this method. Return the watermark that should be normally processed.

Parameters:

watermark - the receiving watermark.

Returns:

the watermark that should be processed. Null if there is no need for following processing.

Throws:

Exception

postProcessWatermark

public Watermark postProcessWatermark(Watermark watermark)
                               throws Exception

A hook that will be invoked after finishing advancing the watermark and right before the watermark being emitting downstream. Here is a chance for customization of the emitting watermark. It is not recommended to perform async state here. Only some synchronous logic is suggested.

Parameters:

watermark - the advanced watermark.

Returns:

the watermark that should be emitted to downstream. Null if there is no need for following emitting.

Throws:

Exception

processWatermark

public final void processWatermark(Watermark mark)
                            throws Exception

Process a watermark when receiving it. Do not override this method since the async processing is difficult to write. Please override the hooks, see preProcessWatermark(Watermark) and postProcessWatermark(Watermark). The basic logic of processWatermark with hooks in sync form would be:

             Watermark watermark = preProcessWatermark(mark);
             if (watermark != null) {
                 super.processWatermark(watermark);
                 postProcessWatermark(watermark);
             }
 

Overrides:

processWatermark in class AbstractStreamOperator<OUT>

Throws:

Exception

processWatermarkStatus

public void processWatermarkStatus(WatermarkStatus watermarkStatus)
                            throws Exception

Overrides:

processWatermarkStatus in class AbstractStreamOperator<OUT>

Throws:

Exception

processWatermarkStatus

protected void processWatermarkStatus(WatermarkStatus watermarkStatus,
                                      int index)
                               throws Exception

Overrides:

processWatermarkStatus in class AbstractStreamOperator<OUT>

Throws:

Exception

processRecordAttributes

@Experimental
public void processRecordAttributes(RecordAttributes recordAttributes)
                             throws Exception

Overrides:

processRecordAttributes in class AbstractStreamOperator<OUT>

Throws:

Exception

processRecordAttributes1

@Experimental
public void processRecordAttributes1(RecordAttributes recordAttributes)

Overrides:

processRecordAttributes1 in class AbstractStreamOperator<OUT>

processRecordAttributes2

@Experimental
public void processRecordAttributes2(RecordAttributes recordAttributes)

Overrides:

processRecordAttributes2 in class AbstractStreamOperator<OUT>

processWatermarkInternal

public void processWatermarkInternal(WatermarkEvent watermark)
                              throws Exception

Throws:

Exception

processWatermark1Internal

public void processWatermark1Internal(WatermarkEvent watermark)
                               throws Exception

Throws:

Exception

processWatermark2Internal

public void processWatermark2Internal(WatermarkEvent watermark)
                               throws Exception

Throws:

Exception

processWatermark

public void processWatermark(WatermarkEvent watermark)
                      throws Exception

Overrides:

processWatermark in class AbstractStreamOperator<OUT>

Throws:

Exception

processWatermark1

public void processWatermark1(WatermarkEvent watermark)
                       throws Exception

Overrides:

processWatermark1 in class AbstractStreamOperator<OUT>

Throws:

Exception

processWatermark2

public void processWatermark2(WatermarkEvent watermark)
                       throws Exception

Overrides:

processWatermark2 in class AbstractStreamOperator<OUT>

Throws:

Exception

getAsyncKeyedStateBackend

public <K> AsyncKeyedStateBackend<K> getAsyncKeyedStateBackend()

drainStateRequests

public void drainStateRequests()

finish

public void finish()
            throws Exception

Description copied from interface: StreamOperator

This method is called at the end of data processing.

The method is expected to flush all remaining buffered data. Exceptions during this flushing of buffered data should be propagated, in order to cause the operation to be recognized as failed, because the last data items are not processed properly.

After this method is called, no more records can be produced for the downstream operators.

WARNING: It is not safe to use this method to commit any transactions or other side effects! You can use this method to flush any buffered data that can later on be committed e.g. in a CheckpointListener.notifyCheckpointComplete(long).

NOTE:This method does not need to close any resources. You should release external resources in the StreamOperator.close() method.

Specified by:

finish in interface StreamOperator<OUT>

Overrides:

finish in class AbstractStreamOperator<OUT>

Throws:

Exception - An exception in this method causes the operator to fail.

close

public void close()
           throws Exception

Description copied from interface: StreamOperator

This method is called at the very end of the operator's life, both in the case of a successful completion of the operation, and in the case of a failure and canceling.

This method is expected to make a thorough effort to release all resources that the operator has acquired.

NOTE:It can not emit any records! If you need to emit records at the end of processing, do so in the StreamOperator.finish() method.

Specified by:

close in interface StreamOperator<OUT>

Overrides:

close in class AbstractStreamOperator<OUT>

Throws:

Exception