Package org.apache.flink.streaming.runtime.operators.windowing

Class WindowOperator<K,IN,ACC,OUT,W extends Window>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator<OUT,InternalWindowFunction<ACC,OUT,K,W>>

org.apache.flink.streaming.runtime.operators.windowing.WindowOperator<K,IN,ACC,OUT,W>

Type Parameters:

K - The type of key returned by the KeySelector.

IN - The type of the incoming elements.

OUT - The type of elements emitted by the InternalWindowFunction.

W - The type of Window that the WindowAssigner assigns.

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.state.CheckpointListener, Input<IN>, KeyContext, KeyContextHandler, OneInputStreamOperator<IN,OUT>, org.apache.flink.streaming.api.operators.OutputTypeConfigurable<OUT>, StreamOperator<OUT>, StreamOperatorStateHandler.CheckpointedStreamOperator, Triggerable<K,W>, UserFunctionProvider<InternalWindowFunction<ACC,OUT,K,W>>, YieldingOperator<OUT>

Direct Known Subclasses:

EvictingWindowOperator

@Internal
public class WindowOperator<K,IN,ACC,OUT,W extends Window>
extends AbstractUdfStreamOperator<OUT,InternalWindowFunction<ACC,OUT,K,W>>
implements OneInputStreamOperator<IN,OUT>, Triggerable<K,W>

An operator that implements the logic for windowing based on a WindowAssigner and Trigger.

When an element arrives it gets assigned a key using a KeySelector and it gets assigned to zero or more windows using a WindowAssigner. Based on this, the element is put into panes. A pane is the bucket of elements that have the same key and same Window. An element can be in multiple panes if it was assigned to multiple windows by the WindowAssigner.

Each pane gets its own instance of the provided Trigger. This trigger determines when the contents of the pane should be processed to emit results. When a trigger fires, the given InternalWindowFunction is invoked to produce the results that are emitted for the pane to which the Trigger belongs.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
class	WindowOperator.AbstractPerWindowStateStore	Base class for per-window KeyedStateStores.
class	WindowOperator.Context	Context is a utility for handling Trigger invocations.
class	WindowOperator.MergingWindowStateStore	Special WindowOperator.AbstractPerWindowStateStore that doesn't allow access to per-window state.
class	WindowOperator.PerWindowStateStore	Regular per-window state store for use with ProcessWindowFunction.
protected static class	WindowOperator.Timer<K,W extends Window>	Internal class for keeping track of in-flight timers.
class	WindowOperator.WindowContext	A utility class for handling ProcessWindowFunction invocations.

Field Summary

Fields

Modifier and Type	Field	Description
protected long	allowedLateness	The allowed lateness for elements.
protected InternalTimerService<W>	internalTimerService
protected org.apache.flink.api.common.typeutils.TypeSerializer<K>	keySerializer	For serializing the key in checkpoints.
protected org.apache.flink.util.OutputTag<IN>	lateDataOutputTag	OutputTag to use for late arriving events.
protected org.apache.flink.metrics.Counter	numLateRecordsDropped
protected WindowOperator.WindowContext	processContext
protected TimestampedCollector<OUT>	timestampedCollector	This is given to the InternalWindowFunction for emitting elements with a given timestamp.
protected WindowOperator.Context	triggerContext
protected WindowAssigner<? super IN,W>	windowAssigner
protected WindowAssigner.WindowAssignerContext	windowAssignerContext
protected org.apache.flink.api.common.typeutils.TypeSerializer<W>	windowSerializer	For serializing the window in checkpoints.

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator

userFunction

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

combinedWatermark, config, lastRecordAttributes1, lastRecordAttributes2, latencyStats, LOG, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager

Constructor Summary

Constructors

Constructor

Description

WindowOperator(WindowAssigner<? super IN,W> windowAssigner, org.apache.flink.api.common.typeutils.TypeSerializer<W> windowSerializer, org.apache.flink.api.java.functions.KeySelector<IN,K> keySelector, org.apache.flink.api.common.typeutils.TypeSerializer<K> keySerializer, org.apache.flink.api.common.state.StateDescriptor<? extends org.apache.flink.api.common.state.AppendingState<IN,ACC>,?> windowStateDescriptor, InternalWindowFunction<ACC,OUT,K,W> windowFunction, Trigger<? super IN,? super W> trigger, long allowedLateness, org.apache.flink.util.OutputTag<IN> lateDataOutputTag)

Creates a new WindowOperator based on the given policies and user functions.

Method Summary

Modifier and Type	Method	Description
void	close()	This method is called at the very end of the operator's life, both in the case of a successful completion of the operation, and in the case of a failure and canceling.
protected void	deleteCleanupTimer(W window)	Deletes the cleanup timer set for the contents of the provided window.
org.apache.flink.api.java.functions.KeySelector<IN,K>	getKeySelector()
protected MergingWindowSet<W>	getMergingWindowSet()	Retrieves the MergingWindowSet for the currently active key.
OperatorAttributes	getOperatorAttributes()	Called to get the OperatorAttributes of the operator.
org.apache.flink.api.common.state.StateDescriptor<? extends org.apache.flink.api.common.state.AppendingState<IN,ACC>,?>	getStateDescriptor()
Trigger<? super IN,? super W>	getTrigger()
WindowAssigner<? super IN,W>	getWindowAssigner()
protected boolean	isCleanupTime(W window, long time)	Returns true if the given time is the cleanup time for the given window.
protected boolean	isElementLate(StreamRecord<IN> element)	Decide if a record is currently late, based on current watermark and allowed lateness.
protected boolean	isWindowLate(W window)	Returns true if the watermark is after the end timestamp plus the allowed lateness of the given window.
void	onEventTime(InternalTimer<K,W> timer)	Invoked when an event-time timer fires.
void	onProcessingTime(InternalTimer<K,W> timer)	Invoked when a processing-time timer fires.
void	open()	This method is called immediately before any elements are processed, it should contain the operator's initialization logic, e.g. state initialization.
void	processElement(StreamRecord<IN> element)	Processes one element that arrived on this input of the MultipleInputStreamOperator.
protected void	registerCleanupTimer(W window)	Registers a timer to cleanup the content of the window.
void	setProcessingTimeService(ProcessingTimeService processingTimeService)
void	setup(StreamTask<?,?> containingTask, StreamConfig config, Output<StreamRecord<OUT>> output)
protected void	sideOutput(StreamRecord<IN> element)	Write skipped late arriving element to SideOutput.

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator

finish, getUserFunction, getUserFunctionParameters, initializeState, notifyCheckpointAborted, notifyCheckpointComplete, setOutputType, snapshotState

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

beforeInitializeStateHandler, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, isAsyncKeyOrderedProcessingEnabled, isUsingCustomRawKeyedState, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark, processWatermark, processWatermark1, processWatermark1, processWatermark2, processWatermark2, processWatermarkStatus, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, snapshotState, useSplittableTimers

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted, notifyCheckpointComplete

Methods inherited from interface org.apache.flink.streaming.api.operators.Input

processLatencyMarker, processRecordAttributes, processWatermark, processWatermark, processWatermarkStatus

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

getCurrentKey, setCurrentKey

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

hasKeyContext

Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

setKeyContextElement

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

finish, getMetricGroup, getOperatorID, initializeState, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState

Field Detail

windowAssigner

protected final WindowAssigner<? super IN,W extends Window> windowAssigner

keySerializer

protected final org.apache.flink.api.common.typeutils.TypeSerializer<K> keySerializer

For serializing the key in checkpoints.

windowSerializer

protected final org.apache.flink.api.common.typeutils.TypeSerializer<W extends Window> windowSerializer

For serializing the window in checkpoints.

allowedLateness

protected final long allowedLateness

The allowed lateness for elements. This is used for:

• Deciding if an element should be dropped from a window due to lateness.
• Clearing the state of a window if the system time passes the window.maxTimestamp + allowedLateness landmark.

lateDataOutputTag

protected final org.apache.flink.util.OutputTag<IN> lateDataOutputTag

OutputTag to use for late arriving events. Elements for which window.maxTimestamp + allowedLateness is smaller than the current watermark will be emitted to this.

numLateRecordsDropped

protected transient org.apache.flink.metrics.Counter numLateRecordsDropped

timestampedCollector

protected transient TimestampedCollector<OUT> timestampedCollector

This is given to the InternalWindowFunction for emitting elements with a given timestamp.

triggerContext

protected transient WindowOperator.Context triggerContext

processContext

protected transient WindowOperator.WindowContext processContext

windowAssignerContext

protected transient WindowAssigner.WindowAssignerContext windowAssignerContext

internalTimerService

protected transient InternalTimerService<W extends Window> internalTimerService

Constructor Detail

WindowOperator

public WindowOperator(WindowAssigner<? super IN,W> windowAssigner,
                      org.apache.flink.api.common.typeutils.TypeSerializer<W> windowSerializer,
                      org.apache.flink.api.java.functions.KeySelector<IN,K> keySelector,
                      org.apache.flink.api.common.typeutils.TypeSerializer<K> keySerializer,
                      org.apache.flink.api.common.state.StateDescriptor<? extends org.apache.flink.api.common.state.AppendingState<IN,ACC>,?> windowStateDescriptor,
                      InternalWindowFunction<ACC,OUT,K,W> windowFunction,
                      Trigger<? super IN,? super W> trigger,
                      long allowedLateness,
                      org.apache.flink.util.OutputTag<IN> lateDataOutputTag)

Creates a new WindowOperator based on the given policies and user functions.

Method Detail

setup

public void setup(StreamTask<?,?> containingTask,
                  StreamConfig config,
                  Output<StreamRecord<OUT>> output)

Overrides:

setup in class AbstractUdfStreamOperator<OUT,InternalWindowFunction<ACC,OUT,K,W extends Window>>

setProcessingTimeService

public void setProcessingTimeService(ProcessingTimeService processingTimeService)

Overrides:

setProcessingTimeService in class AbstractStreamOperator<OUT>

open

public void open()
          throws Exception

Description copied from class: AbstractStreamOperator

This method is called immediately before any elements are processed, it should contain the operator's initialization logic, e.g. state initialization.

The default implementation does nothing.

Specified by:

open in interface StreamOperator<K>

Overrides:

open in class AbstractUdfStreamOperator<OUT,InternalWindowFunction<ACC,OUT,K,W extends Window>>

Throws:

Exception - An exception in this method causes the operator to fail.

close

public void close()
           throws Exception

Description copied from interface: StreamOperator

This method is called at the very end of the operator's life, both in the case of a successful completion of the operation, and in the case of a failure and canceling.

This method is expected to make a thorough effort to release all resources that the operator has acquired.

NOTE:It can not emit any records! If you need to emit records at the end of processing, do so in the StreamOperator.finish() method.

Specified by:

close in interface StreamOperator<K>

Overrides:

close in class AbstractUdfStreamOperator<OUT,InternalWindowFunction<ACC,OUT,K,W extends Window>>

Throws:

Exception

processElement

public void processElement(StreamRecord<IN> element)
                    throws Exception

Description copied from interface: Input

Processes one element that arrived on this input of the MultipleInputStreamOperator. This method is guaranteed to not be called concurrently with other methods of the operator.

Specified by:

processElement in interface Input<K>

Throws:

Exception

onEventTime

public void onEventTime(InternalTimer<K,W> timer)
                 throws Exception

Description copied from interface: Triggerable

Invoked when an event-time timer fires.

Specified by:

onEventTime in interface Triggerable<K,IN>

Throws:

Exception

onProcessingTime

public void onProcessingTime(InternalTimer<K,W> timer)
                      throws Exception

Description copied from interface: Triggerable

Invoked when a processing-time timer fires.

Specified by:

onProcessingTime in interface Triggerable<K,IN>

Throws:

Exception

getOperatorAttributes

public OperatorAttributes getOperatorAttributes()

Description copied from interface: StreamOperator

Called to get the OperatorAttributes of the operator. If there is no defined attribute, a default OperatorAttributes is built.

Specified by:

getOperatorAttributes in interface StreamOperator<K>

Returns:

OperatorAttributes of the operator.

sideOutput

protected void sideOutput(StreamRecord<IN> element)

Write skipped late arriving element to SideOutput.

Parameters:

element - skipped late arriving element to side output

getMergingWindowSet

protected MergingWindowSet<W> getMergingWindowSet()
                                           throws Exception

Retrieves the MergingWindowSet for the currently active key. The caller must ensure that the correct key is set in the state backend.

The caller must also ensure to properly persist changes to state using MergingWindowSet.persist().

Throws:

Exception

isWindowLate

protected boolean isWindowLate(W window)

Returns true if the watermark is after the end timestamp plus the allowed lateness of the given window.

isElementLate

protected boolean isElementLate(StreamRecord<IN> element)

Decide if a record is currently late, based on current watermark and allowed lateness.

Parameters:

element - The element to check

Returns:

The element for which should be considered when sideoutputs

registerCleanupTimer

protected void registerCleanupTimer(W window)

Registers a timer to cleanup the content of the window.

Parameters:

window - the window whose state to discard

deleteCleanupTimer

protected void deleteCleanupTimer(W window)

Deletes the cleanup timer set for the contents of the provided window.

Parameters:

window - the window whose state to discard

isCleanupTime

protected final boolean isCleanupTime(W window,
                                      long time)

Returns true if the given time is the cleanup time for the given window.

getTrigger

@VisibleForTesting
public Trigger<? super IN,? super W> getTrigger()

getKeySelector

@VisibleForTesting
public org.apache.flink.api.java.functions.KeySelector<IN,K> getKeySelector()

getWindowAssigner

@VisibleForTesting
public WindowAssigner<? super IN,W> getWindowAssigner()

getStateDescriptor

@VisibleForTesting
public org.apache.flink.api.common.state.StateDescriptor<? extends org.apache.flink.api.common.state.AppendingState<IN,ACC>,?> getStateDescriptor()