software.amazon.kinesis.connectors.flink.util

Class WatermarkTracker

java.lang.Object

software.amazon.kinesis.connectors.flink.util.WatermarkTracker

All Implemented Interfaces:

Closeable, Serializable, AutoCloseable

Direct Known Subclasses:

JobManagerWatermarkTracker

@PublicEvolving
public abstract class WatermarkTracker
extends Object
implements Closeable, Serializable

The watermark tracker is responsible for aggregating watermarks across distributed operators.

It can be used for sub tasks of a single Flink source as well as multiple heterogeneous sources or other operators.

The class essentially functions like a distributed hash table that enclosing operators can use to adopt their processing / IO rates.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	WatermarkTracker.WatermarkState Watermark state.

Field Summary

Fields

Modifier and Type	Field and Description
static long	DEFAULT_UPDATE_TIMEOUT_MILLIS

Constructor Summary

Constructors

Constructor and Description
WatermarkTracker()

Method Summary

Modifier and Type	Method and Description
void	close()
protected long	getCurrentTime()
protected String	getSubtaskId()
abstract long	getUpdateTimeoutCount()
protected long	getUpdateTimeoutMillis()
void	open(org.apache.flink.api.common.functions.RuntimeContext context)
void	setUpdateTimeoutMillis(long updateTimeoutMillis) Subtasks that have not provided a watermark update within the configured interval will be considered idle and excluded from target watermark calculation.
abstract long	updateWatermark(long localWatermark) Set the current watermark of the owning subtask and return the global low watermark based on the current state snapshot.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_UPDATE_TIMEOUT_MILLIS

public static final long DEFAULT_UPDATE_TIMEOUT_MILLIS

See Also:

Constant Field Values

Constructor Detail

WatermarkTracker

public WatermarkTracker()

Method Detail

getSubtaskId

protected String getSubtaskId()

getUpdateTimeoutMillis

protected long getUpdateTimeoutMillis()

getUpdateTimeoutCount

public abstract long getUpdateTimeoutCount()

setUpdateTimeoutMillis

public void setUpdateTimeoutMillis(long updateTimeoutMillis)

Subtasks that have not provided a watermark update within the configured interval will be considered idle and excluded from target watermark calculation.

Parameters:

updateTimeoutMillis -

updateWatermark

public abstract long updateWatermark(long localWatermark)

Set the current watermark of the owning subtask and return the global low watermark based on the current state snapshot. Periodically called by the enclosing consumer instance, which is responsible for any timer management etc.

Parameters:

localWatermark -

Returns:

getCurrentTime

protected long getCurrentTime()

open

public void open(org.apache.flink.api.common.functions.RuntimeContext context)

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable