Package org.apache.flink.streaming.api.operators.source

Class SourceOutputWithWatermarks<T>

java.lang.Object

org.apache.flink.streaming.api.operators.source.SourceOutputWithWatermarks<T>

Type Parameters:

T - The type of emitted records.

All Implemented Interfaces:

org.apache.flink.api.common.eventtime.WatermarkOutput, org.apache.flink.api.connector.source.SourceOutput<T>

@Internal
public class SourceOutputWithWatermarks<T>
extends Object
implements org.apache.flink.api.connector.source.SourceOutput<T>

Implementation of the SourceOutput. The records emitted to this output are pushed into a given PushingAsyncDataInput.DataOutput. The watermarks are pushed into the same output, or into a separate WatermarkOutput, if one is provided.

Periodic Watermarks

This output does not implement automatic periodic watermark emission. The method emitPeriodicWatermark() needs to be called periodically.

Note on Performance Considerations

The methods SourceOutput.collect(Object) and SourceOutput.collect(Object, long) are highly performance-critical (part of the hot loop). To make the code as JIT friendly as possible, we want to have only a single implementation of these two methods, across all classes. That way, the JIT compiler can de-virtualize (and inline) them better.

Currently, we have one implementation of these methods for the case where we don't need watermarks (see class NoOpTimestampsAndWatermarks) and one for the case where we do (this class). When the JVM is dedicated to a single job (or type of job) only one of these classes will be loaded. In mixed job setups, we still have a bimorphic method (rather than a poly/-/mega-morphic method).

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	SourceOutputWithWatermarks(PushingAsyncDataInput.DataOutput<T> recordsOutput, org.apache.flink.api.common.eventtime.WatermarkOutput onEventWatermarkOutput, org.apache.flink.api.common.eventtime.WatermarkOutput periodicWatermarkOutput, org.apache.flink.api.common.eventtime.TimestampAssigner<T> timestampAssigner, org.apache.flink.api.common.eventtime.WatermarkGenerator<T> watermarkGenerator)	Creates a new SourceOutputWithWatermarks that emits records to the given DataOutput and watermarks to the (possibly different) WatermarkOutput.

Method Summary

Modifier and Type	Method	Description
void	collect(T record)
void	collect(T record, long timestamp)
static <E> SourceOutputWithWatermarks<E>	createWithSeparateOutputs(PushingAsyncDataInput.DataOutput<E> recordsOutput, org.apache.flink.api.common.eventtime.WatermarkOutput onEventWatermarkOutput, org.apache.flink.api.common.eventtime.WatermarkOutput periodicWatermarkOutput, org.apache.flink.api.common.eventtime.TimestampAssigner<E> timestampAssigner, org.apache.flink.api.common.eventtime.WatermarkGenerator<E> watermarkGenerator)	Creates a new SourceOutputWithWatermarks that emits records to the given DataOutput and watermarks to the different WatermarkOutputs.
void	emitPeriodicWatermark()
void	emitWatermark(org.apache.flink.api.common.eventtime.Watermark watermark)
void	markActive()
void	markIdle()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SourceOutputWithWatermarks

protected SourceOutputWithWatermarks(PushingAsyncDataInput.DataOutput<T> recordsOutput,
                                     org.apache.flink.api.common.eventtime.WatermarkOutput onEventWatermarkOutput,
                                     org.apache.flink.api.common.eventtime.WatermarkOutput periodicWatermarkOutput,
                                     org.apache.flink.api.common.eventtime.TimestampAssigner<T> timestampAssigner,
                                     org.apache.flink.api.common.eventtime.WatermarkGenerator<T> watermarkGenerator)

Creates a new SourceOutputWithWatermarks that emits records to the given DataOutput and watermarks to the (possibly different) WatermarkOutput.

Method Detail

collect

public final void collect(T record)

Specified by:

collect in interface org.apache.flink.api.connector.source.SourceOutput<T>

collect

public final void collect(T record,
                          long timestamp)

Specified by:

collect in interface org.apache.flink.api.connector.source.SourceOutput<T>

emitWatermark

public final void emitWatermark(org.apache.flink.api.common.eventtime.Watermark watermark)

Specified by:

emitWatermark in interface org.apache.flink.api.common.eventtime.WatermarkOutput

markIdle

public final void markIdle()

Specified by:

markIdle in interface org.apache.flink.api.common.eventtime.WatermarkOutput

markActive

public void markActive()

Specified by:

markActive in interface org.apache.flink.api.common.eventtime.WatermarkOutput

emitPeriodicWatermark

public final void emitPeriodicWatermark()

createWithSeparateOutputs

public static <E> SourceOutputWithWatermarks<E> createWithSeparateOutputs(PushingAsyncDataInput.DataOutput<E> recordsOutput,
                                                                          org.apache.flink.api.common.eventtime.WatermarkOutput onEventWatermarkOutput,
                                                                          org.apache.flink.api.common.eventtime.WatermarkOutput periodicWatermarkOutput,
                                                                          org.apache.flink.api.common.eventtime.TimestampAssigner<E> timestampAssigner,
                                                                          org.apache.flink.api.common.eventtime.WatermarkGenerator<E> watermarkGenerator)

Creates a new SourceOutputWithWatermarks that emits records to the given DataOutput and watermarks to the different WatermarkOutputs.