org.apache.flink.streaming.api.functions.source

Interface SourceFunction<T>

Type Parameters:

T - The type of the elements produced by this source.

All Superinterfaces:

Function, Serializable

All Known Subinterfaces:

EventTimeSourceFunction<T>, ParallelSourceFunction<OUT>

All Known Implementing Classes:

ConnectorSource, FileMonitoringFunction, FileSourceFunction, FromElementsFunction, FromIteratorFunction, FromSplittableIteratorFunction, MessageAcknowledingSourceBase, RichEventTimeSourceFunction, RichParallelSourceFunction, RichSourceFunction, SocketTextStreamFunction, StatefulSequenceSource

public interface SourceFunction<T>
extends Function, Serializable

Base interface for all stream data sources in Flink. The contract of a stream source is the following: When the source should start emitting elements the run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T>) method is called with a Collector that can be used for emitting elements. The run method can run for as long as necessary. The source must, however, react to an invocation of cancel() by breaking out of its main loop.

Note about checkpointed sources
Sources that also implement the Checkpointed interface must ensure that state checkpointing, updating of internal state and emission of elements are not done concurrently. This is achieved by using the provided checkpointing lock object to protect update of state and emission of elements in a synchronized block.

This is the basic pattern one should follow when implementing a (checkpointed) source:

public class ExampleSource<T> implements SourceFunction<T>, Checkpointed<Long> {
      private long count = 0L;
      private volatile boolean isRunning = true;

      {@literal @}Override
      public void run(SourceContext<T> ctx) {
          while (isRunning && count < 1000) {
              synchronized (ctx.getCheckpointLock()) {
                  ctx.collect(count);
                  count++;
              }
          }
      }

      {@literal @}Override
      public void cancel() {
          isRunning = false;
      }

      {@literal @}Override
      public Long snapshotState(long checkpointId, long checkpointTimestamp) { return count; }

      {@literal @}Override
      public void restoreState(Long state) { this.count = state; }
 }
 

Note about element timestamps and watermarks:
Sources must only manually emit watermarks when they implement EventTimeSourceFunction. Otherwise, elements automatically get the current timestamp assigned at ingress and the system automatically emits watermarks.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static interface	SourceFunction.SourceContext<T> Interface that source functions use to communicate with the outside world.

Method Summary

Methods

Modifier and Type	Method and Description
void	cancel() Cancels the source.
void	run(SourceFunction.SourceContext<T> ctx) Starts the source.

Method Detail

run

void run(SourceFunction.SourceContext<T> ctx)
         throws Exception

Starts the source. You can use the Collector parameter to emit elements. Sources that implement Checkpointed must lock on the checkpoint lock (using a synchronized block) before updating internal state and/or emitting elements. Also, the update of state and emission of elements must happen in the same synchronized block.

Parameters:

ctx - The context for interaction with the outside world.

Throws:

Exception

cancel

void cancel()

Cancels the source. Most sources will have a while loop inside the run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T>) method. You need to ensure that the source will break out of this loop. This can be achieved by having a volatile field "isRunning" that is checked in the loop and that is set to false in this method.