Type - The type of the messages created by the source.Id - The type of the IDs that are used for acknowledging elements.public abstract class MessageAcknowledingSourceBase<Type,Id> extends RichSourceFunction<Type> implements Checkpointed<SerializedCheckpointData[]>, CheckpointNotifier
The mechanism for this source assumes that messages are identified by a unique ID. When messages are taken from the message queue, the message must not be dropped immediately, but must be retained until acknowledged. Messages that are not acknowledged within a certain time interval will be served again (to a different connection, established by the recovered source).
Note that this source can give no guarantees about message order in teh case of failures, because messages that were retrieved but not yet acknowledged will be returned later again, after a set of messages that was not retrieved before the failure.
Internally, this source gathers the IDs of elements it emits. Per checkpoint, the IDs are stored and acknowledged when the checkpoint is complete. That way, no message is acknowledged unless it is certain that it has been successfully processed throughout the topology and the updates to any state caused by that message are persistent.
All messages that are emitted and successfully processed by the streaming program will eventually be acknowledged. In corner cases, the source may acknowledge certain IDs multiple times, if a failure occurs while acknowledging.
A typical way to use this base in a source function is by implementing a run() method as follows:
public void run(SourceContext<Type> ctx) throws Exception {
while (running) {
Message msg = queue.retrieve();
synchronized (ctx.getCheckpointLock()) {
ctx.collect(msg.getMessageData());
addId(msg.getMessageId());
}
}
}
SourceFunction.SourceContext<T>| Modifier | Constructor and Description |
|---|---|
protected |
MessageAcknowledingSourceBase(Class<Id> idClass)
Creates a new MessageAcknowledingSourceBase for IDs of teh given type.
|
protected |
MessageAcknowledingSourceBase(TypeInformation<Id> idTypeInfo)
Creates a new MessageAcknowledingSourceBase for IDs of teh given type.
|
| Modifier and Type | Method and Description |
|---|---|
protected abstract void |
acknowledgeIDs(List<Id> ids)
This method must be implemented to acknowledge the given set of IDs back to the message queue.
|
protected void |
addId(Id id)
Adds an ID to be stored with the current checkpoint.
|
void |
close() |
void |
notifyCheckpointComplete(long checkpointId)
This method is called as a notification once a distributed checkpoint has been completed.
|
void |
open(Configuration parameters) |
void |
restoreState(SerializedCheckpointData[] state)
Restores the state of the function or operator to that of a previous checkpoint.
|
SerializedCheckpointData[] |
snapshotState(long checkpointId,
long checkpointTimestamp)
Gets the current state of the function of operator.
|
getIterationRuntimeContext, getRuntimeContext, setRuntimeContextclone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitcancel, runprotected MessageAcknowledingSourceBase(Class<Id> idClass)
idClass - The class of the message ID type, used to create a serializer for the message IDs.protected MessageAcknowledingSourceBase(TypeInformation<Id> idTypeInfo)
idTypeInfo - The type information of the message ID type, used to create a serializer for the message IDs.public void open(Configuration parameters) throws Exception
open in interface RichFunctionopen in class AbstractRichFunctionExceptionpublic void close()
throws Exception
close in interface RichFunctionclose in class AbstractRichFunctionExceptionprotected abstract void acknowledgeIDs(List<Id> ids)
ids - The list od IDs to acknowledge.protected void addId(Id id)
id - The ID to add.public SerializedCheckpointData[] snapshotState(long checkpointId, long checkpointTimestamp) throws Exception
CheckpointedsnapshotState in interface Checkpointed<SerializedCheckpointData[]>checkpointId - The ID of the checkpoint.checkpointTimestamp - The timestamp of the checkpoint, as derived by
System.currentTimeMillis() on the JobManager.Exception - Thrown if the creation of the state object failed. This causes the
checkpoint to fail. The system may decide to fail the operation (and trigger
recovery), or to discard this checkpoint attempt and to continue running
and to try again with the next checkpoint attempt.public void restoreState(SerializedCheckpointData[] state) throws Exception
CheckpointedrestoreState in interface Checkpointed<SerializedCheckpointData[]>state - The state to be restored.Exceptionpublic void notifyCheckpointComplete(long checkpointId)
throws Exception
CheckpointNotifiernotifyCheckpointComplete in interface CheckpointNotifiercheckpointId - The ID of the checkpoint that has been completed.ExceptionCopyright © 2014–2016 The Apache Software Foundation. All rights reserved.