org.apache.flink.runtime.checkpoint.channel

Class ChannelStateWriterImpl

java.lang.Object

org.apache.flink.runtime.checkpoint.channel.ChannelStateWriterImpl

All Implemented Interfaces:

Closeable, AutoCloseable, ChannelStateWriter

@Internal
 @ThreadSafe
public class ChannelStateWriterImpl
extends Object
implements ChannelStateWriter

ChannelStateWriter implemented using CheckpointStateOutputStreams. Internally, it has by default

• one stream per checkpoint; having multiple streams would mean more files written and more connections opened (and more latency on restore)
• one thread; having multiple threads means more connections, couples with the implementation and increases complexity

Thread-safety: this class is thread-safe when used with a thread-safe executor (e.g. default ChannelStateWriteRequestExecutorImpl.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.runtime.checkpoint.channel.ChannelStateWriter

ChannelStateWriter.ChannelStateWriteResult, ChannelStateWriter.NoOpChannelStateWriter

Field Summary

Fields inherited from interface org.apache.flink.runtime.checkpoint.channel.ChannelStateWriter

NO_OP, SEQUENCE_NUMBER_RESTORED, SEQUENCE_NUMBER_UNKNOWN

Constructor Summary

Constructors

Constructor and Description
ChannelStateWriterImpl(JobVertexID jobVertexID, String taskName, int subtaskIndex, CheckpointStorage checkpointStorage, ChannelStateWriteRequestExecutorFactory channelStateExecutorFactory, int maxSubtasksPerChannelStateFile) Creates a ChannelStateWriterImpl with DEFAULT_MAX_CHECKPOINTS as maxCheckpoints.

Method Summary

Modifier and Type	Method and Description
void	abort(long checkpointId, Throwable cause, boolean cleanup) Aborts the checkpoint and fails pending result for this checkpoint.
void	addInputData(long checkpointId, InputChannelInfo info, int startSeqNum, org.apache.flink.util.CloseableIterator<Buffer> iterator) Add in-flight buffers from the InputChannel.
void	addOutputData(long checkpointId, ResultSubpartitionInfo info, int startSeqNum, Buffer... data) Add in-flight buffers from the ResultSubpartition.
void	addOutputDataFuture(long checkpointId, ResultSubpartitionInfo info, int startSeqNum, CompletableFuture<List<Buffer>> dataFuture) Add in-flight bufferFuture from the ResultSubpartition.
void	close()
void	finishInput(long checkpointId) Finalize write of channel state data for the given checkpoint id.
void	finishOutput(long checkpointId) Finalize write of channel state data for the given checkpoint id.
ChannelStateWriter.ChannelStateWriteResult	getAndRemoveWriteResult(long checkpointId) Must be called after ChannelStateWriter.start(long, CheckpointOptions) once.
ChannelStateWriter.ChannelStateWriteResult	getWriteResult(long checkpointId)
void	start(long checkpointId, CheckpointOptions checkpointOptions) Initiate write of channel state for the given checkpoint id.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ChannelStateWriterImpl

public ChannelStateWriterImpl(JobVertexID jobVertexID,
                              String taskName,
                              int subtaskIndex,
                              CheckpointStorage checkpointStorage,
                              ChannelStateWriteRequestExecutorFactory channelStateExecutorFactory,
                              int maxSubtasksPerChannelStateFile)

Creates a ChannelStateWriterImpl with DEFAULT_MAX_CHECKPOINTS as maxCheckpoints.

Method Detail

start

public void start(long checkpointId,
                  CheckpointOptions checkpointOptions)

Description copied from interface: ChannelStateWriter

Initiate write of channel state for the given checkpoint id.

Specified by:

start in interface ChannelStateWriter

addInputData

public void addInputData(long checkpointId,
                         InputChannelInfo info,
                         int startSeqNum,
                         org.apache.flink.util.CloseableIterator<Buffer> iterator)

Description copied from interface: ChannelStateWriter

Add in-flight buffers from the InputChannel. Must be called after ChannelStateWriter.start(long,CheckpointOptions) and before ChannelStateWriter.finishInput(long). Buffers are recycled after they are written or exception occurs.

Specified by:

addInputData in interface ChannelStateWriter

startSeqNum - sequence number of the 1st passed buffer. It is intended to use for incremental snapshots. If no data is passed it is ignored.

iterator - zero or more data buffers ordered by their sequence numbers

See Also:

ChannelStateWriter.SEQUENCE_NUMBER_RESTORED, ChannelStateWriter.SEQUENCE_NUMBER_UNKNOWN

addOutputData

public void addOutputData(long checkpointId,
                          ResultSubpartitionInfo info,
                          int startSeqNum,
                          Buffer... data)

Description copied from interface: ChannelStateWriter

Add in-flight buffers from the ResultSubpartition. Must be called after ChannelStateWriter.start(long, org.apache.flink.runtime.checkpoint.CheckpointOptions) and before ChannelStateWriter.finishOutput(long). Buffers are recycled after they are written or exception occurs.

Specified by:

addOutputData in interface ChannelStateWriter

startSeqNum - sequence number of the 1st passed buffer. It is intended to use for incremental snapshots. If no data is passed it is ignored.

data - zero or more data buffers ordered by their sequence numbers

See Also:

ChannelStateWriter.SEQUENCE_NUMBER_RESTORED, ChannelStateWriter.SEQUENCE_NUMBER_UNKNOWN

addOutputDataFuture

public void addOutputDataFuture(long checkpointId,
                                ResultSubpartitionInfo info,
                                int startSeqNum,
                                CompletableFuture<List<Buffer>> dataFuture)
                         throws IllegalArgumentException

Description copied from interface: ChannelStateWriter

Add in-flight bufferFuture from the ResultSubpartition. Must be called after ChannelStateWriter.start(long, org.apache.flink.runtime.checkpoint.CheckpointOptions) and before ChannelStateWriter.finishOutput(long). Buffers are recycled after they are written or exception occurs.

The method will be called when the unaligned checkpoint is enabled and received an aligned barrier.

Specified by:

addOutputDataFuture in interface ChannelStateWriter

Throws:

IllegalArgumentException

finishInput

public void finishInput(long checkpointId)

Description copied from interface: ChannelStateWriter

Finalize write of channel state data for the given checkpoint id. Must be called after ChannelStateWriter.start(long, CheckpointOptions) and all of the input data of the given checkpoint added. When both ChannelStateWriter.finishInput(long) and ChannelStateWriter.finishOutput(long) were called the results can be (eventually) obtained using ChannelStateWriter.getAndRemoveWriteResult(long)

Specified by:

finishInput in interface ChannelStateWriter

finishOutput

public void finishOutput(long checkpointId)

Description copied from interface: ChannelStateWriter

Finalize write of channel state data for the given checkpoint id. Must be called after ChannelStateWriter.start(long, CheckpointOptions) and all of the output data of the given checkpoint added. When both ChannelStateWriter.finishInput(long) and ChannelStateWriter.finishOutput(long) were called the results can be (eventually) obtained using ChannelStateWriter.getAndRemoveWriteResult(long)

Specified by:

finishOutput in interface ChannelStateWriter

abort

public void abort(long checkpointId,
                  Throwable cause,
                  boolean cleanup)

Description copied from interface: ChannelStateWriter

Aborts the checkpoint and fails pending result for this checkpoint.

Specified by:

abort in interface ChannelStateWriter

cleanup - true if ChannelStateWriter.getAndRemoveWriteResult(long) is not supposed to be called afterwards.

getAndRemoveWriteResult

public ChannelStateWriter.ChannelStateWriteResult getAndRemoveWriteResult(long checkpointId)

Description copied from interface: ChannelStateWriter

Must be called after ChannelStateWriter.start(long, CheckpointOptions) once.

Specified by:

getAndRemoveWriteResult in interface ChannelStateWriter

getWriteResult

@VisibleForTesting
public ChannelStateWriter.ChannelStateWriteResult getWriteResult(long checkpointId)

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException