org.apache.flink.runtime.checkpoint

Class DefaultCompletedCheckpointStore<R extends ResourceVersion<R>>

java.lang.Object

org.apache.flink.runtime.checkpoint.AbstractCompleteCheckpointStore

org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore<R>

All Implemented Interfaces:

CompletedCheckpointStore

public class DefaultCompletedCheckpointStore<R extends ResourceVersion<R>>
extends AbstractCompleteCheckpointStore

Default implementation of CompletedCheckpointStore. Combined with different StateHandleStore, we could persist the completed checkpoints to various storage.

During recovery, the latest checkpoint is read from StateHandleStore. If there is more than one, only the latest one is used and older ones are discarded (even if the maximum number of retained checkpoints is greater than one).

If there is a network partition and multiple JobManagers run concurrent checkpoints for the same program, it is OK to take any valid successful checkpoint as long as the "history" of checkpoints is consistent. Currently, after recovery we start out with only a single checkpoint to circumvent those situations.

Constructor Summary

Constructors

Constructor and Description
DefaultCompletedCheckpointStore(int maxNumberOfCheckpointsToRetain, StateHandleStore<CompletedCheckpoint,R> stateHandleStore, CheckpointStoreUtil completedCheckpointStoreUtil, Collection<CompletedCheckpoint> completedCheckpoints, SharedStateRegistry sharedStateRegistry, Executor executor) Creates a DefaultCompletedCheckpointStore instance.

Method Summary

Modifier and Type	Method and Description
CompletedCheckpoint	addCheckpointAndSubsumeOldestOne(CompletedCheckpoint checkpoint, CheckpointsCleaner checkpointsCleaner, Runnable postCleanup) Synchronously writes the new checkpoints to state handle store and asynchronously removes older ones.
List<CompletedCheckpoint>	getAllCheckpoints() Returns all CompletedCheckpoint instances.
int	getMaxNumberOfRetainedCheckpoints() Returns the max number of retained checkpoints.
int	getNumberOfRetainedCheckpoints() Returns the current number of retained checkpoints.
boolean	requiresExternalizedCheckpoints() This method returns whether the completed checkpoint store requires checkpoints to be externalized.
void	shutdown(org.apache.flink.api.common.JobStatus jobStatus, CheckpointsCleaner checkpointsCleaner) Shuts down the store.

Methods inherited from class org.apache.flink.runtime.checkpoint.AbstractCompleteCheckpointStore

findLowest, getSharedStateRegistry, unregisterUnusedState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.checkpoint.CompletedCheckpointStore

getLatestCheckpoint, getLatestCheckpointId

Constructor Detail

DefaultCompletedCheckpointStore

public DefaultCompletedCheckpointStore(int maxNumberOfCheckpointsToRetain,
                                       StateHandleStore<CompletedCheckpoint,R> stateHandleStore,
                                       CheckpointStoreUtil completedCheckpointStoreUtil,
                                       Collection<CompletedCheckpoint> completedCheckpoints,
                                       SharedStateRegistry sharedStateRegistry,
                                       Executor executor)

Creates a DefaultCompletedCheckpointStore instance.

Parameters:

maxNumberOfCheckpointsToRetain - The maximum number of checkpoints to retain (at least 1). Adding more checkpoints than this results in older checkpoints being discarded. On recovery, we will only start with a single checkpoint.

stateHandleStore - Completed checkpoints in external store

completedCheckpointStoreUtil - utilities for completed checkpoint store

executor - to execute blocking calls

Method Detail

requiresExternalizedCheckpoints

public boolean requiresExternalizedCheckpoints()

Description copied from interface: CompletedCheckpointStore

This method returns whether the completed checkpoint store requires checkpoints to be externalized. Externalized checkpoints have their meta data persisted, which the checkpoint store can exploit (for example by simply pointing the persisted metadata).

Returns:

True, if the store requires that checkpoints are externalized before being added, false if the store stores the metadata itself.

addCheckpointAndSubsumeOldestOne

public CompletedCheckpoint addCheckpointAndSubsumeOldestOne(CompletedCheckpoint checkpoint,
                                                            CheckpointsCleaner checkpointsCleaner,
                                                            Runnable postCleanup)
                                                     throws Exception

Synchronously writes the new checkpoints to state handle store and asynchronously removes older ones.

Parameters:

checkpoint - Completed checkpoint to add.

Returns:

the subsumed oldest completed checkpoint if possible, return null if no checkpoint needs to be discarded on subsume.

Throws:

PossibleInconsistentStateException - if adding the checkpoint failed and leaving the system in a possibly inconsistent state, i.e. it's uncertain whether the checkpoint metadata was fully written to the underlying systems or not.

Exception

getAllCheckpoints

public List<CompletedCheckpoint> getAllCheckpoints()

Description copied from interface: CompletedCheckpointStore

Returns all CompletedCheckpoint instances.

Returns an empty list if no checkpoint has been added yet.

getNumberOfRetainedCheckpoints

public int getNumberOfRetainedCheckpoints()

Description copied from interface: CompletedCheckpointStore

Returns the current number of retained checkpoints.

getMaxNumberOfRetainedCheckpoints

public int getMaxNumberOfRetainedCheckpoints()

Description copied from interface: CompletedCheckpointStore

Returns the max number of retained checkpoints.

shutdown

public void shutdown(org.apache.flink.api.common.JobStatus jobStatus,
                     CheckpointsCleaner checkpointsCleaner)
              throws Exception

Description copied from interface: CompletedCheckpointStore

Shuts down the store.

The job status is forwarded and used to decide whether state should actually be discarded or kept. SharedStateRegistry.unregisterUnusedState(long) and CheckpointsCleaner.cleanSubsumedCheckpoints(long, java.util.Set<java.lang.Long>, java.lang.Runnable, java.util.concurrent.Executor) should be called here to subsume unused state.

Specified by:

shutdown in interface CompletedCheckpointStore

Overrides:

shutdown in class AbstractCompleteCheckpointStore

Parameters:

jobStatus - Job state on shut down

checkpointsCleaner - that will cleanup completed checkpoints if needed

Throws:

Exception