org.apache.flink.runtime.state.filesystem

Class FsMergingCheckpointStorageLocation

java.lang.Object

org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory

org.apache.flink.runtime.state.filesystem.FsCheckpointStorageLocation

org.apache.flink.runtime.state.filesystem.FsMergingCheckpointStorageLocation

All Implemented Interfaces:

CheckpointStorageLocation, CheckpointStreamFactory

public class FsMergingCheckpointStorageLocation
extends FsCheckpointStorageLocation

An implementation of checkpoint storage location for file merging checkpoints.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory

FsCheckpointStreamFactory.FsCheckpointStateOutputStream

Field Summary

Fields inherited from class org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory

MAX_FILE_STATE_THRESHOLD

Constructor Summary

Constructors

Constructor and Description

FsMergingCheckpointStorageLocation(FileMergingSnapshotManager.SubtaskKey subtaskKey, org.apache.flink.core.fs.FileSystem fileSystem, org.apache.flink.core.fs.Path checkpointDir, org.apache.flink.core.fs.Path sharedStateDir, org.apache.flink.core.fs.Path taskOwnedStateDir, CheckpointStorageLocationReference reference, int fileStateSizeThreshold, int writeBufferSize, FileMergingSnapshotManager fileMergingSnapshotManager, long checkpointId)

Method Summary

Modifier and Type	Method and Description
boolean	canFastDuplicate(StreamStateHandle stateHandle, CheckpointedStateScope scope) Tells if we can duplicate the given StreamStateHandle into the path corresponding to the given CheckpointedStateScope.
boolean	couldReuseStateHandle(StreamStateHandle stateHandle) A pre-check hook before the checkpoint writer want to reuse a state handle, if this returns false, it is not recommended for the writer to rewrite the state file considering the space amplification.
FileMergingCheckpointStateOutputStream	createCheckpointStateOutputStream(CheckpointedStateScope scope) Creates an new CheckpointStateOutputStream.
List<StreamStateHandle>	duplicate(List<StreamStateHandle> stateHandles, CheckpointedStateScope scope) Duplicates StreamStateHandle into the path corresponding to * the given CheckpointedStateScope.
DirectoryStreamStateHandle	getExclusiveStateHandle()
DirectoryStreamStateHandle	getSharedStateHandle()
void	reusePreviousStateHandle(Collection<? extends StreamStateHandle> previousHandle) A callback method when some previous handle is reused.
CheckpointStreamFactory	toNonFileMerging()
String	toString()

Methods inherited from class org.apache.flink.runtime.state.filesystem.FsCheckpointStorageLocation

createMetadataOutputStream, disposeOnFailure, getCheckpointDirectory, getLocationReference, getMetadataFilePath, getSharedStateDirectory, getTaskOwnedStateDirectory

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

FsMergingCheckpointStorageLocation

public FsMergingCheckpointStorageLocation(FileMergingSnapshotManager.SubtaskKey subtaskKey,
                                          org.apache.flink.core.fs.FileSystem fileSystem,
                                          org.apache.flink.core.fs.Path checkpointDir,
                                          org.apache.flink.core.fs.Path sharedStateDir,
                                          org.apache.flink.core.fs.Path taskOwnedStateDir,
                                          CheckpointStorageLocationReference reference,
                                          int fileStateSizeThreshold,
                                          int writeBufferSize,
                                          FileMergingSnapshotManager fileMergingSnapshotManager,
                                          long checkpointId)

Method Detail

toNonFileMerging

public CheckpointStreamFactory toNonFileMerging()

getExclusiveStateHandle

public DirectoryStreamStateHandle getExclusiveStateHandle()

getSharedStateHandle

public DirectoryStreamStateHandle getSharedStateHandle()

canFastDuplicate

public boolean canFastDuplicate(StreamStateHandle stateHandle,
                                CheckpointedStateScope scope)
                         throws IOException

Description copied from interface: CheckpointStreamFactory

Tells if we can duplicate the given StreamStateHandle into the path corresponding to the given CheckpointedStateScope.

This should be a rather cheap operation, preferably not involving any remote accesses.

Specified by:

canFastDuplicate in interface CheckpointStreamFactory

Overrides:

canFastDuplicate in class FsCheckpointStreamFactory

Parameters:

stateHandle - The handle to duplicate

scope - Scope determining the location to duplicate into

Returns:

true, if we can perform the duplication

Throws:

IOException

duplicate

public List<StreamStateHandle> duplicate(List<StreamStateHandle> stateHandles,
                                         CheckpointedStateScope scope)
                                  throws IOException

Description copied from interface: CheckpointStreamFactory

Duplicates StreamStateHandle into the path corresponding to * the given CheckpointedStateScope.

You should first check if you can duplicate with CheckpointStreamFactory.canFastDuplicate(StreamStateHandle, CheckpointedStateScope).

Specified by:

duplicate in interface CheckpointStreamFactory

Overrides:

duplicate in class FsCheckpointStreamFactory

Parameters:

stateHandles - The handles to duplicate

scope - Scope determining the location to duplicate into

Returns:

The duplicated handle

Throws:

IOException

createCheckpointStateOutputStream

public FileMergingCheckpointStateOutputStream createCheckpointStateOutputStream(CheckpointedStateScope scope)
                                                                         throws IOException

Description copied from interface: CheckpointStreamFactory

Creates an new CheckpointStateOutputStream. When the stream is closed, it returns a state handle that can retrieve the state back.

Specified by:

createCheckpointStateOutputStream in interface CheckpointStreamFactory

Overrides:

createCheckpointStateOutputStream in class FsCheckpointStreamFactory

Parameters:

scope - The state's scope, whether it is exclusive or shared.

Returns:

An output stream that writes state for the given checkpoint.

Throws:

IOException - Exceptions may occur while creating the stream and should be forwarded.

reusePreviousStateHandle

public void reusePreviousStateHandle(Collection<? extends StreamStateHandle> previousHandle)

Description copied from interface: CheckpointStreamFactory

A callback method when some previous handle is reused. It is needed by the file merging mechanism (FLIP-306) which will manage the life cycle of underlying files by file-reusing information.

Parameters:

previousHandle - the previous handles that will be reused.

couldReuseStateHandle

public boolean couldReuseStateHandle(StreamStateHandle stateHandle)

Description copied from interface: CheckpointStreamFactory

A pre-check hook before the checkpoint writer want to reuse a state handle, if this returns false, it is not recommended for the writer to rewrite the state file considering the space amplification.

Parameters:

stateHandle - the handle to be reused.

Returns:

true if it can be reused.

toString

public String toString()

Overrides:

toString in class FsCheckpointStorageLocation