org.apache.flink.runtime.checkpoint.filemerging

Class AcrossCheckpointFileMergingSnapshotManager

java.lang.Object

org.apache.flink.runtime.checkpoint.filemerging.FileMergingSnapshotManagerBase

org.apache.flink.runtime.checkpoint.filemerging.AcrossCheckpointFileMergingSnapshotManager

All Implemented Interfaces:

Closeable, AutoCloseable, FileMergingSnapshotManager

public class AcrossCheckpointFileMergingSnapshotManager
extends FileMergingSnapshotManagerBase

A FileMergingSnapshotManager that merging files across checkpoints.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.runtime.checkpoint.filemerging.FileMergingSnapshotManagerBase

FileMergingSnapshotManagerBase.DirectoryHandleWithReferenceTrack

Nested classes/interfaces inherited from interface org.apache.flink.runtime.checkpoint.filemerging.FileMergingSnapshotManager

FileMergingSnapshotManager.SpaceStat, FileMergingSnapshotManager.SubtaskKey

Field Summary

Fields inherited from class org.apache.flink.runtime.checkpoint.filemerging.FileMergingSnapshotManagerBase

checkpointDir, filePoolType, fs, ioExecutor, lock, managedExclusiveStateDir, managedExclusiveStateDirHandle, maxPhysicalFileSize, maxSpaceAmplification, metricGroup, physicalFileDeleter, sharedStateDir, shouldSyncAfterClosingLogicalFile, spaceStat, taskOwnedStateDir, uploadedStates, writeBufferSize

Constructor Summary

Constructors

Constructor and Description
AcrossCheckpointFileMergingSnapshotManager(String id, long maxFileSize, PhysicalFilePool.Type filePoolType, float maxSpaceAmplification, Executor ioExecutor, org.apache.flink.metrics.MetricGroup metricGroup)

Method Summary

Modifier and Type	Method and Description
protected PhysicalFile	getOrCreatePhysicalFileForCheckpoint(FileMergingSnapshotManager.SubtaskKey subtaskKey, long checkpointID, CheckpointedStateScope scope) Get a reused physical file or create one.
protected void	returnPhysicalFileForNextReuse(FileMergingSnapshotManager.SubtaskKey subtaskKey, long checkpointId, PhysicalFile physicalFile) Try to return an existing physical file to the manager for next reuse.

Methods inherited from class org.apache.flink.runtime.checkpoint.filemerging.FileMergingSnapshotManagerBase

close, couldReusePreviousStateHandle, createCheckpointStateOutputStream, createLogicalFile, createPhysicalFile, createPhysicalPool, deletePhysicalFile, discardCheckpoint, discardSingleLogicalFile, generatePhysicalFilePath, getId, getLogicalFile, getManagedDir, getManagedDirStateHandle, initFileSystem, notifyCheckpointAborted, notifyCheckpointComplete, notifyCheckpointStart, notifyCheckpointSubsumed, registerSubtaskForSharedStates, restoreStateHandles, reusePreviousStateHandle, unregisterSubtask

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.checkpoint.filemerging.FileMergingSnapshotManager

isFileMergingHandle

Constructor Detail

AcrossCheckpointFileMergingSnapshotManager

public AcrossCheckpointFileMergingSnapshotManager(String id,
                                                  long maxFileSize,
                                                  PhysicalFilePool.Type filePoolType,
                                                  float maxSpaceAmplification,
                                                  Executor ioExecutor,
                                                  org.apache.flink.metrics.MetricGroup metricGroup)

Method Detail

getOrCreatePhysicalFileForCheckpoint

@Nonnull
protected PhysicalFile getOrCreatePhysicalFileForCheckpoint(FileMergingSnapshotManager.SubtaskKey subtaskKey,
                                                                     long checkpointID,
                                                                     CheckpointedStateScope scope)
                                                              throws IOException

Description copied from class: FileMergingSnapshotManagerBase

Get a reused physical file or create one. This will be called in checkpoint output stream creation logic.

Basic logic of file reusing: whenever a physical file is needed, this method is called with necessary information provided for acquiring a file. The file will not be reused until it is written and returned to the reused pool by calling FileMergingSnapshotManagerBase.returnPhysicalFileForNextReuse(org.apache.flink.runtime.checkpoint.filemerging.FileMergingSnapshotManager.SubtaskKey, long, org.apache.flink.runtime.checkpoint.filemerging.PhysicalFile).

Specified by:

getOrCreatePhysicalFileForCheckpoint in class FileMergingSnapshotManagerBase

Parameters:

subtaskKey - the subtask key for the caller

checkpointID - the checkpoint id

scope - checkpoint scope

Returns:

the requested physical file.

Throws:

IOException - thrown if anything goes wrong with file system.

returnPhysicalFileForNextReuse

protected void returnPhysicalFileForNextReuse(FileMergingSnapshotManager.SubtaskKey subtaskKey,
                                              long checkpointId,
                                              PhysicalFile physicalFile)
                                       throws IOException

Description copied from class: FileMergingSnapshotManagerBase

Try to return an existing physical file to the manager for next reuse. If this physical file is no longer needed (for reusing), it will be closed.

Basic logic of file reusing, see FileMergingSnapshotManagerBase.getOrCreatePhysicalFileForCheckpoint(org.apache.flink.runtime.checkpoint.filemerging.FileMergingSnapshotManager.SubtaskKey, long, org.apache.flink.runtime.state.CheckpointedStateScope).

Specified by:

returnPhysicalFileForNextReuse in class FileMergingSnapshotManagerBase

Parameters:

subtaskKey - the subtask key for the caller

checkpointId - in which checkpoint this physical file is requested.

physicalFile - the returning checkpoint

Throws:

IOException - thrown if anything goes wrong with file system.

See Also:

#getOrCreatePhysicalFileForCheckpoint(SubtaskKey, long, CheckpointedStateScope)