org.apache.flink.runtime.checkpoint.filemerging

Class WithinCheckpointFileMergingSnapshotManager

java.lang.Object

org.apache.flink.runtime.checkpoint.filemerging.FileMergingSnapshotManagerBase

org.apache.flink.runtime.checkpoint.filemerging.WithinCheckpointFileMergingSnapshotManager

All Implemented Interfaces:

Closeable, AutoCloseable, FileMergingSnapshotManager

public class WithinCheckpointFileMergingSnapshotManager
extends FileMergingSnapshotManagerBase

A FileMergingSnapshotManager that merging files within a checkpoint.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.runtime.checkpoint.filemerging.FileMergingSnapshotManagerBase

FileMergingSnapshotManagerBase.DirectoryHandleWithReferenceTrack

Nested classes/interfaces inherited from interface org.apache.flink.runtime.checkpoint.filemerging.FileMergingSnapshotManager

FileMergingSnapshotManager.SpaceStat, FileMergingSnapshotManager.SubtaskKey

Field Summary

Fields inherited from class org.apache.flink.runtime.checkpoint.filemerging.FileMergingSnapshotManagerBase

checkpointDir, filePoolType, fs, ioExecutor, lock, managedExclusiveStateDir, managedExclusiveStateDirHandle, maxPhysicalFileSize, maxSpaceAmplification, metricGroup, physicalFileDeleter, sharedStateDir, shouldSyncAfterClosingLogicalFile, spaceStat, taskOwnedStateDir, uploadedStates, writeBufferSize

Constructor Summary

Constructors

Constructor and Description
WithinCheckpointFileMergingSnapshotManager(String id, long maxFileSize, PhysicalFilePool.Type filePoolType, float maxSpaceAmplification, Executor ioExecutor, org.apache.flink.metrics.MetricGroup metricGroup)

Method Summary

Modifier and Type	Method and Description
void	close()
protected void	discardCheckpoint(long checkpointId) The callback which will be triggered when all subtasks discarded (aborted or subsumed).
protected PhysicalFile	getOrCreatePhysicalFileForCheckpoint(FileMergingSnapshotManager.SubtaskKey subtaskKey, long checkpointId, CheckpointedStateScope scope) Get a reused physical file or create one.
void	notifyCheckpointAborted(FileMergingSnapshotManager.SubtaskKey subtaskKey, long checkpointId) This method is called as a notification once a distributed checkpoint has been aborted.
void	notifyCheckpointComplete(FileMergingSnapshotManager.SubtaskKey subtaskKey, long checkpointId) Notifies the manager that the checkpoint with the given checkpointId completed and was committed.
protected void	returnPhysicalFileForNextReuse(FileMergingSnapshotManager.SubtaskKey subtaskKey, long checkpointId, PhysicalFile physicalFile) Try to return an existing physical file to the manager for next reuse.

Methods inherited from class org.apache.flink.runtime.checkpoint.filemerging.FileMergingSnapshotManagerBase

couldReusePreviousStateHandle, createCheckpointStateOutputStream, createLogicalFile, createPhysicalFile, createPhysicalPool, deletePhysicalFile, discardSingleLogicalFile, generatePhysicalFilePath, getId, getLogicalFile, getManagedDir, getManagedDirStateHandle, initFileSystem, notifyCheckpointStart, notifyCheckpointSubsumed, registerSubtaskForSharedStates, restoreStateHandles, reusePreviousStateHandle, unregisterSubtask

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.checkpoint.filemerging.FileMergingSnapshotManager

isFileMergingHandle

Constructor Detail

WithinCheckpointFileMergingSnapshotManager

public WithinCheckpointFileMergingSnapshotManager(String id,
                                                  long maxFileSize,
                                                  PhysicalFilePool.Type filePoolType,
                                                  float maxSpaceAmplification,
                                                  Executor ioExecutor,
                                                  org.apache.flink.metrics.MetricGroup metricGroup)

Method Detail

notifyCheckpointComplete

public void notifyCheckpointComplete(FileMergingSnapshotManager.SubtaskKey subtaskKey,
                                     long checkpointId)
                              throws Exception

Description copied from interface: FileMergingSnapshotManager

Notifies the manager that the checkpoint with the given checkpointId completed and was committed.

Specified by:

notifyCheckpointComplete in interface FileMergingSnapshotManager

Overrides:

notifyCheckpointComplete in class FileMergingSnapshotManagerBase

Parameters:

subtaskKey - the subtask key identifying the subtask.

checkpointId - The ID of the checkpoint that has been completed.

Throws:

Exception - thrown if anything goes wrong with the listener.

notifyCheckpointAborted

public void notifyCheckpointAborted(FileMergingSnapshotManager.SubtaskKey subtaskKey,
                                    long checkpointId)
                             throws Exception

Description copied from interface: FileMergingSnapshotManager

This method is called as a notification once a distributed checkpoint has been aborted.

Specified by:

notifyCheckpointAborted in interface FileMergingSnapshotManager

Overrides:

notifyCheckpointAborted in class FileMergingSnapshotManagerBase

Parameters:

subtaskKey - the subtask key identifying the subtask.

checkpointId - The ID of the checkpoint that has been completed.

Throws:

Exception - thrown if anything goes wrong with the listener.

getOrCreatePhysicalFileForCheckpoint

@Nonnull
protected PhysicalFile getOrCreatePhysicalFileForCheckpoint(FileMergingSnapshotManager.SubtaskKey subtaskKey,
                                                                     long checkpointId,
                                                                     CheckpointedStateScope scope)
                                                              throws IOException

Description copied from class: FileMergingSnapshotManagerBase

Get a reused physical file or create one. This will be called in checkpoint output stream creation logic.

Basic logic of file reusing: whenever a physical file is needed, this method is called with necessary information provided for acquiring a file. The file will not be reused until it is written and returned to the reused pool by calling FileMergingSnapshotManagerBase.returnPhysicalFileForNextReuse(org.apache.flink.runtime.checkpoint.filemerging.FileMergingSnapshotManager.SubtaskKey, long, org.apache.flink.runtime.checkpoint.filemerging.PhysicalFile).

Specified by:

getOrCreatePhysicalFileForCheckpoint in class FileMergingSnapshotManagerBase

Parameters:

subtaskKey - the subtask key for the caller

checkpointId - the checkpoint id

scope - checkpoint scope

Returns:

the requested physical file.

Throws:

IOException - thrown if anything goes wrong with file system.

returnPhysicalFileForNextReuse

protected void returnPhysicalFileForNextReuse(FileMergingSnapshotManager.SubtaskKey subtaskKey,
                                              long checkpointId,
                                              PhysicalFile physicalFile)
                                       throws IOException

Description copied from class: FileMergingSnapshotManagerBase

Try to return an existing physical file to the manager for next reuse. If this physical file is no longer needed (for reusing), it will be closed.

Basic logic of file reusing, see FileMergingSnapshotManagerBase.getOrCreatePhysicalFileForCheckpoint(org.apache.flink.runtime.checkpoint.filemerging.FileMergingSnapshotManager.SubtaskKey, long, org.apache.flink.runtime.state.CheckpointedStateScope).

Specified by:

returnPhysicalFileForNextReuse in class FileMergingSnapshotManagerBase

Parameters:

subtaskKey - the subtask key for the caller

checkpointId - in which checkpoint this physical file is requested.

physicalFile - the returning checkpoint

Throws:

IOException - thrown if anything goes wrong with file system.

See Also:

#getOrCreatePhysicalFileForCheckpoint(SubtaskKey, long, CheckpointedStateScope)

discardCheckpoint

protected void discardCheckpoint(long checkpointId)
                          throws IOException

Description copied from class: FileMergingSnapshotManagerBase

The callback which will be triggered when all subtasks discarded (aborted or subsumed).

Overrides:

discardCheckpoint in class FileMergingSnapshotManagerBase

Parameters:

checkpointId - the discarded checkpoint id.

Throws:

IOException - if anything goes wrong with file system.

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class FileMergingSnapshotManagerBase

Throws:

IOException