DuplicatingStateChangeFsUploader (Flink : DSTL : DFS 1.20.2 API)

java.lang.Object
- org.apache.flink.changelog.fs.AbstractStateChangeFsUploader
- - org.apache.flink.changelog.fs.DuplicatingStateChangeFsUploader

All Implemented Interfaces:

AutoCloseable, StateChangeUploader
```
public class DuplicatingStateChangeFsUploader
extends AbstractStateChangeFsUploader
```
A StateChangeFsUploader implementation that writes the changes to remote and local.
Local dstl files are only managed by TM side, LocalChangelogRegistry, TaskChangelogRegistry and ChangelogTaskLocalStateStore are responsible for managing them.
Remote dstl files are managed by TM side and JM side, TaskChangelogRegistry is responsible for TM side, and SharedStateRegistry is responsible for JM side.
The total discard logic of local dstl files is:
1. Register files to TaskChangelogRegistry.startTracking(org.apache.flink.runtime.state.StreamStateHandle, long) on AbstractStateChangeFsUploader.upload(java.util.Collection<org.apache.flink.changelog.fs.StateChangeUploadScheduler.UploadTask>).
2. Store the meta of files into ChangelogTaskLocalStateStore by AsyncCheckpointRunnable#reportCompletedSnapshotStates().
3. Pass control of the file to LocalChangelogRegistry.register(org.apache.flink.runtime.state.StreamStateHandle, long) when FsStateChangelogWriter#persist , files of the previous checkpoint will be deleted by LocalChangelogRegistry.discardUpToCheckpoint(long) when the checkpoint is confirmed.
4. When ChangelogTruncateHelper#materialized() or ChangelogTruncateHelper#checkpointSubsumed() is called, TaskChangelogRegistry.release(org.apache.flink.runtime.state.StreamStateHandle) is responsible for deleting local files.
5. When one checkpoint is aborted, all accumulated local dstl files will be deleted at once.

Nested Class Summary
- Nested classes/interfaces inherited from interface org.apache.flink.changelog.fs.StateChangeUploader
  StateChangeUploader.UploadTasksResult

Field Summary
- Fields inherited from class org.apache.flink.changelog.fs.AbstractStateChangeFsUploader
  bufferSize, compression, metrics

Constructor Summary

Constructors
Constructor and Description
`DuplicatingStateChangeFsUploader(org.apache.flink.api.common.JobID jobID, org.apache.flink.core.fs.Path basePath, org.apache.flink.core.fs.FileSystem fileSystem, boolean compression, int bufferSize, ChangelogStorageMetricGroup metrics, TaskChangelogRegistry changelogRegistry, org.apache.flink.runtime.state.LocalSnapshotDirectoryProvider localSnapshotDirectoryProvider)`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type Method and Description

void close()

org.apache.flink.changelog.fs.OutputStreamWithPos prepareStream()
- Methods inherited from class org.apache.flink.changelog.fs.AbstractStateChangeFsUploader
  generateFileName, upload
- Methods inherited from class java.lang.Object
  clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`close()`
`org.apache.flink.changelog.fs.OutputStreamWithPos`	`prepareStream()`

Constructor Detail

DuplicatingStateChangeFsUploader

public DuplicatingStateChangeFsUploader(org.apache.flink.api.common.JobID jobID,
                                        org.apache.flink.core.fs.Path basePath,
                                        org.apache.flink.core.fs.FileSystem fileSystem,
                                        boolean compression,
                                        int bufferSize,
                                        ChangelogStorageMetricGroup metrics,
                                        TaskChangelogRegistry changelogRegistry,
                                        org.apache.flink.runtime.state.LocalSnapshotDirectoryProvider localSnapshotDirectoryProvider)

Method Detail

prepareStream

public org.apache.flink.changelog.fs.OutputStreamWithPos prepareStream()
                                                                throws IOException

Throws:: IOException

close

public void close()
           throws Exception

Throws:: Exception

Class DuplicatingStateChangeFsUploader

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.changelog.fs.StateChangeUploader

Field Summary

Fields inherited from class org.apache.flink.changelog.fs.AbstractStateChangeFsUploader

Constructor Summary

Method Summary

Methods inherited from class org.apache.flink.changelog.fs.AbstractStateChangeFsUploader

Methods inherited from class java.lang.Object

Constructor Detail

DuplicatingStateChangeFsUploader

Method Detail

prepareStream

close