org.apache.flink.runtime.state

Class IncrementalRemoteKeyedStateHandle

java.lang.Object

org.apache.flink.runtime.state.AbstractIncrementalStateHandle

org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle

All Implemented Interfaces:

Serializable, CheckpointBoundKeyedStateHandle, CompositeStateHandle, IncrementalKeyedStateHandle, KeyedStateHandle, StateObject

public class IncrementalRemoteKeyedStateHandle
extends AbstractIncrementalStateHandle

The handle to states of an incremental snapshot.

The states contained in an incremental snapshot include:

• Created shared state which includes shared files produced since the last completed checkpoint. These files can be referenced by succeeding checkpoints if the checkpoint succeeds to complete.
• Referenced shared state which includes the shared files materialized in previous checkpoints. Until we this is registered to a SharedStateRegistry, all referenced shared state handles are only placeholders, so that we do not send state handles twice from which we know that they already exist on the checkpoint coordinator.
• Private state which includes all other files, typically mutable, that cannot be shared by other checkpoints.
• Backend meta state which includes the information of existing states.

When this should become a completed checkpoint on the checkpoint coordinator, it must first be registered with a SharedStateRegistry, so that all placeholder state handles to previously existing state are replaced with the originals.

IMPORTANT: This class currently overrides equals and hash code only for testing purposes. They should not be called from production code. This means this class is also not suited to serve as a key, e.g. in hash maps.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.runtime.state.IncrementalKeyedStateHandle

IncrementalKeyedStateHandle.HandleAndLocalPath

Nested classes/interfaces inherited from interface org.apache.flink.runtime.state.StateObject

StateObject.StateObjectLocation, StateObject.StateObjectSizeStatsCollector

Field Summary

Fields

Modifier and Type	Field and Description
static long	UNKNOWN_CHECKPOINTED_SIZE

Fields inherited from class org.apache.flink.runtime.state.AbstractIncrementalStateHandle

backendIdentifier, checkpointId, keyGroupRange, metaStateHandle, sharedState, stateHandleId

Constructor Summary

Constructors

Modifier	Constructor and Description
	IncrementalRemoteKeyedStateHandle(UUID backendIdentifier, KeyGroupRange keyGroupRange, long checkpointId, List<IncrementalKeyedStateHandle.HandleAndLocalPath> sharedState, List<IncrementalKeyedStateHandle.HandleAndLocalPath> privateState, StreamStateHandle metaStateHandle)
	IncrementalRemoteKeyedStateHandle(UUID backendIdentifier, KeyGroupRange keyGroupRange, long checkpointId, List<IncrementalKeyedStateHandle.HandleAndLocalPath> sharedState, List<IncrementalKeyedStateHandle.HandleAndLocalPath> privateState, StreamStateHandle metaStateHandle, long persistedSizeOfThisCheckpoint)
protected	IncrementalRemoteKeyedStateHandle(UUID backendIdentifier, KeyGroupRange keyGroupRange, long checkpointId, List<IncrementalKeyedStateHandle.HandleAndLocalPath> sharedState, List<IncrementalKeyedStateHandle.HandleAndLocalPath> privateState, StreamStateHandle metaStateHandle, long persistedSizeOfThisCheckpoint, StateHandleID stateHandleId)

Method Summary

Modifier and Type	Method and Description
void	collectSizeStats(StateObject.StateObjectSizeStatsCollector collector) Collects statistics about state size and location from the state object.
void	discardState() Discards the state referred to and solemnly owned by this handle, to free up resources in the persistent storage.
long	getCheckpointedSize() Returns the persisted data size during checkpoint execution in bytes.
List<IncrementalKeyedStateHandle.HandleAndLocalPath>	getPrivateState()
List<IncrementalKeyedStateHandle.HandleAndLocalPath>	getSharedState()
List<IncrementalKeyedStateHandle.HandleAndLocalPath>	getSharedStateHandles() Returns a list of all shared states and the corresponding localPath in the backend at the time this was created.
SharedStateRegistry	getSharedStateRegistry()
long	getStateSize() Returns the size of the state in bytes.
CheckpointBoundKeyedStateHandle	rebound(long checkpointId) Returns a new CheckpointBoundKeyedStateHandle Rebounding checkpoint id to a specific checkpoint id.
void	registerSharedStates(SharedStateRegistry stateRegistry, long checkpointID) Register both newly created and already referenced shared states in the given SharedStateRegistry.
static IncrementalRemoteKeyedStateHandle	restore(UUID backendIdentifier, KeyGroupRange keyGroupRange, long checkpointId, List<IncrementalKeyedStateHandle.HandleAndLocalPath> sharedState, List<IncrementalKeyedStateHandle.HandleAndLocalPath> privateState, StreamStateHandle metaStateHandle, long persistedSizeOfThisCheckpoint, StateHandleID stateHandleId)
Stream<StreamStateHandle>	streamSubHandles()
String	toString()

Methods inherited from class org.apache.flink.runtime.state.AbstractIncrementalStateHandle

equals, getBackendIdentifier, getCheckpointId, getIntersection, getKeyGroupRange, getMetaDataStateHandle, getStateHandleId, hashCode

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

UNKNOWN_CHECKPOINTED_SIZE

public static final long UNKNOWN_CHECKPOINTED_SIZE

See Also:

Constant Field Values

Constructor Detail

IncrementalRemoteKeyedStateHandle

@VisibleForTesting
public IncrementalRemoteKeyedStateHandle(UUID backendIdentifier,
                                                            KeyGroupRange keyGroupRange,
                                                            long checkpointId,
                                                            List<IncrementalKeyedStateHandle.HandleAndLocalPath> sharedState,
                                                            List<IncrementalKeyedStateHandle.HandleAndLocalPath> privateState,
                                                            StreamStateHandle metaStateHandle)

IncrementalRemoteKeyedStateHandle

public IncrementalRemoteKeyedStateHandle(UUID backendIdentifier,
                                         KeyGroupRange keyGroupRange,
                                         long checkpointId,
                                         List<IncrementalKeyedStateHandle.HandleAndLocalPath> sharedState,
                                         List<IncrementalKeyedStateHandle.HandleAndLocalPath> privateState,
                                         StreamStateHandle metaStateHandle,
                                         long persistedSizeOfThisCheckpoint)

IncrementalRemoteKeyedStateHandle

protected IncrementalRemoteKeyedStateHandle(UUID backendIdentifier,
                                            KeyGroupRange keyGroupRange,
                                            long checkpointId,
                                            List<IncrementalKeyedStateHandle.HandleAndLocalPath> sharedState,
                                            List<IncrementalKeyedStateHandle.HandleAndLocalPath> privateState,
                                            StreamStateHandle metaStateHandle,
                                            long persistedSizeOfThisCheckpoint,
                                            StateHandleID stateHandleId)

Method Detail

restore

public static IncrementalRemoteKeyedStateHandle restore(UUID backendIdentifier,
                                                        KeyGroupRange keyGroupRange,
                                                        long checkpointId,
                                                        List<IncrementalKeyedStateHandle.HandleAndLocalPath> sharedState,
                                                        List<IncrementalKeyedStateHandle.HandleAndLocalPath> privateState,
                                                        StreamStateHandle metaStateHandle,
                                                        long persistedSizeOfThisCheckpoint,
                                                        StateHandleID stateHandleId)

rebound

public CheckpointBoundKeyedStateHandle rebound(long checkpointId)

Description copied from interface: CheckpointBoundKeyedStateHandle

Returns a new CheckpointBoundKeyedStateHandle Rebounding checkpoint id to a specific checkpoint id.

Parameters:

checkpointId - rebounded checkpoint id.

getSharedState

public List<IncrementalKeyedStateHandle.HandleAndLocalPath> getSharedState()

getPrivateState

public List<IncrementalKeyedStateHandle.HandleAndLocalPath> getPrivateState()

getSharedStateHandles

@Nonnull
public List<IncrementalKeyedStateHandle.HandleAndLocalPath> getSharedStateHandles()

Description copied from interface: IncrementalKeyedStateHandle

Returns a list of all shared states and the corresponding localPath in the backend at the time this was created.

Specified by:

getSharedStateHandles in interface IncrementalKeyedStateHandle

Overrides:

getSharedStateHandles in class AbstractIncrementalStateHandle

getSharedStateRegistry

public SharedStateRegistry getSharedStateRegistry()

discardState

public void discardState()
                  throws Exception

Description copied from interface: StateObject

Discards the state referred to and solemnly owned by this handle, to free up resources in the persistent storage. This method is called when the state represented by this object will not be used anymore.

Throws:

Exception

getStateSize

public long getStateSize()

Description copied from interface: StateObject

Returns the size of the state in bytes. If the size is not known, this method should return 0.

The values produced by this method are only used for informational purposes and for metrics/monitoring. If this method returns wrong values, the checkpoints and recovery will still behave correctly. However, efficiency may be impacted (wrong space pre-allocation) and functionality that depends on metrics (like monitoring) will be impacted.

Note for implementors: This method should not perform any I/O operations while obtaining the state size (hence it does not declare throwing an IOException). Instead, the state size should be stored in the state object, or should be computable from the state stored in this object. The reason is that this method is called frequently by several parts of the checkpointing and issuing I/O requests from this method accumulates a heavy I/O load on the storage system at higher scale.

Returns:

Size of the state in bytes.

collectSizeStats

public void collectSizeStats(StateObject.StateObjectSizeStatsCollector collector)

Description copied from interface: StateObject

Collects statistics about state size and location from the state object.

Parameters:

collector - the statistics collector.

streamSubHandles

public Stream<StreamStateHandle> streamSubHandles()

getCheckpointedSize

public long getCheckpointedSize()

Description copied from interface: CompositeStateHandle

Returns the persisted data size during checkpoint execution in bytes. If incremental checkpoint is enabled, this value represents the incremental persisted data size, and usually smaller than StateObject.getStateSize(). If the size is unknown, this method would return same result as StateObject.getStateSize().

Returns:

The persisted data size during checkpoint execution in bytes.

registerSharedStates

public void registerSharedStates(SharedStateRegistry stateRegistry,
                                 long checkpointID)

Description copied from interface: CompositeStateHandle

Register both newly created and already referenced shared states in the given SharedStateRegistry. This method is called when the checkpoint successfully completes or is recovered from failures.

After this is completed, newly created shared state is considered as published is no longer owned by this handle. This means that it should no longer be deleted as part of calls to StateObject.discardState(). Instead, StateObject.discardState() will trigger an unregistration from the registry.

Parameters:

stateRegistry - The registry where shared states are registered.

toString

public String toString()

Overrides:

toString in class AbstractIncrementalStateHandle