Type Members

1.  abstract class AbstractFileContextBasedCheckpointFileManager extends CheckpointFileManager with Logging
2.  class AsyncCommitLog extends CommitLog

   Implementation of CommitLog to perform asynchronous writes to storage

3.  class AsyncOffsetSeqLog extends OffsetSeqLog

   Used to write entries to the offset log asynchronously

4.  trait CheckpointFileManager extends AnyRef

   An interface to abstract out all operation related to streaming checkpoints.

   An interface to abstract out all operation related to streaming checkpoints. Most importantly, the key operation this interface provides is createAtomic(path, overwrite) which returns a CancellableFSDataOutputStream. This method is used by HDFSMetadataLog and StateStore implementations to write a complete checkpoint file atomically (i.e. no partial file will be visible), with or without overwrite.

   This higher-level interface above the Hadoop FileSystem is necessary because different implementation of FileSystem/FileContext may have different combination of operations to provide the desired atomic guarantees (e.g. write-to-temp-file-and-rename, direct-write-and-cancel-on-failure) and this abstraction allow different implementations while keeping the usage simple (createAtomic -> close or cancel).

5.  case class Checksum(algorithm: String, value: Int, mainFileSize: Long, timestampMs: Long, creator: ChecksumFileCreatorInfo) extends Product with Serializable

   This is the content of the checksum file.

   This is the content of the checksum file. Holds the checksum value and additional information

6.  class ChecksumCancellableFSDataOutputStream extends CancellableFSDataOutputStream with Logging

   An implementation of CancellableFSDataOutputStream that calculates the checksum of the file that the client is writing (main file) incrementally, while it is being written.

   An implementation of CancellableFSDataOutputStream that calculates the checksum of the file that the client is writing (main file) incrementally, while it is being written. It then writes the main file and an additional checksum file, which will be used for verification by ChecksumFSDataInputStream on file read.

7.  class ChecksumCheckpointFileManager extends CheckpointFileManager with Logging

   A CheckpointFileManager that creates a checksum file for the main file.

   A CheckpointFileManager that creates a checksum file for the main file. This wraps another CheckpointFileManager and adds checksum functionality on top of it. Under the hood, when a file is created, it also creates a checksum file with the same name as the main file but adds a suffix. It returns ChecksumCancellableFSDataOutputStream which handles the writing of the main file and checksum file.

   When a file is opened, it returns ChecksumFSDataInputStream, which handles reading the main file and checksum file and does the checksum verification.

   In order to reduce the impact of reading/writing 2 files instead of 1, it uses a threadpool to read/write both files concurrently.

   Note

   It is able to read files written by other CheckpointFileManager, that don't have checksum. It automatically deletes the checksum file when the main file is deleted. If you delete the main file with a different type of manager, then the checksum file will be left behind (i.e. orphan checksum file), since they don't know about it. It would be your responsibility to delete the orphan checksum files.

8.  class ChecksumFSDataInputStream extends FSDataInputStream with Logging

   An implementation of FSDataInputStream that calculates the checksum of the file that the client is reading (main file) incrementally, while it is being read.

   An implementation of FSDataInputStream that calculates the checksum of the file that the client is reading (main file) incrementally, while it is being read. It then does checksum verification on close, to verify that the computed checksum matches the expected checksum in the checksum file.

   Computing the checksum incrementally and doing the verification after file read is complete is for better performance, instead of first reading the entire file and doing verification before the client starts reading the file.

9.  case class ChecksumFile(path: Path) extends Product with Serializable

   Holds the path of the checksum file and knows the main file path.

10.  case class ChecksumFileCreatorInfo(executorId: String, taskInfo: String) extends Product with Serializable

    Information about the creator of the checksum file.

    Information about the creator of the checksum file. Useful for debugging

11.  class CommitLog extends HDFSMetadataLog[CommitMetadata]

    Used to write log files that represent batch commit points in structured streaming.

    Used to write log files that represent batch commit points in structured streaming. A commit log file will be written immediately after the successful completion of a batch, and before processing the next batch. Here is an execution summary: - trigger batch 1 - obtain batch 1 offsets and write to offset log - process batch 1 - write batch 1 to completion log - trigger batch 2 - obtain batch 2 offsets and write to offset log - process batch 2 - write batch 2 to completion log ....

    The current format of the batch completion log is: line 1: version line 2: metadata (optional json string)

12.  case class CommitMetadata(nextBatchWatermarkMs: Long = 0, stateUniqueIds: Option[Map[Long, Array[Array[String]]]] = None) extends Product with Serializable

    In Checkpoint V2, for a stateful query, the checkpoint structure looks like below: 0 (operator ID) +----+ | 0 (partitionID) +----+ | ......

    In Checkpoint V2, for a stateful query, the checkpoint structure looks like below: 0 (operator ID) +----+ | 0 (partitionID) +----+ | ...... | 1 (partitionID) +----+ | |- default (storeName) | +-----+ | | 20_unique_id_1.zip | | 21_unique_id_2.delta | | 22_unique_id_3.delta | + 23_unique_id_4.delta | 2 (partitionID) +--- ...... In the commit log, in addition to nextBatchWatermarkMs, we also store the unique ids of the state store files.

    nextBatchWatermarkMs

    The watermark of the next batch.

    stateUniqueIds

    Map[Long, Array[Array[String]]] of map OperatorId -> (partitionID -> array of uniqueID)

13.  class FileContextBasedCheckpointFileManager extends AbstractFileContextBasedCheckpointFileManager with RenameHelperMethods
14.  class FileSystemBasedCheckpointFileManager extends CheckpointFileManager with RenameHelperMethods with Logging

    An implementation of CheckpointFileManager using Hadoop's FileSystem API.

15.  class HDFSMetadataLog[T <: AnyRef] extends MetadataLog[T] with Logging

    A MetadataLog implementation based on HDFS.

    A MetadataLog implementation based on HDFS. HDFSMetadataLog uses the specified path as the metadata storage.

    When writing a new batch, HDFSMetadataLog will firstly write to a temp file and then rename it to the final batch file. If the rename step fails, there must be multiple writers and only one of them will succeed and the others will fail.

    Note: HDFSMetadataLog doesn't support S3-like file systems as they don't guarantee listing files in a directory always shows the latest files.

16.  trait MetadataLog[T] extends AnyRef

    A general MetadataLog that supports the following features:

    A general MetadataLog that supports the following features:

    • Allow the user to store a metadata object for each batch.
    • Allow the user to query the latest batch id.
    • Allow the user to query the metadata object of a specified batch id.
    • Allow the user to query metadata objects in a range of batch ids.
    • Allow the user to remove obsolete metadata
17.  case class OffsetSeq(offsets: Seq[Option[connector.read.streaming.Offset]], metadata: Option[OffsetSeqMetadata] = None) extends Product with Serializable

    An ordered collection of offsets, used to track the progress of processing data from one or more Sources that are present in a streaming query.

    An ordered collection of offsets, used to track the progress of processing data from one or more Sources that are present in a streaming query. This is similar to simplified, single-instance vector clock that must progress linearly forward.

18.  class OffsetSeqLog extends HDFSMetadataLog[OffsetSeq]

    This class is used to log offsets to persistent files in HDFS.

    This class is used to log offsets to persistent files in HDFS. Each file corresponds to a specific batch of offsets. The file format contains a version string in the first line, followed by a the JSON string representation of the offsets separated by a newline character. If a source offset is missing, then that line will contain a string value defined in the SERIALIZED_VOID_OFFSET variable in OffsetSeqLog companion object. For instance, when dealing with LongOffset types: v1 // version 1 metadata {0} // LongOffset 0 {3} // LongOffset 3

    • // No offset for this source i.e., an invalid JSON string {2} // LongOffset 2 ...
19.  case class OffsetSeqMetadata(batchWatermarkMs: Long = 0, batchTimestampMs: Long = 0, conf: Map[String, String] = Map.empty) extends Product with Serializable

    Contains metadata associated with a OffsetSeq.

    Contains metadata associated with a OffsetSeq. This information is persisted to the offset log in the checkpoint location via the OffsetSeq metadata field.