Package io.delta.storage.commit

Class CoordinatedCommitsUtils

Object
io.delta.storage.commit.CoordinatedCommitsUtils
public class CoordinatedCommitsUtils extends Object

  • Method Details

    • generateUnbackfilledDeltaFilePath

      public static org.apache.hadoop.fs.Path generateUnbackfilledDeltaFilePath(org.apache.hadoop.fs.Path logPath, long version)
      Creates a new unbackfilled delta file path for the given commit version. The path is of the form: `tablePath/_delta_log/_staged_commits/00000000000000000001.uuid.json`.

    • getBackfilledDeltaFilePath

      public static org.apache.hadoop.fs.Path getBackfilledDeltaFilePath(org.apache.hadoop.fs.Path logPath, Long version)
      Returns the path to the backfilled delta file for the given commit version. The path is of the form `tablePath/_delta_log/00000000000000000001.json`.

    • isCoordinatedCommitsToFSConversion

      public static boolean isCoordinatedCommitsToFSConversion(Long commitVersion, UpdatedActions updatedActions)
      Returns true if the commit is a coordinated commits to filesystem conversion.

    • getTablePath

      public static org.apache.hadoop.fs.Path getTablePath(org.apache.hadoop.fs.Path logPath)
      Get the table path from the provided log path.

    • getUnbackfilledDeltaFile

      public static org.apache.hadoop.fs.Path getUnbackfilledDeltaFile(org.apache.hadoop.fs.Path logPath, long version, Optional<String> uuidString)
      Returns the un-backfilled uuid formatted delta (json format) path for a given version.
      Parameters:
      logPath - The root path of the delta log.
      version - The version of the delta file.
      Returns:
      The path to the un-backfilled delta file: logPath/_staged_commits/version.uuid.json

    • writeUnbackfilledCommitFile

      public static org.apache.hadoop.fs.FileStatus writeUnbackfilledCommitFile(LogStore logStore, org.apache.hadoop.conf.Configuration hadoopConf, String logPath, long commitVersion, Iterator<String> actions, String uuid) throws IOException
      Write a UUID-based commit file for the specified version to the table at logPath.
      Throws:
      IOException

    • logDirPath

      public static org.apache.hadoop.fs.Path logDirPath(org.apache.hadoop.fs.Path tablePath)
      Returns path to the directory which holds the delta log

    • commitDirPath

      public static org.apache.hadoop.fs.Path commitDirPath(org.apache.hadoop.fs.Path logPath)
      Returns path to the directory which holds the unbackfilled commits

    • getCoordinatorName

      public static Optional<String> getCoordinatorName(AbstractMetadata metadata)
      Retrieves the coordinator name from the provided abstract metadata. If no coordinator is set, an empty optional is returned.
      Parameters:
      metadata - The abstract metadata from which to retrieve the coordinator name.
      Returns:
      The coordinator name if set, otherwise an empty optional.

    • getCoordinatorConf

      public static Map<String,String> getCoordinatorConf(AbstractMetadata abstractMetadata)
      Get the coordinated commits owner configuration from the provided abstract metadata.

    • getTableConf

      public static Map<String,String> getTableConf(AbstractMetadata abstractMetadata)
      Get the coordinated commits table configuration from the provided abstract metadata.