Package org.apache.druid.server.compaction

Class CompactionSlotManager

java.lang.Object
org.apache.druid.server.compaction.CompactionSlotManager
public class CompactionSlotManager extends Object
Fetches running compaction tasks from the Overlord and tracks their compaction intervals and task slots.

  • Field Details

    • COMPACTION_TASK_TYPE

      public static final String COMPACTION_TASK_TYPE
      Task type for native compaction tasks.
      See Also:

  • Constructor Details

  • Method Details

    • getNumAvailableTaskSlots

      public int getNumAvailableTaskSlots()

    • getDatasourceIntervalsToSkipCompaction

      public Map<String,List<org.joda.time.Interval>> getDatasourceIntervalsToSkipCompaction()

    • reserveTaskSlots

      public void reserveTaskSlots(int numSlotsToReserve)

    • reserveTaskSlots

      public void reserveTaskSlots(ClientCompactionTaskQuery compactionTaskQuery)
      Reserves task slots for the given task from the overall compaction task capacity.

    • reserveTaskSlotsForRunningCompactionTasks

      public void reserveTaskSlotsForRunningCompactionTasks()
      Reserves task slots for all running compaction tasks.

    • fetchRunningCompactionTasks

      public List<ClientCompactionTaskQuery> fetchRunningCompactionTasks()
      Retrieves currently running tasks of type COMPACTION_TASK_TYPE from the Overlord.

      Also queries the Overlord for the status of all tasks that were submitted recently but are not active anymore. The statuses are then updated in the CompactionStatusTracker.

    • cancelTaskOnlyIfGranularityChanged

      public boolean cancelTaskOnlyIfGranularityChanged(ClientCompactionTaskQuery compactionTaskQuery, DataSourceCompactionConfig dataSourceCompactionConfig)
      Cancels a currently running compaction task only if the segment granularity has changed in the datasource compaction config. Otherwise, the task is retained and its intervals are skipped from the current round of compaction.
      Returns:
      true if the task was canceled, false otherwise.

    • skipLockedIntervals

      public void skipLockedIntervals(List<DataSourceCompactionConfig> compactionConfigs)
      Retrieves the list of intervals locked by higher priority tasks for each datasource. Since compaction tasks submitted for these Intervals would have to wait anyway, we skip these Intervals until the next compaction run by adding them to intervalsToSkipCompaction.

      This method must be called after invalid compaction tasks have already been cancelled using cancelTaskOnlyIfGranularityChanged(org.apache.druid.client.indexing.ClientCompactionTaskQuery, org.apache.druid.server.coordinator.DataSourceCompactionConfig) so that their intervals are not considered locked.

    • getMaxTaskSlotsForNativeCompactionTask

      public static int getMaxTaskSlotsForNativeCompactionTask(@Nullable ClientCompactionTaskQueryTuningConfig tuningConfig)
      Returns:
      Maximum number of task slots used by a native compaction task at any time when the task is run with the given tuning config.

    • getMaxTaskSlotsForMSQCompactionTask

      public static int getMaxTaskSlotsForMSQCompactionTask(@Nullable Map<String,Object> context)
      Maximum number of task slots used by an MSQ compaction task at any time when the task is run with the given context.

    • isParallelMode

      public static boolean isParallelMode(@Nullable ClientCompactionTaskQueryTuningConfig tuningConfig)
      Returns true if the compaction task can run in the parallel mode with the given tuningConfig. This method should be synchronized with ParallelIndexSupervisorTask.isParallelMode(InputSource, ParallelIndexTuningConfig).

    • computeSlotsRequiredForTask

      public int computeSlotsRequiredForTask(ClientCompactionTaskQuery task, DataSourceCompactionConfig config)
      Computes the maximum number of slots required for a compaction task. This is the legacy method used by the Coordinator compaction duty.

      MSQ-based Compaction tasks launched by the Coordinator use up all available task slots if maxNumTasks is not specified in the context. However, computeSlotsRequiredForTask(ClientCompactionTaskQuery), used by compaction supervisors uses ClientMSQContext.DEFAULT_MAX_NUM_TASKS instead.

    • computeSlotsRequiredForTask

      public static int computeSlotsRequiredForTask(ClientCompactionTaskQuery task)
      Computes the number of task slots required to run this compaction task.