Package org.apache.druid.server

Class SegmentManager

java.lang.Object
org.apache.druid.server.SegmentManager
public class SegmentManager extends Object
This class is responsible for managing data sources and their states like timeline, total segment size, and number of segments. All public methods of this class must be thread-safe.

  • Constructor Details

  • Method Details

    • getDataSourceNames

      public Set<String> getDataSourceNames()

    • getDataSourceSizes

      public Map<String,Long> getDataSourceSizes()
      Returns a map of dataSource to the total byte size of segments managed by this segmentManager. This method should be used carefully because the returned map might be different from the actual data source states.
      Returns:
      a map of dataSources and their total byte sizes

    • getAverageRowCountForDatasource

      public Map<String,Long> getAverageRowCountForDatasource()

    • getRowCountDistribution

      public Map<String,SegmentRowCountDistribution> getRowCountDistribution()

    • getDataSourceCounts

      public Map<String,Long> getDataSourceCounts()
      Returns a map of dataSource to the number of segments managed by this segmentManager. This method should be used carefully because the returned map might be different from the actual data source states.
      Returns:
      a map of dataSources and number of segments

    • getTimeline

      public Optional<VersionedIntervalTimeline<String,DataSegment>> getTimeline(TableDataSource dataSource)
      Returns the timeline for a datasource, if it exists. The analysis object passed in must represent a scan-based datasource of a single table.

    • getSegmentsBundle

      public LeafSegmentsBundle getSegmentsBundle(List<DataSegmentAndDescriptor> segments, SegmentMapFunction segmentMapFunction)
      Given a list of DataSegmentAndDescriptor produce a LeafSegmentsBundle which partitions segments into cached, loadable, or missing segments. This gives callers the flexibilty to decide to perform operations on segments which are already cached prior to or alongside the operation to load any segments which are not already present in the cache on demand.

      What this means mechanically, is that for each DataSegmentAndDescriptor we check if it is already cached with acquireCachedSegment(DataSegment) to add to LeafSegmentsBundle.cachedSegments, else if canLoadSegmentOnDemand(DataSegment) is true it is added to LeafSegmentsBundle.loadableSegments or LeafSegmentsBundle.missingSegments if not.

      The segments in LeafSegmentsBundle.loadableSegments can be retrieved with acquireSegment(DataSegment) to ensure they are loaded from deep storage.

    • acquireCachedSegment

      public Optional<Segment> acquireCachedSegment(SegmentId segmentId)
      Returns a Segment transformed with a SegmentMapFunction, if it is available in the cache. The returned Segment must be closed when the caller is finished doing segment things. This method will not download a DataSegment if it is not already present in cacheManager, use acquireSegment(DataSegment) instead.

    • acquireCachedSegment

      public Optional<Segment> acquireCachedSegment(DataSegment dataSegment)
      Convenience overload of acquireCachedSegment(SegmentId) that accepts a DataSegment.

    • acquireSegment

      public AcquireSegmentAction acquireSegment(DataSegment dataSegment)
      Returns a AcquireSegmentAction, where calling AcquireSegmentAction.getSegmentFuture() will either return immediately if the Segment is in the cache, or possibly try to fetch the segment from deep storage if not. The returned Segment, if present, must be closed when the caller is finished doing segment things.

      Calling this method is treated as an intent to acquire and use the segment via resolving the future, and cache manager implementations will place a hold on this segment until the 'loadCleanup' closer is closed - typically after resolving the future to acquire the reference to the actual Segment object.

    • getIndexedTables

      public Optional<Stream<ReferenceCountedIndexedTableProvider>> getIndexedTables(TableDataSource dataSource)
      Returns the collection of IndexedTable for the entire timeline (since join conditions do not currently consider the queries intervals), if the timeline exists for each of its segments that are joinable.

    • hasIndexedTables

      public boolean hasIndexedTables(String dataSourceName)

    • loadSegmentOnBootstrap

      public void loadSegmentOnBootstrap(DataSegment dataSegment, SegmentLazyLoadFailCallback loadFailed) throws SegmentLoadingException, IOException
      Load the supplied segment into segment cache on bootstrap. If the segment is already loaded, this method does not reload the segment into the segment cache.
      Parameters:
      dataSegment - segment to bootstrap
      loadFailed - callback to execute when segment lazy load fails. This applies only when lazy loading is enabled.
      Throws:
      SegmentLoadingException - if the segment cannot be loaded
      IOException - if the segment info cannot be cached on disk

    • loadSegment

      public void loadSegment(DataSegment dataSegment) throws SegmentLoadingException, IOException
      Load the supplied segment into segment cache. If the segment is already loaded, this method does not reload the segment into the segment cache. This method should be called for non-bootstrapping flows. Unlike loadSegmentOnBootstrap(DataSegment, SegmentLazyLoadFailCallback), this method doesn't accept a lazy load fail callback because the segment is loaded immediately.
      Parameters:
      dataSegment - segment to load
      Throws:
      SegmentLoadingException - if the segment cannot be loaded
      IOException - if the segment info cannot be cached on disk

    • dropSegment

      public void dropSegment(DataSegment dataSegment)

    • canHandleSegments

      public boolean canHandleSegments()
      Return whether the cache manager can handle segments or not.

    • canLoadSegmentsOnDemand

      public boolean canLoadSegmentsOnDemand()

    • canLoadSegmentOnDemand

      public boolean canLoadSegmentOnDemand(DataSegment dataSegment)

    • getCachedSegments

      public List<DataSegment> getCachedSegments() throws IOException
      Return a list of cached segments, if any. This should be called only when canHandleSegments() is true.
      Throws:
      IOException

    • shutdownBootstrap

      public void shutdownBootstrap()
      Shutdown the bootstrap executor to save resources. This should be called after loading bootstrap segments into the page cache.

    • shutdown

      public void shutdown()