Package org.apache.druid.segment.loading

Interface SegmentCacheManager

All Known Implementing Classes:

SegmentLocalCacheManager

public interface SegmentCacheManager

A class to fetch segment files from deep storage to local disk and manage the local cache. Implementations must be thread-safe.

Method Summary

Modifier and Type	Method	Description
Optional<Segment>	acquireCachedSegment(SegmentId segmentId)	Applies a SegmentMapFunction to a Segment if it is available in the cache.
AcquireSegmentAction	acquireSegment(DataSegment dataSegment)	Returns a AcquireSegmentAction for a given DataSegment and SegmentDescriptor, which returns a reference provider for the Segment if already present in the cache, or tries to fetch from deep storage and map if not.
void	bootstrap(DataSegment segment, SegmentLazyLoadFailCallback loadFailed)	Similar to load(DataSegment), this method loads segments during startup on data nodes.
boolean	canHandleSegments()	Return whether the cache manager can handle segments or not.
boolean	canLoadSegmentOnDemand(DataSegment segment)
boolean	canLoadSegmentsOnDemand()
void	drop(DataSegment segment)	Cleanup the segment files cache space used by the segment, releasing the StorageLocation reservation
List<DataSegment>	getCachedSegments()	Return a list of cached segments from local disk, if any.
File	getSegmentFiles(DataSegment segment)	Alternative to acquireCachedSegment(SegmentId), to return the File location of the segment files stored in the cache, instead of a Optional<Segment>.
StorageStats	getStorageStats()	Collect StorageStats, if available.
void	load(DataSegment segment)	Given a DataSegment, which contains the instructions for where and how to fetch a Segment from deep storage, this method tries to load and subsequently serve it to callers via acquireCachedSegment(SegmentId) or acquireSegment(DataSegment).
void	removeInfoFile(DataSegment segment)	Remove the segment info file for the supplied segment from disk.
void	shutdown()
void	shutdownBootstrap()	Shutdown any previously set-up bootstrap executor to save resources.
void	storeInfoFile(DataSegment segment)	Store a segment info file for the supplied segment on disk.

Method Details

canHandleSegments

boolean canHandleSegments()

Return whether the cache manager can handle segments or not.

canLoadSegmentsOnDemand

boolean canLoadSegmentsOnDemand()

canLoadSegmentOnDemand

boolean canLoadSegmentOnDemand(DataSegment segment)

getCachedSegments

List<DataSegment> getCachedSegments()
                             throws IOException

Return a list of cached segments from local disk, if any. This should be called only when canHandleSegments() is true.

Throws:

IOException

storeInfoFile

void storeInfoFile(DataSegment segment)
            throws IOException

Store a segment info file for the supplied segment on disk. This operation is idempotent when called multiple times for a given segment.

Throws:

IOException

removeInfoFile

void removeInfoFile(DataSegment segment)

Remove the segment info file for the supplied segment from disk. If the file cannot be deleted, do nothing.

See Also:

• drop(DataSegment)

load

void load(DataSegment segment)
   throws SegmentLoadingException

Given a DataSegment, which contains the instructions for where and how to fetch a Segment from deep storage, this method tries to load and subsequently serve it to callers via acquireCachedSegment(SegmentId) or acquireSegment(DataSegment). If the segment cannot be loaded either due to error or insufficient storage space, this method throws a SegmentLoadingException.

Parameters:

segment - Segment to get on each download (after service bootstrap)

Throws:

SegmentLoadingException - If there is an error in loading the segment or insufficient storage space

See Also:

• bootstrap(DataSegment, SegmentLazyLoadFailCallback)

bootstrap

void bootstrap(DataSegment segment,
 SegmentLazyLoadFailCallback loadFailed)
        throws SegmentLoadingException

Similar to load(DataSegment), this method loads segments during startup on data nodes. Implementations of this method may be configured to use a larger than normal work pool that only exists during startup and is shutdown after startup by calling shutdownBootstrap()

Parameters:

segment - Segment to retrieve during service bootstrap

loadFailed - Callback to execute when segment lazy load failed. This applies only when lazyLoadOnStart is enabled

Throws:

SegmentLoadingException - - If there is an error in loading the segment or insufficient storage space

See Also:

• load(DataSegment)
• shutdownBootstrap()

drop

void drop(DataSegment segment)

Cleanup the segment files cache space used by the segment, releasing the StorageLocation reservation

See Also:

• removeInfoFile(DataSegment)

acquireCachedSegment

Optional<Segment> acquireCachedSegment(SegmentId segmentId)

Applies a SegmentMapFunction to a Segment if it is available in the cache. If not present in any storage location, this method will not attempt to download the DataSegment from deep storage. The Segment returned by this method is considered an open reference, cache implementations must not allow it to be dropped until it has been closed. As such, the returned Segment must be closed when the caller is finished doing segment things.

acquireSegment

AcquireSegmentAction acquireSegment(DataSegment dataSegment)

Returns a AcquireSegmentAction for a given DataSegment and SegmentDescriptor, which returns a reference provider for the Segment if already present in the cache, or tries to fetch from deep storage and map if not. The Segment returned by the provider returned by this method are considered an open reference, cache implementations must not allow the segment to be dropped until it has been closed. As such, the returned Segment from ReferenceCountedObjectProvider.acquireReference() must be closed when the caller is finished doing segment things.

getSegmentFiles

@Nullable
File getSegmentFiles(DataSegment segment)
              throws SegmentLoadingException

Alternative to acquireCachedSegment(SegmentId), to return the File location of the segment files stored in the cache, instead of a Optional<Segment>. Unlike acquireCachedSegment(SegmentId) and acquireSegment(DataSegment), this method does not provide any protections for callers, and should only be used by callers that are in control of when drop(DataSegment) is called. This method will not download the segment files from deep storage if they do not already exist in the cache, callers should use load(DataSegment) before calling this method.

Throws:

SegmentLoadingException

shutdownBootstrap

void shutdownBootstrap()

Shutdown any previously set-up bootstrap executor to save resources. This should be called after loading bootstrap segments.

shutdown

void shutdown()

getStorageStats

@Nullable
StorageStats getStorageStats()

Collect StorageStats, if available.