Package org.apache.druid.segment.loading

Class SegmentLocalCacheManager

java.lang.Object

org.apache.druid.segment.loading.SegmentLocalCacheManager

All Implemented Interfaces:

SegmentCacheManager

public class SegmentLocalCacheManager
extends Object
implements SegmentCacheManager

Constructor Summary

Constructors

Constructor	Description
SegmentLocalCacheManager(List<StorageLocation> locations, SegmentLoaderConfig config, StorageLocationSelectorStrategy strategy, IndexIO indexIO, com.fasterxml.jackson.databind.ObjectMapper mapper)

Method Summary

Modifier and Type	Method	Description
Optional<Segment>	acquireCachedSegment(SegmentId segmentId)	Applies a SegmentMapFunction to a Segment if it is available in the cache.
AcquireSegmentAction	acquireSegment(DataSegment dataSegment)	Returns a AcquireSegmentAction for a given DataSegment and SegmentDescriptor, which returns a reference provider for the Segment if already present in the cache, or tries to fetch from deep storage and map if not.
void	bootstrap(DataSegment dataSegment, SegmentLazyLoadFailCallback loadFailed)	Similar to SegmentCacheManager.load(DataSegment), this method loads segments during startup on data nodes.
boolean	canHandleSegments()	Return whether the cache manager can handle segments or not.
boolean	canLoadSegmentOnDemand(DataSegment dataSegment)
boolean	canLoadSegmentsOnDemand()
void	drop(DataSegment segment)	Cleanup the segment files cache space used by the segment, releasing the StorageLocation reservation
List<DataSegment>	getCachedSegments()	Return a list of cached segments from local disk, if any.
File	getSegmentFiles(DataSegment segment)	Alternative to SegmentCacheManager.acquireCachedSegment(SegmentId), to return the File location of the segment files stored in the cache, instead of a Optional<Segment>.
ConcurrentHashMap<DataSegment,org.apache.druid.segment.loading.SegmentLocalCacheManager.ReferenceCountingLock>	getSegmentLocks()
ReferenceCountedSegmentProvider	getSegmentReferenceProvider(DataSegment segment)	Testing use only please, any callers that want to do stuff with segments should use acquireCachedSegment(SegmentId) or acquireSegment(DataSegment) instead.
StorageStats	getStorageStats()	Collect StorageStats, if available.
void	load(DataSegment dataSegment)	Given a DataSegment, which contains the instructions for where and how to fetch a Segment from deep storage, this method tries to load and subsequently serve it to callers via SegmentCacheManager.acquireCachedSegment(SegmentId) or SegmentCacheManager.acquireSegment(DataSegment).
void	removeInfoFile(DataSegment segment)	Remove the segment info file for the supplied segment from disk.
void	shutdown()
void	shutdownBootstrap()	Shutdown any previously set-up bootstrap executor to save resources.
void	storeInfoFile(DataSegment segment)	Store a segment info file for the supplied segment on disk.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

SegmentLocalCacheManager

@Inject
public SegmentLocalCacheManager(List<StorageLocation> locations,
 SegmentLoaderConfig config,
 @Nonnull
 StorageLocationSelectorStrategy strategy,
 IndexIO indexIO,
 com.fasterxml.jackson.databind.ObjectMapper mapper)

Method Details

canHandleSegments

public boolean canHandleSegments()

Description copied from interface: SegmentCacheManager

Return whether the cache manager can handle segments or not.

Specified by:

canHandleSegments in interface SegmentCacheManager

canLoadSegmentsOnDemand

public boolean canLoadSegmentsOnDemand()

Specified by:

canLoadSegmentsOnDemand in interface SegmentCacheManager

canLoadSegmentOnDemand

public boolean canLoadSegmentOnDemand(DataSegment dataSegment)

Specified by:

canLoadSegmentOnDemand in interface SegmentCacheManager

getCachedSegments

public List<DataSegment> getCachedSegments()
                                    throws IOException

Description copied from interface: SegmentCacheManager

Return a list of cached segments from local disk, if any. This should be called only when SegmentCacheManager.canHandleSegments() is true.

Specified by:

getCachedSegments in interface SegmentCacheManager

Throws:

IOException

storeInfoFile

public void storeInfoFile(DataSegment segment)
                   throws IOException

Description copied from interface: SegmentCacheManager

Store a segment info file for the supplied segment on disk. This operation is idempotent when called multiple times for a given segment.

Specified by:

storeInfoFile in interface SegmentCacheManager

Throws:

IOException

removeInfoFile

public void removeInfoFile(DataSegment segment)

Description copied from interface: SegmentCacheManager

Remove the segment info file for the supplied segment from disk. If the file cannot be deleted, do nothing.

Specified by:

removeInfoFile in interface SegmentCacheManager

See Also:

• SegmentCacheManager.drop(DataSegment)

acquireCachedSegment

public Optional<Segment> acquireCachedSegment(SegmentId segmentId)

Description copied from interface: SegmentCacheManager

Applies a SegmentMapFunction to a Segment if it is available in the cache. If not present in any storage location, this method will not attempt to download the DataSegment from deep storage. The Segment returned by this method is considered an open reference, cache implementations must not allow it to be dropped until it has been closed. As such, the returned Segment must be closed when the caller is finished doing segment things.

Specified by:

acquireCachedSegment in interface SegmentCacheManager

acquireSegment

public AcquireSegmentAction acquireSegment(DataSegment dataSegment)

Description copied from interface: SegmentCacheManager

Returns a AcquireSegmentAction for a given DataSegment and SegmentDescriptor, which returns a reference provider for the Segment if already present in the cache, or tries to fetch from deep storage and map if not. The Segment returned by the provider returned by this method are considered an open reference, cache implementations must not allow the segment to be dropped until it has been closed. As such, the returned Segment from ReferenceCountedObjectProvider.acquireReference() must be closed when the caller is finished doing segment things.

Specified by:

acquireSegment in interface SegmentCacheManager

load

public void load(DataSegment dataSegment)
          throws SegmentLoadingException

Description copied from interface: SegmentCacheManager

Given a DataSegment, which contains the instructions for where and how to fetch a Segment from deep storage, this method tries to load and subsequently serve it to callers via SegmentCacheManager.acquireCachedSegment(SegmentId) or SegmentCacheManager.acquireSegment(DataSegment). If the segment cannot be loaded either due to error or insufficient storage space, this method throws a SegmentLoadingException.

Specified by:

load in interface SegmentCacheManager

Parameters:

dataSegment - Segment to get on each download (after service bootstrap)

Throws:

SegmentLoadingException - If there is an error in loading the segment or insufficient storage space

See Also:

• SegmentCacheManager.bootstrap(DataSegment, SegmentLazyLoadFailCallback)

bootstrap

public void bootstrap(DataSegment dataSegment,
 SegmentLazyLoadFailCallback loadFailed)
               throws SegmentLoadingException

Description copied from interface: SegmentCacheManager

Similar to SegmentCacheManager.load(DataSegment), this method loads segments during startup on data nodes. Implementations of this method may be configured to use a larger than normal work pool that only exists during startup and is shutdown after startup by calling SegmentCacheManager.shutdownBootstrap()

Specified by:

bootstrap in interface SegmentCacheManager

Parameters:

dataSegment - Segment to retrieve during service bootstrap

loadFailed - Callback to execute when segment lazy load failed. This applies only when lazyLoadOnStart is enabled

Throws:

SegmentLoadingException - - If there is an error in loading the segment or insufficient storage space

See Also:

• SegmentCacheManager.load(DataSegment)
• SegmentCacheManager.shutdownBootstrap()

getSegmentFiles

@Nullable
public File getSegmentFiles(DataSegment segment)

Description copied from interface: SegmentCacheManager

Alternative to SegmentCacheManager.acquireCachedSegment(SegmentId), to return the File location of the segment files stored in the cache, instead of a Optional<Segment>. Unlike SegmentCacheManager.acquireCachedSegment(SegmentId) and SegmentCacheManager.acquireSegment(DataSegment), this method does not provide any protections for callers, and should only be used by callers that are in control of when SegmentCacheManager.drop(DataSegment) is called. This method will not download the segment files from deep storage if they do not already exist in the cache, callers should use SegmentCacheManager.load(DataSegment) before calling this method.

Specified by:

getSegmentFiles in interface SegmentCacheManager

drop

public void drop(DataSegment segment)

Description copied from interface: SegmentCacheManager

Cleanup the segment files cache space used by the segment, releasing the StorageLocation reservation

Specified by:

drop in interface SegmentCacheManager

See Also:

• SegmentCacheManager.removeInfoFile(DataSegment)

shutdownBootstrap

public void shutdownBootstrap()

Description copied from interface: SegmentCacheManager

Shutdown any previously set-up bootstrap executor to save resources. This should be called after loading bootstrap segments.

Specified by:

shutdownBootstrap in interface SegmentCacheManager

shutdown

public void shutdown()

Specified by:

shutdown in interface SegmentCacheManager

getStorageStats

@Nullable
public StorageStats getStorageStats()

Description copied from interface: SegmentCacheManager

Collect StorageStats, if available.

Specified by:

getStorageStats in interface SegmentCacheManager

getSegmentLocks

public ConcurrentHashMap<DataSegment,org.apache.druid.segment.loading.SegmentLocalCacheManager.ReferenceCountingLock> getSegmentLocks()

getSegmentReferenceProvider

@Nullable
public ReferenceCountedSegmentProvider getSegmentReferenceProvider(DataSegment segment)

Testing use only please, any callers that want to do stuff with segments should use acquireCachedSegment(SegmentId) or acquireSegment(DataSegment) instead. Does not hold locks and so is not really safe to use while the cache manager is active