Package org.apache.druid.metadata

Class SqlSegmentsMetadataManager

java.lang.Object

org.apache.druid.metadata.SqlSegmentsMetadataManager

All Implemented Interfaces:

SegmentsMetadataManager

public class SqlSegmentsMetadataManager
extends Object
implements SegmentsMetadataManager

Constructor Summary

Constructors

Constructor	Description
SqlSegmentsMetadataManager(com.fasterxml.jackson.databind.ObjectMapper jsonMapper, com.google.common.base.Supplier<SegmentsMetadataManagerConfig> config, com.google.common.base.Supplier<org.apache.druid.metadata.MetadataStorageTablesConfig> dbTables, SQLMetadataConnector connector)

Method Summary

Modifier and Type	Method	Description
Collection<ImmutableDruidDataSource>	getImmutableDataSourcesWithAllUsedSegments()	Returns a set of ImmutableDruidDataSource objects containing information about all used segments.
ImmutableDruidDataSource	getImmutableDataSourceWithUsedSegments(String dataSourceName)	If there are used segments belonging to the given data source this method returns them as an ImmutableDruidDataSource object.
DataSourcesSnapshot	getSnapshotOfDataSourcesWithAllUsedSegments()	Returns a snapshot of DruidDataSources and overshadowed segments
List<org.joda.time.Interval>	getUnusedSegmentIntervals(String dataSource, org.joda.time.DateTime minStartTime, org.joda.time.DateTime maxEndTime, int limit, org.joda.time.DateTime maxUsedFlagLastUpdatedTime)	Returns top N unused segment intervals with the start time no earlier than the specified start time (if not null) and with the end time no later than the specified maxEndTime and with sed_status_last_updated time no later than maxLastUsedTime when ordered by segment start time, end time.
boolean	isPollingDatabasePeriodically()
com.google.common.base.Optional<Iterable<org.apache.druid.timeline.DataSegment>>	iterateAllUsedNonOvershadowedSegmentsForDatasourceInterval(String datasource, org.joda.time.Interval interval, boolean requiresLatest)	Returns an iterable to go over all used and non-overshadowed segments of given data sources over given interval.
Iterable<org.apache.druid.timeline.DataSegment>	iterateAllUsedSegments()	Returns an iterable to go over all segments in all data sources.
int	markAsUnusedAllSegmentsInDataSource(String dataSource)	Returns the number of segment entries in the database whose state was changed as the result of this call (that is, the segments were marked as unused).
int	markAsUnusedSegmentsInInterval(String dataSourceName, org.joda.time.Interval interval)
int	markAsUsedAllNonOvershadowedSegmentsInDataSource(String dataSource)	Returns the number of segment entries in the database whose state was changed as the result of this call (that is, the segments were marked as used).
int	markAsUsedNonOvershadowedSegments(String dataSource, Set<String> segmentIds)
int	markAsUsedNonOvershadowedSegmentsInInterval(String dataSource, org.joda.time.Interval interval)
boolean	markSegmentAsUnused(org.apache.druid.timeline.SegmentId segmentId)	This method does not update dataSourcesSnapshot, see the comments in doPoll() about snapshot update.
boolean	markSegmentAsUsed(String segmentId)	Returns true if the state of the segment entry is changed in the database as the result of this call (that is, the segment was marked as used), false otherwise.
int	markSegmentsAsUnused(Set<org.apache.druid.timeline.SegmentId> segmentIds)
void	poll()
void	populateUsedFlagLastUpdatedAsync()	Populates used_status_last_updated column in the segments table iteratively until there are no segments with a NULL value for that column.
Set<String>	retrieveAllDataSourceNames()	Retrieves all data source names for which there are segment in the database, regardless of whether those segments are used or not.
void	start()	Don't confuse this method with startPollingDatabasePeriodically().
void	startPollingDatabasePeriodically()
void	stop()	Don't confuse this method with stopPollingDatabasePeriodically().
void	stopAsyncUsedFlagLastUpdatedUpdate()
void	stopPollingDatabasePeriodically()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SqlSegmentsMetadataManager

@Inject
public SqlSegmentsMetadataManager(com.fasterxml.jackson.databind.ObjectMapper jsonMapper,
                                  com.google.common.base.Supplier<SegmentsMetadataManagerConfig> config,
                                  com.google.common.base.Supplier<org.apache.druid.metadata.MetadataStorageTablesConfig> dbTables,
                                  SQLMetadataConnector connector)

Method Detail

start

public void start()

Don't confuse this method with startPollingDatabasePeriodically(). This is a lifecycle starting method to be executed just once for an instance of SqlSegmentsMetadataManager.

stop

public void stop()

Don't confuse this method with stopPollingDatabasePeriodically(). This is a lifecycle stopping method to be executed just once for an instance of SqlSegmentsMetadataManager.

startPollingDatabasePeriodically

public void startPollingDatabasePeriodically()

Specified by:

startPollingDatabasePeriodically in interface SegmentsMetadataManager

stopAsyncUsedFlagLastUpdatedUpdate

public void stopAsyncUsedFlagLastUpdatedUpdate()

Specified by:

stopAsyncUsedFlagLastUpdatedUpdate in interface SegmentsMetadataManager

populateUsedFlagLastUpdatedAsync

public void populateUsedFlagLastUpdatedAsync()

Description copied from interface: SegmentsMetadataManager

Populates used_status_last_updated column in the segments table iteratively until there are no segments with a NULL value for that column.

Specified by:

populateUsedFlagLastUpdatedAsync in interface SegmentsMetadataManager

isPollingDatabasePeriodically

public boolean isPollingDatabasePeriodically()

Specified by:

isPollingDatabasePeriodically in interface SegmentsMetadataManager

stopPollingDatabasePeriodically

public void stopPollingDatabasePeriodically()

Specified by:

stopPollingDatabasePeriodically in interface SegmentsMetadataManager

markSegmentAsUsed

public boolean markSegmentAsUsed(String segmentId)

Description copied from interface: SegmentsMetadataManager

Returns true if the state of the segment entry is changed in the database as the result of this call (that is, the segment was marked as used), false otherwise. If the call results in a database error, an exception is relayed to the caller.

Specified by:

markSegmentAsUsed in interface SegmentsMetadataManager

markAsUsedAllNonOvershadowedSegmentsInDataSource

public int markAsUsedAllNonOvershadowedSegmentsInDataSource(String dataSource)

Description copied from interface: SegmentsMetadataManager

Returns the number of segment entries in the database whose state was changed as the result of this call (that is, the segments were marked as used). If the call results in a database error, an exception is relayed to the caller.

Specified by:

markAsUsedAllNonOvershadowedSegmentsInDataSource in interface SegmentsMetadataManager

markAsUsedNonOvershadowedSegmentsInInterval

public int markAsUsedNonOvershadowedSegmentsInInterval(String dataSource,
                                                       org.joda.time.Interval interval)

Specified by:

markAsUsedNonOvershadowedSegmentsInInterval in interface SegmentsMetadataManager

markAsUsedNonOvershadowedSegments

public int markAsUsedNonOvershadowedSegments(String dataSource,
                                             Set<String> segmentIds)
                                      throws UnknownSegmentIdsException

Specified by:

markAsUsedNonOvershadowedSegments in interface SegmentsMetadataManager

Throws:

UnknownSegmentIdsException

markAsUnusedAllSegmentsInDataSource

public int markAsUnusedAllSegmentsInDataSource(String dataSource)

Description copied from interface: SegmentsMetadataManager

Returns the number of segment entries in the database whose state was changed as the result of this call (that is, the segments were marked as unused). If the call results in a database error, an exception is relayed to the caller.

Specified by:

markAsUnusedAllSegmentsInDataSource in interface SegmentsMetadataManager

markSegmentAsUnused

public boolean markSegmentAsUnused(org.apache.druid.timeline.SegmentId segmentId)

This method does not update dataSourcesSnapshot, see the comments in doPoll() about snapshot update. The update of the segment's state will be reflected after the next SqlSegmentsMetadataManager.DatabasePoll.

Specified by:

markSegmentAsUnused in interface SegmentsMetadataManager

markSegmentsAsUnused

public int markSegmentsAsUnused(Set<org.apache.druid.timeline.SegmentId> segmentIds)

Specified by:

markSegmentsAsUnused in interface SegmentsMetadataManager

markAsUnusedSegmentsInInterval

public int markAsUnusedSegmentsInInterval(String dataSourceName,
                                          org.joda.time.Interval interval)

Specified by:

markAsUnusedSegmentsInInterval in interface SegmentsMetadataManager

getImmutableDataSourceWithUsedSegments

@Nullable
public ImmutableDruidDataSource getImmutableDataSourceWithUsedSegments(String dataSourceName)

Description copied from interface: SegmentsMetadataManager

If there are used segments belonging to the given data source this method returns them as an ImmutableDruidDataSource object. If there are no used segments belonging to the given data source this method returns null.

Specified by:

getImmutableDataSourceWithUsedSegments in interface SegmentsMetadataManager

getImmutableDataSourcesWithAllUsedSegments

public Collection<ImmutableDruidDataSource> getImmutableDataSourcesWithAllUsedSegments()

Description copied from interface: SegmentsMetadataManager

Returns a set of ImmutableDruidDataSource objects containing information about all used segments. ImmutableDruidDataSource objects in the returned collection are unique. If there are no used segments, this method returns an empty collection.

Specified by:

getImmutableDataSourcesWithAllUsedSegments in interface SegmentsMetadataManager

getSnapshotOfDataSourcesWithAllUsedSegments

public DataSourcesSnapshot getSnapshotOfDataSourcesWithAllUsedSegments()

Description copied from interface: SegmentsMetadataManager

Returns a snapshot of DruidDataSources and overshadowed segments

Specified by:

getSnapshotOfDataSourcesWithAllUsedSegments in interface SegmentsMetadataManager

iterateAllUsedSegments

public Iterable<org.apache.druid.timeline.DataSegment> iterateAllUsedSegments()

Description copied from interface: SegmentsMetadataManager

Returns an iterable to go over all segments in all data sources. The order in which segments are iterated is unspecified. Note: the iteration may not be as trivially cheap as, for example, iteration over an ArrayList. Try (to some reasonable extent) to organize the code so that it iterates the returned iterable only once rather than several times.

Specified by:

iterateAllUsedSegments in interface SegmentsMetadataManager

iterateAllUsedNonOvershadowedSegmentsForDatasourceInterval

public com.google.common.base.Optional<Iterable<org.apache.druid.timeline.DataSegment>> iterateAllUsedNonOvershadowedSegmentsForDatasourceInterval(String datasource,
                                                                                                                                                   org.joda.time.Interval interval,
                                                                                                                                                   boolean requiresLatest)

Description copied from interface: SegmentsMetadataManager

Returns an iterable to go over all used and non-overshadowed segments of given data sources over given interval. The order in which segments are iterated is unspecified. Note: the iteration may not be as trivially cheap as, for example, iteration over an ArrayList. Try (to some reasonable extent) to organize the code so that it iterates the returned iterable only once rather than several times. If {@param requiresLatest} is true then a force metadatastore poll will be triggered. This can cause a longer response time but will ensure that the latest segment information (at the time this method is called) is returned. If {@param requiresLatest} is false then segment information from stale snapshot of up to the last periodic poll period periodicPollDelay will be used.

Specified by:

iterateAllUsedNonOvershadowedSegmentsForDatasourceInterval in interface SegmentsMetadataManager

retrieveAllDataSourceNames

public Set<String> retrieveAllDataSourceNames()

Description copied from interface: SegmentsMetadataManager

Retrieves all data source names for which there are segment in the database, regardless of whether those segments are used or not. If there are no segments in the database, returns an empty set. Performance warning: this method makes a query into the database. This method might return a different set of data source names than may be observed via SegmentsMetadataManager.getImmutableDataSourcesWithAllUsedSegments() method. This method will include a data source name even if there are no used segments belonging to it, while SegmentsMetadataManager.getImmutableDataSourcesWithAllUsedSegments() won't return such a data source.

Specified by:

retrieveAllDataSourceNames in interface SegmentsMetadataManager

poll

public void poll()

Specified by:

poll in interface SegmentsMetadataManager

getUnusedSegmentIntervals

public List<org.joda.time.Interval> getUnusedSegmentIntervals(String dataSource,
                                                              @Nullable
                                                              org.joda.time.DateTime minStartTime,
                                                              org.joda.time.DateTime maxEndTime,
                                                              int limit,
                                                              org.joda.time.DateTime maxUsedFlagLastUpdatedTime)

Description copied from interface: SegmentsMetadataManager

Returns top N unused segment intervals with the start time no earlier than the specified start time (if not null) and with the end time no later than the specified maxEndTime and with sed_status_last_updated time no later than maxLastUsedTime when ordered by segment start time, end time. Any segment having no used_status_last_updated time due to upgrade from legacy Druid means maxUsedFlagLastUpdatedTime is ignored for that segment.

Specified by:

getUnusedSegmentIntervals in interface SegmentsMetadataManager