Package org.apache.druid.sql.calcite.schema

Class BrokerSegmentMetadataCache

java.lang.Object

org.apache.druid.segment.metadata.AbstractSegmentMetadataCache<DatasourceTable.PhysicalDatasourceMetadata>

org.apache.druid.sql.calcite.schema.BrokerSegmentMetadataCache

public class BrokerSegmentMetadataCache
extends AbstractSegmentMetadataCache<DatasourceTable.PhysicalDatasourceMetadata>

Broker-side cache of segment metadata that combines segments to build datasources which become "tables" in Calcite. This cache provides the "physical" metadata about a dataSource which is blended with catalog "logical" metadata to provide the final user-view of each dataSource.

This class extends AbstractSegmentMetadataCache and introduces following changes,

• The refresh mechanism includes polling the coordinator for datasource schema, and falling back to running SegmentMetadataQuery.
• It builds and caches DatasourceTable.PhysicalDatasourceMetadata object for the table schema

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.druid.segment.metadata.AbstractSegmentMetadataCache

AbstractSegmentMetadataCache.ColumnTypeMergePolicy, AbstractSegmentMetadataCache.FirstTypeMergePolicy, AbstractSegmentMetadataCache.LeastRestrictiveTypeMergePolicy

Field Summary

Fields inherited from class org.apache.druid.segment.metadata.AbstractSegmentMetadataCache

cacheExec, callbackExec, dataSourcesNeedingRebuild, isServerViewInitialized, lock, mutableSegments, ROW_SIGNATURE_INTERNER, SEGMENT_ORDER, segmentMetadataInfo, segmentsNeedingRefresh, tables

Constructor Summary

Constructors

Constructor	Description
BrokerSegmentMetadataCache(QueryLifecycleFactory queryLifecycleFactory, TimelineServerView serverView, BrokerSegmentMetadataCacheConfig config, Escalator escalator, InternalQueryConfig internalQueryConfig, ServiceEmitter emitter, PhysicalDatasourceMetadataFactory dataSourceMetadataFactory, CoordinatorClient coordinatorClient, CentralizedDatasourceSchemaConfig centralizedDatasourceSchemaConfig)

Method Summary

Modifier and Type	Method	Description
void	refresh(Set<SegmentId> segmentsToRefresh, Set<String> dataSourcesToRebuild)	Refreshes the set of segments in two steps: Polls the coordinator for the datasource schema. Refreshes the remaining set of segments by executing a SegmentMetadataQuery and builds datasource schema by combining segment schema.
protected void	removeSegmentAction(SegmentId segmentId)
protected boolean	shouldRefresh()	Execute refresh on the broker in each cycle if CentralizedDatasourceSchema is enabled else if there are segments or datasources to be refreshed.
void	start()
void	stop()

Methods inherited from class org.apache.druid.segment.metadata.AbstractSegmentMetadataCache

addSegment, awaitInitialization, buildDataSourceRowSignature, cacheExecLoop, doInLock, emitMetric, emitMetric, fetchAggregatorsInSegmentMetadataQuery, getAvailableSegmentMetadata, getDatasource, getDataSourceInformationMap, getDatasourceNames, getDataSourcesNeedingRebuild, getMutableSegments, getSegmentMetadataSnapshot, getSegmentsNeedingRefresh, getTotalSegments, iterateSegmentMetadata, markDataSourceAsNeedRebuild, markSegmentAsNeedRefresh, refreshSegments, refreshWaitCondition, removeSegment, removeServerSegment, runSegmentMetadataQuery, setAvailableSegmentMetadata, unmarkSegmentAsMutable, updateSegmentMetadata

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

BrokerSegmentMetadataCache

@Inject
public BrokerSegmentMetadataCache(QueryLifecycleFactory queryLifecycleFactory,
 TimelineServerView serverView,
 BrokerSegmentMetadataCacheConfig config,
 Escalator escalator,
 InternalQueryConfig internalQueryConfig,
 ServiceEmitter emitter,
 PhysicalDatasourceMetadataFactory dataSourceMetadataFactory,
 CoordinatorClient coordinatorClient,
 CentralizedDatasourceSchemaConfig centralizedDatasourceSchemaConfig)

Method Details

start

public void start()
           throws InterruptedException

Specified by:

start in class AbstractSegmentMetadataCache<DatasourceTable.PhysicalDatasourceMetadata>

Throws:

InterruptedException

stop

public void stop()

Specified by:

stop in class AbstractSegmentMetadataCache<DatasourceTable.PhysicalDatasourceMetadata>

shouldRefresh

protected boolean shouldRefresh()

Execute refresh on the broker in each cycle if CentralizedDatasourceSchema is enabled else if there are segments or datasources to be refreshed.

Overrides:

shouldRefresh in class AbstractSegmentMetadataCache<DatasourceTable.PhysicalDatasourceMetadata>

refresh

public void refresh(Set<SegmentId> segmentsToRefresh,
 Set<String> dataSourcesToRebuild)
             throws IOException

Refreshes the set of segments in two steps:

• Polls the coordinator for the datasource schema.
• Refreshes the remaining set of segments by executing a SegmentMetadataQuery and builds datasource schema by combining segment schema.

Specified by:

refresh in class AbstractSegmentMetadataCache<DatasourceTable.PhysicalDatasourceMetadata>

Parameters:

segmentsToRefresh - segments for which the schema might have changed

dataSourcesToRebuild - datasources for which the schema might have changed

Throws:

IOException - when querying segment schema from data nodes and tasks

removeSegmentAction

protected void removeSegmentAction(SegmentId segmentId)

Specified by:

removeSegmentAction in class AbstractSegmentMetadataCache<DatasourceTable.PhysicalDatasourceMetadata>