software.amazon.kinesis.leases.dynamodb

Class DynamoDBLeaseManagementFactory

java.lang.Object

software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseManagementFactory

All Implemented Interfaces:

LeaseManagementFactory

public class DynamoDBLeaseManagementFactory
extends Object
implements LeaseManagementFactory

Constructor Summary

Constructors

Constructor and Description

DynamoDBLeaseManagementFactory(@NotNull software.amazon.awssdk.services.kinesis.KinesisAsyncClient kinesisClient, @NotNull software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient dynamoDBClient, @NotNull String tableName, @NotNull String workerIdentifier, @NotNull ExecutorService executorService, long failoverTimeMillis, boolean enablePriorityLeaseAssignment, long epsilonMillis, int maxLeasesForWorker, int maxLeasesToStealAtOneTime, int maxLeaseRenewalThreads, boolean cleanupLeasesUponShardCompletion, boolean ignoreUnexpectedChildShards, long shardSyncIntervalMillis, boolean consistentReads, long listShardsBackoffTimeMillis, int maxListShardsRetryAttempts, int maxCacheMissesBeforeReload, long listShardsCacheAllowedAgeInSeconds, int cacheMissWarningModulus, long initialLeaseTableReadCapacity, long initialLeaseTableWriteCapacity, TableCreatorCallback tableCreatorCallback, Duration dynamoDbRequestTimeout, software.amazon.awssdk.services.dynamodb.model.BillingMode billingMode, boolean leaseTableDeletionProtectionEnabled, boolean leaseTablePitrEnabled, Collection<software.amazon.awssdk.services.dynamodb.model.Tag> tags, @NotNull LeaseSerializer leaseSerializer, Function<StreamConfig,ShardDetector> customShardDetectorProvider, boolean isMultiStreamMode, LeaseCleanupConfig leaseCleanupConfig, LeaseManagementConfig.WorkerUtilizationAwareAssignmentConfig workerUtilizationAwareAssignmentConfig, LeaseManagementConfig.GracefulLeaseHandoffConfig gracefulLeaseHandoffConfig, long leaseAssignmentIntervalMillis) Constructor.

Method Summary

Modifier and Type	Method and Description
LeaseCleanupManager	createLeaseCleanupManager(MetricsFactory metricsFactory) LeaseCleanupManager cleans up leases in the lease table for shards which have either expired past the stream's retention period or have been completely processed.
LeaseCoordinator	createLeaseCoordinator(@NonNull MetricsFactory metricsFactory)
LeaseCoordinator	createLeaseCoordinator(@NonNull MetricsFactory metricsFactory, @NonNull ConcurrentMap<ShardInfo,ShardConsumer> shardInfoShardConsumerMap)
LeaseCoordinator	createLeaseCoordinator(MetricsFactory metricsFactory, ConcurrentMap<ShardInfo,ShardConsumer> shardInfoShardConsumerMap, StreamIdCacheManager streamIdCacheManager)
DynamoDBLeaseRefresher	createLeaseRefresher()
ShardDetector	createShardDetector(StreamConfig streamConfig) KinesisShardDetector supports reading from service only using streamName.
ShardSyncTaskManager	createShardSyncTaskManager(MetricsFactory metricsFactory, StreamConfig streamConfig, DeletedStreamListProvider deletedStreamListProvider) Create ShardSyncTaskManager from the streamConfig passed
ShardSyncTaskManager	createShardSyncTaskManager(MetricsFactory metricsFactory, StreamConfig streamConfig, DeletedStreamListProvider deletedStreamListProvider, StreamInfoManager streamInfoManager)
StreamIdCacheManager	createStreamIdCacheManager(StreamInfoDAO streamInfoDAO, Map<StreamIdentifier,StreamConfig> currentStreamConfigMap, StreamIdOnboardingState streamIdOnboardingState, boolean isMultiStreamMode, MetricsFactory metricsFactory)
StreamInfoManager	createStreamInfoManager(Map<StreamIdentifier,StreamConfig> currentStreamConfigMap, StreamInfoDAO streamInfoDAO, MetricsFactory metricsFactory, boolean isMultiStreamMode, long streamInfoBackfillIntervalMillis, StreamInfoMode streamInfoMode, StreamIdOnboardingState streamIdOnboardingState)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface software.amazon.kinesis.leases.LeaseManagementFactory

createShardDetector, createShardSyncTaskManager, createShardSyncTaskManager

Constructor Detail

DynamoDBLeaseManagementFactory

public DynamoDBLeaseManagementFactory(@NotNull
                                      @NotNull software.amazon.awssdk.services.kinesis.KinesisAsyncClient kinesisClient,
                                      @NotNull
                                      @NotNull software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient dynamoDBClient,
                                      @NotNull
                                      @NotNull String tableName,
                                      @NotNull
                                      @NotNull String workerIdentifier,
                                      @NotNull
                                      @NotNull ExecutorService executorService,
                                      long failoverTimeMillis,
                                      boolean enablePriorityLeaseAssignment,
                                      long epsilonMillis,
                                      int maxLeasesForWorker,
                                      int maxLeasesToStealAtOneTime,
                                      int maxLeaseRenewalThreads,
                                      boolean cleanupLeasesUponShardCompletion,
                                      boolean ignoreUnexpectedChildShards,
                                      long shardSyncIntervalMillis,
                                      boolean consistentReads,
                                      long listShardsBackoffTimeMillis,
                                      int maxListShardsRetryAttempts,
                                      int maxCacheMissesBeforeReload,
                                      long listShardsCacheAllowedAgeInSeconds,
                                      int cacheMissWarningModulus,
                                      long initialLeaseTableReadCapacity,
                                      long initialLeaseTableWriteCapacity,
                                      TableCreatorCallback tableCreatorCallback,
                                      Duration dynamoDbRequestTimeout,
                                      software.amazon.awssdk.services.dynamodb.model.BillingMode billingMode,
                                      boolean leaseTableDeletionProtectionEnabled,
                                      boolean leaseTablePitrEnabled,
                                      Collection<software.amazon.awssdk.services.dynamodb.model.Tag> tags,
                                      @NotNull
                                      @NotNull LeaseSerializer leaseSerializer,
                                      Function<StreamConfig,ShardDetector> customShardDetectorProvider,
                                      boolean isMultiStreamMode,
                                      LeaseCleanupConfig leaseCleanupConfig,
                                      LeaseManagementConfig.WorkerUtilizationAwareAssignmentConfig workerUtilizationAwareAssignmentConfig,
                                      LeaseManagementConfig.GracefulLeaseHandoffConfig gracefulLeaseHandoffConfig,
                                      long leaseAssignmentIntervalMillis)

Constructor.

Parameters:

kinesisClient -

dynamoDBClient -

tableName -

workerIdentifier -

executorService -

failoverTimeMillis -

enablePriorityLeaseAssignment -

epsilonMillis -

maxLeasesForWorker -

maxLeasesToStealAtOneTime -

maxLeaseRenewalThreads -

cleanupLeasesUponShardCompletion -

ignoreUnexpectedChildShards -

shardSyncIntervalMillis -

consistentReads -

listShardsBackoffTimeMillis -

maxListShardsRetryAttempts -

maxCacheMissesBeforeReload -

listShardsCacheAllowedAgeInSeconds -

cacheMissWarningModulus -

initialLeaseTableReadCapacity -

initialLeaseTableWriteCapacity -

tableCreatorCallback -

dynamoDbRequestTimeout -

billingMode -

leaseTableDeletionProtectionEnabled -

leaseTablePitrEnabled -

leaseSerializer -

customShardDetectorProvider -

isMultiStreamMode -

leaseCleanupConfig -

workerUtilizationAwareAssignmentConfig -

gracefulLeaseHandoffConfig -

leaseAssignmentIntervalMillis -

Method Detail

createLeaseCoordinator

public LeaseCoordinator createLeaseCoordinator(@NonNull
                                               @NonNull MetricsFactory metricsFactory)

Specified by:

createLeaseCoordinator in interface LeaseManagementFactory

createLeaseCoordinator

public LeaseCoordinator createLeaseCoordinator(@NonNull
                                               @NonNull MetricsFactory metricsFactory,
                                               @NonNull
                                               @NonNull ConcurrentMap<ShardInfo,ShardConsumer> shardInfoShardConsumerMap)

Specified by:

createLeaseCoordinator in interface LeaseManagementFactory

createLeaseCoordinator

public LeaseCoordinator createLeaseCoordinator(MetricsFactory metricsFactory,
                                               ConcurrentMap<ShardInfo,ShardConsumer> shardInfoShardConsumerMap,
                                               StreamIdCacheManager streamIdCacheManager)

Specified by:

createLeaseCoordinator in interface LeaseManagementFactory

createShardSyncTaskManager

public ShardSyncTaskManager createShardSyncTaskManager(MetricsFactory metricsFactory,
                                                       StreamConfig streamConfig,
                                                       DeletedStreamListProvider deletedStreamListProvider)

Create ShardSyncTaskManager from the streamConfig passed

Specified by:

createShardSyncTaskManager in interface LeaseManagementFactory

Parameters:

metricsFactory - - factory to get metrics object

streamConfig - - streamConfig for which ShardSyncTaskManager needs to be created

deletedStreamListProvider - - store for capturing the streams which are deleted in kinesis

Returns:

ShardSyncTaskManager

createShardSyncTaskManager

public ShardSyncTaskManager createShardSyncTaskManager(MetricsFactory metricsFactory,
                                                       StreamConfig streamConfig,
                                                       DeletedStreamListProvider deletedStreamListProvider,
                                                       StreamInfoManager streamInfoManager)

Specified by:

createShardSyncTaskManager in interface LeaseManagementFactory

createLeaseRefresher

public DynamoDBLeaseRefresher createLeaseRefresher()

Specified by:

createLeaseRefresher in interface LeaseManagementFactory

createShardDetector

public ShardDetector createShardDetector(StreamConfig streamConfig)

KinesisShardDetector supports reading from service only using streamName. Support for accountId and stream creation epoch is yet to be provided.

Specified by:

createShardDetector in interface LeaseManagementFactory

Parameters:

streamConfig -

Returns:

ShardDetector

createLeaseCleanupManager

public LeaseCleanupManager createLeaseCleanupManager(MetricsFactory metricsFactory)

LeaseCleanupManager cleans up leases in the lease table for shards which have either expired past the stream's retention period or have been completely processed.

Specified by:

createLeaseCleanupManager in interface LeaseManagementFactory

Parameters:

metricsFactory -

Returns:

LeaseCleanupManager

createStreamIdCacheManager

public StreamIdCacheManager createStreamIdCacheManager(StreamInfoDAO streamInfoDAO,
                                                       Map<StreamIdentifier,StreamConfig> currentStreamConfigMap,
                                                       StreamIdOnboardingState streamIdOnboardingState,
                                                       boolean isMultiStreamMode,
                                                       MetricsFactory metricsFactory)

Specified by:

createStreamIdCacheManager in interface LeaseManagementFactory

createStreamInfoManager

public StreamInfoManager createStreamInfoManager(Map<StreamIdentifier,StreamConfig> currentStreamConfigMap,
                                                 StreamInfoDAO streamInfoDAO,
                                                 MetricsFactory metricsFactory,
                                                 boolean isMultiStreamMode,
                                                 long streamInfoBackfillIntervalMillis,
                                                 StreamInfoMode streamInfoMode,
                                                 StreamIdOnboardingState streamIdOnboardingState)

Specified by:

createStreamInfoManager in interface LeaseManagementFactory