Package org.apache.flink.connector.kafka.dynamic.source.enumerator

Class StoppableKafkaEnumContextProxy

  • All Implemented Interfaces:
    AutoCloseable, org.apache.flink.api.connector.source.SplitEnumeratorContext<KafkaPartitionSplit>
    @Internal
public class StoppableKafkaEnumContextProxy
extends Object
implements org.apache.flink.api.connector.source.SplitEnumeratorContext<KafkaPartitionSplit>, AutoCloseable
    A proxy enumerator context that supports life cycle management of underlying threads related to a sub KafkaSourceEnumerator. This is motivated by the need to cancel the periodic partition discovery in scheduled tasks when sub Kafka Enumerators are restarted. The worker thread pool in SourceCoordinatorContext should not contain tasks of inactive KafkaSourceEnumerators, after source restart.

    Due to the inability to cancel scheduled tasks from SourceCoordinatorContext, this enumerator context will safely catch exceptions during enumerator restart and use a closeable proxy scheduler to invoke tasks on the coordinator main thread to maintain the single threaded property.

    • Constructor Detail

      • StoppableKafkaEnumContextProxy

        public StoppableKafkaEnumContextProxy​(String kafkaClusterId,
                                      KafkaMetadataService kafkaMetadataService,
                                      org.apache.flink.api.connector.source.SplitEnumeratorContext<DynamicKafkaSourceSplit> enumContext,
                                      @Nullable
                                      Runnable signalNoMoreSplitsCallback)
        Constructor for the enumerator context.
        Parameters:
        kafkaClusterId - The Kafka cluster id in order to maintain the mapping to the sub KafkaSourceEnumerator
        kafkaMetadataService - the Kafka metadata service to facilitate error handling
        enumContext - the underlying enumerator context
        signalNoMoreSplitsCallback - the callback when signal no more splits is invoked

    • Method Detail

      • metricGroup

        public org.apache.flink.metrics.groups.SplitEnumeratorMetricGroup metricGroup()
        Specified by:
        metricGroup in interface org.apache.flink.api.connector.source.SplitEnumeratorContext<KafkaPartitionSplit>

      • sendEventToSourceReader

        public void sendEventToSourceReader​(int subtaskId,
                                    org.apache.flink.api.connector.source.SourceEvent event)
        Specified by:
        sendEventToSourceReader in interface org.apache.flink.api.connector.source.SplitEnumeratorContext<KafkaPartitionSplit>

      • currentParallelism

        public int currentParallelism()
        Specified by:
        currentParallelism in interface org.apache.flink.api.connector.source.SplitEnumeratorContext<KafkaPartitionSplit>

      • registeredReaders

        public Map<Integer,​org.apache.flink.api.connector.source.ReaderInfo> registeredReaders()
        Specified by:
        registeredReaders in interface org.apache.flink.api.connector.source.SplitEnumeratorContext<KafkaPartitionSplit>

      • assignSplits

        public void assignSplits​(org.apache.flink.api.connector.source.SplitsAssignment<KafkaPartitionSplit> newSplitAssignments)
        Wrap splits with cluster metadata.
        Specified by:
        assignSplits in interface org.apache.flink.api.connector.source.SplitEnumeratorContext<KafkaPartitionSplit>

      • signalNoMoreSplits

        public void signalNoMoreSplits​(int subtask)
        Specified by:
        signalNoMoreSplits in interface org.apache.flink.api.connector.source.SplitEnumeratorContext<KafkaPartitionSplit>

      • callAsync

        public <T> void callAsync​(Callable<T> callable,
                          java.util.function.BiConsumer<T,​Throwable> handler)
        Execute the one time callables in the coordinator.
        Specified by:
        callAsync in interface org.apache.flink.api.connector.source.SplitEnumeratorContext<KafkaPartitionSplit>

      • callAsync

        public <T> void callAsync​(Callable<T> callable,
                          java.util.function.BiConsumer<T,​Throwable> handler,
                          long initialDelay,
                          long period)
        Schedule task via internal thread pool to proxy task so that the task handler callback can execute in the single threaded source coordinator thread pool to avoid synchronization needs.

        Having the scheduled task in the internal thread pool also allows us to cancel the task when the context needs to close due to dynamic enumerator restart.

        In the case of KafkaEnumerator partition discovery, the callback modifies KafkaEnumerator object state.

        Specified by:
        callAsync in interface org.apache.flink.api.connector.source.SplitEnumeratorContext<KafkaPartitionSplit>

      • runInCoordinatorThread

        public void runInCoordinatorThread​(Runnable runnable)
        Specified by:
        runInCoordinatorThread in interface org.apache.flink.api.connector.source.SplitEnumeratorContext<KafkaPartitionSplit>

      • isNoMoreSplits

        public boolean isNoMoreSplits()

      • close

        public void close()
           throws Exception
        Note that we can't close the source coordinator here, because these contexts can be closed during metadata change when the coordinator still needs to continue to run. We can only close the coordinator context in Flink job shutdown, which Flink will do for us. That's why there is the complexity of the internal thread pools in this class.

        TODO: Attach Flink JIRA ticket -- discuss with upstream how to cancel scheduled tasks belonging to enumerator.

        Specified by:
        close in interface AutoCloseable
        Throws:
        Exception

      • wrapCallAsyncCallable

        protected <T> Callable<T> wrapCallAsyncCallable​(Callable<T> callable)
        Wraps callable in call async executed in worker thread pool with exception propagation to optimize on doing IO in non-coordinator thread.

      • wrapCallAsyncCallableHandler

        protected <T> java.util.function.BiConsumer<T,​Throwable> wrapCallAsyncCallableHandler​(java.util.function.BiConsumer<T,​Throwable> mainHandler)
        Handle exception that is propagated by a callable, executed on coordinator thread. Depending on condition(s) the exception may be swallowed or forwarded. This is the Kafka topic partition discovery callable handler.