Package org.apache.flink.connector.file.table.stream.compact

Class CompactCoordinator

  • All Implemented Interfaces:
    Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.Input<CompactMessages.CoordinatorInput>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.OneInputStreamOperator<CompactMessages.CoordinatorInput,​CompactMessages.CoordinatorOutput>, org.apache.flink.streaming.api.operators.StreamOperator<CompactMessages.CoordinatorOutput>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.YieldingOperator<CompactMessages.CoordinatorOutput>
    @Internal
public class CompactCoordinator
extends org.apache.flink.streaming.api.operators.AbstractStreamOperator<CompactMessages.CoordinatorOutput>
implements org.apache.flink.streaming.api.operators.OneInputStreamOperator<CompactMessages.CoordinatorInput,​CompactMessages.CoordinatorOutput>
    This is the single (non-parallel) monitoring task which coordinate input files to compaction units. - Receives in-flight input files inside checkpoint. - Receives all upstream end input messages after the checkpoint completes successfully, starts coordination.

    CompactMessages.CompactionUnit and CompactMessages.EndCompaction must be sent to the downstream in an orderly manner, while CompactMessages.EndCompaction is broadcast emitting, so unit and endCompaction use the broadcast emitting mechanism together. Since unit is broadcast, we want it to be processed by a single task, so we carry the ID in the unit and let the downstream task select its own unit.

    NOTE: The coordination is a stable algorithm, which can ensure that the downstream can perform compaction at any time without worrying about fail over.

    STATE: This operator stores input files in state, after the checkpoint completes successfully, input files are taken out from the state for coordination.

    See Also:
    Serialized Form

    • Field Summary

      • Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

        combinedWatermark, config, lastRecordAttributes1, lastRecordAttributes2, latencyStats, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager

    • Constructor Summary

      Constructors 
      Constructor Description
      CompactCoordinator​(org.apache.flink.util.function.SupplierWithException<org.apache.flink.core.fs.FileSystem,​IOException> fsFactory, long targetFileSize)  

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void initializeState​(org.apache.flink.runtime.state.StateInitializationContext context)  
      void processElement​(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<CompactMessages.CoordinatorInput> element)  
      void snapshotState​(org.apache.flink.runtime.state.StateSnapshotContext context)  

      • Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

        beforeInitializeStateHandler, close, finish, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, isAsyncKeyOrderedProcessingEnabled, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, open, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark, processWatermark, processWatermark1, processWatermark1, processWatermark2, processWatermark2, processWatermarkStatus, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, setup, snapshotState, useInterruptibleTimers

      • Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

        notifyCheckpointAborted, notifyCheckpointComplete

      • Methods inherited from interface org.apache.flink.streaming.api.operators.Input

        processLatencyMarker, processRecordAttributes, processWatermark, processWatermark, processWatermarkStatus

      • Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

        getCurrentKey, setCurrentKey

      • Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

        hasKeyContext

      • Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

        setKeyContextElement

      • Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

        close, finish, getMetricGroup, getOperatorAttributes, getOperatorID, initializeState, open, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState

    • Constructor Detail

      • CompactCoordinator

        public CompactCoordinator​(org.apache.flink.util.function.SupplierWithException<org.apache.flink.core.fs.FileSystem,​IOException> fsFactory,
                          long targetFileSize)

    • Method Detail

      • initializeState

        public void initializeState​(org.apache.flink.runtime.state.StateInitializationContext context)
                     throws Exception
        Specified by:
        initializeState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
        Overrides:
        initializeState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<CompactMessages.CoordinatorOutput>
        Throws:
        Exception

      • snapshotState

        public void snapshotState​(org.apache.flink.runtime.state.StateSnapshotContext context)
                   throws Exception
        Specified by:
        snapshotState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
        Overrides:
        snapshotState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<CompactMessages.CoordinatorOutput>
        Throws:
        Exception