Package org.apache.flink.connector.file.sink.compactor.operator

Class CompactCoordinator

  • All Implemented Interfaces:
    Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.BoundedOneInput, org.apache.flink.streaming.api.operators.Input<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.OneInputStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>,​CompactorRequest>, org.apache.flink.streaming.api.operators.StreamOperator<CompactorRequest>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.YieldingOperator<CompactorRequest>
    @Internal
public class CompactCoordinator
extends org.apache.flink.streaming.api.operators.AbstractStreamOperator<CompactorRequest>
implements org.apache.flink.streaming.api.operators.OneInputStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>,​CompactorRequest>, org.apache.flink.streaming.api.operators.BoundedOneInput
    Coordinator that coordinates file compaction for the FileSink.

    All committable emitted from the writers are collected and packed into CompactorRequests. The FileCompactStrategy defines when the requests can be fired. When a firing condition is met, the requests will be sent to the CompactorOperator.

    The CompactCoordinator stores the non-fired committable as its state, and may emit a request at any time. A CompactorOperator must ensure that the ownership of the committable in a compact request is successfully handed from the coordinator, before it can actually perform the compaction.

    See Also:
    Serialized Form

    • Field Summary

      • Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

        combinedWatermark, config, lastRecordAttributes1, lastRecordAttributes2, latencyStats, LOG, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void endInput()  
      void initializeState​(org.apache.flink.runtime.state.StateInitializationContext context)  
      void prepareSnapshotPreBarrier​(long checkpointId)  
      void processElement​(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>> element)  
      void snapshotState​(org.apache.flink.runtime.state.StateSnapshotContext context)  

      • Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

        beforeInitializeStateHandler, close, finish, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, isAsyncKeyOrderedProcessingEnabled, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, open, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark, processWatermark, processWatermark1, processWatermark1, processWatermark2, processWatermark2, processWatermarkStatus, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, setup, snapshotState, useInterruptibleTimers

      • Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

        notifyCheckpointAborted, notifyCheckpointComplete

      • Methods inherited from interface org.apache.flink.streaming.api.operators.Input

        processLatencyMarker, processRecordAttributes, processWatermark, processWatermark, processWatermarkStatus

      • Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

        getCurrentKey, setCurrentKey

      • Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

        hasKeyContext

      • Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

        setKeyContextElement

      • Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

        close, finish, getMetricGroup, getOperatorAttributes, getOperatorID, initializeState, open, setKeyContextElement1, setKeyContextElement2, snapshotState

    • Constructor Detail

      • CompactCoordinator

        public CompactCoordinator​(org.apache.flink.streaming.api.operators.StreamOperatorParameters<CompactorRequest> parameters,
                          FileCompactStrategy strategy,
                          org.apache.flink.core.io.SimpleVersionedSerializer<FileSinkCommittable> committableSerializer)

    • Method Detail

      • processElement

        public void processElement​(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>> element)
                    throws Exception
        Specified by:
        processElement in interface org.apache.flink.streaming.api.operators.Input<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>
        Throws:
        Exception

      • endInput

        public void endInput()
              throws Exception
        Specified by:
        endInput in interface org.apache.flink.streaming.api.operators.BoundedOneInput
        Throws:
        Exception

      • prepareSnapshotPreBarrier

        public void prepareSnapshotPreBarrier​(long checkpointId)
                               throws Exception
        Specified by:
        prepareSnapshotPreBarrier in interface org.apache.flink.streaming.api.operators.StreamOperator<CompactorRequest>
        Overrides:
        prepareSnapshotPreBarrier in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<CompactorRequest>
        Throws:
        Exception

      • snapshotState

        public void snapshotState​(org.apache.flink.runtime.state.StateSnapshotContext context)
                   throws Exception
        Specified by:
        snapshotState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
        Overrides:
        snapshotState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<CompactorRequest>
        Throws:
        Exception

      • initializeState

        public void initializeState​(org.apache.flink.runtime.state.StateInitializationContext context)
                     throws Exception
        Specified by:
        initializeState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
        Overrides:
        initializeState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<CompactorRequest>
        Throws:
        Exception