Package org.apache.flink.connector.file.sink.compactor.operator

Class CompactorOperator

  • All Implemented Interfaces:
    Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.BoundedOneInput, org.apache.flink.streaming.api.operators.Input<CompactorRequest>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.OneInputStreamOperator<CompactorRequest,​org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>, org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.YieldingOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>
    @Internal
public class CompactorOperator
extends org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>
implements org.apache.flink.streaming.api.operators.OneInputStreamOperator<CompactorRequest,​org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>, org.apache.flink.streaming.api.operators.BoundedOneInput, org.apache.flink.api.common.state.CheckpointListener
    An operator that perform compaction for the FileSink.

    Requests received from the CompactCoordinator will firstly be held in memory, and snapshot into the state of a checkpoint. When the checkpoint is successfully completed, all requests received before can be submitted. The results can be emitted at the next prepareSnapshotPreBarrier(long) invoking after the compaction is finished, to ensure that committers can receive only one CommittableSummary and the corresponding number of Committable for a single checkpoint.

    See Also:
    Serialized Form

    • Field Summary

      • Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

        combinedWatermark, config, lastRecordAttributes1, lastRecordAttributes2, latencyStats, LOG, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager

    • Constructor Summary

      Constructors 
      Constructor Description
      CompactorOperator​(org.apache.flink.streaming.api.operators.StreamOperatorParameters<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>> parameters, FileCompactStrategy strategy, org.apache.flink.core.io.SimpleVersionedSerializer<FileSinkCommittable> committableSerializer, FileCompactor fileCompactor, org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter<?,​String> bucketWriter)  

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void close()  
      void endInput()  
      CompletableFuture<?> getAllTasksFuture()  
      void initializeState​(org.apache.flink.runtime.state.StateInitializationContext context)  
      void notifyCheckpointComplete​(long checkpointId)  
      void open()  
      void prepareSnapshotPreBarrier​(long checkpointId)  
      void processElement​(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<CompactorRequest> element)  
      void snapshotState​(org.apache.flink.runtime.state.StateSnapshotContext context)  

      • Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

        beforeInitializeStateHandler, finish, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, isAsyncKeyOrderedProcessingEnabled, isUsingCustomRawKeyedState, notifyCheckpointAborted, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark, processWatermark, processWatermark1, processWatermark1, processWatermark2, processWatermark2, processWatermarkStatus, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, setup, snapshotState, useInterruptibleTimers

      • Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

        notifyCheckpointAborted

      • Methods inherited from interface org.apache.flink.streaming.api.operators.Input

        processLatencyMarker, processRecordAttributes, processWatermark, processWatermark, processWatermarkStatus

      • Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

        getCurrentKey, setCurrentKey

      • Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

        hasKeyContext

      • Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

        setKeyContextElement

      • Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

        finish, getMetricGroup, getOperatorAttributes, getOperatorID, initializeState, setKeyContextElement1, setKeyContextElement2, snapshotState

    • Constructor Detail

      • CompactorOperator

        public CompactorOperator​(org.apache.flink.streaming.api.operators.StreamOperatorParameters<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>> parameters,
                         FileCompactStrategy strategy,
                         org.apache.flink.core.io.SimpleVersionedSerializer<FileSinkCommittable> committableSerializer,
                         FileCompactor fileCompactor,
                         org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter<?,​String> bucketWriter)

    • Method Detail

      • open

        public void open()
          throws Exception
        Specified by:
        open in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>
        Overrides:
        open in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>
        Throws:
        Exception

      • processElement

        public void processElement​(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<CompactorRequest> element)
                    throws Exception
        Specified by:
        processElement in interface org.apache.flink.streaming.api.operators.Input<CompactorRequest>
        Throws:
        Exception

      • endInput

        public void endInput()
              throws Exception
        Specified by:
        endInput in interface org.apache.flink.streaming.api.operators.BoundedOneInput
        Throws:
        Exception

      • notifyCheckpointComplete

        public void notifyCheckpointComplete​(long checkpointId)
                              throws Exception
        Specified by:
        notifyCheckpointComplete in interface org.apache.flink.api.common.state.CheckpointListener
        Overrides:
        notifyCheckpointComplete in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>
        Throws:
        Exception

      • prepareSnapshotPreBarrier

        public void prepareSnapshotPreBarrier​(long checkpointId)
                               throws Exception
        Specified by:
        prepareSnapshotPreBarrier in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>
        Overrides:
        prepareSnapshotPreBarrier in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>
        Throws:
        Exception

      • snapshotState

        public void snapshotState​(org.apache.flink.runtime.state.StateSnapshotContext context)
                   throws Exception
        Specified by:
        snapshotState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
        Overrides:
        snapshotState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>
        Throws:
        Exception

      • initializeState

        public void initializeState​(org.apache.flink.runtime.state.StateInitializationContext context)
                     throws Exception
        Specified by:
        initializeState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
        Overrides:
        initializeState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>
        Throws:
        Exception

      • close

        public void close()
           throws Exception
        Specified by:
        close in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>
        Overrides:
        close in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>
        Throws:
        Exception

      • getAllTasksFuture

        @VisibleForTesting
public CompletableFuture<?> getAllTasksFuture()