Package org.apache.flink.connector.file.table.batch.compact

Class BatchCompactOperator<T>

  • All Implemented Interfaces:
    Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.BoundedOneInput, org.apache.flink.streaming.api.operators.Input<CompactMessages.CoordinatorOutput>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.OneInputStreamOperator<CompactMessages.CoordinatorOutput,​CompactMessages.CompactOutput>, org.apache.flink.streaming.api.operators.StreamOperator<CompactMessages.CompactOutput>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.YieldingOperator<CompactMessages.CompactOutput>
    public class BatchCompactOperator<T>
extends org.apache.flink.streaming.api.operators.AbstractStreamOperator<CompactMessages.CompactOutput>
implements org.apache.flink.streaming.api.operators.OneInputStreamOperator<CompactMessages.CoordinatorOutput,​CompactMessages.CompactOutput>, org.apache.flink.streaming.api.operators.BoundedOneInput
    CompactOperator for compaction in batch mode. It will compact files to a target file and then emit the compacted file's path to downstream operator. The main logic is similar to CompactOperator but skip some unnecessary operations in batch mode.

    Note: if the size of the files to be compacted is 1, this operator won't do anything and just emit the file to downstream. Also, the name of the files to be compacted is not a hidden file, it's expected these files are in hidden or temporary directory. Please make sure it. This assumption can help skip rename hidden file.

    See Also:
    Serialized Form

    • Field Summary

      Fields 
      Modifier and Type Field Description
      static String ATTEMPT_PREFIX  
      static String COMPACTED_PREFIX  
      static String UNCOMPACTED_PREFIX  

      • Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

        combinedWatermark, config, lastRecordAttributes1, lastRecordAttributes2, latencyStats, LOG, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void close()  
      static org.apache.flink.core.fs.Path convertFromUncompacted​(org.apache.flink.core.fs.Path path)  
      void endInput()  
      void open()  
      void processElement​(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<CompactMessages.CoordinatorOutput> element)  

      • Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

        beforeInitializeStateHandler, finish, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, initializeState, isAsyncKeyOrderedProcessingEnabled, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark, processWatermark, processWatermark1, processWatermark1, processWatermark2, processWatermark2, processWatermarkStatus, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, setup, snapshotState, snapshotState, useInterruptibleTimers

      • Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

        notifyCheckpointAborted, notifyCheckpointComplete

      • Methods inherited from interface org.apache.flink.streaming.api.operators.Input

        processLatencyMarker, processRecordAttributes, processWatermark, processWatermark, processWatermarkStatus

      • Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

        getCurrentKey, setCurrentKey

      • Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

        hasKeyContext

      • Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

        setKeyContextElement

      • Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

        finish, getMetricGroup, getOperatorAttributes, getOperatorID, initializeState, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState

    • Method Detail

      • open

        public void open()
          throws Exception
        Specified by:
        open in interface org.apache.flink.streaming.api.operators.StreamOperator<T>
        Overrides:
        open in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<CompactMessages.CompactOutput>
        Throws:
        Exception

      • processElement

        public void processElement​(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<CompactMessages.CoordinatorOutput> element)
                    throws Exception
        Specified by:
        processElement in interface org.apache.flink.streaming.api.operators.Input<T>
        Throws:
        Exception

      • endInput

        public void endInput()
              throws Exception
        Specified by:
        endInput in interface org.apache.flink.streaming.api.operators.BoundedOneInput
        Throws:
        Exception

      • close

        public void close()
           throws Exception
        Specified by:
        close in interface org.apache.flink.streaming.api.operators.StreamOperator<T>
        Overrides:
        close in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<CompactMessages.CompactOutput>
        Throws:
        Exception

      • convertFromUncompacted

        public static org.apache.flink.core.fs.Path convertFromUncompacted​(org.apache.flink.core.fs.Path path)