Package org.apache.flink.connector.file.table.stream

Class PartitionCommitter

  • All Implemented Interfaces:
    Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.Input<PartitionCommitInfo>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.OneInputStreamOperator<PartitionCommitInfo,​Void>, org.apache.flink.streaming.api.operators.StreamOperator<Void>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.YieldingOperator<Void>
    @Internal
public class PartitionCommitter
extends org.apache.flink.streaming.api.operators.AbstractStreamOperator<Void>
implements org.apache.flink.streaming.api.operators.OneInputStreamOperator<PartitionCommitInfo,​Void>
    Committer operator for partitions. This is the single (non-parallel) task. It collects all the partition information sent from upstream, and triggers the partition submission decision when it judges to collect the partitions from all tasks of a checkpoint.

    NOTE: It processes records after the checkpoint completes successfully. Receive records from upstream CheckpointListener.notifyCheckpointComplete(long).

    Processing steps: 1.Partitions are sent from upstream. Add partition to trigger. 2.TaskTracker say it have already received partition data from all tasks in a checkpoint. 3.Extracting committable partitions from PartitionCommitTrigger. 4.Using PartitionCommitPolicy chain to commit partitions.

    See Also:
    Serialized Form

    • Field Summary

      • Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

        combinedWatermark, config, lastRecordAttributes1, lastRecordAttributes2, latencyStats, LOG, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager

    • Constructor Summary

      Constructors 
      Constructor Description
      PartitionCommitter​(org.apache.flink.core.fs.Path locationPath, org.apache.flink.table.catalog.ObjectIdentifier tableIdentifier, List<String> partitionKeys, TableMetaStoreFactory metaStoreFactory, FileSystemFactory fsFactory, org.apache.flink.configuration.Configuration conf)  

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void initializeState​(org.apache.flink.runtime.state.StateInitializationContext context)  
      void processElement​(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<PartitionCommitInfo> element)  
      void processWatermark​(org.apache.flink.streaming.api.watermark.Watermark mark)  
      void snapshotState​(org.apache.flink.runtime.state.StateSnapshotContext context)  

      • Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

        beforeInitializeStateHandler, close, finish, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, isAsyncKeyOrderedProcessingEnabled, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, open, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark, processWatermark1, processWatermark1, processWatermark2, processWatermark2, processWatermarkStatus, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, setup, snapshotState, useInterruptibleTimers

      • Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

        notifyCheckpointAborted, notifyCheckpointComplete

      • Methods inherited from interface org.apache.flink.streaming.api.operators.Input

        processLatencyMarker, processRecordAttributes, processWatermark, processWatermarkStatus

      • Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

        getCurrentKey, setCurrentKey

      • Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

        hasKeyContext

      • Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

        setKeyContextElement

      • Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

        close, finish, getMetricGroup, getOperatorAttributes, getOperatorID, initializeState, open, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState

    • Constructor Detail

      • PartitionCommitter

        public PartitionCommitter​(org.apache.flink.core.fs.Path locationPath,
                          org.apache.flink.table.catalog.ObjectIdentifier tableIdentifier,
                          List<String> partitionKeys,
                          TableMetaStoreFactory metaStoreFactory,
                          FileSystemFactory fsFactory,
                          org.apache.flink.configuration.Configuration conf)

    • Method Detail

      • initializeState

        public void initializeState​(org.apache.flink.runtime.state.StateInitializationContext context)
                     throws Exception
        Specified by:
        initializeState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
        Overrides:
        initializeState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<Void>
        Throws:
        Exception

      • processElement

        public void processElement​(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<PartitionCommitInfo> element)
                    throws Exception
        Specified by:
        processElement in interface org.apache.flink.streaming.api.operators.Input<PartitionCommitInfo>
        Throws:
        Exception

      • processWatermark

        public void processWatermark​(org.apache.flink.streaming.api.watermark.Watermark mark)
                      throws Exception
        Specified by:
        processWatermark in interface org.apache.flink.streaming.api.operators.Input<PartitionCommitInfo>
        Overrides:
        processWatermark in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<Void>
        Throws:
        Exception

      • snapshotState

        public void snapshotState​(org.apache.flink.runtime.state.StateSnapshotContext context)
                   throws Exception
        Specified by:
        snapshotState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
        Overrides:
        snapshotState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<Void>
        Throws:
        Exception