Package org.apache.flink.connector.file.table.stream

Class AbstractStreamingWriter<IN,​OUT>

  • All Implemented Interfaces:
    Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.BoundedOneInput, org.apache.flink.streaming.api.operators.Input<IN>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.OneInputStreamOperator<IN,​OUT>, org.apache.flink.streaming.api.operators.StreamOperator<OUT>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.YieldingOperator<OUT>
    Direct Known Subclasses:
    CompactFileWriter, StreamingFileWriter
    public abstract class AbstractStreamingWriter<IN,​OUT>
extends org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
implements org.apache.flink.streaming.api.operators.OneInputStreamOperator<IN,​OUT>, org.apache.flink.streaming.api.operators.BoundedOneInput
    Operator for file system sink. It is a operator version of StreamingFileSink. It can send file and bucket information to downstream.
    See Also:
    Serialized Form

    • Field Summary

      Fields 
      Modifier and Type Field Description
      protected org.apache.flink.streaming.api.functions.sink.filesystem.Buckets<IN,​String> buckets  
      protected long currentWatermark  

      • Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

        combinedWatermark, config, lastRecordAttributes1, lastRecordAttributes2, latencyStats, LOG, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager

    • Constructor Summary

      Constructors 
      Constructor Description
      AbstractStreamingWriter​(long bucketCheckInterval, org.apache.flink.streaming.api.functions.sink.filesystem.legacy.StreamingFileSink.BucketsBuilder<IN,​String,​? extends org.apache.flink.streaming.api.functions.sink.filesystem.legacy.StreamingFileSink.BucketsBuilder<IN,​String,​?>> bucketsBuilder)  

    • Method Summary

      All Methods Instance Methods Abstract Methods Concrete Methods 
      Modifier and Type Method Description
      void close()  
      protected void commitUpToCheckpoint​(long checkpointId)
      Commit up to this checkpoint id.
      void endInput()  
      void initializeState​(org.apache.flink.runtime.state.StateInitializationContext context)  
      void notifyCheckpointComplete​(long checkpointId)  
      protected abstract void onPartFileOpened​(String partition, org.apache.flink.core.fs.Path newPath)
      Notifies a new file has been opened.
      protected abstract void partitionCreated​(String partition)
      Notifies a partition created.
      protected abstract void partitionInactive​(String partition)
      Notifies a partition become inactive.
      void processElement​(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN> element)  
      void processWatermark​(org.apache.flink.streaming.api.watermark.Watermark mark)  
      void snapshotState​(org.apache.flink.runtime.state.StateSnapshotContext context)  

      • Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

        beforeInitializeStateHandler, finish, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, isAsyncKeyOrderedProcessingEnabled, isUsingCustomRawKeyedState, notifyCheckpointAborted, open, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark, processWatermark1, processWatermark1, processWatermark2, processWatermark2, processWatermarkStatus, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, setup, snapshotState, useInterruptibleTimers

      • Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

        notifyCheckpointAborted

      • Methods inherited from interface org.apache.flink.streaming.api.operators.Input

        processLatencyMarker, processRecordAttributes, processWatermark, processWatermarkStatus

      • Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

        getCurrentKey, setCurrentKey

      • Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

        hasKeyContext

      • Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

        setKeyContextElement

      • Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

        finish, getMetricGroup, getOperatorAttributes, getOperatorID, initializeState, open, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState

    • Field Detail

      • buckets

        protected transient org.apache.flink.streaming.api.functions.sink.filesystem.Buckets<IN,​String> buckets

      • currentWatermark

        protected transient long currentWatermark

    • Constructor Detail

      • AbstractStreamingWriter

        public AbstractStreamingWriter​(long bucketCheckInterval,
                               org.apache.flink.streaming.api.functions.sink.filesystem.legacy.StreamingFileSink.BucketsBuilder<IN,​String,​? extends org.apache.flink.streaming.api.functions.sink.filesystem.legacy.StreamingFileSink.BucketsBuilder<IN,​String,​?>> bucketsBuilder)

    • Method Detail

      • partitionCreated

        protected abstract void partitionCreated​(String partition)
        Notifies a partition created.

      • partitionInactive

        protected abstract void partitionInactive​(String partition)
        Notifies a partition become inactive. A partition becomes inactive after all the records received so far have been committed.

      • onPartFileOpened

        protected abstract void onPartFileOpened​(String partition,
                                         org.apache.flink.core.fs.Path newPath)
        Notifies a new file has been opened.

        Note that this does not mean that the file has been created in the file system. It is only created logically and the actual file will be generated after it is committed.

      • commitUpToCheckpoint

        protected void commitUpToCheckpoint​(long checkpointId)
                             throws Exception
        Commit up to this checkpoint id.
        Throws:
        Exception

      • initializeState

        public void initializeState​(org.apache.flink.runtime.state.StateInitializationContext context)
                     throws Exception
        Specified by:
        initializeState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
        Overrides:
        initializeState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
        Throws:
        Exception

      • snapshotState

        public void snapshotState​(org.apache.flink.runtime.state.StateSnapshotContext context)
                   throws Exception
        Specified by:
        snapshotState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator
        Overrides:
        snapshotState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
        Throws:
        Exception

      • processWatermark

        public void processWatermark​(org.apache.flink.streaming.api.watermark.Watermark mark)
                      throws Exception
        Specified by:
        processWatermark in interface org.apache.flink.streaming.api.operators.Input<IN>
        Overrides:
        processWatermark in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
        Throws:
        Exception

      • processElement

        public void processElement​(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN> element)
                    throws Exception
        Specified by:
        processElement in interface org.apache.flink.streaming.api.operators.Input<IN>
        Throws:
        Exception

      • notifyCheckpointComplete

        public void notifyCheckpointComplete​(long checkpointId)
                              throws Exception
        Specified by:
        notifyCheckpointComplete in interface org.apache.flink.api.common.state.CheckpointListener
        Overrides:
        notifyCheckpointComplete in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
        Throws:
        Exception

      • endInput

        public void endInput()
              throws Exception
        Specified by:
        endInput in interface org.apache.flink.streaming.api.operators.BoundedOneInput
        Throws:
        Exception

      • close

        public void close()
           throws Exception
        Specified by:
        close in interface org.apache.flink.streaming.api.operators.StreamOperator<IN>
        Overrides:
        close in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
        Throws:
        Exception