Package org.apache.flink.connector.file.table.stream

Class StreamingFileWriter<IN>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

org.apache.flink.connector.file.table.stream.AbstractStreamingWriter<IN,PartitionCommitInfo>

org.apache.flink.connector.file.table.stream.StreamingFileWriter<IN>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.BoundedOneInput, org.apache.flink.streaming.api.operators.Input<IN>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.OneInputStreamOperator<IN,PartitionCommitInfo>, org.apache.flink.streaming.api.operators.StreamOperator<PartitionCommitInfo>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.YieldingOperator<PartitionCommitInfo>

@Internal
public class StreamingFileWriter<IN>
extends AbstractStreamingWriter<IN,PartitionCommitInfo>

Writer for emitting PartitionCommitInfo to downstream.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.flink.connector.file.table.stream.AbstractStreamingWriter

buckets, currentWatermark

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

combinedWatermark, config, lastRecordAttributes1, lastRecordAttributes2, latencyStats, LOG, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager

Constructor Summary

Constructors

Constructor	Description
StreamingFileWriter(long bucketCheckInterval, org.apache.flink.streaming.api.functions.sink.filesystem.legacy.StreamingFileSink.BucketsBuilder<IN,String,? extends org.apache.flink.streaming.api.functions.sink.filesystem.legacy.StreamingFileSink.BucketsBuilder<IN,String,?>> bucketsBuilder, List<String> partitionKeys, org.apache.flink.configuration.Configuration conf)

Method Summary

Modifier and Type	Method	Description
protected void	commitUpToCheckpoint(long checkpointId)	Commit up to this checkpoint id.
void	initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
protected void	onPartFileOpened(String s, org.apache.flink.core.fs.Path newPath)	Notifies a new file has been opened.
protected void	partitionCreated(String partition)	Notifies a partition created.
protected void	partitionInactive(String partition)	Notifies a partition become inactive.
void	snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context)

Methods inherited from class org.apache.flink.connector.file.table.stream.AbstractStreamingWriter

close, endInput, notifyCheckpointComplete, processElement, processWatermark

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

beforeInitializeStateHandler, finish, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, isAsyncKeyOrderedProcessingEnabled, isUsingCustomRawKeyedState, notifyCheckpointAborted, open, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark, processWatermark1, processWatermark1, processWatermark2, processWatermark2, processWatermarkStatus, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, setup, snapshotState, useInterruptibleTimers

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted

Methods inherited from interface org.apache.flink.streaming.api.operators.Input

processLatencyMarker, processRecordAttributes, processWatermark, processWatermarkStatus

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

getCurrentKey, setCurrentKey

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

hasKeyContext

Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

setKeyContextElement

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

finish, getMetricGroup, getOperatorAttributes, getOperatorID, initializeState, open, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState

Constructor Detail

StreamingFileWriter

public StreamingFileWriter(long bucketCheckInterval,
                           org.apache.flink.streaming.api.functions.sink.filesystem.legacy.StreamingFileSink.BucketsBuilder<IN,String,? extends org.apache.flink.streaming.api.functions.sink.filesystem.legacy.StreamingFileSink.BucketsBuilder<IN,String,?>> bucketsBuilder,
                           List<String> partitionKeys,
                           org.apache.flink.configuration.Configuration conf)

Method Detail

initializeState

public void initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
                     throws Exception

Specified by:

initializeState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator

Overrides:

initializeState in class AbstractStreamingWriter<IN,PartitionCommitInfo>

Throws:

Exception

partitionCreated

protected void partitionCreated(String partition)

Description copied from class: AbstractStreamingWriter

Notifies a partition created.

Specified by:

partitionCreated in class AbstractStreamingWriter<IN,PartitionCommitInfo>

partitionInactive

protected void partitionInactive(String partition)

Description copied from class: AbstractStreamingWriter

Notifies a partition become inactive. A partition becomes inactive after all the records received so far have been committed.

Specified by:

partitionInactive in class AbstractStreamingWriter<IN,PartitionCommitInfo>

onPartFileOpened

protected void onPartFileOpened(String s,
                                org.apache.flink.core.fs.Path newPath)

Description copied from class: AbstractStreamingWriter

Notifies a new file has been opened.

Note that this does not mean that the file has been created in the file system. It is only created logically and the actual file will be generated after it is committed.

Specified by:

onPartFileOpened in class AbstractStreamingWriter<IN,PartitionCommitInfo>

snapshotState

public void snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context)
                   throws Exception

Specified by:

snapshotState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator

Overrides:

snapshotState in class AbstractStreamingWriter<IN,PartitionCommitInfo>

Throws:

Exception

commitUpToCheckpoint

protected void commitUpToCheckpoint(long checkpointId)
                             throws Exception

Description copied from class: AbstractStreamingWriter

Commit up to this checkpoint id.

Overrides:

commitUpToCheckpoint in class AbstractStreamingWriter<IN,PartitionCommitInfo>

Throws:

Exception