org.apache.flink.connector.file.sink.compactor.operator

类 CompactorOperator

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>

org.apache.flink.connector.file.sink.compactor.operator.CompactorOperator

所有已实现的接口:

Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.BoundedOneInput, org.apache.flink.streaming.api.operators.Input<CompactorRequest>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.OneInputStreamOperator<CompactorRequest,org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>, org.apache.flink.streaming.api.operators.SetupableStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>, org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator

@Internal
public class CompactorOperator
extends org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>
implements org.apache.flink.streaming.api.operators.OneInputStreamOperator<CompactorRequest,org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>, org.apache.flink.streaming.api.operators.BoundedOneInput, org.apache.flink.api.common.state.CheckpointListener

An operator that perform compaction for the FileSink.

Requests received from the CompactCoordinator will firstly be held in memory, and snapshot into the state of a checkpoint. When the checkpoint is successfully completed, all requests received before can be submitted. The results can be emitted at the next prepareSnapshotPreBarrier(long) invoking after the compaction is finished, to ensure that committers can receive only one CommittableSummary and the corresponding number of Committable for a single checkpoint.

另请参阅:

序列化表格

字段概要

从类继承的字段 org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, config, latencyStats, LOG, metrics, output, processingTimeService

构造器概要

构造器

构造器和说明
CompactorOperator(FileCompactStrategy strategy, org.apache.flink.core.io.SimpleVersionedSerializer<FileSinkCommittable> committableSerializer, FileCompactor fileCompactor, org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter<?,String> bucketWriter)

方法概要

限定符和类型	方法和说明
void	close()
void	endInput()
CompletableFuture<?>	getAllTasksFuture()
void	initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
void	notifyCheckpointComplete(long checkpointId)
void	open()
void	prepareSnapshotPreBarrier(long checkpointId)
void	processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<CompactorRequest> element)
void	snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context)

从类继承的方法 org.apache.flink.streaming.api.operators.AbstractStreamOperator

finish, getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

从接口继承的方法 org.apache.flink.streaming.api.operators.OneInputStreamOperator

setKeyContextElement

从接口继承的方法 org.apache.flink.streaming.api.operators.StreamOperator

finish, getMetricGroup, getOperatorID, initializeState, setKeyContextElement1, setKeyContextElement2, snapshotState

从接口继承的方法 org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted

从接口继承的方法 org.apache.flink.streaming.api.operators.KeyContext

getCurrentKey, setCurrentKey

从接口继承的方法 org.apache.flink.streaming.api.operators.Input

processLatencyMarker, processWatermark, processWatermarkStatus

构造器详细资料

CompactorOperator

public CompactorOperator(FileCompactStrategy strategy,
                         org.apache.flink.core.io.SimpleVersionedSerializer<FileSinkCommittable> committableSerializer,
                         FileCompactor fileCompactor,
                         org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter<?,String> bucketWriter)

方法详细资料

open

public void open()
          throws Exception

指定者:

open 在接口中 org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>

覆盖:

open 在类中 org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>

抛出:

Exception

processElement

public void processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<CompactorRequest> element)
                    throws Exception

指定者:

processElement 在接口中 org.apache.flink.streaming.api.operators.Input<CompactorRequest>

抛出:

Exception

endInput

public void endInput()
              throws Exception

指定者:

endInput 在接口中 org.apache.flink.streaming.api.operators.BoundedOneInput

抛出:

Exception

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)
                              throws Exception

指定者:

notifyCheckpointComplete 在接口中 org.apache.flink.api.common.state.CheckpointListener

覆盖:

notifyCheckpointComplete 在类中 org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>

抛出:

Exception

prepareSnapshotPreBarrier

public void prepareSnapshotPreBarrier(long checkpointId)
                               throws Exception

指定者:

prepareSnapshotPreBarrier 在接口中 org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>

覆盖:

prepareSnapshotPreBarrier 在类中 org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>

抛出:

Exception

snapshotState

public void snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context)
                   throws Exception

指定者:

snapshotState 在接口中 org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator

覆盖:

snapshotState 在类中 org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>

抛出:

Exception

initializeState

public void initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
                     throws Exception

指定者:

initializeState 在接口中 org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator

覆盖:

initializeState 在类中 org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>

抛出:

Exception

close

public void close()
           throws Exception

指定者:

close 在接口中 org.apache.flink.streaming.api.operators.StreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>

覆盖:

close 在类中 org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.flink.streaming.api.connector.sink2.CommittableMessage<FileSinkCommittable>>

抛出:

Exception

getAllTasksFuture

@VisibleForTesting
public CompletableFuture<?> getAllTasksFuture()