org.apache.flink.connector.file.table.stream.compact

类 CompactCoordinator

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<CompactMessages.CoordinatorOutput>

org.apache.flink.connector.file.table.stream.compact.CompactCoordinator

所有已实现的接口:

Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.Input<CompactMessages.CoordinatorInput>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.OneInputStreamOperator<CompactMessages.CoordinatorInput,CompactMessages.CoordinatorOutput>, org.apache.flink.streaming.api.operators.SetupableStreamOperator<CompactMessages.CoordinatorOutput>, org.apache.flink.streaming.api.operators.StreamOperator<CompactMessages.CoordinatorOutput>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator

@Internal
public class CompactCoordinator
extends org.apache.flink.streaming.api.operators.AbstractStreamOperator<CompactMessages.CoordinatorOutput>
implements org.apache.flink.streaming.api.operators.OneInputStreamOperator<CompactMessages.CoordinatorInput,CompactMessages.CoordinatorOutput>

This is the single (non-parallel) monitoring task which coordinate input files to compaction units. - Receives in-flight input files inside checkpoint. - Receives all upstream end input messages after the checkpoint completes successfully, starts coordination.

CompactMessages.CompactionUnit and CompactMessages.EndCompaction must be sent to the downstream in an orderly manner, while CompactMessages.EndCompaction is broadcast emitting, so unit and endCompaction use the broadcast emitting mechanism together. Since unit is broadcast, we want it to be processed by a single task, so we carry the ID in the unit and let the downstream task select its own unit.

NOTE: The coordination is a stable algorithm, which can ensure that the downstream can perform compaction at any time without worrying about fail over.

STATE: This operator stores input files in state, after the checkpoint completes successfully, input files are taken out from the state for coordination.

另请参阅:

序列化表格

字段概要

从类继承的字段 org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, config, latencyStats, metrics, output, processingTimeService

构造器概要

构造器

构造器和说明
CompactCoordinator(org.apache.flink.util.function.SupplierWithException<org.apache.flink.core.fs.FileSystem,IOException> fsFactory, long targetFileSize)

方法概要

限定符和类型	方法和说明
void	initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
void	processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<CompactMessages.CoordinatorInput> element)
void	snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context)

从类继承的方法 org.apache.flink.streaming.api.operators.AbstractStreamOperator

close, finish, getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, open, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

从接口继承的方法 org.apache.flink.streaming.api.operators.OneInputStreamOperator

setKeyContextElement

从接口继承的方法 org.apache.flink.streaming.api.operators.StreamOperator

close, finish, getMetricGroup, getOperatorID, initializeState, open, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState

从接口继承的方法 org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted, notifyCheckpointComplete

从接口继承的方法 org.apache.flink.streaming.api.operators.KeyContext

getCurrentKey, setCurrentKey

从接口继承的方法 org.apache.flink.streaming.api.operators.Input

processLatencyMarker, processWatermark, processWatermarkStatus

构造器详细资料

CompactCoordinator

public CompactCoordinator(org.apache.flink.util.function.SupplierWithException<org.apache.flink.core.fs.FileSystem,IOException> fsFactory,
                          long targetFileSize)

方法详细资料

initializeState

public void initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
                     throws Exception

指定者:

initializeState 在接口中 org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator

覆盖:

initializeState 在类中 org.apache.flink.streaming.api.operators.AbstractStreamOperator<CompactMessages.CoordinatorOutput>

抛出:

Exception

processElement

public void processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<CompactMessages.CoordinatorInput> element)
                    throws Exception

指定者:

processElement 在接口中 org.apache.flink.streaming.api.operators.Input<CompactMessages.CoordinatorInput>

抛出:

Exception

snapshotState

public void snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context)
                   throws Exception

指定者:

snapshotState 在接口中 org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator

覆盖:

snapshotState 在类中 org.apache.flink.streaming.api.operators.AbstractStreamOperator<CompactMessages.CoordinatorOutput>

抛出:

Exception