org.apache.flink.connector.file.table.stream.compact

类 CompactOperator<T>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<PartitionCommitInfo>

org.apache.flink.connector.file.table.stream.compact.CompactOperator<T>

所有已实现的接口:

Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.BoundedOneInput, org.apache.flink.streaming.api.operators.Input<CompactMessages.CoordinatorOutput>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.OneInputStreamOperator<CompactMessages.CoordinatorOutput,PartitionCommitInfo>, org.apache.flink.streaming.api.operators.SetupableStreamOperator<PartitionCommitInfo>, org.apache.flink.streaming.api.operators.StreamOperator<PartitionCommitInfo>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator

@Internal
public class CompactOperator<T>
extends org.apache.flink.streaming.api.operators.AbstractStreamOperator<PartitionCommitInfo>
implements org.apache.flink.streaming.api.operators.OneInputStreamOperator<CompactMessages.CoordinatorOutput,PartitionCommitInfo>, org.apache.flink.streaming.api.operators.BoundedOneInput

Receives compaction units to do compaction. Send partition commit information after compaction finished.

Use BulkFormat to read and use BucketWriter to write.

STATE: This operator stores expired files in state, after the checkpoint completes successfully, We can ensure that these files will not be used again and they can be deleted from the file system.

另请参阅:

序列化表格

字段概要

字段

限定符和类型	字段和说明
static String	COMPACTED_PREFIX
static String	UNCOMPACTED_PREFIX

从类继承的字段 org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, config, latencyStats, LOG, metrics, output, processingTimeService

构造器概要

构造器

构造器和说明
CompactOperator(org.apache.flink.util.function.SupplierWithException<org.apache.flink.core.fs.FileSystem,IOException> fsFactory, CompactReader.Factory<T> readerFactory, CompactWriter.Factory<T> writerFactory)

方法概要

限定符和类型	方法和说明
static org.apache.flink.core.fs.Path	convertFromUncompacted(org.apache.flink.core.fs.Path path)
static String	convertToUncompacted(String path)
void	endInput()
void	initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
void	notifyCheckpointComplete(long checkpointId)
void	processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<CompactMessages.CoordinatorOutput> element)
void	snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context)

从类继承的方法 org.apache.flink.streaming.api.operators.AbstractStreamOperator

close, finish, getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, open, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

从接口继承的方法 org.apache.flink.streaming.api.operators.OneInputStreamOperator

setKeyContextElement

从接口继承的方法 org.apache.flink.streaming.api.operators.StreamOperator

close, finish, getMetricGroup, getOperatorID, initializeState, open, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState

从接口继承的方法 org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted

从接口继承的方法 org.apache.flink.streaming.api.operators.KeyContext

getCurrentKey, setCurrentKey

从接口继承的方法 org.apache.flink.streaming.api.operators.Input

processLatencyMarker, processWatermark, processWatermarkStatus

字段详细资料

UNCOMPACTED_PREFIX

public static final String UNCOMPACTED_PREFIX

另请参阅:

常量字段值

COMPACTED_PREFIX

public static final String COMPACTED_PREFIX

另请参阅:

常量字段值

构造器详细资料

CompactOperator

public CompactOperator(org.apache.flink.util.function.SupplierWithException<org.apache.flink.core.fs.FileSystem,IOException> fsFactory,
                       CompactReader.Factory<T> readerFactory,
                       CompactWriter.Factory<T> writerFactory)

方法详细资料

initializeState

public void initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
                     throws Exception

指定者:

initializeState 在接口中 org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator

覆盖:

initializeState 在类中 org.apache.flink.streaming.api.operators.AbstractStreamOperator<PartitionCommitInfo>

抛出:

Exception

processElement

public void processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<CompactMessages.CoordinatorOutput> element)
                    throws Exception

指定者:

processElement 在接口中 org.apache.flink.streaming.api.operators.Input<CompactMessages.CoordinatorOutput>

抛出:

Exception

snapshotState

public void snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context)
                   throws Exception

指定者:

snapshotState 在接口中 org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator

覆盖:

snapshotState 在类中 org.apache.flink.streaming.api.operators.AbstractStreamOperator<PartitionCommitInfo>

抛出:

Exception

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)
                              throws Exception

指定者:

notifyCheckpointComplete 在接口中 org.apache.flink.api.common.state.CheckpointListener

覆盖:

notifyCheckpointComplete 在类中 org.apache.flink.streaming.api.operators.AbstractStreamOperator<PartitionCommitInfo>

抛出:

Exception

endInput

public void endInput()
              throws Exception

指定者:

endInput 在接口中 org.apache.flink.streaming.api.operators.BoundedOneInput

抛出:

Exception

convertToUncompacted

public static String convertToUncompacted(String path)

convertFromUncompacted

public static org.apache.flink.core.fs.Path convertFromUncompacted(org.apache.flink.core.fs.Path path)