Package org.apache.flink.streaming.connectors.kafka.shuffle

Class FlinkKafkaShuffleProducer<IN,KEY>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.sink.RichSinkFunction<IN>

org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction<IN,FlinkKafkaProducer.KafkaTransactionState,FlinkKafkaProducer.KafkaTransactionContext>

org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer<IN>

org.apache.flink.streaming.connectors.kafka.shuffle.FlinkKafkaShuffleProducer<IN,KEY>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.sink.SinkFunction<IN>

@Internal
public class FlinkKafkaShuffleProducer<IN,KEY>
extends FlinkKafkaProducer<IN>

Flink Kafka Shuffle Producer Function. It is different from FlinkKafkaProducer in the way handling elements and watermarks

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	FlinkKafkaShuffleProducer.KafkaSerializer<IN>	Flink Kafka Shuffle Serializer.

Nested classes/interfaces inherited from class org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer

FlinkKafkaProducer.ContextStateSerializer, FlinkKafkaProducer.KafkaTransactionContext, FlinkKafkaProducer.KafkaTransactionState, FlinkKafkaProducer.NextTransactionalIdHint, FlinkKafkaProducer.NextTransactionalIdHintSerializer, FlinkKafkaProducer.Semantic, FlinkKafkaProducer.TransactionStateSerializer

Nested classes/interfaces inherited from class org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction

org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.State<TXN extends Object,CONTEXT extends Object>, org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.StateSerializer<TXN extends Object,CONTEXT extends Object>, org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.StateSerializerSnapshot<TXN extends Object,CONTEXT extends Object>, org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.TransactionHolder<TXN extends Object>

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.sink.SinkFunction

org.apache.flink.streaming.api.functions.sink.SinkFunction.Context

Field Summary

Fields inherited from class org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer

asyncException, callback, DEFAULT_KAFKA_PRODUCERS_POOL_SIZE, DEFAULT_KAFKA_TRANSACTION_TIMEOUT, defaultTopicId, KEY_DISABLE_METRICS, pendingRecords, producerConfig, SAFE_SCALE_DOWN_FACTOR, semantic, topicPartitionsMap, writeTimestampToKafka

Fields inherited from class org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction

pendingCommitTransactions, state, userContext

Method Summary

Modifier and Type	Method	Description
void	invoke(org.apache.flink.streaming.api.watermark.Watermark watermark)	This is the function invoked to handle each watermark.
void	invoke(FlinkKafkaProducer.KafkaTransactionState transaction, IN next, org.apache.flink.streaming.api.functions.sink.SinkFunction.Context context)	This is the function invoked to handle each element.

Methods inherited from class org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer

abort, acknowledgeMessage, beginTransaction, checkErroneous, close, commit, createProducer, finishRecoveringContext, getPartitionsByTopic, getTransactionTimeout, ignoreFailuresAfterTransactionTimeout, initializeState, initializeUserContext, open, preCommit, recoverAndAbort, recoverAndCommit, setLogFailuresOnly, setTransactionalIdPrefix, setWriteTimestampToKafka, snapshotState

Methods inherited from class org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction

currentTransaction, enableTransactionTimeoutWarnings, finish, finishProcessing, getUserContext, invoke, invoke, notifyCheckpointAborted, notifyCheckpointComplete, pendingTransactions, setTransactionTimeout

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.functions.sink.SinkFunction

writeWatermark

Method Detail

invoke

public void invoke(FlinkKafkaProducer.KafkaTransactionState transaction,
                   IN next,
                   org.apache.flink.streaming.api.functions.sink.SinkFunction.Context context)
            throws FlinkKafkaException

This is the function invoked to handle each element.

Overrides:

invoke in class FlinkKafkaProducer<IN>

Parameters:

transaction - Transaction state; elements are written to Kafka in transactions to guarantee different level of data consistency

next - Element to handle

context - Context needed to handle the element

Throws:

FlinkKafkaException - for kafka error

invoke

public void invoke(org.apache.flink.streaming.api.watermark.Watermark watermark)
            throws FlinkKafkaException

This is the function invoked to handle each watermark.

Parameters:

watermark - Watermark to handle

Throws:

FlinkKafkaException - For kafka error