Package org.apache.flink.streaming.connectors.kafka.partitioner

Class FlinkFixedPartitioner<T>

java.lang.Object

org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner<T>

org.apache.flink.streaming.connectors.kafka.partitioner.FlinkFixedPartitioner<T>

All Implemented Interfaces:

Serializable

@PublicEvolving
public class FlinkFixedPartitioner<T>
extends FlinkKafkaPartitioner<T>

A partitioner ensuring that each internal Flink partition ends up in one Kafka partition.

Note, one Kafka partition can contain multiple Flink partitions.

There are a couple of cases to consider.

More Flink partitions than kafka partitions

                Flink Sinks:            Kafka Partitions
                        1       ---------------->        1
                        2   --------------/
                        3   -------------/
                        4       ------------/
 

Some (or all) kafka partitions contain the output of more than one flink partition

Fewer Flink partitions than Kafka

                Flink Sinks:            Kafka Partitions
                        1       ---------------->        1
                        2       ---------------->        2
                                                                                3
                                                                                4
                                                                                5
 

Not all Kafka partitions contain data To avoid such an unbalanced partitioning, use a round-robin kafka partitioner (note that this will cause a lot of network connections between all the Flink instances and all the Kafka brokers).

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor	Description
FlinkFixedPartitioner()

Method Summary

Modifier and Type	Method	Description
boolean	equals(Object o)
int	hashCode()
void	open(int parallelInstanceId, int parallelInstances)	Initializer for the partitioner.
int	partition(T record, byte[] key, byte[] value, String targetTopic, int[] partitions)	Determine the id of the partition that the record should be written to.

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FlinkFixedPartitioner

public FlinkFixedPartitioner()

Method Detail

open

public void open(int parallelInstanceId,
                 int parallelInstances)

Description copied from class: FlinkKafkaPartitioner

Initializer for the partitioner. This is called once on each parallel sink instance of the Flink Kafka producer. This method should be overridden if necessary.

Overrides:

open in class FlinkKafkaPartitioner<T>

Parameters:

parallelInstanceId - 0-indexed id of the parallel sink instance in Flink

parallelInstances - the total number of parallel instances

partition

public int partition(T record,
                     byte[] key,
                     byte[] value,
                     String targetTopic,
                     int[] partitions)

Description copied from class: FlinkKafkaPartitioner

Determine the id of the partition that the record should be written to.

Specified by:

partition in class FlinkKafkaPartitioner<T>

Parameters:

record - the record value

key - serialized key of the record

value - serialized value of the record

targetTopic - target topic for the record

partitions - found partitions for the target topic

Returns:

the id of the target partition

equals

public boolean equals(Object o)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object