Package org.apache.flink.streaming.runtime.partitioner

Class ForwardForUnspecifiedPartitioner<T>

java.lang.Object

org.apache.flink.streaming.runtime.partitioner.StreamPartitioner<T>

org.apache.flink.streaming.runtime.partitioner.ForwardPartitioner<T>

org.apache.flink.streaming.runtime.partitioner.ForwardForUnspecifiedPartitioner<T>

Type Parameters:

T - Type of the elements in the Stream

All Implemented Interfaces:

Serializable, ChannelSelector<SerializationDelegate<StreamRecord<T>>>

@Internal
public class ForwardForUnspecifiedPartitioner<T>
extends ForwardPartitioner<T>

When the parallelism of both upstream and downstream is ExecutionConfig.PARALLELISM_DEFAULT and the edge's partitioner is not specified (partitioner==null), the edge's partitioner will be set to FORWARD by default(See StreamGraph.createActualEdge(java.lang.Integer, java.lang.Integer, int, org.apache.flink.streaming.runtime.partitioner.StreamPartitioner<?>, org.apache.flink.util.OutputTag, org.apache.flink.streaming.api.transformations.StreamExchangeMode, org.apache.flink.runtime.jobgraph.IntermediateDataSetID) method for details). When using the AdaptiveBatchScheduler, this will result in the parallelism of many job vertices is not calculated based on the amount of data but has to align with the parallelism of their upstream vertices due to forward edges, which is contrary to the original intention of the AdaptiveBatchScheduler.

To solve it, we introduce the ForwardForUnspecifiedPartitioner. This partitioner will be set for unspecified edges(partitioner==null), and then the runtime framework will change it to FORWARD/RESCALE after the operator chain creation:

1. Convert to ForwardPartitioner if the partitioner is intra-chain.

2. Convert to RescalePartitioner if the partitioner is inter-chain.

This partitioner should only be used when using AdaptiveBatchScheduler.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.flink.streaming.runtime.partitioner.StreamPartitioner

numberOfChannels

Constructor Summary

Constructors

Constructor	Description
ForwardForUnspecifiedPartitioner()

Method Summary

Modifier and Type	Method	Description
StreamPartitioner<T>	copy()
SubtaskStateMapper	getDownstreamSubtaskStateMapper()	Defines the behavior of this partitioner, when downstream rescaled during recovery of in-flight data.
int	selectChannel(SerializationDelegate<StreamRecord<T>> record)	Returns the logical channel index, to which the given record should be written.

Methods inherited from class org.apache.flink.streaming.runtime.partitioner.ForwardPartitioner

getUpstreamSubtaskStateMapper, isPointwise, toString

Methods inherited from class org.apache.flink.streaming.runtime.partitioner.StreamPartitioner

disableUnalignedCheckpoints, equals, hashCode, isBroadcast, isSupportsUnalignedCheckpoint, setup

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

ForwardForUnspecifiedPartitioner

public ForwardForUnspecifiedPartitioner()

Method Detail

copy

public StreamPartitioner<T> copy()

Overrides:

copy in class ForwardPartitioner<T>

getDownstreamSubtaskStateMapper

public SubtaskStateMapper getDownstreamSubtaskStateMapper()

Description copied from class: StreamPartitioner

Defines the behavior of this partitioner, when downstream rescaled during recovery of in-flight data.

Overrides:

getDownstreamSubtaskStateMapper in class ForwardPartitioner<T>

selectChannel

public int selectChannel(SerializationDelegate<StreamRecord<T>> record)

Description copied from interface: ChannelSelector

Returns the logical channel index, to which the given record should be written. It is illegal to call this method for broadcast channel selectors and this method can remain not implemented in that case (for example by throwing UnsupportedOperationException).

Specified by:

selectChannel in interface ChannelSelector<T>

Overrides:

selectChannel in class ForwardPartitioner<T>

Parameters:

record - the record to determine the output channels for.

Returns:

an integer number which indicates the index of the output channel through which the record shall be forwarded.