org.apache.flink.datastream.api.stream

Interface KeyedPartitionStream<K,T>

All Superinterfaces:

DataStream

All Known Subinterfaces:

KeyedPartitionStream.ProcessConfigurableAndKeyedPartitionStream<K,T>

@Experimental
public interface KeyedPartitionStream<K,T>
extends DataStream

This interface represents a kind of partitioned data stream. For this stream, each key is a partition, and the partition to which the data belongs is deterministic.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static interface	KeyedPartitionStream.ProcessConfigurableAndKeyedPartitionStream<K,T> This interface represents a configurable KeyedPartitionStream.
static interface	KeyedPartitionStream.TwoKeyedPartitionStreams<K,T1,T2> This class represents a combination of two KeyedPartitionStream.

Method Summary

Modifier and Type	Method and Description
BroadcastStream<T>	broadcast() Transform this stream to a new BroadcastStream.
<T_OTHER,OUT> NonKeyedPartitionStream.ProcessConfigurableAndNonKeyedPartitionStream<OUT>	connectAndProcess(BroadcastStream<T_OTHER> other, TwoInputBroadcastStreamProcessFunction<T,T_OTHER,OUT> processFunction) Apply a two input operation to this and other BroadcastStream.
<T_OTHER,OUT> KeyedPartitionStream.ProcessConfigurableAndKeyedPartitionStream<K,OUT>	connectAndProcess(BroadcastStream<T_OTHER> other, TwoInputBroadcastStreamProcessFunction<T,T_OTHER,OUT> processFunction, org.apache.flink.api.java.functions.KeySelector<OUT,K> newKeySelector) Apply a two input operation to this and other BroadcastStream.
<T_OTHER,OUT> NonKeyedPartitionStream.ProcessConfigurableAndNonKeyedPartitionStream<OUT>	connectAndProcess(KeyedPartitionStream<K,T_OTHER> other, TwoInputNonBroadcastStreamProcessFunction<T,T_OTHER,OUT> processFunction) Apply a two input operation to this and other KeyedPartitionStream.
<T_OTHER,OUT> KeyedPartitionStream.ProcessConfigurableAndKeyedPartitionStream<K,OUT>	connectAndProcess(KeyedPartitionStream<K,T_OTHER> other, TwoInputNonBroadcastStreamProcessFunction<T,T_OTHER,OUT> processFunction, org.apache.flink.api.java.functions.KeySelector<OUT,K> newKeySelector) Apply a two input operation to this and other KeyedPartitionStream.The two keyed streams must have the same partitions, otherwise it makes no sense to connect them.
GlobalStream<T>	global() Coalesce this stream to a GlobalStream.
<NEW_KEY> KeyedPartitionStream<NEW_KEY,T>	keyBy(org.apache.flink.api.java.functions.KeySelector<T,NEW_KEY> keySelector) Transform this stream to a new KeyedPartitionStream.
<OUT> NonKeyedPartitionStream.ProcessConfigurableAndNonKeyedPartitionStream<OUT>	process(OneInputStreamProcessFunction<T,OUT> processFunction) Apply an operation to this KeyedPartitionStream.
<OUT> KeyedPartitionStream.ProcessConfigurableAndKeyedPartitionStream<K,OUT>	process(OneInputStreamProcessFunction<T,OUT> processFunction, org.apache.flink.api.java.functions.KeySelector<OUT,K> newKeySelector) Apply an operation to this KeyedPartitionStream.
<OUT1,OUT2> NonKeyedPartitionStream.TwoNonKeyedPartitionStreams<OUT1,OUT2>	process(TwoOutputStreamProcessFunction<T,OUT1,OUT2> processFunction) Apply a two output operation to this KeyedPartitionStream.
<OUT1,OUT2> KeyedPartitionStream.TwoKeyedPartitionStreams<K,OUT1,OUT2>	process(TwoOutputStreamProcessFunction<T,OUT1,OUT2> processFunction, org.apache.flink.api.java.functions.KeySelector<OUT1,K> keySelector1, org.apache.flink.api.java.functions.KeySelector<OUT2,K> keySelector2) Apply a two output operation to this KeyedPartitionStream.
NonKeyedPartitionStream<T>	shuffle() Transform this stream to a new NonKeyedPartitionStream, data will be shuffled between these two streams.
ProcessConfigurable<?>	toSink(org.apache.flink.api.connector.dsv2.Sink<T> sink)

Method Detail

process

<OUT> KeyedPartitionStream.ProcessConfigurableAndKeyedPartitionStream<K,OUT> process(OneInputStreamProcessFunction<T,OUT> processFunction,
                                                                                     org.apache.flink.api.java.functions.KeySelector<OUT,K> newKeySelector)

Apply an operation to this KeyedPartitionStream.

This method is used to avoid shuffle after applying the process function. It is required that for the same record, the new KeySelector must extract the same key as the original KeySelector on this KeyedPartitionStream. Otherwise, the partition of data will be messy.

Parameters:

processFunction - to perform operation.

newKeySelector - to select the key after process.

Returns:

new KeyedPartitionStream with this operation.

process

<OUT> NonKeyedPartitionStream.ProcessConfigurableAndNonKeyedPartitionStream<OUT> process(OneInputStreamProcessFunction<T,OUT> processFunction)

Apply an operation to this KeyedPartitionStream.

Generally, apply an operation to a KeyedPartitionStream will result in a NonKeyedPartitionStream, and you can manually generate a KeyedPartitionStream via keyBy partitioning. In some cases, you can guarantee that the partition on which the data is processed will not change, then you can use process(OneInputStreamProcessFunction, KeySelector) to avoid shuffling.

Parameters:

processFunction - to perform operation.

Returns:

new NonKeyedPartitionStream with this operation.

process

<OUT1,OUT2> KeyedPartitionStream.TwoKeyedPartitionStreams<K,OUT1,OUT2> process(TwoOutputStreamProcessFunction<T,OUT1,OUT2> processFunction,
                                                                               org.apache.flink.api.java.functions.KeySelector<OUT1,K> keySelector1,
                                                                               org.apache.flink.api.java.functions.KeySelector<OUT2,K> keySelector2)

Apply a two output operation to this KeyedPartitionStream.

This method is used to avoid shuffle after applying the process function. It is required that for the same record, these new two KeySelectors must extract the same key as the original KeySelectors on this KeyedPartitionStream. Otherwise, the partition of data will be messy.

Parameters:

processFunction - to perform two output operation.

keySelector1 - to select the key of first output.

keySelector2 - to select the key of second output.

Returns:

new KeyedPartitionStream.TwoKeyedPartitionStreams with this operation.

process

<OUT1,OUT2> NonKeyedPartitionStream.TwoNonKeyedPartitionStreams<OUT1,OUT2> process(TwoOutputStreamProcessFunction<T,OUT1,OUT2> processFunction)

Apply a two output operation to this KeyedPartitionStream.

Parameters:

processFunction - to perform two output operation.

Returns:

new NonKeyedPartitionStream.TwoNonKeyedPartitionStreams with this operation.

connectAndProcess

<T_OTHER,OUT> NonKeyedPartitionStream.ProcessConfigurableAndNonKeyedPartitionStream<OUT> connectAndProcess(KeyedPartitionStream<K,T_OTHER> other,
                                                                                                           TwoInputNonBroadcastStreamProcessFunction<T,T_OTHER,OUT> processFunction)

Apply a two input operation to this and other KeyedPartitionStream. The two keyed streams must have the same partitions, otherwise it makes no sense to connect them.

Generally, concatenating two KeyedPartitionStream will result in a NonKeyedPartitionStream, and you can manually generate a KeyedPartitionStream via keyBy partitioning. In some cases, you can guarantee that the partition on which the data is processed will not change, then you can use connectAndProcess(KeyedPartitionStream, TwoInputNonBroadcastStreamProcessFunction, KeySelector) to avoid shuffling.

Parameters:

other - KeyedPartitionStream to perform operation with two input.

processFunction - to perform operation.

Returns:

new NonKeyedPartitionStream with this operation.

connectAndProcess

<T_OTHER,OUT> KeyedPartitionStream.ProcessConfigurableAndKeyedPartitionStream<K,OUT> connectAndProcess(KeyedPartitionStream<K,T_OTHER> other,
                                                                                                       TwoInputNonBroadcastStreamProcessFunction<T,T_OTHER,OUT> processFunction,
                                                                                                       org.apache.flink.api.java.functions.KeySelector<OUT,K> newKeySelector)

Apply a two input operation to this and other KeyedPartitionStream.The two keyed streams must have the same partitions, otherwise it makes no sense to connect them.

This method is used to avoid shuffle after applying the process function. It is required that for the same record, the new KeySelector must extract the same key as the original KeySelectors on these two KeyedPartitionStreams. Otherwise, the partition of data will be messy.

Parameters:

other - KeyedPartitionStream to perform operation with two input.

processFunction - to perform operation.

newKeySelector - to select the key after process.

Returns:

new KeyedPartitionStream with this operation.

connectAndProcess

<T_OTHER,OUT> NonKeyedPartitionStream.ProcessConfigurableAndNonKeyedPartitionStream<OUT> connectAndProcess(BroadcastStream<T_OTHER> other,
                                                                                                           TwoInputBroadcastStreamProcessFunction<T,T_OTHER,OUT> processFunction)

Apply a two input operation to this and other BroadcastStream.

Generally, concatenating KeyedPartitionStream and BroadcastStream will result in a NonKeyedPartitionStream, and you can manually generate a KeyedPartitionStream via keyBy partitioning. In some cases, you can guarantee that the partition on which the data is processed will not change, then you can use connectAndProcess(BroadcastStream, TwoInputBroadcastStreamProcessFunction, KeySelector) to avoid shuffling.

Parameters:

processFunction - to perform operation.

Returns:

new stream with this operation.

connectAndProcess

<T_OTHER,OUT> KeyedPartitionStream.ProcessConfigurableAndKeyedPartitionStream<K,OUT> connectAndProcess(BroadcastStream<T_OTHER> other,
                                                                                                       TwoInputBroadcastStreamProcessFunction<T,T_OTHER,OUT> processFunction,
                                                                                                       org.apache.flink.api.java.functions.KeySelector<OUT,K> newKeySelector)

Apply a two input operation to this and other BroadcastStream.

This method is used to avoid shuffle after applying the process function. It is required that for the record from non-broadcast input, the new KeySelector must extract the same key as the original KeySelectors on the KeyedPartitionStream. Otherwise, the partition of data will be messy. As for the record from broadcast input, the output key from keyed partition itself instead of the new key selector, so the data it outputs will not affect the partition.

Parameters:

other - BroadcastStream to perform operation with two input.

processFunction - to perform operation.

newKeySelector - to select the key after process.

Returns:

new KeyedPartitionStream with this operation.

global

GlobalStream<T> global()

Coalesce this stream to a GlobalStream.

Returns:

the coalesced global stream.

keyBy

<NEW_KEY> KeyedPartitionStream<NEW_KEY,T> keyBy(org.apache.flink.api.java.functions.KeySelector<T,NEW_KEY> keySelector)

Transform this stream to a new KeyedPartitionStream.

Parameters:

keySelector - to decide how to map data to partition.

Returns:

the transformed stream partitioned by key.

shuffle

NonKeyedPartitionStream<T> shuffle()

Transform this stream to a new NonKeyedPartitionStream, data will be shuffled between these two streams.

Returns:

the transformed stream after shuffle.

broadcast

BroadcastStream<T> broadcast()

Transform this stream to a new BroadcastStream.

Returns:

the transformed BroadcastStream.

toSink

ProcessConfigurable<?> toSink(org.apache.flink.api.connector.dsv2.Sink<T> sink)