org.apache.flink.streaming.api.datastream

Class SingleOutputStreamOperator<T,O extends SingleOutputStreamOperator<T,O>>

java.lang.Object

org.apache.flink.streaming.api.datastream.DataStream<T>

org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator<T,O>

Type Parameters:

T - The type of the elements in this Stream

O - Type of the operator.

Direct Known Subclasses:

DataStreamSource, IterativeStream

public class SingleOutputStreamOperator<T,O extends SingleOutputStreamOperator<T,O>>
extends DataStream<T>

The SingleOutputStreamOperator represents a user defined transformation applied on a DataStream with one predefined output type.

Field Summary

Fields inherited from class org.apache.flink.streaming.api.datastream.DataStream

environment, transformation

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	SingleOutputStreamOperator(StreamExecutionEnvironment environment, StreamTransformation<T> transformation)

Method Summary

Methods

Modifier and Type	Method and Description
SingleOutputStreamOperator<T,O>	broadcast() Sets the partitioning of the DataStream so that the output tuples are broadcasted to every parallel instance of the next component.
SingleOutputStreamOperator<T,O>	disableChaining() Turns off chaining for this operator so thread co-location will not be used as an optimization.
SingleOutputStreamOperator<T,O>	forward() Sets the partitioning of the DataStream so that the output tuples are forwarded to the local subtask of the next component (whenever possible).
String	getName() Gets the name of the current data stream.
SingleOutputStreamOperator<T,O>	global() Sets the partitioning of the DataStream so that the output values all go to the first instance of the next processing operator.
SingleOutputStreamOperator<T,O>	isolateResources() Isolates the operator in its own resource group.
SingleOutputStreamOperator<T,O>	name(String name) Sets the name of the current data stream.
SingleOutputStreamOperator<T,O>	rebalance() Sets the partitioning of the DataStream so that the output tuples are distributed evenly to instances of the next component in a Round-robin fashion.
O	returns(Class<T> typeClass) Adds a type information hint about the return type of this operator.
O	returns(String typeInfoString) Adds a type information hint about the return type of this operator.
O	returns(TypeInformation<T> typeInfo) Adds a type information hint about the return type of this operator.
SingleOutputStreamOperator<T,O>	setBufferTimeout(long timeoutMillis) Sets the maximum time frequency (ms) for the flushing of the output buffer.
protected DataStream<T>	setConnectionType(StreamPartitioner<T> partitioner) Internal function for setting the partitioner for the DataStream
SingleOutputStreamOperator<T,O>	setParallelism(int parallelism) Sets the parallelism for this operator.
SingleOutputStreamOperator<T,O>	shuffle() Sets the partitioning of the DataStream so that the output tuples are shuffled uniformly randomly to the next component.
SingleOutputStreamOperator<T,O>	startNewChain() Starts a new task chain beginning at this operator.
SingleOutputStreamOperator<T,O>	startNewResourceGroup() By default all operators in a streaming job share the same resource group.

Methods inherited from class org.apache.flink.streaming.api.datastream.DataStream

addSink, assignTimestamps, clean, coGroup, connect, countWindowAll, countWindowAll, filter, flatMap, getExecutionConfig, getExecutionEnvironment, getId, getParallelism, getTransformation, getType, iterate, iterate, join, keyBy, keyBy, keyBy, map, partitionByHash, partitionByHash, partitionByHash, partitionCustom, partitionCustom, partitionCustom, print, printToErr, project, split, timeWindowAll, timeWindowAll, transform, union, windowAll, write, writeAsCsv, writeAsCsv, writeAsCsv, writeAsCsv, writeAsText, writeAsText, writeAsText, writeAsText, writeToSocket

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SingleOutputStreamOperator

protected SingleOutputStreamOperator(StreamExecutionEnvironment environment,
                          StreamTransformation<T> transformation)

Method Detail

getName

public String getName()

Gets the name of the current data stream. This name is used by the visualization and logging during runtime.

Returns:

Name of the stream.

name

public SingleOutputStreamOperator<T,O> name(String name)

Sets the name of the current data stream. This name is used by the visualization and logging during runtime.

Returns:

The named operator.

setParallelism

public SingleOutputStreamOperator<T,O> setParallelism(int parallelism)

Sets the parallelism for this operator. The degree must be 1 or more.

Parameters:

parallelism - The parallelism for this operator.

Returns:

The operator with set parallelism.

setBufferTimeout

public SingleOutputStreamOperator<T,O> setBufferTimeout(long timeoutMillis)

Sets the maximum time frequency (ms) for the flushing of the output buffer. By default the output buffers flush only when they are full.

Parameters:

timeoutMillis - The maximum time between two output flushes.

Returns:

The operator with buffer timeout set.

broadcast

public SingleOutputStreamOperator<T,O> broadcast()

Description copied from class: DataStream

Sets the partitioning of the DataStream so that the output tuples are broadcasted to every parallel instance of the next component.

This setting only effects the how the outputs will be distributed between the parallel instances of the next processing operator.

Overrides:

broadcast in class DataStream<T>

Returns:

The DataStream with broadcast partitioning set.

shuffle

public SingleOutputStreamOperator<T,O> shuffle()

Description copied from class: DataStream

Sets the partitioning of the DataStream so that the output tuples are shuffled uniformly randomly to the next component.

This setting only effects the how the outputs will be distributed between the parallel instances of the next processing operator.

Overrides:

shuffle in class DataStream<T>

Returns:

The DataStream with shuffle partitioning set.

forward

public SingleOutputStreamOperator<T,O> forward()

Description copied from class: DataStream

Sets the partitioning of the DataStream so that the output tuples are forwarded to the local subtask of the next component (whenever possible).

This setting only effects the how the outputs will be distributed between the parallel instances of the next processing operator.

Overrides:

forward in class DataStream<T>

Returns:

The DataStream with forward partitioning set.

rebalance

public SingleOutputStreamOperator<T,O> rebalance()

Description copied from class: DataStream

Sets the partitioning of the DataStream so that the output tuples are distributed evenly to instances of the next component in a Round-robin fashion.

This setting only effects the how the outputs will be distributed between the parallel instances of the next processing operator.

Overrides:

rebalance in class DataStream<T>

Returns:

The DataStream with rebalance partitioning set.

global

public SingleOutputStreamOperator<T,O> global()

Description copied from class: DataStream

Sets the partitioning of the DataStream so that the output values all go to the first instance of the next processing operator. Use this setting with care since it might cause a serious performance bottleneck in the application.

Overrides:

global in class DataStream<T>

Returns:

The DataStream with shuffle partitioning set.

disableChaining

public SingleOutputStreamOperator<T,O> disableChaining()

Turns off chaining for this operator so thread co-location will not be used as an optimization.

Chaining can be turned off for the whole job by StreamExecutionEnvironment.disableOperatorChaining() however it is not advised for performance considerations.

Returns:

The operator with chaining disabled

startNewChain

public SingleOutputStreamOperator<T,O> startNewChain()

Starts a new task chain beginning at this operator. This operator will not be chained (thread co-located for increased performance) to any previous tasks even if possible.

Returns:

The operator with chaining set.

returns

public O returns(String typeInfoString)

Adds a type information hint about the return type of this operator.

Type hints are important in cases where the Java compiler throws away generic type information necessary for efficient execution.

This method takes a type information string that will be parsed. A type information string can contain the following types:

• Basic types such as Integer, String, etc.
• Basic type arrays such as Integer[], String[], etc.
• Tuple types such as Tuple1<TYPE0>, Tuple2<TYPE0, TYPE1>, etc.
• Pojo types such as org.my.MyPojo<myFieldName=TYPE0,myFieldName2=TYPE1>, etc.
• Generic types such as java.lang.Class, etc.
• Custom type arrays such as org.my.CustomClass[], org.my.CustomClass$StaticInnerClass[], etc.
• Value types such as DoubleValue, StringValue, IntegerValue, etc.
• Tuple array types such as Tuple2<TYPE0,TYPE1>[], etc.
• Writable types such as Writable<org.my.CustomWritable>
• Enum types such as Enum<org.my.CustomEnum>

Example: "Tuple2<String,Tuple2<Integer,org.my.MyJob$Pojo<word=String>>>"

Parameters:

typeInfoString - type information string to be parsed

Returns:

This operator with a given return type hint.

returns

public O returns(TypeInformation<T> typeInfo)

Adds a type information hint about the return type of this operator.

Type hints are important in cases where the Java compiler throws away generic type information necessary for efficient execution.

This method takes an instance of TypeInformation such as:

• BasicTypeInfo
• BasicArrayTypeInfo
• TupleTypeInfo
• PojoTypeInfo
• WritableTypeInfo
• ValueTypeInfo
• etc.

Parameters:

typeInfo - type information as a return type hint

Returns:

This operator with a given return type hint.

returns

public O returns(Class<T> typeClass)

Adds a type information hint about the return type of this operator.

Type hints are important in cases where the Java compiler throws away generic type information necessary for efficient execution.

This method takes a class that will be analyzed by Flink's type extraction capabilities.

Examples for classes are:

• Basic types such as Integer.class, String.class, etc.
• POJOs such as MyPojo.class
• Classes that extend tuples. Classes like Tuple1.class,Tuple2.class, etc. are not sufficient.
• Arrays such as String[].class, etc.

Parameters:

typeClass - class as a return type hint

Returns:

This operator with a given return type hint.

setConnectionType

protected DataStream<T> setConnectionType(StreamPartitioner<T> partitioner)

Description copied from class: DataStream

Internal function for setting the partitioner for the DataStream

Overrides:

setConnectionType in class DataStream<T>

Parameters:

partitioner - Partitioner to set.

Returns:

The modified DataStream.

startNewResourceGroup

public SingleOutputStreamOperator<T,O> startNewResourceGroup()

By default all operators in a streaming job share the same resource group. Each resource group takes as many task manager slots as the maximum parallelism operator in that group. Task chaining is only possible within one resource group. By calling this method, this operators starts a new resource group and all subsequent operators will be added to this group unless specified otherwise.

Please note that local executions have by default as many available task slots as the environment parallelism, so in order to start a new resource group the degree of parallelism for the operators must be decreased from the default.

Returns:

The operator as a part of a new resource group.

isolateResources

public SingleOutputStreamOperator<T,O> isolateResources()

Isolates the operator in its own resource group. This will cause the operator to grab as many task slots as its degree of parallelism. If there are no free resources available, the job will fail to start. It also disables chaining for this operator.

All subsequent operators are assigned to the default resource group.

Returns:

The operator with isolated resource group.