org.apache.flink.streaming.api.transformations

Class StreamTransformation<T>

java.lang.Object

org.apache.flink.streaming.api.transformations.StreamTransformation<T>

Type Parameters:

T - The type of the elements that result from this StreamTransformation

Direct Known Subclasses:

CoFeedbackTransformation, FeedbackTransformation, OneInputTransformation, PartitionTransformation, SelectTransformation, SinkTransformation, SourceTransformation, SplitTransformation, TwoInputTransformation, UnionTransformation

public abstract class StreamTransformation<T>
extends Object

A StreamTransformation represents the operation that creates a DataStream. Every DataStream has an underlying StreamTransformation that is the origin of said DataStream.

API operations such as DataStream.map(org.apache.flink.api.common.functions.MapFunction<T, R>) create a tree of StreamTransformations underneath. When the stream program is to be executed this graph is translated to a StreamGraph using StreamGraphGenerator.

A StreamTransformation does not necessarily correspond to a physical operation at runtime. Some operations are only logical concepts. Examples of this are union, split/select data stream, partitioning.

The following graph of StreamTransformations:

Source              Source        
      +                   +           
      |                   |           
      v                   v           
  Rebalance          HashPartition    
      +                   +           
      |                   |           
      |                   |           
      +------>Union<------+           
                +                     
                |                     
                v                     
              Split                   
                +                     
                |                     
                v                     
              Select                  
                +                     
                v                     
               Map                    
                +                     
                |                     
                v                     
              Sink 
 

Would result in this graph of operations at runtime:

Source              Source
    +                   +
    |                   |
    |                   |
    +------->Map<-------+
              +
              |
              v
             Sink
 

The information about partitioning, union, split/select end up being encoded in the edges that connect the sources to the map operation.

Field Summary

Fields

Modifier and Type	Field and Description
protected long	bufferTimeout
protected int	id
protected static Integer	idCounter
protected String	name
protected TypeInformation<T>	outputType
protected StreamGraph.ResourceStrategy	resourceStrategy
protected boolean	typeUsed

Constructor Summary

Constructors

Constructor and Description
StreamTransformation(String name, TypeInformation<T> outputType, int parallelism) Creates a new StreamTransformation with the given name, output type and parallelism.

Method Summary

Methods

Modifier and Type	Method and Description
boolean	equals(Object o)
long	getBufferTimeout() Returns the buffer timeout of this StreamTransformation.
int	getId() Returns the unique ID of this StreamTransformation.
String	getName() Returns the name of this StreamTransformation.
static int	getNewNodeId()
TypeInformation<T>	getOutputType() Returns the output type of this StreamTransformation as a TypeInformation.
int	getParallelism() Returns the parallelism of this StreamTransformation
StreamGraph.ResourceStrategy	getResourceStrategy() Returns the ResourceStrategy of this StreamTransformation.
abstract Collection<StreamTransformation<?>>	getTransitivePredecessors() Returns all transitive predecessor StreamTransformations of this StreamTransformation.
int	hashCode()
void	setBufferTimeout(long bufferTimeout) Set the buffer timeout of this StreamTransformation.
abstract void	setChainingStrategy(ChainingStrategy strategy) Sets the chaining strategy of this StreamTransformation.
void	setName(String name) Changes the name of this StreamTransformation.
void	setOutputType(TypeInformation<T> outputType) Tries to fill in the type information.
void	setParallelism(int parallelism) Sets the parallelism of this StreamTransformation
void	setResourceStrategy(StreamGraph.ResourceStrategy resourceStrategy) Sets the StreamGraph.ResourceStrategy of this StreamTransformation.
String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

idCounter

protected static Integer idCounter

id

protected final int id

name

protected String name

outputType

protected TypeInformation<T> outputType

typeUsed

protected boolean typeUsed

bufferTimeout

protected long bufferTimeout

resourceStrategy

protected StreamGraph.ResourceStrategy resourceStrategy

Constructor Detail

StreamTransformation

public StreamTransformation(String name,
                    TypeInformation<T> outputType,
                    int parallelism)

Creates a new StreamTransformation with the given name, output type and parallelism.

Parameters:

name - The name of the StreamTransformation, this will be shown in Visualizations and the Log

outputType - The output type of this StreamTransformation

parallelism - The parallelism of this StreamTransformation

Method Detail

getNewNodeId

public static int getNewNodeId()

getId

public int getId()

Returns the unique ID of this StreamTransformation.

setName

public void setName(String name)

Changes the name of this StreamTransformation.

getName

public String getName()

Returns the name of this StreamTransformation.

getParallelism

public int getParallelism()

Returns the parallelism of this StreamTransformation

setParallelism

public void setParallelism(int parallelism)

Sets the parallelism of this StreamTransformation

Parameters:

parallelism - The new parallelism to set on this StreamTransformation

setOutputType

public void setOutputType(TypeInformation<T> outputType)

Tries to fill in the type information. Type information can be filled in later when the program uses a type hint. This method checks whether the type information has ever been accessed before and does not allow modifications if the type was accessed already. This ensures consistency by making sure different parts of the operation do not assume different type information.

Parameters:

outputType - The type information to fill in.

Throws:

IllegalStateException - Thrown, if the type information has been accessed before.

getOutputType

public TypeInformation<T> getOutputType()

Returns the output type of this StreamTransformation as a TypeInformation. Once this is used once the output type cannot be changed anymore using setOutputType(org.apache.flink.api.common.typeinfo.TypeInformation<T>).

Returns:

The output type of this StreamTransformation

setChainingStrategy

public abstract void setChainingStrategy(ChainingStrategy strategy)

Sets the chaining strategy of this StreamTransformation.

setBufferTimeout

public void setBufferTimeout(long bufferTimeout)

Set the buffer timeout of this StreamTransformation. The timeout is used when sending elements over the network. The timeout specifies how long a network buffer should be kept waiting before sending. A higher timeout means that more elements will be sent in one buffer, this increases throughput. The latency, however, is negatively affected by a higher timeout.

getBufferTimeout

public long getBufferTimeout()

Returns the buffer timeout of this StreamTransformation.

See Also:

setBufferTimeout(long)

setResourceStrategy

public void setResourceStrategy(StreamGraph.ResourceStrategy resourceStrategy)

Sets the StreamGraph.ResourceStrategy of this StreamTransformation. The resource strategy is used when scheduling operations on actual workers when transforming the StreamTopology to an ExecutionGraph.

getResourceStrategy

public StreamGraph.ResourceStrategy getResourceStrategy()

Returns the ResourceStrategy of this StreamTransformation.

See Also:

setResourceStrategy(StreamGraph.ResourceStrategy)

getTransitivePredecessors

public abstract Collection<StreamTransformation<?>> getTransitivePredecessors()

Returns all transitive predecessor StreamTransformations of this StreamTransformation. This is, for example, used when determining whether a feedback edge of an iteration actually has the iteration head as a predecessor.

Returns:

The list of transitive predecessors.

toString

public String toString()

Overrides:

toString in class Object

equals

public boolean equals(Object o)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object