org.apache.flink.runtime.io.network.partition

Class UnionResultSubpartitionView

java.lang.Object

org.apache.flink.runtime.io.network.partition.UnionResultSubpartitionView

All Implemented Interfaces:

BufferAvailabilityListener, ResultSubpartitionView

public class UnionResultSubpartitionView
extends Object
implements ResultSubpartitionView, BufferAvailabilityListener

A wrapper to union the output from multiple ResultSubpartitionViews. This class provides the following guarantees to the output buffers.

• Each output buffer corresponds to a buffer in one of the subpartitions.
• Buffers in the same subpartition are output without their order changed.
• If a record is split and placed into multiple adjacent buffers due to the capacity limit of the buffer, these buffers will be output consecutively without the entry of buffers from other subpartitions in between.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.runtime.io.network.partition.ResultSubpartitionView

ResultSubpartitionView.AvailabilityWithBacklog

Constructor Summary

Constructors

Constructor and Description
UnionResultSubpartitionView(BufferAvailabilityListener availabilityListener, int numTotalViews)

Method Summary

Modifier and Type	Method and Description
void	acknowledgeAllDataProcessed()
ResultSubpartitionView.AvailabilityWithBacklog	getAvailabilityAndBacklog(boolean isCreditAvailable) Get the availability and backlog of the view.
Throwable	getFailureCause() ResultSubpartitionView can decide whether the failure cause should be reported to consumer as failure (primary failure) or ProducerFailedException (secondary failure).
ResultSubpartition.BufferAndBacklog	getNextBuffer() Returns the next Buffer instance of this queue iterator.
int	getNumberOfQueuedBuffers()
boolean	isReleased()
void	notifyDataAvailable()
void	notifyDataAvailable(ResultSubpartitionView view) Called whenever there might be new data available.
void	notifyNewBufferSize(int newBufferSize)
void	notifyPriorityEvent(int priorityBufferNumber) Called when the first priority event is added to the head of the buffer queue.
void	notifyRequiredSegmentId(int subpartitionId, int segmentId) In tiered storage shuffle mode, only required segments will be sent to prevent the redundant buffer usage.
void	notifyViewCreated(int subpartitionId, ResultSubpartitionView view)
int	peekNextBufferSubpartitionId() Returns the index of the subpartition where the next buffer locates, or -1 if there is no buffer available and the subpartition to be consumed is not determined.
void	releaseAllResources()
void	resumeConsumption()
int	unsynchronizedGetNumberOfQueuedBuffers()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

UnionResultSubpartitionView

public UnionResultSubpartitionView(BufferAvailabilityListener availabilityListener,
                                   int numTotalViews)

Method Detail

notifyViewCreated

public void notifyViewCreated(int subpartitionId,
                              ResultSubpartitionView view)

peekNextBufferSubpartitionId

public int peekNextBufferSubpartitionId()
                                 throws IOException

Description copied from interface: ResultSubpartitionView

Returns the index of the subpartition where the next buffer locates, or -1 if there is no buffer available and the subpartition to be consumed is not determined.

Specified by:

peekNextBufferSubpartitionId in interface ResultSubpartitionView

Throws:

IOException

getNextBuffer

@Nullable
public ResultSubpartition.BufferAndBacklog getNextBuffer()
                                                            throws IOException

Description copied from interface: ResultSubpartitionView

Returns the next Buffer instance of this queue iterator.

If there is currently no instance available, it will return null. This might happen for example when a pipelined queue producer is slower than the consumer or a spilled queue needs to read in more data.

Important: The consumer has to make sure that each buffer instance will eventually be recycled with Buffer.recycleBuffer() after it has been consumed.

Specified by:

getNextBuffer in interface ResultSubpartitionView

Throws:

IOException

notifyDataAvailable

public void notifyDataAvailable()

Specified by:

notifyDataAvailable in interface ResultSubpartitionView

notifyDataAvailable

public void notifyDataAvailable(ResultSubpartitionView view)

Description copied from interface: BufferAvailabilityListener

Called whenever there might be new data available.

Specified by:

notifyDataAvailable in interface BufferAvailabilityListener

Parameters:

view - the ResultSubpartitionView containing available data.

notifyPriorityEvent

public void notifyPriorityEvent(int priorityBufferNumber)

Description copied from interface: BufferAvailabilityListener

Called when the first priority event is added to the head of the buffer queue.

Specified by:

notifyPriorityEvent in interface BufferAvailabilityListener

Specified by:

notifyPriorityEvent in interface ResultSubpartitionView

Parameters:

priorityBufferNumber - the sequence number that identifies the priority buffer.

releaseAllResources

public void releaseAllResources()
                         throws IOException

Specified by:

releaseAllResources in interface ResultSubpartitionView

Throws:

IOException

isReleased

public boolean isReleased()

Specified by:

isReleased in interface ResultSubpartitionView

resumeConsumption

public void resumeConsumption()

Specified by:

resumeConsumption in interface ResultSubpartitionView

acknowledgeAllDataProcessed

public void acknowledgeAllDataProcessed()

Specified by:

acknowledgeAllDataProcessed in interface ResultSubpartitionView

getFailureCause

public Throwable getFailureCause()

Description copied from interface: ResultSubpartitionView

ResultSubpartitionView can decide whether the failure cause should be reported to consumer as failure (primary failure) or ProducerFailedException (secondary failure). Secondary failure can be reported only if producer (upstream task) is guaranteed to failover.

BEWARE: Incorrectly reporting failure cause as primary failure, can hide the root cause of the failure from the user.

Specified by:

getFailureCause in interface ResultSubpartitionView

getAvailabilityAndBacklog

public ResultSubpartitionView.AvailabilityWithBacklog getAvailabilityAndBacklog(boolean isCreditAvailable)

Description copied from interface: ResultSubpartitionView

Get the availability and backlog of the view. The availability represents if the view is ready to get buffer from it. The backlog represents the number of available data buffers.

Specified by:

getAvailabilityAndBacklog in interface ResultSubpartitionView

Parameters:

isCreditAvailable - the availability of credits for this ResultSubpartitionView.

Returns:

availability and backlog.

notifyRequiredSegmentId

public void notifyRequiredSegmentId(int subpartitionId,
                                    int segmentId)

Description copied from interface: ResultSubpartitionView

In tiered storage shuffle mode, only required segments will be sent to prevent the redundant buffer usage. Downstream will notify the upstream by this method to send required segments.

Specified by:

notifyRequiredSegmentId in interface ResultSubpartitionView

Parameters:

subpartitionId - The id of the corresponding subpartition.

segmentId - The id of required segment.

unsynchronizedGetNumberOfQueuedBuffers

public int unsynchronizedGetNumberOfQueuedBuffers()

Specified by:

unsynchronizedGetNumberOfQueuedBuffers in interface ResultSubpartitionView

getNumberOfQueuedBuffers

public int getNumberOfQueuedBuffers()

Specified by:

getNumberOfQueuedBuffers in interface ResultSubpartitionView

notifyNewBufferSize

public void notifyNewBufferSize(int newBufferSize)

Specified by:

notifyNewBufferSize in interface ResultSubpartitionView