org.apache.flink.runtime.io.network.partition

Class PipelinedSubpartition

java.lang.Object

org.apache.flink.runtime.io.network.partition.ResultSubpartition

org.apache.flink.runtime.io.network.partition.PipelinedSubpartition

All Implemented Interfaces:

ChannelStateHolder

Direct Known Subclasses:

PipelinedApproximateSubpartition

public class PipelinedSubpartition
extends ResultSubpartition
implements ChannelStateHolder

A pipelined in-memory only subpartition, which can be consumed once.

Whenever ResultSubpartition.add(BufferConsumer) adds a finished BufferConsumer or a second BufferConsumer (in which case we will assume the first one finished), we will notify a read view created via ResultSubpartition.createReadView(BufferAvailabilityListener) of new data availability. Except by calling flush() explicitly, we always only notify when the first finished buffer turns up and then, the reader has to drain the buffers via pollBuffer() until its return value shows no more buffers being available. This results in a buffer queue which is either empty or has an unfinished BufferConsumer left from which the notifications will eventually start again.

Explicit calls to flush() will force this notification for any BufferConsumer present in the queue.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.runtime.io.network.partition.ResultSubpartition

ResultSubpartition.BufferAndBacklog

Field Summary

Fields inherited from class org.apache.flink.runtime.io.network.partition.ResultSubpartition

ADD_BUFFER_ERROR_CODE, parent, subpartitionInfo

Method Summary

Modifier and Type	Method and Description
void	abortCheckpoint(long checkpointId, CheckpointException cause)
void	acknowledgeAllDataProcessed()
int	add(BufferConsumer bufferConsumer, int partialRecordLength) Adds the given buffer.
void	alignedBarrierTimeout(long checkpointId)
void	bufferSize(int desirableNewBufferSize)
PipelinedSubpartitionView	createReadView(BufferAvailabilityListener availabilityListener)
int	finish() Writing of data is finished.
void	flush()
ResultSubpartitionView.AvailabilityWithBacklog	getAvailabilityAndBacklog(boolean isCreditAvailable)
int	getBuffersInBacklogUnsafe() Gets the number of non-event buffers in this subpartition.
long	getChannelStateCheckpointId()
int	getNumberOfQueuedBuffers() Get the current size of the queue.
protected long	getTotalNumberOfBuffersUnsafe() Gets the total numbers of buffers (data buffers plus events).
protected long	getTotalNumberOfBytesUnsafe()
boolean	isReleased()
boolean	isSupportChannelStateRecover()
void	release()
void	setChannelStateWriter(ChannelStateWriter channelStateWriter) Injects the ChannelStateWriter.
String	toString()
int	unsynchronizedGetNumberOfQueuedBuffers() Makes a best effort to get the current size of the queue.

Methods inherited from class org.apache.flink.runtime.io.network.partition.ResultSubpartition

add, getSubPartitionIndex, getSubpartitionInfo, onConsumedSubpartition

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Method Detail

setChannelStateWriter

public void setChannelStateWriter(ChannelStateWriter channelStateWriter)

Description copied from interface: ChannelStateHolder

Injects the ChannelStateWriter. Must only be called once.

Specified by:

setChannelStateWriter in interface ChannelStateHolder

add

public int add(BufferConsumer bufferConsumer,
               int partialRecordLength)

Description copied from class: ResultSubpartition

Adds the given buffer.

The request may be executed synchronously, or asynchronously, depending on the implementation.

IMPORTANT: Before adding new BufferConsumer previously added must be in finished state. Because of the performance reasons, this is only enforced during the data reading. Priority events can be added while the previous buffer consumer is still open, in which case the open buffer consumer is overtaken.

Specified by:

add in class ResultSubpartition

Parameters:

bufferConsumer - the buffer to add (transferring ownership to this writer)

partialRecordLength - the length of bytes to skip in order to start with a complete record, from position index 0 of the underlying .

Returns:

the preferable buffer size for this subpartition or ResultSubpartition.ADD_BUFFER_ERROR_CODE if the add operation fails.

isSupportChannelStateRecover

public boolean isSupportChannelStateRecover()

finish

public int finish()
           throws IOException

Description copied from class: ResultSubpartition

Writing of data is finished.

Specified by:

finish in class ResultSubpartition

Returns:

the size of data written for this subpartition inside of finish.

Throws:

IOException

alignedBarrierTimeout

public void alignedBarrierTimeout(long checkpointId)
                           throws IOException

Specified by:

alignedBarrierTimeout in class ResultSubpartition

Throws:

IOException

abortCheckpoint

public void abortCheckpoint(long checkpointId,
                            CheckpointException cause)

Specified by:

abortCheckpoint in class ResultSubpartition

release

public void release()

Specified by:

release in class ResultSubpartition

acknowledgeAllDataProcessed

public void acknowledgeAllDataProcessed()

isReleased

public boolean isReleased()

Specified by:

isReleased in class ResultSubpartition

createReadView

public PipelinedSubpartitionView createReadView(BufferAvailabilityListener availabilityListener)

Specified by:

createReadView in class ResultSubpartition

getAvailabilityAndBacklog

public ResultSubpartitionView.AvailabilityWithBacklog getAvailabilityAndBacklog(boolean isCreditAvailable)

getNumberOfQueuedBuffers

public int getNumberOfQueuedBuffers()

Description copied from class: ResultSubpartition

Get the current size of the queue.

Specified by:

getNumberOfQueuedBuffers in class ResultSubpartition

bufferSize

public void bufferSize(int desirableNewBufferSize)

Specified by:

bufferSize in class ResultSubpartition

toString

public String toString()

Overrides:

toString in class Object

unsynchronizedGetNumberOfQueuedBuffers

public int unsynchronizedGetNumberOfQueuedBuffers()

Description copied from class: ResultSubpartition

Makes a best effort to get the current size of the queue. This method must not acquire locks or interfere with the task and network threads in any way.

Specified by:

unsynchronizedGetNumberOfQueuedBuffers in class ResultSubpartition

flush

public void flush()

Specified by:

flush in class ResultSubpartition

getTotalNumberOfBuffersUnsafe

protected long getTotalNumberOfBuffersUnsafe()

Description copied from class: ResultSubpartition

Gets the total numbers of buffers (data buffers plus events).

Specified by:

getTotalNumberOfBuffersUnsafe in class ResultSubpartition

getTotalNumberOfBytesUnsafe

protected long getTotalNumberOfBytesUnsafe()

Specified by:

getTotalNumberOfBytesUnsafe in class ResultSubpartition

getBuffersInBacklogUnsafe

public int getBuffersInBacklogUnsafe()

Gets the number of non-event buffers in this subpartition.

getChannelStateCheckpointId

@VisibleForTesting
public long getChannelStateCheckpointId()