Class SortMergeResultPartition
- java.lang.Object
-
- org.apache.flink.runtime.io.network.partition.ResultPartition
-
- org.apache.flink.runtime.io.network.partition.SortMergeResultPartition
-
- All Implemented Interfaces:
AutoCloseable,AvailabilityProvider,ResultPartitionWriter
@NotThreadSafe public class SortMergeResultPartition extends ResultPartition
SortMergeResultPartitionappends records and events toDataBufferand after theDataBufferis full, all data in theDataBufferwill be copied and spilled to aPartitionedFilein subpartition index order sequentially. Large records that can not be appended to an emptyDataBufferwill be spilled to the resultPartitionedFileseparately.
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from interface org.apache.flink.runtime.io.AvailabilityProvider
AvailabilityProvider.AvailabilityHelper
-
-
Field Summary
-
Fields inherited from class org.apache.flink.runtime.io.network.partition.ResultPartition
bufferCompressor, bufferPool, LOG, numBuffersOut, numBytesOut, numSubpartitions, partitionId, partitionManager, partitionType, resultPartitionBytes
-
Fields inherited from interface org.apache.flink.runtime.io.AvailabilityProvider
AVAILABLE
-
-
Constructor Summary
Constructors Constructor Description SortMergeResultPartition(String owningTaskName, int partitionIndex, ResultPartitionID partitionId, ResultPartitionType partitionType, int numSubpartitions, int numTargetKeyGroups, BatchShuffleReadBufferPool readBufferPool, Executor readIOExecutor, ResultPartitionManager partitionManager, String resultFileBasePath, BufferCompressor bufferCompressor, org.apache.flink.util.function.SupplierWithException<BufferPool,IOException> bufferPoolFactory)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description voidabortCheckpoint(long checkpointId, CheckpointException cause)Abort the checkpoint.voidalignedBarrierTimeout(long checkpointId)Timeout the aligned barrier to unaligned barrier.voidbroadcastEvent(AbstractEvent event, boolean isPriorityEvent)Writes the givenAbstractEventto all subpartitions.voidbroadcastRecord(ByteBuffer record)Writes the given serialized record to all subpartitions.voidclose()Closes the partition writer which releases the allocated resource, for example the buffer pool.protected ResultSubpartitionViewcreateSubpartitionView(int subpartitionIndex, BufferAvailabilityListener availabilityListener)Returns a reader for the subpartition with the given index.voidemitRecord(ByteBuffer record, int targetSubpartition)Writes the given serialized record to the target subpartition.voidfinish()Finishes the result partition.voidflush(int subpartitionIndex)Manually trigger the consumption of data from the given subpartitions.voidflushAll()Manually trigger the consumption of data from all subpartitions.CompletableFuture<?>getAvailableFuture()intgetNumberOfQueuedBuffers()Returns the total number of queued buffers of all subpartitions.intgetNumberOfQueuedBuffers(int targetSubpartition)Returns the number of queued buffers of the given target subpartition.longgetSizeOfQueuedBuffersUnsafe()Returns the total size in bytes of queued buffers of all subpartitions.voidnotifyEndOfData(StopMode mode)Notifies the downstream tasks that thisResultPartitionWriterhave emitted all the user records.protected voidreleaseInternal()Releases all produced data including both those stored in memory and persisted on disk.protected voidsetupInternal()Do the subclass's own setup operation.-
Methods inherited from class org.apache.flink.runtime.io.network.partition.ResultPartition
canBeCompressed, checkInProduceState, createSubpartitionView, fail, getAllDataProcessedFuture, getBufferPool, getFailureCause, getNumberOfSubpartitions, getNumTargetKeyGroups, getOwningTaskName, getPartitionId, getPartitionIndex, getPartitionManager, getPartitionType, getResultPartitionBytes, isFinished, isNumberOfPartitionConsumerUndefined, isNumberOfPartitionConsumerUndefined, isReleased, onSubpartitionAllDataProcessed, release, release, setMaxOverdraftBuffersPerGate, setMetricGroup, setup, toString
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
-
Methods inherited from interface org.apache.flink.runtime.io.AvailabilityProvider
isApproximatelyAvailable, isAvailable
-
-
-
-
Constructor Detail
-
SortMergeResultPartition
public SortMergeResultPartition(String owningTaskName, int partitionIndex, ResultPartitionID partitionId, ResultPartitionType partitionType, int numSubpartitions, int numTargetKeyGroups, BatchShuffleReadBufferPool readBufferPool, Executor readIOExecutor, ResultPartitionManager partitionManager, String resultFileBasePath, @Nullable BufferCompressor bufferCompressor, org.apache.flink.util.function.SupplierWithException<BufferPool,IOException> bufferPoolFactory)
-
-
Method Detail
-
setupInternal
protected void setupInternal() throws IOExceptionDescription copied from class:ResultPartitionDo the subclass's own setup operation.- Specified by:
setupInternalin classResultPartition- Throws:
IOException
-
releaseInternal
protected void releaseInternal()
Description copied from class:ResultPartitionReleases all produced data including both those stored in memory and persisted on disk.- Specified by:
releaseInternalin classResultPartition
-
emitRecord
public void emitRecord(ByteBuffer record, int targetSubpartition) throws IOException
Description copied from interface:ResultPartitionWriterWrites the given serialized record to the target subpartition.- Throws:
IOException
-
broadcastRecord
public void broadcastRecord(ByteBuffer record) throws IOException
Description copied from interface:ResultPartitionWriterWrites the given serialized record to all subpartitions. One can also achieve the same effect by emitting the same record to all subpartitions one by one, however, this method can have better performance for the underlying implementation can do some optimizations, for example coping the given serialized record only once to a shared channel which can be consumed by all subpartitions.- Throws:
IOException
-
broadcastEvent
public void broadcastEvent(AbstractEvent event, boolean isPriorityEvent) throws IOException
Description copied from interface:ResultPartitionWriterWrites the givenAbstractEventto all subpartitions.- Throws:
IOException
-
alignedBarrierTimeout
public void alignedBarrierTimeout(long checkpointId) throws IOExceptionDescription copied from interface:ResultPartitionWriterTimeout the aligned barrier to unaligned barrier.- Throws:
IOException
-
abortCheckpoint
public void abortCheckpoint(long checkpointId, CheckpointException cause)Description copied from interface:ResultPartitionWriterAbort the checkpoint.
-
notifyEndOfData
public void notifyEndOfData(StopMode mode) throws IOException
Description copied from interface:ResultPartitionWriterNotifies the downstream tasks that thisResultPartitionWriterhave emitted all the user records.- Specified by:
notifyEndOfDatain interfaceResultPartitionWriter- Overrides:
notifyEndOfDatain classResultPartition- Parameters:
mode- tells if we should flush all records or not (it is false in case of stop-with-savepoint (--no-drain))- Throws:
IOException
-
finish
public void finish() throws IOExceptionDescription copied from class:ResultPartitionFinishes the result partition.After this operation, it is not possible to add further data to the result partition.
For BLOCKING results, this will trigger the deployment of consuming tasks.
- Specified by:
finishin interfaceResultPartitionWriter- Overrides:
finishin classResultPartition- Throws:
IOException
-
close
public void close()
Description copied from interface:ResultPartitionWriterCloses the partition writer which releases the allocated resource, for example the buffer pool.- Specified by:
closein interfaceAutoCloseable- Specified by:
closein interfaceResultPartitionWriter- Overrides:
closein classResultPartition
-
createSubpartitionView
protected ResultSubpartitionView createSubpartitionView(int subpartitionIndex, BufferAvailabilityListener availabilityListener) throws IOException
Description copied from class:ResultPartitionReturns a reader for the subpartition with the given index.Given that the function to merge outputs from multiple subpartition views is supported uniformly in
UnionResultSubpartitionView, subclasses ofResultPartitiononly needs to take care of creating subpartition view for a single subpartition.- Specified by:
createSubpartitionViewin classResultPartition- Throws:
IOException
-
flushAll
public void flushAll()
Description copied from interface:ResultPartitionWriterManually trigger the consumption of data from all subpartitions.
-
flush
public void flush(int subpartitionIndex)
Description copied from interface:ResultPartitionWriterManually trigger the consumption of data from the given subpartitions.
-
getAvailableFuture
public CompletableFuture<?> getAvailableFuture()
- Specified by:
getAvailableFuturein interfaceAvailabilityProvider- Overrides:
getAvailableFuturein classResultPartition- Returns:
- a future that is completed if the respective provider is available.
-
getNumberOfQueuedBuffers
public int getNumberOfQueuedBuffers()
Description copied from class:ResultPartitionReturns the total number of queued buffers of all subpartitions.- Specified by:
getNumberOfQueuedBuffersin classResultPartition
-
getSizeOfQueuedBuffersUnsafe
public long getSizeOfQueuedBuffersUnsafe()
Description copied from class:ResultPartitionReturns the total size in bytes of queued buffers of all subpartitions.- Specified by:
getSizeOfQueuedBuffersUnsafein classResultPartition
-
getNumberOfQueuedBuffers
public int getNumberOfQueuedBuffers(int targetSubpartition)
Description copied from class:ResultPartitionReturns the number of queued buffers of the given target subpartition.- Specified by:
getNumberOfQueuedBuffersin classResultPartition
-
-