Package org.apache.flink.runtime.io.network.partition

Class SortBuffer

  • All Implemented Interfaces:
    DataBuffer
    Direct Known Subclasses:
    SortBasedDataBuffer, TieredStorageSortBuffer
    @NotThreadSafe
public abstract class SortBuffer
extends Object
implements DataBuffer
    A DataBuffer implementation which sorts all appended records only by subpartition index. Records of the same subpartition keep the appended order.

    It maintains a list of MemorySegments as a joint buffer. Data will be appended to the joint buffer sequentially. When writing a record, an index entry will be appended first. An index entry consists of 4 fields: 4 bytes for record length, 4 bytes for Buffer.DataType and 8 bytes for address pointing to the next index entry of the same subpartition which will be used to index the next record to read when coping data from this DataBuffer. For simplicity, no index entry can span multiple segments. The corresponding record data is seated right after its index entry and different from the index entry, records have variable length thus may span multiple segments.

    • Field Detail

      • INDEX_ENTRY_SIZE

        protected static final int INDEX_ENTRY_SIZE
        Size of an index entry: 4 bytes for record length, 4 bytes for data type and 8 bytes for pointer to next entry.
        See Also:
        Constant Field Values

      • freeSegments

        protected final LinkedList<org.apache.flink.core.memory.MemorySegment> freeSegments
        A list of MemorySegments used to store data in memory.

      • segments

        public final ArrayList<org.apache.flink.core.memory.MemorySegment> segments
        A segment list as a joint buffer which stores all records and index entries.

      • lastIndexEntryAddresses

        protected final long[] lastIndexEntryAddresses
        Addresses of the last record's index entry for each subpartition.

      • numTotalBytesRead

        protected long numTotalBytesRead
        Total number of bytes already read from this sort buffer.

      • isFinished

        protected boolean isFinished
        Whether this sort buffer is finished. One can only read a finished sort buffer.

      • isReleased

        protected boolean isReleased
        Whether this sort buffer is released. A released sort buffer can not be used.

      • subpartitionReadOrder

        protected final int[] subpartitionReadOrder
        Data of different subpartitions in this sort buffer will be read in this order.

      • readIndexEntryAddress

        protected long readIndexEntryAddress
        Index entry address of the current record or event to be read.

      • recordRemainingBytes

        protected int recordRemainingBytes
        Record bytes remaining after last copy, which must be read first in next copy.

      • readOrderIndex

        protected int readOrderIndex
        Used to index the current available subpartition to read data from.

    • Constructor Detail

      • SortBuffer

        protected SortBuffer​(LinkedList<org.apache.flink.core.memory.MemorySegment> freeSegments,
                     BufferRecycler bufferRecycler,
                     int numSubpartitions,
                     int bufferSize,
                     int numGuaranteedBuffers,
                     @Nullable
                     int[] customReadOrder)

    • Method Detail

      • copyRecordOrEvent

        protected int copyRecordOrEvent​(org.apache.flink.core.memory.MemorySegment targetSegment,
                                int targetSegmentOffset,
                                int sourceSegmentIndex,
                                int sourceSegmentOffset,
                                int recordLength)

      • updateReadSubpartitionAndIndexEntryAddress

        protected void updateReadSubpartitionAndIndexEntryAddress()

      • getSegmentIndexFromPointer

        protected int getSegmentIndexFromPointer​(long value)

      • getSegmentOffsetFromPointer

        protected int getSegmentOffsetFromPointer​(long value)

      • hasRemaining

        public boolean hasRemaining()
        Description copied from interface: DataBuffer
        Returns true if not all data appended to this DataBuffer is consumed.
        Specified by:
        hasRemaining in interface DataBuffer

      • finish

        public void finish()
        Description copied from interface: DataBuffer
        Finishes this DataBuffer which means no record can be appended anymore.
        Specified by:
        finish in interface DataBuffer