SortBuffer (Flink : Runtime 1.20.1 API)

java.lang.Object
- org.apache.flink.runtime.io.network.partition.SortBuffer

All Implemented Interfaces:

DataBuffer

Direct Known Subclasses:

SortBasedDataBuffer, TieredStorageSortBuffer
```
@NotThreadSafe
public abstract class SortBuffer
extends Object
implements DataBuffer
```
A DataBuffer implementation which sorts all appended records only by subpartition index. Records of the same subpartition keep the appended order.
It maintains a list of MemorySegments as a joint buffer. Data will be appended to the joint buffer sequentially. When writing a record, an index entry will be appended first. An index entry consists of 4 fields: 4 bytes for record length, 4 bytes for Buffer.DataType and 8 bytes for address pointing to the next index entry of the same subpartition which will be used to index the next record to read when coping data from this DataBuffer. For simplicity, no index entry can span multiple segments. The corresponding record data is seated right after its index entry and different from the index entry, records have variable length thus may span multiple segments.

Field Summary

Fields
Modifier and Type	Field and Description
`protected BufferRecycler`	`bufferRecycler` `BufferRecycler` used to recycle `freeSegments`.
`protected LinkedList<org.apache.flink.core.memory.MemorySegment>`	`freeSegments` A list of `MemorySegment`s used to store data in memory.
`protected static int`	`INDEX_ENTRY_SIZE` Size of an index entry: 4 bytes for record length, 4 bytes for data type and 8 bytes for pointer to next entry.
`protected boolean`	`isFinished` Whether this sort buffer is finished.
`protected boolean`	`isReleased` Whether this sort buffer is released.
`protected long[]`	`lastIndexEntryAddresses` Addresses of the last record's index entry for each subpartition.
`protected long`	`numTotalBytesRead` Total number of bytes already read from this sort buffer.
`protected long`	`readIndexEntryAddress` Index entry address of the current record or event to be read.
`protected int`	`readOrderIndex` Used to index the current available subpartition to read data from.
`protected int`	`recordRemainingBytes` Record bytes remaining after last copy, which must be read first in next copy.
`ArrayList<org.apache.flink.core.memory.MemorySegment>`	`segments` A segment list as a joint buffer which stores all records and index entries.
`protected int[]`	`subpartitionReadOrder` Data of different subpartitions in this sort buffer will be read in this order.

Constructor Summary

Constructors
Modifier	Constructor and Description
`protected`	`SortBuffer(LinkedList<org.apache.flink.core.memory.MemorySegment> freeSegments, BufferRecycler bufferRecycler, int numSubpartitions, int bufferSize, int numGuaranteedBuffers, int[] customReadOrder)`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`boolean`	`append(ByteBuffer source, int targetSubpartition, Buffer.DataType dataType)` No partial record will be written to this `SortBasedDataBuffer`, which means that either all data of target record will be written or nothing will be written.
`protected int`	`copyRecordOrEvent(org.apache.flink.core.memory.MemorySegment targetSegment, int targetSegmentOffset, int sourceSegmentIndex, int sourceSegmentOffset, int recordLength)`
`void`	`finish()` Finishes this `DataBuffer` which means no record can be appended anymore.
`protected int`	`getSegmentIndexFromPointer(long value)`
`protected int`	`getSegmentOffsetFromPointer(long value)`
`boolean`	`hasRemaining()` Returns true if not all data appended to this `DataBuffer` is consumed.
`boolean`	`isFinished()` Whether this `DataBuffer` is finished or not.
`boolean`	`isReleased()` Whether this `DataBuffer` is released or not.
`long`	`numTotalBytes()` Returns the total number of bytes written to this `DataBuffer`.
`long`	`numTotalRecords()` Returns the total number of records written to this `DataBuffer`.
`void`	`release()` Releases this `DataBuffer` which releases all resources.
`protected void`	`updateReadSubpartitionAndIndexEntryAddress()`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.io.network.partition.DataBuffer
getNextBuffer

- Field Detail
  - INDEX_ENTRY_SIZE
```
protected static final int INDEX_ENTRY_SIZE
```
    Size of an index entry: 4 bytes for record length, 4 bytes for data type and 8 bytes for pointer to next entry.
    
    See Also:
    
    Constant Field Values
  - freeSegments
```
protected final LinkedList<org.apache.flink.core.memory.MemorySegment> freeSegments
```
    A list of MemorySegments used to store data in memory.
  - bufferRecycler
```
protected final BufferRecycler bufferRecycler
```
    BufferRecycler used to recycle freeSegments.
  - segments
```
public final ArrayList<org.apache.flink.core.memory.MemorySegment> segments
```
    A segment list as a joint buffer which stores all records and index entries.
  - lastIndexEntryAddresses
```
protected final long[] lastIndexEntryAddresses
```
    Addresses of the last record's index entry for each subpartition.
  - numTotalBytesRead
```
protected long numTotalBytesRead
```
    Total number of bytes already read from this sort buffer.
  - isFinished
```
protected boolean isFinished
```
    Whether this sort buffer is finished. One can only read a finished sort buffer.
  - isReleased
```
protected boolean isReleased
```
    Whether this sort buffer is released. A released sort buffer can not be used.
  - subpartitionReadOrder
```
protected final int[] subpartitionReadOrder
```
    Data of different subpartitions in this sort buffer will be read in this order.
  - readIndexEntryAddress
```
protected long readIndexEntryAddress
```
    Index entry address of the current record or event to be read.
  - recordRemainingBytes
```
protected int recordRemainingBytes
```
    Record bytes remaining after last copy, which must be read first in next copy.
  - readOrderIndex
```
protected int readOrderIndex
```
    Used to index the current available subpartition to read data from.
- Constructor Detail
  - SortBuffer
```
protected SortBuffer(LinkedList<org.apache.flink.core.memory.MemorySegment> freeSegments,
                     BufferRecycler bufferRecycler,
                     int numSubpartitions,
                     int bufferSize,
                     int numGuaranteedBuffers,
                     @Nullable
                     int[] customReadOrder)
```
- Method Detail
  - append
```
public boolean append(ByteBuffer source,
                      int targetSubpartition,
                      Buffer.DataType dataType)
               throws IOException
```
    No partial record will be written to this SortBasedDataBuffer, which means that either all data of target record will be written or nothing will be written.
    
    Specified by:
    
    append in interface DataBuffer
    
    Throws:
    
    IOException
  - copyRecordOrEvent
```
protected int copyRecordOrEvent(org.apache.flink.core.memory.MemorySegment targetSegment,
                                int targetSegmentOffset,
                                int sourceSegmentIndex,
                                int sourceSegmentOffset,
                                int recordLength)
```
  - updateReadSubpartitionAndIndexEntryAddress
```
protected void updateReadSubpartitionAndIndexEntryAddress()
```
  - getSegmentIndexFromPointer
```
protected int getSegmentIndexFromPointer(long value)
```
  - getSegmentOffsetFromPointer
```
protected int getSegmentOffsetFromPointer(long value)
```
  - numTotalRecords
```
public long numTotalRecords()
```
    Description copied from interface: DataBuffer
    
    Returns the total number of records written to this DataBuffer.
    
    Specified by:
    
    numTotalRecords in interface DataBuffer
  - numTotalBytes
```
public long numTotalBytes()
```
    Description copied from interface: DataBuffer
    
    Returns the total number of bytes written to this DataBuffer.
    
    Specified by:
    
    numTotalBytes in interface DataBuffer
  - hasRemaining
```
public boolean hasRemaining()
```
    Description copied from interface: DataBuffer
    
    Returns true if not all data appended to this DataBuffer is consumed.
    
    Specified by:
    
    hasRemaining in interface DataBuffer
  - finish
```
public void finish()
```
    Description copied from interface: DataBuffer
    
    Finishes this DataBuffer which means no record can be appended anymore.
    
    Specified by:
    
    finish in interface DataBuffer
  - isFinished
```
public boolean isFinished()
```
    Description copied from interface: DataBuffer
    
    Whether this DataBuffer is finished or not.
    
    Specified by:
    
    isFinished in interface DataBuffer
  - release
```
public void release()
```
    Description copied from interface: DataBuffer
    
    Releases this DataBuffer which releases all resources.
    
    Specified by:
    
    release in interface DataBuffer
  - isReleased
```
public boolean isReleased()
```
    Description copied from interface: DataBuffer
    
    Whether this DataBuffer is released or not.
    
    Specified by:
    
    isReleased in interface DataBuffer

Class SortBuffer

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Methods inherited from interface org.apache.flink.runtime.io.network.partition.DataBuffer

Field Detail

INDEX_ENTRY_SIZE

freeSegments

bufferRecycler

segments

lastIndexEntryAddresses

numTotalBytesRead

isFinished

isReleased

subpartitionReadOrder

readIndexEntryAddress

recordRemainingBytes

readOrderIndex

Constructor Detail

SortBuffer

Method Detail

append

copyRecordOrEvent

updateReadSubpartitionAndIndexEntryAddress

getSegmentIndexFromPointer

getSegmentOffsetFromPointer

numTotalRecords

numTotalBytes

hasRemaining

finish

isFinished

release

isReleased