org.apache.flink.runtime.io.network.partition.hybrid.index

Class FileDataIndexSpilledRegionManagerImpl<T extends FileDataIndexRegionHelper.Region>

java.lang.Object

org.apache.flink.runtime.io.network.partition.hybrid.index.FileDataIndexSpilledRegionManagerImpl<T>

All Implemented Interfaces:

AutoCloseable, FileDataIndexSpilledRegionManager<T>

public class FileDataIndexSpilledRegionManagerImpl<T extends FileDataIndexRegionHelper.Region>
extends Object
implements FileDataIndexSpilledRegionManager<T>

Default implementation of FileDataIndexSpilledRegionManager. This manager will handle and spill regions in the following way:

• All regions will be written to the same file, namely index file.
• Multiple regions belonging to the same subpartition form a region group.
• The regions in the same region group have no special relationship, but are only related to the order in which they are spilled.
• Each region group is independent. Even if the previous region group is not full, the next region group can still be allocated.
• If a region has been written to the index file already, spill it again will overwrite the previous region.
• The very large region will monopolize a single region group.

The relationships between index file and region group are shown below.

         - - - - - - - - - Index File - - — - - - - - - - - -
        |                                                     |
        | - - — -RegionGroup1 - -   - - RegionGroup2- - - -   |
        ||SP1 R1｜｜SP1 R2｜ Free | |SP2 R3| SP2 R1| SP2 R2 |  |
        | - - - - - - - - - - - -   - - - - - - - - - - - -   |
        |                                                     |
        | - - - - - - - -RegionGroup3 - - - - -               |
        ||              Big Region             |              |
        | - - - - - - - - - - - - - - - - - - -               |
         - - - - - - - - - - - - - - - - - - - - - -- - - - -
 

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FileDataIndexSpilledRegionManagerImpl.Factory<T extends FileDataIndexRegionHelper.Region> Factory of FileDataIndexSpilledRegionManager.

Constructor Summary

Constructors

Constructor and Description
FileDataIndexSpilledRegionManagerImpl(int numSubpartitions, Path indexFilePath, int regionGroupSizeInBytes, long maxCacheCapacity, int regionHeaderSize, BiConsumer<Integer,T> cacheRegionConsumer, FileDataIndexRegionHelper<T> fileDataIndexRegionHelper)

Method Summary

Modifier and Type	Method and Description
void	appendOrOverwriteRegion(int subpartition, T newRegion) Write this region to index file.
void	close() Close this spilled region manager.
long	findRegion(int subpartition, int bufferIndex, boolean loadToCache) Find the region contains target bufferIndex and belong to target subpartition.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FileDataIndexSpilledRegionManagerImpl

public FileDataIndexSpilledRegionManagerImpl(int numSubpartitions,
                                             Path indexFilePath,
                                             int regionGroupSizeInBytes,
                                             long maxCacheCapacity,
                                             int regionHeaderSize,
                                             BiConsumer<Integer,T> cacheRegionConsumer,
                                             FileDataIndexRegionHelper<T> fileDataIndexRegionHelper)

Method Detail

findRegion

public long findRegion(int subpartition,
                       int bufferIndex,
                       boolean loadToCache)

Description copied from interface: FileDataIndexSpilledRegionManager

Find the region contains target bufferIndex and belong to target subpartition.

Specified by:

findRegion in interface FileDataIndexSpilledRegionManager<T extends FileDataIndexRegionHelper.Region>

Parameters:

subpartition - the subpartition id that target region belong to.

bufferIndex - the buffer index that target region contains.

loadToCache - whether to load the found region into the cache.

Returns:

if target region can be founded, return it's offset in index file. Otherwise, return -1.

appendOrOverwriteRegion

public void appendOrOverwriteRegion(int subpartition,
                                    T newRegion)
                             throws IOException

Description copied from interface: FileDataIndexSpilledRegionManager

Write this region to index file. If target region already spilled, overwrite it.

Specified by:

appendOrOverwriteRegion in interface FileDataIndexSpilledRegionManager<T extends FileDataIndexRegionHelper.Region>

Parameters:

subpartition - the subpartition id of this region.

newRegion - the region to be spilled to index file.

Throws:

IOException

close

public void close()
           throws IOException

Description copied from interface: FileDataIndexSpilledRegionManager

Close this spilled region manager.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface FileDataIndexSpilledRegionManager<T extends FileDataIndexRegionHelper.Region>

Throws:

IOException