org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote

Class RemoteTierConsumerAgent

java.lang.Object

org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote.RemoteTierConsumerAgent

All Implemented Interfaces:

TierConsumerAgent

public class RemoteTierConsumerAgent
extends Object
implements TierConsumerAgent

The data client is used to fetch data from remote tier.

Constructor Summary

Constructors

Constructor and Description
RemoteTierConsumerAgent(List<TieredStorageConsumerSpec> tieredStorageConsumerSpecs, RemoteStorageScanner remoteStorageScanner, PartitionFileReader partitionFileReader, int bufferSizeBytes)

Method Summary

Modifier and Type	Method and Description
void	close() Close the consumer agent.
Optional<Buffer>	getNextBuffer(TieredStoragePartitionId partitionId, TieredStorageSubpartitionId subpartitionId, int segmentId) Get buffer from the consumer agent.
void	notifyAvailable(TieredStoragePartitionId partitionId, TieredStorageSubpartitionId subpartitionId)
int	peekNextBufferSubpartitionId(TieredStoragePartitionId partitionId, ResultSubpartitionIndexSet indexSet) Returns the index of the subpartition where the next buffer locates, or -1 if there is no buffer available or the subpartition index does not belong to the specified indexSet.
void	registerAvailabilityNotifier(AvailabilityNotifier notifier) Register the notifier to notify the availability of a subpartition.
void	setup(TieredStorageMemoryManager memoryManager) The consumer agent may request buffers from the memory manager.
void	start() Start the consumer agent.
void	updateTierShuffleDescriptor(TieredStoragePartitionId partitionId, TieredStorageInputChannelId inputChannelId, TieredStorageSubpartitionId subpartitionId, TierShuffleDescriptor tierShuffleDescriptor) Update the TierShuffleDescriptor for the consumer agent.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RemoteTierConsumerAgent

public RemoteTierConsumerAgent(List<TieredStorageConsumerSpec> tieredStorageConsumerSpecs,
                               RemoteStorageScanner remoteStorageScanner,
                               PartitionFileReader partitionFileReader,
                               int bufferSizeBytes)

Method Detail

setup

public void setup(TieredStorageMemoryManager memoryManager)

Description copied from interface: TierConsumerAgent

The consumer agent may request buffers from the memory manager. Therefore, the TieredStorageMemoryManager should be integrated into the tier consumer agent. Since the buffer pool is initialized after the creation of the client, the memory manager need to be assigned after the buffer pool becomes available.

Specified by:

setup in interface TierConsumerAgent

start

public void start()

Description copied from interface: TierConsumerAgent

Start the consumer agent.

Specified by:

start in interface TierConsumerAgent

peekNextBufferSubpartitionId

public int peekNextBufferSubpartitionId(TieredStoragePartitionId partitionId,
                                        ResultSubpartitionIndexSet indexSet)
                                 throws IOException

Description copied from interface: TierConsumerAgent

Returns the index of the subpartition where the next buffer locates, or -1 if there is no buffer available or the subpartition index does not belong to the specified indexSet.

Specified by:

peekNextBufferSubpartitionId in interface TierConsumerAgent

Parameters:

partitionId - The index of the partition which the returned subpartition should belong to.

indexSet - The indexes of the subpartitions expected.

Throws:

IOException

getNextBuffer

public Optional<Buffer> getNextBuffer(TieredStoragePartitionId partitionId,
                                      TieredStorageSubpartitionId subpartitionId,
                                      int segmentId)

Description copied from interface: TierConsumerAgent

Get buffer from the consumer agent.

Specified by:

getNextBuffer in interface TierConsumerAgent

Parameters:

partitionId - the id of partition.

subpartitionId - the id of subpartition.

segmentId - the id of segment.

Returns:

buffer.

registerAvailabilityNotifier

public void registerAvailabilityNotifier(AvailabilityNotifier notifier)

Description copied from interface: TierConsumerAgent

Register the notifier to notify the availability of a subpartition.

Specified by:

registerAvailabilityNotifier in interface TierConsumerAgent

Parameters:

notifier - to notify availability.

updateTierShuffleDescriptor

public void updateTierShuffleDescriptor(TieredStoragePartitionId partitionId,
                                        TieredStorageInputChannelId inputChannelId,
                                        TieredStorageSubpartitionId subpartitionId,
                                        TierShuffleDescriptor tierShuffleDescriptor)

Description copied from interface: TierConsumerAgent

Update the TierShuffleDescriptor for the consumer agent.

Specified by:

updateTierShuffleDescriptor in interface TierConsumerAgent

close

public void close()
           throws IOException

Description copied from interface: TierConsumerAgent

Close the consumer agent.

Specified by:

close in interface TierConsumerAgent

Throws:

IOException

notifyAvailable

public void notifyAvailable(TieredStoragePartitionId partitionId,
                            TieredStorageSubpartitionId subpartitionId)