org.apache.flink.runtime.io.network.partition.consumer

类 SingleInputGate

java.lang.Object

org.apache.flink.runtime.io.network.partition.consumer.InputGate

org.apache.flink.runtime.io.network.partition.consumer.IndexedInputGate

org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate

所有已实现的接口:

AutoCloseable, AvailabilityProvider, ChannelStateHolder, CheckpointableInput, PullingAsyncDataInput<BufferOrEvent>

public class SingleInputGate
extends IndexedInputGate

An input gate consumes one or more partitions of a single produced intermediate result.

Each intermediate result is partitioned over its producing parallel subtasks; each of these partitions is furthermore partitioned into one or more subpartitions.

As an example, consider a map-reduce program, where the map operator produces data and the reduce operator consumes the produced data.

 +-----+              +---------------------+              +--------+
 | Map | = produce => | Intermediate Result | <= consume = | Reduce |
 +-----+              +---------------------+              +--------+
 

When deploying such a program in parallel, the intermediate result will be partitioned over its producing parallel subtasks; each of these partitions is furthermore partitioned into one or more subpartitions.

                            Intermediate result
               +-----------------------------------------+
               |                      +----------------+ |              +-----------------------+
 +-------+     | +-------------+  +=> | Subpartition 1 | | <=======+=== | Input Gate | Reduce 1 |
 | Map 1 | ==> | | Partition 1 | =|   +----------------+ |         |    +-----------------------+
 +-------+     | +-------------+  +=> | Subpartition 2 | | <==+    |
               |                      +----------------+ |    |    | Subpartition request
               |                                         |    |    |
               |                      +----------------+ |    |    |
 +-------+     | +-------------+  +=> | Subpartition 1 | | <==+====+
 | Map 2 | ==> | | Partition 2 | =|   +----------------+ |    |         +-----------------------+
 +-------+     | +-------------+  +=> | Subpartition 2 | | <==+======== | Input Gate | Reduce 2 |
               |                      +----------------+ |              +-----------------------+
               +-----------------------------------------+
 

In the above example, two map subtasks produce the intermediate result in parallel, resulting in two partitions (Partition 1 and 2). Each of these partitions is further partitioned into two subpartitions -- one for each parallel reduce subtask.

嵌套类概要

从类继承的嵌套类/接口 org.apache.flink.runtime.io.network.partition.consumer.InputGate

InputGate.InputWithData<INPUT,DATA>

从接口继承的嵌套类/接口 org.apache.flink.runtime.io.PullingAsyncDataInput

PullingAsyncDataInput.EndOfDataStatus

从接口继承的嵌套类/接口 org.apache.flink.runtime.io.AvailabilityProvider

AvailabilityProvider.AvailabilityHelper

字段概要

从类继承的字段 org.apache.flink.runtime.io.network.partition.consumer.InputGate

availabilityHelper, priorityAvailabilityHelper

从接口继承的字段 org.apache.flink.runtime.io.AvailabilityProvider

AVAILABLE

构造器概要

构造器

构造器和说明

SingleInputGate(String owningTaskName, int gateIndex, IntermediateDataSetID consumedResultId, ResultPartitionType consumedPartitionType, int numberOfInputChannels, PartitionProducerStateProvider partitionProducerStateProvider, org.apache.flink.util.function.SupplierWithException<BufferPool,IOException> bufferPoolFactory, BufferDecompressor bufferDecompressor, org.apache.flink.core.memory.MemorySegmentProvider memorySegmentProvider, int segmentSize, ThroughputCalculator throughputCalculator, BufferDebloater bufferDebloater)

方法概要

限定符和类型	方法和说明
void	acknowledgeAllRecordsProcessed(InputChannelInfo channelInfo)
void	announceBufferSize(int newBufferSize)
void	close()
void	convertRecoveredInputChannels()
void	finishReadRecoveredState()
BufferPool	getBufferPool()
InputChannel	getChannel(int channelIndex) Returns the channel of this gate.
CompletableFuture<Void>	getCloseFuture()
ResultPartitionType	getConsumedPartitionType() Returns the type of this input channel's consumed result partition.
int	getGateIndex() Returns the index of this input gate.
Map<org.apache.flink.api.java.tuple.Tuple2<IntermediateResultPartitionID,InputChannelInfo>,InputChannel>	getInputChannels()
protected PrioritizedDeque<InputChannel>	getInputChannelsWithData()
Duration	getLastEstimatedTimeToConsume()
Optional<BufferOrEvent>	getNext() Blocking call waiting for next BufferOrEvent.
int	getNumberOfInputChannels()
int	getNumberOfQueuedBuffers()
String	getOwningTaskName()
long	getSizeOfQueuedBuffers()
CompletableFuture<Void>	getStateConsumedFuture()
List<InputChannelInfo>	getUnfinishedChannels() Returns the list of channels that have not received EndOfPartitionEvent.
PullingAsyncDataInput.EndOfDataStatus	hasReceivedEndOfData() Tells if we consumed all available data.
Iterable<InputChannel>	inputChannels()
boolean	isFinished()
Optional<BufferOrEvent>	pollNext() Poll the BufferOrEvent.
void	requestPartitions()
void	resumeConsumption(InputChannelInfo channelInfo)
void	retriggerPartitionRequest(IntermediateResultPartitionID partitionId, InputChannelInfo inputChannelInfo) Retriggers a partition request.
void	sendTaskEvent(TaskEvent event)
void	setBufferPool(BufferPool bufferPool)
void	setInputChannels(InputChannel... channels)
void	setTieredStorageService(List<TieredStorageConsumerSpec> tieredStorageConsumerSpecs, TieredStorageConsumerClient client, TieredStorageNettyServiceImpl nettyService)
void	setup() Setup gate, potentially heavy-weight, blocking operation comparing to just creation.
void	setupChannels() Assign the exclusive buffers to all remote input channels directly for credit-based mode.
String	toString()
void	triggerDebloating()
void	updateInputChannel(ResourceID localLocation, NettyShuffleDescriptor shuffleDescriptor)

从类继承的方法 org.apache.flink.runtime.io.network.partition.consumer.IndexedInputGate

blockConsumption, checkpointStarted, checkpointStopped, convertToPriorityEvent, getInputGateIndex

从类继承的方法 org.apache.flink.runtime.io.network.partition.consumer.InputGate

getAvailableFuture, getChannelInfos, getPriorityEventAvailableFuture, setChannelStateWriter

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

从接口继承的方法 org.apache.flink.runtime.io.network.partition.consumer.CheckpointableInput

getChannelInfos

从接口继承的方法 org.apache.flink.runtime.io.AvailabilityProvider

and, isApproximatelyAvailable, isAvailable, or

构造器详细资料

SingleInputGate

public SingleInputGate(String owningTaskName,
                       int gateIndex,
                       IntermediateDataSetID consumedResultId,
                       ResultPartitionType consumedPartitionType,
                       int numberOfInputChannels,
                       PartitionProducerStateProvider partitionProducerStateProvider,
                       org.apache.flink.util.function.SupplierWithException<BufferPool,IOException> bufferPoolFactory,
                       @Nullable
                       BufferDecompressor bufferDecompressor,
                       org.apache.flink.core.memory.MemorySegmentProvider memorySegmentProvider,
                       int segmentSize,
                       ThroughputCalculator throughputCalculator,
                       @Nullable
                       BufferDebloater bufferDebloater)

方法详细资料

getInputChannelsWithData

protected PrioritizedDeque<InputChannel> getInputChannelsWithData()

setup

public void setup()
           throws IOException

从类复制的说明: InputGate

Setup gate, potentially heavy-weight, blocking operation comparing to just creation.

指定者:

setup 在类中 InputGate

抛出:

IOException

getStateConsumedFuture

public CompletableFuture<Void> getStateConsumedFuture()

指定者:

getStateConsumedFuture 在类中 InputGate

requestPartitions

public void requestPartitions()

指定者:

requestPartitions 在类中 InputGate

convertRecoveredInputChannels

@VisibleForTesting
public void convertRecoveredInputChannels()

finishReadRecoveredState

public void finishReadRecoveredState()
                              throws IOException

指定者:

finishReadRecoveredState 在类中 InputGate

抛出:

IOException

getNumberOfInputChannels

public int getNumberOfInputChannels()

指定者:

getNumberOfInputChannels 在接口中 CheckpointableInput

指定者:

getNumberOfInputChannels 在类中 InputGate

getGateIndex

public int getGateIndex()

从类复制的说明: IndexedInputGate

Returns the index of this input gate. Only supported on

指定者:

getGateIndex 在类中 IndexedInputGate

getUnfinishedChannels

public List<InputChannelInfo> getUnfinishedChannels()

从类复制的说明: IndexedInputGate

Returns the list of channels that have not received EndOfPartitionEvent.

指定者:

getUnfinishedChannels 在类中 IndexedInputGate

announceBufferSize

@VisibleForTesting
public void announceBufferSize(int newBufferSize)

triggerDebloating

public void triggerDebloating()

指定者:

triggerDebloating 在类中 IndexedInputGate

getLastEstimatedTimeToConsume

public Duration getLastEstimatedTimeToConsume()

getConsumedPartitionType

public ResultPartitionType getConsumedPartitionType()

Returns the type of this input channel's consumed result partition.

返回:

consumed result partition type

getBufferPool

public BufferPool getBufferPool()

getOwningTaskName

public String getOwningTaskName()

getNumberOfQueuedBuffers

public int getNumberOfQueuedBuffers()

getSizeOfQueuedBuffers

public long getSizeOfQueuedBuffers()

getCloseFuture

public CompletableFuture<Void> getCloseFuture()

getChannel

public InputChannel getChannel(int channelIndex)

从类复制的说明: InputGate

Returns the channel of this gate.

指定者:

getChannel 在类中 InputGate

setBufferPool

public void setBufferPool(BufferPool bufferPool)

setupChannels

@VisibleForTesting
public void setupChannels()
                                      throws IOException

Assign the exclusive buffers to all remote input channels directly for credit-based mode.

抛出:

IOException

setInputChannels

public void setInputChannels(InputChannel... channels)

setTieredStorageService

public void setTieredStorageService(List<TieredStorageConsumerSpec> tieredStorageConsumerSpecs,
                                    TieredStorageConsumerClient client,
                                    TieredStorageNettyServiceImpl nettyService)

updateInputChannel

public void updateInputChannel(ResourceID localLocation,
                               NettyShuffleDescriptor shuffleDescriptor)
                        throws IOException,
                               InterruptedException

抛出:

IOException

InterruptedException

retriggerPartitionRequest

public void retriggerPartitionRequest(IntermediateResultPartitionID partitionId,
                                      InputChannelInfo inputChannelInfo)
                               throws IOException

Retriggers a partition request.

抛出:

IOException

close

public void close()
           throws IOException

抛出:

IOException

isFinished

public boolean isFinished()

指定者:

isFinished 在接口中 PullingAsyncDataInput<BufferOrEvent>

指定者:

isFinished 在类中 InputGate

返回:

true if is finished and for example end of input was reached, false otherwise.

hasReceivedEndOfData

public PullingAsyncDataInput.EndOfDataStatus hasReceivedEndOfData()

从接口复制的说明: PullingAsyncDataInput

Tells if we consumed all available data.

Moreover it tells us the reason why there is no more data incoming. If any of the upstream subtasks finished because of the stop-with-savepoint --no-drain, we should not drain the input. See also StopMode.

toString

public String toString()

覆盖:

toString 在类中 Object

getNext

public Optional<BufferOrEvent> getNext()
                                throws IOException,
                                       InterruptedException

从类复制的说明: InputGate

Blocking call waiting for next BufferOrEvent.

Note: It should be guaranteed that the previous returned buffer has been recycled before getting next one.

指定者:

getNext 在类中 InputGate

返回:

Optional.empty() if InputGate.isFinished() returns true.

抛出:

IOException

InterruptedException

pollNext

public Optional<BufferOrEvent> pollNext()
                                 throws IOException,
                                        InterruptedException

从类复制的说明: InputGate

Poll the BufferOrEvent.

Note: It should be guaranteed that the previous returned buffer has been recycled before polling next one.

指定者:

pollNext 在接口中 PullingAsyncDataInput<BufferOrEvent>

指定者:

pollNext 在类中 InputGate

返回:

Optional.empty() if there is no data to return or if InputGate.isFinished() returns true.

抛出:

IOException

InterruptedException

sendTaskEvent

public void sendTaskEvent(TaskEvent event)
                   throws IOException

指定者:

sendTaskEvent 在类中 InputGate

抛出:

IOException

resumeConsumption

public void resumeConsumption(InputChannelInfo channelInfo)
                       throws IOException

指定者:

resumeConsumption 在接口中 CheckpointableInput

指定者:

resumeConsumption 在类中 InputGate

抛出:

IOException

acknowledgeAllRecordsProcessed

public void acknowledgeAllRecordsProcessed(InputChannelInfo channelInfo)
                                    throws IOException

指定者:

acknowledgeAllRecordsProcessed 在类中 InputGate

抛出:

IOException

getInputChannels

@VisibleForTesting
public Map<org.apache.flink.api.java.tuple.Tuple2<IntermediateResultPartitionID,InputChannelInfo>,InputChannel> getInputChannels()

inputChannels

public Iterable<InputChannel> inputChannels()