org.apache.flink.runtime.io.network

Class NettyShuffleEnvironment

java.lang.Object

org.apache.flink.runtime.io.network.NettyShuffleEnvironment

All Implemented Interfaces:

AutoCloseable, ShuffleEnvironment<ResultPartition,SingleInputGate>

public class NettyShuffleEnvironment
extends Object
implements ShuffleEnvironment<ResultPartition,SingleInputGate>

The implementation of ShuffleEnvironment based on netty network communication, local memory and disk files. The network environment contains the data structures that keep track of all intermediate results and shuffle data exchanges.

Method Summary

Modifier and Type	Method and Description
void	close() Tries to shut down all network I/O components.
List<SingleInputGate>	createInputGates(ShuffleIOOwnerContext ownerContext, PartitionProducerStateProvider partitionProducerStateProvider, List<InputGateDeploymentDescriptor> inputGateDeploymentDescriptors) Factory method for the InputGates to consume result partitions.
List<ResultPartition>	createResultPartitionWriters(ShuffleIOOwnerContext ownerContext, List<ResultPartitionDeploymentDescriptor> resultPartitionDeploymentDescriptors) Factory method for the ResultPartitionWriters to produce result partitions.
ShuffleIOOwnerContext	createShuffleIOOwnerContext(String ownerName, ExecutionAttemptID executionAttemptID, org.apache.flink.metrics.MetricGroup parentGroup) Create a context of the shuffle input/output owner used to create partitions or gates belonging to the owner.
BatchShuffleReadBufferPool	getBatchShuffleReadBufferPool()
ScheduledExecutorService	getBatchShuffleReadIOExecutor()
NettyShuffleEnvironmentConfiguration	getConfiguration()
ConnectionManager	getConnectionManager()
Optional<Collection<SingleInputGate>>	getInputGate(InputGateID id)
Optional<ShuffleMetrics>	getMetricsIfPartitionOccupyingLocalResource(ResultPartitionID partitionId) Get metrics of the partition if it still occupies some resources locally and have not been released yet.
NetworkBufferPool	getNetworkBufferPool()
Collection<ResultPartitionID>	getPartitionsOccupyingLocalResources() Report unreleased partitions.
ResultPartitionManager	getResultPartitionManager()
boolean	isClosed()
void	registerLegacyNetworkMetrics(org.apache.flink.metrics.MetricGroup metricGroup, ResultPartitionWriter[] producedPartitions, InputGate[] inputGates) Deprecated. should be removed in future
void	releasePartitionsLocally(Collection<ResultPartitionID> partitionIds) Release local resources occupied by the given partitions.
int	start() Start the internal related services before using the shuffle service environment.
boolean	updatePartitionInfo(ExecutionAttemptID consumerID, PartitionInfo partitionInfo) Update a gate with the newly available partition information, previously unknown.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

getResultPartitionManager

@VisibleForTesting
public ResultPartitionManager getResultPartitionManager()

getConnectionManager

@VisibleForTesting
public ConnectionManager getConnectionManager()

getNetworkBufferPool

@VisibleForTesting
public NetworkBufferPool getNetworkBufferPool()

getBatchShuffleReadBufferPool

@VisibleForTesting
public BatchShuffleReadBufferPool getBatchShuffleReadBufferPool()

getBatchShuffleReadIOExecutor

@VisibleForTesting
public ScheduledExecutorService getBatchShuffleReadIOExecutor()

getConfiguration

@VisibleForTesting
public NettyShuffleEnvironmentConfiguration getConfiguration()

getInputGate

@VisibleForTesting
public Optional<Collection<SingleInputGate>> getInputGate(InputGateID id)

releasePartitionsLocally

public void releasePartitionsLocally(Collection<ResultPartitionID> partitionIds)

Description copied from interface: ShuffleEnvironment

Release local resources occupied by the given partitions.

This is called for partitions which occupy resources locally (can be checked by ShuffleDescriptor.storesLocalResourcesOn()).

Specified by:

releasePartitionsLocally in interface ShuffleEnvironment<ResultPartition,SingleInputGate>

Parameters:

partitionIds - identifying the partitions to be released

getPartitionsOccupyingLocalResources

public Collection<ResultPartitionID> getPartitionsOccupyingLocalResources()

Report unreleased partitions.

Specified by:

getPartitionsOccupyingLocalResources in interface ShuffleEnvironment<ResultPartition,SingleInputGate>

Returns:

collection of partitions which still occupy some resources locally on this task executor and have been not released yet.

getMetricsIfPartitionOccupyingLocalResource

public Optional<ShuffleMetrics> getMetricsIfPartitionOccupyingLocalResource(ResultPartitionID partitionId)

Description copied from interface: ShuffleEnvironment

Get metrics of the partition if it still occupies some resources locally and have not been released yet.

Specified by:

getMetricsIfPartitionOccupyingLocalResource in interface ShuffleEnvironment<ResultPartition,SingleInputGate>

Parameters:

partitionId - the partition id

Returns:

An Optional of ShuffleMetrics, if found, of the given partition

createShuffleIOOwnerContext

public ShuffleIOOwnerContext createShuffleIOOwnerContext(String ownerName,
                                                         ExecutionAttemptID executionAttemptID,
                                                         org.apache.flink.metrics.MetricGroup parentGroup)

Description copied from interface: ShuffleEnvironment

Create a context of the shuffle input/output owner used to create partitions or gates belonging to the owner.

This method has to be called only once to avoid duplicated internal metric group registration.

Specified by:

createShuffleIOOwnerContext in interface ShuffleEnvironment<ResultPartition,SingleInputGate>

Parameters:

ownerName - the owner name, used for logs

executionAttemptID - execution attempt id of the producer or consumer

parentGroup - parent of shuffle specific metric group

Returns:

context of the shuffle input/output owner used to create partitions or gates belonging to the owner

createResultPartitionWriters

public List<ResultPartition> createResultPartitionWriters(ShuffleIOOwnerContext ownerContext,
                                                          List<ResultPartitionDeploymentDescriptor> resultPartitionDeploymentDescriptors)

Description copied from interface: ShuffleEnvironment

Factory method for the ResultPartitionWriters to produce result partitions.

The order of the ResultPartitionWriters in the returned collection should be the same as the iteration order of the passed resultPartitionDeploymentDescriptors.

Specified by:

createResultPartitionWriters in interface ShuffleEnvironment<ResultPartition,SingleInputGate>

Parameters:

ownerContext - the owner context relevant for partition creation

resultPartitionDeploymentDescriptors - descriptors of the partition, produced by the owner

Returns:

list of the ResultPartitionWriters

createInputGates

public List<SingleInputGate> createInputGates(ShuffleIOOwnerContext ownerContext,
                                              PartitionProducerStateProvider partitionProducerStateProvider,
                                              List<InputGateDeploymentDescriptor> inputGateDeploymentDescriptors)

Description copied from interface: ShuffleEnvironment

Factory method for the InputGates to consume result partitions.

The order of the InputGates in the returned collection should be the same as the iteration order of the passed inputGateDeploymentDescriptors.

Specified by:

createInputGates in interface ShuffleEnvironment<ResultPartition,SingleInputGate>

Parameters:

ownerContext - the owner context relevant for gate creation

partitionProducerStateProvider - producer state provider to query whether the producer is ready for consumption

inputGateDeploymentDescriptors - descriptors of the input gates to consume

Returns:

list of the InputGates

registerLegacyNetworkMetrics

@Deprecated
public void registerLegacyNetworkMetrics(org.apache.flink.metrics.MetricGroup metricGroup,
                                                     ResultPartitionWriter[] producedPartitions,
                                                     InputGate[] inputGates)

Deprecated. should be removed in future

Registers legacy network metric groups before shuffle service refactoring.

Registers legacy metric groups if shuffle service implementation is original default one.

updatePartitionInfo

public boolean updatePartitionInfo(ExecutionAttemptID consumerID,
                                   PartitionInfo partitionInfo)
                            throws IOException,
                                   InterruptedException

Description copied from interface: ShuffleEnvironment

Update a gate with the newly available partition information, previously unknown.

Specified by:

updatePartitionInfo in interface ShuffleEnvironment<ResultPartition,SingleInputGate>

Parameters:

consumerID - execution id to distinguish gates with the same id from the different consumer executions

partitionInfo - information needed to consume the updated partition, e.g. network location

Returns:

true if the partition has been updated or false if the partition is not available anymore.

Throws:

IOException - IO problem by the update

InterruptedException - potentially blocking operation was interrupted

start

public int start()
          throws IOException

Description copied from interface: ShuffleEnvironment

Start the internal related services before using the shuffle service environment.

Specified by:

start in interface ShuffleEnvironment<ResultPartition,SingleInputGate>

Returns:

a port to connect for the shuffle data exchange, -1 if only local connection is possible.

Throws:

IOException

close

public void close()

Tries to shut down all network I/O components.

Specified by:

close in interface AutoCloseable

isClosed

public boolean isClosed()