org.apache.tez.dag.library.vertexmanager
Class ShuffleVertexManager

java.lang.Object
  org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager

All Implemented Interfaces:

org.apache.tez.dag.api.VertexManagerPlugin

public class ShuffleVertexManager
extends Object
implements org.apache.tez.dag.api.VertexManagerPlugin

Starts scheduling tasks when number of completed source tasks crosses slowStartMinSrcCompletionFraction and schedules all tasks when slowStartMaxSrcCompletionFraction is reached

Nested Class Summary

static class

ShuffleVertexManager.CustomShuffleEdgeManager

Field Summary

static String	TEZ_AM_SHUFFLE_VERTEX_MANAGER_DESIRED_TASK_INPUT_SIZE
static long	TEZ_AM_SHUFFLE_VERTEX_MANAGER_DESIRED_TASK_INPUT_SIZE_DEFAULT
static String	TEZ_AM_SHUFFLE_VERTEX_MANAGER_ENABLE_AUTO_PARALLEL
static boolean	TEZ_AM_SHUFFLE_VERTEX_MANAGER_ENABLE_AUTO_PARALLEL_DEFAULT
static String	TEZ_AM_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION In case of a ScatterGather connection, once this fraction of source tasks have completed, all tasks on the current vertex can be scheduled.
static float	TEZ_AM_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION_DEFAULT
static String	TEZ_AM_SHUFFLE_VERTEX_MANAGER_MIN_SRC_FRACTION In case of a ScatterGather connection, the fraction of source tasks which should complete before tasks for the current vertex are scheduled
static float	TEZ_AM_SHUFFLE_VERTEX_MANAGER_MIN_SRC_FRACTION_DEFAULT
static String	TEZ_AM_SHUFFLE_VERTEX_MANAGER_MIN_TASK_PARALLELISM
static int	TEZ_AM_SHUFFLE_VERTEX_MANAGER_MIN_TASK_PARALLELISM_DEFAULT

Constructor Summary

ShuffleVertexManager()

Method Summary

void	initialize(org.apache.tez.dag.api.VertexManagerPluginContext context)
void	onRootVertexInitialized(String inputName, org.apache.tez.dag.api.InputDescriptor inputDescriptor, List<org.apache.tez.runtime.api.Event> events)
void	onSourceTaskCompleted(String srcVertexName, Integer srcTaskId)
void	onVertexManagerEventReceived(org.apache.tez.runtime.api.events.VertexManagerEvent vmEvent)
void	onVertexStarted(Map<String,List<Integer>> completions)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TEZ_AM_SHUFFLE_VERTEX_MANAGER_MIN_SRC_FRACTION

public static final String TEZ_AM_SHUFFLE_VERTEX_MANAGER_MIN_SRC_FRACTION

In case of a ScatterGather connection, the fraction of source tasks which should complete before tasks for the current vertex are scheduled

See Also:

Constant Field Values

TEZ_AM_SHUFFLE_VERTEX_MANAGER_MIN_SRC_FRACTION_DEFAULT

public static final float TEZ_AM_SHUFFLE_VERTEX_MANAGER_MIN_SRC_FRACTION_DEFAULT

See Also:

Constant Field Values

TEZ_AM_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION

public static final String TEZ_AM_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION

In case of a ScatterGather connection, once this fraction of source tasks have completed, all tasks on the current vertex can be scheduled. Number of tasks ready for scheduling on the current vertex scales linearly between min-fraction and max-fraction

See Also:

Constant Field Values

TEZ_AM_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION_DEFAULT

public static final float TEZ_AM_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION_DEFAULT

See Also:

Constant Field Values

TEZ_AM_SHUFFLE_VERTEX_MANAGER_ENABLE_AUTO_PARALLEL

public static final String TEZ_AM_SHUFFLE_VERTEX_MANAGER_ENABLE_AUTO_PARALLEL

See Also:

Constant Field Values

TEZ_AM_SHUFFLE_VERTEX_MANAGER_ENABLE_AUTO_PARALLEL_DEFAULT

public static final boolean TEZ_AM_SHUFFLE_VERTEX_MANAGER_ENABLE_AUTO_PARALLEL_DEFAULT

See Also:

Constant Field Values

TEZ_AM_SHUFFLE_VERTEX_MANAGER_DESIRED_TASK_INPUT_SIZE

public static final String TEZ_AM_SHUFFLE_VERTEX_MANAGER_DESIRED_TASK_INPUT_SIZE

See Also:

Constant Field Values

TEZ_AM_SHUFFLE_VERTEX_MANAGER_DESIRED_TASK_INPUT_SIZE_DEFAULT

public static final long TEZ_AM_SHUFFLE_VERTEX_MANAGER_DESIRED_TASK_INPUT_SIZE_DEFAULT

See Also:

Constant Field Values

TEZ_AM_SHUFFLE_VERTEX_MANAGER_MIN_TASK_PARALLELISM

public static final String TEZ_AM_SHUFFLE_VERTEX_MANAGER_MIN_TASK_PARALLELISM

See Also:

Constant Field Values

TEZ_AM_SHUFFLE_VERTEX_MANAGER_MIN_TASK_PARALLELISM_DEFAULT

public static final int TEZ_AM_SHUFFLE_VERTEX_MANAGER_MIN_TASK_PARALLELISM_DEFAULT

See Also:

Constant Field Values

Constructor Detail

ShuffleVertexManager

public ShuffleVertexManager()

Method Detail

onVertexStarted

public void onVertexStarted(Map<String,List<Integer>> completions)

Specified by:

onVertexStarted in interface org.apache.tez.dag.api.VertexManagerPlugin

onSourceTaskCompleted

public void onSourceTaskCompleted(String srcVertexName,
                                  Integer srcTaskId)

Specified by:

onSourceTaskCompleted in interface org.apache.tez.dag.api.VertexManagerPlugin

onVertexManagerEventReceived

public void onVertexManagerEventReceived(org.apache.tez.runtime.api.events.VertexManagerEvent vmEvent)

Specified by:

onVertexManagerEventReceived in interface org.apache.tez.dag.api.VertexManagerPlugin

initialize

public void initialize(org.apache.tez.dag.api.VertexManagerPluginContext context)

Specified by:

initialize in interface org.apache.tez.dag.api.VertexManagerPlugin

onRootVertexInitialized

public void onRootVertexInitialized(String inputName,
                                    org.apache.tez.dag.api.InputDescriptor inputDescriptor,
                                    List<org.apache.tez.runtime.api.Event> events)

Specified by:

onRootVertexInitialized in interface org.apache.tez.dag.api.VertexManagerPlugin