org.apache.tez.mapreduce.hadoop
Class MultiStageMRConfToTezTranslator

java.lang.Object
  org.apache.tez.mapreduce.hadoop.MultiStageMRConfToTezTranslator

public class MultiStageMRConfToTezTranslator
extends Object

Constructor Summary

MultiStageMRConfToTezTranslator()

Method Summary

static org.apache.hadoop.conf.Configuration	convertMRToLinearTez(org.apache.hadoop.conf.Configuration srcConf) Converts a single completely configured MR* job to something that can be understood by the Tez MR runtime.
static org.apache.hadoop.conf.Configuration	getInputConfOnEdge(org.apache.hadoop.conf.Configuration srcVertex, org.apache.hadoop.conf.Configuration destVertex) Given a source and destination vertex, returns the config which should be used for the Input on this edge.
static org.apache.hadoop.conf.Configuration	getOutputConfOnEdge(org.apache.hadoop.conf.Configuration srcVertex, org.apache.hadoop.conf.Configuration destVertex) Given a source and destination vertex, returns the config which should be used for the Output on this edge.
static org.apache.hadoop.conf.Configuration[]	getStageConfs(org.apache.hadoop.conf.Configuration conf) Given a single base MRR config, returns a list of complete stage configurations.
static void	translateVertexConfToTez(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.conf.Configuration predecessorConf) Translates MR keys to Tez for the provided vertex conf.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MultiStageMRConfToTezTranslator

public MultiStageMRConfToTezTranslator()

Method Detail

convertMRToLinearTez

public static org.apache.hadoop.conf.Configuration convertMRToLinearTez(org.apache.hadoop.conf.Configuration srcConf)

Converts a single completely configured MR* job to something that can be understood by the Tez MR runtime.

Parameters:

srcConf - the configuration for the entire MR* job. Configs for the first and last stage are expected to be set at root level. Configs for intermediate stages will be prefixed with the stage number.

Returns:

A translated MR* config with keys translated over to Tez.

translateVertexConfToTez

@InterfaceAudience.LimitedPrivate(value="Hive, Pig")
@InterfaceStability.Unstable
public static void translateVertexConfToTez(org.apache.hadoop.conf.Configuration conf,
                                                                                                         org.apache.hadoop.conf.Configuration predecessorConf)

Translates MR keys to Tez for the provided vertex conf. The conversion is done in place. This method should be called for each stage config of the MR* job. The call for the first vertex should set the predecessorConf as null. Since there's no separation of input / output params at the moment, the config generated by this can be set as the configuration for Tez Input/Output and Processor.

Parameters:

conf - Configuration for the vertex being configured.

predecessorConf - Configuration for the previous vertex in the MR* chain

getOutputConfOnEdge

@InterfaceAudience.LimitedPrivate(value="Hive, Pig")
@InterfaceStability.Unstable
public static org.apache.hadoop.conf.Configuration getOutputConfOnEdge(org.apache.hadoop.conf.Configuration srcVertex,
                                                                                                                                    org.apache.hadoop.conf.Configuration destVertex)

Given a source and destination vertex, returns the config which should be used for the Output on this edge. The configs must be configured with tez keys - or run through translateVertexConfToTez.

Parameters:

srcVertex - The tez configuration for the source vertex.

destVertex - The tez configuration for the destination vertex.

Returns:

the output Configuration object for the edge

getInputConfOnEdge

@InterfaceAudience.LimitedPrivate(value="Hive, Pig")
@InterfaceStability.Unstable
public static org.apache.hadoop.conf.Configuration getInputConfOnEdge(org.apache.hadoop.conf.Configuration srcVertex,
                                                                                                                                   org.apache.hadoop.conf.Configuration destVertex)

Given a source and destination vertex, returns the config which should be used for the Input on this edge. The configs must be configured with tez keys - or run through translateVertexConfToTez.

Parameters:

srcVertex - The tez configuration for the source vertex.

destVertex - The tez configuration for the destination vertex.

Returns:

the input Configuration object for the edge

getStageConfs

@InterfaceAudience.Private
public static org.apache.hadoop.conf.Configuration[] getStageConfs(org.apache.hadoop.conf.Configuration conf)

Given a single base MRR config, returns a list of complete stage configurations.

Parameters:

conf -

Returns:

list of complete stage configurations given Conifiguration