org.apache.tez.mapreduce.hadoop
Class MRHelpers

java.lang.Object
  org.apache.tez.mapreduce.hadoop.MRHelpers

public class MRHelpers
extends Object

Constructor Summary

MRHelpers()

Method Summary

static void	addLog4jSystemProperties(String logLevel, List<String> vargs) Add the JVM system properties necessary to configure ContainerLogAppender.
static void	addMRInput(org.apache.tez.dag.api.Vertex vertex, byte[] userPayload, Class<? extends org.apache.tez.runtime.api.TezRootInputInitializer> initClazz) Convenience method to add an MR Input to the specified vertex.
static void	addMROutput(org.apache.tez.dag.api.Vertex vertex, byte[] userPayload) Convenience method to add an MR Output to the specified vertex.
static void	addMROutputLegacy(org.apache.tez.dag.api.Vertex vertex, byte[] userPayload)
static com.google.protobuf.ByteString	createByteStringFromConf(org.apache.hadoop.conf.Configuration conf)
static org.apache.hadoop.conf.Configuration	createConfFromByteString(com.google.protobuf.ByteString bs)
static org.apache.hadoop.conf.Configuration	createConfFromUserPayload(byte[] bb)
static byte[]	createMRInputPayload(byte[] configurationBytes, MRRuntimeProtos.MRSplitsProto mrSplitsProto)
static byte[]	createMRInputPayload(org.apache.hadoop.conf.Configuration conf, MRRuntimeProtos.MRSplitsProto mrSplitsProto)
static byte[]	createMRInputPayloadWithGrouping(byte[] configurationBytes, String inputFormatName) Called to specify that grouping of input splits be performed by Tez The configurationBytes conf should have the input format class configuration set to the TezGroupedSplitsInputFormat.
static byte[]	createMRInputPayloadWithGrouping(org.apache.hadoop.conf.Configuration conf, String inputFormatName) Called to specify that grouping of input splits be performed by Tez The conf should have the input format class configuration set to the TezGroupedSplitsInputFormat.
static org.apache.hadoop.mapreduce.InputSplit	createNewFormatSplitFromUserPayload(MRRuntimeProtos.MRSplitProto splitProto, org.apache.hadoop.io.serializer.SerializationFactory serializationFactory)
static org.apache.hadoop.mapred.InputSplit	createOldFormatSplitFromUserPayload(MRRuntimeProtos.MRSplitProto splitProto, org.apache.hadoop.io.serializer.SerializationFactory serializationFactory)
static MRRuntimeProtos.MRSplitProto	createSplitProto(org.apache.hadoop.mapred.InputSplit oldSplit)
static <T extends org.apache.hadoop.mapreduce.InputSplit> MRRuntimeProtos.MRSplitProto	createSplitProto(T newSplit, org.apache.hadoop.io.serializer.SerializationFactory serializationFactory)
static byte[]	createUserPayloadFromConf(org.apache.hadoop.conf.Configuration conf)
static void	doJobClientMagic(org.apache.hadoop.conf.Configuration conf) Sets up parameters which used to be set by the MR JobClient.
static InputSplitInfoDisk	generateInputSplits(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path inputSplitsDir) Helper api to generate splits
static InputSplitInfoMem	generateInputSplitsToMem(org.apache.hadoop.conf.Configuration conf) Generates Input splits and stores them in a MRProtos instance.
static org.apache.hadoop.mapreduce.InputSplit[]	generateNewSplits(org.apache.hadoop.mapreduce.JobContext jobContext, String inputFormatName, int numTasks)
static org.apache.hadoop.mapred.InputSplit[]	generateOldSplits(org.apache.hadoop.mapred.JobConf jobConf, String inputFormatName, int numTasks)
static org.apache.hadoop.conf.Configuration	getBaseMRConfiguration() Get default initialize JobConf-based configuration
static org.apache.hadoop.conf.Configuration	getBaseMRConfiguration(org.apache.hadoop.conf.Configuration conf) Get default initialize JobConf-based configuration
static String	getMapJavaOpts(org.apache.hadoop.conf.Configuration conf) Generate JVM options to be used to launch map tasks Uses mapreduce.admin.map.child.java.opts, mapreduce.map.java.opts and mapreduce.map.log.level from config to generate the opts.
static org.apache.hadoop.yarn.api.records.Resource	getMapResource(org.apache.hadoop.conf.Configuration conf) Extract the map task's container resource requirements from the provided configuration.
static String	getMRAMJavaOpts(org.apache.hadoop.conf.Configuration conf) Extract Java Opts for the AM based on MR-based configuration
static String	getReduceJavaOpts(org.apache.hadoop.conf.Configuration conf) Generate JVM options to be used to launch reduce tasks Uses mapreduce.admin.reduce.child.java.opts, mapreduce.reduce.java.opts and mapreduce.reduce.log.level from config to generate the opts.
static org.apache.hadoop.yarn.api.records.Resource	getReduceResource(org.apache.hadoop.conf.Configuration conf) Extract the reduce task's container resource requirements from the provided configuration.
static void	mergeMRBinaryTokens(org.apache.hadoop.security.Credentials creds, org.apache.hadoop.conf.Configuration conf) Merge tokens from a configured binary file into provided Credentials object.
static MRRuntimeProtos.MRInputUserPayloadProto	parseMRInputPayload(byte[] bytes)
static void	updateEnvironmentForMRAM(org.apache.hadoop.conf.Configuration conf, Map<String,String> environment) Setup environment for the AM based on MR-based configuration
static void	updateEnvironmentForMRTasks(org.apache.hadoop.conf.Configuration conf, Map<String,String> environment, boolean isMap) Setup classpath and other environment variables
static void	updateLocalResourcesForInputSplits(org.apache.hadoop.fs.FileSystem fs, InputSplitInfo inputSplitInfo, Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources) Update provided localResources collection with the required local resources needed by MapReduce tasks with respect to Input splits.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MRHelpers

public MRHelpers()

Method Detail

generateNewSplits

@InterfaceAudience.Private
public static org.apache.hadoop.mapreduce.InputSplit[] generateNewSplits(org.apache.hadoop.mapreduce.JobContext jobContext,
                                                                                                   String inputFormatName,
                                                                                                   int numTasks)
                                                                  throws ClassNotFoundException,
                                                                         IOException,
                                                                         InterruptedException

Throws:

ClassNotFoundException

IOException

InterruptedException

generateOldSplits

@InterfaceAudience.Private
public static org.apache.hadoop.mapred.InputSplit[] generateOldSplits(org.apache.hadoop.mapred.JobConf jobConf,
                                                                                                String inputFormatName,
                                                                                                int numTasks)
                                                               throws IOException

Throws:

IOException

generateInputSplits

public static InputSplitInfoDisk generateInputSplits(org.apache.hadoop.conf.Configuration conf,
                                                     org.apache.hadoop.fs.Path inputSplitsDir)
                                              throws IOException,
                                                     InterruptedException,
                                                     ClassNotFoundException

Helper api to generate splits

Parameters:

conf - Configuration with all necessary information set to generate splits. The following are required at a minimum: - mapred.mapper.new-api: determine whether mapred.InputFormat or mapreduce.InputFormat is to be used - mapred.input.format.class or mapreduce.job.inputformat.class: determines the InputFormat class to be used In addition to this, all the configs needed by the InputFormat class also have to be set. For example, FileInputFormat needs the input directory paths to be set in the config.

inputSplitsDir - Directory in which the splits file and meta info file will be generated. job.split and job.splitmetainfo files in this directory will be overwritten. Should be a fully-qualified path.

Returns:

InputSplitInfo containing the split files' information and the number of splits generated to be used to determining parallelism of the map stage.

Throws:

IOException

InterruptedException

ClassNotFoundException

generateInputSplitsToMem

public static InputSplitInfoMem generateInputSplitsToMem(org.apache.hadoop.conf.Configuration conf)
                                                  throws IOException,
                                                         ClassNotFoundException,
                                                         InterruptedException

Generates Input splits and stores them in a MRProtos instance. Returns an instance of InputSplitInfoMem

Parameters:

conf - an instance of Configuration which is used to determine whether the mapred of mapreduce API is being used. This Configuration instance should also contain adequate information to be able to generate splits - like the InputFormat being used and related configuration.

Returns:

an instance of InputSplitInfoMem which supports a subset of the APIs defined on InputSplitInfo

Throws:

IOException

ClassNotFoundException

InterruptedException

createSplitProto

@InterfaceAudience.Private
public static <T extends org.apache.hadoop.mapreduce.InputSplit> MRRuntimeProtos.MRSplitProto createSplitProto(T newSplit,
                                                                                                                                         org.apache.hadoop.io.serializer.SerializationFactory serializationFactory)
                                                     throws IOException,
                                                            InterruptedException

Throws:

IOException

InterruptedException

createSplitProto

@InterfaceAudience.Private
public static MRRuntimeProtos.MRSplitProto createSplitProto(org.apache.hadoop.mapred.InputSplit oldSplit)
                                                     throws IOException

Throws:

IOException

addLog4jSystemProperties

public static void addLog4jSystemProperties(String logLevel,
                                            List<String> vargs)

Add the JVM system properties necessary to configure ContainerLogAppender.

Parameters:

logLevel - the desired log level (eg INFO/WARN/DEBUG)

vargs - the argument list to append to

getMapJavaOpts

public static String getMapJavaOpts(org.apache.hadoop.conf.Configuration conf)

Generate JVM options to be used to launch map tasks Uses mapreduce.admin.map.child.java.opts, mapreduce.map.java.opts and mapreduce.map.log.level from config to generate the opts.

Parameters:

conf - Configuration to be used to extract JVM opts specific info

Returns:

JAVA_OPTS string to be used in launching the JVM

getReduceJavaOpts

public static String getReduceJavaOpts(org.apache.hadoop.conf.Configuration conf)

Generate JVM options to be used to launch reduce tasks Uses mapreduce.admin.reduce.child.java.opts, mapreduce.reduce.java.opts and mapreduce.reduce.log.level from config to generate the opts.

Parameters:

conf - Configuration to be used to extract JVM opts specific info

Returns:

JAVA_OPTS string to be used in launching the JVM

doJobClientMagic

public static void doJobClientMagic(org.apache.hadoop.conf.Configuration conf)
                             throws IOException

Sets up parameters which used to be set by the MR JobClient. Includes setting whether to use the new api or the old api. Note: Must be called before generating InputSplits

Parameters:

conf - configuration for the vertex.

Throws:

IOException

createUserPayloadFromConf

@InterfaceAudience.LimitedPrivate(value="Hive, Pig")
@InterfaceStability.Unstable
public static byte[] createUserPayloadFromConf(org.apache.hadoop.conf.Configuration conf)
                                        throws IOException

Throws:

IOException

createByteStringFromConf

@InterfaceAudience.LimitedPrivate(value="Hive, Pig")
public static com.google.protobuf.ByteString createByteStringFromConf(org.apache.hadoop.conf.Configuration conf)
                                                               throws IOException

Throws:

IOException

createConfFromUserPayload

@InterfaceAudience.LimitedPrivate(value="Hive, Pig")
@InterfaceStability.Unstable
public static org.apache.hadoop.conf.Configuration createConfFromUserPayload(byte[] bb)
                                                                      throws IOException

Throws:

IOException

createConfFromByteString

@InterfaceAudience.LimitedPrivate(value="Hive, Pig")
public static org.apache.hadoop.conf.Configuration createConfFromByteString(com.google.protobuf.ByteString bs)
                                                                     throws IOException

Throws:

IOException

createMRInputPayload

public static byte[] createMRInputPayload(byte[] configurationBytes,
                                          MRRuntimeProtos.MRSplitsProto mrSplitsProto)
                                   throws IOException

Throws:

IOException

createMRInputPayload

public static byte[] createMRInputPayload(org.apache.hadoop.conf.Configuration conf,
                                          MRRuntimeProtos.MRSplitsProto mrSplitsProto)
                                   throws IOException

Throws:

IOException

createMRInputPayloadWithGrouping

public static byte[] createMRInputPayloadWithGrouping(byte[] configurationBytes,
                                                      String inputFormatName)
                                               throws IOException

Called to specify that grouping of input splits be performed by Tez The configurationBytes conf should have the input format class configuration set to the TezGroupedSplitsInputFormat. The real input format class name should be passed as an argument to this method.

Throws:

IOException

createMRInputPayloadWithGrouping

public static byte[] createMRInputPayloadWithGrouping(org.apache.hadoop.conf.Configuration conf,
                                                      String inputFormatName)
                                               throws IOException

Called to specify that grouping of input splits be performed by Tez The conf should have the input format class configuration set to the TezGroupedSplitsInputFormat. The real input format class name should be passed as an argument to this method.

Throws:

IOException

parseMRInputPayload

public static MRRuntimeProtos.MRInputUserPayloadProto parseMRInputPayload(byte[] bytes)
                                                                   throws IOException

Throws:

IOException

updateLocalResourcesForInputSplits

public static void updateLocalResourcesForInputSplits(org.apache.hadoop.fs.FileSystem fs,
                                                      InputSplitInfo inputSplitInfo,
                                                      Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources)
                                               throws IOException

Update provided localResources collection with the required local resources needed by MapReduce tasks with respect to Input splits.

Parameters:

fs - Filesystem instance to access status of splits related files

inputSplitInfo - Information on location of split files

localResources - LocalResources collection to be updated

Throws:

IOException

getMapResource

public static org.apache.hadoop.yarn.api.records.Resource getMapResource(org.apache.hadoop.conf.Configuration conf)

Extract the map task's container resource requirements from the provided configuration. Uses mapreduce.map.memory.mb and mapreduce.map.cpu.vcores from the provided configuration.

Parameters:

conf - Configuration with MR specific settings used to extract information from

Returns:

Resource object used to define requirements for containers running Map tasks

getReduceResource

public static org.apache.hadoop.yarn.api.records.Resource getReduceResource(org.apache.hadoop.conf.Configuration conf)

Extract the reduce task's container resource requirements from the provided configuration. Uses mapreduce.reduce.memory.mb and mapreduce.reduce.cpu.vcores from the provided configuration.

Parameters:

conf - Configuration with MR specific settings used to extract information from

Returns:

Resource object used to define requirements for containers running Reduce tasks

updateEnvironmentForMRTasks

public static void updateEnvironmentForMRTasks(org.apache.hadoop.conf.Configuration conf,
                                               Map<String,String> environment,
                                               boolean isMap)

Setup classpath and other environment variables

Parameters:

conf - Configuration to retrieve settings from

environment - Environment to update

isMap - Whether task is a map or reduce task

getBaseMRConfiguration

public static org.apache.hadoop.conf.Configuration getBaseMRConfiguration(org.apache.hadoop.conf.Configuration conf)

Get default initialize JobConf-based configuration

Parameters:

conf - Conf to initialize JobConf with.

Returns:

Base configuration for MR-based jobs

getBaseMRConfiguration

public static org.apache.hadoop.conf.Configuration getBaseMRConfiguration()

Get default initialize JobConf-based configuration

Returns:

Base configuration for MR-based jobs

updateEnvironmentForMRAM

public static void updateEnvironmentForMRAM(org.apache.hadoop.conf.Configuration conf,
                                            Map<String,String> environment)

Setup environment for the AM based on MR-based configuration

Parameters:

conf - Configuration from which to extract information

environment - Environment map to update

getMRAMJavaOpts

public static String getMRAMJavaOpts(org.apache.hadoop.conf.Configuration conf)

Extract Java Opts for the AM based on MR-based configuration

Parameters:

conf - Configuration from which to extract information

Returns:

Java opts for the AM

addMRInput

public static void addMRInput(org.apache.tez.dag.api.Vertex vertex,
                              byte[] userPayload,
                              Class<? extends org.apache.tez.runtime.api.TezRootInputInitializer> initClazz)

Convenience method to add an MR Input to the specified vertex. The name of the Input is "MRInput"

This should only be called for one vertex in a DAG

Parameters:

vertex -

userPayload -

initClazz - class to init the input in the AM

addMROutput

public static void addMROutput(org.apache.tez.dag.api.Vertex vertex,
                               byte[] userPayload)

Convenience method to add an MR Output to the specified vertex. The name of the Output is "MROutput"

This should only be called for one vertex in a DAG

Parameters:

vertex -

userPayload -

addMROutputLegacy

@InterfaceAudience.Private
public static void addMROutputLegacy(org.apache.tez.dag.api.Vertex vertex,
                                                               byte[] userPayload)

createOldFormatSplitFromUserPayload

public static org.apache.hadoop.mapred.InputSplit createOldFormatSplitFromUserPayload(MRRuntimeProtos.MRSplitProto splitProto,
                                                                                      org.apache.hadoop.io.serializer.SerializationFactory serializationFactory)
                                                                               throws IOException

Throws:

IOException

createNewFormatSplitFromUserPayload

public static org.apache.hadoop.mapreduce.InputSplit createNewFormatSplitFromUserPayload(MRRuntimeProtos.MRSplitProto splitProto,
                                                                                         org.apache.hadoop.io.serializer.SerializationFactory serializationFactory)
                                                                                  throws IOException

Throws:

IOException

mergeMRBinaryTokens

public static void mergeMRBinaryTokens(org.apache.hadoop.security.Credentials creds,
                                       org.apache.hadoop.conf.Configuration conf)
                                throws IOException

Merge tokens from a configured binary file into provided Credentials object. Uses "mapreduce.job.credentials.binary" property to find location of token file.

Parameters:

creds - Credentials object to add new tokens to

conf - Configuration containing location of token file. TezClient reads credentials from the property - TezJobConfig.TEZ_CREDENTIALS_PATH. This method is not required if that property is set.

Throws:

IOException