org.apache.gobblin.util

Class JobLauncherUtils

java.lang.Object

org.apache.gobblin.util.JobLauncherUtils

public class JobLauncherUtils
extends Object

Utility class for the job scheduler and job launchers.

Constructor Summary

Constructors

Constructor and Description
JobLauncherUtils()

Method Summary

Modifier and Type	Method and Description
static void	cleanJobStagingData(State state, org.slf4j.Logger logger) Cleanup staging data of all tasks of a job.
static void	cleanStagingData(List<? extends State> states, org.slf4j.Logger logger) Cleanup the staging data for a list of Gobblin tasks.
static void	cleanTaskStagingData(State state, org.slf4j.Logger logger) Cleanup staging data of a Gobblin task.
static void	cleanTaskStagingData(State state, org.slf4j.Logger logger, com.google.common.io.Closer closer, Map<String,ParallelRunner> parallelRunners) Cleanup staging data of a Gobblin task using a ParallelRunner.
static void	cleanUpOldJobData(State state, org.slf4j.Logger logger, boolean stagingDirProvided, boolean outputDirProvided)
static List<WorkUnit>	flattenWorkUnits(Collection<WorkUnit> workUnits) Utility method that takes in a List of WorkUnits, and flattens them.
static org.apache.hadoop.fs.FileSystem	getFsWithProxy(State state, String fsUri, org.apache.hadoop.conf.Configuration conf)
static String	newJobId(String jobName) Create a new job ID.
static String	newMultiTaskId(String jobId, int sequence) Create an ID for a new multi-task (corresponding to a MultiWorkUnit) for the job with the given job ID.
static String	newTaskId(String jobId, int sequence) Create a new task ID for the job with the given job ID.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

JobLauncherUtils

public JobLauncherUtils()

Method Detail

newJobId

public static String newJobId(String jobName)

Create a new job ID.

Parameters:

jobName - job name

Returns:

new job ID

newTaskId

public static String newTaskId(String jobId,
                               int sequence)

Create a new task ID for the job with the given job ID.

Parameters:

jobId - job ID

sequence - task sequence number

Returns:

new task ID

newMultiTaskId

public static String newMultiTaskId(String jobId,
                                    int sequence)

Create an ID for a new multi-task (corresponding to a MultiWorkUnit) for the job with the given job ID.

Parameters:

jobId - job ID

sequence - multi-task sequence number

Returns:

new multi-task ID

flattenWorkUnits

public static List<WorkUnit> flattenWorkUnits(Collection<WorkUnit> workUnits)

Utility method that takes in a List of WorkUnits, and flattens them. It builds up the flattened list by checking each element of the given list, and seeing if it is an instance of MultiWorkUnit. If it is then it calls itself on the WorkUnits returned by MultiWorkUnit.getWorkUnits(). If not, then it simply adds the WorkUnit to the flattened list.

Parameters:

workUnits - is a List containing either WorkUnits or MultiWorkUnits

Returns:

a List of flattened WorkUnits

cleanStagingData

public static void cleanStagingData(List<? extends State> states,
                                    org.slf4j.Logger logger)
                             throws IOException

Cleanup the staging data for a list of Gobblin tasks. This method calls the cleanTaskStagingData(State, Logger) method.

Parameters:

states - a List of States that need their staging data cleaned

Throws:

IOException

cleanJobStagingData

public static void cleanJobStagingData(State state,
                                       org.slf4j.Logger logger)
                                throws IOException

Cleanup staging data of all tasks of a job.

Parameters:

state - a State instance storing job configuration properties

logger - a Logger used for logging

Throws:

IOException

cleanTaskStagingData

public static void cleanTaskStagingData(State state,
                                        org.slf4j.Logger logger)
                                 throws IOException

Cleanup staging data of a Gobblin task.

Parameters:

state - a State instance storing task configuration properties

logger - a Logger used for logging

Throws:

IOException

cleanTaskStagingData

public static void cleanTaskStagingData(State state,
                                        org.slf4j.Logger logger,
                                        com.google.common.io.Closer closer,
                                        Map<String,ParallelRunner> parallelRunners)
                                 throws IOException

Cleanup staging data of a Gobblin task using a ParallelRunner.

Parameters:

state - workunit state.

logger - a Logger used for logging.

closer - a closer that registers the given map of ParallelRunners. The caller is responsible for closing the closer after the cleaning is done.

parallelRunners - a map from FileSystem URI to ParallelRunner.

Throws:

IOException - if it fails to cleanup the task staging data.

cleanUpOldJobData

public static void cleanUpOldJobData(State state,
                                     org.slf4j.Logger logger,
                                     boolean stagingDirProvided,
                                     boolean outputDirProvided)
                              throws IOException

Throws:

IOException

getFsWithProxy

public static org.apache.hadoop.fs.FileSystem getFsWithProxy(State state,
                                                             String fsUri,
                                                             org.apache.hadoop.conf.Configuration conf)
                                                      throws IOException

Parameters:

state -

fsUri -

Returns:

Throws:

IOException