public class JobLauncherUtils extends Object
| Constructor and Description |
|---|
JobLauncherUtils() |
| Modifier and Type | Method and Description |
|---|---|
static void |
cleanJobStagingData(State state,
org.slf4j.Logger logger)
Cleanup staging data of all tasks of a job.
|
static void |
cleanStagingData(List<? extends State> states,
org.slf4j.Logger logger)
Cleanup the staging data for a list of Gobblin tasks.
|
static void |
cleanTaskStagingData(State state,
org.slf4j.Logger logger)
Cleanup staging data of a Gobblin task.
|
static void |
cleanTaskStagingData(State state,
org.slf4j.Logger logger,
com.google.common.io.Closer closer,
Map<String,ParallelRunner> parallelRunners)
Cleanup staging data of a Gobblin task using a
ParallelRunner. |
static void |
cleanUpOldJobData(State state,
org.slf4j.Logger logger,
boolean stagingDirProvided,
boolean outputDirProvided) |
static List<WorkUnit> |
flattenWorkUnits(Collection<WorkUnit> workUnits)
|
static org.apache.hadoop.fs.FileSystem |
getFsWithProxy(State state,
String fsUri,
org.apache.hadoop.conf.Configuration conf) |
static String |
newJobId(String jobName)
Create a new job ID.
|
static String |
newMultiTaskId(String jobId,
int sequence)
Create an ID for a new multi-task (corresponding to a
MultiWorkUnit)
for the job with the given job ID. |
static String |
newTaskId(String jobId,
int sequence)
Create a new task ID for the job with the given job ID.
|
public static String newJobId(String jobName)
jobName - job namepublic static String newTaskId(String jobId, int sequence)
jobId - job IDsequence - task sequence numberpublic static String newMultiTaskId(String jobId, int sequence)
MultiWorkUnit)
for the job with the given job ID.jobId - job IDsequence - multi-task sequence numberpublic static List<WorkUnit> flattenWorkUnits(Collection<WorkUnit> workUnits)
List of WorkUnits, and flattens them. It builds up
the flattened list by checking each element of the given list, and seeing if it is an instance of
MultiWorkUnit. If it is then it calls itself on the WorkUnits returned by
MultiWorkUnit.getWorkUnits(). If not, then it simply adds the WorkUnit to the
flattened list.workUnits - is a List containing either WorkUnits or MultiWorkUnitsList of flattened WorkUnitspublic static void cleanStagingData(List<? extends State> states, org.slf4j.Logger logger) throws IOException
cleanTaskStagingData(State, Logger) method.states - a List of States that need their staging data cleanedIOExceptionpublic static void cleanJobStagingData(State state, org.slf4j.Logger logger) throws IOException
state - a State instance storing job configuration propertieslogger - a Logger used for loggingIOExceptionpublic static void cleanTaskStagingData(State state, org.slf4j.Logger logger) throws IOException
state - a State instance storing task configuration propertieslogger - a Logger used for loggingIOExceptionpublic static void cleanTaskStagingData(State state, org.slf4j.Logger logger, com.google.common.io.Closer closer, Map<String,ParallelRunner> parallelRunners) throws IOException
ParallelRunner.state - workunit state.logger - a Logger used for logging.closer - a closer that registers the given map of ParallelRunners. The caller is responsible
for closing the closer after the cleaning is done.parallelRunners - a map from FileSystem URI to ParallelRunner.IOException - if it fails to cleanup the task staging data.public static void cleanUpOldJobData(State state, org.slf4j.Logger logger, boolean stagingDirProvided, boolean outputDirProvided) throws IOException
IOExceptionpublic static org.apache.hadoop.fs.FileSystem getFsWithProxy(State state, String fsUri, org.apache.hadoop.conf.Configuration conf) throws IOException
state - fsUri - IOException