alluxio.master.job

Class JobMaster

java.lang.Object

alluxio.master.AbstractMaster

alluxio.master.job.JobMaster

All Implemented Interfaces:

Checkpointed, Journaled, JournalEntryIterable, NoopJournaled, Master, Server<java.lang.Boolean>

@ThreadSafe
public class JobMaster
extends AbstractMaster
implements NoopJournaled

The master that handles all job managing operations.

Field Summary

Fields inherited from class alluxio.master.AbstractMaster

mClock, mMasterContext

Constructor Summary

Constructors

Constructor and Description
JobMaster(MasterContext masterContext, FileSystem filesystem, FileSystemContext fsContext, UfsManager ufsManager) Creates a new instance of JobMaster.

Method Summary

Modifier and Type	Method and Description
void	cancel(long jobId) Cancels a job.
java.util.List<JobInfo>	failed(int limit, long before, long after)
java.util.List<JobWorkerHealth>	getAllWorkerHealth()
java.lang.String	getName()
long	getNewJobId()
java.util.Map<ServiceType,GrpcService>	getServices()
JobInfo	getStatus(long jobId) Gets information of the given job id (verbose = True).
JobInfo	getStatus(long jobId, boolean verbose) Gets information of the given job id.
JobServiceSummary	getSummary() Gets summary of the job service.
java.util.List<WorkerInfo>	getWorkerInfoList()
java.util.List<java.lang.Long>	list(ListAllPOptions options)
java.util.List<JobInfo>	listDetailed()
long	registerWorker(WorkerNetAddress workerNetAddress) Returns a worker id for the given worker.
long	run(JobConfig jobConfig) Runs a job with the given configuration.
void	run(JobConfig jobConfig, long jobId) Runs a job with the given configuration and job id.
void	setTaskPoolSize(int taskPoolSize)
void	start(java.lang.Boolean isLeader)
void	stop()
java.util.List<JobCommand>	workerHeartbeat(JobWorkerHealth jobWorkerHealth, java.util.List<TaskInfo> taskInfoList) Updates the tasks' status when a worker periodically heartbeats with the master, and sends the commands for the worker to execute.

Methods inherited from class alluxio.master.AbstractMaster

close, createJournalContext, getDependencies, getExecutorService, getMasterContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface alluxio.master.journal.NoopJournaled

getCheckpointName, getJournalEntryIterator, processJournalEntry, resetState, restoreFromCheckpoint, writeToCheckpoint

Methods inherited from interface alluxio.master.journal.Journaled

applyAndJournal

Constructor Detail

JobMaster

public JobMaster(MasterContext masterContext,
                 FileSystem filesystem,
                 FileSystemContext fsContext,
                 UfsManager ufsManager)

Creates a new instance of JobMaster.

Parameters:

masterContext - the context for Alluxio master

filesystem - the Alluxio filesystem client the job master uses to communicate

fsContext - the filesystem client's underlying context

ufsManager - the ufs manager

Method Detail

getNewJobId

public long getNewJobId()

Returns:

new job id

start

public void start(java.lang.Boolean isLeader)
           throws java.io.IOException

Specified by:

start in interface Server<java.lang.Boolean>

Overrides:

start in class AbstractMaster

Throws:

java.io.IOException

stop

public void stop()
          throws java.io.IOException

Specified by:

stop in interface Server<java.lang.Boolean>

Overrides:

stop in class AbstractMaster

Throws:

java.io.IOException

getServices

public java.util.Map<ServiceType,GrpcService> getServices()

Specified by:

getServices in interface Server<java.lang.Boolean>

getName

public java.lang.String getName()

Specified by:

getName in interface Server<java.lang.Boolean>

run

public long run(JobConfig jobConfig)
         throws JobDoesNotExistException,
                ResourceExhaustedException

Runs a job with the given configuration.

Parameters:

jobConfig - the job configuration

Returns:

the job id tracking the progress

Throws:

JobDoesNotExistException - when the job doesn't exist

ResourceExhaustedException - if the job master is too busy to run the job

run

public void run(JobConfig jobConfig,
                long jobId)
         throws JobDoesNotExistException,
                ResourceExhaustedException

Runs a job with the given configuration and job id.

Parameters:

jobConfig - the job configuration

jobId - the job id

Throws:

JobDoesNotExistException - when the job doesn't exist

ResourceExhaustedException - if the job master is too busy to run the job

cancel

public void cancel(long jobId)
            throws JobDoesNotExistException

Cancels a job.

Parameters:

jobId - the id of the job

Throws:

JobDoesNotExistException - when the job does not exist

list

public java.util.List<java.lang.Long> list(ListAllPOptions options)

Parameters:

options - listing options

Returns:

list of all job ids

listDetailed

public java.util.List<JobInfo> listDetailed()

Returns:

list of all job infos

failed

public java.util.List<JobInfo> failed(int limit,
                                      long before,
                                      long after)

Parameters:

limit - maximum number of jobInfos to return

before - filters out on or after this timestamp (in ms) (-1 to disable)

after - filter out on or before this timestamp (in ms) (-1 to disable)

Returns:

list of all failed job infos ordered by when it failed (recently failed first)

getStatus

public JobInfo getStatus(long jobId)
                  throws JobDoesNotExistException

Gets information of the given job id (verbose = True).

Parameters:

jobId - the id of the job

Returns:

the job information

Throws:

JobDoesNotExistException - if the job does not exist

getStatus

public JobInfo getStatus(long jobId,
                         boolean verbose)
                  throws JobDoesNotExistException

Gets information of the given job id.

Parameters:

jobId - the id of the job

verbose - whether the job info should be verbose

Returns:

the job information

Throws:

JobDoesNotExistException - if the job does not exist

getSummary

public JobServiceSummary getSummary()

Gets summary of the job service.

Returns:

JobServiceSummary

getAllWorkerHealth

public java.util.List<JobWorkerHealth> getAllWorkerHealth()

Returns:

health metrics for each of the job workers

registerWorker

public long registerWorker(WorkerNetAddress workerNetAddress)

Returns a worker id for the given worker.

Parameters:

workerNetAddress - the worker WorkerNetAddress

Returns:

the worker id for this worker

getWorkerInfoList

public java.util.List<WorkerInfo> getWorkerInfoList()

Returns:

a list of WorkerInfo objects representing the workers in Alluxio

setTaskPoolSize

public void setTaskPoolSize(int taskPoolSize)

Parameters:

taskPoolSize - the task pool size for the job workers

workerHeartbeat

public java.util.List<JobCommand> workerHeartbeat(JobWorkerHealth jobWorkerHealth,
                                                  java.util.List<TaskInfo> taskInfoList)
                                           throws ResourceExhaustedException

Updates the tasks' status when a worker periodically heartbeats with the master, and sends the commands for the worker to execute.

Parameters:

jobWorkerHealth - the job worker health info

taskInfoList - the list of the task information

Returns:

the list of JobCommand to the worker

Throws:

ResourceExhaustedException