org.apache.flink.runtime.scheduler.adaptivebatch

Class AdaptiveBatchScheduler

java.lang.Object

org.apache.flink.runtime.scheduler.SchedulerBase

org.apache.flink.runtime.scheduler.DefaultScheduler

org.apache.flink.runtime.scheduler.adaptivebatch.AdaptiveBatchScheduler

All Implemented Interfaces:

AutoCloseable, CheckpointScheduling, GlobalFailureHandler, SchedulerNG, SchedulerOperations, org.apache.flink.util.AutoCloseableAsync

public class AdaptiveBatchScheduler
extends DefaultScheduler

This scheduler decides the parallelism of JobVertex according to the data volume it consumes. A dynamically built up ExecutionGraph is used for this purpose.

Field Summary

Fields inherited from class org.apache.flink.runtime.scheduler.DefaultScheduler

executionDeployer, executionSlotAllocator, failoverStrategy, log, schedulingStrategy, shuffleMaster

Fields inherited from class org.apache.flink.runtime.scheduler.SchedulerBase

executionVertexVersioner, inputsLocationsRetriever, jobInfo, jobManagerJobMetricGroup, operatorCoordinatorHandler, stateLocationRetriever

Constructor Summary

Constructors

Constructor and Description

AdaptiveBatchScheduler(org.slf4j.Logger log, JobGraph jobGraph, Executor ioExecutor, org.apache.flink.configuration.Configuration jobMasterConfiguration, Consumer<org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor> startUpAction, org.apache.flink.util.concurrent.ScheduledExecutor delayExecutor, ClassLoader userCodeLoader, CheckpointsCleaner checkpointsCleaner, CheckpointRecoveryFactory checkpointRecoveryFactory, JobManagerJobMetricGroup jobManagerJobMetricGroup, SchedulingStrategyFactory schedulingStrategyFactory, FailoverStrategy.Factory failoverStrategyFactory, RestartBackoffTimeStrategy restartBackoffTimeStrategy, ExecutionOperations executionOperations, ExecutionVertexVersioner executionVertexVersioner, ExecutionSlotAllocatorFactory executionSlotAllocatorFactory, long initializationTimestamp, org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor mainThreadExecutor, JobStatusListener jobStatusListener, Collection<org.apache.flink.core.failure.FailureEnricher> failureEnrichers, ExecutionGraphFactory executionGraphFactory, ShuffleMaster<?> shuffleMaster, org.apache.flink.api.common.time.Time rpcTimeout, VertexParallelismAndInputInfosDecider vertexParallelismAndInputInfosDecider, int defaultMaxParallelism, BlocklistOperations blocklistOperations, org.apache.flink.configuration.JobManagerOptions.HybridPartitionDataConsumeConstraint hybridPartitionDataConsumeConstraint, Map<JobVertexID,ForwardGroup> forwardGroupsByJobVertexId, BatchJobRecoveryHandler jobRecoveryHandler)

Method Summary

Modifier and Type	Method and Description
void	allocateSlotsAndDeploy(List<ExecutionVertexID> verticesToDeploy) Allocate slots and deploy the vertex when slots are returned.
CompletableFuture<Void>	closeAsync()
List<CompletableFuture<Integer>>	computeDynamicSourceParallelism()
static VertexParallelismStore	computeVertexParallelismStoreForDynamicGraph(Iterable<JobVertex> vertices, int defaultMaxParallelism) Compute the VertexParallelismStore for all given vertices in a dynamic graph, which will set defaults and ensure that the returned store contains valid parallelisms, with the configured default max parallelism.
protected MarkPartitionFinishedStrategy	getMarkPartitionFinishedStrategy()
protected void	handleTaskFailure(Execution failedExecution, Throwable error)
void	initializeVerticesIfPossible()
protected void	maybeRestartTasks(FailureHandlingResult failureHandlingResult) Modifies the vertices which need to be restarted.
protected void	onTaskFailed(Execution execution)
protected void	onTaskFinished(Execution execution, IOMetrics ioMetrics)
protected void	resetForNewExecution(ExecutionVertexID executionVertexId)
protected void	resetForNewExecutions(Collection<ExecutionVertexID> vertices)
protected void	startSchedulingInternal()

Methods inherited from class org.apache.flink.runtime.scheduler.DefaultScheduler

cancelAllPendingSlotRequestsForVertex, cancelAllPendingSlotRequestsInternal, cancelExecution, createFailureHandlingResultSnapshot, getNumberOfRestarts, handleGlobalFailure, notifyCoordinatorsAboutTaskFailure, recordTaskFailure

Methods inherited from class org.apache.flink.runtime.scheduler.SchedulerBase

acknowledgeCheckpoint, archiveFromFailureHandlingResult, archiveGlobalFailure, cancel, computeVertexParallelismStore, computeVertexParallelismStore, computeVertexParallelismStore, computeVertexParallelismStore, declineCheckpoint, deliverCoordinationRequestToCoordinator, deliverOperatorEventToCoordinator, failJob, getDefaultMaxParallelism, getExceptionHistory, getExecutionGraph, getExecutionJobVertex, getExecutionVertex, getJobGraph, getJobId, getJobTerminationFuture, getMainThreadExecutor, getResultPartitionAvailabilityChecker, getSchedulingTopology, notifyEndOfData, notifyKvStateRegistered, notifyKvStateUnregistered, registerJobMetrics, reportCheckpointMetrics, reportInitializationMetrics, requestCheckpointStats, requestJob, requestJobStatus, requestKvStateLocation, requestNextInputSplit, requestPartitionState, restoreState, setGlobalFailureCause, startCheckpointScheduler, startScheduling, stopCheckpointScheduler, stopWithSavepoint, transitionExecutionGraphState, transitionToRunning, transitionToScheduled, triggerCheckpoint, triggerSavepoint, updateAccumulators, updateTaskExecutionState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.scheduler.SchedulerNG

requestJobResourceRequirements, updateJobResourceRequirements, updateTaskExecutionState

Methods inherited from interface org.apache.flink.util.AutoCloseableAsync

close

Constructor Detail

AdaptiveBatchScheduler

public AdaptiveBatchScheduler(org.slf4j.Logger log,
                              JobGraph jobGraph,
                              Executor ioExecutor,
                              org.apache.flink.configuration.Configuration jobMasterConfiguration,
                              Consumer<org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor> startUpAction,
                              org.apache.flink.util.concurrent.ScheduledExecutor delayExecutor,
                              ClassLoader userCodeLoader,
                              CheckpointsCleaner checkpointsCleaner,
                              CheckpointRecoveryFactory checkpointRecoveryFactory,
                              JobManagerJobMetricGroup jobManagerJobMetricGroup,
                              SchedulingStrategyFactory schedulingStrategyFactory,
                              FailoverStrategy.Factory failoverStrategyFactory,
                              RestartBackoffTimeStrategy restartBackoffTimeStrategy,
                              ExecutionOperations executionOperations,
                              ExecutionVertexVersioner executionVertexVersioner,
                              ExecutionSlotAllocatorFactory executionSlotAllocatorFactory,
                              long initializationTimestamp,
                              org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor mainThreadExecutor,
                              JobStatusListener jobStatusListener,
                              Collection<org.apache.flink.core.failure.FailureEnricher> failureEnrichers,
                              ExecutionGraphFactory executionGraphFactory,
                              ShuffleMaster<?> shuffleMaster,
                              org.apache.flink.api.common.time.Time rpcTimeout,
                              VertexParallelismAndInputInfosDecider vertexParallelismAndInputInfosDecider,
                              int defaultMaxParallelism,
                              BlocklistOperations blocklistOperations,
                              org.apache.flink.configuration.JobManagerOptions.HybridPartitionDataConsumeConstraint hybridPartitionDataConsumeConstraint,
                              Map<JobVertexID,ForwardGroup> forwardGroupsByJobVertexId,
                              BatchJobRecoveryHandler jobRecoveryHandler)
                       throws Exception

Throws:

Exception

Method Detail

startSchedulingInternal

protected void startSchedulingInternal()

Overrides:

startSchedulingInternal in class DefaultScheduler

maybeRestartTasks

protected void maybeRestartTasks(FailureHandlingResult failureHandlingResult)

Modifies the vertices which need to be restarted. If any task needing restarting belongs to job vertices with unrecovered operator coordinators, all tasks within those job vertices need to be restarted once.

Overrides:

maybeRestartTasks in class DefaultScheduler

resetForNewExecutions

protected void resetForNewExecutions(Collection<ExecutionVertexID> vertices)

Overrides:

resetForNewExecutions in class SchedulerBase

closeAsync

public CompletableFuture<Void> closeAsync()

Specified by:

closeAsync in interface org.apache.flink.util.AutoCloseableAsync

Overrides:

closeAsync in class SchedulerBase

onTaskFinished

protected void onTaskFinished(Execution execution,
                              IOMetrics ioMetrics)

Overrides:

onTaskFinished in class DefaultScheduler

onTaskFailed

protected void onTaskFailed(Execution execution)

Overrides:

onTaskFailed in class DefaultScheduler

handleTaskFailure

protected void handleTaskFailure(Execution failedExecution,
                                 @Nullable
                                 Throwable error)

Overrides:

handleTaskFailure in class DefaultScheduler

allocateSlotsAndDeploy

public void allocateSlotsAndDeploy(List<ExecutionVertexID> verticesToDeploy)

Description copied from interface: SchedulerOperations

Allocate slots and deploy the vertex when slots are returned. Vertices will be deployed only after all of them have been assigned slots. The given order will be respected, i.e. tasks with smaller indices will be deployed earlier. Only vertices in CREATED state will be accepted. Errors will happen if scheduling Non-CREATED vertices.

Specified by:

allocateSlotsAndDeploy in interface SchedulerOperations

Overrides:

allocateSlotsAndDeploy in class DefaultScheduler

Parameters:

verticesToDeploy - The execution vertices to deploy

resetForNewExecution

protected void resetForNewExecution(ExecutionVertexID executionVertexId)

Overrides:

resetForNewExecution in class SchedulerBase

getMarkPartitionFinishedStrategy

protected MarkPartitionFinishedStrategy getMarkPartitionFinishedStrategy()

Overrides:

getMarkPartitionFinishedStrategy in class SchedulerBase

computeDynamicSourceParallelism

public List<CompletableFuture<Integer>> computeDynamicSourceParallelism()

initializeVerticesIfPossible

@VisibleForTesting
public void initializeVerticesIfPossible()

computeVertexParallelismStoreForDynamicGraph

@VisibleForTesting
public static VertexParallelismStore computeVertexParallelismStoreForDynamicGraph(Iterable<JobVertex> vertices,
                                                                                                     int defaultMaxParallelism)

Compute the VertexParallelismStore for all given vertices in a dynamic graph, which will set defaults and ensure that the returned store contains valid parallelisms, with the configured default max parallelism.

Parameters:

vertices - the vertices to compute parallelism for

defaultMaxParallelism - the global default max parallelism

Returns:

the computed parallelism store