org.apache.gobblin.util

Class ExecutorsUtils

java.lang.Object

org.apache.gobblin.util.ExecutorsUtils

public class ExecutorsUtils
extends Object

A utility class to use with Executors in cases such as when creating new thread pools.

Field Summary

Fields

Modifier and Type	Field and Description
static long	EXECUTOR_SERVICE_SHUTDOWN_TIMEOUT
static TimeUnit	EXECUTOR_SERVICE_SHUTDOWN_TIMEOUT_TIMEUNIT

Constructor Summary

Constructors

Constructor and Description
ExecutorsUtils()

Method Summary

Modifier and Type	Method and Description
static ThreadFactory	defaultThreadFactory() Get a default ThreadFactory.
static <T> Callable<T>	loggingDecorator(Callable<T> callable) Creates an Callable which propagates the MDC information across thread boundaries.
static com.google.common.util.concurrent.ListeningExecutorService	loggingDecorator(ExecutorService executorService) Creates an ListeningExecutorService whose submit and execute methods propagate the MDC information across thread boundaries.
static Runnable	loggingDecorator(Runnable runnable) Creates an Runnable which propagates the MDC information across thread boundaries.
static com.google.common.util.concurrent.ListeningScheduledExecutorService	loggingDecorator(ScheduledExecutorService scheduledExecutorService) Creates an ListeningScheduledExecutorService whose submit, execute, schedule, scheduleAtFixedRate, and scheduleWithFixedDelay methods propagate the MDC information across thread boundaries.
static ThreadFactory	newDaemonThreadFactory(com.google.common.base.Optional<org.slf4j.Logger> logger, com.google.common.base.Optional<String> nameFormat) Get a new ThreadFactory that uses a LoggingUncaughtExceptionHandler to handle uncaught exceptions, uses the given thread name format, and produces daemon threads.
static ThreadFactory	newThreadFactory(com.google.common.base.Optional<org.slf4j.Logger> logger) Get a new ThreadFactory that uses a LoggingUncaughtExceptionHandler to handle uncaught exceptions.
static ThreadFactory	newThreadFactory(com.google.common.base.Optional<org.slf4j.Logger> logger, com.google.common.base.Optional<String> nameFormat) Get a new ThreadFactory that uses a LoggingUncaughtExceptionHandler to handle uncaught exceptions and the given thread name format.
static <F,T> List<T>	parallelize(List<F> list, com.google.common.base.Function<F,T> function, int threadCount, int timeoutInSecs, com.google.common.base.Optional<org.slf4j.Logger> logger) A utility method to parallelize loops.
static void	shutdownExecutorService(ExecutorService executorService, com.google.common.base.Optional<org.slf4j.Logger> logger) Shutdown an ExecutorService gradually, first disabling new task submissions and later cancelling existing tasks.
static void	shutdownExecutorService(ExecutorService executorService, com.google.common.base.Optional<org.slf4j.Logger> logger, long timeout, TimeUnit unit) Shutdown an ExecutorService gradually, first disabling new task submissions and later cancelling existing tasks.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

EXECUTOR_SERVICE_SHUTDOWN_TIMEOUT

public static final long EXECUTOR_SERVICE_SHUTDOWN_TIMEOUT

See Also:

Constant Field Values

EXECUTOR_SERVICE_SHUTDOWN_TIMEOUT_TIMEUNIT

public static final TimeUnit EXECUTOR_SERVICE_SHUTDOWN_TIMEOUT_TIMEUNIT

Constructor Detail

ExecutorsUtils

public ExecutorsUtils()

Method Detail

defaultThreadFactory

public static ThreadFactory defaultThreadFactory()

Get a default ThreadFactory.

Returns:

the default ThreadFactory

newThreadFactory

public static ThreadFactory newThreadFactory(com.google.common.base.Optional<org.slf4j.Logger> logger)

Get a new ThreadFactory that uses a LoggingUncaughtExceptionHandler to handle uncaught exceptions.

Parameters:

logger - an Optional wrapping the Logger that the LoggingUncaughtExceptionHandler uses to log uncaught exceptions thrown in threads

Returns:

a new ThreadFactory

newThreadFactory

public static ThreadFactory newThreadFactory(com.google.common.base.Optional<org.slf4j.Logger> logger,
                                             com.google.common.base.Optional<String> nameFormat)

Get a new ThreadFactory that uses a LoggingUncaughtExceptionHandler to handle uncaught exceptions and the given thread name format.

Parameters:

logger - an Optional wrapping the Logger that the LoggingUncaughtExceptionHandler uses to log uncaught exceptions thrown in threads

nameFormat - an Optional wrapping a thread naming format

Returns:

a new ThreadFactory

newDaemonThreadFactory

public static ThreadFactory newDaemonThreadFactory(com.google.common.base.Optional<org.slf4j.Logger> logger,
                                                   com.google.common.base.Optional<String> nameFormat)

Get a new ThreadFactory that uses a LoggingUncaughtExceptionHandler to handle uncaught exceptions, uses the given thread name format, and produces daemon threads.

Parameters:

logger - an Optional wrapping the Logger that the LoggingUncaughtExceptionHandler uses to log uncaught exceptions thrown in threads

nameFormat - an Optional wrapping a thread naming format

Returns:

a new ThreadFactory

loggingDecorator

public static com.google.common.util.concurrent.ListeningExecutorService loggingDecorator(ExecutorService executorService)

Creates an ListeningExecutorService whose submit and execute methods propagate the MDC information across thread boundaries.

Parameters:

executorService - the ExecutorService to wrap

Returns:

a new instance of ListeningExecutorService

loggingDecorator

public static com.google.common.util.concurrent.ListeningScheduledExecutorService loggingDecorator(ScheduledExecutorService scheduledExecutorService)

Creates an ListeningScheduledExecutorService whose submit, execute, schedule, scheduleAtFixedRate, and scheduleWithFixedDelay methods propagate the MDC information across thread boundaries.

Parameters:

scheduledExecutorService - the ScheduledExecutorService to wrap

Returns:

a new instance of ListeningScheduledExecutorService

loggingDecorator

public static Runnable loggingDecorator(Runnable runnable)

Creates an Runnable which propagates the MDC information across thread boundaries.

Parameters:

runnable - the Runnable to wrap

Returns:

a new instance of Runnable

loggingDecorator

public static <T> Callable<T> loggingDecorator(Callable<T> callable)

Creates an Callable which propagates the MDC information across thread boundaries.

Parameters:

callable - the Callable to wrap

Returns:

a new instance of Callable

shutdownExecutorService

public static void shutdownExecutorService(ExecutorService executorService,
                                           com.google.common.base.Optional<org.slf4j.Logger> logger,
                                           long timeout,
                                           TimeUnit unit)

Shutdown an ExecutorService gradually, first disabling new task submissions and later cancelling existing tasks.

The implementation is based on the implementation of Guava's MoreExecutors.shutdownAndAwaitTermination, which is available since version 17.0. We cannot use Guava version 17.0 or after directly, however, as it cannot be used with Hadoop 2.6.0 or after due to the issue reported in HADOOP-10961.

Parameters:

executorService - the ExecutorService to shutdown

logger - an Optional wrapping the Logger that is used to log metadata of the executorService if it cannot shutdown all its threads

timeout - the maximum time to wait for the ExecutorService to terminate

unit - the time unit of the timeout argument

shutdownExecutorService

public static void shutdownExecutorService(ExecutorService executorService,
                                           com.google.common.base.Optional<org.slf4j.Logger> logger)

Shutdown an ExecutorService gradually, first disabling new task submissions and later cancelling existing tasks.

This method calls shutdownExecutorService(ExecutorService, Optional, long, TimeUnit) with default timeout time EXECUTOR_SERVICE_SHUTDOWN_TIMEOUT and time unit EXECUTOR_SERVICE_SHUTDOWN_TIMEOUT_TIMEUNIT.

Parameters:

executorService - the ExecutorService to shutdown

logger - an Optional wrapping a Logger to be used during shutdown

parallelize

public static <F,T> List<T> parallelize(List<F> list,
                                        com.google.common.base.Function<F,T> function,
                                        int threadCount,
                                        int timeoutInSecs,
                                        com.google.common.base.Optional<org.slf4j.Logger> logger)
                                 throws ExecutionException

A utility method to parallelize loops. Applies the Function to every element in the List in parallel by spawning threads. A list containing the result obtained by applying the function is returned. The method is a blocking call and will wait for all the elements in the list to be processed or timeoutInSecs which ever is earlier.

NOTE: The method is an all or none implementation. Meaning, if any of the thread fails, the method will throw an ExecutionException even if other threads completed successfully

• Uses a Fixed thread pool of size threadCount.
• Uses shutdownExecutorService(ExecutorService, Optional, long, TimeUnit) to shutdown the executor service
• All threads are daemon threads

Parameters:

list - input list on which the function is applied in parallel

function - to be applied on every element of the list

threadCount - to be used to process the list

timeoutInSecs - to wait for all the threads to complete

logger - an Optional wrapping a Logger to be used during shutdown

Returns:

a list containing the result obtained by applying the function on each element of the input list in the same order

Throws:

IllegalArgumentException - if input list or function is null

ExecutionException -

• if any computation threw an exception
• if any computation was cancelled
• if any thread was interrupted while waiting