org.apache.gobblin.util

Class ParallelRunner

java.lang.Object

org.apache.gobblin.util.ParallelRunner

All Implemented Interfaces:

Closeable, AutoCloseable

public class ParallelRunner
extends Object
implements Closeable

A class that is responsible for running certain methods in parallel. Methods in this class returns immediately and are run in a fixed-size thread pool.

This class is intended to be used in the following pattern. This example uses the serialize() method.

 
     Closer closer = Closer.create();
     try {
       // Do stuff
       ParallelRunner runner = closer.register(new ParallelRunner(threads, fs));
       runner.serialize(state1, outputFilePath1);
       // Submit more serialization tasks
       runner.serialize(stateN, outputFilePathN);
       // Do stuff
     } catch (Throwable e) {
       throw closer.rethrow(e);
     } finally {
       closer.close();
     }
   

Note that calling close() will wait for all submitted tasks to complete and then stop the ParallelRunner by shutting down the ExecutorService.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ParallelRunner.FailPolicy Policies indicating how ParallelRunner should handle failure of tasks.
static class	ParallelRunner.NamedFuture A future with a name / message for reporting.

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_PARALLEL_RUNNER_THREADS
static String	PARALLEL_RUNNER_THREADS_KEY

Constructor Summary

Constructors

Constructor and Description
ParallelRunner(int threads, org.apache.hadoop.fs.FileSystem fs)
ParallelRunner(int threads, org.apache.hadoop.fs.FileSystem fs, ParallelRunner.FailPolicy failPolicy)

Method Summary

Modifier and Type	Method and Description
void	close()
void	deletePath(org.apache.hadoop.fs.Path path, boolean recursive) Delete a Path.
<T extends State> void	deserializeFromFile(T state, org.apache.hadoop.fs.Path inputFilePath) Deserialize a State object from a file.
<T extends State> void	deserializeFromSequenceFile(Class<? extends org.apache.hadoop.io.Writable> keyClass, Class<T> stateClass, org.apache.hadoop.fs.Path inputFilePath, Collection<T> states, boolean deleteAfter) Deserialize a list of State objects from a Hadoop SequenceFile.
void	movePath(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.FileSystem dstFs, org.apache.hadoop.fs.Path dst, boolean overwrite, com.google.common.base.Optional<String> group) Move a Path.
void	movePath(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.FileSystem dstFs, org.apache.hadoop.fs.Path dst, com.google.common.base.Optional<String> group) Move a Path.
void	renamePath(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst, com.google.common.base.Optional<String> group) Rename a Path.
<T extends State> void	serializeToFile(T state, org.apache.hadoop.fs.Path outputFilePath) Serialize a State object into a file.
void	submitCallable(Callable<Void> callable, String name) Submit a callable to the thread pool
void	waitForTasks() Wait until default timeout(infinite long, if not specified) for all tasks under this parallel runner.
void	waitForTasks(long timeoutInMills) Wait for all submitted tasks to complete.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PARALLEL_RUNNER_THREADS_KEY

public static final String PARALLEL_RUNNER_THREADS_KEY

See Also:

Constant Field Values

DEFAULT_PARALLEL_RUNNER_THREADS

public static final int DEFAULT_PARALLEL_RUNNER_THREADS

See Also:

Constant Field Values

Constructor Detail

ParallelRunner

public ParallelRunner(int threads,
                      org.apache.hadoop.fs.FileSystem fs)

ParallelRunner

public ParallelRunner(int threads,
                      org.apache.hadoop.fs.FileSystem fs,
                      ParallelRunner.FailPolicy failPolicy)

Method Detail

serializeToFile

public <T extends State> void serializeToFile(T state,
                                              org.apache.hadoop.fs.Path outputFilePath)

Serialize a State object into a file.

This method submits a task to serialize the State object and returns immediately after the task is submitted.

Type Parameters:

T - the State object type

Parameters:

state - the State object to be serialized

outputFilePath - the file to write the serialized State object to

deserializeFromFile

public <T extends State> void deserializeFromFile(T state,
                                                  org.apache.hadoop.fs.Path inputFilePath)

Deserialize a State object from a file.

This method submits a task to deserialize the State object and returns immediately after the task is submitted.

Type Parameters:

T - the State object type

Parameters:

state - an empty State object to which the deserialized content will be populated

inputFilePath - the input file to read from

deserializeFromSequenceFile

public <T extends State> void deserializeFromSequenceFile(Class<? extends org.apache.hadoop.io.Writable> keyClass,
                                                          Class<T> stateClass,
                                                          org.apache.hadoop.fs.Path inputFilePath,
                                                          Collection<T> states,
                                                          boolean deleteAfter)

Deserialize a list of State objects from a Hadoop SequenceFile.

This method submits a task to deserialize the State objects and returns immediately after the task is submitted.

Type Parameters:

T - the State object type

Parameters:

stateClass - the Class object of the State class

inputFilePath - the input SequenceFile to read from

states - a Collection object to store the deserialized State objects

deleteAfter - a flag telling whether to delete the SequenceFile afterwards

deletePath

public void deletePath(org.apache.hadoop.fs.Path path,
                       boolean recursive)

Delete a Path.

This method submits a task to delete a Path and returns immediately after the task is submitted.

Parameters:

path - path to be deleted.

renamePath

public void renamePath(org.apache.hadoop.fs.Path src,
                       org.apache.hadoop.fs.Path dst,
                       com.google.common.base.Optional<String> group)

Rename a Path.

This method submits a task to rename a Path and returns immediately after the task is submitted.

Parameters:

src - path to be renamed

dst - new path after rename

group - an optional group name for the destination path

movePath

public void movePath(org.apache.hadoop.fs.Path src,
                     org.apache.hadoop.fs.FileSystem dstFs,
                     org.apache.hadoop.fs.Path dst,
                     com.google.common.base.Optional<String> group)

Move a Path.

This method submits a task to move a Path and returns immediately after the task is submitted.

Parameters:

src - path to be moved

dstFs - the destination FileSystem

dst - the destination path

group - an optional group name for the destination path

movePath

public void movePath(org.apache.hadoop.fs.Path src,
                     org.apache.hadoop.fs.FileSystem dstFs,
                     org.apache.hadoop.fs.Path dst,
                     boolean overwrite,
                     com.google.common.base.Optional<String> group)

Move a Path.

This method submits a task to move a Path and returns immediately after the task is submitted.

Parameters:

src - path to be moved

dstFs - the destination FileSystem

dst - the destination path

overwrite - true to overwrite the destination

group - an optional group name for the destination path

submitCallable

public void submitCallable(Callable<Void> callable,
                           String name)

Submit a callable to the thread pool

This method submits a task and returns immediately

Parameters:

callable - the callable to submit

name - for the future

waitForTasks

public void waitForTasks(long timeoutInMills)
                  throws IOException

Wait for all submitted tasks to complete. The ParallelRunner can be reused after this call.

Throws:

IOException

waitForTasks

public void waitForTasks()
                  throws IOException

Wait until default timeout(infinite long, if not specified) for all tasks under this parallel runner.

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException