org.apache.gobblin.util.logs

Class LogCopier

java.lang.Object

com.google.common.util.concurrent.AbstractScheduledService

org.apache.gobblin.util.logs.LogCopier

All Implemented Interfaces:

com.google.common.util.concurrent.Service

public class LogCopier
extends com.google.common.util.concurrent.AbstractScheduledService

A utility service that periodically reads log files in a source log file directory for changes since the last reads and appends the changes to destination log files with the same names as the source log files in a destination log directory. The source and destination log files can be on different FileSystems.

This class extends the AbstractScheduledService so it can be used with a ServiceManager that manages the lifecycle of a LogCopier.

This class is intended to be used in the following pattern:

     
       LogCopier.Builder logCopierBuilder = LogCopier.newBuilder();
       LogCopier logCopier = logCopierBuilder
           .useSrcFileSystem(FileSystem.getLocal(new Configuration()))
           .useDestFileSystem(FileSystem.get(URI.create(destFsUri), new Configuration()))
           .readFrom(new Path(srcLogDir))
           .writeTo(new Path(destLogDir))
           .useSourceLogFileMonitorInterval(60)
           .useTimeUnit(TimeUnit.SECONDS)
           .build();

       ServiceManager serviceManager = new ServiceManager(Lists.newArrayList(logCopier));
       serviceManager.startAsync();

       // ...
       serviceManager.stopAsync().awaitStopped(60, TimeUnit.SECONDS);
     
   

Checkout the Javadoc of LogCopier.Builder to see the available options for customization.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	LogCopier.Builder A builder class for LogCopier.

Nested classes/interfaces inherited from class com.google.common.util.concurrent.AbstractScheduledService

com.google.common.util.concurrent.AbstractScheduledService.CustomScheduler, com.google.common.util.concurrent.AbstractScheduledService.Scheduler

Nested classes/interfaces inherited from interface com.google.common.util.concurrent.Service

com.google.common.util.concurrent.Service.Listener, com.google.common.util.concurrent.Service.State

Method Summary

Modifier and Type	Method and Description
static LogCopier.Builder	newBuilder() Get a new LogCopier.Builder instance for building a LogCopier.
protected void	runOneIteration()
protected com.google.common.util.concurrent.AbstractScheduledService.Scheduler	scheduler()
protected void	shutDown()

Methods inherited from class com.google.common.util.concurrent.AbstractScheduledService

addListener, awaitRunning, awaitRunning, awaitTerminated, awaitTerminated, executor, failureCause, isRunning, serviceName, start, startAndWait, startAsync, startUp, state, stop, stopAndWait, stopAsync, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Method Detail

shutDown

protected void shutDown()
                 throws Exception

Overrides:

shutDown in class com.google.common.util.concurrent.AbstractScheduledService

Throws:

Exception

runOneIteration

protected void runOneIteration()
                        throws IOException

Specified by:

runOneIteration in class com.google.common.util.concurrent.AbstractScheduledService

Throws:

IOException

scheduler

protected com.google.common.util.concurrent.AbstractScheduledService.Scheduler scheduler()

Specified by:

scheduler in class com.google.common.util.concurrent.AbstractScheduledService

newBuilder

public static LogCopier.Builder newBuilder()

Get a new LogCopier.Builder instance for building a LogCopier.

Returns:

a new LogCopier.Builder instance