org.apache.tez.runtime.library.common.shuffle.impl
Class MergeManager

java.lang.Object
  org.apache.tez.runtime.library.common.shuffle.impl.MergeManager

@InterfaceAudience.Private
@InterfaceStability.Unstable
public class MergeManager
extends Object

Usage. Create instance. setInitialMemoryAvailable(long), configureAndStart()

Constructor Summary

MergeManager(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem localFS, org.apache.hadoop.fs.LocalDirAllocator localDirAllocator, org.apache.tez.runtime.api.TezInputContext inputContext, Combiner combiner, org.apache.tez.common.counters.TezCounter spilledRecordsCounter, org.apache.tez.common.counters.TezCounter reduceCombineInputCounter, org.apache.tez.common.counters.TezCounter mergedMapOutputsCounter, org.apache.tez.runtime.library.common.shuffle.impl.ExceptionReporter exceptionReporter, long initialMemoryAvailable, org.apache.hadoop.io.compress.CompressionCodec codec, boolean ifileReadAheadEnabled, int ifileReadAheadLength)
Construct the MergeManager.

Method Summary

TezRawKeyValueIterator	close()
void	closeInMemoryFile(org.apache.tez.runtime.library.common.shuffle.impl.MapOutput mapOutput)
void	closeInMemoryMergedFile(org.apache.tez.runtime.library.common.shuffle.impl.MapOutput mapOutput)
void	closeOnDiskFile(org.apache.hadoop.fs.Path file)
boolean	isMergeComplete() Should only be used after the Shuffle phaze is complete, otherwise can return an invalid state since a merge may not be in progress dur to inadequate inputs
org.apache.tez.runtime.library.common.shuffle.impl.MapOutput	reserve(InputAttemptIdentifier srcAttemptIdentifier, long requestedSize, int fetcher)
void	waitForInMemoryMerge()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MergeManager

public MergeManager(org.apache.hadoop.conf.Configuration conf,
                    org.apache.hadoop.fs.FileSystem localFS,
                    org.apache.hadoop.fs.LocalDirAllocator localDirAllocator,
                    org.apache.tez.runtime.api.TezInputContext inputContext,
                    Combiner combiner,
                    org.apache.tez.common.counters.TezCounter spilledRecordsCounter,
                    org.apache.tez.common.counters.TezCounter reduceCombineInputCounter,
                    org.apache.tez.common.counters.TezCounter mergedMapOutputsCounter,
                    org.apache.tez.runtime.library.common.shuffle.impl.ExceptionReporter exceptionReporter,
                    long initialMemoryAvailable,
                    org.apache.hadoop.io.compress.CompressionCodec codec,
                    boolean ifileReadAheadEnabled,
                    int ifileReadAheadLength)

Construct the MergeManager. Must call start before it becomes usable.

Method Detail

waitForInMemoryMerge

public void waitForInMemoryMerge()
                          throws InterruptedException

Throws:

InterruptedException

reserve

public org.apache.tez.runtime.library.common.shuffle.impl.MapOutput reserve(InputAttemptIdentifier srcAttemptIdentifier,
                                                                            long requestedSize,
                                                                            int fetcher)
                                                                     throws IOException

Throws:

IOException

closeInMemoryFile

public void closeInMemoryFile(org.apache.tez.runtime.library.common.shuffle.impl.MapOutput mapOutput)

closeInMemoryMergedFile

public void closeInMemoryMergedFile(org.apache.tez.runtime.library.common.shuffle.impl.MapOutput mapOutput)

closeOnDiskFile

public void closeOnDiskFile(org.apache.hadoop.fs.Path file)

isMergeComplete

@InterfaceAudience.Private
public boolean isMergeComplete()

Should only be used after the Shuffle phaze is complete, otherwise can return an invalid state since a merge may not be in progress dur to inadequate inputs

Returns:

true if the merge process is complete, otherwise false

close

public TezRawKeyValueIterator close()
                             throws Throwable

Throws:

Throwable