org.apache.tez.runtime.library.common.sort.impl.dflt
Class DefaultSorter

java.lang.Object
  org.apache.tez.runtime.library.common.sort.impl.ExternalSorter
      org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter

All Implemented Interfaces:

org.apache.hadoop.util.IndexedSortable

public class DefaultSorter
extends ExternalSorter
implements org.apache.hadoop.util.IndexedSortable

Nested Class Summary

protected class	DefaultSorter.BlockingBuffer Inner class managing the spill of serialized records to disk.
class	DefaultSorter.Buffer
protected class	DefaultSorter.MRResultIterator
protected class	DefaultSorter.SpillThread

Nested classes/interfaces inherited from class org.apache.tez.runtime.library.common.sort.impl.ExternalSorter

ExternalSorter.MapBufferTooSmallException

Field Summary

protected static int	KEYSTART
static int	MAP_OUTPUT_INDEX_RECORD_LENGTH The size of each record in the index file for the map-outputs.
protected static int	METASIZE
protected static int	NMETA
protected static int	PARTITION
protected static int	VALLEN
protected static int	VALSTART

Fields inherited from class org.apache.tez.runtime.library.common.sort.impl.ExternalSorter

additionalSpillBytesRead, additionalSpillBytesWritten, availableMemoryMb, codec, combiner, comparator, conf, fileOutputByteCounter, ifileBufferSize, ifileReadAhead, ifileReadAheadLength, keyClass, keySerializer, mapOutputByteCounter, mapOutputFile, mapOutputRecordCounter, nullProgressable, numAdditionalSpills, outputBytesWithOverheadCounter, outputContext, partitioner, partitions, rfs, serializationFactory, sorter, spilledRecordsCounter, valClass, valSerializer

Constructor Summary

DefaultSorter(org.apache.tez.runtime.api.TezOutputContext outputContext, org.apache.hadoop.conf.Configuration conf, int numOutputs, long initialMemoryAvailable)

Method Summary

void	close()
int	compare(int mi, int mj) Compare logical range, st i, j MOD offset capacity.
void	flush()
protected int	getInMemVBytesLength(int kvoff)
protected void	sortAndSpill()
protected void	spill(int mstart, int mend)
void	swap(int mi, int mj) Swap metadata for items i,j
void	write(Object key, Object value)

Methods inherited from class org.apache.tez.runtime.library.common.sort.impl.ExternalSorter

getInitialMemoryRequirement, getMapOutput, getShuffleHeader, getSortedStream, runCombineProcessor, sameVolRename

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAP_OUTPUT_INDEX_RECORD_LENGTH

public static final int MAP_OUTPUT_INDEX_RECORD_LENGTH

The size of each record in the index file for the map-outputs.

See Also:

Constant Field Values

VALSTART

protected static final int VALSTART

See Also:

Constant Field Values

KEYSTART

protected static final int KEYSTART

See Also:

Constant Field Values

PARTITION

protected static final int PARTITION

See Also:

Constant Field Values

VALLEN

protected static final int VALLEN

See Also:

Constant Field Values

NMETA

protected static final int NMETA

See Also:

Constant Field Values

METASIZE

protected static final int METASIZE

See Also:

Constant Field Values

Constructor Detail

DefaultSorter

public DefaultSorter(org.apache.tez.runtime.api.TezOutputContext outputContext,
                     org.apache.hadoop.conf.Configuration conf,
                     int numOutputs,
                     long initialMemoryAvailable)
              throws IOException

Throws:

IOException

Method Detail

write

public void write(Object key,
                  Object value)
           throws IOException

Specified by:

write in class ExternalSorter

Throws:

IOException

compare

public int compare(int mi,
                   int mj)

Compare logical range, st i, j MOD offset capacity. Compare by partition, then by key.

Specified by:

compare in interface org.apache.hadoop.util.IndexedSortable

See Also:

IndexedSortable.compare(int, int)

swap

public void swap(int mi,
                 int mj)

Swap metadata for items i,j

Specified by:

swap in interface org.apache.hadoop.util.IndexedSortable

See Also:

IndexedSortable.swap(int, int)

flush

public void flush()
           throws IOException

Specified by:

flush in class ExternalSorter

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in class ExternalSorter

Throws:

IOException

sortAndSpill

protected void sortAndSpill()
                     throws IOException,
                            InterruptedException

Throws:

IOException

InterruptedException

spill

protected void spill(int mstart,
                     int mend)
              throws IOException,
                     InterruptedException

Throws:

IOException

InterruptedException

getInMemVBytesLength

protected int getInMemVBytesLength(int kvoff)