org.apache.tez.runtime.library.input
Class ShuffledMergedInput

java.lang.Object
  org.apache.tez.runtime.library.input.ShuffledMergedInput

All Implemented Interfaces:

org.apache.tez.runtime.api.Input, org.apache.tez.runtime.api.LogicalInput

Direct Known Subclasses:

ShuffledMergedInputLegacy

public class ShuffledMergedInput
extends Object
implements org.apache.tez.runtime.api.LogicalInput

ShuffleMergedInput in a LogicalInput which shuffles intermediate sorted data, merges them and provides key/ to the consumer. The Copy and Merge will be triggered by the initialization - which is handled by the Tez framework. Input is not consumable until the Copy and Merge are complete. Methods are provided to check for this, as well as to wait for completion. Attempting to get a reader on a non-complete input will block.

Field Summary

protected org.apache.hadoop.conf.Configuration	conf
protected org.apache.tez.runtime.api.TezInputContext	inputContext
protected MemoryUpdateCallbackHandler	memoryUpdateCallbackHandler
protected int	numInputs
protected TezRawKeyValueIterator	rawIter
protected Shuffle	shuffle
protected ValuesIterator	vIter

Constructor Summary

ShuffledMergedInput()

Method Summary

List<org.apache.tez.runtime.api.Event>	close()
protected void	createValuesIterator()
KeyValuesReader	getReader() Get a KVReader for the Input.
void	handleEvents(List<org.apache.tez.runtime.api.Event> inputEvents)
List<org.apache.tez.runtime.api.Event>	initialize(org.apache.tez.runtime.api.TezInputContext inputContext)
boolean	isInputReady() Check if the input is ready for consumption
void	setNumPhysicalInputs(int numInputs)
void	start()
void	waitForInputReady() Waits for the input to become ready for consumption

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

inputContext

protected org.apache.tez.runtime.api.TezInputContext inputContext

rawIter

protected TezRawKeyValueIterator rawIter

conf

protected org.apache.hadoop.conf.Configuration conf

numInputs

protected int numInputs

shuffle

protected Shuffle shuffle

memoryUpdateCallbackHandler

protected MemoryUpdateCallbackHandler memoryUpdateCallbackHandler

vIter

protected ValuesIterator vIter

Constructor Detail

ShuffledMergedInput

public ShuffledMergedInput()

Method Detail

initialize

public List<org.apache.tez.runtime.api.Event> initialize(org.apache.tez.runtime.api.TezInputContext inputContext)
                                                  throws IOException

Specified by:

initialize in interface org.apache.tez.runtime.api.Input

Throws:

IOException

start

public void start()
           throws IOException

Specified by:

start in interface org.apache.tez.runtime.api.Input

Throws:

IOException

isInputReady

public boolean isInputReady()

Check if the input is ready for consumption

Returns:

true if the input is ready for consumption, or if an error occurred processing fetching the input. false if the shuffle and merge are still in progress

waitForInputReady

public void waitForInputReady()
                       throws IOException,
                              InterruptedException

Waits for the input to become ready for consumption

Throws:

IOException

InterruptedException

close

public List<org.apache.tez.runtime.api.Event> close()
                                             throws IOException

Specified by:

close in interface org.apache.tez.runtime.api.Input

Throws:

IOException

getReader

public KeyValuesReader getReader()
                          throws IOException

Get a KVReader for the Input.

This method will block until the input is ready - i.e. the copy and merge stages are complete. Users can use the isInputReady method to check if the input is ready, which gives an indication of whether this method will block or not. NOTE: All values for the current K-V pair must be read prior to invoking moveToNext. Once moveToNext() is called, the valueIterator from the previous K-V pair will throw an Exception

Specified by:

getReader in interface org.apache.tez.runtime.api.Input

Returns:

a KVReader over the sorted input.

Throws:

IOException

handleEvents

public void handleEvents(List<org.apache.tez.runtime.api.Event> inputEvents)

Specified by:

handleEvents in interface org.apache.tez.runtime.api.Input

setNumPhysicalInputs

public void setNumPhysicalInputs(int numInputs)

Specified by:

setNumPhysicalInputs in interface org.apache.tez.runtime.api.LogicalInput

createValuesIterator

protected void createValuesIterator()
                             throws IOException

Throws:

IOException