org.datavec.api.split.partition

Class NumberOfRecordsPartitioner

java.lang.Object

org.datavec.api.split.partition.NumberOfRecordsPartitioner

All Implemented Interfaces:

Partitioner

public class NumberOfRecordsPartitioner
extends Object
implements Partitioner

Partition relative to number of records written per file. This partitioner will ensure that no more than recordsPerFile number of records is written per file when outputting to the various locations of the InputSplit locations.

Author:

Adam Gibson

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_RECORDS_PER_FILE
static String	RECORDS_PER_FILE_CONFIG

Constructor Summary

Constructors

Constructor and Description
NumberOfRecordsPartitioner()

Method Summary

Modifier and Type	Method and Description
OutputStream	currentOutputStream() Get the current output stream
void	init(Configuration configuration, InputSplit split) Initializes this partitioner with the given configuration and input split
void	init(InputSplit inputSplit) Initializes this partitioner with the given configuration and input split
boolean	needsNewPartition() Returns true if the partition needs to be moved to the next.
int	numPartitions() Returns the number of partitions
int	numRecordsWritten() Number of records written so far
OutputStream	openNewStream() "Increment" to the next stream
int	totalRecordsWritten() Returns the total records written
void	updatePartitionInfo(PartitionMetaData metadata) Updates the metadata for this partitioner (to indicate whether the next partition is needed or not)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_RECORDS_PER_FILE

public static final int DEFAULT_RECORDS_PER_FILE

See Also:

Constant Field Values

RECORDS_PER_FILE_CONFIG

public static final String RECORDS_PER_FILE_CONFIG

See Also:

Constant Field Values

Constructor Detail

NumberOfRecordsPartitioner

public NumberOfRecordsPartitioner()

Method Detail

totalRecordsWritten

public int totalRecordsWritten()

Description copied from interface: Partitioner

Returns the total records written

Specified by:

totalRecordsWritten in interface Partitioner

Returns:

numRecordsWritten

public int numRecordsWritten()

Description copied from interface: Partitioner

Number of records written so far

Specified by:

numRecordsWritten in interface Partitioner

Returns:

numPartitions

public int numPartitions()

Description copied from interface: Partitioner

Returns the number of partitions

Specified by:

numPartitions in interface Partitioner

Returns:

init

public void init(InputSplit inputSplit)

Description copied from interface: Partitioner

Initializes this partitioner with the given configuration and input split

Specified by:

init in interface Partitioner

Parameters:

inputSplit - the input split to use with this partitioner

init

public void init(Configuration configuration,
                 InputSplit split)

Description copied from interface: Partitioner

Initializes this partitioner with the given configuration and input split

Specified by:

init in interface Partitioner

Parameters:

configuration - the configuration to configure this partitioner with

split - the input split to use with this partitioner

updatePartitionInfo

public void updatePartitionInfo(PartitionMetaData metadata)

Description copied from interface: Partitioner

Updates the metadata for this partitioner (to indicate whether the next partition is needed or not)

Specified by:

updatePartitionInfo in interface Partitioner

needsNewPartition

public boolean needsNewPartition()

Description copied from interface: Partitioner

Returns true if the partition needs to be moved to the next. This is controlled with Partitioner.updatePartitionInfo(PartitionMetaData) which handles incrementing counters and the like to determine whether the current partition has been exhausted.

Specified by:

needsNewPartition in interface Partitioner

Returns:

openNewStream

public OutputStream openNewStream()

Description copied from interface: Partitioner

"Increment" to the next stream

Specified by:

openNewStream in interface Partitioner

Returns:

the new opened output stream

currentOutputStream

public OutputStream currentOutputStream()

Description copied from interface: Partitioner

Get the current output stream

Specified by:

currentOutputStream in interface Partitioner

Returns: