org.datavec.api.split.partition

Interface Partitioner

All Known Implementing Classes:

NumberOfRecordsPartitioner

public interface Partitioner

A partitioner for iterating thorugh files for RecordWriter. This allows for a configurable log rotation like algorithm for partitioning files by number of recodrds, sizes among other things.

Method Summary

Modifier and Type	Method and Description
OutputStream	currentOutputStream() Get the current output stream
void	init(Configuration configuration, InputSplit split) Initializes this partitioner with the given configuration and input split
void	init(InputSplit inputSplit) Initializes this partitioner with the given configuration and input split
boolean	needsNewPartition() Returns true if the partition needs to be moved to the next.
int	numPartitions() Returns the number of partitions
int	numRecordsWritten() Number of records written so far
OutputStream	openNewStream() "Increment" to the next stream
int	totalRecordsWritten() Returns the total records written
void	updatePartitionInfo(PartitionMetaData metadata) Updates the metadata for this partitioner (to indicate whether the next partition is needed or not)

Method Detail

totalRecordsWritten

int totalRecordsWritten()

Returns the total records written

Returns:

numRecordsWritten

int numRecordsWritten()

Number of records written so far

Returns:

numPartitions

int numPartitions()

Returns the number of partitions

Returns:

init

void init(InputSplit inputSplit)

Initializes this partitioner with the given configuration and input split

Parameters:

inputSplit - the input split to use with this partitioner

init

void init(Configuration configuration,
          InputSplit split)

Initializes this partitioner with the given configuration and input split

Parameters:

configuration - the configuration to configure this partitioner with

split - the input split to use with this partitioner

updatePartitionInfo

void updatePartitionInfo(PartitionMetaData metadata)

Updates the metadata for this partitioner (to indicate whether the next partition is needed or not)

Parameters:

metadata -

needsNewPartition

boolean needsNewPartition()

Returns true if the partition needs to be moved to the next. This is controlled with updatePartitionInfo(PartitionMetaData) which handles incrementing counters and the like to determine whether the current partition has been exhausted.

Returns:

openNewStream

OutputStream openNewStream()

"Increment" to the next stream

Returns:

the new opened output stream

currentOutputStream

OutputStream currentOutputStream()

Get the current output stream

Returns: