org.datavec.api.records.reader.impl

Class FileRecordReader

java.lang.Object

org.datavec.api.records.reader.BaseRecordReader

org.datavec.api.records.reader.impl.FileRecordReader

All Implemented Interfaces:

Closeable, Serializable, AutoCloseable, Configurable, RecordReader

Direct Known Subclasses:

CSVSequenceRecordReader, JacksonLineSequenceRecordReader, MatlabRecordReader, RegexSequenceRecordReader

public class FileRecordReader
extends BaseRecordReader

File reader/writer

Author:

Adam Gibson

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	appendLabel
protected String	charset
protected Configuration	conf
protected URI	currentUri
protected List<String>	labels
protected Iterator<URI>	locationsIterator

Fields inherited from class org.datavec.api.records.reader.BaseRecordReader

inputSplit, listeners, streamCreatorFn

Fields inherited from interface org.datavec.api.records.reader.RecordReader

APPEND_LABEL, LABELS, NAME_SPACE

Constructor Summary

Constructors

Constructor and Description
FileRecordReader()

Method Summary

Modifier and Type	Method and Description
void	close()
protected void	doInitialize(InputSplit split)
Configuration	getConf() Return the configuration used by this object.
int	getCurrentLabel() Return the current label.
int	getLabel(URI uri)
List<String>	getLabels() List of label strings
boolean	hasNext() Whether there are anymore records
void	initialize(Configuration conf, InputSplit split) Called once at initialization.
void	initialize(InputSplit split) Called once at initialization.
List<Record>	loadFromMetaData(List<RecordMetaData> recordMetaDatas) Load multiple records from the given a list of RecordMetaData instances
Record	loadFromMetaData(RecordMetaData recordMetaData) Load a single record from the given RecordMetaData instance Note: that for data that isn't splittable (i.e., text data that needs to be scanned/split), it is more efficient to load multiple records at once using RecordReader.loadFromMetaData(List)
List<Writable>	next() Get the next record
List<List<Writable>>	next(int num) This method will be used, if batchesSupported() returns true.
Record	nextRecord() Similar to RecordReader.next(), but returns a Record object, that may include metadata such as the source of the data
List<Writable>	record(URI uri, DataInputStream dataInputStream) Load the record from the given DataInputStream Unlike RecordReader.next() the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStream
void	reset() Reset record reader iterator
boolean	resetSupported()
void	setConf(Configuration conf) Set the configuration to be used by this object.
void	setLabels(List<String> labels)

Methods inherited from class org.datavec.api.records.reader.BaseRecordReader

batchesSupported, getListeners, invokeListeners, setListeners, setListeners

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

locationsIterator

protected Iterator<URI> locationsIterator

conf

protected Configuration conf

currentUri

protected URI currentUri

labels

protected List<String> labels

appendLabel

protected boolean appendLabel

charset

protected String charset

Constructor Detail

FileRecordReader

public FileRecordReader()

Method Detail

initialize

public void initialize(InputSplit split)
                throws IOException,
                       InterruptedException

Description copied from interface: RecordReader

Called once at initialization.

Specified by:

initialize in interface RecordReader

Overrides:

initialize in class BaseRecordReader

Parameters:

split - the split that defines the range of records to read

Throws:

IOException

InterruptedException

doInitialize

protected void doInitialize(InputSplit split)

initialize

public void initialize(Configuration conf,
                       InputSplit split)
                throws IOException,
                       InterruptedException

Description copied from interface: RecordReader

Called once at initialization.

Parameters:

conf - a configuration for initialization

split - the split that defines the range of records to read

Throws:

IOException

InterruptedException

next

public List<Writable> next()

Description copied from interface: RecordReader

Get the next record

Returns:

getCurrentLabel

public int getCurrentLabel()

Return the current label. The index of the current file's parent directory in the label list

Returns:

The index of the current file's parent directory

getLabel

public int getLabel(URI uri)

getLabels

public List<String> getLabels()

Description copied from interface: RecordReader

List of label strings

Returns:

setLabels

public void setLabels(List<String> labels)

hasNext

public boolean hasNext()

Description copied from interface: RecordReader

Whether there are anymore records

Returns:

close

public void close()
           throws IOException

Throws:

IOException

setConf

public void setConf(Configuration conf)

Description copied from interface: Configurable

Set the configuration to be used by this object.

getConf

public Configuration getConf()

Description copied from interface: Configurable

Return the configuration used by this object.

next

public List<List<Writable>> next(int num)

Description copied from interface: RecordReader

This method will be used, if batchesSupported() returns true.

Specified by:

next in interface RecordReader

Overrides:

next in class BaseRecordReader

Returns:

reset

public void reset()

Description copied from interface: RecordReader

Reset record reader iterator

resetSupported

public boolean resetSupported()

Returns:

True if the record reader can be reset, false otherwise. Note that some record readers cannot be reset - for example, if they are backed by a non-resettable input split (such as certain types of streams)

record

public List<Writable> record(URI uri,
                             DataInputStream dataInputStream)
                      throws IOException

Description copied from interface: RecordReader

Load the record from the given DataInputStream Unlike RecordReader.next() the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStream

Throws:

IOException - if error occurs during reading from the input stream

nextRecord

public Record nextRecord()

Description copied from interface: RecordReader

Similar to RecordReader.next(), but returns a Record object, that may include metadata such as the source of the data

Returns:

next record

loadFromMetaData

public Record loadFromMetaData(RecordMetaData recordMetaData)
                        throws IOException

Description copied from interface: RecordReader

Load a single record from the given RecordMetaData instance
Note: that for data that isn't splittable (i.e., text data that needs to be scanned/split), it is more efficient to load multiple records at once using RecordReader.loadFromMetaData(List)

Parameters:

recordMetaData - Metadata for the record that we want to load from

Returns:

Single record for the given RecordMetaData instance

Throws:

IOException - If I/O error occurs during loading

loadFromMetaData

public List<Record> loadFromMetaData(List<RecordMetaData> recordMetaDatas)
                              throws IOException

Description copied from interface: RecordReader

Load multiple records from the given a list of RecordMetaData instances

Parameters:

recordMetaDatas - Metadata for the records that we want to load from

Returns:

Multiple records for the given RecordMetaData instances

Throws:

IOException - If I/O error occurs during loading