org.datavec.api.records.reader.impl.csv

Class CSVRecordReader

java.lang.Object

org.datavec.api.records.reader.BaseRecordReader

org.datavec.api.records.reader.impl.LineRecordReader

org.datavec.api.records.reader.impl.csv.CSVRecordReader

All Implemented Interfaces:

Closeable, Serializable, AutoCloseable, Configurable, RecordReader

Direct Known Subclasses:

CSVLineSequenceRecordReader, CSVMultiSequenceRecordReader, CSVNLinesSequenceRecordReader, CSVRegexRecordReader, CSVVariableSlidingWindowRecordReader

public class CSVRecordReader
extends LineRecordReader

Simple csv record reader.

Author:

Adam Gibson

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static char	DEFAULT_DELIMITER
static char	DEFAULT_QUOTE
static String	DELIMITER
static String	QUOTE
static String	SKIP_NUM_LINES
protected int	skipNumLines

Fields inherited from class org.datavec.api.records.reader.impl.LineRecordReader

charset, conf, initialized, lineIndex, locations, splitIndex

Fields inherited from class org.datavec.api.records.reader.BaseRecordReader

inputSplit, listeners, streamCreatorFn

Fields inherited from interface org.datavec.api.records.reader.RecordReader

APPEND_LABEL, LABELS, NAME_SPACE

Constructor Summary

Constructors

Constructor and Description
CSVRecordReader()
CSVRecordReader(char delimiter) Create a CSVRecordReader with the specified delimiter
CSVRecordReader(int skipNumLines) Skip first n lines
CSVRecordReader(int skipNumLines, char delimiter) Skip lines and use delimiter
CSVRecordReader(int skipNumLines, char delimiter, char quote) Skip lines, use delimiter, and strip quotes
CSVRecordReader(int skipNumLines, String delimiter) Deprecated. This constructor is deprecated; use CSVRecordReader(int, char) or CSVRecordReader(int, char, char)
CSVRecordReader(int skipNumLines, String delimiter, String quote) Deprecated. This constructor is deprecated; use CSVRecordReader(int, char) or CSVRecordReader(int, char, char)

Method Summary

Modifier and Type	Method and Description
boolean	batchesSupported() This method returns true, if next(int) signature is supported by this RecordReader implementation.
boolean	hasNext() Whether there are anymore records
void	initialize(Configuration conf, InputSplit split) Called once at initialization.
List<Record>	loadFromMetaData(List<RecordMetaData> recordMetaDatas) Load multiple records from the given a list of RecordMetaData instances
Record	loadFromMetaData(RecordMetaData recordMetaData) Load a single record from the given RecordMetaData instance Note: that for data that isn't splittable (i.e., text data that needs to be scanned/split), it is more efficient to load multiple records at once using RecordReader.loadFromMetaData(List)
List<Writable>	next() Get the next record
List<List<Writable>>	next(int num) This method will be used, if batchesSupported() returns true.
Record	nextRecord() Similar to RecordReader.next(), but returns a Record object, that may include metadata such as the source of the data
protected void	onLocationOpen(URI location)
protected List<Writable>	parseLine(String line)
protected String	readStringLine()
List<Writable>	record(URI uri, DataInputStream dataInputStream) Load the record from the given DataInputStream Unlike RecordReader.next() the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStream
void	reset() Reset record reader iterator

Methods inherited from class org.datavec.api.records.reader.impl.LineRecordReader

close, closeIfRequired, getConf, getIterator, getLabels, initialize, resetSupported, setConf

Methods inherited from class org.datavec.api.records.reader.BaseRecordReader

getListeners, invokeListeners, setListeners, setListeners

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

skipNumLines

protected int skipNumLines

DEFAULT_DELIMITER

public static final char DEFAULT_DELIMITER

See Also:

Constant Field Values

DEFAULT_QUOTE

public static final char DEFAULT_QUOTE

See Also:

Constant Field Values

SKIP_NUM_LINES

public static final String SKIP_NUM_LINES

DELIMITER

public static final String DELIMITER

QUOTE

public static final String QUOTE

Constructor Detail

CSVRecordReader

public CSVRecordReader(int skipNumLines)

Skip first n lines

Parameters:

skipNumLines - the number of lines to skip

CSVRecordReader

public CSVRecordReader(char delimiter)

Create a CSVRecordReader with the specified delimiter

Parameters:

delimiter - Delimiter character for CSV

CSVRecordReader

public CSVRecordReader(int skipNumLines,
                       char delimiter)

Skip lines and use delimiter

Parameters:

skipNumLines - the number of lines to skip

delimiter - the delimiter

CSVRecordReader

@Deprecated
public CSVRecordReader(int skipNumLines,
                                   String delimiter)

Deprecated. This constructor is deprecated; use CSVRecordReader(int, char) or CSVRecordReader(int, char, char)

Parameters:

skipNumLines - Number of lines to skip

delimiter - Delimiter to use

CSVRecordReader

public CSVRecordReader(int skipNumLines,
                       char delimiter,
                       char quote)

Skip lines, use delimiter, and strip quotes

Parameters:

skipNumLines - the number of lines to skip

delimiter - the delimiter

quote - the quote to strip

CSVRecordReader

@Deprecated
public CSVRecordReader(int skipNumLines,
                                   String delimiter,
                                   String quote)

Deprecated. This constructor is deprecated; use CSVRecordReader(int, char) or CSVRecordReader(int, char, char)

Skip lines, use delimiter, and strip quotes

Parameters:

skipNumLines - the number of lines to skip

delimiter - the delimiter

quote - the quote to strip

CSVRecordReader

public CSVRecordReader()

Method Detail

initialize

public void initialize(Configuration conf,
                       InputSplit split)
                throws IOException,
                       InterruptedException

Description copied from interface: RecordReader

Called once at initialization.

Specified by:

initialize in interface RecordReader

Overrides:

initialize in class LineRecordReader

Parameters:

conf - a configuration for initialization

split - the split that defines the range of records to read

Throws:

IOException

InterruptedException

batchesSupported

public boolean batchesSupported()

Description copied from interface: RecordReader

This method returns true, if next(int) signature is supported by this RecordReader implementation.

Specified by:

batchesSupported in interface RecordReader

Overrides:

batchesSupported in class BaseRecordReader

Returns:

hasNext

public boolean hasNext()

Description copied from interface: RecordReader

Whether there are anymore records

Specified by:

hasNext in interface RecordReader

Overrides:

hasNext in class LineRecordReader

Returns:

next

public List<List<Writable>> next(int num)

Description copied from interface: RecordReader

This method will be used, if batchesSupported() returns true.

Specified by:

next in interface RecordReader

Overrides:

next in class BaseRecordReader

Returns:

next

public List<Writable> next()

Description copied from interface: RecordReader

Get the next record

Specified by:

next in interface RecordReader

Overrides:

next in class LineRecordReader

Returns:

parseLine

protected List<Writable> parseLine(String line)

readStringLine

protected String readStringLine()

nextRecord

public Record nextRecord()

Description copied from interface: RecordReader

Similar to RecordReader.next(), but returns a Record object, that may include metadata such as the source of the data

Specified by:

nextRecord in interface RecordReader

Overrides:

nextRecord in class LineRecordReader

Returns:

next record

loadFromMetaData

public Record loadFromMetaData(RecordMetaData recordMetaData)
                        throws IOException

Description copied from interface: RecordReader

Load a single record from the given RecordMetaData instance
Note: that for data that isn't splittable (i.e., text data that needs to be scanned/split), it is more efficient to load multiple records at once using RecordReader.loadFromMetaData(List)

Specified by:

loadFromMetaData in interface RecordReader

Overrides:

loadFromMetaData in class LineRecordReader

Parameters:

recordMetaData - Metadata for the record that we want to load from

Returns:

Single record for the given RecordMetaData instance

Throws:

IOException - If I/O error occurs during loading

loadFromMetaData

public List<Record> loadFromMetaData(List<RecordMetaData> recordMetaDatas)
                              throws IOException

Description copied from interface: RecordReader

Load multiple records from the given a list of RecordMetaData instances

Specified by:

loadFromMetaData in interface RecordReader

Overrides:

loadFromMetaData in class LineRecordReader

Parameters:

recordMetaDatas - Metadata for the records that we want to load from

Returns:

Multiple records for the given RecordMetaData instances

Throws:

IOException - If I/O error occurs during loading

record

public List<Writable> record(URI uri,
                             DataInputStream dataInputStream)
                      throws IOException

Description copied from interface: RecordReader

Load the record from the given DataInputStream Unlike RecordReader.next() the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStream

Specified by:

record in interface RecordReader

Overrides:

record in class LineRecordReader

Throws:

IOException - if error occurs during reading from the input stream

reset

public void reset()

Description copied from interface: RecordReader

Reset record reader iterator

Specified by:

reset in interface RecordReader

Overrides:

reset in class LineRecordReader

onLocationOpen

protected void onLocationOpen(URI location)

Overrides:

onLocationOpen in class LineRecordReader