org.datavec.api.records.reader.impl.csv

Class CSVMultiSequenceRecordReader

java.lang.Object

org.datavec.api.records.reader.BaseRecordReader

org.datavec.api.records.reader.impl.LineRecordReader

org.datavec.api.records.reader.impl.csv.CSVRecordReader

org.datavec.api.records.reader.impl.csv.CSVMultiSequenceRecordReader

All Implemented Interfaces:

Closeable, Serializable, AutoCloseable, Configurable, RecordReader, SequenceRecordReader

public class CSVMultiSequenceRecordReader
extends CSVRecordReader
implements SequenceRecordReader

CSVMultiSequenceRecordReader: Used to read CSV-format time series (sequence) data where there are multiple independent sequences in each file.
The assumption is that each sequence is separated by some delimiter - for example, a blank line between sequences, or some other line that can be detected by a regex.
Note that the number of columns (i.e., number of lines in the CSV per sequence) must be the same for all sequences.

It supports 3 CSVMultiSequenceRecordReader.Modes:
(a) CONCAT mode: the output is a univariate (single column) sequence with the values from all lines (b) EQUAL_LENGTH: Require that all lines have the exact same number of tokens
(c) PAD: For any shorter lines (fewer tokens), a user-specified padding Writable value will be used to make them the same length as the other sequences

Example:
Input data:

  a,b,c
   1,2

   A,B,C
   D,E,F
 

Output:
(a) CONCAT: two sequences of length 5 and 6 respectively: [a,b,c,1,2] and [A,B,C,D,E,F]
(b) EQUAL_LENGTH: Exception: because lines (a,b,c) and (1,2) have different lengths. If the second line was "1,2,3" instead, the output would be two sequences with 2 columns each, sequence length 3: [[a,b,c],[1,2,3]] and [[A,B,C],[D,E,F]]
(c) PAD: two sequences with 2 columns each, sequence length 3: [[a,b,c],[1,2,PAD]] and [[A,B,C],[D,E,F]], where "PAD" is a user-specified padding value

Note that the user has to specify a sequence separator regex: for "sequences are separated by an empty line" use "^$"

Author:

Alex Black

See Also:

CSVLineSequenceRecordReader for the edge case - a univariate version, Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CSVMultiSequenceRecordReader.Mode

Field Summary

Fields inherited from class org.datavec.api.records.reader.impl.csv.CSVRecordReader

DEFAULT_DELIMITER, DEFAULT_QUOTE, DELIMITER, QUOTE, SKIP_NUM_LINES, skipNumLines

Fields inherited from class org.datavec.api.records.reader.impl.LineRecordReader

charset, conf, initialized, lineIndex, locations, splitIndex

Fields inherited from class org.datavec.api.records.reader.BaseRecordReader

inputSplit, listeners, streamCreatorFn

Fields inherited from interface org.datavec.api.records.reader.RecordReader

APPEND_LABEL, LABELS, NAME_SPACE

Constructor Summary

Constructors

Constructor and Description
CSVMultiSequenceRecordReader(int skipNumLines, char elementDelimiter, char quote, String sequenceSeparatorRegex, CSVMultiSequenceRecordReader.Mode mode, Writable padValue) Create a sequence reader using the default value for skip lines (0), the default delimiter (',') and the default quote character ('"')
CSVMultiSequenceRecordReader(String sequenceSeparatorRegex, CSVMultiSequenceRecordReader.Mode mode) Create a sequence reader using the default value for skip lines (0), the default delimiter (',') and the default quote character ('"'). Note that this constructor cannot be used with CSVMultiSequenceRecordReader.Mode.PAD as the padding value cannot be specified
CSVMultiSequenceRecordReader(String sequenceSeparatorRegex, CSVMultiSequenceRecordReader.Mode mode, Writable padValue) Create a sequence reader using the default value for skip lines (0), the default delimiter (',') and the default quote character ('"')

Method Summary

Modifier and Type	Method and Description
boolean	batchesSupported() This method returns true, if next(int) signature is supported by this RecordReader implementation.
List<SequenceRecord>	loadSequenceFromMetaData(List<RecordMetaData> recordMetaDatas) Load multiple sequence records from the given a list of RecordMetaData instances
SequenceRecord	loadSequenceFromMetaData(RecordMetaData recordMetaData) Load a single sequence record from the given RecordMetaData instance Note: that for data that isn't splittable (i.e., text data that needs to be scanned/split), it is more efficient to load multiple records at once using SequenceRecordReader.loadSequenceFromMetaData(List)
SequenceRecord	nextSequence() Similar to SequenceRecordReader.sequenceRecord(), but returns a Record object, that may include metadata such as the source of the data
List<List<Writable>>	sequenceRecord() Returns a sequence record.
List<List<Writable>>	sequenceRecord(URI uri, DataInputStream dataInputStream) Load a sequence record from the given DataInputStream Unlike RecordReader.next() the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStream

Methods inherited from class org.datavec.api.records.reader.impl.csv.CSVRecordReader

hasNext, initialize, loadFromMetaData, loadFromMetaData, next, next, nextRecord, onLocationOpen, parseLine, readStringLine, record, reset

Methods inherited from class org.datavec.api.records.reader.impl.LineRecordReader

close, closeIfRequired, getConf, getIterator, getLabels, initialize, resetSupported, setConf

Methods inherited from class org.datavec.api.records.reader.BaseRecordReader

getListeners, invokeListeners, setListeners, setListeners

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.datavec.api.records.reader.RecordReader

getLabels, getListeners, hasNext, initialize, initialize, loadFromMetaData, loadFromMetaData, next, next, nextRecord, record, reset, resetSupported, setListeners, setListeners

Methods inherited from interface java.io.Closeable

close

Methods inherited from interface org.datavec.api.conf.Configurable

getConf, setConf

Constructor Detail

CSVMultiSequenceRecordReader

public CSVMultiSequenceRecordReader(String sequenceSeparatorRegex,
                                    CSVMultiSequenceRecordReader.Mode mode)

Create a sequence reader using the default value for skip lines (0), the default delimiter (',') and the default quote character ('"').
Note that this constructor cannot be used with CSVMultiSequenceRecordReader.Mode.PAD as the padding value cannot be specified

Parameters:

sequenceSeparatorRegex - The sequence separator regex. Use "^$" for "sequences are separated by an empty line

mode - Mode: see CSVMultiSequenceRecordReader javadoc

CSVMultiSequenceRecordReader

public CSVMultiSequenceRecordReader(String sequenceSeparatorRegex,
                                    CSVMultiSequenceRecordReader.Mode mode,
                                    Writable padValue)

Create a sequence reader using the default value for skip lines (0), the default delimiter (',') and the default quote character ('"')

Parameters:

sequenceSeparatorRegex - The sequence separator regex. Use "^$" for "sequences are separated by an empty line

mode - Mode: see CSVMultiSequenceRecordReader javadoc

padValue - Padding value for padding short sequences. Only used/allowable with CSVMultiSequenceRecordReader.Mode.PAD, should be null otherwise

CSVMultiSequenceRecordReader

public CSVMultiSequenceRecordReader(int skipNumLines,
                                    char elementDelimiter,
                                    char quote,
                                    String sequenceSeparatorRegex,
                                    CSVMultiSequenceRecordReader.Mode mode,
                                    Writable padValue)

Create a sequence reader using the default value for skip lines (0), the default delimiter (',') and the default quote character ('"')

Parameters:

skipNumLines - Number of lines to skip

elementDelimiter - Delimiter for elements - i.e., ',' if lines are comma separated

sequenceSeparatorRegex - The sequence separator regex. Use "^$" for "sequences are separated by an empty line

mode - Mode: see CSVMultiSequenceRecordReader javadoc

padValue - Padding value for padding short sequences. Only used/allowable with CSVMultiSequenceRecordReader.Mode.PAD, should be null otherwise

Method Detail

sequenceRecord

public List<List<Writable>> sequenceRecord()

Description copied from interface: SequenceRecordReader

Returns a sequence record.

Specified by:

sequenceRecord in interface SequenceRecordReader

Returns:

a sequence of records

nextSequence

public SequenceRecord nextSequence()

Description copied from interface: SequenceRecordReader

Similar to SequenceRecordReader.sequenceRecord(), but returns a Record object, that may include metadata such as the source of the data

Specified by:

nextSequence in interface SequenceRecordReader

Returns:

next sequence record

sequenceRecord

public List<List<Writable>> sequenceRecord(URI uri,
                                           DataInputStream dataInputStream)
                                    throws IOException

Description copied from interface: SequenceRecordReader

Load a sequence record from the given DataInputStream Unlike RecordReader.next() the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStream

Specified by:

sequenceRecord in interface SequenceRecordReader

Throws:

IOException - if error occurs during reading from the input stream

loadSequenceFromMetaData

public SequenceRecord loadSequenceFromMetaData(RecordMetaData recordMetaData)
                                        throws IOException

Description copied from interface: SequenceRecordReader

Load a single sequence record from the given RecordMetaData instance
Note: that for data that isn't splittable (i.e., text data that needs to be scanned/split), it is more efficient to load multiple records at once using SequenceRecordReader.loadSequenceFromMetaData(List)

Specified by:

loadSequenceFromMetaData in interface SequenceRecordReader

Parameters:

recordMetaData - Metadata for the sequence record that we want to load from

Returns:

Single sequence record for the given RecordMetaData instance

Throws:

IOException - If I/O error occurs during loading

loadSequenceFromMetaData

public List<SequenceRecord> loadSequenceFromMetaData(List<RecordMetaData> recordMetaDatas)
                                              throws IOException

Description copied from interface: SequenceRecordReader

Load multiple sequence records from the given a list of RecordMetaData instances

Specified by:

loadSequenceFromMetaData in interface SequenceRecordReader

Parameters:

recordMetaDatas - Metadata for the records that we want to load from

Returns:

Multiple sequence record for the given RecordMetaData instances

Throws:

IOException - If I/O error occurs during loading

batchesSupported

public boolean batchesSupported()

Description copied from interface: RecordReader

This method returns true, if next(int) signature is supported by this RecordReader implementation.

Specified by:

batchesSupported in interface RecordReader

Overrides:

batchesSupported in class CSVRecordReader

Returns: