org.datavec.api.records.reader.impl.regex

Class RegexSequenceRecordReader

java.lang.Object

org.datavec.api.records.reader.BaseRecordReader

org.datavec.api.records.reader.impl.FileRecordReader

org.datavec.api.records.reader.impl.regex.RegexSequenceRecordReader

All Implemented Interfaces:

Closeable, Serializable, AutoCloseable, Configurable, RecordReader, SequenceRecordReader

public class RegexSequenceRecordReader
extends FileRecordReader
implements SequenceRecordReader

RegexSequenceRecordReader: Read an entire file (as a sequence), one line at a time and split each line into fields using a regex. Specifically, we are using Pattern and Matcher to do the splitting into groups Example: Data in format "2016-01-01 23:59:59.001 1 DEBUG First entry message!"
using regex String "(\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\.\\d{3}) (\\d+) ([A-Z]+) (.*)"
would be split into 4 Text writables: ["2016-01-01 23:59:59.001", "1", "DEBUG", "First entry message!"]
Note: RegexSequenceRecordReader supports multiple error handling modes, via RegexSequenceRecordReader.LineErrorHandling. Invalid lines that don't match the provided regex can result in an exception (FailOnInvalid), can be skipped silently (SkipInvalid), or skip invalid but log a warning (SkipInvalidWithWarning)

Author:

Alex Black

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	RegexSequenceRecordReader.LineErrorHandling Error handling mode: How should invalid lines (i.e., those that don't match the provided regex) be handled? FailOnInvalid: Throw an IllegalStateException when an invalid line is found SkipInvalid: Skip invalid lines (quietly, with no warning) SkipInvalidWithWarning: Skip invalid lines, but log a warning

Field Summary

Fields

Modifier and Type	Field and Description
static Charset	DEFAULT_CHARSET
static RegexSequenceRecordReader.LineErrorHandling	DEFAULT_ERROR_HANDLING
static org.slf4j.Logger	LOG
static String	SKIP_NUM_LINES

Fields inherited from class org.datavec.api.records.reader.impl.FileRecordReader

appendLabel, conf, currentUri, labels, locationsIterator

Fields inherited from class org.datavec.api.records.reader.BaseRecordReader

inputSplit, listeners, streamCreatorFn

Fields inherited from interface org.datavec.api.records.reader.RecordReader

APPEND_LABEL, LABELS, NAME_SPACE

Constructor Summary

Constructors

Constructor and Description
RegexSequenceRecordReader(String regex, int skipNumLines)
RegexSequenceRecordReader(String regex, int skipNumLines, Charset encoding, RegexSequenceRecordReader.LineErrorHandling errorHandling)

Method Summary

Modifier and Type	Method and Description
void	initialize(Configuration conf, InputSplit split) Called once at initialization.
List<SequenceRecord>	loadSequenceFromMetaData(List<RecordMetaData> recordMetaDatas) Load multiple sequence records from the given a list of RecordMetaData instances
SequenceRecord	loadSequenceFromMetaData(RecordMetaData recordMetaData) Load a single sequence record from the given RecordMetaData instance Note: that for data that isn't splittable (i.e., text data that needs to be scanned/split), it is more efficient to load multiple records at once using SequenceRecordReader.loadSequenceFromMetaData(List)
SequenceRecord	nextSequence() Similar to SequenceRecordReader.sequenceRecord(), but returns a Record object, that may include metadata such as the source of the data
void	reset() Reset record reader iterator
List<List<Writable>>	sequenceRecord() Returns a sequence record.
List<List<Writable>>	sequenceRecord(URI uri, DataInputStream dataInputStream) Load a sequence record from the given DataInputStream Unlike RecordReader.next() the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStream

Methods inherited from class org.datavec.api.records.reader.impl.FileRecordReader

close, doInitialize, getConf, getCurrentLabel, getLabel, getLabels, hasNext, initialize, loadFromMetaData, loadFromMetaData, next, next, nextRecord, record, resetSupported, setConf, setLabels

Methods inherited from class org.datavec.api.records.reader.BaseRecordReader

batchesSupported, getListeners, invokeListeners, setListeners, setListeners

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.datavec.api.records.reader.RecordReader

batchesSupported, getLabels, getListeners, hasNext, initialize, loadFromMetaData, loadFromMetaData, next, next, nextRecord, record, resetSupported, setListeners, setListeners

Methods inherited from interface java.io.Closeable

close

Methods inherited from interface org.datavec.api.conf.Configurable

getConf, setConf

Field Detail

SKIP_NUM_LINES

public static final String SKIP_NUM_LINES

DEFAULT_CHARSET

public static final Charset DEFAULT_CHARSET

DEFAULT_ERROR_HANDLING

public static final RegexSequenceRecordReader.LineErrorHandling DEFAULT_ERROR_HANDLING

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

RegexSequenceRecordReader

public RegexSequenceRecordReader(String regex,
                                 int skipNumLines)

RegexSequenceRecordReader

public RegexSequenceRecordReader(String regex,
                                 int skipNumLines,
                                 Charset encoding,
                                 RegexSequenceRecordReader.LineErrorHandling errorHandling)

Method Detail

initialize

public void initialize(Configuration conf,
                       InputSplit split)
                throws IOException,
                       InterruptedException

Description copied from interface: RecordReader

Called once at initialization.

Specified by:

initialize in interface RecordReader

Overrides:

initialize in class FileRecordReader

Parameters:

conf - a configuration for initialization

split - the split that defines the range of records to read

Throws:

IOException

InterruptedException

sequenceRecord

public List<List<Writable>> sequenceRecord()

Description copied from interface: SequenceRecordReader

Returns a sequence record.

Specified by:

sequenceRecord in interface SequenceRecordReader

Returns:

a sequence of records

sequenceRecord

public List<List<Writable>> sequenceRecord(URI uri,
                                           DataInputStream dataInputStream)
                                    throws IOException

Description copied from interface: SequenceRecordReader

Load a sequence record from the given DataInputStream Unlike RecordReader.next() the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStream

Specified by:

sequenceRecord in interface SequenceRecordReader

Throws:

IOException - if error occurs during reading from the input stream

reset

public void reset()

Description copied from interface: RecordReader

Reset record reader iterator

Specified by:

reset in interface RecordReader

Overrides:

reset in class FileRecordReader

nextSequence

public SequenceRecord nextSequence()

Description copied from interface: SequenceRecordReader

Similar to SequenceRecordReader.sequenceRecord(), but returns a Record object, that may include metadata such as the source of the data

Specified by:

nextSequence in interface SequenceRecordReader

Returns:

next sequence record

loadSequenceFromMetaData

public SequenceRecord loadSequenceFromMetaData(RecordMetaData recordMetaData)
                                        throws IOException

Description copied from interface: SequenceRecordReader

Load a single sequence record from the given RecordMetaData instance
Note: that for data that isn't splittable (i.e., text data that needs to be scanned/split), it is more efficient to load multiple records at once using SequenceRecordReader.loadSequenceFromMetaData(List)

Specified by:

loadSequenceFromMetaData in interface SequenceRecordReader

Parameters:

recordMetaData - Metadata for the sequence record that we want to load from

Returns:

Single sequence record for the given RecordMetaData instance

Throws:

IOException - If I/O error occurs during loading

loadSequenceFromMetaData

public List<SequenceRecord> loadSequenceFromMetaData(List<RecordMetaData> recordMetaDatas)
                                              throws IOException

Description copied from interface: SequenceRecordReader

Load multiple sequence records from the given a list of RecordMetaData instances

Specified by:

loadSequenceFromMetaData in interface SequenceRecordReader

Parameters:

recordMetaDatas - Metadata for the records that we want to load from

Returns:

Multiple sequence record for the given RecordMetaData instances

Throws:

IOException - If I/O error occurs during loading