org.datavec.api.records.reader.impl.jackson

Class JacksonLineRecordReader

java.lang.Object

org.datavec.api.records.reader.BaseRecordReader

org.datavec.api.records.reader.impl.LineRecordReader

org.datavec.api.records.reader.impl.jackson.JacksonLineRecordReader

All Implemented Interfaces:

Closeable, Serializable, AutoCloseable, Configurable, RecordReader

public class JacksonLineRecordReader
extends LineRecordReader

JacksonLineRecordReader will read a single file line-by-line when .next() is
called. It uses Jackson ObjectMapper and FieldSelection to read the fields in
each line.

Each line should be a valid JSON entry without separator at the end. This is similar
to other readers and follows Hadoop convention. Hadoop and Spark use this format to
to make sure splits work properly in a cluster environment. For those new to Hadoop
file format convention, the reason is a large file can be split into chunks and
sent to different nodes in a cluster. If a record spanned multiple lines, split
might not get the complete record, which will result in runtime errors and calculation
errors. Where and how a job splits a file varies depending on the job configuration
and cluster size.

A couple of important notes. The reader doesn't automatically create labels for each
record like JacksonRecordReader. JacksonRecordReader uses the folder name for the label
at runtime. It assumes a top level folder has multiple subfolders. The labels are the subfolder names.

In the case of JacksonLineRecordReader, you have to provide the labels in the configuration
for the training. Please look at the examples in dl4j-examples repository on how to provide labels.

Author:

peter

See Also:

Serialized Form

Field Summary

Fields inherited from class org.datavec.api.records.reader.impl.LineRecordReader

charset, conf, initialized, lineIndex, locations, splitIndex

Fields inherited from class org.datavec.api.records.reader.BaseRecordReader

inputSplit, listeners, streamCreatorFn

Fields inherited from interface org.datavec.api.records.reader.RecordReader

APPEND_LABEL, LABELS, NAME_SPACE

Constructor Summary

Constructors

Constructor and Description
JacksonLineRecordReader(FieldSelection selection, org.nd4j.shade.jackson.databind.ObjectMapper mapper)

Method Summary

Modifier and Type	Method and Description
List<String>	getLabels() List of label strings
List<Writable>	next() Get the next record
protected List<Writable>	parseLine(String line)

Methods inherited from class org.datavec.api.records.reader.impl.LineRecordReader

close, closeIfRequired, getConf, getIterator, hasNext, initialize, initialize, loadFromMetaData, loadFromMetaData, nextRecord, onLocationOpen, record, reset, resetSupported, setConf

Methods inherited from class org.datavec.api.records.reader.BaseRecordReader

batchesSupported, getListeners, invokeListeners, next, setListeners, setListeners

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

JacksonLineRecordReader

public JacksonLineRecordReader(FieldSelection selection,
                               org.nd4j.shade.jackson.databind.ObjectMapper mapper)

Method Detail

next

public List<Writable> next()

Description copied from interface: RecordReader

Get the next record

Specified by:

next in interface RecordReader

Overrides:

next in class LineRecordReader

Returns:

parseLine

protected List<Writable> parseLine(String line)

getLabels

public List<String> getLabels()

Description copied from interface: RecordReader

List of label strings

Specified by:

getLabels in interface RecordReader

Overrides:

getLabels in class LineRecordReader

Returns: