Package ai.djl.training.dataset

Class ArrayDataset

java.lang.Object

ai.djl.training.dataset.RandomAccessDataset

ai.djl.training.dataset.ArrayDataset

All Implemented Interfaces:

Dataset

public class ArrayDataset
extends RandomAccessDataset

ArrayDataset is an implementation of RandomAccessDataset that consist entirely of large NDArrays. It is recommended only for datasets small enough to fit in memory that come in array formats. Otherwise, consider directly using the RandomAccessDataset instead.

There can be multiple data and label NDArrays within the dataset. Each sample will be retrieved by indexing each NDArray along the first dimension.

The following is an example of how to use ArrayDataset:

     ArrayDataset dataset = new ArrayDataset.Builder()
                              .setData(data1, data2)
                              .optLabels(labels1, labels2, labels3)
                              .setSampling(20, false)
                              .build();
 

Suppose you get a Batch from trainer.iterateDataset(dataset) or dataset.getData(manager). In the data of this batch, it will be an NDList with one NDArray for each data input. In this case, it would be 2 arrays. Similarly, the labels would have 3 arrays.

See Also:

Dataset

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	ArrayDataset.Builder	The Builder to construct an ArrayDataset.

Nested classes/interfaces inherited from class ai.djl.training.dataset.RandomAccessDataset

RandomAccessDataset.BaseBuilder<T extends RandomAccessDataset.BaseBuilder<T>>

Nested classes/interfaces inherited from interface ai.djl.training.dataset.Dataset

Dataset.Usage

Field Summary

Fields

Modifier and Type	Field	Description
protected NDArray[]	data
protected NDArray[]	labels

Fields inherited from class ai.djl.training.dataset.RandomAccessDataset

dataBatchifier, device, labelBatchifier, limit, pipeline, prefetchNumber, sampler, targetPipeline

Constructor Summary

Constructors

Constructor	Description
ArrayDataset(RandomAccessDataset.BaseBuilder<?> builder)	Creates a new instance of ArrayDataset with the arguments in ArrayDataset.Builder.

Method Summary

Modifier and Type	Method	Description
protected long	availableSize()	Returns the number of records available to be read in this Dataset.
Record	get(NDManager manager, long index)	Gets the Record for the given index from the dataset.
void	prepare(ai.djl.util.Progress progress)	Prepares the dataset for use with tracked progress.

Methods inherited from class ai.djl.training.dataset.RandomAccessDataset

getData, getData, getData, getData, randomSplit, size, subDataset, toArray

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface ai.djl.training.dataset.Dataset

prepare

Field Detail

data

protected NDArray[] data

labels

protected NDArray[] labels

Constructor Detail

ArrayDataset

public ArrayDataset(RandomAccessDataset.BaseBuilder<?> builder)

Creates a new instance of ArrayDataset with the arguments in ArrayDataset.Builder.

Parameters:

builder - a builder with the required arguments

Method Detail

availableSize

protected long availableSize()

Returns the number of records available to be read in this Dataset.

Specified by:

availableSize in class RandomAccessDataset

Returns:

the number of records available to be read in this Dataset

get

public Record get(NDManager manager,
                  long index)

Gets the Record for the given index from the dataset.

Specified by:

get in class RandomAccessDataset

Parameters:

manager - the manager used to create the arrays

index - the index of the requested data item

Returns:

a Record that contains the data and label of the requested data item

prepare

public void prepare(ai.djl.util.Progress progress)
             throws java.io.IOException

Prepares the dataset for use with tracked progress.

Parameters:

progress - the progress tracker

Throws:

java.io.IOException - for various exceptions depending on the dataset