org.bytedeco.pytorch

Class ChunkDataset

java.lang.Object

org.bytedeco.javacpp.Pointer

org.bytedeco.pytorch.ChunkBatchDataset

org.bytedeco.pytorch.ChunkStatefulDataset

org.bytedeco.pytorch.ChunkDataset

All Implemented Interfaces:

AutoCloseable

@Name(value="torch::data::datasets::ChunkDataset<JavaCPP_torch_0003a_0003adata_0003a_0003adatasets_0003a_0003aChunkDataReader_0003ctorch_0003a_0003adata_0003a_0003aExample_0003ctorch_0003a_0003aTensor_0002ctorch_0003a_0003aTensor_0003e_0002cstd_0003a_0003avector_0003ctorch_0003a_0003adata_0003a_0003aExample_0003ctorch_0003a_0003aTensor_0002ctorch_0003a_0003aTensor_0003e_00020_0003e_00020_0003e,torch::data::samplers::RandomSampler,torch::data::samplers::RandomSampler>")
 @NoOffset
 @Properties(inherit=torch.class)
public class ChunkDataset
extends ChunkStatefulDataset

A stateful dataset that support hierarchical sampling and prefetching of entre chunks. Unlike regular dataset, chunk dataset require two samplers to operate and keeps an internal state. ChunkSampler selects, which chunk to load next, while the ExampleSampler determins the order of Examples that are returned in each get_batch call. The hierarchical sampling approach used here is inspired by this paper http://martin.zinkevich.org/publications/nips2010.pdf

Nested Class Summary

Nested classes/interfaces inherited from class org.bytedeco.javacpp.Pointer

Pointer.CustomDeallocator, Pointer.Deallocator, Pointer.NativeDeallocator, Pointer.ReferenceCounter

Field Summary

Fields inherited from class org.bytedeco.pytorch.ChunkBatchDataset

is_stateful

Fields inherited from class org.bytedeco.javacpp.Pointer

address, capacity, limit, position

Constructor Summary

Constructors

Constructor and Description
ChunkDataset(ChunkDataReader chunk_reader, RandomSampler chunk_sampler, RandomSampler example_sampler, ChunkDatasetOptions options)
ChunkDataset(ChunkDataReader chunk_reader, RandomSampler chunk_sampler, RandomSampler example_sampler, ChunkDatasetOptions options, Pointer preprocessing_policy)
ChunkDataset(Pointer p) Pointer cast constructor.

Method Summary

Modifier and Type	Method and Description
RandomSampler	chunk_sampler()
ExampleVectorOptional	get_batch() Helper method around get_batch as batch_size is not strictly necessary
ExampleVectorOptional	get_batch(long batch_size) Default get_batch method of BatchDataset.
void	load(InputArchive archive) Deserializes the statefulDataset's state from the archive.
void	reset() This will clear any internal state and starts the internal prefetching mechanism for the chunk dataset.
void	save(OutputArchive archive) Saves the statefulDataset's state to OutputArchive.
SizeTOptional	size() size is not used for chunk dataset.

Methods inherited from class org.bytedeco.pytorch.ChunkBatchDataset

is_stateful

Methods inherited from class org.bytedeco.javacpp.Pointer

address, asBuffer, asByteBuffer, availablePhysicalBytes, calloc, capacity, capacity, close, deallocate, deallocate, deallocateReferences, deallocator, deallocator, equals, fill, formatBytes, free, getDirectBufferAddress, getPointer, getPointer, getPointer, getPointer, hashCode, interruptDeallocatorThread, isNull, isNull, limit, limit, malloc, maxBytes, maxPhysicalBytes, memchr, memcmp, memcpy, memmove, memset, offsetAddress, offsetof, offsetof, parseBytes, physicalBytes, physicalBytesInaccurate, position, position, put, realloc, referenceCount, releaseReference, retainReference, setNull, sizeof, sizeof, toString, totalBytes, totalCount, totalPhysicalBytes, withDeallocator, zero

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

ChunkDataset

public ChunkDataset(Pointer p)

Pointer cast constructor. Invokes Pointer(Pointer).

ChunkDataset

public ChunkDataset(ChunkDataReader chunk_reader,
                    RandomSampler chunk_sampler,
                    RandomSampler example_sampler,
                    ChunkDatasetOptions options)

ChunkDataset

public ChunkDataset(ChunkDataReader chunk_reader,
                    RandomSampler chunk_sampler,
                    RandomSampler example_sampler,
                    ChunkDatasetOptions options,
                    Pointer preprocessing_policy)

Method Detail

get_batch

@ByVal
public ExampleVectorOptional get_batch(@Cast(value="size_t")
                                              long batch_size)

Default get_batch method of BatchDataset. This method returns Example batches created from the preloaded chunks. The implemenation is dataset agnostic and does not need overriding in different chunk datasets.

Overrides:

get_batch in class ChunkBatchDataset

get_batch

@ByVal
public ExampleVectorOptional get_batch()

Helper method around get_batch as batch_size is not strictly necessary

reset

public void reset()

This will clear any internal state and starts the internal prefetching mechanism for the chunk dataset.

Overrides:

reset in class ChunkStatefulDataset

size

@ByVal
public SizeTOptional size()

size is not used for chunk dataset.

Overrides:

size in class ChunkBatchDataset

chunk_sampler

@Cast(value="torch::data::datasets::ChunkDataset<JavaCPP_torch_0003a_0003adata_0003a_0003adatasets_0003a_0003aChunkDataReader_0003ctorch_0003a_0003adata_0003a_0003aExample_0003ctorch_0003a_0003aTensor_0002ctorch_0003a_0003aTensor_0003e_0002cstd_0003a_0003avector_0003ctorch_0003a_0003adata_0003a_0003aExample_0003ctorch_0003a_0003aTensor_0002ctorch_0003a_0003aTensor_0003e_00020_0003e_00020_0003e,torch::data::samplers::RandomSampler,torch::data::samplers::RandomSampler>::ChunkSamplerType*")
 @ByRef
public RandomSampler chunk_sampler()

save

public void save(@ByRef
                 OutputArchive archive)

Description copied from class: ChunkStatefulDataset

Saves the statefulDataset's state to OutputArchive.

Overrides:

save in class ChunkStatefulDataset

load

public void load(@ByRef
                 InputArchive archive)

Description copied from class: ChunkStatefulDataset

Deserializes the statefulDataset's state from the archive.

Overrides:

load in class ChunkStatefulDataset