org.datavec.api.split

Class TransformSplit

java.lang.Object

org.datavec.api.split.BaseInputSplit

org.datavec.api.split.TransformSplit

All Implemented Interfaces:

InputSplit

public class TransformSplit
extends BaseInputSplit

InputSplit implementation that maps the URIs of a given BaseInputSplit to new URIs. Useful when features and labels are in different files sharing a common naming scheme, and the name of the output file can be determined given the name of the input file.

Author:

Ede Meijer

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	TransformSplit.URITransform

Field Summary

Fields inherited from class org.datavec.api.split.BaseInputSplit

iterationOrder, length, uriStrings

Constructor Summary

Constructors

Constructor and Description
TransformSplit(@NonNull BaseInputSplit sourceSplit, @NonNull TransformSplit.URITransform transform) Apply a given transformation to the raw URI objects

Method Summary

Modifier and Type	Method and Description
void	bootStrapForWrite() Bootstrap this input split for writing.
boolean	needsBootstrapForWrite() Returns true if this InputSplit needs bootstrapping for writing.
static TransformSplit	ofSearchReplace(@NonNull BaseInputSplit sourceSplit, @NonNull String search, @NonNull String replace) Static factory method, replace the string version of the URI with a simple search-replace pair
InputStream	openInputStreamFor(String location) Open an InputStream for the given location.
OutputStream	openOutputStreamFor(String location) Open an OutputStream for the given location.
void	reset() Reset the InputSplit without reinitializing it from scratch.
boolean	resetSupported()
void	updateSplitLocations(boolean reset) Refreshes the split locations if needed in memory.

Methods inherited from class org.datavec.api.split.BaseInputSplit

addNewLocation, addNewLocation, canWriteToLocation, length, locations, locationsIterator, locationsPathIterator, sample

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TransformSplit

public TransformSplit(@NonNull
                      @NonNull BaseInputSplit sourceSplit,
                      @NonNull
                      @NonNull TransformSplit.URITransform transform)
               throws URISyntaxException

Apply a given transformation to the raw URI objects

Parameters:

sourceSplit - the split with URIs to transform

transform - transform operation that returns a new URI based on an input URI

Throws:

URISyntaxException - thrown if the transformed URI is malformed

Method Detail

ofSearchReplace

public static TransformSplit ofSearchReplace(@NonNull
                                             @NonNull BaseInputSplit sourceSplit,
                                             @NonNull
                                             @NonNull String search,
                                             @NonNull
                                             @NonNull String replace)
                                      throws URISyntaxException

Static factory method, replace the string version of the URI with a simple search-replace pair

Parameters:

sourceSplit - the split with URIs to transform

search - the string to search

replace - the string to replace with

Throws:

URISyntaxException - thrown if the transformed URI is malformed

updateSplitLocations

public void updateSplitLocations(boolean reset)

Description copied from interface: InputSplit

Refreshes the split locations if needed in memory. (Think a few file gets added)

needsBootstrapForWrite

public boolean needsBootstrapForWrite()

Description copied from interface: InputSplit

Returns true if this InputSplit needs bootstrapping for writing. A simple example of needing bootstrapping is for FileSplit where there is only a directory existing, but no file to write to

Returns:

true if this input split needs bootstrapping for writing to or not

bootStrapForWrite

public void bootStrapForWrite()

Description copied from interface: InputSplit

Bootstrap this input split for writing. This is for use with RecordWriter

openOutputStreamFor

public OutputStream openOutputStreamFor(String location)
                                 throws Exception

Description copied from interface: InputSplit

Open an OutputStream for the given location. Note that the user is responsible for closing the associated output stream.

Parameters:

location - the location to open the output stream for

Returns:

the output input stream

Throws:

Exception

openInputStreamFor

public InputStream openInputStreamFor(String location)
                               throws Exception

Description copied from interface: InputSplit

Open an InputStream for the given location. Note that the user is responsible for closing the associated input stream.

Parameters:

location - the location to open the input stream for

Returns:

the opened input stream

Throws:

Exception

reset

public void reset()

Description copied from interface: InputSplit

Reset the InputSplit without reinitializing it from scratch. In many cases, this is a no-op. For InputSplits that have randomization: reset should shuffle the order.

resetSupported

public boolean resetSupported()

Returns:

True if the reset() method is supported (or is a no-op), false otherwise. If false is returned, reset() may throw an exception