org.datavec.api.transform.transform.string

Class StringListToCountsNDArrayTransform

java.lang.Object

org.datavec.api.transform.transform.BaseTransform

org.datavec.api.transform.transform.string.StringListToCountsNDArrayTransform

All Implemented Interfaces:

Serializable, ColumnOp, Transform

Direct Known Subclasses:

StringListToIndicesNDArrayTransform

public class StringListToCountsNDArrayTransform
extends BaseTransform

Converts String column into a bag-of-words (BOW) represented as an NDArray of "counts."
Note that the original column is removed in the process

Author:

dave@skymind.io

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	binary
protected int	columnIdx
protected String	columnName
protected String	delimiter
protected boolean	ignoreUnknown
protected Map<String,Integer>	map
protected String	newColumnName
protected List<String>	vocabulary

Fields inherited from class org.datavec.api.transform.transform.BaseTransform

inputSchema

Constructor Summary

Constructors

Constructor and Description
StringListToCountsNDArrayTransform(String columnName, List<String> vocabulary, String delimiter, boolean binary, boolean ignoreUnknown)
StringListToCountsNDArrayTransform(String columnName, String newColumnName, List<String> vocabulary, String delimiter, boolean binary, boolean ignoreUnknown)

Method Summary

Modifier and Type	Method and Description
String	columnName() Returns a singular column name this op is meant to run on
String[]	columnNames() Returns column names this op is meant to run on
protected Collection<Integer>	getIndices(String text)
protected org.nd4j.linalg.api.ndarray.INDArray	makeBOWNDArray(Collection<Integer> indices)
List<Writable>	map(List<Writable> writables) Transform a writable in to another writable
Object	map(Object input) Transform an object in to another object
Object	mapSequence(Object sequence) Transform a sequence
String	outputColumnName() The output column name after the operation has been applied
String[]	outputColumnNames() The output column names This will often be the same as the input
static List<String>	readVocabFromFile(String path)
void	setInputSchema(Schema inputSchema) Set the input schema.
String	toString()
Schema	transform(Schema inputSchema) Get the output schema for this transformation, given an input schema

Methods inherited from class org.datavec.api.transform.transform.BaseTransform

getInputSchema, mapSequence

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

columnName

protected final String columnName

newColumnName

protected final String newColumnName

vocabulary

protected final List<String> vocabulary

delimiter

protected final String delimiter

binary

protected final boolean binary

ignoreUnknown

protected final boolean ignoreUnknown

map

protected final Map<String,Integer> map

columnIdx

protected int columnIdx

Constructor Detail

StringListToCountsNDArrayTransform

public StringListToCountsNDArrayTransform(String columnName,
                                          List<String> vocabulary,
                                          String delimiter,
                                          boolean binary,
                                          boolean ignoreUnknown)

Parameters:

columnName - The name of the column to convert

vocabulary - The possible tokens that may be present.

delimiter - The delimiter for the Strings to convert

ignoreUnknown - Whether to ignore unknown tokens

StringListToCountsNDArrayTransform

public StringListToCountsNDArrayTransform(String columnName,
                                          String newColumnName,
                                          List<String> vocabulary,
                                          String delimiter,
                                          boolean binary,
                                          boolean ignoreUnknown)

Parameters:

columnName - The name of the column to convert

vocabulary - The possible tokens that may be present.

delimiter - The delimiter for the Strings to convert

ignoreUnknown - Whether to ignore unknown tokens

Method Detail

readVocabFromFile

public static List<String> readVocabFromFile(String path)
                                      throws IOException

Throws:

IOException

transform

public Schema transform(Schema inputSchema)

Description copied from interface: ColumnOp

Get the output schema for this transformation, given an input schema

setInputSchema

public void setInputSchema(Schema inputSchema)

Description copied from interface: ColumnOp

Set the input schema.

Specified by:

setInputSchema in interface ColumnOp

Overrides:

setInputSchema in class BaseTransform

toString

public String toString()

Specified by:

toString in class BaseTransform

getIndices

protected Collection<Integer> getIndices(String text)

makeBOWNDArray

protected org.nd4j.linalg.api.ndarray.INDArray makeBOWNDArray(Collection<Integer> indices)

map

public List<Writable> map(List<Writable> writables)

Description copied from interface: Transform

Transform a writable in to another writable

Parameters:

writables - the record to transform

Returns:

the transformed writable

map

public Object map(Object input)

Transform an object in to another object

Parameters:

input - the record to transform

Returns:

the transformed writable

mapSequence

public Object mapSequence(Object sequence)

Transform a sequence

Parameters:

sequence -

outputColumnName

public String outputColumnName()

The output column name after the operation has been applied

Returns:

the output column name

outputColumnNames

public String[] outputColumnNames()

The output column names This will often be the same as the input

Returns:

the output column names

columnNames

public String[] columnNames()

Returns column names this op is meant to run on

Returns:

columnName

public String columnName()

Returns a singular column name this op is meant to run on

Returns: