Type Members

1.  abstract class BaseDenseTensorIterator extends TensorIterator

   The base for custom iterators over dense tensors of ranks 1+.

   The base for custom iterators over dense tensors of ranks 1+. Encapsulates the logic for traversing nested collections, so that it can be shared between e.g. Avro and Spark representations.

   Examples:

   - A regular tensor of the shape [2, 3]. Numbering of positions: [ [0, 1, 2], [3, 4, 5] ]. The sequence of indices: [(0, 0), (0, 1), (0, 2), (1, 0), (1, 1), (1, 2)].

   - A ragged tensor of rank 2. Numbering of positions: [ [0, 1], [], [2, 3, 4] ] The sequence of indices: [(0, 0), (0, 1), (2, 0), (2, 1), (2, 2)].

   Subclasses are expected to cache their current path through the tree of nested collections. There are as many collections in the cache as the rank of the tensor, starting from the root and ending with the level containing leaves/values: cache[0] = root cache[1] = cache[0][_indices[0]] ... cache[rank] = cache[rank-1][_indices[rank-1]] = current value

2.  class DenseTensorList extends AbstractList[AnyRef]

   A list that is backed by a dense tensor.

   A list that is backed by a dense tensor. Exposes only the values of the tensor in their natural order. Warning: is only reasonably efficient for a single-threaded sequential access pattern. Random access is quite inefficient.

3.  trait FeatureDeserializer extends AnyRef

   A converter for a single feature from Spark value to a TensorData of a specific TensorType.

4.  final class InternalFeaturizedDatasetMetadataUtils extends AnyRef

   Utility methods for working with FeaturizedDatasetMetadata.

   Utility methods for working with FeaturizedDatasetMetadata.

   Annotations

   @InternalApi()

5.  final class SchemaMetadataUtils extends AnyRef

   Internal utilities for handling FDS metadata in Avro schema.

6.  final class SparkDeserializerFactory extends AnyRef

   A converter from FDS Spark objects to TensorData.

   A converter from FDS Spark objects to TensorData. Instead of creating a new memory copy, a thin wrapper (view) around the Spark structure is created. This reduces GC pressure, but some operations (e.g., cardinality or isEmpty) are more expensive than usual.

   For working with individual feature values, use #getFeatureDeserializer.

   NOTE that this class directly operates with scala.collection.Seq values coming from Spark DataFrame. An alternative would be converting those to java.util.List, but given the performance requirements of this class, that was not chosen.