Type Members

1.  final class ByteArray extends AnyRef
2.  final class CalendarInterval extends Serializable with Comparable[CalendarInterval]

   The class representing calendar intervals.

   The class representing calendar intervals. The calendar interval is stored internally in three components:

   • an integer value representing the number of months in this interval,
   • an integer value representing the number of days in this interval,
   • a long value representing the number of microseconds in this interval.

   The months and days are not units of time with a constant length (unlike hours, seconds), so they are two separated fields from microseconds. One month may be equal to 28, 29, 30 or 31 days and one day may be equal to 23, 24 or 25 hours (daylight saving).

   Annotations

   @Unstable()

   Since

   3.0.0

3.  final class UTF8String extends Comparable[UTF8String] with Externalizable with KryoSerializable with Cloneable

   A UTF-8 String for internal Spark use.

   A UTF-8 String for internal Spark use.

   A String encoded in UTF-8 as an Array[Byte], which can be used for comparison, search, see http://en.wikipedia.org/wiki/UTF-8 for details.

   Note: This is not designed for general use cases, should not be used outside SQL.

4.  class VariantVal extends Serializable

   The physical data representation of org.apache.spark.sql.types.VariantType that represents a semi-structured value.

   The physical data representation of org.apache.spark.sql.types.VariantType that represents a semi-structured value. It consists of two binary values: VariantVal#value and VariantVal#metadata. The value encodes types and values, but not field names. The metadata currently contains a version flag and a list of field names. We can extend/modify the detailed binary format given the version flag.

   A VariantVal can be produced by casting another value into the Variant type or parsing a JSON string in the org.apache.spark.sql.catalyst.expressions.variant.ParseJson expression. We can extract a path consisting of field names and array indices from it, cast it into a concrete data type, or rebuild a JSON string from it.

   The storage layout of this class in org.apache.spark.sql.catalyst.expressions.UnsafeRow and org.apache.spark.sql.catalyst.expressions.UnsafeArrayData is: the fixed-size part is a long value "offsetAndSize". The upper 32 bits is the offset that points to the start position of the actual binary content. The lower 32 bits is the total length of the binary content. The binary content contains: 4 bytes representing the length of VariantVal#value, content of VariantVal#value, content of VariantVal#metadata. This is an internal and transient format and can be modified at any time.