case object SparkShreddingUtils extends Product with Serializable
- Alphabetic
- By Inheritance
- SparkShreddingUtils
- Serializable
- Product
- Equals
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Type Members
- class SparkShreddedResult extends ShreddedResult
- class SparkShreddedResultBuilder extends ShreddedResultBuilder
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- val MetadataFieldName: String
- val TypedValueFieldName: String
- val VARIANT_WRITE_SHREDDING_KEY: String
- val VariantValueFieldName: String
- def addWriteShreddingMetadata(schema: StructType): StructType
Given a schema that represents a valid shredding schema (e.g.
Given a schema that represents a valid shredding schema (e.g. constructed by SparkShreddingUtils.variantShreddingSchema), add metadata to the top-level fields to mark it as a shredding schema for writers.
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def assembleVariant(row: InternalRow, schema: VariantSchema): VariantVal
- def assembleVariantBatch(input: WritableColumnVector, output: WritableColumnVector, schema: VariantSchema): Unit
- def assembleVariantStruct(inputRow: InternalRow, schema: VariantSchema, fields: Array[FieldToExtract]): InternalRow
- def assembleVariantStructBatch(input: WritableColumnVector, output: WritableColumnVector, schema: VariantSchema, fields: Array[FieldToExtract]): Unit
- def buildVariantSchema(schema: DataType): VariantSchema
- def castShredded(v: Variant, schema: VariantSchema): InternalRow
Converts an input variant into shredded components.
Converts an input variant into shredded components. Returns the shredded result.
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def getFieldsToExtract(targetType: DataType, inputSchema: VariantSchema): Array[FieldToExtract]
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- def isVariantShreddingStruct(s: StructType): Boolean
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- def parquetTypeToSparkType(parquetType: Type): DataType
- def productElementName(n: Int): String
- Definition Classes
- Product
- def productElementNames: Iterator[String]
- Definition Classes
- Product
- def scalarSchemaToSparkType(scalar: ScalarType): DataType
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def variantShreddingSchema(dataType: DataType, isTopLevel: Boolean = true, isObjectField: Boolean = false): StructType
Given an expected schema of a Variant value, returns a suitable schema for shredding, by inserting appropriate intermediate value/typed_value fields at each level.
Given an expected schema of a Variant value, returns a suitable schema for shredding, by inserting appropriate intermediate value/typed_value fields at each level. For example, to represent the JSON {"a": 1, "b": "hello"}, the schema struct<a: int, b: string> could be passed into this function, and it would return the shredding schema: struct< metadata: binary, value: binary, typed_value: struct< a: struct<typed_value: int, value: binary>, b: struct<typed_value: string, value: binary>>>
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)