Packages

package util

Linear Supertypes
Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. util
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Type Members

  1. class ArrayBasedMapBuilder extends Serializable

    A builder of ArrayBasedMapData, which fails if a null map key is detected, and removes duplicated map keys w.r.t.

    A builder of ArrayBasedMapData, which fails if a null map key is detected, and removes duplicated map keys w.r.t. the last wins policy.

  2. class ArrayBasedMapData extends MapData

    A simple MapData implementation which is backed by 2 arrays.

    A simple MapData implementation which is backed by 2 arrays.

    Note that, user is responsible to guarantee that the key array does not have duplicated elements, otherwise the behavior is undefined.

  3. abstract class ArrayData extends SpecializedGetters with Serializable
  4. class ArrayDataIndexedSeq[T] extends IndexedSeq[T]

    Implements an IndexedSeq interface for ArrayData.

    Implements an IndexedSeq interface for ArrayData. Notice that if the original ArrayData is a primitive array and contains null elements, it is better to ask for IndexedSeq[Any], instead of IndexedSeq[Int], in order to keep the null elements.

  5. case class BadRecordException(record: () => UTF8String, partialResults: () => Array[InternalRow] = () => Array.empty[InternalRow], cause: Throwable) extends Exception with Product with Serializable

    Exception thrown when the underlying parser met a bad record and can't parse it.

    Exception thrown when the underlying parser met a bad record and can't parse it. The stacktrace is not collected for better performance, and thus, this exception should not be used in a user-facing context.

    record

    a function to return the record that cause the parser to fail

    partialResults

    a function that returns an row array, which is the partial results of parsing this bad record.

    cause

    the actual exception about why the record is bad and can't be parsed. It's better to use LazyBadRecordCauseWrapper here to delay heavy cause construction until it's needed.

  6. case class CannotParseJSONFieldException(fieldName: String, fieldValue: String, jsonType: JsonToken, dataType: DataType) extends RuntimeException with Product with Serializable

    No-stacktrace equivalent of QueryExecutionErrors.cannotParseJSONFieldError.

    No-stacktrace equivalent of QueryExecutionErrors.cannotParseJSONFieldError. Used for code control flow in the parser without overhead of creating a full exception.

  7. class CharVarcharCodegenUtils extends AnyRef
  8. case class EmptyJsonFieldValueException(dataType: DataType) extends RuntimeException with Product with Serializable

    No-stacktrace equivalent of QueryExecutionErrors.emptyJsonFieldValueError.

    No-stacktrace equivalent of QueryExecutionErrors.emptyJsonFieldValueError. Used for code control flow in the parser without overhead of creating a full exception.

  9. class FailureSafeParser[IN] extends AnyRef
  10. class GenericArrayData extends ArrayData
  11. final class Geography extends Geo
  12. final class Geometry extends Geo
  13. sealed final class HadoopCompressionCodec extends Enum[HadoopCompressionCodec]

    A mapper class from Spark supported hadoop compression codecs to hadoop compression codecs.

  14. class HyperLogLogPlusPlusHelper extends Serializable
  15. class InternalRowComparableWrapper extends AnyRef

    Wraps the InternalRow with the corresponding DataType to make it comparable with the values in InternalRow.

    Wraps the InternalRow with the corresponding DataType to make it comparable with the values in InternalRow. It uses Spark's internal murmur hash to compute hash code from an row, and uses RowOrdering to perform equality checks.

  16. case class JsonArraysAsStructsException() extends RuntimeException with Product with Serializable

    Exception thrown when the underlying parser parses a JSON array as a struct.

  17. case class LazyBadRecordCauseWrapper(cause: () => Throwable) extends Exception with Product with Serializable

    Exception to use as BadRecordException cause to delay heavy user-facing exception construction.

    Exception to use as BadRecordException cause to delay heavy user-facing exception construction. Does not contain stacktrace and used only for control flow

  18. implicit class LogStringContext extends AnyRef
    Definition Classes
    Logging
  19. abstract class MapData extends Serializable

    This is an internal data representation for map type in Spark SQL.

    This is an internal data representation for map type in Spark SQL. This should not implement equals and hashCode because the type cannot be used as join keys, grouping keys, or in equality tests. See SPARK-9415 and PR#13847 for the discussions.

  20. implicit class MetadataColumnHelper extends AnyRef
  21. sealed trait ParseMode extends AnyRef
  22. case class PartialArrayDataResultException(partialResult: ArrayData, cause: Throwable) extends PartialValueException with Product with Serializable

    Exception thrown when the underlying parser returns a partial array result.

    Exception thrown when the underlying parser returns a partial array result.

    partialResult

    the partial array result.

    cause

    the actual exception about why the parser cannot return full result.

  23. case class PartialMapDataResultException(partialResult: MapData, cause: Throwable) extends PartialValueException with Product with Serializable

    Exception thrown when the underlying parser returns a partial map result.

    Exception thrown when the underlying parser returns a partial map result.

    partialResult

    the partial map result.

    cause

    the actual exception about why the parser cannot return full result.

  24. case class PartialResultArrayException(partialResults: Array[InternalRow], cause: Throwable) extends Exception with Product with Serializable

    Exception thrown when the underlying parser returns partial result list of parsing.

    Exception thrown when the underlying parser returns partial result list of parsing.

    partialResults

    the partial result list of parsing bad records.

    cause

    the actual exception about why the parser cannot return full result.

  25. case class PartialResultException(partialResult: InternalRow, cause: Throwable) extends PartialValueException with Product with Serializable

    Exception thrown when the underlying parser returns a partial result of parsing an object/row.

    Exception thrown when the underlying parser returns a partial result of parsing an object/row.

    partialResult

    the partial result of parsing a bad record.

    cause

    the actual exception about why the parser cannot return full result.

  26. abstract class PartialValueException extends Exception
  27. class QuantileSummaries extends Serializable

    Helper class to compute approximate quantile summary.

    Helper class to compute approximate quantile summary. This implementation is based on the algorithm proposed in the paper: "Space-efficient Online Computation of Quantile Summaries" by Greenwald, Michael and Khanna, Sanjeev. (https://doi.org/10.1145/375663.375670)

    In order to optimize for speed, it maintains an internal buffer of the last seen samples, and only inserts them after crossing a certain size threshold. This guarantees a near-constant runtime complexity compared to the original algorithm.

  28. case class RandomIndicesGenerator(randomSeed: Long) extends Product with Serializable

    This class is used to generate a random indices of given length.

    This class is used to generate a random indices of given length.

    This implementation uses the "inside-out" version of Fisher-Yates algorithm. Reference: https://en.wikipedia.org/wiki/Fisher%E2%80%93Yates_shuffle#The_%22inside-out%22_algorithm

  29. case class RandomUUIDGenerator(randomSeed: Long) extends Product with Serializable

    This class is used to generate a UUID from Pseudo-Random Numbers.

    This class is used to generate a UUID from Pseudo-Random Numbers.

    For the algorithm, see RFC 4122: A Universally Unique IDentifier (UUID) URN Namespace, section 4.4 "Algorithms for Creating a UUID from Truly Random or Pseudo-Random Numbers".

  30. case class ReplaceDataProjections(rowProjection: ProjectingInternalRow, metadataProjection: Option[ProjectingInternalRow]) extends Product with Serializable
  31. final class STUtils extends AnyRef
  32. case class StringAsDataTypeException(fieldName: String, fieldValue: String, dataType: DataType) extends RuntimeException with Product with Serializable

    Exception thrown when the underlying parser can not parses a String as a datatype.

  33. class StringKeyHashMap[T] extends AnyRef
  34. class ToNumberParser extends Serializable

    This class represents a parser to implement the to_number or try_to_number SQL functions.

    This class represents a parser to implement the to_number or try_to_number SQL functions.

    It works by consuming an input string and a format string. This class accepts the format string as a field, and proceeds to iterate through the format string to generate a sequence of tokens (or throw an exception if the format string is invalid). Then when the function is called with an input string, this class steps through the sequence of tokens and compares them against the input string, returning a Spark Decimal object if they match (or throwing an exception otherwise).

  35. class V2ExpressionBuilder extends Logging

    The builder to generate V2 expressions from catalyst expressions.

  36. case class WriteDeltaProjections(rowProjection: Option[ProjectingInternalRow], rowIdProjection: ProjectingInternalRow, metadataProjection: Option[ProjectingInternalRow]) extends Product with Serializable

Value Members

  1. val AGGREGATED_ACCESS_ONLY: String

    If set, this metadata column can only be accessed under AggregateExpression.

    If set, this metadata column can only be accessed under AggregateExpression. This is important when resolving columns in ORDER BY and HAVING clauses on top of Aggregate. In this case we can only reference attributes from grouping expressions, or attributes marked as "aggregated_access_only" under AggregateExpression.

  2. val AUTO_GENERATED_ALIAS: String
  3. val INTERNAL_METADATA_KEYS: Seq[String]
  4. def MDC(key: LogKey, value: Any): MDC
    Attributes
    protected
    Definition Classes
    Logging
  5. val METADATA_COL_ATTR_KEY: String
  6. val QUALIFIED_ACCESS_ONLY: String

    If set, this metadata column can only be accessed with qualifiers, e.g.

    If set, this metadata column can only be accessed with qualifiers, e.g. qualifiers.col or qualifiers.*. If not set, metadata columns cannot be accessed via star.

  7. def escapeSingleQuotedString(str: String): String
  8. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  9. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  10. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  11. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  12. def logBasedOnLevel(level: Level)(f: => MessageWithContext): Unit
    Attributes
    protected
    Definition Classes
    Logging
  13. def logDebug(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  14. def logDebug(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  15. def logDebug(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  16. def logDebug(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  17. def logError(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  18. def logError(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  19. def logError(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  20. def logError(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  21. def logInfo(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  22. def logInfo(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  23. def logInfo(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  24. def logInfo(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  25. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  26. def logTrace(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  27. def logTrace(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  28. def logTrace(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  29. def logTrace(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  30. def logWarning(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  31. def logWarning(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  32. def logWarning(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  33. def logWarning(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  34. def quietly[A](f: => A): A

    Silences output to stderr or stdout for the duration of f

  35. def quoteIdentifier(name: String): String
  36. def quoteIfNeeded(part: String): String
  37. def quoteNameParts(name: Seq[String]): String
  38. def removeInternalMetadata(schema: StructType): StructType
  39. def resourceToBytes(resource: String, classLoader: ClassLoader = Utils.getSparkClassLoader): Array[Byte]
  40. def resourceToString(resource: String, encoding: String = UTF_8.name(), classLoader: ClassLoader = Utils.getSparkClassLoader): String
  41. def sideBySide(left: Seq[String], right: Seq[String]): Seq[String]
  42. def sideBySide(left: String, right: String): Seq[String]
  43. def stackTraceToString(t: Throwable): String
  44. def stringToFile(file: File, str: String): File
  45. def toPrettySQL(e: Expression, shouldTrimTempResolvedColumn: Boolean = false): String
  46. def trimTempResolvedColumn(input: Expression): Expression

    Helper method used to remove all the TempResolvedColumns from the provided expression tree.

  47. def truncatedString[T](seq: Seq[T], sep: String, maxFields: Int): String

    Shorthand for calling truncatedString() without start or end strings.

  48. def truncatedString[T](seq: Seq[T], start: String, sep: String, end: String, maxFields: Int): String

    Format a sequence with semantics similar to calling .mkString().

    Format a sequence with semantics similar to calling .mkString(). Any elements beyond maxNumToStringFields will be dropped and replaced by a "... N more fields" placeholder.

    returns

    the trimmed and formatted string.

  49. def withLogContext(context: Map[String, String])(body: => Unit): Unit
    Attributes
    protected
    Definition Classes
    Logging
  50. object ArrayBasedMapData extends Serializable
  51. object ArrayData extends Serializable
  52. object CharVarcharUtils extends Logging with SparkCharVarcharUtils
  53. object ColumnOrField
  54. object CompressionCodecs
  55. object DateTimeUtils extends SparkDateTimeUtils

    Helper functions for converting between internal and external date and time representations.

    Helper functions for converting between internal and external date and time representations. Dates are exposed externally as java.sql.Date and are represented internally as the number of dates since the Unix epoch (1970-01-01). Timestamps are exposed externally as java.sql.Timestamp and are stored internally as longs, which are capable of storing timestamps with microsecond precision.

  56. case object DropMalformedMode extends ParseMode with Product with Serializable

    This mode ignores the whole corrupted records.

  57. object EvaluateUnresolvedInlineTable extends SQLConfHelper with AliasHelper with EvalHelper with CastSupport

    Utility object used to replace UnresolvedInlineTable with ResolvedInlineTable or (whenever possible) with LocalRelation.

    Utility object used to replace UnresolvedInlineTable with ResolvedInlineTable or (whenever possible) with LocalRelation. Typically, UnresolvedInlineTable is a child of InsertIntoStatement. Use the method: EvaluateUnresolvedInlineTable.evaluate as the entry point for this transformation.

  58. case object FailFastMode extends ParseMode with Product with Serializable

    This mode throws an exception when it meets corrupted records.

  59. object GeneratedColumn

    This object contains utility methods and values for Generated Columns

  60. object GeneratedColumnAnalyzer extends Analyzer

    Analyzer for processing generated column expressions using built-in functions only.

  61. object HyperLogLogPlusPlusHelper extends Serializable

    Constants used in the implementation of the HyperLogLogPlusPlus aggregate function.

    Constants used in the implementation of the HyperLogLogPlusPlus aggregate function.

    See the Appendix to HyperLogLog in Practice: Algorithmic Engineering of a State of the Art Cardinality (https://docs.google.com/document/d/1gyjfMHy43U9OWBXxfaeG-3MjGzejW1dlpyMwEYAAWEI/view?fullscreen) for more information.

  62. object IdentityColumn

    This object contains utility methods and values for Identity Columns

  63. object InternalRowComparableWrapper
  64. object IntervalMathUtils

    Helper functions for interval arithmetic operations with overflow.

  65. object IntervalUtils extends SparkIntervalUtils
  66. object NumberConverter
  67. object ParseMode extends Logging
  68. case object PermissiveMode extends ParseMode with Product with Serializable

    This mode permissively parses the records.

  69. object PushableExpression

    Get the expression of DS V2 to represent catalyst expression that can be pushed down.

  70. object QuantileSummaries extends Serializable
  71. object ResolveDefaultColumns extends QueryErrorsBase with ResolveDefaultColumnsUtils with SQLConfHelper with Logging

    This object contains fields to help process DEFAULT columns.

  72. object ResolveTableConstraints
  73. object RowDeltaUtils

    A utility that holds constants for handling deltas of rows.

  74. object SQLOrderingUtil
  75. object StringKeyHashMap

    Build a map with String type of key, and it also supports either key case sensitive or insensitive.

  76. object StringUtils extends Logging
  77. object ThetaSketchUtils
  78. object ToNumberParser extends Serializable
  79. object TypeUtils extends QueryErrorsBase

    Functions to help with checking for valid data types and value comparison of various types.

  80. object UTF8StringUtils

    Helper functions for casting string to numeric values.

  81. object UnsafeRowUtils

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped