Packages

final class UTF8String extends Comparable[UTF8String] with Externalizable with KryoSerializable with Cloneable

A UTF-8 String for internal Spark use.

A String encoded in UTF-8 as an Array[Byte], which can be used for comparison, search, see http://en.wikipedia.org/wiki/UTF-8 for details.

Note: This is not designed for general use cases, should not be used outside SQL.

Linear Supertypes
Cloneable, KryoSerializable, Externalizable, Serializable, Comparable[UTF8String], AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. UTF8String
  2. Cloneable
  3. KryoSerializable
  4. Externalizable
  5. Serializable
  6. Comparable
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new UTF8String()

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def binaryCompare(other: UTF8String): Int

    Binary comparison of two UTF8String.

    Binary comparison of two UTF8String. Can only be used for default UTF8_BINARY collation.

  6. def binaryEquals(other: UTF8String): Boolean

    Binary equality check of two UTF8String.

    Binary equality check of two UTF8String. Note that binary equality is not the same as equality under given collation. E.g. if string is collated in case-insensitive two strings are considered equal even if they are different in binary comparison.

  7. def bytePosToChar(bytePos: Int): Int
  8. def charPosToByte(charPos: Int): Int
  9. def clone(): UTF8String
    Definition Classes
    UTF8String → AnyRef
    Annotations
    @Override()
  10. def codePointFrom(byteIndex: Int): Int

    Returns the code point starting from the byte at position byteIndex.

    Returns the code point starting from the byte at position byteIndex. If byte index is invalid, throws exception.

  11. def codePointIterator(iteratorMode: CodePointIteratorType): Iterator[Integer]
  12. def codePointIterator(): Iterator[Integer]

    Returns a code point iterator for this UTF8String.

  13. def compareTo(other: UTF8String): Int

    Implementation of Comparable interface.

    Implementation of Comparable interface. This method is kept for backwards compatibility. It should not be used in spark code base, given that string comparison requires passing collation id. Either explicitly use binaryCompare or use semanticCompare.

    Definition Classes
    UTF8String → Comparable
    Annotations
    @Override()
  14. def contains(substring: UTF8String): Boolean

    Returns whether this contains substring or not.

  15. def copy(): UTF8String
  16. def copyUTF8String(start: Int, end: Int): UTF8String

    Copy the bytes from the current UTF8String, and make a new UTF8String.

    Copy the bytes from the current UTF8String, and make a new UTF8String.

    start

    the start position of the current UTF8String in bytes.

    end

    the end position of the current UTF8String in bytes.

    returns

    a new UTF8String in the position of [start, end] of current UTF8String bytes.

  17. def endsWith(suffix: UTF8String): Boolean
  18. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  19. def equals(other: AnyRef): Boolean

    Binary equality check of two UTF8String.

    Binary equality check of two UTF8String. Note that binary equality is not the same as equality under given collation. E.g. if string is collated in case-insensitive two strings are considered equal even if they are different in binary comparison.

    Definition Classes
    UTF8String → AnyRef → Any
    Annotations
    @Override()
  20. def find(str: UTF8String, start: Int): Int

    Find the str from left to right.

  21. def findInSet(match: UTF8String): Int
  22. def getBaseObject(): AnyRef
  23. def getBaseOffset(): Long
  24. def getByte(byteIndex: Int): Byte

    Returns the byte at (byte) position byteIndex.

    Returns the byte at (byte) position byteIndex. If byte index is invalid, returns 0.

  25. def getByteBuffer(): ByteBuffer

    Returns a ByteBuffer wrapping the base object if it is a byte array or a copy of the data if the base object is not a byte array.

    Returns a ByteBuffer wrapping the base object if it is a byte array or a copy of the data if the base object is not a byte array.

    Unlike getBytes this will not create a copy the array if this is a slice.

    Annotations
    @Nonnull()
  26. def getBytes(): Array[Byte]

    Returns the underline bytes, will be a copy of it if it's part of another array.

  27. def getChar(charIndex: Int): Int

    Returns the code point at (char) position charIndex.

    Returns the code point at (char) position charIndex. If char index is invalid, throws exception. Note that this method is not efficient as it needs to traverse the UTF-8 string. If byteIndex of the first byte in the code point is known, use codePointFrom instead.

  28. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  29. def getPrefix(): Long

    Returns a 64-bit integer that can be used as the prefix used in sorting.

  30. def hashCode(): Int
    Definition Classes
    UTF8String → AnyRef → Any
    Annotations
    @Override()
  31. def indexOf(v: UTF8String, start: Int): Int

    Returns the position of the first occurrence of substr in current string from the specified position (0-based index).

    Returns the position of the first occurrence of substr in current string from the specified position (0-based index).

    v

    the string to be searched

    start

    the start position of the current string for searching

    returns

    the position of the first occurrence of substr, if not found, -1 returned.

  32. def indexOfEmpty(start: Int): Int

    Returns the (default) position of the first occurrence of an empty substr in the current string from the specified position (0-based index).

    Returns the (default) position of the first occurrence of an empty substr in the current string from the specified position (0-based index).

    start

    the start position of the current string for searching

    returns

    the position of the first occurrence of the empty substr (now, always 0)

  33. def isFullAscii(): Boolean
  34. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  35. def isValid(): Boolean

    Checks if the current UTF8String is valid.

    Checks if the current UTF8String is valid.

    returns

    If string represents a valid UTF8 string.

  36. def levenshteinDistance(other: UTF8String, threshold: Int): Int
  37. def levenshteinDistance(other: UTF8String): Int

    Levenshtein distance is a metric for measuring the distance of two strings.

    Levenshtein distance is a metric for measuring the distance of two strings. The distance is defined by the minimum number of single-character edits (i.e. insertions, deletions or substitutions) that are required to change one of the strings into the other.

  38. def lpad(len: Int, pad: UTF8String): UTF8String

    Returns str, left-padded with pad to a length of len.

    Returns str, left-padded with pad to a length of len. For example: ('hi', 5, '??') => '???hi' ('hi', 1, '??') => 'h'

  39. def makeValid(): UTF8String

    Returns a validated version of the current UTF-8 string by replacing invalid UTF-8 sequences with the Unicode replacement character (U+FFFD), as per the rules defined in the Unicode standard 15, Section 3.9, Paragraph D86, Table 3-7.

    Returns a validated version of the current UTF-8 string by replacing invalid UTF-8 sequences with the Unicode replacement character (U+FFFD), as per the rules defined in the Unicode standard 15, Section 3.9, Paragraph D86, Table 3-7. This behaviour is consistent with the behaviour of UnicodeString in ICU4C.

    returns

    A new UTF8String that is a valid UTF8 string.

  40. def matchAt(s: UTF8String, pos: Int): Boolean
  41. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  42. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  43. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  44. def numBytes(): Int

    Returns the number of bytes

  45. def numChars(): Int

    Returns the number of code points in it.

  46. def read(kryo: Kryo, in: Input): Unit
    Definition Classes
    UTF8String → KryoSerializable
    Annotations
    @Override()
  47. def readExternal(in: ObjectInput): Unit
    Definition Classes
    UTF8String → Externalizable
    Annotations
    @Override()
  48. def repeat(times: Int): UTF8String
  49. def replace(search: UTF8String, replace: UTF8String): UTF8String
  50. def reverse(): UTF8String
  51. def reverseCodePointIterator(iteratorMode: CodePointIteratorType): Iterator[Integer]
  52. def reverseCodePointIterator(): Iterator[Integer]

    Reverse version of the code point iterator for this UTF8String, returns code points in the current string one by one, as integers, in reverse order.

    Reverse version of the code point iterator for this UTF8String, returns code points in the current string one by one, as integers, in reverse order. The logic is similar to the above.

  53. def rfind(str: UTF8String, start: Int): Int

    Find the str from right to left.

  54. def rpad(len: Int, pad: UTF8String): UTF8String

    Returns str, right-padded with pad to a length of len For example: ('hi', 5, '??') => 'hi???' ('hi', 1, '??') => 'h'

  55. def semanticCompare(other: UTF8String, collationId: Int): Int

    Collation-aware comparison of two UTF8String.

    Collation-aware comparison of two UTF8String. The collation to use is specified by the collationId parameter.

  56. def semanticEquals(other: UTF8String, collationId: Int): Boolean

    Collation-aware equality comparison of two UTF8String.

  57. def soundex(): UTF8String

    Encodes a string into a Soundex value.

    Encodes a string into a Soundex value. Soundex is an encoding used to relate similar names, but can also be used as a general purpose scheme to find word with similar phonemes. https://en.wikipedia.org/wiki/Soundex

  58. def split(pattern: UTF8String, limit: Int): Array[UTF8String]
  59. def splitSQL(delimiter: UTF8String, limit: Int): Array[UTF8String]
  60. def startsWith(prefix: UTF8String): Boolean
  61. def subStringIndex(delim: UTF8String, count: Int): UTF8String

    Returns the substring from string str before count occurrences of the delimiter delim.

    Returns the substring from string str before count occurrences of the delimiter delim. If count is positive, everything the left of the final delimiter (counting from left) is returned. If count is negative, every to the right of the final delimiter (counting from the right) is returned. subStringIndex performs a case-sensitive match when searching for delim.

  62. def substring(start: Int, until: Int): UTF8String

    Returns a substring of this.

    Returns a substring of this.

    start

    the position of first code point

    until

    the position after last code point, exclusive.

  63. def substringSQL(pos: Int, length: Int): UTF8String
  64. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  65. def toByte(intWrapper: IntWrapper): Boolean
  66. def toByteExact(): Byte
  67. def toInt(intWrapper: IntWrapper): Boolean

    Parses this UTF8String(trimmed if needed) to int.

    Parses this UTF8String(trimmed if needed) to int.

    Note that, in this method we accumulate the result in negative format, and convert it to positive format at the end, if this string is not started with '-'. This is because min value is bigger than max value in digits, e.g. Integer.MAX_VALUE is '2147483647' and Integer.MIN_VALUE is '-2147483648'.

    This code is mostly copied from LazyInt.parseInt in Hive.

    Note that, this method is almost same as toLong, but we leave it duplicated for performance reasons, like Hive does.

    intWrapper

    If a valid int was parsed from this UTF8String, then its value would be set in intWrapper

    returns

    true if the parsing was successful else false

  68. def toIntExact(): Int

    Parses UTF8String(trimmed if needed) to int.

    Parses UTF8String(trimmed if needed) to int. This method is used when ANSI is enabled.

    returns

    If string contains valid numeric value then it returns the int value otherwise a NumberFormatException is thrown.

  69. def toLong(toLongResult: LongWrapper): Boolean

    Parses this UTF8String(trimmed if needed) to long.

    Parses this UTF8String(trimmed if needed) to long.

    Note that, in this method we accumulate the result in negative format, and convert it to positive format at the end, if this string is not started with '-'. This is because min value is bigger than max value in digits, e.g. Long.MAX_VALUE is '9223372036854775807' and Long.MIN_VALUE is '-9223372036854775808'.

    This code is mostly copied from LazyLong.parseLong in Hive.

    toLongResult

    If a valid long was parsed from this UTF8String, then its value would be set in toLongResult

    returns

    true if the parsing was successful else false

  70. def toLongExact(): Long

    Parses UTF8String(trimmed if needed) to long.

    Parses UTF8String(trimmed if needed) to long. This method is used when ANSI is enabled.

    returns

    If string contains valid numeric value then it returns the long value otherwise a NumberFormatException is thrown.

  71. def toLowerCase(): UTF8String

    Returns the lower case of this string

  72. def toLowerCaseAscii(): UTF8String
  73. def toShort(intWrapper: IntWrapper): Boolean
  74. def toShortExact(): Short
  75. def toString(): String

    Returns a string representation of this UTF8String object.

    Returns a string representation of this UTF8String object. The string representation consists of the string's characters encoded in UTF-8 and the result of this method is always a valid UTF-8 string. However, if the current UTF8String contains illegal UTF-8 byte sequences, the method will replace the illegal byte sequences with the Unicode replacement character U+FFFD, according to Java specification. Using this method with invalid UTF8Strings is NOT RECOMMENDED.

    Definition Classes
    UTF8String → AnyRef → Any
    Annotations
    @Override()
  76. def toTitleCase(): UTF8String

    Returns the title case of this string, that could be used as title.

    Returns the title case of this string, that could be used as title. There are essentially two different version of this method - one using the JVM case mapping rules, and the other using the ICU case mapping rules. ASCII implementation is the same for both, but please refer to the respective methods for the slow (non-ASCII) implementation for more details on the differences.

  77. def toTitleCaseICU(): UTF8String
  78. def toUpperCase(): UTF8String

    Returns the upper case of this string

  79. def toUpperCaseAscii(): UTF8String
  80. def toValidString(): String

    Returns a string representation of this UTF8String object, but uses our custom implementation for invalid UTF-8 byte sequence replacement, as per the specification defined in the Unicode standard 15, Section 3.9, Paragraph D86, Table 3-7.

    Returns a string representation of this UTF8String object, but uses our custom implementation for invalid UTF-8 byte sequence replacement, as per the specification defined in the Unicode standard 15, Section 3.9, Paragraph D86, Table 3-7. Hence, the result of this method is always a valid UTF-8 string. This is the recommended method to use with invalid UTF8Strings.

  81. def translate(dict: Map[String, String]): UTF8String
  82. def trim(trimString: UTF8String): UTF8String

    Trims instances of the given trim string from both ends of this string.

    Trims instances of the given trim string from both ends of this string.

    trimString

    the trim character string

    returns

    this string with no occurrences of the trim string at the start or end, or null if trimString is null

  83. def trim(): UTF8String

    Trims space characters (ASCII 32) from both ends of this string.

    Trims space characters (ASCII 32) from both ends of this string.

    returns

    this string with no spaces at the start or end

  84. def trimAll(): UTF8String

    Trims whitespace ASCII characters from both ends of this string.

    Trims whitespace ASCII characters from both ends of this string.

    Note that, this method is different from UTF8String#trim() which removes only spaces(= ASCII 32) from both ends.

    returns

    A UTF8String whose value is this UTF8String, with any leading and trailing white space removed, or this UTF8String if it has no leading or trailing whitespace.

  85. def trimLeft(trimString: UTF8String): UTF8String

    Trims instances of the given trim string from the start of this string.

    Trims instances of the given trim string from the start of this string.

    trimString

    the trim character string

    returns

    this string with no occurrences of the trim string at the start, or null if trimString is null

  86. def trimLeft(): UTF8String

    Trims space characters (ASCII 32) from the start of this string.

    Trims space characters (ASCII 32) from the start of this string.

    returns

    this string with no spaces at the start

  87. def trimRight(trimString: UTF8String): UTF8String

    Trims instances of the given trim string from the end of this string.

    Trims instances of the given trim string from the end of this string.

    trimString

    the trim character string

    returns

    this string with no occurrences of the trim string at the end, or null if trimString is null

  88. def trimRight(): UTF8String

    Trims space characters (ASCII 32) from the end of this string.

    Trims space characters (ASCII 32) from the end of this string.

    returns

    this string with no spaces at the end

  89. def trimTrailingSpaces(numSpaces: Int): UTF8String

    Trims at most numSpaces space characters (ASCII 32) from the end of this string.

  90. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  91. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  92. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  93. def write(kryo: Kryo, out: Output): Unit
    Definition Classes
    UTF8String → KryoSerializable
    Annotations
    @Override()
  94. def writeExternal(out: ObjectOutput): Unit
    Definition Classes
    UTF8String → Externalizable
    Annotations
    @Override()
  95. def writeTo(out: OutputStream): Unit
  96. def writeTo(buffer: ByteBuffer): Unit
  97. def writeToMemory(target: AnyRef, targetOffset: Long): Unit

    Writes the content of this string into a memory address, identified by an object and an offset.

    Writes the content of this string into a memory address, identified by an object and an offset. The target memory address must already been allocated, and have enough space to hold all the bytes in this string.

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from Cloneable

Inherited from KryoSerializable

Inherited from Externalizable

Inherited from Serializable

Inherited from Comparable[UTF8String]

Inherited from AnyRef

Inherited from Any

Ungrouped