object DateTimeUtils extends SparkDateTimeUtils
Helper functions for converting between internal and external date and time representations. Dates are exposed externally as java.sql.Date and are represented internally as the number of dates since the Unix epoch (1970-01-01). Timestamps are exposed externally as java.sql.Timestamp and are stored internally as longs, which are capable of storing timestamps with microsecond precision.
- Alphabetic
- By Inheritance
- DateTimeUtils
- SparkDateTimeUtils
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final val JULIAN_DAY_OF_EPOCH: Int(2440588)
- val TIMEZONE_OPTION: String
- final val TimeZoneUTC: TimeZone
- Definition Classes
- SparkDateTimeUtils
- def anyToDays(obj: Any): Int
- Definition Classes
- SparkDateTimeUtils
- def anyToMicros(obj: Any): Long
- Definition Classes
- SparkDateTimeUtils
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def cleanLegacyTimestampStr(s: UTF8String): UTF8String
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- def convertSpecialDate(input: String, zoneId: ZoneId): Option[Int]
Converts notational shorthands that are converted to ordinary dates.
Converts notational shorthands that are converted to ordinary dates.
- input
A string to parse. It can contain trailing or leading whitespaces.
- zoneId
Zone identifier used to get the current date.
- returns
Some of days since the epoch if the conversion completed successfully otherwise None.
- def convertSpecialTimestamp(input: String, zoneId: ZoneId): Option[Long]
Converts notational shorthands that are converted to ordinary timestamps.
Converts notational shorthands that are converted to ordinary timestamps.
- input
A string to parse. It can contain trailing or leading whitespaces.
- zoneId
Zone identifier used to get the current timestamp.
- returns
Some of microseconds since the epoch if the conversion completed successfully otherwise None.
- def convertSpecialTimestampNTZ(input: String, zoneId: ZoneId): Option[Long]
Converts notational shorthands that are converted to ordinary timestamps without time zone.
Converts notational shorthands that are converted to ordinary timestamps without time zone.
- input
A string to parse. It can contain trailing or leading whitespaces.
- zoneId
Zone identifier used to get the current local timestamp.
- returns
Some of microseconds since the epoch if the conversion completed successfully otherwise None.
- def convertTimestampNtzToAnotherTz(sourceTz: String, targetTz: String, micros: Long): Long
Converts a timestamp without time zone from a source to target time zone.
Converts a timestamp without time zone from a source to target time zone.
- sourceTz
The time zone for the input timestamp without time zone.
- targetTz
The time zone to which the input timestamp should be converted.
- micros
The offset in microseconds represents a local timestamp.
- returns
The timestamp without time zone represents the same moment (physical time) as the input timestamp in the input time zone, but in the destination time zone.
- def convertTz(micros: Long, fromZone: ZoneId, toZone: ZoneId): Long
- Definition Classes
- SparkDateTimeUtils
- def currentDate(zoneId: ZoneId): Int
Obtains the current date as days since the epoch in the specified time-zone.
- def dateAddInterval(start: Int, interval: CalendarInterval): Int
Adds the interval's months and days to a date expressed as days since the epoch.
Adds the interval's months and days to a date expressed as days since the epoch.
- returns
A date value, expressed in days since 1970-01-01.
- Exceptions thrown
DateTimeExceptionif the result exceeds the supported date rangeIllegalArgumentExceptionif the interval hasmicrosecondspart
- def dateAddMonths(days: Int, months: Int): Int
Adds an year-month interval to a date represented as days since 1970-01-01.
Adds an year-month interval to a date represented as days since 1970-01-01.
- returns
a date value, expressed in days since 1970-01-01.
- def daysToLocalDate(days: Int): LocalDate
- Definition Classes
- SparkDateTimeUtils
- def daysToMicros(days: Int, zoneId: ZoneId): Long
- Definition Classes
- SparkDateTimeUtils
- def doubleToTimestampAnsi(d: Double, context: QueryContext): Long
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def fromJavaDate(date: Date): Int
- Definition Classes
- SparkDateTimeUtils
- def fromJavaTimestamp(timeZoneId: String, t: Timestamp): Long
- Definition Classes
- SparkDateTimeUtils
- def fromJavaTimestamp(t: Timestamp): Long
- Definition Classes
- SparkDateTimeUtils
- def fromJavaTimestampNoRebase(t: Timestamp): Long
- Definition Classes
- SparkDateTimeUtils
- def fromJulianDay(days: Int, nanos: Long): Long
Returns the number of microseconds since epoch from Julian day and nanoseconds in a day.
- def fromUTCTime(micros: Long, timeZone: String): Long
Returns a timestamp of given timezone from UTC timestamp, with the same string representation in their timezone.
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def getDayInYear(days: Int): Int
Returns the 'day in year' value for the given number of days since 1970-01-01.
- def getDayName(days: Int): UTF8String
Returns the three-letter abbreviated day name for the given number of days since 1970-01-01.
- def getDayOfMonth(days: Int): Int
Returns the 'day of month' value for the given number of days since 1970-01-01.
- def getDayOfWeek(days: Int): Int
Returns the day of the week for the given number of days since 1970-01-01 (1 = Sunday, 2 = Monday, ..., 7 = Saturday).
- def getDayOfWeekFromString(string: UTF8String): Int
Returns day of week from String.
Returns day of week from String. Starting from Thursday, marked as 0. (Because 1970-01-01 is Thursday).
- Exceptions thrown
SparkIllegalArgumentExceptionif the input is not a valid day of week.
- def getHours(micros: Long, zoneId: ZoneId): Int
Returns the hour value of a given timestamp value.
Returns the hour value of a given timestamp value. The timestamp is expressed in microseconds.
- def getHoursOfTime(nanos: Long): Int
Returns the hour value of a given TIME (TimeType) value.
- def getLastDayOfMonth(days: Int): Int
Returns last day of the month for the given number of days since 1970-01-01.
- def getLocalDateTime(micros: Long, zoneId: ZoneId): LocalDateTime
- Attributes
- protected
- Definition Classes
- SparkDateTimeUtils
- def getMicroseconds(micros: Long, zoneId: ZoneId): Int
Returns local seconds, including fractional parts, multiplied by 1000000.
Returns local seconds, including fractional parts, multiplied by 1000000.
- micros
The number of microseconds since the epoch.
- zoneId
The time zone id which milliseconds should be obtained in.
- def getMinutes(micros: Long, zoneId: ZoneId): Int
Returns the minute value of a given timestamp value.
Returns the minute value of a given timestamp value. The timestamp is expressed in microseconds since the epoch.
- def getMinutesOfTime(nanos: Long): Int
Returns the minute value of a given TIME (TimeType) value.
- def getMonth(days: Int): Int
Returns the month value for the given number of days since 1970-01-01.
Returns the month value for the given number of days since 1970-01-01. January is month 1.
- def getMonthName(days: Int): UTF8String
Returns the three-letter abbreviated month name for the given number of days since 1970-01-01.
- def getNextDateForDayOfWeek(startDay: Int, dayOfWeek: Int): Int
Returns the first date which is later than startDate and is of the given dayOfWeek.
Returns the first date which is later than startDate and is of the given dayOfWeek. dayOfWeek is an integer ranges in [0, 6], and 0 is Thu, 1 is Fri, etc,.
- def getQuarter(days: Int): Int
Returns the quarter for the given number of days since 1970-01-01.
- def getSeconds(micros: Long, zoneId: ZoneId): Int
Returns the second value of a given timestamp value.
Returns the second value of a given timestamp value. The timestamp is expressed in microseconds since the epoch.
- def getSecondsOfTime(nanos: Long): Int
Returns the second value of a given TIME (TimeType) value.
- def getSecondsOfTimeWithFraction(nanos: Long, precision: Int): Decimal
Returns the second value with fraction from a given TIME (TimeType) value.
Returns the second value with fraction from a given TIME (TimeType) value.
- nanos
The number of nanoseconds since the epoch.
- precision
The time fractional seconds precision, which indicates the number of decimal digits maintained.
- def getSecondsWithFraction(micros: Long, zoneId: ZoneId): Decimal
Returns the seconds part and its fractional part with microseconds.
- def getTimeZone(timeZoneId: String): TimeZone
- Definition Classes
- SparkDateTimeUtils
- def getWeekBasedYear(days: Int): Int
Returns the year which conforms to ISO 8601.
Returns the year which conforms to ISO 8601. Each ISO 8601 week-numbering year begins with the Monday of the week containing the 4th of January.
- def getWeekDay(days: Int): Int
Returns the day of the week for the given number of days since 1970-01-01 (0 = Monday, 1 = Tuesday, ..., 6 = Sunday).
- def getWeekOfYear(days: Int): Int
Returns the week of the year of the given date expressed as the number of days from 1970-01-01.
Returns the week of the year of the given date expressed as the number of days from 1970-01-01. A week is considered to start on a Monday and week 1 is the first week with > 3 days.
- def getYear(days: Int): Int
Returns the year value for the given number of days since 1970-01-01.
- def getZoneId(timeZoneId: String): ZoneId
- Definition Classes
- SparkDateTimeUtils
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def instantToMicros(instant: Instant): Long
- Definition Classes
- SparkDateTimeUtils
- def instantToNanosOfDay(instant: Instant, zoneId: ZoneId): Long
- Definition Classes
- SparkDateTimeUtils
- def instantToNanosOfDay(instant: Instant, timezone: String): Long
- Definition Classes
- SparkDateTimeUtils
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- def localDateTimeToMicros(localDateTime: LocalDateTime): Long
- Definition Classes
- SparkDateTimeUtils
- def localDateToDays(localDate: LocalDate): Int
- Definition Classes
- SparkDateTimeUtils
- def localTimeToNanos(localTime: LocalTime): Long
- Definition Classes
- SparkDateTimeUtils
- def makeTime(hours: Int, minutes: Int, secsAndMicros: Decimal): Long
Converts separate time fields in a long that represents nanoseconds since the start of the day
Converts separate time fields in a long that represents nanoseconds since the start of the day
- hours
the hour, from 0 to 23
- minutes
the minute, from 0 to 59
- secsAndMicros
the second, from 0 to 59.999999
- returns
A time value represented as nanoseconds since the start of the day
- def makeTimestamp(days: Int, nanos: Long, timezone: UTF8String): Long
Makes a timestamp from a date and a local time with timezone string.
Makes a timestamp from a date and a local time with timezone string.
- days
The number of days since the epoch 1970-01-01. Negative numbers represent earlier days.
- nanos
The number of nanoseconds within the day since the midnight.
- timezone
The time zone string.
- returns
The number of microseconds since the epoch 1970-01-01 00:00:00Z.
- def makeTimestamp(days: Int, nanos: Long, zoneId: ZoneId): Long
Makes a timestamp from a date and a local time.
Makes a timestamp from a date and a local time.
- days
The number of days since the epoch 1970-01-01. Negative numbers represent earlier days.
- nanos
The number of nanoseconds within the day since the midnight.
- zoneId
The time zone ID at which the operation is performed.
- returns
The number of microseconds since the epoch 1970-01-01 00:00:00Z.
- def makeTimestampNTZ(days: Int, nanos: Long): Long
Makes a timestamp without time zone from a date and a local time.
Makes a timestamp without time zone from a date and a local time.
- days
The number of days since the epoch 1970-01-01. Negative numbers represent earlier days.
- nanos
The number of nanoseconds within the day since the midnight.
- returns
The number of microseconds since the epoch 1970-01-01 00:00:00Z.
- def microsToDays(micros: Long, zoneId: ZoneId): Int
- Definition Classes
- SparkDateTimeUtils
- def microsToInstant(micros: Long): Instant
- Definition Classes
- SparkDateTimeUtils
- def microsToLocalDateTime(micros: Long): LocalDateTime
- Definition Classes
- SparkDateTimeUtils
- def microsToMillis(micros: Long): Long
- Definition Classes
- SparkDateTimeUtils
- def microsToNanos(micros: Long): Long
- Definition Classes
- SparkDateTimeUtils
- def millisToMicros(millis: Long): Long
- Definition Classes
- SparkDateTimeUtils
- def monthsBetween(micros1: Long, micros2: Long, roundOff: Boolean, zoneId: ZoneId): Double
Returns number of months between micros1 and micros2.
Returns number of months between micros1 and micros2. micros1 and micros2 are expressed in microseconds since 1970-01-01. If micros1 is later than micros2, the result is positive.
If micros1 and micros2 are on the same day of month, or both are the last day of month, returns, time of day will be ignored.
Otherwise, the difference is calculated based on 31 days per month. The result is rounded to 8 decimal places if
roundOffis set to true. - def nanosToLocalTime(nanos: Long): LocalTime
- Definition Classes
- SparkDateTimeUtils
- def nanosToMicros(nanos: Long): Long
- Definition Classes
- SparkDateTimeUtils
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- def parseTimestampString(s: UTF8String): (Array[Int], Option[ZoneId], Boolean)
- Definition Classes
- SparkDateTimeUtils
- def parseTruncLevel(format: UTF8String): Int
Returns the truncate level, could be from TRUNC_TO_MICROSECOND to TRUNC_TO_YEAR, or TRUNC_INVALID, TRUNC_INVALID means unsupported truncate level.
- final val singleHourTz: Pattern
- Definition Classes
- SparkDateTimeUtils
- final val singleMinuteTz: Pattern
- Definition Classes
- SparkDateTimeUtils
- def stringToDate(s: UTF8String): Option[Int]
- Definition Classes
- SparkDateTimeUtils
- def stringToDateAnsi(s: UTF8String, context: QueryContext): Int
- Definition Classes
- SparkDateTimeUtils
- def stringToTime(s: UTF8String): Option[Long]
- Definition Classes
- SparkDateTimeUtils
- def stringToTimeAnsi(s: UTF8String, context: QueryContext): Long
- Definition Classes
- SparkDateTimeUtils
- def stringToTimestamp(s: UTF8String, timeZoneId: ZoneId): Option[Long]
- Definition Classes
- SparkDateTimeUtils
- def stringToTimestampAnsi(s: UTF8String, timeZoneId: ZoneId, context: QueryContext): Long
- Definition Classes
- SparkDateTimeUtils
- def stringToTimestampWithoutTimeZone(s: UTF8String): Option[Long]
Trims and parses a given UTF8 string to a corresponding Long value which representing the number of microseconds since the epoch.
Trims and parses a given UTF8 string to a corresponding Long value which representing the number of microseconds since the epoch. The result is independent of time zones. Zone id component will be ignored. The return type is Option in order to distinguish between 0L and null. Please refer to
parseTimestampStringfor the allowed formats. - def stringToTimestampWithoutTimeZone(s: UTF8String, allowTimeZone: Boolean): Option[Long]
- Definition Classes
- SparkDateTimeUtils
- def stringToTimestampWithoutTimeZoneAnsi(s: UTF8String, context: QueryContext): Long
- def subtractDates(endDay: Int, startDay: Int): CalendarInterval
Subtracts two dates expressed as days since 1970-01-01.
Subtracts two dates expressed as days since 1970-01-01.
- endDay
The end date, exclusive
- startDay
The start date, inclusive
- returns
An interval between two dates. The interval can be negative if the end date is before the start date.
- def subtractTimes(endNanos: Long, startNanos: Long): Long
Subtracts two time values expressed as nanoseconds since 00:00:00, and returns the difference in microseconds.
Subtracts two time values expressed as nanoseconds since 00:00:00, and returns the difference in microseconds.
- endNanos
The end time as nanoseconds since the midnight, exclusive
- startNanos
The end time as nanoseconds since the midnight, inclusive
- returns
The difference in microseconds between local time corresponded to the input
endNanosandstartNanos.
- def subtractTimestamps(endMicros: Long, startMicros: Long, zoneId: ZoneId): Long
Subtracts two timestamps expressed as microseconds since 1970-01-01 00:00:00Z, and returns the difference in microseconds between local timestamps at the given time zone.
Subtracts two timestamps expressed as microseconds since 1970-01-01 00:00:00Z, and returns the difference in microseconds between local timestamps at the given time zone.
- endMicros
The end timestamp as microseconds since the epoch, exclusive
- startMicros
The end timestamp as microseconds since the epoch, inclusive
- zoneId
The time zone ID in which the subtraction is performed
- returns
The difference in microseconds between local timestamps corresponded to the input instants
endandstart.
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def timeAddInterval(time: Long, timePrecision: Int, interval: Long, intervalEndField: Byte, targetPrecision: Int): Long
Adds a day-time interval to a time.
Adds a day-time interval to a time.
- time
A time in nanoseconds.
- timePrecision
The number of digits of the fraction part of time.
- interval
A day-time interval in microseconds.
- intervalEndField
The rightmost field which the interval comprises of. Valid values: 0 (DAY), 1 (HOUR), 2 (MINUTE), 3 (SECOND).
- targetPrecision
The number of digits of the fraction part of the resulting time.
- returns
A time value in nanoseconds or throw an arithmetic overflow if the result out of valid time range [00:00, 24:00).
- def timeDiff(unit: UTF8String, startNanos: Long, endNanos: Long): Long
Gets the difference between two time values in the specified unit.
Gets the difference between two time values in the specified unit.
- unit
Specifies the interval units in which to express the difference between the two time parameters. Supported units are: MICROSECOND, MILLISECOND, SECOND, MINUTE, HOUR.
- startNanos
A time value expressed as nanoseconds since the start of the day, which the function subtracts from
endNanos.- endNanos
A time value expressed as nanoseconds since the start of the day, from which the function subtracts
startNanos.- returns
The time span between two time values, in the units specified.
- def timeTrunc(level: UTF8String, nanos: Long): Long
Returns time truncated to the unit specified by the level.
Returns time truncated to the unit specified by the level. Trunc level is parsed directly to corresponding ChronoUnits. Note that only levels from 'MICROSECOND' to 'HOUR' are supported.
- def timestampAdd(unit: String, quantity: Long, micros: Long, zoneId: ZoneId): Long
Adds the specified number of units to a timestamp.
Adds the specified number of units to a timestamp.
- unit
A keyword that specifies the interval units to add to the input timestamp.
- quantity
The amount of
units to add. It can be positive or negative.- micros
The input timestamp value, expressed in microseconds since 1970-01-01 00:00:00Z.
- zoneId
The time zone ID at which the operation is performed.
- returns
A timestamp value, expressed in microseconds since 1970-01-01 00:00:00Z.
- def timestampAddDayTime(micros: Long, dayTime: Long, zoneId: ZoneId): Long
Adds a day-time interval expressed in microseconds to a timestamp at the given time zone.
Adds a day-time interval expressed in microseconds to a timestamp at the given time zone. It converts the input timestamp to a local timestamp, and adds the interval by:
- Splitting the interval to days and microsecond adjustment in a day, and
- First of all, it adds days and then the time part. The resulted local timestamp is converted back to an instant at the given time zone.
- micros
The input timestamp value, expressed in microseconds since 1970-01-01 00:00:00Z.
- dayTime
The amount of microseconds to add. It can be positive or negative.
- zoneId
The time zone ID at which the operation is performed.
- returns
A timestamp value, expressed in microseconds since 1970-01-01 00:00:00Z.
- def timestampAddInterval(start: Long, months: Int, days: Int, microseconds: Long, zoneId: ZoneId): Long
Adds a full interval (months, days, microseconds) to a timestamp represented as the number of microseconds since 1970-01-01 00:00:00Z.
Adds a full interval (months, days, microseconds) to a timestamp represented as the number of microseconds since 1970-01-01 00:00:00Z.
- returns
A timestamp value, expressed in microseconds since 1970-01-01 00:00:00Z.
- def timestampAddMonths(micros: Long, months: Int, zoneId: ZoneId): Long
Adds months to a timestamp at the given time zone.
Adds months to a timestamp at the given time zone. It converts the input timestamp to a local timestamp at the given time zone, adds months, and converts the resulted local timestamp back to a timestamp, expressed in microseconds since 1970-01-01 00:00:00Z.
- micros
The input timestamp value, expressed in microseconds since 1970-01-01 00:00:00Z
- months
The amount of months to add. It can be positive or negative.
- zoneId
The time zone ID at which the operation is performed.
- returns
A timestamp value, expressed in microseconds since 1970-01-01 00:00:00Z.
- def timestampDiff(unit: String, startTs: Long, endTs: Long, zoneId: ZoneId): Long
Gets the difference between two timestamps.
Gets the difference between two timestamps.
- unit
Specifies the interval units in which to express the difference between the two timestamp parameters.
- startTs
A timestamp which the function subtracts from
endTs.- endTs
A timestamp from which the function subtracts
startTs.- zoneId
The time zone ID at which the operation is performed.
- returns
The time span between two timestamp values, in the units specified.
- def timestampNTZAddInterval(start: Long, months: Int, days: Int, microseconds: Long, zoneId: ZoneId): Long
Adds a full interval (months, days, microseconds) to a timestamp without time zone represented as a local time in microsecond precision, which is independent of time zone.
Adds a full interval (months, days, microseconds) to a timestamp without time zone represented as a local time in microsecond precision, which is independent of time zone.
- returns
A timestamp without time zone value, expressed in range [0001-01-01T00:00:00.000000, 9999-12-31T23:59:59.999999].
- def toJavaDate(days: Int): Date
- Definition Classes
- SparkDateTimeUtils
- def toJavaTimestamp(timeZoneId: String, micros: Long): Timestamp
- Definition Classes
- SparkDateTimeUtils
- def toJavaTimestamp(micros: Long): Timestamp
- Definition Classes
- SparkDateTimeUtils
- def toJavaTimestampNoRebase(micros: Long): Timestamp
- Definition Classes
- SparkDateTimeUtils
- def toJulianDay(micros: Long): (Int, Long)
Returns Julian day and nanoseconds in a day from the number of microseconds
Returns Julian day and nanoseconds in a day from the number of microseconds
Note: support timestamp since 4717 BC (without negative nanoseconds, compatible with Hive).
- def toString(): String
- Definition Classes
- AnyRef → Any
- def toUTCTime(micros: Long, timeZone: String): Long
Returns a utc timestamp from a given timestamp from a given timezone, with the same string representation in their timezone.
- def truncDate(days: Int, level: Int): Int
Returns the trunc date from original date and trunc level.
Returns the trunc date from original date and trunc level. Trunc level should be generated using
parseTruncLevel(), should be between 6 and 9. - def truncTimestamp(micros: Long, level: Int, zoneId: ZoneId): Long
Returns the trunc date time from original date time and trunc level.
Returns the trunc date time from original date time and trunc level. Trunc level should be generated using
parseTruncLevel(), should be between 0 and 9. - def truncateTimeToPrecision(nanos: Long, p: Int): Long
- Definition Classes
- SparkDateTimeUtils
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)