Package io.delta.kernel.internal.skipping

Class StatsSchemaHelper

Object
io.delta.kernel.internal.skipping.StatsSchemaHelper
public class StatsSchemaHelper extends Object
Provides information and utilities for statistics columns given a table schema. Specifically, it is used to:
  1. Get the expected statistics schema given a table schema
  2. Check if a Literal or Column is skipping eligible
  3. Get the statistics column for a given stat type and logical column

  • Field Details

  • Constructor Details

    • StatsSchemaHelper

      public StatsSchemaHelper(StructType dataSchema)

  • Method Details

    • isSkippingEligibleLiteral

      public static boolean isSkippingEligibleLiteral(Literal literal)
      Returns true if the given literal is skipping-eligible. Delta tracks min/max stats for a limited set of data types and only literals of those types are skipping eligible.

    • isSkippingEligibleDataType

      public static boolean isSkippingEligibleDataType(DataType dataType)
      Returns true if the given data type is eligible for MIN/MAX data skipping.

    • getStatsSchema

      public static StructType getStatsSchema(StructType dataSchema, Set<CollationIdentifier> collationIdentifiers)
      Returns the expected statistics schema given a table schema.

      Here is an example of a data schema along with the schema of the statistics that would be collected.

      Data Schema: 

       |-- a: struct (nullable = true)
 |  |-- b: struct (nullable = true)
 |  |  |-- c: long (nullable = true)
 |  |  |-- d: string (nullable = true)

      Collected Statistics: 

       |-- stats: struct (nullable = true)
 |  |-- numRecords: long (nullable = false)
 |  |-- minValues: struct (nullable = false)
 |  |  |-- a: struct (nullable = false)
 |  |  |  |-- b: struct (nullable = false)
 |  |  |  |  |-- c: long (nullable = true)
 |  |  |  |  |-- d: string (nullable = true)
 |  |-- maxValues: struct (nullable = false)
 |  |  |-- a: struct (nullable = false)
 |  |  |  |-- b: struct (nullable = false)
 |  |  |  |  |-- c: long (nullable = true)
 |  |  |  |  |-- d: string (nullable = true)
 |  |-- nullCount: struct (nullable = false)
 |  |  |-- a: struct (nullable = false)
 |  |  |  |-- b: struct (nullable = false)
 |  |  |  |  |-- c: long (nullable = true)
 |  |  |  |  |-- d: string (nullable = true)
 |  |-- tightBounds: boolean (nullable = true)
 |  |-- statsWithCollation: struct (nullable = true)
 |  |  |-- collationName: struct (nullable = true)
 |  |  |  |-- min: struct (nullable = true)
 |  |  |  |  |-- a: struct (nullable = true)
 |  |  |  |  |  |-- b: struct (nullable = true)
 |  |  |  |  |  |  |-- d: string (nullable = true)
 |  |  |  |-- max: struct (nullable = true)
 |  |  |  |  |-- a: struct (nullable = true)
 |  |  |  |  |  |-- b: struct (nullable = true)
 |  |  |  |  |  |  |-- d: string (nullable = true)

    • getMinColumn

      public Tuple2<Column,Optional<Expression>> getMinColumn(Column column, Optional<CollationIdentifier> collationIdentifier)
      Given a logical column in the data schema provided when creating this, return the corresponding MIN column and an optional column adjustment expression from the statistic schema that stores the MIN values for the provided logical column.
      Parameters:
      column - the logical column name.
      collationIdentifier - optional collation identifier if getting a collated stats column.
      Returns:
      a tuple of the MIN column and an optional adjustment expression.

    • getMaxColumn

      public Tuple2<Column,Optional<Expression>> getMaxColumn(Column column, Optional<CollationIdentifier> collationIdentifier)
      Given a logical column in the data schema provided when creating this, return the corresponding MAX column and an optional column adjustment expression from the statistic schema that stores the MAX values for the provided logical column.
      Parameters:
      column - the logical column name.
      collationIdentifier - optional collation identifier if getting a collated stats column.
      Returns:
      a tuple of the MAX column and an optional adjustment expression.

    • getNullCountColumn

      public Column getNullCountColumn(Column column)
      Given a logical column in the data schema provided when creating this, return the corresponding NULL_COUNT column in the statistic schema that stores the null count values for the provided logical column.

    • getNumRecordsColumn

      public Column getNumRecordsColumn()
      Returns the NUM_RECORDS column in the statistic schema

    • isSkippingEligibleMinMaxColumn

      public boolean isSkippingEligibleMinMaxColumn(Column column)
      Returns true if the given column is skipping-eligible using min/max statistics. This means the column exists, is a leaf column, and is of a skipping-eligible data-type.

    • isSkippingEligibleNullCountColumn

      public boolean isSkippingEligibleNullCountColumn(Column column)
      Returns true if the given column is skipping-eligible using null count statistics. This means the column exists and is a leaf column as we only collect stats for leaf columns.