Package io.delta.kernel.statistics

Class DataFileStatistics

Object
io.delta.kernel.statistics.DataFileStatistics
public class DataFileStatistics extends Object
Encapsulates statistics for a data file in a Delta Lake table and provides methods to serialize those stats to JSON with basic physical-type validation. Note that connectors (e.g. Spark, Flink) are responsible for ensuring the correctness of collected stats, including any necessary string truncation, prior to constructing this class.

  • Field Details

  • Constructor Details

    • DataFileStatistics

      public DataFileStatistics(long numRecords, Map<Column,Literal> minValues, Map<Column,Literal> maxValues, Map<Column,Long> nullCount, Optional<Boolean> tightBounds)
      Create a new instance of DataFileStatistics. The minValues, maxValues, and nullCount are required fields. The tightBounds field is optional - pass Optional.empty() if not specified, Optional.of(true) or Optional.of(false) for explicit values.
      Parameters:
      numRecords - Number of records in the data file.
      minValues - Map of column to minimum value of it in the data file. If the data file has all nulls for the column, the value will be null or not present in the map.
      maxValues - Map of column to maximum value of it in the data file. If the data file has all nulls for the column, the value will be null or not present in the map.
      nullCount - Map of column to number of nulls in the data file.
      tightBounds - Optional boolean indicating if bounds are tight (accurate). Pass Optional.empty() if not specified.

  • Method Details

    • getNumRecords

      public static Optional<Long> getNumRecords(String json)
      Utility method to extract only the numRecords field from a statistics JSON string.
      Parameters:
      json - Data statistics JSON string to deserialize.
      Returns:
      An Optional containing the numRecords value if present.
      Throws:
      KernelException - if JSON parsing fails

    • deserializeFromJson

      public static Optional<DataFileStatistics> deserializeFromJson(String json, StructType physicalSchema)
      Utility method to deserialize statistics from a JSON string with full type information. This overloaded version uses the provided schema to correctly parse min/max values and null counts with their appropriate data types.
      Parameters:
      json - Data statistics JSON string to deserialize.
      physicalSchema - The physical schema providing type information for columns. Must match the schema used during serialization.
      Returns:
      An Optional containing the deserialized DataFileStatistics if present.
      Throws:
      KernelException - if JSON parsing fails or if values don't match expected types

    • getNumRecords

      public long getNumRecords()
      Get the number of records in the data file.
      Returns:
      Number of records in the data file.

    • getMinValues

      public Map<Column,Literal> getMinValues()
      Get the minimum values of the columns in the data file. The map may contain statistics for only a subset of columns in the data file.
      Returns:
      Map of column to minimum value of it in the data file.

    • getMaxValues

      public Map<Column,Literal> getMaxValues()
      Get the maximum values of the columns in the data file. The map may contain statistics for only a subset of columns in the data file.
      Returns:
      Map of column to minimum value of it in the data file.

    • getNullCount

      public Map<Column,Long> getNullCount()
      Get the number of nulls of columns in the data file. The map may contain statistics for only a subset of columns in the data file.
      Returns:
      Map of column to number of nulls in the data file.

    • getTightBounds

      public Optional<Boolean> getTightBounds()
      Get the tight bounds information for the data file. Tight bounds indicate whether the values are guaranteed to be accurate bounds for the data.
      Returns:
      The tight bounds boolean value.

    • withoutTightBounds

      public DataFileStatistics withoutTightBounds()
      Returns a new DataFileStatistics instance with tightBounds set to false. This is useful when the statistics bounds are no longer guaranteed to be tight, such as after applying deletion vectors.
      Returns:
      A new DataFileStatistics with tightBounds set to false

    • serializeAsJson

      public String serializeAsJson(StructType physicalSchema)
      Serializes the statistics as a JSON string.

      Example: For nested column structures: 

       Input:
   minValues = {
     new Column(new String[]{"a", "b", "c"}) mapped to Literal.ofInt(10),
     new Column("d") mapped to Literal.ofString("value")
   }

 Output JSON:
   {
     "minValues": {
       "a": {
         "b": {
           "c": 10
         }
       },
       "d": "value"
     }
   }
      Parameters:
      physicalSchema - the optional physical schema. If provided, all min/max values and null counts will be included and validated against their physical types. If null, only numRecords will be serialized without validation.
      Returns:
      a JSON representation of the statistics.
      Throws:
      KernelException - if dataSchema is provided and there's a type mismatch between the Literal values and the expected types in the schema, or if an unsupported data type is found.

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class Object

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object

    • toString

      public String toString()
      Overrides:
      toString in class Object