Class Statistics<T extends Comparable<T>>

java.lang.Object
org.apache.parquet.column.statistics.Statistics<T>
Type Parameters:
T - the Java type described by this Statistics instance
Direct Known Subclasses:
BinaryStatistics, BooleanStatistics, DoubleStatistics, FloatStatistics, IntStatistics, LongStatistics

public abstract class Statistics<T extends Comparable<T>> extends Object
Statistics class to keep track of statistics in parquet pages and column chunks
  • Method Details

    • getStatsBasedOnType

      @Deprecated public static Statistics getStatsBasedOnType(PrimitiveType.PrimitiveTypeName type)
      Deprecated.
      Use createStats(Type) instead
      Returns the typed statistics object based on the passed type parameter
      Parameters:
      type - PrimitiveTypeName type of the column
      Returns:
      instance of a typed statistics class
    • createStats

      public static Statistics<?> createStats(Type type)
      Creates an empty Statistics instance for the specified type to be used for reading/writing the new min/max statistics used in the V2 format.
      Parameters:
      type - type of the column
      Returns:
      instance of a typed statistics class
    • noopStats

      public static Statistics<?> noopStats(Type type)
      Creates a noop NoopStatistics statistics instance. This is only used when the user disables statistics for the specified column.
      Parameters:
      type - type of the column
      Returns:
      a noop statistics
    • getBuilderForReading

      public static Statistics.Builder getBuilderForReading(PrimitiveType type)
      Returns a builder to create new statistics object. Used to read the statistics from the parquet file.
      Parameters:
      type - type of the column
      Returns:
      builder to create new statistics object
    • updateStats

      public void updateStats(int value)
      updates statistics min and max using the passed value
      Parameters:
      value - value to use to update min and max
    • updateStats

      public void updateStats(long value)
      updates statistics min and max using the passed value
      Parameters:
      value - value to use to update min and max
    • updateStats

      public void updateStats(float value)
      updates statistics min and max using the passed value
      Parameters:
      value - value to use to update min and max
    • updateStats

      public void updateStats(double value)
      updates statistics min and max using the passed value
      Parameters:
      value - value to use to update min and max
    • updateStats

      public void updateStats(boolean value)
      updates statistics min and max using the passed value
      Parameters:
      value - value to use to update min and max
    • updateStats

      public void updateStats(Binary value)
      updates statistics min and max using the passed value
      Parameters:
      value - value to use to update min and max
    • equals

      public boolean equals(Object other)
      Equality comparison method to compare two statistics objects.
      Overrides:
      equals in class Object
      Parameters:
      other - Object to compare against
      Returns:
      true if objects are equal, false otherwise
    • hashCode

      public int hashCode()
      Hash code for the statistics object
      Overrides:
      hashCode in class Object
      Returns:
      hash code int
    • mergeStatistics

      public void mergeStatistics(Statistics stats)
      Method to merge this statistics object with the object passed as parameter. Merging keeps the smallest of min values, largest of max values and combines the number of null counts.
      Parameters:
      stats - Statistics object to merge with
    • mergeStatisticsMinMax

      protected abstract void mergeStatisticsMinMax(Statistics stats)
      Abstract method to merge this statistics min and max with the values of the parameter object. Does not do any checks, only called internally.
      Parameters:
      stats - Statistics object to merge with
    • setMinMaxFromBytes

      @Deprecated public abstract void setMinMaxFromBytes(byte[] minBytes, byte[] maxBytes)
      Deprecated.
      will be removed in 2.0.0. Use getBuilderForReading(PrimitiveType) instead.
      Abstract method to set min and max values from byte arrays.
      Parameters:
      minBytes - byte array to set the min value to
      maxBytes - byte array to set the max value to
    • genericGetMin

      public abstract T genericGetMin()
      Returns the min value in the statistics. The java natural order of the returned type defined by Comparable.compareTo(Object) might not be the proper one. For example, UINT_32 requires unsigned comparison instead of the natural signed one. Use compareMinToValue(Comparable) or the comparator returned by comparator() to always get the proper ordering.
      Returns:
      the min value
    • genericGetMax

      public abstract T genericGetMax()
      Returns the max value in the statistics. The java natural order of the returned type defined by Comparable.compareTo(Object) might not be the proper one. For example, UINT_32 requires unsigned comparison instead of the natural signed one. Use compareMaxToValue(Comparable) or the comparator returned by comparator() to always get the proper ordering.
      Returns:
      the max value
    • comparator

      public final PrimitiveComparator<T> comparator()
      Returns the PrimitiveComparator implementation to be used to compare two generic values in the proper way (for example, unsigned comparison for UINT_32).
      Returns:
      the comparator for data described by this Statistics instance
    • compareMinToValue

      public final int compareMinToValue(T value)
      Compares min to the specified value in the proper way. It does the same as invoking comparator().compare(genericGetMin(), value). The corresponding statistics implementations overload this method so the one with the primitive argument shall be used to avoid boxing/unboxing.
      Parameters:
      value - the value which min is to be compared to
      Returns:
      a negative integer, zero, or a positive integer as min is less than, equal to, or greater than value.
    • compareMaxToValue

      public final int compareMaxToValue(T value)
      Compares max to the specified value in the proper way. It does the same as invoking comparator().compare(genericGetMax(), value). The corresponding statistics implementations overload this method so the one with the primitive argument shall be used to avoid boxing/unboxing.
      Parameters:
      value - the value which max is to be compared to
      Returns:
      a negative integer, zero, or a positive integer as max is less than, equal to, or greater than value.
    • getMaxBytes

      public abstract byte[] getMaxBytes()
      Abstract method to return the max value as a byte array
      Returns:
      byte array corresponding to the max value
    • getMinBytes

      public abstract byte[] getMinBytes()
      Abstract method to return the min value as a byte array
      Returns:
      byte array corresponding to the min value
    • minAsString

      public String minAsString()
      Returns the string representation of min for debugging/logging purposes.
      Returns:
      the min value as a string
    • maxAsString

      public String maxAsString()
      Returns the string representation of max for debugging/logging purposes.
      Returns:
      the max value as a string
    • isSmallerThan

      public abstract boolean isSmallerThan(long size)
      Abstract method to return whether the min and max values fit in the given size.
      Parameters:
      size - a size in bytes
      Returns:
      true iff the min and max values are less than size bytes
    • toString

      public String toString()
      Overrides:
      toString in class Object
    • incrementNumNulls

      public void incrementNumNulls()
      Increments the null count by one
    • incrementNumNulls

      public void incrementNumNulls(long increment)
      Increments the null count by the parameter value
      Parameters:
      increment - value to increment the null count by
    • incrementNanCount

      public void incrementNanCount()
      Increments the NaN count by one. If nan_count was not set (-1), initializes it to 1.
    • incrementNanCount

      public void incrementNanCount(long increment)
      Increments the NaN count by the parameter value. If nan_count was not set (-1), initializes it to increment.
      Parameters:
      increment - value to increment the NaN count by
    • getNanCount

      public long getNanCount()
      Returns the NaN count
      Returns:
      NaN count or -1 if the NaN count is not set
    • isNanCountSet

      public boolean isNanCountSet()
      Returns:
      whether nanCount is set and can be used
    • unsetNanCount

      public void unsetNanCount()
      Unsets the NaN count to -1.
    • getNumNulls

      public long getNumNulls()
      Returns the null count
      Returns:
      null count or -1 if the null count is not set
    • setNumNulls

      @Deprecated public void setNumNulls(long nulls)
      Deprecated.
      will be removed in 2.0.0. Use getBuilderForReading(PrimitiveType) instead.
      Sets the number of nulls to the parameter value
      Parameters:
      nulls - null count to set the count to
    • isEmpty

      public boolean isEmpty()
      Returns a boolean specifying if the Statistics object is empty, i.e does not contain valid statistics for the page/column yet
      Returns:
      true if object is empty, false otherwise
    • hasNonNullValue

      public boolean hasNonNullValue()
      Returns whether there have been non-null values added to this statistics
      Returns:
      true if the values contained at least one non-null value
    • isNumNullsSet

      public boolean isNumNullsSet()
      Returns:
      whether numNulls is set and can be used
    • markAsNotEmpty

      protected void markAsNotEmpty()
      Sets the page/column as having a valid non-null value kind of misnomer here
    • copy

      public abstract Statistics<T> copy()
      Returns:
      a new independent statistics instance of this class.
    • type

      public PrimitiveType type()
      Returns:
      the primitive type object which this statistics is created for