All Classes and Interfaces
Class
Description
The purpose of this is to finally generate a bloom filter with the optimal bit size according to the number
of real data distinct values.
Provides ability to chain two filters together.
Deprecated.
Plain encoding reader for BINARY values.
Class for truncating min/max values for binary types.
a column reader that packs the ints in the number of bits required based on the maximum size.
a column writer that packs the ints in the number of bits required based on the maximum size.
A Bloom filter is a compact structure to indicate whether an item is not in a set or probably
in a set.
Contains all writers for all columns of a row group
Decodes PLAIN-encoded booleans: one bit per value, packed 8 per byte, little-endian
bit order (bit 0 of each byte is the first value).
An implementation of the PLAIN encoding for BOOLEAN values.
Enum for
BoundaryOrder.Describes a column's type as well as its position in its containing schema.
Column index containing min/max and null count values for the pages in a column chunk.
Builder implementation to create
ColumnIndex objects.Filter implementation based on column indexes.
Provides the
ColumnIndex and OffsetIndex objects for a row-group.Exception thrown in case of an offset index is missing for any of the columns.
a structure used to serialize deserialize records
Factory constructing the ColumnIO structure from the schema
Class representing the column order with all the related parameters.
The enum type of the column order.
ColumnPredicates class provides checks for column values.
Reader for (repetition level, definition level, values) triplets.
ColumnReader implementation for the scenario when column indexes are not used (all values are read)
Container which can produce a ColumnReader for any given column in a schema.
Implementation of the ColumnReadStore
Record filter which applies the supplied predicate to the specified column.
writer for (repetition level, definition level, values) triplets
Container which can construct writers for multiple columns to be stored
together.
thrown when a problem occured while compiling the column reader
Recursively rewrites Contains predicates composed using And or Or into a single Contains predicate
containing all predicate assertions.
Utility functions to convert from Java-like map and list types
to equivalent Parquet types.
Represent a tree of converters
that materializes tuples
There was a bug (https://github.com/apache/parquet-java/issues/1433) that caused the statistics metadata
for binary columns to be corrupted in the write path.
one data page in a chunk
Deprecated.
Handles ValuesWriter creation statically based on the types of the columns and the writer version.
Read values written by
DeltaBinaryPackingValuesWriterWrite integers with delta encoding and binary packing
The format is as follows:
Write integers (INT32) with delta encoding and binary packing.
Write longs (INT64) with delta encoding and binary packing.
Reads binary data written by
DeltaByteArrayWriterWrite prefix lengths using delta encoding, followed by suffixes with Delta length byte arrays
Reads binary data written by
DeltaLengthByteArrayValuesWriterWrite lengths of byte-arrays using delta encoding, followed by concatenated byte-arrays
This is a special writer that doesn't write anything.
a dictionary to decode dictionary based encodings
Data for a dictionary page
Interface to read dictionary pages for all the columns of a row group
Reads values that have been dictionary encoded
Will attempt to encode values using a dictionary and fall back to plain encoding
if the dictionary gets too big
Dummy implementation for perf tests
Edge interpolation algorithm for Geography logical type
encoding of the data
EncodingStats track dictionary and data page encodings for a single column within a row group.
Used to build
EncodingStats from metadata or to accumulate stats as pages are written.The Filter API is expressed through these static methods.
Parquet currently has two ways to specify a filter for dropping records at read time.
Anyone wanting to use a
FilterCompat.Filter need only implement this interface,
per the visitor pattern.A pass-through proxy for a
RecordMaterializer that updates a IncrementallyUpdatedFilterPredicate
as it receives concrete values for the current record.A FilterPredicate is an expression tree describing the criteria for which records to keep when loading data from
a parquet file.
A FilterPredicate Visitor must visit all the operators in a FilterPredicate expression tree,
and must handle recursion itself, per the visitor pattern.
ValuesReader for FIXED_LEN_BYTE_ARRAY.
ValuesWriter for FIXED_LEN_BYTE_ARRAY.
The class is a utility class to manipulate half-precision 16-bit
IEEE 754
floating point data types (also called fp16 or binary16).
A structure for capturing metadata for estimating the unencoded,
uncompressed size of geospatial data written.
Builder to create a GeospatialStatistics.
Group level of the IO structure
converter for group nodes
Represents a group type: a list of fields
A interface contains a set of hash functions used by Bloom filter.
thrown when we are trying to read together files with incompatible schemas.
A rewritten version of a
FilterPredicate which receives
the values for a record's columns one by one and internally tracks whether the predicate is
satisfied, unsatisfied, or unknown.A ValueInspector implementation that keeps state for one or more delegate inspectors.
This is the leaf node of a filter predicate.
A Visitor for an
IncrementallyUpdatedFilterPredicate, per the visitor pattern.This class is auto-generated by org.apache.parquet.filter2.IncrementallyUpdatedFilterPredicateGenerator
Do not manually edit!
The implementation of this abstract class is auto-generated by
org.apache.parquet.filter2.IncrementallyUpdatedFilterPredicateGenerator
Determines whether an
IncrementallyUpdatedFilterPredicate is satisfied or not.Resets all the
IncrementallyUpdatedFilterPredicate.ValueInspectors in a IncrementallyUpdatedFilterPredicate.Iterator implementation for page indexes.
An append-only integer list
avoids autoboxing and buffer resizing
to iterate on the content of the list
not an actual iterator to avoid autoboxing
thrown when an invalid record is encountered
Recursively removes all use of the not() operator in a predicate
by replacing all instances of not(x) with the inverse(x),
eg: not(and(eq(), not(eq(y))) -> or(notEq(), eq(y))
Converts a
FilterPredicate to its logical inverse.Implement this interface to visit a logical type annotation in the schema.
Message level of the IO structure
The root of a schema
Parses a schema from a textual format similar to that described in the Dremel paper.
This class calculates the max and min values of an iterable collection.
Provides ability to negate the result of a filter.
Offset index containing the offset and size of the page and the index of the first row in the page.
Builder implementation to create
OffsetIndex objects during writing a parquet file.These are the operators in a filter predicate expression tree.
Base class for
Operators.In and Operators.NotIn.Operators.UserDefinedByInstance<T extends Comparable<T>,U extends UserDefinedPredicate<T> & Serializable>
Deprecated.
use
LogicalTypeAnnotation to represent logical types insteadProvides ability to chain two filters together.
one page in a chunk
Filter which will only materialize a page worth of results.
Reader for a sequence a page from a given column chunk
contains all the readers for all the columns of the corresponding row group
a writer for all the pages of a given column chunk
contains all the writers for the columns in the corresponding row group
Examples from the Dremel Paper
thrown when an encoding problem occured
thrown when a decoding problem occured
This class represents all the configurable Parquet properties.
a simple implementation of dictionary for plain encoded values
a simple implementation of dictionary for plain encoded binary
a simple implementation of dictionary for plain encoded boolean values
a simple implementation of dictionary for plain encoded double values
a simple implementation of dictionary for plain encoded float values
a simple implementation of dictionary for plain encoded integer values
a simple implementation of dictionary for plain encoded long values
Plain encoding for float, double, int, long.
Plain encoding except for booleans
Primitive level of the IO structure
Comparator implementation that also supports the comparison of the related primitive type to avoid the
performance penalty of boxing/unboxing.converter for leaves of the schema
Class that provides string representations for the primitive values.
Converts a
Class<primitive> to it's corresponding Class<Boxed>, eg
Class<int> to Class<Integer>Representation of a Primitive type
Supported Primitive types
Abstraction for writing records
It decouples the striping algorithm from the actual record model
example:
This class can be used to wrap an actual RecordConsumer and log all calls
Filter to be applied to a record to work out whether to skip it.
Top-level class which should be implemented in order to materialize objects from
a stream of Parquet data.
This exception signals that the current record is cannot be converted from parquet columns to a materialized
record, but can be skipped if requested.
used to read reassembled records
Used to add extra behavior to a ValuesWriter that requires fallback
Class representing row ranges in a row-group.
Deprecated.
moved to
RowRanges.Constructs a
RowRanges by appending selected row indices in strictly increasing
order.Decodes values written in the grammar described in
RunLengthBitPackingHybridEncoderEncodes values using a combination of run length encoding and bit packing,
according to the following grammar:
This ValuesReader does all the reading in
RunLengthBitPackingHybridValuesReader.initFromPage(int, org.apache.parquet.bytes.ByteBufferInputStream)
and stores the values in an in memory buffer, which is less than ideal.Inspects the column types found in the provided
FilterPredicate and compares them
to the actual schema found in the parquet file.A structure for capturing metadata for estimating the unencoded,
uncompressed size of data written.
Builder to create a SizeStatistics.
Statistics class to keep track of statistics in parquet pages and column chunks
Contains statistics about a group of records
Builder class to build Statistics objects.
Thrown if the two Statistics objects have mismatching types
Represents the declared type for a field in a schema.
represents a field ID
Constraint on the repetition of a field
to convert a MessageType tree
This class provides fluent builders that produce Parquet schema Types.
A base builder for
Type objects.A builder for
GroupType objects.A builder for
PrimitiveType objects.Implement this interface to visit a schema
Builder for a record filter.
Thrown if the specified column is unknown in the underlying storage
Thrown if the specified column type is unknown in the underlying storage
A UserDefinedPredicate decides whether a record should be kept or dropped, first by
inspecting meta data about a group of records to see if the entire group can be dropped,
then by inspecting actual values of a single column.
Wraps a record consumer
Validates the record written against the schema and pass down the event to the wrapped consumer
Contains all valid mappings from class -> parquet type (and vice versa) for use in
FilterPredicatesBase class to implement an encoding for a given column type.
The different type of values we can store in columns
base class to implement an encoding for a given column
Can be overridden to allow users to manually test different strategies to create ValuesWriters.
The implementation of HashFunction interface.
ColumnReader which does not read any actual data, but rather simply produces
an endless stream of constant values.
LogicalTypeAnnotation.DecimalLogicalTypeAnnotationwith proper precision and scale parameters instead