Package org.apache.druid.segment.data

Class CompressionFactory

java.lang.Object
org.apache.druid.segment.data.CompressionFactory
public class CompressionFactory extends Object
Compression of metrics is done by using a combination of CompressionStrategy and Encoding(such as CompressionFactory.LongEncodingStrategy for type Long). CompressionStrategy is unaware of the data type and is based on byte operations. It must compress and decompress in block of bytes. Encoding refers to compression method relies on data format, so a different set of Encodings exist for each data type.

Storage Format : Byte 1 : version (currently 0x02) Byte 2 - 5 : number of values Byte 6 - 9 : size per block (even if block format isn't used, this is needed for backward compatibility) Byte 10 : compression strategy (contains a flag if there's an encoding byte, see below for how the flag is defined) Byte 11(optional) : encoding type

Encoding specific header (described below)

Block related header (if block compression is used, described in GenericIndexed)

Values

  • Field Details

    • DEFAULT_LONG_ENCODING_STRATEGY

      public static final CompressionFactory.LongEncodingStrategy DEFAULT_LONG_ENCODING_STRATEGY

    • LEGACY_LONG_ENCODING_FORMAT

      public static final CompressionFactory.LongEncodingFormat LEGACY_LONG_ENCODING_FORMAT

    • DELTA_ENCODING_VERSION

      public static final byte DELTA_ENCODING_VERSION
      Delta Encoding Header v1: Byte 1 : version Byte 2 - 9 : base value Byte 10 - 13 : number of bits per value
      See Also:

    • TABLE_ENCODING_VERSION

      public static final byte TABLE_ENCODING_VERSION
      Table Encoding Header v1 : Byte 1 : version Byte 2 - 5 : table size Byte 6 - (6 + 8 * table size - 1) : table of encoding, where the ith 8-byte value is encoded as i
      See Also:

    • MAX_TABLE_SIZE

      public static final int MAX_TABLE_SIZE
      See Also:

  • Method Details

    • hasEncodingFlag

      public static boolean hasEncodingFlag(byte strategyId)

    • setEncodingFlag

      public static byte setEncodingFlag(byte strategyId)

    • clearEncodingFlag

      public static byte clearEncodingFlag(byte strategyId)

    • getLongSupplier

      public static com.google.common.base.Supplier<ColumnarLongs> getLongSupplier(int totalSize, int sizePer, ByteBuffer fromBuffer, ByteOrder order, CompressionFactory.LongEncodingFormat encodingFormat, CompressionStrategy strategy, @Nullable SmooshedFileMapper smooshMapper)
      Reads a column from a ByteBuffer, possibly using additional secondary files from a SmooshedFileMapper.
      Parameters:
      totalSize - number of rows in the column
      sizePer - number of values per compression buffer, for compressed columns
      fromBuffer - primary buffer to read from
      order - byte order
      encodingFormat - encoding of each long value
      strategy - compression strategy, for compressed columns
      smooshMapper - required for reading version 2 (multi-file) indexed. May be null if you know you are reading a single-file column. Generally, this should only be null in tests, not production code.

    • getLongSerializer

      public static ColumnarLongsSerializer getLongSerializer(String columnName, SegmentWriteOutMedium segmentWriteOutMedium, String filenameBase, ByteOrder order, CompressionFactory.LongEncodingStrategy encodingStrategy, CompressionStrategy compressionStrategy, Closer closer)

    • getFloatSupplier

      public static com.google.common.base.Supplier<ColumnarFloats> getFloatSupplier(int totalSize, int sizePer, ByteBuffer fromBuffer, ByteOrder order, CompressionStrategy strategy, @Nullable SmooshedFileMapper smooshMapper)
      Reads a column from a ByteBuffer, possibly using additional secondary files from a SmooshedFileMapper.
      Parameters:
      totalSize - number of rows in the column
      sizePer - number of values per compression buffer, for compressed columns
      fromBuffer - primary buffer to read from
      order - byte order
      strategy - compression strategy, for compressed columns
      smooshMapper - required for reading version 2 (multi-file) indexed. May be null if you know you are reading a single-file column. Generally, this should only be null in tests, not production code.

    • getFloatSerializer

      public static ColumnarFloatsSerializer getFloatSerializer(String columnName, SegmentWriteOutMedium segmentWriteOutMedium, String filenameBase, ByteOrder order, CompressionStrategy compressionStrategy, Closer closer)

    • getDoubleSupplier

      public static com.google.common.base.Supplier<ColumnarDoubles> getDoubleSupplier(int totalSize, int sizePer, ByteBuffer fromBuffer, ByteOrder byteOrder, CompressionStrategy strategy, SmooshedFileMapper smooshMapper)
      Reads a column from a ByteBuffer, possibly using additional secondary files from a SmooshedFileMapper.
      Parameters:
      totalSize - number of rows in the column
      sizePer - number of values per compression buffer, for compressed columns
      fromBuffer - primary buffer to read from
      byteOrder - byte order
      strategy - compression strategy, for compressed columns
      smooshMapper - required for reading version 2 (multi-file) indexed. May be null if you know you are reading a single-file column. Generally, this should only be null in tests, not production code.

    • getDoubleSerializer

      public static ColumnarDoublesSerializer getDoubleSerializer(String columnName, SegmentWriteOutMedium segmentWriteOutMedium, String filenameBase, ByteOrder byteOrder, CompressionStrategy compression, Closer closer)