Class DeltaBinaryPackingValuesWriter
java.lang.Object
org.apache.parquet.column.values.ValuesWriter
org.apache.parquet.column.values.delta.DeltaBinaryPackingValuesWriter
- All Implemented Interfaces:
AutoCloseable
- Direct Known Subclasses:
DeltaBinaryPackingValuesWriterForInteger,DeltaBinaryPackingValuesWriterForLong
Write integers with delta encoding and binary packing
The format is as follows:
delta-binary-packing: <page-header> <block>*
page-header := <block size in values> <number of miniblocks in a block> <total value count> <first value>
block := <min delta> <list of bitwidths of miniblocks> <miniblocks>
min delta : zig-zag var int encoded
bitWidthsOfMiniBlock : 1 byte little endian
blockSizeInValues,blockSizeInValues,totalValueCount,firstValue : unsigned varint
The algorithm and format is inspired by D. Lemire's paper: http://lemire.me/blog/archives/2012/09/12/fast-integer-compression-decoding-billions-of-integers-per-second/
-
Field Summary
FieldsModifier and TypeFieldDescriptionprotected final org.apache.parquet.bytes.CapacityByteArrayOutputStreamprotected final int[]bit width for each mini block, reused between flushesprotected final org.apache.parquet.column.values.delta.DeltaBinaryPackingConfigstores blockSizeInValues, miniBlockNumInABlock and miniBlockSizeInValuesstatic final intstatic final intprotected inta pointer to deltaBlockBuffer indicating the end of deltaBlockBuffer the number of values in the deltaBlockBuffer that haven't flushed to baos it will be reset after each flushprotected byte[]bytes buffer for a mini block, it is reused for each mini block.protected int -
Constructor Summary
ConstructorsConstructorDescriptionDeltaBinaryPackingValuesWriter(int blockSizeInValues, int miniBlockNum, int slabSize, int pageSize, org.apache.parquet.bytes.ByteBufferAllocator allocator) DeltaBinaryPackingValuesWriter(int slabSize, int pageSize, org.apache.parquet.bytes.ByteBufferAllocator allocator) -
Method Summary
Modifier and TypeMethodDescriptionvoidclose()Called to close the values writer.longlongused to decide if we want to work to the next pagecalled after getBytes() and before reset()protected intgetMiniBlockCountToFlush(double numberCount) memUsageString(String prefix) voidreset()called after getBytes() to reset the current buffer and start writing the next pageprotected voidwriteBitWidthForMiniBlock(int i) Methods inherited from class org.apache.parquet.column.values.ValuesWriter
getBytes, resetDictionary, toDictPageAndClose, writeBoolean, writeByte, writeBytes, writeDouble, writeFloat, writeInteger, writeLong
-
Field Details
-
DEFAULT_NUM_BLOCK_VALUES
public static final int DEFAULT_NUM_BLOCK_VALUES- See Also:
-
DEFAULT_NUM_MINIBLOCKS
public static final int DEFAULT_NUM_MINIBLOCKS- See Also:
-
baos
protected final org.apache.parquet.bytes.CapacityByteArrayOutputStream baos -
config
protected final org.apache.parquet.column.values.delta.DeltaBinaryPackingConfig configstores blockSizeInValues, miniBlockNumInABlock and miniBlockSizeInValues -
bitWidths
protected final int[] bitWidthsbit width for each mini block, reused between flushes -
totalValueCount
protected int totalValueCount -
deltaValuesToFlush
protected int deltaValuesToFlusha pointer to deltaBlockBuffer indicating the end of deltaBlockBuffer the number of values in the deltaBlockBuffer that haven't flushed to baos it will be reset after each flush -
miniBlockByteBuffer
protected byte[] miniBlockByteBufferbytes buffer for a mini block, it is reused for each mini block. Therefore the size of biggest miniblock with bitwidth of MAX_BITWIDTH is allocated
-
-
Constructor Details
-
DeltaBinaryPackingValuesWriter
public DeltaBinaryPackingValuesWriter(int slabSize, int pageSize, org.apache.parquet.bytes.ByteBufferAllocator allocator) -
DeltaBinaryPackingValuesWriter
public DeltaBinaryPackingValuesWriter(int blockSizeInValues, int miniBlockNum, int slabSize, int pageSize, org.apache.parquet.bytes.ByteBufferAllocator allocator)
-
-
Method Details
-
getBufferedSize
public long getBufferedSize()Description copied from class:ValuesWriterused to decide if we want to work to the next page- Specified by:
getBufferedSizein classValuesWriter- Returns:
- the size of the currently buffered data (in bytes)
-
writeBitWidthForMiniBlock
protected void writeBitWidthForMiniBlock(int i) -
getMiniBlockCountToFlush
protected int getMiniBlockCountToFlush(double numberCount) -
getEncoding
Description copied from class:ValuesWritercalled after getBytes() and before reset()- Specified by:
getEncodingin classValuesWriter- Returns:
- the encoding that was used to encode the bytes
-
reset
public void reset()Description copied from class:ValuesWritercalled after getBytes() to reset the current buffer and start writing the next page- Specified by:
resetin classValuesWriter
-
close
public void close()Description copied from class:ValuesWriterCalled to close the values writer. Any output stream is closed and can no longer be used. All resources are released.- Specified by:
closein interfaceAutoCloseable- Overrides:
closein classValuesWriter
-
getAllocatedSize
public long getAllocatedSize()Description copied from class:ValuesWriter- Specified by:
getAllocatedSizein classValuesWriter- Returns:
- the allocated size of the buffer
-
memUsageString
- Specified by:
memUsageStringin classValuesWriter
-