Package org.apache.druid.frame

Class Frame

java.lang.Object
org.apache.druid.frame.Frame
public class Frame extends Object
A data frame. Frames are split into contiguous "regions". With columnar frames (FrameType.isColumnar()) each region is a column. With row-based frames (FrameType.isRowBased()) there are always two regions: row offsets and row data. This object is lightweight. It has constant overhead regardless of the number of rows or regions. Frames wrap some Memory. If the memory is backed by a resource that requires explicit releasing, such as direct off-heap memory or a memory-mapped file, the creator of the Memory is responsible for releasing that resource when the frame is no longer needed. Frames are written with FrameWriter and read with FrameReader. Frame format: - 1 byte: FrameType.version() - 8 bytes: size in bytes of the frame, little-endian long - 4 bytes: number of rows, little-endian int - 4 bytes: number of regions, little-endian int - 1 byte: 0 if frame is nonpermuted, 1 if frame is permuted - 4 bytes x numRows: permutation section; only present for permuted frames. Array of little-endian ints mapping logical row numbers to physical row numbers. - 8 bytes x numRegions: region offsets. Array of end offsets of each region (exclusive), relative to start of frame, as little-endian longs. - NNN bytes: regions, back-to-back. There is also a compressed frame format. Compressed frames are written by writeTo(java.nio.channels.WritableByteChannel, boolean, java.nio.ByteBuffer, org.apache.druid.frame.channel.ByteTracker) when "compress" is true, and decompressed by decompress(org.apache.datasketches.memory.Memory, long, long). Format: - 1 byte: compression type: CompressionStrategy.getId(). Currently, only LZ4 is supported. - 8 bytes: compressed frame length, little-endian long - 8 bytes: uncompressed frame length (numBytes), little-endian long - NNN bytes: LZ4-compressed frame - 8 bytes: 64-bit xxhash checksum of prior content, including 16-byte header and compressed frame, little-endian long Note to developers: if we end up needing to add more fields here, consider introducing a Smile (or Protobuf, etc) header to make it simpler to add more fields.

  • Field Details

    • HEADER_SIZE

      public static final long HEADER_SIZE
      See Also:

    • COMPRESSED_FRAME_HEADER_SIZE

      public static final int COMPRESSED_FRAME_HEADER_SIZE
      See Also:

    • COMPRESSED_FRAME_TRAILER_SIZE

      public static final int COMPRESSED_FRAME_TRAILER_SIZE
      See Also:

    • COMPRESSED_FRAME_ENVELOPE_SIZE

      public static final int COMPRESSED_FRAME_ENVELOPE_SIZE
      See Also:

  • Method Details

    • wrap

      public static Frame wrap(org.apache.datasketches.memory.Memory memory)
      Returns a frame backed by the provided Memory. This operation does not do any copies or allocations. The Memory must be in little-endian byte order. Behavior is undefined if the memory is modified anytime during the lifetime of the Frame object.

    • wrap

      public static Frame wrap(ByteBuffer buffer)
      Returns a frame backed by the provided ByteBuffer. This operation does not do any copies or allocations. The position and limit of the buffer are ignored. If you need them to be respected, call ByteBuffer.slice() first, or use wrap(Memory) to wrap a particular region.

    • wrap

      public static Frame wrap(byte[] bytes)
      Returns a frame backed by the provided byte array. This operation does not do any copies or allocations. The position and limit of the buffer are ignored. If you need them to be respected, call ByteBuffer.slice() first, or use wrap(Memory) to wrap a particular region.

    • decompress

      public static Frame decompress(org.apache.datasketches.memory.Memory memory, long position, long length)
      Decompresses the provided memory and returns a frame backed by that decompressed memory. This operation is safe even on corrupt data: it validates position, length, and checksum prior to decompressing. This operation allocates memory on-heap to store the decompressed frame.

    • compressionBufferSize

      public static int compressionBufferSize(long frameBytes)
      Minimum size of compression buffer required by writeTo(java.nio.channels.WritableByteChannel, boolean, java.nio.ByteBuffer, org.apache.druid.frame.channel.ByteTracker) when "compress" is true.

    • type

      public FrameType type()

    • numBytes

      public long numBytes()

    • numRows

      public int numRows()

    • numRegions

      public int numRegions()

    • isPermuted

      public boolean isPermuted()

    • physicalRow

      public int physicalRow(int logicalRow)
      Maps a logical row number to a physical row number. If the frame is non-permuted, these are the same. If the frame is permuted, this uses the sorted-row mappings to remap the row number.
      Throws:
      IllegalArgumentException - if "logicalRow" is out of bounds

    • region

      public org.apache.datasketches.memory.Memory region(int regionNumber)
      Returns memory corresponding to a particular region of this frame.

    • writableMemory

      public org.apache.datasketches.memory.WritableMemory writableMemory()
      Direct, writable access to this frame's memory. Used by operations that modify the frame in-place, like FrameSort. Most callers should use region(int) and physicalRow(int), rather than this direct-access method.
      Throws:
      IllegalStateException - if this frame wraps non-writable memory

    • writeTo

      public long writeTo(WritableByteChannel channel, boolean compress, @Nullable ByteBuffer compressionBuffer, ByteTracker byteTracker) throws IOException
      Writes this frame to a channel, optionally compressing it as well. Returns the number of bytes written. The provided compressionBuffer is used to hold compressed data temporarily, prior to writing it to the channel. It must be at least as large as compressionBufferSize(numBytes()), or else an IllegalStateException is thrown. It may be null if "compress" is false.
      Throws:
      IOException

    • ensureRowBased

      public Frame ensureRowBased()
      Returns this object if type() is row-based, otherwise throws an error.

    • ensureColumnar

      public Frame ensureColumnar()
      Returns this object if type() is columnar, otherwise throws an error.