Package org.apache.druid.frame.file

Class FrameFile

java.lang.Object
org.apache.druid.frame.file.FrameFile
All Implemented Interfaces:
Closeable, AutoCloseable
public class FrameFile extends Object implements Closeable
A file containing Frame data. Frame files are written by FrameFileWriter. Frame files can optionally be partitioned, by providing partition numbers to the FrameFileWriter.write(org.apache.druid.query.rowsandcols.RowsAndColumns, int) method when creating the file. Partitions are contiguous within the frame file. Frame files can contain up to Integer.MAX_VALUE frames. Generally, frames are on the order of 1 MB in size, so this allows well over a petabyte of data per file. Ought to be enough for anyone. Format: - 2 bytes: FrameFileWriter.MAGIC - NNN bytes: sequence of FrameFileWriter.MARKER_FRAME followed by one compressed frame (see Frame) - 1 byte: FrameFileWriter.MARKER_NO_MORE_FRAMES - 4 bytes * numPartitions: end frame number of each partition (exclusive), as little-endian ints. Note that partitions may be empty. In this case, certain adjacent values in this array will be equal. Only present if the file is partitioned. - 8 bytes * numFrames: end of each compressed frame (exclusive), relative to start of file, as little-endian longs - 4 bytes: number of frames, as little-endian int - 4 bytes: number of partitions, as little-endian int - 4 bytes: length of footer, from FrameFileWriter.MARKER_NO_MORE_FRAMES to EOF - 4 bytes: checksum of footer (xxhash64, truncated to 32 bits), not considering these final 4 bytes Instances of this class are not thread-safe. For sharing across threads, use newReference() to create an additional reference.
See Also:

  • Method Details

    • open

      public static FrameFile open(File file, @Nullable ByteTracker byteTracker, FrameFile.Flag... flags) throws IOException
      Open a frame file with certain optional flags.
      Parameters:
      file - ƒrame file
      byteTracker -
      flags - optional flags
      Throws:
      IOException

    • numFrames

      public int numFrames()
      Number of frames in the file.

    • numPartitions

      public int numPartitions()
      Number of partitions in the file, or zero if the file is unpartitioned.

    • getPartitionStartFrame

      public int getPartitionStartFrame(int partition)
      First frame of a given partition. Partitions beyond numPartitions() are treated as empty: if provided, this method returns numFrames().

    • rac

      public RowsAndColumns rac(int batchNumber, @Nullable WireTransferable.ConcreteDeserializer deserializer)
      Reads a RowsAndColumns from the file.
      Parameters:
      batchNumber - the batch number to read. Starts at 0, ends at numFrames() (exclusive)
      deserializer - deserializer for WireTransferable format
      Returns:
      the RowsAndColumns

    • newReference

      public FrameFile newReference()
      Creates a new reference to this file. Calling close() releases the reference. The original file is closed when it, and all additional references, are closed. The new FrameFile instance may be used concurrently with the original FrameFile instance.

    • file

      public File file()
      Returns the file that this instance is backed by.

    • close

      public void close() throws IOException
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException