Package org.kairosdb.bigqueue

Class BigArrayImpl

  • All Implemented Interfaces:
    Closeable, AutoCloseable, IBigArray
    public class BigArrayImpl
extends Object
implements IBigArray
    A big array implementation supporting sequential append and random read. Main features: 1. FAST : close to the speed of direct memory access, extremely fast in append only and sequential read modes, sequential append and read are close to O(1) memory access, random read is close to O(1) memory access if data is in cache and is close to O(1) disk access if data is not in cache. 2. MEMORY-EFFICIENT : automatic paging and swapping algorithm, only most-recently accessed data is kept in memory. 3. THREAD-SAFE : multiple threads can concurrently read/append the array without data corruption. 4. PERSISTENT - all array data is persisted on disk, and is crash resistant. 5. BIG(HUGE) - the total size of the array data is only limited by the available disk space.
    Author:
    bulldog

    • Constructor Summary

      Constructors 
      Constructor Description
      BigArrayImpl​(String arrayDir, String arrayName)
      A big array implementation supporting sequential write and random read, use default back data file size per page, see DEFAULT_DATA_PAGE_SIZE.
      BigArrayImpl​(String arrayDir, String arrayName, int pageSize)
      A big array implementation supporting sequential write and random read.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      long append​(byte[] data)
      Append the data into the head of the array
      void close()  
      long findClosestIndex​(long timestamp)
      Find an index closest to the specific timestamp when the corresponding item was appended
      void flush()
      Force to persist newly appended data, normally, you don't need to flush explicitly since: 1.)
      byte[] get​(long index)
      Get the data at specific index
      String getArrayDirectory()  
      long getBackFileSize()
      Get total size of back files(index and data files) of the big array
      int getDataPageSize()
      Get the back data file size per page.
      long getHeadIndex()
      The head of the array.
      int getItemLength​(long index)
      Get the data item length at specific index
      long getTailIndex()
      The tail of the array.
      long getTimestamp​(long index)
      Get the timestamp of data at specific index, this is the timestamp when the data was appended.
      boolean isEmpty()
      Check if the array is empty or not
      boolean isFull()
      Check if the ring space of java long type has all been used up.
      void limitBackFileSize​(long sizeLimit)
      limit the back file size, truncate back file and advance array tail index accordingly, Note, this is a best effort call, exact size limit can't be guaranteed
      void removeAll()
      Remove all data in this array, this will empty the array and delete all back page files.
      void removeBefore​(long timestamp)
      Remove all data before specific timestamp, this will advance the array tail and delete back page files accordingly.
      void removeBeforeIndex​(long index)
      Remove all data before specific index, this will advance the array tail to index and delete back page files before index.
      void setClock​(Clock clock)
      Used for changing the clock for unit tests
      long size()
      The total number of items has been appended into the array

    • Constructor Detail

      • BigArrayImpl

        public BigArrayImpl​(String arrayDir,
                    String arrayName)
             throws IOException
        A big array implementation supporting sequential write and random read, use default back data file size per page, see DEFAULT_DATA_PAGE_SIZE.
        Parameters:
        arrayDir - directory for array data store
        arrayName - the name of the array, will be appended as last part of the array directory
        Throws:
        IOException - exception throws during array initialization

      • BigArrayImpl

        public BigArrayImpl​(String arrayDir,
                    String arrayName,
                    int pageSize)
             throws IOException
        A big array implementation supporting sequential write and random read.
        Parameters:
        arrayDir - directory for array data store
        arrayName - the name of the array, will be appended as last part of the array directory
        pageSize - the back data file size per page in bytes, see minimum allowed MINIMUM_DATA_PAGE_SIZE.
        Throws:
        IOException - exception throws during array initialization

    • Method Detail

      • setClock

        public void setClock​(Clock clock)
        Used for changing the clock for unit tests
        Parameters:
        clock - Clock instance to use

      • getArrayDirectory

        public String getArrayDirectory()

      • removeAll

        public void removeAll()
               throws IOException
        Description copied from interface: IBigArray
        Remove all data in this array, this will empty the array and delete all back page files.
        Specified by:
        removeAll in interface IBigArray
        Throws:
        IOException - exception thrown if IO error occurs durring remove all

      • removeBeforeIndex

        public void removeBeforeIndex​(long index)
                       throws IOException
        Description copied from interface: IBigArray
        Remove all data before specific index, this will advance the array tail to index and delete back page files before index.
        Specified by:
        removeBeforeIndex in interface IBigArray
        Parameters:
        index - an index
        Throws:
        IOException - exception thrown if there was any IO error during the removal operation

      • removeBefore

        public void removeBefore​(long timestamp)
                  throws IOException
        Description copied from interface: IBigArray
        Remove all data before specific timestamp, this will advance the array tail and delete back page files accordingly.
        Specified by:
        removeBefore in interface IBigArray
        Parameters:
        timestamp - a timestamp
        Throws:
        IOException - exception thrown if there was any IO error during the removal operation

      • append

        public long append​(byte[] data)
            throws IOException
        Append the data into the head of the array
        Specified by:
        append in interface IBigArray
        Parameters:
        data - binary data to append
        Returns:
        appended index
        Throws:
        IOException - if there is any IO error

      • flush

        public void flush()
        Description copied from interface: IBigArray
        Force to persist newly appended data, normally, you don't need to flush explicitly since: 1.) BigArray will automatically flush a cached page when it is replaced out, 2.) BigArray uses memory mapped file technology internally, and the OS will flush the changes even your process crashes, call this periodically only if you need transactional reliability and you are aware of the cost to performance.
        Specified by:
        flush in interface IBigArray

      • get

        public byte[] get​(long index)
           throws IOException
        Description copied from interface: IBigArray
        Get the data at specific index
        Specified by:
        get in interface IBigArray
        Parameters:
        index - valid data index
        Returns:
        binary data if the index is valid
        Throws:
        IOException - if there is any IO error

      • getTimestamp

        public long getTimestamp​(long index)
                  throws IOException
        Description copied from interface: IBigArray
        Get the timestamp of data at specific index, this is the timestamp when the data was appended.
        Specified by:
        getTimestamp in interface IBigArray
        Parameters:
        index - valid data index
        Returns:
        timestamp when the data was appended
        Throws:
        IOException - if there is any IO error

      • size

        public long size()
        Description copied from interface: IBigArray
        The total number of items has been appended into the array
        Specified by:
        size in interface IBigArray
        Returns:
        total number

      • getHeadIndex

        public long getHeadIndex()
        Description copied from interface: IBigArray
        The head of the array. This is the next to append index, the index of the last appended data is [headIndex - 1] if the array is not empty.
        Specified by:
        getHeadIndex in interface IBigArray
        Returns:
        an index

      • getTailIndex

        public long getTailIndex()
        Description copied from interface: IBigArray
        The tail of the array. The is the index of the first appended data
        Specified by:
        getTailIndex in interface IBigArray
        Returns:
        an index

      • isEmpty

        public boolean isEmpty()
        Description copied from interface: IBigArray
        Check if the array is empty or not
        Specified by:
        isEmpty in interface IBigArray
        Returns:
        true if empty false otherwise

      • isFull

        public boolean isFull()
        Description copied from interface: IBigArray
        Check if the ring space of java long type has all been used up. can always assume false, if true, the world is end:)
        Specified by:
        isFull in interface IBigArray
        Returns:
        array full or not

      • getDataPageSize

        public int getDataPageSize()
        Description copied from interface: IBigArray
        Get the back data file size per page.
        Specified by:
        getDataPageSize in interface IBigArray
        Returns:
        size per page

      • findClosestIndex

        public long findClosestIndex​(long timestamp)
                      throws IOException
        Description copied from interface: IBigArray
        Find an index closest to the specific timestamp when the corresponding item was appended
        Specified by:
        findClosestIndex in interface IBigArray
        Parameters:
        timestamp - when the corresponding item was appended
        Returns:
        an index
        Throws:
        IOException - exception thrown if there was any IO error during the getClosestIndex operation

      • getBackFileSize

        public long getBackFileSize()
                     throws IOException
        Description copied from interface: IBigArray
        Get total size of back files(index and data files) of the big array
        Specified by:
        getBackFileSize in interface IBigArray
        Returns:
        total size of back files
        Throws:
        IOException - exception thrown if there was any IO error during the getBackFileSize operation

      • limitBackFileSize

        public void limitBackFileSize​(long sizeLimit)
                       throws IOException
        Description copied from interface: IBigArray
        limit the back file size, truncate back file and advance array tail index accordingly, Note, this is a best effort call, exact size limit can't be guaranteed
        Specified by:
        limitBackFileSize in interface IBigArray
        Parameters:
        sizeLimit - the size to limit
        Throws:
        IOException - exception thrown if there was any IO error during the limitBackFileSize operation

      • getItemLength

        public int getItemLength​(long index)
                  throws IOException
        Description copied from interface: IBigArray
        Get the data item length at specific index
        Specified by:
        getItemLength in interface IBigArray
        Parameters:
        index - valid data index
        Returns:
        the length of binary data if the index is valid
        Throws:
        IOException - if there is any IO error