Class AdaptiveBlockSplitBloomFilter

java.lang.Object
org.apache.parquet.column.values.bloomfilter.AdaptiveBlockSplitBloomFilter
All Implemented Interfaces:
BloomFilter

public class AdaptiveBlockSplitBloomFilter extends Object implements BloomFilter
The purpose of this is to finally generate a bloom filter with the optimal bit size according to the number of real data distinct values. `AdaptiveBlockSplitBloomFilter` contains multiple `BlockSplitBloomFilter` as candidates and inserts values in the candidates at the same time. Some candidates that are too small will be eliminated during the insertion process. Finally we will choose the most appropriate size candidate to write out.
  • Constructor Details

    • AdaptiveBlockSplitBloomFilter

      public AdaptiveBlockSplitBloomFilter(int maximumBytes, int numCandidates, double fpp, ColumnDescriptor column)
      Parameters:
      maximumBytes - the maximum bytes size of candidate
      numCandidates - the number of candidates
      fpp - the false positive probability
    • AdaptiveBlockSplitBloomFilter

      public AdaptiveBlockSplitBloomFilter(int maximumBytes, BloomFilter.HashStrategy hashStrategy, double fpp, int numCandidates, ColumnDescriptor column)
  • Method Details

    • optimalCandidate

      Used at the end of the insertion, select the candidate of the smallest size. At least one of the largest candidates will be kept when inserting data.
      Returns:
      the smallest and optimal candidate
    • getCandidates

    • writeTo

      public void writeTo(OutputStream out) throws IOException
      Description copied from interface: BloomFilter
      Write the Bloom filter to an output stream. It writes the Bloom filter header including the bitset's length in bytes, the hash strategy, the algorithm, and the bitset.
      Specified by:
      writeTo in interface BloomFilter
      Parameters:
      out - the output stream to write
      Throws:
      IOException
    • insertHash

      public void insertHash(long hash)
      Insert an element to the multiple bloom filter candidates and remove the bad candidate if the number of distinct values exceeds its expected size.
      Specified by:
      insertHash in interface BloomFilter
      Parameters:
      hash - the hash result of element.
    • getBitsetSize

      public int getBitsetSize()
      Description copied from interface: BloomFilter
      Get the number of bytes for bitset in this Bloom filter.
      Specified by:
      getBitsetSize in interface BloomFilter
      Returns:
      The number of bytes for bitset in this Bloom filter.
    • findHash

      public boolean findHash(long hash)
      Description copied from interface: BloomFilter
      Determine whether an element is in set or not.
      Specified by:
      findHash in interface BloomFilter
      Parameters:
      hash - the hash value of element plain encoding result.
      Returns:
      false if element is must not in set, true if element probably in set.
    • hash

      public long hash(Object value)
      Description copied from interface: BloomFilter
      Compute hash for Object value by using its plain encoding result.
      Specified by:
      hash in interface BloomFilter
      Parameters:
      value - the value to hash
      Returns:
      hash result
    • getHashStrategy

      public BloomFilter.HashStrategy getHashStrategy()
      Description copied from interface: BloomFilter
      Return the hash strategy that the bloom filter apply.
      Specified by:
      getHashStrategy in interface BloomFilter
      Returns:
      hash strategy that the bloom filter apply
    • getAlgorithm

      public BloomFilter.Algorithm getAlgorithm()
      Description copied from interface: BloomFilter
      Return the algorithm that the bloom filter apply.
      Specified by:
      getAlgorithm in interface BloomFilter
      Returns:
      algorithm that the bloom filter apply
    • getCompression

      public BloomFilter.Compression getCompression()
      Description copied from interface: BloomFilter
      Return the compress algorithm that the bloom filter apply.
      Specified by:
      getCompression in interface BloomFilter
      Returns:
      compress algorithm that the bloom filter apply
    • hash

      public long hash(int value)
      Description copied from interface: BloomFilter
      Compute hash for int value by using its plain encoding result.
      Specified by:
      hash in interface BloomFilter
      Parameters:
      value - the value to hash
      Returns:
      hash result
    • hash

      public long hash(long value)
      Description copied from interface: BloomFilter
      Compute hash for long value by using its plain encoding result.
      Specified by:
      hash in interface BloomFilter
      Parameters:
      value - the value to hash
      Returns:
      hash result
    • hash

      public long hash(double value)
      Description copied from interface: BloomFilter
      Compute hash for double value by using its plain encoding result.
      Specified by:
      hash in interface BloomFilter
      Parameters:
      value - the value to hash
      Returns:
      hash result
    • hash

      public long hash(float value)
      Description copied from interface: BloomFilter
      Compute hash for float value by using its plain encoding result.
      Specified by:
      hash in interface BloomFilter
      Parameters:
      value - the value to hash
      Returns:
      hash result
    • hash

      public long hash(Binary value)
      Description copied from interface: BloomFilter
      Compute hash for Binary value by using its plain encoding result.
      Specified by:
      hash in interface BloomFilter
      Parameters:
      value - the value to hash
      Returns:
      hash result