org.apache.hudi.common.bloom

Class InternalBloomFilter

java.lang.Object

org.apache.hudi.common.bloom.InternalBloomFilter

public class InternalBloomFilter
extends Object

Implements a Bloom filter, as defined by Bloom in 1970.

The code in class is adapted from org.apache.hadoop.util.bloom.BloomFilter in Apache Hadoop. The serialization and deserialization are completely the same as and compatible with Hadoop's org.apache.hadoop.util.bloom.BloomFilter, so that this class correctly reads bloom filters serialized by older Hudi versions using Hadoop's BloomFilter.

Hudi serializes bloom filter(s) and write them to Parquet file footers and metadata table's bloom filter partition containing bloom filters for all data files. We want to maintain the serde of a bloom filter and thus the code in Hudi repo to avoid breaking changes in storage format and bytes.

The Bloom filter is a data structure that was introduced in 1970 and that has been adopted by the networking research community in the past decade thanks to the bandwidth efficiencies that it offers for the transmission of set membership information between networked hosts. A sender encodes the information into a bit vector, the Bloom filter, that is more compact than a conventional representation. Computation and space costs for construction are linear in the number of elements. The receiver uses the filter to test whether various elements are members of the set. Though the filter will occasionally return a false positive, it will never return a false negative. When creating the filter, the sender can choose its desired point in a trade-off between the false positive rate and the size.

Originally created by European Commission One-Lab Project 034819.

See Also:

Space/Time Trade-Offs in Hash Coding with Allowable Errors

Field Summary

Fields

Modifier and Type	Field and Description
protected HashFunction	hash The hash function used to map a key to several positions in the vector.
protected int	hashType Type of hashing function to use.
protected int	nbHash The number of hash function to consider.
protected int	vectorSize The vector size of this filter.

Constructor Summary

Constructors

Constructor and Description
InternalBloomFilter() Default constructor - use with readFields
InternalBloomFilter(int vectorSize, int nbHash, int hashType) Constructor

Method Summary

Modifier and Type	Method and Description
void	add(Collection<Key> keys) Adds a collection of keys to this filter.
void	add(Key key) Adds a key to this filter.
void	add(Key[] keys) Adds an array of keys to this filter.
void	add(List<Key> keys) Adds a list of keys to this filter.
void	and(org.apache.hudi.common.bloom.InternalFilter filter) Performs a logical AND between this filter and a specified filter.
int	getVectorSize()
boolean	membershipTest(Key key) Determines whether a specified key belongs to this filter.
void	not() Performs a logical NOT on this filter.
void	or(org.apache.hudi.common.bloom.InternalFilter filter) Performs a logical OR between this filter and a specified filter.
void	readFields(DataInput in) Deserialize the fields of this object from in.
String	toString()
void	write(DataOutput out) Serialize the fields of this object to out.
void	xor(org.apache.hudi.common.bloom.InternalFilter filter) Performs a logical XOR between this filter and a specified filter.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

vectorSize

protected int vectorSize

The vector size of this filter.

hash

protected HashFunction hash

The hash function used to map a key to several positions in the vector.

nbHash

protected int nbHash

The number of hash function to consider.

hashType

protected int hashType

Type of hashing function to use.

Constructor Detail

InternalBloomFilter

public InternalBloomFilter()

Default constructor - use with readFields

InternalBloomFilter

public InternalBloomFilter(int vectorSize,
                           int nbHash,
                           int hashType)

Constructor

Parameters:

vectorSize - The vector size of this filter.

nbHash - The number of hash function to consider.

hashType - type of the hashing function (see Hash).

Method Detail

add

public void add(Key key)

Adds a key to this filter.

Parameters:

key - The key to add.

and

public void and(org.apache.hudi.common.bloom.InternalFilter filter)

Performs a logical AND between this filter and a specified filter.

Invariant: The result is assigned to this filter.

Parameters:

filter - The filter to AND with.

membershipTest

public boolean membershipTest(Key key)

Determines whether a specified key belongs to this filter.

Parameters:

key - The key to test.

Returns:

boolean True if the specified key belongs to this filter. False otherwise.

not

public void not()

Performs a logical NOT on this filter.

The result is assigned to this filter.

or

public void or(org.apache.hudi.common.bloom.InternalFilter filter)

Performs a logical OR between this filter and a specified filter.

Invariant: The result is assigned to this filter.

Parameters:

filter - The filter to OR with.

xor

public void xor(org.apache.hudi.common.bloom.InternalFilter filter)

Performs a logical XOR between this filter and a specified filter.

Invariant: The result is assigned to this filter.

Parameters:

filter - The filter to XOR with.

toString

public String toString()

Overrides:

toString in class Object

getVectorSize

public int getVectorSize()

Returns:

size of the bloomfilter

write

public void write(DataOutput out)
           throws IOException

Serialize the fields of this object to out.

Parameters:

out - DataOuput to serialize this object into.

Throws:

IOException

readFields

public void readFields(DataInput in)
                throws IOException

Deserialize the fields of this object from in.

For efficiency, implementations should attempt to re-use storage in the existing object where possible.

Parameters:

in - DataInput to deserialize this object from.

Throws:

IOException

add

public void add(List<Key> keys)

Adds a list of keys to this filter.

Parameters:

keys - The list of keys.

add

public void add(Collection<Key> keys)

Adds a collection of keys to this filter.

Parameters:

keys - The collection of keys.

add

public void add(Key[] keys)

Adds an array of keys to this filter.

Parameters:

keys - The array of keys.