com.bluejeans.common.bigqueue

Class BigArray

java.lang.Object

com.bluejeans.common.bigqueue.BigArray

All Implemented Interfaces:

Closeable, AutoCloseable

public class BigArray
extends Object
implements Closeable

A big array implementation supporting sequential append and random read. Main features: 1. FAST : close to the speed of direct memory access, extremely fast in append only and sequential read modes, sequential append and read are close to O(1) memory access, random read is close to O(1) memory access if data is in cache and is close to O(1) disk access if data is not in cache. 2. MEMORY-EFFICIENT : automatic paging and swapping algorithm, only most-recently accessed data is kept in memory. 3. THREAD-SAFE : multiple threads can concurrently read/append the array without data corruption. 4. PERSISTENT - all array data is persisted on disk, and is crash resistant. 5. BIG(HUGE) - the total size of the array data is only limited by the available disk space.

Author:

bulldog

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_DATA_PAGE_SIZE
static int	MINIMUM_DATA_PAGE_SIZE
static long	NOT_FOUND

Constructor Summary

Constructors

Constructor and Description
BigArray(String arrayDir, String arrayName) A big array implementation supporting sequential write and random read, use default back data file size per page, see DEFAULT_DATA_PAGE_SIZE.
BigArray(String arrayDir, String arrayName, int pageSize) A big array implementation supporting sequential write and random read.

Method Summary

Modifier and Type	Method and Description
long	append(byte[] data) Append the data into the head of the array
void	close()
long	findClosestIndex(long timestamp) Find an index closest to the specific timestamp when the corresponding item was appended
void	flush() Force to persist newly appended data, normally, you don't need to flush explicitly since: 1.) BigArray will automatically flush a cached page when it is replaced out, 2.) BigArray uses memory mapped file technology internally, and the OS will flush the changes even your process crashes, call this periodically only if you need transactional reliability and you are aware of the cost to performance.
byte[]	get(long index) Get the data at specific index
long	getBackFileSize() Get total size of back files(index and data files) of the big array
int	getDataPageSize() Get the back data file size per page.
long	getHeadIndex() The head of the array.
int	getItemLength(long index) Get the data item length at specific index
long	getTailIndex() The tail of the array.
long	getTimestamp(long index) Get the timestamp of data at specific index, this is the timestamp when the data was appended.
boolean	isEmpty() Check if the array is empty or not
boolean	isFull() Check if the ring space of java long type has all been used up.
void	limitBackFileSize(long sizeLimit) limit the back file size, truncate back file and advance array tail index accordingly, Note, this is a best effort call, exact size limit can't be guaranteed
void	removeAll()
void	removeBefore(long timestamp) Remove all data before specific timestamp, this will advance the array tail and delete back page files accordingly.
void	removeBeforeIndex(long index) Remove all data before specific index, this will advance the array tail to index and delete back page files before index.
long	size() The total number of items has been appended into the array

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NOT_FOUND

public static final long NOT_FOUND

See Also:

Constant Field Values

DEFAULT_DATA_PAGE_SIZE

public static final int DEFAULT_DATA_PAGE_SIZE

See Also:

Constant Field Values

MINIMUM_DATA_PAGE_SIZE

public static final int MINIMUM_DATA_PAGE_SIZE

See Also:

Constant Field Values

Constructor Detail

BigArray

public BigArray(String arrayDir,
                String arrayName)

A big array implementation supporting sequential write and random read, use default back data file size per page, see DEFAULT_DATA_PAGE_SIZE.

Parameters:

arrayDir - directory for array data store

arrayName - the name of the array, will be appended as last part of the array directory

BigArray

public BigArray(String arrayDir,
                String arrayName,
                int pageSize)

A big array implementation supporting sequential write and random read.

Parameters:

arrayDir - directory for array data store

arrayName - the name of the array, will be appended as last part of the array directory

pageSize - the back data file size per page in bytes, see minimum allowed MINIMUM_DATA_PAGE_SIZE.

Method Detail

removeAll

public void removeAll()

removeBeforeIndex

public void removeBeforeIndex(long index)

Remove all data before specific index, this will advance the array tail to index and delete back page files before index.

Parameters:

index - an index

removeBefore

public void removeBefore(long timestamp)

Remove all data before specific timestamp, this will advance the array tail and delete back page files accordingly.

Parameters:

timestamp - a timestamp

append

public long append(byte[] data)

Append the data into the head of the array

Parameters:

data - binary data to append

Returns:

appended index

flush

public void flush()

Force to persist newly appended data, normally, you don't need to flush explicitly since: 1.) BigArray will automatically flush a cached page when it is replaced out, 2.) BigArray uses memory mapped file technology internally, and the OS will flush the changes even your process crashes, call this periodically only if you need transactional reliability and you are aware of the cost to performance.

get

public byte[] get(long index)

Get the data at specific index

Parameters:

index - valid data index

Returns:

binary data if the index is valid

getTimestamp

public long getTimestamp(long index)

Get the timestamp of data at specific index, this is the timestamp when the data was appended.

Parameters:

index - valid data index

Returns:

timestamp when the data was appended

size

public long size()

The total number of items has been appended into the array

Returns:

total number

getHeadIndex

public long getHeadIndex()

The head of the array. This is the next to append index, the index of the last appended data is [headIndex - 1] if the array is not empty.

Returns:

an index

getTailIndex

public long getTailIndex()

The tail of the array. The is the index of the first appended data

Returns:

an index

isEmpty

public boolean isEmpty()

Check if the array is empty or not

Returns:

true if empty false otherwise

isFull

public boolean isFull()

Check if the ring space of java long type has all been used up. can always assume false, if true, the world is end:)

Returns:

array full or not

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

getDataPageSize

public int getDataPageSize()

Get the back data file size per page.

Returns:

size per page

findClosestIndex

public long findClosestIndex(long timestamp)

Find an index closest to the specific timestamp when the corresponding item was appended

Parameters:

timestamp - when the corresponding item was appended

Returns:

an index

getBackFileSize

public long getBackFileSize()

Get total size of back files(index and data files) of the big array

Returns:

total size of back files

limitBackFileSize

public void limitBackFileSize(long sizeLimit)

limit the back file size, truncate back file and advance array tail index accordingly, Note, this is a best effort call, exact size limit can't be guaranteed

Parameters:

sizeLimit - the size to limit

getItemLength

public int getItemLength(long index)

Get the data item length at specific index

Parameters:

index - valid data index

Returns:

the length of binary data if the index is valid