Vec (h2o-core 0.1.4 API)

java.lang.Object
- water.Iced
- - water.Keyed
  - - water.fvec.Vec

All Implemented Interfaces:: java.lang.Cloneable, Freezable

Direct Known Subclasses:: AppendableVec, ByteVec, WrappedVec

public class Vec
extends Keyed

A single distributed vector column.

A distributed vector has a count of elements, an element-to-chunk mapping, a Java type (mostly determines rounding on store and display), and functions to directly load elements without further indirections. The data is compressed, or backed by disk or both. *Writing* to elements may throw if the backing data is read-only (file backed).

  Vec Key format is: Key. VEC - byte, 0 - byte,   0    - int, normal Key bytes.
 DVec Key format is: Key.DVEC - byte, 0 - byte, chunk# - int, normal Key bytes.

The main API is at, set, and isNA:

   double  at  ( long row );  // Returns the value expressed as a double.  NaN if missing.
   long    at8 ( long row );  // Returns the value expressed as a long.  Throws if missing.
   boolean isNA( long row );  // True if the value is missing.
   set( long row, double d ); // Stores a double; NaN will be treated as missing.
   set( long row, long l );   // Stores a long; throws if l exceeds what fits in a double & any floats are ever set.
   setNA( long row );         // Sets the value as missing.

Note this dangerous scenario: loading a missing value as a double, and setting it as a long:

   set(row,(long)at(row)); // Danger!

The cast from a Double.NaN to a long produces a zero! This code will replace a missing value with a zero.

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static class`	`Vec.VectorGroup` Class representing the group of vectors.
`static class`	`Vec.Writer` More efficient way to write randomly to a Vec - still slow, but much faster than Vec.set().

Field Summary

Fields
Modifier and Type	Field and Description
`protected java.lang.String[]`	`_domain` Enum/factor/categorical names.
`protected boolean`	`_isString` String flag
`protected boolean`	`_isUUID` UUID flag
`protected byte`	`_time` Time parse, index into Utils.TIME_PARSE, or -1 for not-a-time
`static int`	`CHUNK_SZ` Chunk size.
`static int`	`KEY_PREFIX_LEN`

Fields inherited from class water.Keyed
_key

Constructor Summary

Constructors
Modifier	Constructor and Description
	`Vec(Key key, long[] espc)` Main default constructor; requires the caller understand Chunk layout already, along with count of missing elements.
	`Vec(Key key, long[] espc, java.lang.String[] domain)`
	`Vec(Key key, long[] espc, java.lang.String[] domain, boolean hasUUID, boolean hasString, byte time)`
`protected`	`Vec(Key key, Vec v)`

Method Summary

Methods
Modifier and Type	Method and Description
`Vec`	`align(Vec vec)` Always makes a copy of the given vector which shares the same group.
`double`	`at(long i)` Fetch element the slow way, as a double.
`long`	`at16h(long i)`
`long`	`at16l(long i)` Fetch element the slow way, as a long.
`long`	`at8(long i)` Fetch element the slow way, as a long.
`ValueString`	`atStr(ValueString vstr, long i)`
`double`	`base()`
`long[]`	`bins()` Histogram bins.
`int`	`cardinality()` Returns cardinality for enum domain or -1 for other types.
`protected boolean`	`checkMissing(int cidx, Value val)`
`Chunk`	`chunkForChunkIdx(int cidx)` The Chunk for a chunk#.
`Chunk`	`chunkForRow(long i)` The Chunk for a row#.
`protected Value`	`chunkIdx(int cidx)` Get a Chunk's Value by index.
`Key`	`chunkKey(int cidx)` Get a Chunk Key from a chunk-index.
`static Key`	`chunkKey(Key veckey, int cidx)`
`int`	`chunkLen(int cidx)` Number of rows in chunk.
`java.lang.String[]`	`domain()` Return an array of domains.
`boolean`	`equals(java.lang.Object o)`
`java.lang.String[]`	`factors()`
`static Key`	`getVecKey(Key key)` Get a Vec Key from Chunk Key, without loading the Chunk
`Vec.VectorGroup`	`group()` Get the group this vector belongs to.
`int`	`hashCode()`
`boolean`	`isConst()` Is the column constant.
`boolean`	`isEnum()` Is the column a factor/categorical/enum? Note: all "isEnum()" columns are are also "isInt()" but not vice-versa.
`boolean`	`isInt()` Is all integers?
`boolean`	`isNA(long row)` Fetch the missing-status the slow way.
`boolean`	`isString()`
`boolean`	`isTime()` Whether or not this column parsed as a time, and if so what pattern was used.
`boolean`	`isUUID()`
`long`	`length()` Number of elements in the vector.
`protected Vec`	`makeCon(double d)`
`static Vec`	`makeCon(long l, java.lang.String[] domain, long len)`
`static Vec`	`makeCon(long l, java.lang.String[] domain, Vec.VectorGroup group, long[] espc)`
`static Vec`	`makeConSeq(double x, long len)`
`static Vec`	`makeSeq(long len)`
`Vec`	`makeTransf(int[][] map, java.lang.String[] finalDomain)` Create a vector transforming values according given domain map.
`Vec`	`makeTransf(int[] values, int[] indexes, java.lang.String[] domain)` Creates a new transformation from given values to given indexes of given domain.
`Vec`	`makeZero()` Make a new vector with the same size and data layout as the old one, and initialized to zero.
`Vec`	`makeZero(java.lang.String[] domain)`
`protected Vec`	`masterVec()` This Vec does not have dependent hidden Vec it uses.
`double`	`max()` Return column max - lazily computed as needed.
`double[]`	`maxs()`
`double`	`mean()` Return column mean - lazily computed as needed.
`double`	`min()` Return column min - lazily computed as needed.
`double[]`	`mins()`
`long`	`naCnt()` Return column missing-element-count - lazily computed as needed.
`int`	`nChunks()` Number of chunks.
`static Key`	`newKey()` Make a new random Key that fits the requirements for a Vec key.
`long`	`ninfs()`
`long`	`nzCnt()` Return column zero-element-count - lazily computed as needed.
`Vec.Writer`	`open()` Create a writer for bulk serial writes into this Vec
`long`	`pinfs()` Positive and negative infinity counts
`Futures`	`postWrite(Futures fs)` Stop writing into this Vec.
`void`	`preWriting()` Writing into this Vector from some chunk.
`protected boolean`	`readable()` Default read/write behavior for Vecs.
`Futures`	`remove_impl(Futures fs)`
`RollupStats`	`rollupStats()` Compute the roll-up stats as-needed
`Key`	`rollupStatsKey()`
`void`	`set_factors(java.lang.String[] factors)`
`void`	`set(long i, double d)` Write element the slow way, as a double.
`void`	`set(long i, float f)` Write element the slow way, as a float.
`void`	`set(long i, long l)` Write element the slow way, as a long.
`void`	`set(long i, java.lang.String str)`
`double`	`sigma()` Return column standard deviation - lazily computed as needed.
`double`	`stride()`
`java.lang.String`	`timeParse()`
`Vec`	`toEnum()` Transform this vector to enum.
`java.lang.String`	`toString()` Pretty print the Vec: [#elems, min/mean/max]{chunks,...}
`protected boolean`	`writable()` Default read/write behavior for Vecs.

Methods inherited from class water.Keyed
remove, remove, remove, remove

Methods inherited from class water.Iced
clone, frozenType, read_impl, read, readJSON_impl, readJSON, write_impl, write, writeHTML_impl, writeHTML, writeJSON_impl, writeJSON

Methods inherited from class java.lang.Object
finalize, getClass, notify, notifyAll, wait, wait, wait

- Field Detail
  - CHUNK_SZ
```
public static final int CHUNK_SZ
```
    Chunk size. Bigger increases batch sizes, lowers overhead costs, lower increases fine-grained parallelism.
    
    See Also:
    Constant Field Values
  - _domain
```
protected java.lang.String[] _domain
```
    Enum/factor/categorical names.
  - _time
```
protected byte _time
```
    Time parse, index into Utils.TIME_PARSE, or -1 for not-a-time
  - _isUUID
```
protected boolean _isUUID
```
    UUID flag
  - _isString
```
protected boolean _isString
```
    String flag
  - KEY_PREFIX_LEN
```
public static final int KEY_PREFIX_LEN
```
    See Also:
    Constant Field Values
- Constructor Detail
  - Vec
```
public Vec(Key key,
   long[] espc)
```
    Main default constructor; requires the caller understand Chunk layout already, along with count of missing elements.
  - Vec
```
public Vec(Key key,
   long[] espc,
   java.lang.String[] domain)
```
  - Vec
```
public Vec(Key key,
   long[] espc,
   java.lang.String[] domain,
   boolean hasUUID,
   boolean hasString,
   byte time)
```
  - Vec
```
protected Vec(Key key,
   Vec v)
```
- Method Detail
  - factors
```
public final java.lang.String[] factors()
```
  - set_factors
```
public final void set_factors(java.lang.String[] factors)
```
  - makeZero
```
public Vec makeZero()
```
    Make a new vector with the same size and data layout as the old one, and initialized to zero.
  - makeZero
```
public Vec makeZero(java.lang.String[] domain)
```
  - makeCon
```
public static Vec makeCon(long l,
          java.lang.String[] domain,
          long len)
```
  - makeCon
```
public static Vec makeCon(long l,
          java.lang.String[] domain,
          Vec.VectorGroup group,
          long[] espc)
```
  - makeCon
```
protected Vec makeCon(double d)
```
  - makeSeq
```
public static Vec makeSeq(long len)
```
  - makeConSeq
```
public static Vec makeConSeq(double x,
             long len)
```
  - length
```
public long length()
```
    Number of elements in the vector. Overridden by subclasses that compute length in an alternative way, such as file-backed Vecs.
  - nChunks
```
public int nChunks()
```
    Number of chunks. Overridden by subclasses that compute chunks in an alternative way, such as file-backed Vecs.
  - chunkLen
```
public int chunkLen(int cidx)
```
    Number of rows in chunk. Does not fetch chunk content.
  - isEnum
```
public final boolean isEnum()
```
    Is the column a factor/categorical/enum? Note: all "isEnum()" columns are are also "isInt()" but not vice-versa.
  - isUUID
```
public final boolean isUUID()
```
  - isString
```
public final boolean isString()
```
  - isTime
```
public final boolean isTime()
```
    Whether or not this column parsed as a time, and if so what pattern was used.
  - timeParse
```
public final java.lang.String timeParse()
```
  - isConst
```
public final boolean isConst()
```
    Is the column constant.
    Returns true if the column contains only constant values and it is not full of NAs.
  - domain
```
public java.lang.String[] domain()
```
    Return an array of domains. This is eagerly manifested for enum or categorical columns. Returns null for non-Enum/factor columns.
  - cardinality
```
public int cardinality()
```
    Returns cardinality for enum domain or -1 for other types.
  - readable
```
protected boolean readable()
```
    Default read/write behavior for Vecs. File-backed Vecs are read-only.
  - writable
```
protected boolean writable()
```
    Default read/write behavior for Vecs. AppendableVecs are write-only.
  - min
```
public double min()
```
    Return column min - lazily computed as needed.
  - mins
```
public double[] mins()
```
  - max
```
public double max()
```
    Return column max - lazily computed as needed.
  - maxs
```
public double[] maxs()
```
  - mean
```
public double mean()
```
    Return column mean - lazily computed as needed.
  - sigma
```
public double sigma()
```
    Return column standard deviation - lazily computed as needed.
  - naCnt
```
public long naCnt()
```
    Return column missing-element-count - lazily computed as needed.
  - nzCnt
```
public long nzCnt()
```
    Return column zero-element-count - lazily computed as needed.
  - pinfs
```
public long pinfs()
```
    Positive and negative infinity counts
  - ninfs
```
public long ninfs()
```
  - isInt
```
public boolean isInt()
```
    Is all integers?
  - bins
```
public long[] bins()
```
    Histogram bins. Computed on-demand to 1st call to these methods. bins[] are row-counts in each bin base - start of bin 0 stride - relative start of next bin
  - base
```
public double base()
```
  - stride
```
public double stride()
```
  - rollupStats
```
public RollupStats rollupStats()
```
    Compute the roll-up stats as-needed
  - preWriting
```
public void preWriting()
```
    Writing into this Vector from *some* chunk. Immediately clear all caches (_min, _max, _mean, etc). Can be called repeatedly from one or all chunks. Per-chunk row-counts will not be changing, just row contents and caches of row contents.
  - postWrite
```
public Futures postWrite(Futures fs)
```
    Stop writing into this Vec. Rollup stats will again (lazily) be computed.
  - getVecKey
```
public static Key getVecKey(Key key)
```
    Get a Vec Key from Chunk Key, without loading the Chunk
  - chunkKey
```
public Key chunkKey(int cidx)
```
    Get a Chunk Key from a chunk-index. Basically the index-to-key map.
  - chunkKey
```
public static Key chunkKey(Key veckey,
           int cidx)
```
  - rollupStatsKey
```
public Key rollupStatsKey()
```
  - chunkIdx
```
protected Value chunkIdx(int cidx)
```
    Get a Chunk's Value by index. Basically the index-to-key map, plus the DKV.get(). Warning: this pulls the data locally; using this call on every Chunk index on the same node will probably trigger an OOM!
  - checkMissing
```
protected boolean checkMissing(int cidx,
                   Value val)
```
  - newKey
```
public static Key newKey()
```
    Make a new random Key that fits the requirements for a Vec key.
  - group
```
public final Vec.VectorGroup group()
```
    Get the group this vector belongs to. In case of a group with only one vector, the object actually does not exist in KV store.
    
    Returns:
    VectorGroup this vector belongs to.
  - chunkForChunkIdx
```
public Chunk chunkForChunkIdx(int cidx)
```
    The Chunk for a chunk#. Warning: this loads the data locally!
  - chunkForRow
```
public final Chunk chunkForRow(long i)
```
    The Chunk for a row#. Warning: this loads the data locally!
  - at8
```
public final long at8(long i)
```
    Fetch element the slow way, as a long. Floating point values are silently rounded to an integer. Throws if the value is missing.
  - at
```
public final double at(long i)
```
    Fetch element the slow way, as a double. Missing values are returned as Double.NaN instead of throwing.
  - isNA
```
public final boolean isNA(long row)
```
    Fetch the missing-status the slow way.
  - at16l
```
public final long at16l(long i)
```
    Fetch element the slow way, as a long. Throws if the value is missing or not a UUID.
  - at16h
```
public final long at16h(long i)
```
  - atStr
```
public final ValueString atStr(ValueString vstr,
                long i)
```
  - set
```
public final void set(long i,
       long l)
```
    Write element the slow way, as a long. There is no way to write a missing value with this call. Under rare circumstances this can throw: if the long does not fit in a double (value is larger magnitude than 2^52), AND float values are stored in Vector. In this case, there is no common compatible data representation.
  - set
```
public final void set(long i,
       double d)
```
    Write element the slow way, as a double. Double.NaN will be treated as a set of a missing element. Slow to do this for every set -> use Writer if writing many values
  - set
```
public final void set(long i,
       float f)
```
    Write element the slow way, as a float. Float.NaN will be treated as a set of a missing element.
  - set
```
public final void set(long i,
       java.lang.String str)
```
  - open
```
public final Vec.Writer open()
```
    Create a writer for bulk serial writes into this Vec
  - toString
```
public java.lang.String toString()
```
    Pretty print the Vec: [#elems, min/mean/max]{chunks,...}
    
    Overrides:
    
    toString in class java.lang.Object
  - remove_impl
```
public Futures remove_impl(Futures fs)
```
    Overrides:
    
    remove_impl in class Keyed
  - equals
```
public boolean equals(java.lang.Object o)
```
    Overrides:
    
    equals in class java.lang.Object
  - hashCode
```
public int hashCode()
```
    Overrides:
    
    hashCode in class java.lang.Object
  - align
```
public Vec align(Vec vec)
```
    Always makes a copy of the given vector which shares the same group. The user is responsible for deleting the returned vector. This can be expensive operation since it can force copy of data among nodes.
    
    Parameters:
    vec - vector which is intended to be copied
    
    Returns:
    a copy of vec which shared the same Vec.VectorGroup with this vector
  - toEnum
```
public Vec toEnum()
```
    Transform this vector to enum. If the vector is integer vector then its domain is collected and transformed to corresponding strings. If the vector is enum an identity transformation vector is returned. Transformation is done by a TransfVec which provides a mapping between values.
    
    Returns:
    always returns a new vector and the caller is responsible for vector deletion!
  - makeTransf
```
public Vec makeTransf(int[][] map,
             java.lang.String[] finalDomain)
```
    Create a vector transforming values according given domain map.
    
    See Also:
    makeTransf(int[], int[], String[])
  - makeTransf
```
public Vec makeTransf(int[] values,
             int[] indexes,
             java.lang.String[] domain)
```
    Creates a new transformation from given values to given indexes of given domain.
    
    Parameters:
    values - values being mapped from
    indexes - values being mapped to
    domain - domain of new vector
    
    Returns:
    always return a new vector which maps given values into a new domain
  - masterVec
```
protected Vec masterVec()
```
    This Vec does not have dependent hidden Vec it uses.
    
    Returns:
    dependent hidden vector or null

Class Vec

Nested Class Summary

Field Summary

Fields inherited from class water.Keyed

Constructor Summary

Method Summary

Methods inherited from class water.Keyed

Methods inherited from class water.Iced

Methods inherited from class java.lang.Object

Field Detail

CHUNK_SZ

_domain

_time

_isUUID

_isString

KEY_PREFIX_LEN

Constructor Detail

Vec

Vec

Vec

Vec

Method Detail

factors

set_factors

makeZero

makeZero

makeCon

makeCon

makeCon

makeSeq

makeConSeq

length

nChunks

chunkLen

isEnum

isUUID

isString

isTime

timeParse

isConst

domain

cardinality

readable

writable

min

mins

max

maxs

mean

sigma

naCnt

nzCnt

pinfs

ninfs

isInt

bins

base

stride

rollupStats

preWriting

postWrite

getVecKey

chunkKey

chunkKey

rollupStatsKey

chunkIdx

checkMissing

newKey

group

chunkForChunkIdx

chunkForRow

at8

at

isNA

at16l

at16h

atStr

set

set

set