HoodieData (hudi-common 0.15.0 API)

Type Parameters:

T - type of object

All Superinterfaces:

Serializable

All Known Implementing Classes:

HoodieListData
```
public interface HoodieData<T>
extends Serializable
```
An interface abstracting a container holding a collection of objects of type T allowing to perform common transformation on it. This abstraction provides common API implemented by
1. In-memory implementation (HoodieListData, HoodieListPairData), where all objects are held in-memory by the executing process
2. RDD-based implementation (HoodieJavaRDD, etc)
All implementations provide for consistent semantic, where
- All non-terminal* operations are executed lazily (for ex, map, filter, etc)
- All terminal operations are executed eagerly, executing all previously accumulated transformations. Note that, collection could not be re-used after invoking terminal operation on it.

Nested Class Summary

Nested Classes
Modifier and Type Interface and Description

static class HoodieData.HoodieDataCacheKey
The key used in a caching map to identify a HoodieData.

Nested Classes
Modifier and Type	Interface and Description
`static class`	`HoodieData.HoodieDataCacheKey` The key used in a caching map to identify a `HoodieData`.

Method Summary

All Methods Instance Methods Abstract Methods Default Methods
Modifier and Type	Method and Description
`List<T>`	`collectAsList()` Collects results of the underlying collection into a `List` This is a terminal operation
`long`	`count()` Returns number of objects held in the collection
`HoodieData<T>`	`distinct()` Returns new `HoodieData` collection holding only distinct objects of the original one This is a stateful intermediate operation
`HoodieData<T>`	`distinct(int parallelism)` Returns new `HoodieData` collection holding only distinct objects of the original one This is a stateful intermediate operation
`default <O> HoodieData<T>`	`distinctWithKey(SerializableFunction<T,O> keyGetter, int parallelism)`
`HoodieData<T>`	`filter(SerializableFunction<T,Boolean> filterFunc)` Returns new instance of `HoodieData` collection only containing elements matching provided `filterFunc` (ie ones it returns true on)
`<O> HoodieData<O>`	`flatMap(SerializableFunction<T,Iterator<O>> func)` Maps every element in the collection into a collection of the new elements using provided mapping `func`, subsequently flattening the result (by concatenating) into a single collection This is an intermediate operation
`<K,V> HoodiePairData<K,V>`	`flatMapToPair(SerializableFunction<T,Iterator<? extends Pair<K,V>>> func)` Maps every element in the collection into a collection of the `Pair`s of new elements using provided mapping `func`, subsequently flattening the result (by concatenating) into a single collection NOTE: That this operation will convert container from `HoodieData` to `HoodiePairData` This is an intermediate operation
`int`	`getId()` Get the `HoodieData`'s unique non-negative identifier.
`int`	`getNumPartitions()`
`boolean`	`isEmpty()` Returns whether the collection is empty.
`<O> HoodieData<O>`	`map(SerializableFunction<T,O> func)` Maps every element in the collection using provided mapping `func`.
`<O> HoodieData<O>`	`mapPartitions(SerializableFunction<Iterator<T>,Iterator<O>> func, boolean preservesPartitioning)` Maps every element in the collection's partition (if applicable) by applying provided mapping `func` to every collection's partition This is an intermediate operation
`<K,V> HoodiePairData<K,V>`	`mapToPair(SerializablePairFunction<T,K,V> func)` Maps every element in the collection using provided mapping `func` into a `Pair` of elements `K` and `V`
`void`	`persist(String level)` Persists the data w/ provided `level` (if applicable).
`void`	`persist(String level, HoodieEngineContext engineContext, HoodieData.HoodieDataCacheKey cacheKey)` Persists the data w/ provided `level` (if applicable), and cache the data's ids within the `engineContext`.
`HoodieData<T>`	`repartition(int parallelism)` Re-partitions underlying collection (if applicable) making sure new `HoodieData` has exactly `parallelism` partitions
`HoodieData<T>`	`union(HoodieData<T> other)` Unions `HoodieData` with another instance of `HoodieData`.
`void`	`unpersist()` Un-persists the data (if previously persisted)

- Method Detail
  - getId
```
int getId()
```
    Get the HoodieData's unique non-negative identifier. -1 indicates invalid id.
  - persist
```
void persist(String level)
```
    Persists the data w/ provided level (if applicable). Use this method only when you call unpersist() at some later point for the same HoodieData. Otherwise, use persist(String, HoodieEngineContext, HoodieDataCacheKey) instead for auto-unpersist at the end of a client write operation.
  - persist
```
void persist(String level,
             HoodieEngineContext engineContext,
             HoodieData.HoodieDataCacheKey cacheKey)
```
    Persists the data w/ provided level (if applicable), and cache the data's ids within the engineContext.
  - unpersist
```
void unpersist()
```
    Un-persists the data (if previously persisted)
  - isEmpty
```
boolean isEmpty()
```
    Returns whether the collection is empty.
  - count
```
long count()
```
    Returns number of objects held in the collection
    NOTE: This is a terminal operation
  - getNumPartitions
```
int getNumPartitions()
```
    Returns:
    
    the number of data partitions in the engine-specific representation.
  - map
```
<O> HoodieData<O> map(SerializableFunction<T,O> func)
```
    Maps every element in the collection using provided mapping func.
    This is an intermediate operation
    
    Type Parameters:
    
    O - output object type
    
    Parameters:
    
    func - serializable map function
    
    Returns:
    
    HoodieData holding mapped elements
  - mapPartitions
```
<O> HoodieData<O> mapPartitions(SerializableFunction<Iterator<T>,Iterator<O>> func,
                                boolean preservesPartitioning)
```
    Maps every element in the collection's partition (if applicable) by applying provided mapping func to every collection's partition This is an intermediate operation
    
    Type Parameters:
    
    O - output object type
    
    Parameters:
    
    func - serializable map function accepting Iterator of a single partition's elements and returning a new Iterator mapping every element of the partition into a new one
    
    preservesPartitioning - whether to preserve partitioning in the resulting collection
    
    Returns:
    
    HoodieData holding mapped elements
  - flatMap
```
<O> HoodieData<O> flatMap(SerializableFunction<T,Iterator<O>> func)
```
    Maps every element in the collection into a collection of the new elements using provided mapping func, subsequently flattening the result (by concatenating) into a single collection This is an intermediate operation
    
    Type Parameters:
    
    O - output object type
    
    Parameters:
    
    func - serializable function mapping every element T into Iterator<O>
    
    Returns:
    
    HoodieData holding mapped elements
  - flatMapToPair
```
<K,V> HoodiePairData<K,V> flatMapToPair(SerializableFunction<T,Iterator<? extends Pair<K,V>>> func)
```
    Maps every element in the collection into a collection of the Pairs of new elements using provided mapping func, subsequently flattening the result (by concatenating) into a single collection NOTE: That this operation will convert container from HoodieData to HoodiePairData This is an intermediate operation
  - mapToPair
```
<K,V> HoodiePairData<K,V> mapToPair(SerializablePairFunction<T,K,V> func)
```
    Maps every element in the collection using provided mapping func into a Pair of elements K and V
    This is an intermediate operation
    
    Type Parameters:
    
    K - key type of the pair
    
    V - value type of the pair
    
    Parameters:
    
    func - serializable map function
    
    Returns:
    
    HoodiePairData holding mapped elements
  - distinct
```
HoodieData<T> distinct()
```
    Returns new HoodieData collection holding only distinct objects of the original one This is a stateful intermediate operation
  - distinct
```
HoodieData<T> distinct(int parallelism)
```
    Returns new HoodieData collection holding only distinct objects of the original one This is a stateful intermediate operation
  - filter
```
HoodieData<T> filter(SerializableFunction<T,Boolean> filterFunc)
```
    Returns new instance of HoodieData collection only containing elements matching provided filterFunc (ie ones it returns true on)
    
    Parameters:
    
    filterFunc - filtering func either accepting or rejecting the elements
    
    Returns:
    
    HoodieData holding filtered elements
  - union
```
HoodieData<T> union(HoodieData<T> other)
```
    Unions HoodieData with another instance of HoodieData. Note that, it's only able to union same underlying collection implementations. This is a stateful intermediate operation
    
    Parameters:
    
    other - HoodieData collection
    
    Returns:
    
    HoodieData holding superset of elements of this and other collections
  - collectAsList
```
List<T> collectAsList()
```
    Collects results of the underlying collection into a List This is a terminal operation
  - repartition
```
HoodieData<T> repartition(int parallelism)
```
    Re-partitions underlying collection (if applicable) making sure new HoodieData has exactly parallelism partitions
    
    Parameters:
    
    parallelism - target number of partitions in the underlying collection
    
    Returns:
    
    HoodieData holding re-partitioned collection
  - distinctWithKey
```
default <O> HoodieData<T> distinctWithKey(SerializableFunction<T,O> keyGetter,
                                          int parallelism)
```

Interface HoodieData<T>

Nested Class Summary

Method Summary

Method Detail

getId

persist

persist

unpersist

isEmpty

count

getNumPartitions

map

mapPartitions

flatMap

flatMapToPair

mapToPair

distinct

distinct

filter

union

collectAsList

repartition

distinctWithKey