org.apache.hudi.common.util

Class InternalSchemaCache

java.lang.Object

org.apache.hudi.common.util.InternalSchemaCache

public class InternalSchemaCache
extends Object

An internal cache implementation for managing different version of schemas. This is a Global cache; all threads in one container/executor share the same cache. A map of (tablePath, HistorySchemas) is maintained.

Constructor Summary

Constructors

Constructor and Description
InternalSchemaCache()

Method Summary

Modifier and Type	Method and Description
static Pair<Option<String>,Option<String>>	getInternalSchemaAndAvroSchemaForClusteringAndCompaction(HoodieTableMetaClient metaClient, String compactionAndClusteringInstant) Get internalSchema and avroSchema for compaction/cluster operation.
static InternalSchema	getInternalSchemaByVersionId(long versionId, HoodieTableMetaClient metaClient)
static InternalSchema	getInternalSchemaByVersionId(long versionId, String tablePath, HoodieStorage storage, String validCommits) Give a schema versionId return its internalSchema.
static InternalSchema	searchSchemaAndCache(long versionID, HoodieTableMetaClient metaClient, boolean cacheEnable) Search internalSchema based on versionID.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

InternalSchemaCache

public InternalSchemaCache()

Method Detail

searchSchemaAndCache

public static InternalSchema searchSchemaAndCache(long versionID,
                                                  HoodieTableMetaClient metaClient,
                                                  boolean cacheEnable)

Search internalSchema based on versionID. first step: try to get internalSchema from hoodie commit files, we no need to add lock. if we cannot get internalSchema by first step, then we try to get internalSchema from cache.

Parameters:

versionID - schema version_id need to search

metaClient - current hoodie metaClient

Returns:

internalSchema

getInternalSchemaAndAvroSchemaForClusteringAndCompaction

public static Pair<Option<String>,Option<String>> getInternalSchemaAndAvroSchemaForClusteringAndCompaction(HoodieTableMetaClient metaClient,
                                                                                                           String compactionAndClusteringInstant)

Get internalSchema and avroSchema for compaction/cluster operation.

Parameters:

metaClient - current hoodie metaClient

compactionAndClusteringInstant - first instant before current compaction/cluster instant

Returns:

(internalSchemaStrOpt, avroSchemaStrOpt) a pair of InternalSchema/avroSchema

getInternalSchemaByVersionId

public static InternalSchema getInternalSchemaByVersionId(long versionId,
                                                          String tablePath,
                                                          HoodieStorage storage,
                                                          String validCommits)

Give a schema versionId return its internalSchema. This method will be called by spark tasks, we should minimize time cost. We try our best to not use metaClient， since the initialization of metaClient is time cost step1： try to parser internalSchema from HoodieInstant directly step2： if we cannot parser internalSchema in step1， (eg: current versionId HoodieInstant has been archived) try to find internalSchema in historySchema. step3: if we cannot parser internalSchema in step2 (eg: schema evolution is not enabled when we create hoodie table, however after some inserts we enable schema evolution) try to convert table schema to internalSchema.

Parameters:

versionId - the internalSchema version to be search.

tablePath - table path

storage - HoodieStorage instance.

validCommits - current validate commits, use to make up the commit file path/verify the validity of the history schema files

Returns:

a internalSchema.

getInternalSchemaByVersionId

public static InternalSchema getInternalSchemaByVersionId(long versionId,
                                                          HoodieTableMetaClient metaClient)