org.apache.gobblin.util

Class AvroUtils

java.lang.Object

org.apache.gobblin.util.AvroUtils

public class AvroUtils
extends Object

A Utils class for dealing with Avro objects

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AvroUtils.AvroPathFilter
static class	AvroUtils.SchemaEntry

Field Summary

Fields

Modifier and Type	Field and Description
static String	FIELD_LOCATION_DELIMITER

Constructor Summary

Constructors

Constructor and Description
AvroUtils()

Method Summary

Modifier and Type	Method and Description
static org.apache.avro.Schema	addSchemaCreationTime(org.apache.avro.Schema inputSchema, org.apache.avro.Schema outputSchema)
static boolean	checkReaderWriterCompatibility(org.apache.avro.Schema readerSchema, org.apache.avro.Schema writerSchema, boolean ignoreNamespace) Validates that the provided reader schema can be used to decode avro data written with the provided writer schema.
static void	convertFieldToSchemaWithProps(Map<String,org.codehaus.jackson.JsonNode> fieldProps, org.apache.avro.Schema targetSchemaObj) Generate a Schema object from Schema.Field with Field's properties carried over to the new object.
static org.apache.avro.generic.GenericRecord	convertRecordSchema(org.apache.avro.generic.GenericRecord record, org.apache.avro.Schema newSchema) Change the schema of an Avro record.
static org.apache.avro.generic.GenericRecord	decorateRecord(org.apache.avro.generic.GenericRecord inputRecord, Map<String,Object> fieldMap, org.apache.avro.Schema outputSchema) Decorate a GenericRecord with additional fields and make it conform to an extended Schema It is the caller's responsibility to ensure that the outputSchema is the merge of the inputRecord's schema and the additional fields.
static org.apache.avro.Schema	decorateRecordSchema(org.apache.avro.Schema inputSchema, List<org.apache.avro.Schema.Field> fieldList) Decorate the Schema for a record with additional Schema.Fields.
static List<org.apache.avro.Schema.Field>	deepCopySchemaFields(org.apache.avro.Schema readerSchema)
static org.apache.commons.math3.util.Pair<org.apache.avro.Schema,List<AvroUtils.SchemaEntry>>	dropRecursiveFields(org.apache.avro.Schema schema) Drop recursive fields from a Schema.
static org.apache.avro.Schema	getDirectorySchema(org.apache.hadoop.fs.Path directory, org.apache.hadoop.conf.Configuration conf, boolean latest) Get the latest avro schema for a directory
static org.apache.avro.Schema	getDirectorySchema(org.apache.hadoop.fs.Path directory, org.apache.hadoop.fs.FileSystem fs, boolean latest) Get the latest avro schema for a directory
static com.google.common.base.Optional<org.apache.avro.Schema.Field>	getField(org.apache.avro.Schema schema, String fieldLocation) Given a GenericRecord, this method will return the field specified by the path parameter.
static com.google.common.base.Optional<org.apache.avro.Schema>	getFieldSchema(org.apache.avro.Schema schema, String fieldLocation) Given a GenericRecord, this method will return the schema of the field specified by the path parameter.
static com.google.common.base.Optional<Object>	getFieldValue(org.apache.avro.generic.GenericRecord record, String fieldLocation) Given a GenericRecord, this method will return the field specified by the path parameter.
static Map<String,Object>	getMultiFieldValue(org.apache.avro.generic.GenericRecord record, String fieldLocation)
static String	getSchemaCreationTime(org.apache.avro.Schema inputSchema)
static org.apache.avro.Schema	getSchemaFromDataFile(org.apache.hadoop.fs.Path dataFile, org.apache.hadoop.fs.FileSystem fs) Get Avro schema from an Avro data file.
static boolean	isSchemaRecursive(org.apache.avro.Schema schema, com.google.common.base.Optional<org.slf4j.Logger> logger) Check if a schema has recursive fields inside it
static org.apache.avro.Schema	nullifyFieldsForSchemaMerge(org.apache.avro.Schema oldSchema, org.apache.avro.Schema newSchema) Merge oldSchema and newSchame.
static org.apache.avro.generic.GenericRecord	overrideNameAndNamespace(org.apache.avro.generic.GenericRecord input, String nameOverride, com.google.common.base.Optional<Map<String,String>> namespaceOverride) Given a generic record, Override the name and namespace of the schema and return a new generic record
static org.apache.avro.Schema	overrideNameAndNamespace(org.apache.avro.Schema input, String nameOverride, com.google.common.base.Optional<Map<String,String>> namespaceOverride) Given a input schema, Override the name and namespace of the schema and return a new schema
static org.apache.avro.Schema	parseSchemaFromFile(org.apache.hadoop.fs.Path filePath, org.apache.hadoop.fs.FileSystem fs) Parse Avro schema from a schema file.
static byte[]	recordToByteArray(org.apache.avro.generic.GenericRecord record) Convert a GenericRecord to a byte array.
static com.google.common.base.Optional<org.apache.avro.Schema>	removeUncomparableFields(org.apache.avro.Schema schema) Remove map, array, enum fields, as well as union fields that contain map, array or enum, from an Avro schema.
static String	sanitizeSchemaString(String schemaString) Escaping "\", """, ";" and "'" character in the schema string when it is being used in DDL.
static org.apache.hadoop.fs.Path	serializeAsPath(org.apache.avro.generic.GenericRecord record, boolean includeFieldNames, boolean replacePathSeparators) Serialize a generic record as a relative Path.
static org.apache.avro.Schema	setSchemaCreationTime(org.apache.avro.Schema inputSchema, String creationTime)
static org.apache.avro.generic.GenericRecord	slowDeserializeGenericRecord(byte[] serializedRecord, org.apache.avro.Schema schema) Deserialize a GenericRecord from a byte array.
static org.apache.avro.Schema	switchName(org.apache.avro.Schema schema, String newName) Copies the input Schema but changes the schema name.
static org.apache.avro.Schema	switchNamespace(org.apache.avro.Schema schema, Map<String,String> namespaceOverride) Copies the input Schema but changes the schema namespace.
static Map<String,String>	toStringMap(Object map) Given a map: key -> value, return a map: key.toString() -> value.toString().
static void	writeSchemaToFile(org.apache.avro.Schema schema, org.apache.hadoop.fs.Path filePath, org.apache.hadoop.fs.FileSystem fs, boolean overwrite)
static void	writeSchemaToFile(org.apache.avro.Schema schema, org.apache.hadoop.fs.Path filePath, org.apache.hadoop.fs.FileSystem fs, boolean overwrite, org.apache.hadoop.fs.permission.FsPermission perm)
static void	writeSchemaToFile(org.apache.avro.Schema schema, org.apache.hadoop.fs.Path filePath, org.apache.hadoop.fs.Path tempFilePath, org.apache.hadoop.fs.FileSystem fs, boolean overwrite)
static void	writeSchemaToFile(org.apache.avro.Schema schema, org.apache.hadoop.fs.Path filePath, org.apache.hadoop.fs.Path tempFilePath, org.apache.hadoop.fs.FileSystem fs, boolean overwrite, org.apache.hadoop.fs.permission.FsPermission perm) Write a schema to a file

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

FIELD_LOCATION_DELIMITER

public static final String FIELD_LOCATION_DELIMITER

See Also:

Constant Field Values

Constructor Detail

AvroUtils

public AvroUtils()

Method Detail

checkReaderWriterCompatibility

public static boolean checkReaderWriterCompatibility(org.apache.avro.Schema readerSchema,
                                                     org.apache.avro.Schema writerSchema,
                                                     boolean ignoreNamespace)

Validates that the provided reader schema can be used to decode avro data written with the provided writer schema.

Parameters:

readerSchema - schema to check.

writerSchema - schema to check.

ignoreNamespace - whether name and namespace should be ignored in validation

Returns:

true if validation passes

addSchemaCreationTime

public static org.apache.avro.Schema addSchemaCreationTime(org.apache.avro.Schema inputSchema,
                                                           org.apache.avro.Schema outputSchema)

getSchemaCreationTime

public static String getSchemaCreationTime(org.apache.avro.Schema inputSchema)

setSchemaCreationTime

public static org.apache.avro.Schema setSchemaCreationTime(org.apache.avro.Schema inputSchema,
                                                           String creationTime)

deepCopySchemaFields

public static List<org.apache.avro.Schema.Field> deepCopySchemaFields(org.apache.avro.Schema readerSchema)

convertFieldToSchemaWithProps

public static void convertFieldToSchemaWithProps(Map<String,org.codehaus.jackson.JsonNode> fieldProps,
                                                 org.apache.avro.Schema targetSchemaObj)

Generate a Schema object from Schema.Field with Field's properties carried over to the new object. Common use cases for this method is in traversing Schema object into nested level and create Schema object for non-root level.

getFieldSchema

public static com.google.common.base.Optional<org.apache.avro.Schema> getFieldSchema(org.apache.avro.Schema schema,
                                                                                     String fieldLocation)

Given a GenericRecord, this method will return the schema of the field specified by the path parameter. The fieldLocation parameter is an ordered string specifying the location of the nested field to retrieve. For example, field1.nestedField1 takes the the schema of the field "field1", and retrieves the schema "nestedField1" from it.

Parameters:

schema - is the record to retrieve the schema from

fieldLocation - is the location of the field

Returns:

the schema of the field

getField

public static com.google.common.base.Optional<org.apache.avro.Schema.Field> getField(org.apache.avro.Schema schema,
                                                                                     String fieldLocation)

Given a GenericRecord, this method will return the field specified by the path parameter. The fieldLocation parameter is an ordered string specifying the location of the nested field to retrieve. For example, field1.nestedField1 takes field "field1", and retrieves "nestedField1" from it.

Parameters:

schema - is the record to retrieve the schema from

fieldLocation - is the location of the field

Returns:

the field

getFieldValue

public static com.google.common.base.Optional<Object> getFieldValue(org.apache.avro.generic.GenericRecord record,
                                                                    String fieldLocation)

Given a GenericRecord, this method will return the field specified by the path parameter. The fieldLocation parameter is an ordered string specifying the location of the nested field to retrieve. For example, field1.nestedField1 takes the the value of the field "field1", and retrieves the field "nestedField1" from it.

Parameters:

record - is the record to retrieve the field from

fieldLocation - is the location of the field

Returns:

the value of the field

getMultiFieldValue

public static Map<String,Object> getMultiFieldValue(org.apache.avro.generic.GenericRecord record,
                                                    String fieldLocation)

toStringMap

public static Map<String,String> toStringMap(Object map)

Given a map: key -> value, return a map: key.toString() -> value.toString(). Avro serializer wraps a String into Utf8. This method helps to restore the original string map object

Parameters:

map - a map object

Returns:

a map of strings

convertRecordSchema

public static org.apache.avro.generic.GenericRecord convertRecordSchema(org.apache.avro.generic.GenericRecord record,
                                                                        org.apache.avro.Schema newSchema)
                                                                 throws IOException

Change the schema of an Avro record.

Parameters:

record - The Avro record whose schema is to be changed.

newSchema - The target schema. It must be compatible as reader schema with record.getSchema() as writer schema.

Returns:

a new Avro record with the new schema.

Throws:

IOException - if conversion failed.

recordToByteArray

public static byte[] recordToByteArray(org.apache.avro.generic.GenericRecord record)
                                throws IOException

Convert a GenericRecord to a byte array.

Throws:

IOException

getSchemaFromDataFile

public static org.apache.avro.Schema getSchemaFromDataFile(org.apache.hadoop.fs.Path dataFile,
                                                           org.apache.hadoop.fs.FileSystem fs)
                                                    throws IOException

Get Avro schema from an Avro data file.

Throws:

IOException

parseSchemaFromFile

public static org.apache.avro.Schema parseSchemaFromFile(org.apache.hadoop.fs.Path filePath,
                                                         org.apache.hadoop.fs.FileSystem fs)
                                                  throws IOException

Parse Avro schema from a schema file.

Throws:

IOException

writeSchemaToFile

public static void writeSchemaToFile(org.apache.avro.Schema schema,
                                     org.apache.hadoop.fs.Path filePath,
                                     org.apache.hadoop.fs.FileSystem fs,
                                     boolean overwrite)
                              throws IOException

Throws:

IOException

writeSchemaToFile

public static void writeSchemaToFile(org.apache.avro.Schema schema,
                                     org.apache.hadoop.fs.Path filePath,
                                     org.apache.hadoop.fs.Path tempFilePath,
                                     org.apache.hadoop.fs.FileSystem fs,
                                     boolean overwrite)
                              throws IOException

Throws:

IOException

writeSchemaToFile

public static void writeSchemaToFile(org.apache.avro.Schema schema,
                                     org.apache.hadoop.fs.Path filePath,
                                     org.apache.hadoop.fs.FileSystem fs,
                                     boolean overwrite,
                                     org.apache.hadoop.fs.permission.FsPermission perm)
                              throws IOException

Throws:

IOException

writeSchemaToFile

public static void writeSchemaToFile(org.apache.avro.Schema schema,
                                     org.apache.hadoop.fs.Path filePath,
                                     org.apache.hadoop.fs.Path tempFilePath,
                                     org.apache.hadoop.fs.FileSystem fs,
                                     boolean overwrite,
                                     org.apache.hadoop.fs.permission.FsPermission perm)
                              throws IOException

Write a schema to a file

Parameters:

schema - the schema

filePath - the target file

tempFilePath - if not null then this path is used for a temporary file used to stage the write

fs - a FileSystem

overwrite - should any existing target file be overwritten?

perm - permissions

Throws:

IOException

getDirectorySchema

public static org.apache.avro.Schema getDirectorySchema(org.apache.hadoop.fs.Path directory,
                                                        org.apache.hadoop.fs.FileSystem fs,
                                                        boolean latest)
                                                 throws IOException

Get the latest avro schema for a directory

Parameters:

directory - the input dir that contains avro files

fs - the FileSystem for the given directory.

latest - true to return latest schema, false to return oldest schema

Returns:

the latest/oldest schema in the directory

Throws:

IOException

getDirectorySchema

public static org.apache.avro.Schema getDirectorySchema(org.apache.hadoop.fs.Path directory,
                                                        org.apache.hadoop.conf.Configuration conf,
                                                        boolean latest)
                                                 throws IOException

Get the latest avro schema for a directory

Parameters:

directory - the input dir that contains avro files

conf - configuration

latest - true to return latest schema, false to return oldest schema

Returns:

the latest/oldest schema in the directory

Throws:

IOException

nullifyFieldsForSchemaMerge

public static org.apache.avro.Schema nullifyFieldsForSchemaMerge(org.apache.avro.Schema oldSchema,
                                                                 org.apache.avro.Schema newSchema)

Merge oldSchema and newSchame. Set a field default value to null, if this field exists in the old schema but not in the new schema.

Parameters:

oldSchema -

newSchema -

Returns:

schema that contains all the fields in both old and new schema.

removeUncomparableFields

public static com.google.common.base.Optional<org.apache.avro.Schema> removeUncomparableFields(org.apache.avro.Schema schema)

Remove map, array, enum fields, as well as union fields that contain map, array or enum, from an Avro schema. A schema with these fields cannot be used as Mapper key in a MapReduce job.

switchName

public static org.apache.avro.Schema switchName(org.apache.avro.Schema schema,
                                                String newName)

Copies the input Schema but changes the schema name.

Parameters:

schema - Schema to copy.

newName - name for the copied Schema.

Returns:

A Schema that is a copy of schema, but has the name newName.

switchNamespace

public static org.apache.avro.Schema switchNamespace(org.apache.avro.Schema schema,
                                                     Map<String,String> namespaceOverride)

Copies the input Schema but changes the schema namespace.

Parameters:

schema - Schema to copy.

namespaceOverride - namespace for the copied Schema.

Returns:

A Schema that is a copy of schema, but has the new namespace.

serializeAsPath

public static org.apache.hadoop.fs.Path serializeAsPath(org.apache.avro.generic.GenericRecord record,
                                                        boolean includeFieldNames,
                                                        boolean replacePathSeparators)

Serialize a generic record as a relative Path. Useful for converting GenericRecord type keys into file system locations. For example {field1=v1, field2=v2} returns field1=v1/field2=v2 if includeFieldNames is true, or v1/v2 if it is false. Illegal HDFS tokens such as ':' and '\\' will be replaced with '_'. Additionally, parameter replacePathSeparators controls whether to replace path separators ('/') with '_'.

Parameters:

record - GenericRecord to serialize.

includeFieldNames - If true, each token in the path will be of the form key=value, otherwise, only the value will be included.

replacePathSeparators - If true, path separators ('/') in each token will be replaced with '_'.

Returns:

A relative path where each level is a field in the input record.

sanitizeSchemaString

public static String sanitizeSchemaString(String schemaString)

Escaping "\", """, ";" and "'" character in the schema string when it is being used in DDL. These characters are not allowed to show as part of column name but could possibly appear in documentation field. Therefore the escaping behavior won't cause correctness issues.

slowDeserializeGenericRecord

public static org.apache.avro.generic.GenericRecord slowDeserializeGenericRecord(byte[] serializedRecord,
                                                                                 org.apache.avro.Schema schema)
                                                                          throws IOException

Deserialize a GenericRecord from a byte array. This method is not intended for high performance.

Throws:

IOException

decorateRecordSchema

public static org.apache.avro.Schema decorateRecordSchema(org.apache.avro.Schema inputSchema,
                                                          @Nonnull
                                                          List<org.apache.avro.Schema.Field> fieldList)

Decorate the Schema for a record with additional Schema.Fields.

Parameters:

inputSchema: - must be a GenericData.Record schema.

Returns:

the decorated Schema. Fields are appended to the inputSchema.

decorateRecord

public static org.apache.avro.generic.GenericRecord decorateRecord(org.apache.avro.generic.GenericRecord inputRecord,
                                                                   @Nonnull
                                                                   Map<String,Object> fieldMap,
                                                                   org.apache.avro.Schema outputSchema)

Decorate a GenericRecord with additional fields and make it conform to an extended Schema It is the caller's responsibility to ensure that the outputSchema is the merge of the inputRecord's schema and the additional fields. The method does not check this for performance reasons, because it is expected to be called in the critical path of processing a record. Use decorateRecordSchema(Schema, List) to generate such a Schema before calling this method.

Parameters:

inputRecord: - record with data to be copied into the output record

fieldMap: - values can be primitive types or GenericRecords if nested

outputSchema: - the schema that the decoratedRecord will conform to

Returns:

an outputRecord that contains a union of the fields in the inputRecord and the field-values in the fieldMap

overrideNameAndNamespace

public static org.apache.avro.generic.GenericRecord overrideNameAndNamespace(org.apache.avro.generic.GenericRecord input,
                                                                             String nameOverride,
                                                                             com.google.common.base.Optional<Map<String,String>> namespaceOverride)

Given a generic record, Override the name and namespace of the schema and return a new generic record

Parameters:

input - input record who's name and namespace need to be overridden

nameOverride - new name for the record schema

namespaceOverride - Optional map containing namespace overrides

Returns:

an output record with overridden name and possibly namespace

overrideNameAndNamespace

public static org.apache.avro.Schema overrideNameAndNamespace(org.apache.avro.Schema input,
                                                              String nameOverride,
                                                              com.google.common.base.Optional<Map<String,String>> namespaceOverride)

Given a input schema, Override the name and namespace of the schema and return a new schema

Parameters:

input -

nameOverride -

namespaceOverride -

Returns:

a schema with overridden name and possibly namespace

isSchemaRecursive

public static boolean isSchemaRecursive(org.apache.avro.Schema schema,
                                        com.google.common.base.Optional<org.slf4j.Logger> logger)

Check if a schema has recursive fields inside it

Parameters:

schema -

logger - : Optional logger if you want the method to log why it thinks the schema was recursive

Returns:

true / false

dropRecursiveFields

public static org.apache.commons.math3.util.Pair<org.apache.avro.Schema,List<AvroUtils.SchemaEntry>> dropRecursiveFields(org.apache.avro.Schema schema)

Drop recursive fields from a Schema. Recursive fields are fields that refer to types that are part of the parent tree. e.g. consider this Schema for a User { "type": "record", "name": "User", "fields": [ {"name": "name", "type": "string", {"name": "friend", "type": "User"} ] } the friend field is a recursive field. After recursion has been eliminated we expect the output Schema to look like { "type": "record", "name": "User", "fields": [ {"name": "name", "type": "string"} ] }

Parameters:

schema -

Returns:

a Pair of (The transformed schema with recursion eliminated, A list of @link{SchemaEntry} objects which represent the fields that were removed from the original schema)