org.apache.gobblin.util

Class WriterUtils

java.lang.Object

org.apache.gobblin.util.WriterUtils

public class WriterUtils
extends Object

Utility class for use with the DataWriter class.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	WriterUtils.WriterFilePathType

Field Summary

Fields

Modifier and Type	Field and Description
static com.typesafe.config.Config	NO_RETRY_CONFIG
static String	WRITER_ENCRYPTED_CONFIG_PATH

Constructor Summary

Constructors

Constructor and Description
WriterUtils()

Method Summary

Modifier and Type	Method and Description
static org.apache.avro.file.CodecFactory	getCodecFactory(com.google.common.base.Optional<String> codecName, com.google.common.base.Optional<String> deflateLevel) Creates a CodecFactory based on the specified codec name and deflate level.
static org.apache.hadoop.fs.Path	getDataPublisherFinalDir(State state, int numBranches, int branchId) Get the Path corresponding the to the directory a given BaseDataPublisher should commits its output data.
static org.apache.hadoop.fs.Path	getDefaultWriterFilePath(State state, int numBranches, int branchId) Creates the default Path for the ConfigurationKeys.WRITER_FILE_PATH key.
static org.apache.hadoop.conf.Configuration	getFsConfiguration(State state)
static org.apache.hadoop.fs.Path	getNamespaceTableWriterFilePath(State state) Creates Path for case WriterUtils.WriterFilePathType.NAMESPACE_TABLE with configurations ConfigurationKeys.EXTRACT_NAMESPACE_NAME_KEY and ConfigurationKeys.EXTRACT_TABLE_NAME_KEY
static org.apache.hadoop.fs.Path	getTableNameWriterFilePath(State state) Creates Path for the ConfigurationKeys.WRITER_FILE_PATH key according to ConfigurationKeys.EXTRACT_TABLE_NAME_KEY.
static String	getWriterFileName(State state, int numBranches, int branchId, String writerId, String formatExtension) Get the value of ConfigurationKeys.WRITER_FILE_NAME for the a given DataWriter.
static org.apache.hadoop.fs.Path	getWriterFilePath(State state, int numBranches, int branchId) Get the Path corresponding the the relative file path for a given DataWriter.
static org.apache.hadoop.fs.FileSystem	getWriterFs(State state)
static org.apache.hadoop.fs.FileSystem	getWriterFS(State state, int numBranches, int branchId)
static URI	getWriterFsUri(State state, int numBranches, int branchId)
static org.apache.hadoop.fs.Path	getWriterOutputDir(State state, int numBranches, int branchId) Get the Path corresponding the to the directory a given DataWriter should be writing its output data.
static org.apache.hadoop.fs.Path	getWriterStagingDir(State state, int numBranches, int branchId) Get the Path corresponding the to the directory a given DataWriter should be writing its staging data.
static org.apache.hadoop.fs.Path	getWriterStagingDir(State state, int numBranches, int branchId, String attemptId) Get the staging Path for DataWriter that has attemptId in the path.
static void	mkdirsWithRecursivePermission(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission perm) Create the given dir as well as all missing ancestor dirs.
static void	mkdirsWithRecursivePermissionWithRetry(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission perm, com.typesafe.config.Config retrierConfig)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

WRITER_ENCRYPTED_CONFIG_PATH

public static final String WRITER_ENCRYPTED_CONFIG_PATH

See Also:

Constant Field Values

NO_RETRY_CONFIG

public static final com.typesafe.config.Config NO_RETRY_CONFIG

Constructor Detail

WriterUtils

public WriterUtils()

Method Detail

getWriterStagingDir

public static org.apache.hadoop.fs.Path getWriterStagingDir(State state,
                                                            int numBranches,
                                                            int branchId)

Get the Path corresponding the to the directory a given DataWriter should be writing its staging data. The staging data directory is determined by combining the ConfigurationKeys.WRITER_STAGING_DIR and the ConfigurationKeys.WRITER_FILE_PATH.

Parameters:

state - is the State corresponding to a specific DataWriter.

numBranches - is the total number of branches for the given State.

branchId - is the id for the specific branch that the DataWriter will write to.

Returns:

a Path specifying the directory where the DataWriter will write to.

getWriterStagingDir

public static org.apache.hadoop.fs.Path getWriterStagingDir(State state,
                                                            int numBranches,
                                                            int branchId,
                                                            String attemptId)

Get the staging Path for DataWriter that has attemptId in the path.

getWriterOutputDir

public static org.apache.hadoop.fs.Path getWriterOutputDir(State state,
                                                           int numBranches,
                                                           int branchId)

Get the Path corresponding the to the directory a given DataWriter should be writing its output data. The output data directory is determined by combining the ConfigurationKeys.WRITER_OUTPUT_DIR and the ConfigurationKeys.WRITER_FILE_PATH.

Parameters:

state - is the State corresponding to a specific DataWriter.

numBranches - is the total number of branches for the given State.

branchId - is the id for the specific branch that the DataWriter will write to.

Returns:

a Path specifying the directory where the DataWriter will write to.

getDataPublisherFinalDir

public static org.apache.hadoop.fs.Path getDataPublisherFinalDir(State state,
                                                                 int numBranches,
                                                                 int branchId)

Get the Path corresponding the to the directory a given BaseDataPublisher should commits its output data. The final output data directory is determined by combining the ConfigurationKeys.DATA_PUBLISHER_FINAL_DIR and the ConfigurationKeys.WRITER_FILE_PATH.

Parameters:

state - is the State corresponding to a specific DataWriter.

numBranches - is the total number of branches for the given State.

branchId - is the id for the specific branch that the BaseDataPublisher will publish.

Returns:

a Path specifying the directory where the BaseDataPublisher will publish.

getWriterFilePath

public static org.apache.hadoop.fs.Path getWriterFilePath(State state,
                                                          int numBranches,
                                                          int branchId)

Get the Path corresponding the the relative file path for a given DataWriter. This method retrieves the value of ConfigurationKeys.WRITER_FILE_PATH from the given State. It also constructs the default value of the ConfigurationKeys.WRITER_FILE_PATH if not is not specified in the given State.

Parameters:

state - is the State corresponding to a specific DataWriter.

numBranches - is the total number of branches for the given State.

branchId - is the id for the specific branch that the {DataWriter will write to.

Returns:

a Path specifying the relative directory where the DataWriter will write to.

getNamespaceTableWriterFilePath

public static org.apache.hadoop.fs.Path getNamespaceTableWriterFilePath(State state)

Creates Path for case WriterUtils.WriterFilePathType.NAMESPACE_TABLE with configurations ConfigurationKeys.EXTRACT_NAMESPACE_NAME_KEY and ConfigurationKeys.EXTRACT_TABLE_NAME_KEY

Parameters:

state -

Returns:

a path

getTableNameWriterFilePath

public static org.apache.hadoop.fs.Path getTableNameWriterFilePath(State state)

Creates Path for the ConfigurationKeys.WRITER_FILE_PATH key according to ConfigurationKeys.EXTRACT_TABLE_NAME_KEY.

Parameters:

state -

Returns:

getDefaultWriterFilePath

public static org.apache.hadoop.fs.Path getDefaultWriterFilePath(State state,
                                                                 int numBranches,
                                                                 int branchId)

Creates the default Path for the ConfigurationKeys.WRITER_FILE_PATH key.

Parameters:

numBranches - is the total number of branches for the given State.

branchId - is the id for the specific branch that the DataWriter will write to.

Returns:

a Path specifying the directory where the DataWriter will write to.

getWriterFileName

public static String getWriterFileName(State state,
                                       int numBranches,
                                       int branchId,
                                       String writerId,
                                       String formatExtension)

Get the value of ConfigurationKeys.WRITER_FILE_NAME for the a given DataWriter. The method also constructs the default value of the ConfigurationKeys.WRITER_FILE_NAME if it is not set in the State

Parameters:

state - is the State corresponding to a specific DataWriter.

numBranches - is the total number of branches for the given State.

branchId - is the id for the specific branch that the {DataWriter will write to.

writerId - is the id for a specific DataWriter.

formatExtension - is the format extension for the file (e.g. ".avro").

Returns:

a String representation of the file name.

getCodecFactory

public static org.apache.avro.file.CodecFactory getCodecFactory(com.google.common.base.Optional<String> codecName,
                                                                com.google.common.base.Optional<String> deflateLevel)

Creates a CodecFactory based on the specified codec name and deflate level. If codecName is absent, then a CodecFactory.deflateCodec(int) is returned. Otherwise the codecName is converted into a CodecFactory via the CodecFactory.fromString(String) method.

Parameters:

codecName - the name of the codec to use (e.g. deflate, snappy, xz, etc.).

deflateLevel - must be an integer from [0-9], and is only applicable if the codecName is "deflate".

Returns:

a CodecFactory.

mkdirsWithRecursivePermission

public static void mkdirsWithRecursivePermission(org.apache.hadoop.fs.FileSystem fs,
                                                 org.apache.hadoop.fs.Path path,
                                                 org.apache.hadoop.fs.permission.FsPermission perm)
                                          throws IOException

Create the given dir as well as all missing ancestor dirs. All created dirs will have the given permission. This should be used instead of FileSystem.mkdirs(Path, FsPermission), since that method only sets the permission for the given dir, and not recursively for the ancestor dirs.

Parameters:

fs - FileSystem

path - The dir to be created

perm - The permission to be set

Throws:

IOException - if failing to create dir or set permission.

mkdirsWithRecursivePermissionWithRetry

public static void mkdirsWithRecursivePermissionWithRetry(org.apache.hadoop.fs.FileSystem fs,
                                                          org.apache.hadoop.fs.Path path,
                                                          org.apache.hadoop.fs.permission.FsPermission perm,
                                                          com.typesafe.config.Config retrierConfig)
                                                   throws IOException

Throws:

IOException

getWriterFsUri

public static URI getWriterFsUri(State state,
                                 int numBranches,
                                 int branchId)

getWriterFS

public static org.apache.hadoop.fs.FileSystem getWriterFS(State state,
                                                          int numBranches,
                                                          int branchId)
                                                   throws IOException

Throws:

IOException

getWriterFs

public static org.apache.hadoop.fs.FileSystem getWriterFs(State state)
                                                   throws IOException

Throws:

IOException

getFsConfiguration

public static org.apache.hadoop.conf.Configuration getFsConfiguration(State state)