org.apache.gobblin.util

Class PullFileLoader

java.lang.Object

org.apache.gobblin.util.PullFileLoader

public class PullFileLoader
extends Object

Used to load pull files from the file system.

Field Summary

Fields

Modifier and Type	Field and Description
static Set<String>	DEFAULT_HOCON_PULL_FILE_EXTENSIONS
static Set<String>	DEFAULT_JAVA_PROPS_PULL_FILE_EXTENSIONS
static boolean	DEFAULT_PROPERTY_DELIMITER_PARSING_ENABLED_KEY
static String	GLOBAL_HOCON_EXTENSION
static org.apache.hadoop.fs.PathFilter	GLOBAL_HOCON_PATH_FILTER
static org.apache.hadoop.fs.PathFilter	GLOBAL_PATH_FILTER
static String	GLOBAL_PROPS_EXTENSION
static org.apache.hadoop.fs.PathFilter	GLOBAL_PROPS_PATH_FILTER
static String	PROPERTY_DELIMITER_PARSING_ENABLED_KEY

Constructor Summary

Constructors

Constructor and Description
PullFileLoader(org.apache.hadoop.fs.Path rootDirectory, org.apache.hadoop.fs.FileSystem fs, Collection<String> javaPropsPullFileExtensions, Collection<String> hoconPullFileExtensions)

Method Summary

Modifier and Type	Method and Description
List<org.apache.hadoop.fs.Path>	fetchJobFilesRecursively(org.apache.hadoop.fs.Path path)
com.typesafe.config.Config	loadPullFile(org.apache.hadoop.fs.Path path, com.typesafe.config.Config sysProps, boolean loadGlobalProperties)
com.typesafe.config.Config	loadPullFile(org.apache.hadoop.fs.Path path, com.typesafe.config.Config sysProps, boolean loadGlobalProperties, boolean resolve) Load a single pull file.
List<com.typesafe.config.Config>	loadPullFilesRecursively(org.apache.hadoop.fs.Path path, com.typesafe.config.Config sysProps, boolean loadGlobalProperties) Find and load all pull files under a base Path recursively in an order sorted by last modified date.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

GLOBAL_PROPS_EXTENSION

public static final String GLOBAL_PROPS_EXTENSION

See Also:

Constant Field Values

GLOBAL_PROPS_PATH_FILTER

public static final org.apache.hadoop.fs.PathFilter GLOBAL_PROPS_PATH_FILTER

GLOBAL_HOCON_EXTENSION

public static final String GLOBAL_HOCON_EXTENSION

See Also:

Constant Field Values

GLOBAL_HOCON_PATH_FILTER

public static final org.apache.hadoop.fs.PathFilter GLOBAL_HOCON_PATH_FILTER

GLOBAL_PATH_FILTER

public static final org.apache.hadoop.fs.PathFilter GLOBAL_PATH_FILTER

DEFAULT_JAVA_PROPS_PULL_FILE_EXTENSIONS

public static final Set<String> DEFAULT_JAVA_PROPS_PULL_FILE_EXTENSIONS

DEFAULT_HOCON_PULL_FILE_EXTENSIONS

public static final Set<String> DEFAULT_HOCON_PULL_FILE_EXTENSIONS

PROPERTY_DELIMITER_PARSING_ENABLED_KEY

public static final String PROPERTY_DELIMITER_PARSING_ENABLED_KEY

See Also:

Constant Field Values

DEFAULT_PROPERTY_DELIMITER_PARSING_ENABLED_KEY

public static final boolean DEFAULT_PROPERTY_DELIMITER_PARSING_ENABLED_KEY

See Also:

Constant Field Values

Constructor Detail

PullFileLoader

public PullFileLoader(org.apache.hadoop.fs.Path rootDirectory,
                      org.apache.hadoop.fs.FileSystem fs,
                      Collection<String> javaPropsPullFileExtensions,
                      Collection<String> hoconPullFileExtensions)

Method Detail

loadPullFile

public com.typesafe.config.Config loadPullFile(org.apache.hadoop.fs.Path path,
                                               com.typesafe.config.Config sysProps,
                                               boolean loadGlobalProperties,
                                               boolean resolve)
                                        throws IOException

Load a single pull file.

Parameters:

path - The Path to the pull file to load, full path

sysProps - A Config used as fallback.

loadGlobalProperties - if true, will also load at most one *.properties file per directory from the rootDirectory to the pull file Path.

resolve - if true, call Config.resolve() on the config after loading it

Returns:

The loaded Config.

Throws:

IOException

loadPullFile

public com.typesafe.config.Config loadPullFile(org.apache.hadoop.fs.Path path,
                                               com.typesafe.config.Config sysProps,
                                               boolean loadGlobalProperties)
                                        throws IOException

Throws:

IOException

loadPullFilesRecursively

public List<com.typesafe.config.Config> loadPullFilesRecursively(org.apache.hadoop.fs.Path path,
                                                                 com.typesafe.config.Config sysProps,
                                                                 boolean loadGlobalProperties)

Find and load all pull files under a base Path recursively in an order sorted by last modified date.

Parameters:

path - base Path where pull files should be found recursively.

sysProps - A Config used as fallback.

loadGlobalProperties - if true, will also load at most one *.properties file per directory from the rootDirectory to the pull file Path for each pull file.

Returns:

The loaded Configs. Files that fail to parse successfully will be logged, but will not result in a Config object

fetchJobFilesRecursively

public List<org.apache.hadoop.fs.Path> fetchJobFilesRecursively(org.apache.hadoop.fs.Path path)