org.apache.gobblin.util

Class FileListUtils

java.lang.Object

org.apache.gobblin.util.FileListUtils

public class FileListUtils
extends Object

Utility class for listing files on a FileSystem.

See Also:

FileSystem

Field Summary

Fields

Modifier and Type	Field and Description
static Comparator<org.apache.hadoop.fs.FileStatus>	LATEST_MOD_TIME_ORDER
static org.apache.hadoop.fs.PathFilter	NO_OP_PATH_FILTER

Constructor Summary

Constructors

Constructor and Description
FileListUtils()

Method Summary

Modifier and Type	Method and Description
static org.apache.hadoop.fs.FileStatus	getAnyNonHiddenFile(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Get any data file, which is not hidden or a directory, from the given path
static List<org.apache.hadoop.fs.FileStatus>	listFilesRecursively(org.apache.hadoop.fs.FileSystem fs, Iterable<org.apache.hadoop.fs.Path> paths)
static List<org.apache.hadoop.fs.FileStatus>	listFilesRecursively(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path)
static List<org.apache.hadoop.fs.FileStatus>	listFilesRecursively(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.PathFilter fileFilter) Helper method to list out all files under a specified path.
static List<org.apache.hadoop.fs.FileStatus>	listFilesRecursively(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.PathFilter fileFilter, boolean applyFilterToDirectories) Helper method to list out all files under a specified path.
static List<org.apache.hadoop.fs.FileStatus>	listFilesToCopyAtPath(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.PathFilter fileFilter, boolean includeEmptyDirectories) Given a path to copy, list all files rooted at the given path to copy
static List<org.apache.hadoop.fs.FileStatus>	listFilesToCopyAtPath(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.PathFilter fileFilter, boolean applyFilterToDirectories, boolean includeEmptyDirectories) Given a path to copy, list all files rooted at the given path to copy
static List<org.apache.hadoop.fs.FileStatus>	listMostNestedPathRecursively(org.apache.hadoop.fs.FileSystem fs, Iterable<org.apache.hadoop.fs.Path> paths)
static List<org.apache.hadoop.fs.FileStatus>	listMostNestedPathRecursively(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Method to list out all files, or directory if no file exists, under a specified path.
static List<org.apache.hadoop.fs.FileStatus>	listMostNestedPathRecursively(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.PathFilter fileFilter) Method to list out all files, or directory if no file exists, under a specified path.
static List<org.apache.hadoop.fs.FileStatus>	listPathsRecursively(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.PathFilter fileFilter) Helper method to list out all paths under a specified path.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LATEST_MOD_TIME_ORDER

public static final Comparator<org.apache.hadoop.fs.FileStatus> LATEST_MOD_TIME_ORDER

NO_OP_PATH_FILTER

public static final org.apache.hadoop.fs.PathFilter NO_OP_PATH_FILTER

Constructor Detail

FileListUtils

public FileListUtils()

Method Detail

listFilesRecursively

public static List<org.apache.hadoop.fs.FileStatus> listFilesRecursively(org.apache.hadoop.fs.FileSystem fs,
                                                                         org.apache.hadoop.fs.Path path)
                                                                  throws IOException

Throws:

IOException

listFilesRecursively

public static List<org.apache.hadoop.fs.FileStatus> listFilesRecursively(org.apache.hadoop.fs.FileSystem fs,
                                                                         Iterable<org.apache.hadoop.fs.Path> paths)
                                                                  throws IOException

Throws:

IOException

listFilesToCopyAtPath

public static List<org.apache.hadoop.fs.FileStatus> listFilesToCopyAtPath(org.apache.hadoop.fs.FileSystem fs,
                                                                          org.apache.hadoop.fs.Path path,
                                                                          org.apache.hadoop.fs.PathFilter fileFilter,
                                                                          boolean includeEmptyDirectories)
                                                                   throws IOException

Given a path to copy, list all files rooted at the given path to copy

Parameters:

fs - the file system of the path

path - root path to copy

fileFilter - a filter only applied to root

includeEmptyDirectories - a control to include empty directories for copy

Throws:

IOException

listFilesToCopyAtPath

public static List<org.apache.hadoop.fs.FileStatus> listFilesToCopyAtPath(org.apache.hadoop.fs.FileSystem fs,
                                                                          org.apache.hadoop.fs.Path path,
                                                                          org.apache.hadoop.fs.PathFilter fileFilter,
                                                                          boolean applyFilterToDirectories,
                                                                          boolean includeEmptyDirectories)
                                                                   throws IOException

Given a path to copy, list all files rooted at the given path to copy

Parameters:

fs - the file system of the path

path - root path to copy

fileFilter - a filter only applied to root

applyFilterToDirectories - a control to decide whether to apply filter to directories

includeEmptyDirectories - a control to include empty directories for copy

Throws:

IOException

listFilesRecursively

public static List<org.apache.hadoop.fs.FileStatus> listFilesRecursively(org.apache.hadoop.fs.FileSystem fs,
                                                                         org.apache.hadoop.fs.Path path,
                                                                         org.apache.hadoop.fs.PathFilter fileFilter)
                                                                  throws IOException

Helper method to list out all files under a specified path. The specified PathFilter is treated as a file filter, that is it is only applied to file Paths.

Throws:

IOException

listFilesRecursively

public static List<org.apache.hadoop.fs.FileStatus> listFilesRecursively(org.apache.hadoop.fs.FileSystem fs,
                                                                         org.apache.hadoop.fs.Path path,
                                                                         org.apache.hadoop.fs.PathFilter fileFilter,
                                                                         boolean applyFilterToDirectories)
                                                                  throws IOException

Helper method to list out all files under a specified path. If applyFilterToDirectories is false, the supplied PathFilter will only be applied to files.

Throws:

IOException

listMostNestedPathRecursively

public static List<org.apache.hadoop.fs.FileStatus> listMostNestedPathRecursively(org.apache.hadoop.fs.FileSystem fs,
                                                                                  org.apache.hadoop.fs.Path path)
                                                                           throws IOException

Method to list out all files, or directory if no file exists, under a specified path.

Throws:

IOException

listMostNestedPathRecursively

public static List<org.apache.hadoop.fs.FileStatus> listMostNestedPathRecursively(org.apache.hadoop.fs.FileSystem fs,
                                                                                  Iterable<org.apache.hadoop.fs.Path> paths)
                                                                           throws IOException

Throws:

IOException

listMostNestedPathRecursively

public static List<org.apache.hadoop.fs.FileStatus> listMostNestedPathRecursively(org.apache.hadoop.fs.FileSystem fs,
                                                                                  org.apache.hadoop.fs.Path path,
                                                                                  org.apache.hadoop.fs.PathFilter fileFilter)
                                                                           throws IOException

Method to list out all files, or directory if no file exists, under a specified path. The specified PathFilter is treated as a file filter, that is it is only applied to file Paths.

Throws:

IOException

listPathsRecursively

public static List<org.apache.hadoop.fs.FileStatus> listPathsRecursively(org.apache.hadoop.fs.FileSystem fs,
                                                                         org.apache.hadoop.fs.Path path,
                                                                         org.apache.hadoop.fs.PathFilter fileFilter)
                                                                  throws IOException

Helper method to list out all paths under a specified path. If the FileSystem is unable to list the contents of a relevant directory, will log an error and skip.

Throws:

IOException

getAnyNonHiddenFile

public static org.apache.hadoop.fs.FileStatus getAnyNonHiddenFile(org.apache.hadoop.fs.FileSystem fs,
                                                                  org.apache.hadoop.fs.Path path)
                                                           throws IOException

Get any data file, which is not hidden or a directory, from the given path

Throws:

IOException