org.apache.hadoop.hive.common
Class FileUtils

java.lang.Object
  org.apache.hadoop.hive.common.FileUtils

public final class FileUtils
extends Object

Collection of file manipulation utilities common across Hive.

Method Summary

static String	escapePathName(String path)
static String	escapePathName(String path, String defaultPath) Escapes a path name.
static org.apache.hadoop.fs.Path	getPathOrParentThatExists(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Find the parent of path that exists, if path does not exist
static boolean	isActionPermittedForFileHierarchy(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.FileStatus fileStatus, String userName, org.apache.hadoop.fs.permission.FsAction action) Check if user userName has permissions to perform the given FsAction action on all files under the file whose FileStatus fileStatus is provided
static boolean	isActionPermittedForUser(String userName, org.apache.hadoop.fs.FileStatus fsStatus, org.apache.hadoop.fs.permission.FsAction action) Check if the given FileStatus indicates that the action is allowed for userName.
static boolean	isLocalFile(HiveConf conf, String fileName) A best effort attempt to determine if if the file is a local file
static boolean	isLocalFile(HiveConf conf, URI fileUri) A best effort attempt to determine if if the file is a local file
static boolean	isOwnerOfFileHierarchy(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.FileStatus fileStatus, String userName)
static void	listStatusRecursively(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.FileStatus fileStatus, List<org.apache.hadoop.fs.FileStatus> results) Recursively lists status for all files starting from a particular directory (or individual file as base case).
static String	makeDefaultListBucketingDirName(List<String> skewedCols, String name) default directory will have the same depth as number of skewed columns this will make future operation easy like DML merge, concatenate merge
static String	makeListBucketingDirName(List<String> lbCols, List<String> vals) Makes a valid list bucketing directory name.
static String	makePartName(List<String> partCols, List<String> vals)
static String	makePartName(List<String> partCols, List<String> vals, String defaultStr) Makes a valid partition name.
static org.apache.hadoop.fs.Path	makeQualified(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf) Variant of Path.makeQualified that qualifies the input path against the default file system indicated by the configuration This does not require a FileSystem handle in most cases - only requires the Filesystem URI.
static String	unescapePathName(String path)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

makeQualified

public static org.apache.hadoop.fs.Path makeQualified(org.apache.hadoop.fs.Path path,
                                                      org.apache.hadoop.conf.Configuration conf)
                                               throws IOException

Variant of Path.makeQualified that qualifies the input path against the default file system indicated by the configuration This does not require a FileSystem handle in most cases - only requires the Filesystem URI. This saves the cost of opening the Filesystem - which can involve RPCs - as well as cause errors

Parameters:

path - path to be fully qualified

conf - Configuration file

Returns:

path qualified relative to default file system

Throws:

IOException

makePartName

public static String makePartName(List<String> partCols,
                                  List<String> vals)

makePartName

public static String makePartName(List<String> partCols,
                                  List<String> vals,
                                  String defaultStr)

Makes a valid partition name.

Parameters:

partCols - The partition keys' names

vals - The partition values

defaultStr - The default name given to a partition value if the respective value is empty or null.

Returns:

An escaped, valid partition name.

makeDefaultListBucketingDirName

public static String makeDefaultListBucketingDirName(List<String> skewedCols,
                                                     String name)

default directory will have the same depth as number of skewed columns this will make future operation easy like DML merge, concatenate merge

Parameters:

skewedCols -

name -

Returns:

makeListBucketingDirName

public static String makeListBucketingDirName(List<String> lbCols,
                                              List<String> vals)

Makes a valid list bucketing directory name.

Parameters:

lbCols - The skewed keys' names

vals - The skewed values

Returns:

An escaped, valid list bucketing directory name.

escapePathName

public static String escapePathName(String path)

escapePathName

public static String escapePathName(String path,
                                    String defaultPath)

Escapes a path name.

Parameters:

path - The path to escape.

defaultPath - The default name for the path, if the given path is empty or null.

Returns:

An escaped path name.

unescapePathName

public static String unescapePathName(String path)

listStatusRecursively

public static void listStatusRecursively(org.apache.hadoop.fs.FileSystem fs,
                                         org.apache.hadoop.fs.FileStatus fileStatus,
                                         List<org.apache.hadoop.fs.FileStatus> results)
                                  throws IOException

Recursively lists status for all files starting from a particular directory (or individual file as base case).

Parameters:

fs - file system

fileStatus - starting point in file system

results - receives enumeration of all files found

Throws:

IOException

getPathOrParentThatExists

public static org.apache.hadoop.fs.Path getPathOrParentThatExists(org.apache.hadoop.fs.FileSystem fs,
                                                                  org.apache.hadoop.fs.Path path)
                                                           throws IOException

Find the parent of path that exists, if path does not exist

Parameters:

fs - file system

path -

Returns:

the argument path if it exists or a parent path exists. Returns NULL root is only parent that exists

Throws:

IOException

isActionPermittedForUser

public static boolean isActionPermittedForUser(String userName,
                                               org.apache.hadoop.fs.FileStatus fsStatus,
                                               org.apache.hadoop.fs.permission.FsAction action)

Check if the given FileStatus indicates that the action is allowed for userName. It checks the group and other permissions also to determine this.

Parameters:

userName -

fsStatus -

action -

Returns:

true if it is writable for userName

isActionPermittedForFileHierarchy

public static boolean isActionPermittedForFileHierarchy(org.apache.hadoop.fs.FileSystem fs,
                                                        org.apache.hadoop.fs.FileStatus fileStatus,
                                                        String userName,
                                                        org.apache.hadoop.fs.permission.FsAction action)
                                                 throws IOException

Check if user userName has permissions to perform the given FsAction action on all files under the file whose FileStatus fileStatus is provided

Parameters:

fs -

fileStatus -

userName -

action -

Returns:

Throws:

IOException

isLocalFile

public static boolean isLocalFile(HiveConf conf,
                                  String fileName)

A best effort attempt to determine if if the file is a local file

Parameters:

conf -

fileName -

Returns:

true if it was successfully able to determine that it is a local file

isLocalFile

public static boolean isLocalFile(HiveConf conf,
                                  URI fileUri)

A best effort attempt to determine if if the file is a local file

Parameters:

conf -

fileUri -

Returns:

true if it was successfully able to determine that it is a local file

isOwnerOfFileHierarchy

public static boolean isOwnerOfFileHierarchy(org.apache.hadoop.fs.FileSystem fs,
                                             org.apache.hadoop.fs.FileStatus fileStatus,
                                             String userName)
                                      throws IOException

Throws:

IOException