final def !=(arg0: Any): Boolean

Definition Classes: AnyRef → Any

final def ##(): Int

Definition Classes: AnyRef → Any

final def ==(arg0: Any): Boolean

Definition Classes: AnyRef → Any

def appendDateFormatted(outputPath: String, dateTimeFormatter: DateTimeFormatter, localDateTime: LocalDateTime): String

Append a date formatted string to the back of an output path

outputPath: output path that should be appended to.
dateTimeFormatter: DateTimeFormatter to use.
returns: modified output path with a date representation.

final def asInstanceOf[T0]: T0

Definition Classes: Any

def clone(): AnyRef

Attributes: protected[lang]
Definition Classes: AnyRef
Annotations: @throws( ... ) @native()

def createDailyOutputPath(outputPath: String, localDateTime: LocalDateTime = LocalDateTime.now()): String

Return a daily directory for an outputPath.

Return a daily directory for an outputPath. Daily directories by default follow the [path]/daily/yyyy/MM/dd format, but may be overriden.

def createLatestOutputPath(outputPath: String, localDateTime: LocalDateTime = LocalDateTime.now()): String

Return a latest directory for an outputPath.

Return a latest directory for an outputPath. Snapshot directories follow a timeStampFormatter containing the year, month, day, and time of day.

def createStringPath(pathParts: String*): String

Creates a File Path given one or multiple parts and returns it normalized (no multiple consecutive or trailing "/")

pathParts: A Varargs with the parts used to create the String path
returns: A normalized (no multiple consecutive or trailing "/") String path given for the provided path parts

val dailyDirFormatter: DateTimeFormatter

DateTimeFormatter used to form the commonly-used path for a date strings.

val dateStampFormatter: DateTimeFormatter

DateTimeFormatter used to format strings on the form yyyy-MM-dd

def deleteOlderPaths(basePath: String, maxPathsToKeep: Int, conf: Configuration = conf): Unit

Deletes older paths.

Deletes older paths. A path is old if there are at least maxPathsToKeep paths generated after it. Paths are sorted in a consistent way with the getLatestPath() method: reverse lexicographic order

basePath: The input path to clean up
maxPathsToKeep: Max number of folders that will be kept in the basePath

def deletePath(path: String, recursive: Boolean = true, conf: Configuration = conf): Boolean

Deletes the given path in the FileSystem (typically HDFS) specified by the given configuration.

path: The String path to be deleted
recursive: Whether to delete recursively or not. Defaults to true.
conf: Configuration
returns: true if delete is successful else false.

final def eq(arg0: AnyRef): Boolean

Definition Classes: AnyRef

def equals(arg0: Any): Boolean

Definition Classes: AnyRef → Any

def exists(path: String, conf: Configuration = conf): Boolean

Checks if the given path in exists.

path: The String path to check
conf: Configuration
returns: true if exists else false.

def filterPath(path: String, excludeFilesPrefixList: Seq[String] = List(".", "_")): Boolean

Returns a boolean values to test if the file or directory name starts with any character in the excludeFilesPrefixList.

path: The file/directory path that needs to be analysed for filtering.
excludeFilesPrefixList: Sequence of file prefixes that needs to be excluded while reading.
returns: boolean (true) if the file or directory starts with any character in excludeFilesPrefixList, false otherwise.

def finalize(): Unit

Attributes: protected[lang]
Definition Classes: AnyRef
Annotations: @throws( classOf[java.lang.Throwable] )

def getAllFilesOfGivenType(path: String, fileType: String = "", recursive: Boolean = true, errorOnMissingFiles: Boolean = true, excludeFilesPrefixList: Seq[String] = List(".", "_"), conf: Configuration = conf): Seq[String]

Returns all files under path with a given fileType.

Returns all files under path with a given fileType. Pass empty fileType to return all files. If errorOnMissingFiles is set to true, a FileNotFoundException exception is thrown if the path does not have files of type fileType. If a file, or any of its parent directories under path, starts with any char in excludeFilesPrefixList, it will be excluded.

final def getClass(): Class[_]

Definition Classes: AnyRef → Any
Annotations: @native()

def getLatestPath(inputPath: String, conf: Configuration = conf): String

Replaces the latestSubdirectory identifier with the latest subdirectory in the path.

Replaces the latestSubdirectory identifier with the latest subdirectory in the path. The latestSubdirectory identifier can appear multiple times with or without leading/training "/" An inputPath must not be null or empty. It might not contain the identifier.

inputPath: The inputPath
conf: Config
returns: A normalized (no multiple consecutive or trailing "/") String with all the identifiers replaced with the latest subdirectory.

def getPaths(basePath: String, startInclusive: LocalDateTime, endExclusive: LocalDateTime, unit: ChronoUnit, formatter: DateTimeFormatter): Seq[String]

Returns a Seq of paths that are prefixed with basePath, and that cover the period [startInclusive, endExclusive).

Returns a Seq of paths that are prefixed with basePath, and that cover the period [startInclusive, endExclusive). The dates are formatted using formatter.

basePath: The prefix path under which the dated directories exist.
startInclusive: The start time (e.g., 2017/01/01). Included in the returned paths.
endExclusive: The end time (e.g., 2017/01/03). Not included in the returned paths.
unit: The time unit (e.g., ChronoUnit.DAYS).
formatter: A DateTimeFormatter for formatting the dates in the paths.
returns: A Seq of paths under basePath covering the period [startInclusive, endExclusive).

def getPaths(basePath: String, startInclusive: LocalDateTime, endExclusive: LocalDateTime, unit: ChronoUnit): Seq[String]

Returns a Seq of paths that are prefixed with basePath, and that cover the period [startInclusive, endExclusive).

Returns a Seq of paths that are prefixed with basePath, and that cover the period [startInclusive, endExclusive). The dates are formatted using TemporalPathFormats or any object similarly defined.

basePath: The prefix path under which the dated directories exist.
startInclusive: The start time (e.g., 2017/01/01). Included in the returned paths.
endExclusive: The end time (e.g., 2017/01/03). Not included in the returned paths.
unit: The time unit (e.g., ChronoUnit.DAYS).
returns: A Seq of paths under basePath covering the period [startInclusive, endExclusive).

def getPrintStream(path: String): PrintStream

Returns PrintStream from path

def hashCode(): Int

Definition Classes: AnyRef → Any
Annotations: @native()

def hdfsConvertIteratorToInputStream(User: String, tempDirPrefix: String, iter: Iterator[String], attempts: Int = maxAttempts, conf: Configuration = conf): Option[InputStream]

Converts Iterator to InputStream on HDFS

def hdfsCreateDirectoriesAsNeeded(dir: String, conf: Configuration = conf): Unit

Creaete directories as needed

def hdfsCreateTempFile(User: String, tempDir: String, prefix: String, suffix: String, attempts: Int = maxAttempts, conf: Configuration = conf): Option[String]

Returns temp file path

def hdfsSubdir(inputPath: String, excludePathsPrefixList: Seq[String] = List(".", "_"), conf: Configuration = conf): Array[String]

For a given input path, provides sub folders.

inputPath: input path
conf: Hadoop Configuration
returns: Array[String]

final def isInstanceOf[T0]: Boolean

Definition Classes: Any

val latestSubdirectory: String

Identifier for latest subdirectory.

Identifier for latest subdirectory. Used by getLatestPath() e.g. "/data/path/#LATEST"

def listFiles(path: String, recursive: Boolean = true, excludeFilesPrefixList: Seq[String] = List(".", "_"), conf: Configuration = conf): Seq[String]

Gets a listing of all files rooted at the given directory.

Gets a listing of all files rooted at the given directory. Files starting with a prefix in excludeFilesPrefixList are excluded. By default, the function filters out the files/directories that starts with either . or _.

val maxAttempts: Int

Default number of attempts to create a temporary file

final def ne(arg0: AnyRef): Boolean

Definition Classes: AnyRef

final def notify(): Unit

Definition Classes: AnyRef
Annotations: @native()

final def notifyAll(): Unit

Definition Classes: AnyRef
Annotations: @native()

val pstZoneId: ZoneId

PST ZoneId

def renamePath(src: String, dst: String, conf: Configuration = conf): Boolean

Rename the src path to dst path in the FileSystem (typically HDFS) specified by the given configuration.

Rename the src path to dst path in the FileSystem (typically HDFS) specified by the given configuration. Rename could be a tricky action. please refer to the HDFS official documentation for behaviors. E.g. hadoop-project-dist/hadoop-common/filesystem/filesystem.html

src: The src path as string
dst: The dst path as string
conf: Configuration
returns: true if rename is successful else false.

def replaceTimestamp(inputPath: String, localDateTime: LocalDateTime = LocalDateTime.now(ZoneOffset.UTC), dateTimeFormatterPattern: DateTimeFormatter = timeStampFormatter): String

Replaces the timestampIdentifier identifier with the given LocalDateTime in the provided format It could be used to partition different outputs of a flow within the same run and have a nicer folder structure.

Replaces the timestampIdentifier identifier with the given LocalDateTime in the provided format It could be used to partition different outputs of a flow within the same run and have a nicer folder structure. e.g.

/path/2016-05-25-10-15/trainingDataset
/path/2016-05-25-10-15/testDataset
/path/2016-05-25-10-15/scoringDataset

NOTE: The localDateTime defaults to UTC. To conform to existing datasets (e.g. Tracking Data) you might want to use the Pacific time zone instead (e.g. replaceTimestamp("path", LocalDateTime.now(pstZoneId)))

inputPath: The input path
localDateTime: The LocalDateTime object used to replace the timestampIdentifier. Defaults to current time in UTC and can be overridden (e.g. replaceTimestamp("path", LocalDateTime.now(pstZoneId)))
dateTimeFormatterPattern: The DateTimeFormatter used to format the localDateTime above. Defaults to timeStampFormatter
returns: A normalized (no multiple consecutive or trailing "/") String with all the identifiers replaced with the provided localDateTime in the given format

def setupLoggerForHDFS(conf: Configuration, logger: Logger, path: String): Unit

Set up to write logs to HDFS at given path.

def stripPrefix(prefixedPath: String, basePath: String): String

Strips hdfs://..., and similar prefixes.

Strips hdfs://..., and similar prefixes. Example: stripPrefix("hdfs://xyz.linkedin/a/b/c/foo/bar", "/a/b/c/") => "/a/b/c/foo/bar"

final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes: AnyRef

val timeStampFormatter: DateTimeFormatter

DateTimeFormatter used to format strings on the form yyyy-MM-dd-HH-mm

val timestampIdentifier: String

Identifier for using a datetime in the path Used by replaceTimestamp() e.g.

Identifier for using a datetime in the path Used by replaceTimestamp() e.g. "/output/path/#TIMESTAMP/data" || "/output/path/#TIMESTAMP/metadata"

def toString(): String

Definition Classes: AnyRef → Any

final def wait(): Unit

Definition Classes: AnyRef
Annotations: @throws( ... )

final def wait(arg0: Long, arg1: Int): Unit

Definition Classes: AnyRef
Annotations: @throws( ... )

final def wait(arg0: Long): Unit

Definition Classes: AnyRef
Annotations: @throws( ... ) @native()

object TemporalPathFormats

An object for defining the date/hour formats in the dated directories.

An object for defining the date/hour formats in the dated directories. This object is used by getPaths to format the returned paths. An example of how this object can be overridden to change the format exists in the test.

Packages

HdfsUtils

object HdfsUtils

Value Members

Inherited from AnyRef

Inherited from Any

Ungrouped