org.apache.gobblin.util.recordcount

Class LateFileRecordCountProvider

java.lang.Object

org.apache.gobblin.util.RecordCountProvider

org.apache.gobblin.util.recordcount.LateFileRecordCountProvider

public class LateFileRecordCountProvider
extends RecordCountProvider

Constructor Summary

Constructors

Constructor and Description
LateFileRecordCountProvider()

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.fs.Path	constructLateFilePath(String originalFilename, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path outputDir) Construct filename for a late file.
long	getRecordCount(org.apache.hadoop.fs.Path path) Get record count from a given filename (possibly having LATE_COMPONENT{RandomInteger}), using the original FileNameWithRecordCountFormat.
static org.apache.hadoop.fs.Path	restoreFilePath(org.apache.hadoop.fs.Path path) Remove the late components in the path added by LateFileRecordCountProvider.

Methods inherited from class org.apache.gobblin.util.RecordCountProvider

convertPath, getNotImplementedException, getRecordCount

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LateFileRecordCountProvider

public LateFileRecordCountProvider()

Method Detail

constructLateFilePath

public org.apache.hadoop.fs.Path constructLateFilePath(String originalFilename,
                                                       org.apache.hadoop.fs.FileSystem fs,
                                                       org.apache.hadoop.fs.Path outputDir)
                                                throws IOException

Construct filename for a late file. If the file does not exists in the output dir, retain the original name. Otherwise, append a LATE_COMPONENT{RandomInteger} to the original file name. For example, if file "part1.123.avro" exists in dir "/a/b/", the returned path will be "/a/b/part1.123.late12345.avro".

Throws:

IOException

restoreFilePath

public static org.apache.hadoop.fs.Path restoreFilePath(org.apache.hadoop.fs.Path path)

Remove the late components in the path added by LateFileRecordCountProvider.

getRecordCount

public long getRecordCount(org.apache.hadoop.fs.Path path)

Get record count from a given filename (possibly having LATE_COMPONENT{RandomInteger}), using the original FileNameWithRecordCountFormat.

Specified by:

getRecordCount in class RecordCountProvider