org.apache.gobblin.util.recordcount

Class CompactionRecordCountProvider

java.lang.Object

org.apache.gobblin.util.RecordCountProvider

org.apache.gobblin.util.recordcount.CompactionRecordCountProvider

public class CompactionRecordCountProvider
extends RecordCountProvider

Implementation of RecordCountProvider, which provides record count from file path. The file name should follow the pattern: {Prefix}{RecordCount}.{SystemCurrentTimeInMills}.{RandomInteger}{SUFFIX}. The prefix should be either M_OUTPUT_FILE_PREFIX or MR_OUTPUT_FILE_PREFIX. For example, given a file path: "/a/b/c/part-m-123.1444437036.12345.avro", the record count will be 123.

Field Summary

Fields

Modifier and Type	Field and Description
static String	M_OUTPUT_FILE_PREFIX
static String	MR_OUTPUT_FILE_PREFIX

Constructor Summary

Constructors

Constructor and Description
CompactionRecordCountProvider()

Method Summary

Modifier and Type	Method and Description
static String	constructFileName(String filenamePrefix, long recordCount) Deprecated. discouraged since default behavior is not obvious from API itself.
static String	constructFileName(String filenamePrefix, String extension, long recordCount) Construct the file name as {filenamePrefix}{recordCount}.{SystemCurrentTimeInMills}.{RandomInteger}{extension}.
org.apache.hadoop.fs.Path	convertPath(org.apache.hadoop.fs.Path path, String extension, RecordCountProvider src) This method currently supports converting the given Path from IngestionRecordCountProvider.
long	getRecordCount(org.apache.hadoop.fs.Path filepath) Get the record count through filename.

Methods inherited from class org.apache.gobblin.util.RecordCountProvider

getNotImplementedException, getRecordCount

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MR_OUTPUT_FILE_PREFIX

public static final String MR_OUTPUT_FILE_PREFIX

See Also:

Constant Field Values

M_OUTPUT_FILE_PREFIX

public static final String M_OUTPUT_FILE_PREFIX

See Also:

Constant Field Values

Constructor Detail

CompactionRecordCountProvider

public CompactionRecordCountProvider()

Method Detail

constructFileName

@Deprecated
public static String constructFileName(String filenamePrefix,
                                                   long recordCount)

Deprecated. discouraged since default behavior is not obvious from API itself.

Construct the file name as {filenamePrefix}{recordCount}.{SystemCurrentTimeInMills}.{RandomInteger}{SUFFIX}.

constructFileName

public static String constructFileName(String filenamePrefix,
                                       String extension,
                                       long recordCount)

Construct the file name as {filenamePrefix}{recordCount}.{SystemCurrentTimeInMills}.{RandomInteger}{extension}.

getRecordCount

public long getRecordCount(org.apache.hadoop.fs.Path filepath)

Get the record count through filename.

Specified by:

getRecordCount in class RecordCountProvider

convertPath

public org.apache.hadoop.fs.Path convertPath(org.apache.hadoop.fs.Path path,
                                             String extension,
                                             RecordCountProvider src)

This method currently supports converting the given Path from IngestionRecordCountProvider. The converted Path will start with M_OUTPUT_FILE_PREFIX.

Overrides:

convertPath in class RecordCountProvider