Packages

o

com.linkedin.feathr.offline.job

FeatureJoinJob

object FeatureJoinJob

Join features to some observations for training/testing

Output Format: For some input observations having schema S, the output will have the following schema 

{
S,
featureGroupA: [ { name: string, term: string, value: double } ],
featureGroupB: [ { name: string, term: string, value: double } ],
...
}

Note: Most of the actual join logic is found in FeathrClient.joinFeatures. This class is mostly concerned with input/output serialization and formatting, configuration, and wrapping the join logic with a job we can invoke via Azkaban.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. FeatureJoinJob
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. type FeatureName = String
  2. type KeyTag = Seq[String]

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. val SKIP_OUTPUT: String
  5. val SPARK_JOIN_LIMIT_PARTITION_FACTOR: Int
  6. val SPARK_JOIN_MAX_PARALLELISM: String
  7. val SPARK_JOIN_MIN_PARALLELISM: String
  8. val SPARK_JOIN_PARALLELISM_DEFAULT: String
  9. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  10. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  11. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  12. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  13. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  14. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  15. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  16. def hdfsFileReader(ss: SparkSession, path: String): String
  17. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  18. def loadSourceDataframe(args: Array[String], featureNamesInAnchorSet: Set[String]): Map[String, DataFrame]

    Load the DataFrames for sources that needs preprocessing by Pyspark.

    Load the DataFrames for sources that needs preprocessing by Pyspark.

    args

    Same arguments for the main job.

    featureNamesInAnchorSet

    A set of feature names of an anchor sorted and joined by comma. For example, anchor1 -> f1, f2, anchor2 -> f3, f4. Then the set is ("f1,f2", "f3,f4")

    returns

    A Java map whose key is Feature names of an anchor sorted and joined by comma and value is the DataFrame for this anchor source. For example, anchor1 -> f1, f2, anchor2 -> f3, f4. Then the result is Map("f1,f2" -> df1, "f3,f4" -> df2).

  19. val log: Logger
  20. val logger: Logger
  21. def main(args: Array[String]): Unit
  22. def mainWithPreprocessedDataFrame(args: Array[String], preprocessedDfMap: Map[String, DataFrame]): Unit
  23. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  24. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  25. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  26. def parseInputArgument(args: Array[String]): FeathrJoinJobContext

    Parse command line arguments into the join job context.

    Parse command line arguments into the join job context.

    args

    command line arguments

    returns

    join job context

  27. def parseJoinConfig(joinConfString: String): FeatureJoinConfig

  28. def prepareSparkSession(args: Array[String]): FeathrJoinPreparationInfo
  29. def run(ss: SparkSession, hadoopConf: Configuration, jobContext: FeathrJoinJobContext, dataPathHandlers: List[DataPathHandler]): Unit
  30. def stringifyFeatureNames(nameSet: Set[String]): String
  31. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  32. def toString(): String
    Definition Classes
    AnyRef → Any
  33. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  34. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  35. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from AnyRef

Inherited from Any

Ungrouped