Packages

package job

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. All

Type Members

  1. class DataFrameStatFunctions extends AnyRef

    Stat functions for dataframe

  2. case class FeathrGenPreparationInfo(sparkSession: SparkSession, applicationConfigPath: String, featureDefs: FeatureDefinitionsInput, jobContext: FeatureGenJobContext) extends Product with Serializable
  3. case class FeathrJoinJobContext(joinConfig: String, jobJoinContext: JoinJobContext, dataSourceConfigs: DataSourceConfigs) extends Product with Serializable
  4. case class FeathrJoinPreparationInfo(sparkSession: SparkSession, hadoopConf: Configuration, jobContext: FeathrJoinJobContext) extends Product with Serializable
  5. case class FeathrJointFeatureRecord[L](observation: L, joinedFeatures: Map[StringTaggedFeatureName, FeatureValue])(implicit evidence$1: ClassTag[L]) extends Product with Serializable

    This case class describes feature record after join process

  6. class FeatureDefinitionsInput extends AnyRef

  7. class FeatureGenJobContext extends AnyRef

    Feature generation job context

  8. class FeatureGenSpec extends AnyRef

    wrapper of generation config and context, also does not further processing on them.

  9. case class JoinJobContext(feathrLocalConfig: Option[String] = None, feathrFeatureConfig: Option[String] = None, inputData: Option[InputData] = None, outputPath: DataLocation = SimplePath("/join_output"), numParts: Int = 1) extends Product with Serializable

    Class the holds the context for the join job.

    Class the holds the context for the join job. This includes parameters specifying configs and observation data, among other parameters.

    feathrLocalConfig

    Optional. Path to local FeatureDef config, that is, config in user's own repo.

    feathrFeatureConfig

    Optional. Path to FeatureDef config.

    inputData

    Observation data

    outputPath

    HDFS path to which the joined observation and feature data will be written to

    numParts

    Number of output partitions

  10. case class LocalTestConfig(localConfig: Option[String], featureConfig: Option[String]) extends Product with Serializable

    Convenient class to hold config strings for local testing of workflows.

    Convenient class to hold config strings for local testing of workflows. See LocalFeatureJoinJob.

    localConfig

    Optional local config string

    featureConfig

    FeatureDef config string

Value Members

  1. object DataSourceUtils
  2. object FeatureGenJob
  3. object FeatureGenJobContext
  4. object FeatureGenSpec
  5. object FeatureJoinJob

    Join features to some observations for training/testing

    Join features to some observations for training/testing

    Output Format: For some input observations having schema S, the output will have the following schema 

    {
S,
featureGroupA: [ { name: string, term: string, value: double } ],
featureGroupB: [ { name: string, term: string, value: double } ],
...
}

    Note: Most of the actual join logic is found in FeathrClient.joinFeatures. This class is mostly concerned with input/output serialization and formatting, configuration, and wrapping the join logic with a job we can invoke via Azkaban.

  6. object JoinJobContext extends Serializable
  7. object LocalFeatureGenJob

    This object is used to help user to test/debug their workflow locally with sample data.

    This object is used to help user to test/debug their workflow locally with sample data. This class serves as a wrapper around FeathrClient class and invokes the FeathrClient#generateFeatures method by creating a spark session.

  8. object LocalFeatureJoinJob

    This object is used to help user to test/debug their workflow locally with sample data.

    This object is used to help user to test/debug their workflow locally with sample data. This class serves as a wrapper around FeathrClient class and invokes the FeathrClient#joinFeatures method by creating a spark session.

  9. object PreprocessedDataFrameManager

    Object to keep the preprocessed DataFrames by Pyspark.

Ungrouped