Packages

c

com.linkedin.feathr.sparkcommon

FeatureDerivationFunctionSpark

abstract class FeatureDerivationFunctionSpark extends FeatureDerivationFunctionBase

Spark DataFrame based derived feature base class, Feathr user can extend this class to implement their customized derivation function to produce derived features

see TestDataFrameDerivationFunctionExtractor in feathr repo for example

Linear Supertypes
FeatureDerivationFunctionBase, Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. FeatureDerivationFunctionSpark
  2. FeatureDerivationFunctionBase
  3. Serializable
  4. Serializable
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new FeatureDerivationFunctionSpark()

Abstract Value Members

  1. abstract def transform(dataframe: DataFrame): DataFrame

    Append the derived feature column, the column name should be the same as declared in the Feathr featureDef file E.g, in the following TestDataFrameDerivationFunctionExtractor (extends FeatureDerivationFunctionSpark), the input dataframe will contain dependent feature columns "f1" and "f2", output DataFrame should contain the output column 'E' derivations: { E: { key: [x, y] inputs: { f1: { key: x, feature: C }, f2: { key: y, feature: D } } class: "com.linkedin.feathr.offline.TestDataFrameDerivationFunctionExtractor" }

    Append the derived feature column, the column name should be the same as declared in the Feathr featureDef file E.g, in the following TestDataFrameDerivationFunctionExtractor (extends FeatureDerivationFunctionSpark), the input dataframe will contain dependent feature columns "f1" and "f2", output DataFrame should contain the output column 'E' derivations: { E: { key: [x, y] inputs: { f1: { key: x, feature: C }, f2: { key: y, feature: D } } class: "com.linkedin.feathr.offline.TestDataFrameDerivationFunctionExtractor" }

    dataframe

    context DataFrame with all dependency feature columns ready to use, the input columns could be of any valid org.apache.spark.sql.types.DataType

    returns

    the transformed DataFrame

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. var _params: Option[Config]
    Definition Classes
    FeatureDerivationFunctionBase
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  9. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  10. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. def getInputFeatureList(): Seq[JoiningFeatureParams]

    return dependency/input feature list This function is only used in derivation function that produces multiple derived features, i.e., derived features defined within 'advancedDerivations' section.

    return dependency/input feature list This function is only used in derivation function that produces multiple derived features, i.e., derived features defined within 'advancedDerivations' section. Derived features defined in 'derivations' section do not need to override this.

    Definition Classes
    FeatureDerivationFunctionBase
  12. def getOutputFeatureList(): Seq[String]

    return output feature list Used together with getInputFeatureList()

    return output feature list Used together with getInputFeatureList()

    Definition Classes
    FeatureDerivationFunctionBase
  13. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  14. def init(params: Config): Unit

    init with parameter map

    init with parameter map

    Definition Classes
    FeatureDerivationFunctionBase
  15. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  16. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  17. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  18. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  19. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  20. def toString(): String
    Definition Classes
    AnyRef → Any
  21. def validateInputSchema(schema: StructType): Unit

    Check the validity of the input DataFrame, raise an exception if the schema is invalid, e.g, does not contain required input columns or has incorrect column types It is the developer's responsibility to validate the input schema's validity

  22. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  23. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from FeatureDerivationFunctionBase

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped