Packages

c

com.linkedin.feathr.sparkcommon

SourceKeyExtractor

abstract class SourceKeyExtractor extends Serializable

This is the base class for user customized key extractor for a source (feature dataset, rather than observation dataset) in an anchor, it is used in the 'keyExtractor' of an anchor, 'keyExtractor' is a replacement for 'key' section in the anchor. Note that, though this is called 'keyExtractor', it is actually a general purpose interface used to generate the fact table that will be passed to anchorExtractors later. Arbitrary transformations are allowed in appendKeyColumns(), only requirement is that it generates the key columns in the output dataframe

For example, swaAnchorWithKeyExtractor: { source: "swaSource" keyExtractor: "com.linkedin.feathr.offline.SimpleSampleKeyExtractor" features: { f: { def: "aggregationWindow" aggregation: SUM window: 3d } } }

We provide a few default source key extractors, such as MVELSourceKeyExtractor, SQLSourceKeyExtractor, SpecificRecordSourceKeyExtractor in repo 'feathr'. So internally, Feathr will create one of these source key extractors, if 'key' section is being used instead of 'keyExtractor' in the anchor.

There're also some simple examples such as SimpleSampleKeyExtractor, SimpleSampleKeyExtractor2 in repo 'feathr'

Linear Supertypes
Serializable, Serializable, AnyRef, Any
Known Subclasses
SQLSourceKeyExtractor
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SourceKeyExtractor
  2. Serializable
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new SourceKeyExtractor()

Abstract Value Members

  1. abstract def appendKeyColumns(dataFrame: DataFrame): DataFrame

    Append the key columns specified in getKeyColumnNames to the input dataframe This function will be called on the feature source dataframe before join with observation, to prepare the 'dataFrameWithKeyColumns' dataframe which is the input parameters of GenericAnchorExtractorSpark and SimpleAnchorExtractorSpark You may do any necessary transformations (e.g, explode/filter/group the dataframe, even run a spark job) in this function to prepare the input dataframe (or fact table) for your anchor extractor.

    Append the key columns specified in getKeyColumnNames to the input dataframe This function will be called on the feature source dataframe before join with observation, to prepare the 'dataFrameWithKeyColumns' dataframe which is the input parameters of GenericAnchorExtractorSpark and SimpleAnchorExtractorSpark You may do any necessary transformations (e.g, explode/filter/group the dataframe, even run a spark job) in this function to prepare the input dataframe (or fact table) for your anchor extractor.

    dataFrame

    source dataframe to work with

    returns

    input dataframe with key columns appended

  2. abstract def getKeyColumnNames(datum: Option[Any] = None): Seq[String]

    Get key column names in the source dataframe, the key columns will be used as join keys to join with observation data.

    Get key column names in the source dataframe, the key columns will be used as join keys to join with observation data. Note: Some extractors such as MVELSourceKeyExtractor and SpecificRecordSourceKeyExtractor need a sample row of the dataset to determine number of key columns, for most of user customized source key extractor, the input datum may be ignored.

    datum

    a sample row of the dataset, could be None if the dataset is empty

    returns

    the key column names of the source in the current anchor

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def getKeyColumnAlias(datum: Option[Any] = None): Seq[String]

    return list of alias for each key column name an alias should contain only alphaNumeric characters

    return list of alias for each key column name an alias should contain only alphaNumeric characters

    datum

    a sample row of the dataset, could be None if the dataset is empty

  11. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. def init(_params: Config): Unit

    init with parameter map

  13. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  14. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. var params: Option[Config]
  18. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  19. def toString(): String
    Definition Classes
    SourceKeyExtractor → AnyRef → Any
  20. def validateInputSchema(schema: StructType): Unit

    Check the validity of the input DataFrame, raise an exception if the schema is invalid, e.g, does not contain required input columns or has incorrect column types It is the developer's responsibility to validate the input schema's validity

  21. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  22. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  23. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped