Value Members

1.  final def !=(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

2.  final def ##(): Int

   Definition Classes

   AnyRef → Any

3.  final def ==(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

4.  def appendKeyColumns(dataFrame: DataFrame, evaluateLateralViewParams: Boolean): DataFrame

   append key columns considering lateral view parameters

   append key columns considering lateral view parameters

   dataFrame

   source dataframe

   evaluateLateralViewParams

   evaluate laterView parameters or not, in feature generation, set to true, in feature join, set to false. Because in feature generation, we simply treat lateral view as a special keyExtractor, but in feature join, the spark SWJ library will handle it in different way.

   returns

   source dataframe with key columns appended

5.  def appendKeyColumns(dataFrame: DataFrame): DataFrame

   Append the key columns specified in getKeyColumnNames to the input dataframe This function will be called on the feature source dataframe before join with observation, to prepare the 'dataFrameWithKeyColumns' dataframe which is the input parameters of GenericAnchorExtractorSpark and SimpleAnchorExtractorSpark You may do any necessary transformations (e.g, explode/filter/group the dataframe, even run a spark job) in this function to prepare the input dataframe (or fact table) for your anchor extractor.

   Append the key columns specified in getKeyColumnNames to the input dataframe This function will be called on the feature source dataframe before join with observation, to prepare the 'dataFrameWithKeyColumns' dataframe which is the input parameters of GenericAnchorExtractorSpark and SimpleAnchorExtractorSpark You may do any necessary transformations (e.g, explode/filter/group the dataframe, even run a spark job) in this function to prepare the input dataframe (or fact table) for your anchor extractor.

   dataFrame

   source dataframe to work with

   returns

   input dataframe with key columns appended

   Definition Classes

   SQLSourceKeyExtractor → SourceKeyExtractor

6.  final def asInstanceOf[T0]: T0

   Definition Classes

   Any

7.  def clone(): AnyRef

   Attributes

   protected[lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... ) @native()

8.  final def eq(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

9.  def equals(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

10.  def finalize(): Unit

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

11.  final def getClass(): Class[_]

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

12.  def getKeyColumnAlias(datum: Option[Any]): Seq[String]

    return key column alias, if not set explicitly, will return the key sql expression with non-alphanumeric removed

    return key column alias, if not set explicitly, will return the key sql expression with non-alphanumeric removed

    datum

    sample row of the dataset, not used in this case

    Definition Classes

    SQLSourceKeyExtractor → SourceKeyExtractor

13.  def getKeyColumnNames(datum: Option[Any]): Seq[String]

    Get key column names in the source dataframe, the key columns will be used as join keys to join with observation data.

    Get key column names in the source dataframe, the key columns will be used as join keys to join with observation data. Note: Some extractors such as MVELSourceKeyExtractor and SpecificRecordSourceKeyExtractor need a sample row of the dataset to determine number of key columns, for most of user customized source key extractor, the input datum may be ignored.

    datum

    a sample row of the dataset, could be None if the dataset is empty

    returns

    the key column names of the source in the current anchor

    Definition Classes

    SQLSourceKeyExtractor → SourceKeyExtractor

14.  def hashCode(): Int

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

15.  def init(_params: Config): Unit

    init with parameter map

    init with parameter map

    Definition Classes

    SourceKeyExtractor

16.  final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

17.  val keyExprs: Seq[String]
18.  final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

19.  final def notify(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

20.  final def notifyAll(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

21.  var params: Option[Config]

    Definition Classes

    SourceKeyExtractor

22.  lazy val ss: SparkSession
23.  final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes

    AnyRef

24.  def toString(): String

    Definition Classes

    SQLSourceKeyExtractor → SourceKeyExtractor → AnyRef → Any

25.  def validateInputSchema(schema: StructType): Unit

    Check the validity of the input DataFrame, raise an exception if the schema is invalid, e.g, does not contain required input columns or has incorrect column types It is the developer's responsibility to validate the input schema's validity

    Check the validity of the input DataFrame, raise an exception if the schema is invalid, e.g, does not contain required input columns or has incorrect column types It is the developer's responsibility to validate the input schema's validity

    Definition Classes

    SourceKeyExtractor

26.  final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

27.  final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

28.  final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... ) @native()