Packages

  • package root
    Definition Classes
    root
  • package ai
    Definition Classes
    root
  • package catboost
    Definition Classes
    ai
  • package spark

    CatBoost is a machine learning algorithm that uses gradient boosting on decision trees.

    CatBoost is a machine learning algorithm that uses gradient boosting on decision trees.

    Overview

    This package provides classes that implement interfaces from Apache Spark Machine Learning Library (MLLib).

    For binary and multi- classification problems use CatBoostClassifier, for regression use CatBoostRegressor.

    These classes implement usual fit method of org.apache.spark.ml.Predictor that accept a single org.apache.spark.sql.DataFrame for training, but you can also use other fit method that accepts additional datasets for computing evaluation metrics and overfitting detection similarily to CatBoost's other APIs.

    This package also contains Pool class that is CatBoost's abstraction of a dataset. It contains additional information compared to simple org.apache.spark.sql.DataFrame.

    It is also possible to create Pool with quantized features before training by calling quantize method. This is useful if this dataset is used for training multiple times and quantization parameters do not change. Pre-quantized Pool allows to cache quantized features data and so do not re-run feature quantization step at the start of an each training.

    Detailed documentation is available on https://catboost.ai/docs/

    Definition Classes
    catboost
  • package impl
    Definition Classes
    spark
  • package params
    Definition Classes
    spark
  • CatBoostClassificationModel
  • CatBoostClassifier
  • CatBoostPredictorTrait
  • CatBoostRegressionModel
  • CatBoostRegressor
  • Pool

class Pool extends Params with HasLabelCol with HasFeaturesCol with HasWeightCol

CatBoost's abstraction of a dataset.

Features data can be stored in raw (features column has org.apache.spark.ml.linalg.Vector type) or quantized (float feature values are quantized into integer bin values, features column has Array[Byte] type) form.

Raw Pool can be transformed to quantized form using quantize method. This is useful if this dataset is used for training multiple times and quantization parameters do not change. Pre-quantized Pool allows to cache quantized features data and so do not re-run feature quantization step at the start of an each training.

Linear Supertypes
HasWeightCol, HasFeaturesCol, HasLabelCol, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Pool
  2. HasWeightCol
  3. HasFeaturesCol
  4. HasLabelCol
  5. Params
  6. Serializable
  7. Serializable
  8. Identifiable
  9. AnyRef
  10. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new Pool(data: DataFrame, pairsData: DataFrame)

    Construct Pool from DataFrame also specifying pairs data in an additional DataFrame

    Construct Pool from DataFrame also specifying pairs data in an additional DataFrame

    Example: 

    1. val spark = SparkSession.builder()
  .master("local[4]")
  .appName("PoolWithPairsTest")
  .getOrCreate();

val srcData = Seq(
  Row(Vectors.dense(0.1, 0.2, 0.11), "0.12", 0x0L, 0.12f, 0),
  Row(Vectors.dense(0.97, 0.82, 0.33), "0.22", 0x0L, 0.18f, 1),
  Row(Vectors.dense(0.13, 0.22, 0.23), "0.34", 0x1L, 1.0f, 2),
  Row(Vectors.dense(0.23, 0.01, 0.0), "0.0", 0x1L, 1.2f, 3)
)

val srcDataSchema = Seq(
  StructField("features", SQLDataTypes.VectorType),
  StructField("label", StringType),
  StructField("groupId", LongType),
  StructField("weight", FloatType)
  StructField("sampleId", LongType)
)

val df = spark.createDataFrame(spark.sparkContext.parallelize(srcData), StructType(srcDataSchema))

val srcPairsData = Seq(
  Row(0x0L, 0, 1),
  Row(0x1L, 3, 2)
)

val srcPairsDataSchema = Seq(
  StructField("groupId", LongType),
  StructField("winnerId", IntegerType),
  StructField("loserId", IntegerType)
)

val pairsDf = spark.createDataFrame(
  spark.sparkContext.parallelize(srcPairsData),
  StructType(srcPairsDataSchema)
)

val pool = new Pool(df, pairsDf)
  .setGroupIdCol("groupId")
  .setWeightCol("weight")
  .setSampleIdCol("sampleId")

pool.data.show()
pool.pairsData.show()
  2. new Pool(data: DataFrame)

    Construct Pool from DataFrame Call set*Col methods to specify non-default columns.

    Construct Pool from DataFrame Call set*Col methods to specify non-default columns. Only features and label columns with "features" and "label" names are assumed by default.

    Example: 

    1. val spark = SparkSession.builder()
  .master("local[4]")
  .appName("PoolTest")
  .getOrCreate();

val srcData = Seq(
  Row(Vectors.dense(0.1, 0.2, 0.11), "0.12", 0x0L, 0.12f),
  Row(Vectors.dense(0.97, 0.82, 0.33), "0.22", 0x0L, 0.18f),
  Row(Vectors.dense(0.13, 0.22, 0.23), "0.34", 0x1L, 1.0f)
)

val srcDataSchema = Seq(
  StructField("features", SQLDataTypes.VectorType),
  StructField("label", StringType),
  StructField("groupId", LongType),
  StructField("weight", FloatType)
)

val df = spark.createDataFrame(spark.sparkContext.parallelize(srcData), StructType(srcDataSchema))

val pool = new Pool(df)
  .setGroupIdCol("groupId")
  .setWeightCol("weight")

pool.data.show()
  3. new Pool(uid: String, data: DataFrame = null, featuresLayout: TFeaturesLayout = null, quantizedFeaturesInfo: QuantizedFeaturesInfoPtr = null, pairsData: DataFrame = null)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def $[T](param: Param[T]): T
    Attributes
    protected
    Definition Classes
    Params
  4. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. final val baselineCol: Param[String]
  7. def calcNanModesAndBorders(nanModeAndBordersBuilder: TNanModeAndBordersBuilder, quantizationParams: QuantizationParamsTrait): Unit
    Attributes
    protected
  8. final def clear(param: Param[_]): Pool.this.type
    Definition Classes
    Params
  9. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  10. def copy(extra: ParamMap): Pool
    Definition Classes
    Pool → Params
  11. def copyValues[T <: Params](to: T, extra: ParamMap): T
    Attributes
    protected
    Definition Classes
    Params
  12. def count: Long

    returns

    Number of objects in the dataset, similar to the same method of org.apache.spark.sql.Dataset

  13. def createQuantizationSchema(quantizationParams: QuantizationParamsTrait): QuantizedFeaturesInfoPtr
    Attributes
    protected
  14. def createQuantized(quantizedFeaturesInfo: QuantizedFeaturesInfoPtr): Pool
    Attributes
    protected
  15. val data: DataFrame
  16. final def defaultCopy[T <: Params](extra: ParamMap): T
    Attributes
    protected
    Definition Classes
    Params
  17. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  18. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  19. def explainParam(param: Param[_]): String
    Definition Classes
    Params
  20. def explainParams(): String
    Definition Classes
    Params
  21. final def extractParamMap(): ParamMap
    Definition Classes
    Params
  22. final def extractParamMap(extra: ParamMap): ParamMap
    Definition Classes
    Params
  23. final val featuresCol: Param[String]
    Definition Classes
    HasFeaturesCol
  24. var featuresLayout: TFeaturesLayout
    Attributes
    protected
  25. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  26. final def get[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  27. final def getBaselineCol: String
  28. def getBaselineCount: Int

    returns

    dimension of formula baseline, 0 if no baseline specified

  29. def getCatFeaturesUniqValueCounts: Array[Int]
  30. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  31. final def getDefault[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  32. def getEstimatedFeatureCount: Int
  33. def getFeatureCount: Int
  34. def getFeatureNames: Array[String]
  35. final def getFeaturesCol: String
    Definition Classes
    HasFeaturesCol
  36. def getFeaturesLayout: TFeaturesLayout
  37. final def getGroupIdCol: String
  38. final def getGroupWeightCol: String
  39. final def getLabelCol: String
    Definition Classes
    HasLabelCol
  40. final def getOrDefault[T](param: Param[T]): T
    Definition Classes
    Params
  41. def getParam(paramName: String): Param[Any]
    Definition Classes
    Params
  42. final def getSampleIdCol: String
  43. final def getSubgroupIdCol: String
  44. def getTargetType: ERawTargetType
  45. final def getTimestampCol: String
  46. final def getWeightCol: String
    Definition Classes
    HasWeightCol
  47. final val groupIdCol: Param[String]
  48. final val groupWeightCol: Param[String]
  49. final def hasDefault[T](param: Param[T]): Boolean
    Definition Classes
    Params
  50. def hasParam(paramName: String): Boolean
    Definition Classes
    Params
  51. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  52. final def isDefined(param: Param[_]): Boolean
    Definition Classes
    Params
  53. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  54. def isQuantized: Boolean
  55. final def isSet(param: Param[_]): Boolean
    Definition Classes
    Params
  56. final val labelCol: Param[String]
    Definition Classes
    HasLabelCol
  57. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  58. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  59. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  60. def pairsCount: Long

    returns

    Number of pairs in the dataset

  61. val pairsData: DataFrame
  62. lazy val params: Array[Param[_]]
    Definition Classes
    Params
  63. def quantize(quantizedFeaturesInfo: QuantizedFeaturesInfoPtr): Pool

    Create Pool with quantized features from Pool with raw features.

    Create Pool with quantized features from Pool with raw features. This variant of the method is useful if QuantizedFeaturesInfo with data for quantization (borders and nan modes) has already been computed. Used, for example, to quantize evaluation datasets after the training dataset has been quantized.

  64. def quantize(quantizationParams: QuantizationParamsTrait): Pool

    Create Pool with quantized features from Pool with raw features

    Create Pool with quantized features from Pool with raw features

    Example: 

    1. val spark = SparkSession.builder()
  .master("local[*]")
  .appName("QuantizationTest")
  .getOrCreate();

val srcData = Seq(
  Row(Vectors.dense(0.1, 0.2, 0.11), "0.12"),
  Row(Vectors.dense(0.97, 0.82, 0.33), "0.22"),
  Row(Vectors.dense(0.13, 0.22, 0.23), "0.34")
)

val srcDataSchema = Seq(
  StructField("features", SQLDataTypes.VectorType),
  StructField("label", StringType)
)

val df = spark.createDataFrame(spark.sparkContext.parallelize(srcData), StructType(srcDataSchema))

val pool = new Pool(df)

val quantizedPool = pool.quantize(new QuantizationParams)
val quantizedPoolWithTwoBinsPerFeature = pool.quantize(new QuantizationParams().setBorderCount(1))
quantizedPool.data.show()
quantizedPoolWithTwoBinsPerFeature.data.show()
  65. val quantizedFeaturesInfo: QuantizedFeaturesInfoPtr
  66. def repartition(partitionCount: Int, byGroupColumnsIfPresent: Boolean = true): Pool

    Repartion data to the specified number of partitions.

    Repartion data to the specified number of partitions. Useful to repartition data to create one partition per executor for training (where each executor gets its' own CatBoost worker with a part of the training data).

  67. final val sampleIdCol: Param[String]
  68. final def set(paramPair: ParamPair[_]): Pool.this.type
    Attributes
    protected
    Definition Classes
    Params
  69. final def set(param: String, value: Any): Pool.this.type
    Attributes
    protected
    Definition Classes
    Params
  70. final def set[T](param: Param[T], value: T): Pool.this.type
    Definition Classes
    Params
  71. final def setBaselineCol(value: String): Pool.this.type
  72. final def setDefault(paramPairs: ParamPair[_]*): Pool.this.type
    Attributes
    protected
    Definition Classes
    Params
  73. final def setDefault[T](param: Param[T], value: T): Pool.this.type
    Attributes
    protected
    Definition Classes
    Params
  74. def setFeaturesCol(value: String): Pool

  75. final def setGroupIdCol(value: String): Pool.this.type
  76. final def setGroupWeightCol(value: String): Pool.this.type
  77. def setLabelCol(value: String): Pool

  78. final def setSampleIdCol(value: String): Pool.this.type
  79. final def setSubgroupIdCol(value: String): Pool.this.type
  80. final def setTimestampCol(value: String): Pool.this.type
  81. def setWeightCol(value: String): Pool

  82. final val subgroupIdCol: Param[String]
  83. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  84. final val timestampCol: Param[String]
  85. def toString(): String
    Definition Classes
    Identifiable → AnyRef → Any
  86. val uid: String
    Definition Classes
    Pool → Identifiable
  87. def updateCatFeaturesInfo(quantizedFeaturesInfo: QuantizedFeaturesInfoPtr): Unit
    Attributes
    protected
  88. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  89. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  90. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  91. final val weightCol: Param[String]
    Definition Classes
    HasWeightCol

Inherited from HasWeightCol

Inherited from HasFeaturesCol

Inherited from HasLabelCol

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

setParam

Ungrouped