Packages

class Builder extends Logging

Linear Supertypes
Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Builder
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Type Members

  1. implicit class LogStringContext extends AnyRef
    Definition Classes
    Logging

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def MDC(key: LogKey, value: Any): MDC
    Attributes
    protected
    Definition Classes
    Logging
  5. def addCompiledArtifacts(uri: URI): Builder

    Upload the compiled artifacts to the Databricks cluster and synchronize as part of session initialization.

    Upload the compiled artifacts to the Databricks cluster and synchronize as part of session initialization.

    This is required when the program requires external dependencies to evaluate the Spark query. For any Spark query using a typed Dataset API such as filter(), map(), foreach(), etc., or if the query is using a UDF, the enclosing classes should be synchronized with the cluster.

    For example: the following code snippet includes the compiled artifact of class Foo. The uri in this example will either point to a JAR file or to a folder that contains the compiled class files.

    val uri = Foo.getClass.getProtectionDomain.getCodeSource.getLocation.toURI
val spark = DatabricksSession.builder()
    .addCompiledArtifacts(uri)
    ...
    uri

    Specify the class file, JAR or directory to upload and synchronize. When a directory is specified, all deep nested JARs and class files will be synchronized.

    returns

    instance of this Builder

  6. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  8. def clusterId(clusterId: String): Builder

    The Databricks cluster id to connect.

    The Databricks cluster id to connect. Can't be used with serverless(enabled=True) at the same time.

    clusterId

    cluster id

    returns

    instance of Builder with the cluster id configured

  9. def configureSessionAfterCreation(spark: SparkSession): Unit
    Attributes
    protected
  10. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  11. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  12. def fromSparkClientConf(conf: Configuration): SparkSession
    Attributes
    protected[connect]
  13. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  14. def getOrCreate(): SparkSession

    Get a spark session if one was already created with the given configuration.

    Get a spark session if one was already created with the given configuration. Otherwise, create a new one.

    Connection parameters for connecting to Databricks Connect is collected in the following order. When one of the configuration parameters is collected, the lookup for that parameter stops; collection for other parameters continues down the chain.

    • Parameters configured directly in code via host(), token(), etc.
    • Parameters configured in the Databricks SDK's DatabricksConfig specified via sdkConfig()
    • The SPARK_REMOTE environment variable. For details on the connection string, see https://github.com/apache/spark/blob/master/sql/connect/docs/client-connection-string.md
    • Use the "default" configuration profile from the Databricks config file. For details on Databricks configuration profiles, see https://docs.databricks.com/dev-tools/auth.html

    By default, validates the used configuration by retrieving the used Databricks Runtime version with the Databricks SDK. Logs a warning if the authentication fails or if there is an unsupported combination of Databricks Runtime & Databricks Connect versions. Change this behaviour with DatabricksSession.builder().validateSession().

    returns

    spark session

  15. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  16. def header(header: String, value: String): Builder

    Add a header to the Spark Connect GRPC requests.

    Add a header to the Spark Connect GRPC requests. This method is cumulative (can be called repeatedly to add more headers).

    header

    Name of the header to set

    value

    The value to set

    returns

    The same instance of this class with a header set.

  17. def host(host: String): Builder

    The URL of the Databricks workspace to connect.

    The URL of the Databricks workspace to connect.

    host

    Databricks workspace URL

    returns

    instance of Builder with the host configured

  18. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  19. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  20. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  21. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  22. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  23. def logBasedOnLevel(level: Level)(f: => MessageWithContext): Unit
    Attributes
    protected
    Definition Classes
    Logging
  24. def logDebug(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  25. def logDebug(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  26. def logDebug(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  27. def logDebug(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  28. def logError(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  29. def logError(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  30. def logError(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  31. def logError(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  32. def logInfo(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  33. def logInfo(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  34. def logInfo(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  35. def logInfo(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  36. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  37. def logNativeProto(dbrLogBytes: Array[Byte]): Unit
    Attributes
    protected
    Definition Classes
    Logging
  38. def logTrace(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  39. def logTrace(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  40. def logTrace(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  41. def logTrace(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  42. def logWarning(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  43. def logWarning(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  44. def logWarning(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  45. def logWarning(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  46. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  47. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  48. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  49. def remote(): Builder

    API returns the same instance of the builder unchanged.

    API returns the same instance of the builder unchanged. This API is here for backwards compatibility.

    returns

    instance of Builder

  50. def sdkConfig(config: DatabricksConfig): Builder

    Provide a custom SDK Config object to use for connection configuration.

    Provide a custom SDK Config object to use for connection configuration. Can be used to override the default SDK Config that will be used.

    config

    Databricks SDK Config

    returns

    instance of Builder with the host configured

  51. def serverless(enabled: Boolean = true, usagePolicyId: Option[String] = None): Builder

    [IMPORTANT]: This API and support for Scala in serverless is in early PRIVATE PREVIEW.

    [IMPORTANT]: This API and support for Scala in serverless is in early PRIVATE PREVIEW. Some operations may not work.

    Connect to the serverless endpoint of the workspace. Can't be used with clusterId at the same time.

    enabled

    Boolean flag that enables serverless mode.

    usagePolicyId

    Private Preview API. Not yet supported in Databricks notebooks and jobs. The usage policy for running serverless workloads on Databricks compute. If no usage policy is specified, the workload is run using the unrestricted usage policy or the workspace default usage policy, if the user is entitled to one of them. If the specified policy does not exist or if the user is not entitled to it, an error will be thrown.

    returns

    instance of Builder with the serverless mode configured

  52. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  53. def toString(): String
    Definition Classes
    AnyRef → Any
  54. def token(token: String): Builder

    The access token to use while for connecting.

    The access token to use while for connecting. The user associated with this token should have access to the workspace and cluster. Subsequent queries will be executed on behalf of this user.

    token

    Databricks access token

    returns

    instance of Builder with the access token configured

  55. def userAgent(userAgent: String): Builder

    A user agent string identifying the application using the Databricks Connect module.

    A user agent string identifying the application using the Databricks Connect module. Databricks Connect sends a set of standard information, such as, OS, Python version and the version of Databricks Connect included as a user agent to the service. The value provided here will be included along with the rest of the information. It is recommended to provide this value in the format "<product-name>/<product-version>" as described in https://datatracker.ietf.org/doc/html/rfc7231#section-5.5.3, but is not required.

    userAgent

    The user agent string identifying the application that is using this module.

    returns

    instance of Builder with the cluster id configured

  56. def validateSession(enabled: Boolean = true): Builder

    Setting this option will run validations and throw an error if any fail.

    Setting this option will run validations and throw an error if any fail. Validations are run only when connecting to a Databricks cluster.

    • the specified connection parameters are valid and can communicate with the cluster
    • the Databricks Runtime version of the cluster is greater than or equal to the Databricks Connect version. By default, these validations are run and a warning is logged. Unsetting this option will turn off these validations. Skipping validations for connections strings.
    enabled

    Boolean, optional

    returns

    instance of Builder with validateSession option configured

  57. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  58. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  59. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  60. def withLogContext(context: Map[String, String])(body: => Unit): Unit
    Attributes
    protected
    Definition Classes
    Logging

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped