Package com.lancedb.lance.spark.utils

Class SparkUtil

java.lang.Object
com.lancedb.lance.spark.utils.SparkUtil
public class SparkUtil extends Object

  • Constructor Summary

    Constructors
    Constructor
    Description
    SparkUtil()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    static org.apache.hadoop.conf.Configuration
    hadoopConfCatalogOverrides(org.apache.spark.sql.SparkSession spark, String catalogName)
    Pulls any Catalog specific overrides for the Hadoop conf from the current SparkSession, which can be set via `spark.sql.catalog.$catalogName.hadoop.*`

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • SparkUtil

      public SparkUtil()

  • Method Details

    • hadoopConfCatalogOverrides

      public static org.apache.hadoop.conf.Configuration hadoopConfCatalogOverrides(org.apache.spark.sql.SparkSession spark, String catalogName)
      Pulls any Catalog specific overrides for the Hadoop conf from the current SparkSession, which can be set via `spark.sql.catalog.$catalogName.hadoop.*`

      Mirrors the override of hadoop configurations for a given spark session using `spark.hadoop.*`.

      The SparkCatalog allows for hadoop configurations to be overridden per catalog, by setting them on the SQLConf, where the following will add the property "fs.default.name" with value "hdfs://hanksnamenode:8020" to the catalog's hadoop configuration. SparkSession.builder() .config(s"spark.sql.catalog.$catalogName.hadoop.fs.default.name", "hdfs://hanksnamenode:8020") .getOrCreate()

      Parameters:
      spark - The current Spark session
      catalogName - Name of the catalog to find overrides for.
      Returns:
      the Hadoop Configuration that should be used for this catalog, with catalog specific overrides applied.