Packages

•  package root

  Definition Classes

  root

•  package io

  Definition Classes

  root

•  package delta

  Definition Classes

  io

•  package sql

  Definition Classes

  delta

•  class DeltaSparkSessionExtension extends AbstractDeltaSparkSessionExtension

  An extension for Spark SQL to activate Delta SQL parser to support Delta SQL grammar.

  An extension for Spark SQL to activate Delta SQL parser to support Delta SQL grammar.

  Scala example to create a SparkSession with the Delta SQL parser:

  import org.apache.spark.sql.SparkSession
  
  val spark = SparkSession
     .builder()
     .appName("...")
     .master("...")
     .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
     .getOrCreate()

  Java example to create a SparkSession with the Delta SQL parser:

  import org.apache.spark.sql.SparkSession;
  
  SparkSession spark = SparkSession
               .builder()
               .appName("...")
               .master("...")
               .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
               .getOrCreate();

  Python example to create a SparkSession with the Delta SQL parser (PySpark doesn't pick up the SQL conf "spark.sql.extensions" in Apache Spark 2.4.x, hence we need to activate it manually in 2.4.x. However, because SparkSession has been created and everything has been materialized, we need to clone a new session to trigger the initialization. See SPARK-25003):

  from pyspark.sql import SparkSession
  
  spark = SparkSession \
      .builder \
      .appName("...") \
      .master("...") \
      .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
      .getOrCreate()
  if spark.sparkContext().version < "3.":
      spark.sparkContext()._jvm.io.delta.sql.DeltaSparkSessionExtension() \
          .apply(spark._jsparkSession.extensions())
      spark = SparkSession(spark.sparkContext(), spark._jsparkSession.cloneSession())

  Definition Classes

  sql

  Since

  0.4.0

• NoOpRule

io.delta.sql.DeltaSparkSessionExtension

NoOpRule