trait ScalaDataFrameSuiteBase extends AnyFunSuite with DataFrameSuiteBase
Base trait for testing Spark DataFrames in Scala.
- Alphabetic
- By Inheritance
- ScalaDataFrameSuiteBase
- DataFrameSuiteBase
- DataFrameSuiteBaseLike
- SharedSparkContext
- SparkContextProvider
- BeforeAndAfterAll
- SuiteMixin
- TestSuite
- TestSuiteLike
- AnyFunSuite
- AnyFunSuiteLike
- Documenting
- Alerting
- Notifying
- Informing
- TestRegistration
- TestSuite
- Suite
- Serializable
- Assertions
- TripleEquals
- TripleEqualsSupport
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Type Members
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- def !==[T](right: Spread[T]): TripleEqualsInvocationOnSpread[T]
- Definition Classes
- TripleEqualsSupport
- def !==(right: Null): TripleEqualsInvocation[Null]
- Definition Classes
- TripleEqualsSupport
- def !==[T](right: T): TripleEqualsInvocation[T]
- Definition Classes
- TripleEqualsSupport
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- def ===[T](right: Spread[T]): TripleEqualsInvocationOnSpread[T]
- Definition Classes
- TripleEqualsSupport
- def ===(right: Null): TripleEqualsInvocation[Null]
- Definition Classes
- TripleEqualsSupport
- def ===[T](right: T): TripleEqualsInvocation[T]
- Definition Classes
- TripleEqualsSupport
- def afterAll(): Unit
- Definition Classes
- DataFrameSuiteBase → SharedSparkContext → BeforeAndAfterAll
- def alert: Alerter
- Attributes
- protected
- Definition Classes
- AnyFunSuiteLike → Alerting
- def appID: String
- Definition Classes
- SparkContextProvider
- def approxEquals(r1: Row, r2: Row, tol: Double): Boolean
- Definition Classes
- DataFrameSuiteBaseLike
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def assert[U](message: String, expected: U, actual: U)(implicit CT: ClassTag[U]): Unit
- Definition Classes
- TestSuite → TestSuiteLike
- def assert[U](expected: U, actual: U)(implicit CT: ClassTag[U]): Unit
- Definition Classes
- TestSuite → TestSuiteLike
- macro def assert(condition: Boolean, clue: Any)(implicit prettifier: Prettifier, pos: Position): Assertion
- Definition Classes
- Assertions
- macro def assert(condition: Boolean)(implicit prettifier: Prettifier, pos: Position): Assertion
- Definition Classes
- Assertions
- macro def assertCompiles(code: String)(implicit pos: Position): Assertion
- Definition Classes
- Assertions
- def assertDataFrameApproximateEquals(expected: DataFrame, result: DataFrame, tol: Double): Unit
Compares if two DataFrames are equal, checks that the schemas are the same.
Compares if two DataFrames are equal, checks that the schemas are the same. When comparing inexact fields uses tol.
- tol
max acceptable tolerance, should be less than 1.
- Definition Classes
- DataFrameSuiteBaseLike
- def assertDataFrameDataEquals(expected: DataFrame, result: DataFrame): Unit
Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other.
Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other. The resulting DataFrame should be empty inferring the two DataFrames have the same elements. Does not compare the schema.
- Definition Classes
- DataFrameSuiteBaseLike
- def assertDataFrameEquals(expected: DataFrame, result: DataFrame): Unit
Compares if two DataFrames are equal, checks the schema and then if that matches checks if the rows are equal.
Compares if two DataFrames are equal, checks the schema and then if that matches checks if the rows are equal.
- Definition Classes
- DataFrameSuiteBaseLike
- def assertDataFrameNoOrderEquals(expected: DataFrame, result: DataFrame): Unit
Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other.
Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other. The resulting DataFrame should be empty inferring the two DataFrames have the same elements. Also verifies that the schema is identical.
- Definition Classes
- DataFrameSuiteBaseLike
- macro def assertDoesNotCompile(code: String)(implicit pos: Position): Assertion
- Definition Classes
- Assertions
- def assertEmpty[U](arr: Array[U])(implicit CT: ClassTag[U]): Unit
- Definition Classes
- TestSuite → TestSuiteLike
- def assertResult(expected: Any)(actual: Any)(implicit prettifier: Prettifier, pos: Position): Assertion
- Definition Classes
- Assertions
- def assertResult(expected: Any, clue: Any)(actual: Any)(implicit prettifier: Prettifier, pos: Position): Assertion
- Definition Classes
- Assertions
- def assertSchemasEqual(expected: StructType, result: StructType): Unit
Compare if two schemas are equal, ignoring autoGeneratedAlias magic
Compare if two schemas are equal, ignoring autoGeneratedAlias magic
- Definition Classes
- DataFrameSuiteBaseLike
- def assertSmallDataFrameDataEquals(expected: DataFrame, result: DataFrame): Unit
Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other.
Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other. Similar to the function assertDataFrameDataEquals but for small DataFrame that can be collected in memory for the comparison.
- Definition Classes
- DataFrameSuiteBaseLike
- def assertThrows[T <: AnyRef](f: => Any)(implicit classTag: ClassTag[T], pos: Position): Assertion
- Definition Classes
- Assertions
- def assertTrue(expected: Boolean): Unit
- Definition Classes
- TestSuite → TestSuiteLike
- macro def assertTypeError(code: String)(implicit pos: Position): Assertion
- Definition Classes
- Assertions
- macro def assume(condition: Boolean, clue: Any)(implicit prettifier: Prettifier, pos: Position): Assertion
- Definition Classes
- Assertions
- macro def assume(condition: Boolean)(implicit prettifier: Prettifier, pos: Position): Assertion
- Definition Classes
- Assertions
- def beforeAll(): Unit
- Definition Classes
- DataFrameSuiteBase → SharedSparkContext → BeforeAndAfterAll
- def builder(): Builder
Constructs a configuration for hive or iceberg, where the metastore is located in a temp directory.
Constructs a configuration for hive or iceberg, where the metastore is located in a temp directory.
- Definition Classes
- DataFrameSuiteBaseLike
- def cancel(cause: Throwable)(implicit pos: Position): Nothing
- Definition Classes
- Assertions
- def cancel(message: String, cause: Throwable)(implicit pos: Position): Nothing
- Definition Classes
- Assertions
- def cancel(message: String)(implicit pos: Position): Nothing
- Definition Classes
- Assertions
- def cancel()(implicit pos: Position): Nothing
- Definition Classes
- Assertions
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native() @HotSpotIntrinsicCandidate()
- def conf: SparkConf
- Definition Classes
- SparkContextProvider
- def convertEquivalenceToAToBConstraint[A, B](equivalenceOfB: Equivalence[B])(implicit ev: <:<[A, B]): CanEqual[A, B]
- Definition Classes
- TripleEquals → TripleEqualsSupport
- def convertEquivalenceToBToAConstraint[A, B](equivalenceOfA: Equivalence[A])(implicit ev: <:<[B, A]): CanEqual[A, B]
- Definition Classes
- TripleEquals → TripleEqualsSupport
- def convertToCheckingEqualizer[T](left: T): CheckingEqualizer[T]
- Definition Classes
- TripleEquals → TripleEqualsSupport
- implicit def convertToEqualizer[T](left: T): Equalizer[T]
- Definition Classes
- TripleEquals → TripleEqualsSupport
- def defaultEquality[A]: Equality[A]
- Definition Classes
- TripleEqualsSupport
- def enableHiveSupport: Boolean
- Attributes
- protected
- Definition Classes
- DataFrameSuiteBaseLike
- def enableIcebergSupport: Boolean
- Attributes
- protected
- Definition Classes
- DataFrameSuiteBaseLike
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def execute(testName: String, configMap: ConfigMap, color: Boolean, durations: Boolean, shortstacks: Boolean, fullstacks: Boolean, stats: Boolean): Unit
- Definition Classes
- Suite
- def expectedTestCount(filter: Filter): Int
- Definition Classes
- Suite
- def fail(message: String): Unit
- Definition Classes
- TestSuite → TestSuiteLike
- def fail(cause: Throwable)(implicit pos: Position): Nothing
- Definition Classes
- Assertions
- def fail(message: String, cause: Throwable)(implicit pos: Position): Nothing
- Definition Classes
- Assertions
- def fail(message: String)(implicit pos: Position): Nothing
- Definition Classes
- Assertions
- def fail()(implicit pos: Position): Nothing
- Definition Classes
- Assertions
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @HotSpotIntrinsicCandidate()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @HotSpotIntrinsicCandidate()
- val icebergWarehouse: String
- Definition Classes
- DataFrameSuiteBaseLike
- def ignore(testName: String, testTags: Tag*)(testFun: => Any)(implicit pos: Position): Unit
- Attributes
- protected
- Definition Classes
- AnyFunSuiteLike
- implicit def impSqlContext: SQLContext
- Attributes
- protected
- Definition Classes
- DataFrameSuiteBaseLike
- def info: Informer
- Attributes
- protected
- Definition Classes
- AnyFunSuiteLike → Informing
- def intercept[T <: AnyRef](f: => Any)(implicit classTag: ClassTag[T], pos: Position): T
- Definition Classes
- Assertions
- val invokeBeforeAllAndAfterAllEvenIfNoTestsAreExpected: Boolean
- Definition Classes
- BeforeAndAfterAll
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- lazy val localMetastorePath: String
- Definition Classes
- DataFrameSuiteBaseLike
- lazy val localWarehousePath: String
- Definition Classes
- DataFrameSuiteBaseLike
- def lowPriorityTypeCheckedConstraint[A, B](implicit equivalenceOfB: Equivalence[B], ev: <:<[A, B]): CanEqual[A, B]
- Definition Classes
- TripleEquals → TripleEqualsSupport
- def markup: Documenter
- Attributes
- protected
- Definition Classes
- AnyFunSuiteLike → Documenting
- val maxUnequalRowsToShow: Int
- Definition Classes
- DataFrameSuiteBaseLike
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def nestedSuites: IndexedSeq[Suite]
- Definition Classes
- Suite
- def note: Notifier
- Attributes
- protected
- Definition Classes
- AnyFunSuiteLike → Notifying
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @HotSpotIntrinsicCandidate()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @HotSpotIntrinsicCandidate()
- def pending: Assertion with PendingStatement
- Definition Classes
- Assertions
- def pendingUntilFixed(f: => Unit)(implicit pos: Position): Assertion with PendingStatement
- Definition Classes
- Assertions
- final def registerIgnoredTest(testText: String, testTags: Tag*)(testFun: => Any)(implicit pos: Position): Unit
- Definition Classes
- AnyFunSuiteLike → TestRegistration
- final def registerTest(testText: String, testTags: Tag*)(testFun: => Any)(implicit pos: Position): Unit
- Definition Classes
- AnyFunSuiteLike → TestRegistration
- def rerunner: Option[String]
- Definition Classes
- Suite
- implicit def reuseContextIfPossible: Boolean
- Attributes
- protected
- Definition Classes
- SharedSparkContext
- def run(testName: Option[String], args: Args): Status
- Definition Classes
- BeforeAndAfterAll → SuiteMixin
- def runNestedSuites(args: Args): Status
- Attributes
- protected
- Definition Classes
- Suite
- def runTest(testName: String, args: Args): Status
- Attributes
- protected
- Definition Classes
- AnyFunSuiteLike → TestSuite → Suite
- def runTests(testName: Option[String], args: Args): Status
- Attributes
- protected
- Definition Classes
- AnyFunSuiteLike → Suite
- def sc: SparkContext
- Definition Classes
- SharedSparkContext → SparkContextProvider
- def setup(sc: SparkContext): Unit
Setup work to be called when creating a new SparkContext.
Setup work to be called when creating a new SparkContext. Default implementation currently sets a checkpoint directory.
This _should_ be called by the context provider automatically.
- Definition Classes
- SparkContextProvider
- lazy val spark: SparkSession
- Definition Classes
- DataFrameSuiteBaseLike
- Annotations
- @transient()
- def sqlBeforeAllTestCases(): Unit
- Definition Classes
- DataFrameSuiteBaseLike
- lazy val sqlContext: SQLContext
- Definition Classes
- DataFrameSuiteBaseLike
- Annotations
- @transient()
- final val succeed: Assertion
- Definition Classes
- Assertions
- def suiteId: String
- Definition Classes
- Suite
- def suiteName: String
- Definition Classes
- Suite
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def tags: Map[String, Set[String]]
- Definition Classes
- AnyFunSuiteLike → Suite
- lazy val tempDir: File
- Definition Classes
- DataFrameSuiteBaseLike
- def test(testName: String, testTags: Tag*)(testFun: => Any)(implicit pos: Position): Unit
- Attributes
- protected
- Definition Classes
- AnyFunSuiteLike
- def testCodegenOnly(testName: String, testTags: Tag*)(testFun: => Any)(implicit pos: Position): Unit
- def testCombined(testName: String, testTags: Tag*)(testFun: => Any)(implicit pos: Position): Unit
- def testDataFor(testName: String, theConfigMap: ConfigMap): TestData
- Definition Classes
- AnyFunSuiteLike → Suite
- def testNames: Set[String]
- Definition Classes
- AnyFunSuiteLike → Suite
- def testNonCodegen(testName: String, testTags: Tag*)(testFun: => Any)(implicit pos: Position): Unit
- def testsFor(unit: Unit): Unit
- Attributes
- protected
- Definition Classes
- AnyFunSuiteLike
- def toString(): String
- Definition Classes
- AnyFunSuite → AnyFunSuiteLike → AnyRef → Any
- def typeCheckedConstraint[A, B](implicit equivalenceOfA: Equivalence[A], ev: <:<[B, A]): CanEqual[A, B]
- Definition Classes
- TripleEquals → TripleEqualsSupport
- implicit def unconstrainedEquality[A, B](implicit equalityOfA: Equality[A]): CanEqual[A, B]
- Definition Classes
- TripleEquals → TripleEqualsSupport
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- def withClue[T](clue: Any)(fun: => T): T
- Definition Classes
- Assertions
- def withFixture(test: NoArgTest): Outcome
- Attributes
- protected
- Definition Classes
- TestSuite
- def withSQLConf(pairs: (String, String)*)(f: => Unit): Unit
Sets all SQL configurations specified in
pairs, callsf, and then restores all SQL configurations.Sets all SQL configurations specified in
pairs, callsf, and then restores all SQL configurations. Taken from Spark SQLHelper.- Attributes
- protected
- Definition Classes
- DataFrameSuiteBase
Deprecated Value Members
- def conversionCheckedConstraint[A, B](implicit equivalenceOfA: Equivalence[A], cnv: (B) => A): CanEqual[A, B]
- Definition Classes
- TripleEquals → TripleEqualsSupport
- Annotations
- @deprecated
- Deprecated
(Since version 3.1.0) The conversionCheckedConstraint method has been deprecated and will be removed in a future version of ScalaTest. It is no longer needed now that the deprecation period of ConversionCheckedTripleEquals has expired. It will not be replaced.
- def convertEquivalenceToAToBConversionConstraint[A, B](equivalenceOfB: Equivalence[B])(implicit ev: (A) => B): CanEqual[A, B]
- Definition Classes
- TripleEquals → TripleEqualsSupport
- Annotations
- @deprecated
- Deprecated
(Since version 3.1.0) The convertEquivalenceToAToBConversionConstraint method has been deprecated and will be removed in a future version of ScalaTest. It is no longer needed now that the deprecation period of ConversionCheckedTripleEquals has expired. It will not be replaced.
- def convertEquivalenceToBToAConversionConstraint[A, B](equivalenceOfA: Equivalence[A])(implicit ev: (B) => A): CanEqual[A, B]
- Definition Classes
- TripleEquals → TripleEqualsSupport
- Annotations
- @deprecated
- Deprecated
(Since version 3.1.0) The convertEquivalenceToBToAConversionConstraint method has been deprecated and will be removed in a future version of ScalaTest. It is no longer needed now that the deprecation period of ConversionCheckedTripleEquals has expired. It will not be replaced.
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
- def lowPriorityConversionCheckedConstraint[A, B](implicit equivalenceOfB: Equivalence[B], cnv: (A) => B): CanEqual[A, B]
- Definition Classes
- TripleEquals → TripleEqualsSupport
- Annotations
- @deprecated
- Deprecated
(Since version 3.1.0) The lowPriorityConversionCheckedConstraint method has been deprecated and will be removed in a future version of ScalaTest. It is no longer needed now that the deprecation period of ConversionCheckedTripleEquals has expired. It will not be replaced.
- final val styleName: String
- Definition Classes
- AnyFunSuiteLike → Suite
- Annotations
- @deprecated
- Deprecated
(Since version 3.1.0) The styleName lifecycle method has been deprecated and will be removed in a future version of ScalaTest with no replacement.