Trait

com.holdenkarau.spark.testing

ScalaDataFrameSuiteBase

Related Doc: package testing

Permalink

trait ScalaDataFrameSuiteBase extends AnyFunSuite with DataFrameSuiteBase

Base trait for testing Spark DataFrames in Scala.

Linear Supertypes
DataFrameSuiteBase, DataFrameSuiteBaseLike, SharedSparkContext, SparkContextProvider, BeforeAndAfterAll, SuiteMixin, TestSuite, TestSuiteLike, AnyFunSuite, AnyFunSuiteLike, Documenting, Alerting, Notifying, Informing, TestRegistration, org.scalatest.TestSuite, Suite, Serializable, Serializable, Assertions, TripleEquals, TripleEqualsSupport, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ScalaDataFrameSuiteBase
  2. DataFrameSuiteBase
  3. DataFrameSuiteBaseLike
  4. SharedSparkContext
  5. SparkContextProvider
  6. BeforeAndAfterAll
  7. SuiteMixin
  8. TestSuite
  9. TestSuiteLike
  10. AnyFunSuite
  11. AnyFunSuiteLike
  12. Documenting
  13. Alerting
  14. Notifying
  15. Informing
  16. TestRegistration
  17. TestSuite
  18. Suite
  19. Serializable
  20. Serializable
  21. Assertions
  22. TripleEquals
  23. TripleEqualsSupport
  24. AnyRef
  25. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. class CheckingEqualizer[L] extends AnyRef

    Permalink
    Definition Classes
    TripleEqualsSupport
  2. class Equalizer[L] extends AnyRef

    Permalink
    Definition Classes
    TripleEqualsSupport
  3. trait NoArgTest extends () ⇒ Outcome with TestData

    Permalink
    Attributes
    protected
    Definition Classes
    TestSuite

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. def !==[T](right: Spread[T]): TripleEqualsInvocationOnSpread[T]

    Permalink
    Definition Classes
    TripleEqualsSupport
  3. def !==(right: Null): TripleEqualsInvocation[Null]

    Permalink
    Definition Classes
    TripleEqualsSupport
  4. def !==[T](right: T): TripleEqualsInvocation[T]

    Permalink
    Definition Classes
    TripleEqualsSupport
  5. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  6. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  7. def ===[T](right: Spread[T]): TripleEqualsInvocationOnSpread[T]

    Permalink
    Definition Classes
    TripleEqualsSupport
  8. def ===(right: Null): TripleEqualsInvocation[Null]

    Permalink
    Definition Classes
    TripleEqualsSupport
  9. def ===[T](right: T): TripleEqualsInvocation[T]

    Permalink
    Definition Classes
    TripleEqualsSupport
  10. def afterAll(): Unit

    Permalink
    Definition Classes
    DataFrameSuiteBase → SharedSparkContext → BeforeAndAfterAll
  11. def alert: Alerter

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → Alerting
  12. def appID: String

    Permalink
    Definition Classes
    SparkContextProvider
  13. def approxEquals(r1: Row, r2: Row, tol: Double): Boolean

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  14. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  15. def assert[U](message: String, expected: U, actual: U)(implicit CT: ClassTag[U]): Unit

    Permalink
    Definition Classes
    TestSuite → TestSuiteLike
  16. def assert[U](expected: U, actual: U)(implicit CT: ClassTag[U]): Unit

    Permalink
    Definition Classes
    TestSuite → TestSuiteLike
  17. macro def assert(condition: Boolean, clue: Any)(implicit prettifier: Prettifier, pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  18. macro def assert(condition: Boolean)(implicit prettifier: Prettifier, pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  19. macro def assertCompiles(code: String)(implicit pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  20. def assertDataFrameApproximateEquals(expected: DataFrame, result: DataFrame, tol: Double): Unit

    Permalink

    Compares if two DataFrames are equal, checks that the schemas are the same.

    Compares if two DataFrames are equal, checks that the schemas are the same. When comparing inexact fields uses tol.

    tol

    max acceptable tolerance, should be less than 1.

    Definition Classes
    DataFrameSuiteBaseLike
  21. def assertDataFrameDataEquals(expected: DataFrame, result: DataFrame): Unit

    Permalink

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other.

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other. The resulting DataFrame should be empty inferring the two DataFrames have the same elements. Does not compare the schema.

    Definition Classes
    DataFrameSuiteBaseLike
  22. def assertDataFrameEquals(expected: DataFrame, result: DataFrame): Unit

    Permalink

    Compares if two DataFrames are equal, checks the schema and then if that matches checks if the rows are equal.

    Compares if two DataFrames are equal, checks the schema and then if that matches checks if the rows are equal.

    Definition Classes
    DataFrameSuiteBaseLike
  23. def assertDataFrameNoOrderEquals(expected: DataFrame, result: DataFrame): Unit

    Permalink

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other.

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other. The resulting DataFrame should be empty inferring the two DataFrames have the same elements. Also verifies that the schema is identical.

    Definition Classes
    DataFrameSuiteBaseLike
  24. macro def assertDoesNotCompile(code: String)(implicit pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  25. def assertEmpty[U](arr: Array[U])(implicit CT: ClassTag[U]): Unit

    Permalink
    Definition Classes
    TestSuite → TestSuiteLike
  26. def assertResult(expected: Any)(actual: Any)(implicit prettifier: Prettifier, pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  27. def assertResult(expected: Any, clue: Any)(actual: Any)(implicit prettifier: Prettifier, pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  28. def assertSchemasEqual(expected: StructType, result: StructType): Unit

    Permalink

    Compare if two schemas are equal, ignoring autoGeneratedAlias magic

    Compare if two schemas are equal, ignoring autoGeneratedAlias magic

    Definition Classes
    DataFrameSuiteBaseLike
  29. def assertSmallDataFrameDataEquals(expected: DataFrame, result: DataFrame): Unit

    Permalink

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other.

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other. Similar to the function assertDataFrameDataEquals but for small DataFrame that can be collected in memory for the comparison.

    Definition Classes
    DataFrameSuiteBaseLike
  30. def assertThrows[T <: AnyRef](f: ⇒ Any)(implicit classTag: ClassTag[T], pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  31. def assertTrue(expected: Boolean): Unit

    Permalink
    Definition Classes
    TestSuite → TestSuiteLike
  32. macro def assertTypeError(code: String)(implicit pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  33. macro def assume(condition: Boolean, clue: Any)(implicit prettifier: Prettifier, pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  34. macro def assume(condition: Boolean)(implicit prettifier: Prettifier, pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  35. def beforeAll(): Unit

    Permalink
    Definition Classes
    DataFrameSuiteBase → SharedSparkContext → BeforeAndAfterAll
  36. def builder(): Builder

    Permalink

    Constructs a configuration for hive or iceberg, where the metastore is located in a temp directory.

    Constructs a configuration for hive or iceberg, where the metastore is located in a temp directory.

    Definition Classes
    DataFrameSuiteBaseLike
  37. def cancel(cause: Throwable)(implicit pos: Position): Nothing

    Permalink
    Definition Classes
    Assertions
  38. def cancel(message: String, cause: Throwable)(implicit pos: Position): Nothing

    Permalink
    Definition Classes
    Assertions
  39. def cancel(message: String)(implicit pos: Position): Nothing

    Permalink
    Definition Classes
    Assertions
  40. def cancel()(implicit pos: Position): Nothing

    Permalink
    Definition Classes
    Assertions
  41. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @HotSpotIntrinsicCandidate() @throws( ... )
  42. def conf: SparkConf

    Permalink
    Definition Classes
    SparkContextProvider
  43. def convertEquivalenceToAToBConstraint[A, B](equivalenceOfB: Equivalence[B])(implicit ev: <:<[A, B]): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
  44. def convertEquivalenceToBToAConstraint[A, B](equivalenceOfA: Equivalence[A])(implicit ev: <:<[B, A]): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
  45. def convertToCheckingEqualizer[T](left: T): CheckingEqualizer[T]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
  46. implicit def convertToEqualizer[T](left: T): Equalizer[T]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
  47. def defaultEquality[A]: Equality[A]

    Permalink
    Definition Classes
    TripleEqualsSupport
  48. def enableHiveSupport: Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    DataFrameSuiteBaseLike
  49. def enableIcebergSupport: Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    DataFrameSuiteBaseLike
  50. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  51. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  52. final def execute(testName: String, configMap: ConfigMap, color: Boolean, durations: Boolean, shortstacks: Boolean, fullstacks: Boolean, stats: Boolean): Unit

    Permalink
    Definition Classes
    Suite
  53. def expectedTestCount(filter: Filter): Int

    Permalink
    Definition Classes
    Suite
  54. def fail(message: String): Unit

    Permalink
    Definition Classes
    TestSuite → TestSuiteLike
  55. def fail(cause: Throwable)(implicit pos: Position): Nothing

    Permalink
    Definition Classes
    Assertions
  56. def fail(message: String, cause: Throwable)(implicit pos: Position): Nothing

    Permalink
    Definition Classes
    Assertions
  57. def fail(message: String)(implicit pos: Position): Nothing

    Permalink
    Definition Classes
    Assertions
  58. def fail()(implicit pos: Position): Nothing

    Permalink
    Definition Classes
    Assertions
  59. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
    Annotations
    @HotSpotIntrinsicCandidate()
  60. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
    Annotations
    @HotSpotIntrinsicCandidate()
  61. val icebergWarehouse: String

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  62. def ignore(testName: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike
  63. implicit def impSqlContext: SQLContext

    Permalink
    Attributes
    protected
    Definition Classes
    DataFrameSuiteBaseLike
  64. def info: Informer

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → Informing
  65. def intercept[T <: AnyRef](f: ⇒ Any)(implicit classTag: ClassTag[T], pos: Position): T

    Permalink
    Definition Classes
    Assertions
  66. val invokeBeforeAllAndAfterAllEvenIfNoTestsAreExpected: Boolean

    Permalink
    Definition Classes
    BeforeAndAfterAll
  67. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  68. lazy val localMetastorePath: String

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  69. lazy val localWarehousePath: String

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  70. def lowPriorityTypeCheckedConstraint[A, B](implicit equivalenceOfB: Equivalence[B], ev: <:<[A, B]): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
  71. def markup: Documenter

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → Documenting
  72. val maxUnequalRowsToShow: Int

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  73. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  74. def nestedSuites: IndexedSeq[Suite]

    Permalink
    Definition Classes
    Suite
  75. def note: Notifier

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → Notifying
  76. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @HotSpotIntrinsicCandidate()
  77. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @HotSpotIntrinsicCandidate()
  78. def pending: Assertion with PendingStatement

    Permalink
    Definition Classes
    Assertions
  79. def pendingUntilFixed(f: ⇒ Unit)(implicit pos: Position): Assertion with PendingStatement

    Permalink
    Definition Classes
    Assertions
  80. final def registerIgnoredTest(testText: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit

    Permalink
    Definition Classes
    AnyFunSuiteLike → TestRegistration
  81. final def registerTest(testText: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit

    Permalink
    Definition Classes
    AnyFunSuiteLike → TestRegistration
  82. def rerunner: Option[String]

    Permalink
    Definition Classes
    Suite
  83. implicit def reuseContextIfPossible: Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    SharedSparkContext
  84. def run(testName: Option[String], args: Args): Status

    Permalink
    Definition Classes
    BeforeAndAfterAll → SuiteMixin
  85. def runNestedSuites(args: Args): Status

    Permalink
    Attributes
    protected
    Definition Classes
    Suite
  86. def runTest(testName: String, args: Args): Status

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → TestSuite → Suite
  87. def runTests(testName: Option[String], args: Args): Status

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → Suite
  88. def sc: SparkContext

    Permalink
  89. def setup(sc: SparkContext): Unit

    Permalink

    Setup work to be called when creating a new SparkContext.

    Setup work to be called when creating a new SparkContext. Default implementation currently sets a checkpoint directory.

    This _should_ be called by the context provider automatically.

    Definition Classes
    SparkContextProvider
  90. lazy val spark: SparkSession

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  91. def sqlBeforeAllTestCases(): Unit

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  92. lazy val sqlContext: SQLContext

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  93. final val succeed: Assertion

    Permalink
    Definition Classes
    Assertions
  94. def suiteId: String

    Permalink
    Definition Classes
    Suite
  95. def suiteName: String

    Permalink
    Definition Classes
    Suite
  96. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  97. def tags: Map[String, Set[String]]

    Permalink
    Definition Classes
    AnyFunSuiteLike → Suite
  98. lazy val tempDir: File

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  99. def test(testName: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike
  100. def testCodegenOnly(testName: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit

    Permalink
  101. def testCombined(testName: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit

    Permalink
  102. def testDataFor(testName: String, theConfigMap: ConfigMap): TestData

    Permalink
    Definition Classes
    AnyFunSuiteLike → Suite
  103. def testNames: Set[String]

    Permalink
    Definition Classes
    AnyFunSuiteLike → Suite
  104. def testNonCodegen(testName: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit

    Permalink
  105. def testsFor(unit: Unit): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike
  106. def toString(): String

    Permalink
    Definition Classes
    AnyFunSuite → AnyFunSuiteLike → AnyRef → Any
  107. def typeCheckedConstraint[A, B](implicit equivalenceOfA: Equivalence[A], ev: <:<[B, A]): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
  108. implicit def unconstrainedEquality[A, B](implicit equalityOfA: Equality[A]): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
  109. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  110. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  111. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  112. def withClue[T](clue: Any)(fun: ⇒ T): T

    Permalink
    Definition Classes
    Assertions
  113. def withFixture(test: NoArgTest): Outcome

    Permalink
    Attributes
    protected
    Definition Classes
    TestSuite
  114. def withSQLConf(pairs: (String, String)*)(f: ⇒ Unit): Unit

    Permalink

    Sets all SQL configurations specified in pairs, calls f, and then restores all SQL configurations.

    Sets all SQL configurations specified in pairs, calls f, and then restores all SQL configurations. Taken from Spark SQLHelper.

    Attributes
    protected
    Definition Classes
    DataFrameSuiteBase

Deprecated Value Members

  1. def conversionCheckedConstraint[A, B](implicit equivalenceOfA: Equivalence[A], cnv: (B) ⇒ A): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
    Annotations
    @deprecated
    Deprecated

    (Since version 3.1.0) The conversionCheckedConstraint method has been deprecated and will be removed in a future version of ScalaTest. It is no longer needed now that the deprecation period of ConversionCheckedTripleEquals has expired. It will not be replaced.

  2. def convertEquivalenceToAToBConversionConstraint[A, B](equivalenceOfB: Equivalence[B])(implicit ev: (A) ⇒ B): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
    Annotations
    @deprecated
    Deprecated

    (Since version 3.1.0) The convertEquivalenceToAToBConversionConstraint method has been deprecated and will be removed in a future version of ScalaTest. It is no longer needed now that the deprecation period of ConversionCheckedTripleEquals has expired. It will not be replaced.

  3. def convertEquivalenceToBToAConversionConstraint[A, B](equivalenceOfA: Equivalence[A])(implicit ev: (B) ⇒ A): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
    Annotations
    @deprecated
    Deprecated

    (Since version 3.1.0) The convertEquivalenceToBToAConversionConstraint method has been deprecated and will be removed in a future version of ScalaTest. It is no longer needed now that the deprecation period of ConversionCheckedTripleEquals has expired. It will not be replaced.

  4. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @Deprecated @deprecated @throws( classOf[java.lang.Throwable] )
    Deprecated

    (Since version ) see corresponding Javadoc for more information.

  5. def lowPriorityConversionCheckedConstraint[A, B](implicit equivalenceOfB: Equivalence[B], cnv: (A) ⇒ B): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
    Annotations
    @deprecated
    Deprecated

    (Since version 3.1.0) The lowPriorityConversionCheckedConstraint method has been deprecated and will be removed in a future version of ScalaTest. It is no longer needed now that the deprecation period of ConversionCheckedTripleEquals has expired. It will not be replaced.

  6. final val styleName: String

    Permalink
    Definition Classes
    AnyFunSuiteLike → Suite
    Annotations
    @deprecated
    Deprecated

    (Since version 3.1.0) The styleName lifecycle method has been deprecated and will be removed in a future version of ScalaTest with no replacement.

Inherited from DataFrameSuiteBase

Inherited from DataFrameSuiteBaseLike

Inherited from SharedSparkContext

Inherited from SparkContextProvider

Inherited from BeforeAndAfterAll

Inherited from SuiteMixin

Inherited from TestSuite

Inherited from TestSuiteLike

Inherited from AnyFunSuite

Inherited from AnyFunSuiteLike

Inherited from Documenting

Inherited from Alerting

Inherited from Notifying

Inherited from Informing

Inherited from TestRegistration

Inherited from org.scalatest.TestSuite

Inherited from Suite

Inherited from Serializable

Inherited from Serializable

Inherited from Assertions

Inherited from TripleEquals

Inherited from TripleEqualsSupport

Inherited from AnyRef

Inherited from Any

Ungrouped