Trait

com.holdenkarau.spark.testing

ScalaDataFrameSuiteBase

Related Doc: package testing

Permalink

trait ScalaDataFrameSuiteBase extends AnyFunSuite with DataFrameSuiteBase

Base trait for testing Spark DataFrames in Scala.

Linear Supertypes
DataFrameSuiteBase, DataFrameSuiteBaseLike, SharedSparkContext, SparkContextProvider, BeforeAndAfterAll, SuiteMixin, TestSuite, TestSuiteLike, AnyFunSuite, AnyFunSuiteLike, Documenting, Alerting, Notifying, Informing, TestRegistration, org.scalatest.TestSuite, Suite, Serializable, Serializable, Assertions, TripleEquals, TripleEqualsSupport, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ScalaDataFrameSuiteBase
  2. DataFrameSuiteBase
  3. DataFrameSuiteBaseLike
  4. SharedSparkContext
  5. SparkContextProvider
  6. BeforeAndAfterAll
  7. SuiteMixin
  8. TestSuite
  9. TestSuiteLike
  10. AnyFunSuite
  11. AnyFunSuiteLike
  12. Documenting
  13. Alerting
  14. Notifying
  15. Informing
  16. TestRegistration
  17. TestSuite
  18. Suite
  19. Serializable
  20. Serializable
  21. Assertions
  22. TripleEquals
  23. TripleEqualsSupport
  24. AnyRef
  25. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. class CheckingEqualizer[L] extends AnyRef

    Permalink
    Definition Classes
    TripleEqualsSupport
  2. class Equalizer[L] extends AnyRef

    Permalink
    Definition Classes
    TripleEqualsSupport
  3. trait NoArgTest extends () ⇒ Outcome with TestData

    Permalink
    Attributes
    protected
    Definition Classes
    TestSuite

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. def !==[T](right: Spread[T]): TripleEqualsInvocationOnSpread[T]

    Permalink
    Definition Classes
    TripleEqualsSupport
  3. def !==(right: Null): TripleEqualsInvocation[Null]

    Permalink
    Definition Classes
    TripleEqualsSupport
  4. def !==[T](right: T): TripleEqualsInvocation[T]

    Permalink
    Definition Classes
    TripleEqualsSupport
  5. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  6. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  7. def ===[T](right: Spread[T]): TripleEqualsInvocationOnSpread[T]

    Permalink
    Definition Classes
    TripleEqualsSupport
  8. def ===(right: Null): TripleEqualsInvocation[Null]

    Permalink
    Definition Classes
    TripleEqualsSupport
  9. def ===[T](right: T): TripleEqualsInvocation[T]

    Permalink
    Definition Classes
    TripleEqualsSupport
  10. def afterAll(): Unit

    Permalink
    Definition Classes
    DataFrameSuiteBase → SharedSparkContext → BeforeAndAfterAll
  11. def alert: Alerter

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → Alerting
  12. def appID: String

    Permalink
    Definition Classes
    SparkContextProvider
  13. def appName: String

    Permalink
    Definition Classes
    SparkContextProvider
  14. def approxEquals(r1: Row, r2: Row, tol: Double, tolTimestamp: Duration): Boolean

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  15. def approxEquals(r1: Row, r2: Row, tolTimestamp: Duration): Boolean

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  16. def approxEquals(r1: Row, r2: Row, tol: Double): Boolean

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  17. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  18. def assert[U](message: String, expected: U, actual: U)(implicit CT: ClassTag[U]): Unit

    Permalink
    Definition Classes
    TestSuite → TestSuiteLike
  19. def assert[U](expected: U, actual: U)(implicit CT: ClassTag[U]): Unit

    Permalink
    Definition Classes
    TestSuite → TestSuiteLike
  20. macro def assert(condition: Boolean, clue: Any)(implicit prettifier: Prettifier, pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  21. macro def assert(condition: Boolean)(implicit prettifier: Prettifier, pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  22. macro def assertCompiles(code: String)(implicit pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  23. def assertDataFrameApproximateEquals(expected: DataFrame, result: DataFrame, tol: Double, tolTimestamp: Duration, customShow: (DataFrame) ⇒ Unit = _.show()): Unit

    Permalink

    Compares if two DataFrames are equal, checks that the schemas are the same.

    Compares if two DataFrames are equal, checks that the schemas are the same. When comparing inexact fields uses tol & tolTimestamp.

    tol

    max acceptable numeric tolerance, should be less than 1.

    tolTimestamp

    max acceptable timestamp tolerance.

    customShow

    unit function to customize the show method when dataframes are not equal. IE: df.show(false) or df.toJSON.show(false).

    Definition Classes
    DataFrameSuiteBaseLike
  24. def assertDataFrameDataEquals(expected: DataFrame, result: DataFrame): Unit

    Permalink

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other.

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other. The resulting DataFrame should be empty inferring the two DataFrames have the same elements. Does not compare the schema.

    Definition Classes
    DataFrameSuiteBaseLike
  25. def assertDataFrameEquals(expected: DataFrame, result: DataFrame, customShow: (DataFrame) ⇒ Unit = _.show()): Unit

    Permalink

    Compares if two DataFrames are equal, checks the schema and then if that matches checks if the rows are equal.

    Compares if two DataFrames are equal, checks the schema and then if that matches checks if the rows are equal.

    customShow

    unit function to customize the show method when dataframes are not equal. IE: df.show(false) or df.toJSON.show(false).

    Definition Classes
    DataFrameSuiteBaseLike
  26. def assertDataFrameNoOrderEquals(expected: DataFrame, result: DataFrame): Unit

    Permalink

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other.

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other. The resulting DataFrame should be empty inferring the two DataFrames have the same elements. Also verifies that the schema is identical.

    Definition Classes
    DataFrameSuiteBaseLike
  27. macro def assertDoesNotCompile(code: String)(implicit pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  28. def assertEmpty[U](arr: Array[U])(implicit CT: ClassTag[U]): Unit

    Permalink
    Definition Classes
    TestSuite → TestSuiteLike
  29. def assertResult(expected: Any)(actual: Any)(implicit prettifier: Prettifier, pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  30. def assertResult(expected: Any, clue: Any)(actual: Any)(implicit prettifier: Prettifier, pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  31. def assertSchemasEqual(expected: StructType, result: StructType): Unit

    Permalink

    Compare if two schemas are equal, ignoring autoGeneratedAlias magic

    Compare if two schemas are equal, ignoring autoGeneratedAlias magic

    Definition Classes
    DataFrameSuiteBaseLike
  32. def assertSmallDataFrameDataEquals(expected: DataFrame, result: DataFrame): Unit

    Permalink

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other.

    Compares if two DataFrames are equal without caring about order of rows, by finding elements in one DataFrame that is not in the other. Similar to the function assertDataFrameDataEquals but for small DataFrame that can be collected in memory for the comparison.

    Definition Classes
    DataFrameSuiteBaseLike
  33. def assertThrows[T <: AnyRef](f: ⇒ Any)(implicit classTag: ClassTag[T], pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  34. def assertTrue(expected: Boolean): Unit

    Permalink
    Definition Classes
    TestSuite → TestSuiteLike
  35. macro def assertTypeError(code: String)(implicit pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  36. macro def assume(condition: Boolean, clue: Any)(implicit prettifier: Prettifier, pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  37. macro def assume(condition: Boolean)(implicit prettifier: Prettifier, pos: Position): Assertion

    Permalink
    Definition Classes
    Assertions
  38. def beforeAll(): Unit

    Permalink
    Definition Classes
    DataFrameSuiteBase → SharedSparkContext → BeforeAndAfterAll
  39. def builder(): Builder

    Permalink

    Constructs a configuration for hive or iceberg, where the metastore is located in a temp directory.

    Constructs a configuration for hive or iceberg, where the metastore is located in a temp directory.

    Definition Classes
    DataFrameSuiteBaseLike
  40. def cancel(cause: Throwable)(implicit pos: Position): Nothing

    Permalink
    Definition Classes
    Assertions
  41. def cancel(message: String, cause: Throwable)(implicit pos: Position): Nothing

    Permalink
    Definition Classes
    Assertions
  42. def cancel(message: String)(implicit pos: Position): Nothing

    Permalink
    Definition Classes
    Assertions
  43. def cancel()(implicit pos: Position): Nothing

    Permalink
    Definition Classes
    Assertions
  44. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @throws( ... )
  45. def conf: SparkConf

    Permalink
    Definition Classes
    SparkContextProvider
  46. def convertEquivalenceToAToBConstraint[A, B](equivalenceOfB: Equivalence[B])(implicit ev: <:<[A, B]): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
  47. def convertEquivalenceToBToAConstraint[A, B](equivalenceOfA: Equivalence[A])(implicit ev: <:<[B, A]): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
  48. def convertToCheckingEqualizer[T](left: T): CheckingEqualizer[T]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
  49. implicit def convertToEqualizer[T](left: T): Equalizer[T]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
  50. def defaultEquality[A]: Equality[A]

    Permalink
    Definition Classes
    TripleEqualsSupport
  51. def enableHiveSupport: Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    DataFrameSuiteBaseLike
  52. def enableIcebergSupport: Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    DataFrameSuiteBaseLike
  53. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  54. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  55. final def execute(testName: String, configMap: ConfigMap, color: Boolean, durations: Boolean, shortstacks: Boolean, fullstacks: Boolean, stats: Boolean): Unit

    Permalink
    Definition Classes
    Suite
  56. def expectedTestCount(filter: Filter): Int

    Permalink
    Definition Classes
    Suite
  57. def fail(message: String): Unit

    Permalink
    Definition Classes
    TestSuite → TestSuiteLike
  58. def fail(cause: Throwable)(implicit pos: Position): Nothing

    Permalink
    Definition Classes
    Assertions
  59. def fail(message: String, cause: Throwable)(implicit pos: Position): Nothing

    Permalink
    Definition Classes
    Assertions
  60. def fail(message: String)(implicit pos: Position): Nothing

    Permalink
    Definition Classes
    Assertions
  61. def fail()(implicit pos: Position): Nothing

    Permalink
    Definition Classes
    Assertions
  62. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate()
  63. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate()
  64. val icebergWarehouse: String

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  65. def ignore(testName: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike
  66. implicit def impSqlContext: SQLContext

    Permalink
    Attributes
    protected
    Definition Classes
    DataFrameSuiteBaseLike
  67. def info: Informer

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → Informing
  68. def intercept[T <: AnyRef](f: ⇒ Any)(implicit classTag: ClassTag[T], pos: Position): T

    Permalink
    Definition Classes
    Assertions
  69. val invokeBeforeAllAndAfterAllEvenIfNoTestsAreExpected: Boolean

    Permalink
    Definition Classes
    BeforeAndAfterAll
  70. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  71. lazy val localMetastorePath: String

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  72. lazy val localWarehousePath: String

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  73. def lowPriorityTypeCheckedConstraint[A, B](implicit equivalenceOfB: Equivalence[B], ev: <:<[A, B]): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
  74. def markup: Documenter

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → Documenting
  75. val maxUnequalRowsToShow: Int

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  76. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  77. def nestedSuites: IndexedSeq[Suite]

    Permalink
    Definition Classes
    Suite
  78. def note: Notifier

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → Notifying
  79. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate()
  80. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate()
  81. def pending: Assertion with PendingStatement

    Permalink
    Definition Classes
    Assertions
  82. def pendingUntilFixed(f: ⇒ Unit)(implicit pos: Position): Assertion with PendingStatement

    Permalink
    Definition Classes
    Assertions
  83. final def registerIgnoredTest(testText: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit

    Permalink
    Definition Classes
    AnyFunSuiteLike → TestRegistration
  84. final def registerTest(testText: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit

    Permalink
    Definition Classes
    AnyFunSuiteLike → TestRegistration
  85. def rerunner: Option[String]

    Permalink
    Definition Classes
    Suite
  86. implicit def reuseContextIfPossible: Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    SharedSparkContext
  87. def run(testName: Option[String], args: Args): Status

    Permalink
    Definition Classes
    BeforeAndAfterAll → SuiteMixin
  88. def runNestedSuites(args: Args): Status

    Permalink
    Attributes
    protected
    Definition Classes
    Suite
  89. def runTest(testName: String, args: Args): Status

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → TestSuite → Suite
  90. def runTests(testName: Option[String], args: Args): Status

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike → Suite
  91. def sc: SparkContext

    Permalink
  92. def setup(sc: SparkContext): Unit

    Permalink

    Setup work to be called when creating a new SparkContext.

    Setup work to be called when creating a new SparkContext. Default implementation currently sets a checkpoint directory.

    This _should_ be called by the context provider automatically.

    Definition Classes
    SparkContextProvider
  93. lazy val spark: SparkSession

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  94. def sqlBeforeAllTestCases(): Unit

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  95. lazy val sqlContext: SQLContext

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  96. final val succeed: Assertion

    Permalink
    Definition Classes
    Assertions
  97. def suiteId: String

    Permalink
    Definition Classes
    Suite
  98. def suiteName: String

    Permalink
    Definition Classes
    Suite
  99. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  100. def tags: Map[String, Set[String]]

    Permalink
    Definition Classes
    AnyFunSuiteLike → Suite
  101. lazy val tempDir: File

    Permalink
    Definition Classes
    DataFrameSuiteBaseLike
  102. def test(testName: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike
  103. def testCodegenOnly(testName: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit

    Permalink
  104. def testCombined(testName: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit

    Permalink
  105. def testDataFor(testName: String, theConfigMap: ConfigMap): TestData

    Permalink
    Definition Classes
    AnyFunSuiteLike → Suite
  106. def testNames: Set[String]

    Permalink
    Definition Classes
    AnyFunSuiteLike → Suite
  107. def testNonCodegen(testName: String, testTags: Tag*)(testFun: ⇒ Any)(implicit pos: Position): Unit

    Permalink
  108. def testsFor(unit: Unit): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    AnyFunSuiteLike
  109. def toString(): String

    Permalink
    Definition Classes
    AnyFunSuite → AnyFunSuiteLike → AnyRef → Any
  110. def typeCheckedConstraint[A, B](implicit equivalenceOfA: Equivalence[A], ev: <:<[B, A]): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
  111. implicit def unconstrainedEquality[A, B](implicit equalityOfA: Equality[A]): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
  112. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  113. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  114. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  115. def withClue[T](clue: Any)(fun: ⇒ T): T

    Permalink
    Definition Classes
    Assertions
  116. def withFixture(test: NoArgTest): Outcome

    Permalink
    Attributes
    protected
    Definition Classes
    TestSuite
  117. def withSQLConf(pairs: (String, String)*)(f: ⇒ Unit): Unit

    Permalink

    Sets all SQL configurations specified in pairs, calls f, and then restores all SQL configurations.

    Sets all SQL configurations specified in pairs, calls f, and then restores all SQL configurations. Taken from Spark SQLHelper.

    Attributes
    protected
    Definition Classes
    DataFrameSuiteBase

Deprecated Value Members

  1. def assertDataFrameApproximateEquals(expected: DataFrame, result: DataFrame, tol: Double): Unit

    Permalink

    Compares if two DataFrames are equal, checks that the schemas are the same.

    Compares if two DataFrames are equal, checks that the schemas are the same. When comparing inexact fields uses tol.

    tol

    max acceptable tolerance for numeric (between(0, 1)) & timestamp (millis).

    Definition Classes
    DataFrameSuiteBaseLike
    Annotations
    @deprecated
    Deprecated

    (Since version 1.5.0) Use assertDataFrameApproximateEquals with timestamp tolerance

  2. def conversionCheckedConstraint[A, B](implicit equivalenceOfA: Equivalence[A], cnv: (B) ⇒ A): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
    Annotations
    @deprecated
    Deprecated

    (Since version 3.1.0) The conversionCheckedConstraint method has been deprecated and will be removed in a future version of ScalaTest. It is no longer needed now that the deprecation period of ConversionCheckedTripleEquals has expired. It will not be replaced.

  3. def convertEquivalenceToAToBConversionConstraint[A, B](equivalenceOfB: Equivalence[B])(implicit ev: (A) ⇒ B): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
    Annotations
    @deprecated
    Deprecated

    (Since version 3.1.0) The convertEquivalenceToAToBConversionConstraint method has been deprecated and will be removed in a future version of ScalaTest. It is no longer needed now that the deprecation period of ConversionCheckedTripleEquals has expired. It will not be replaced.

  4. def convertEquivalenceToBToAConversionConstraint[A, B](equivalenceOfA: Equivalence[A])(implicit ev: (B) ⇒ A): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
    Annotations
    @deprecated
    Deprecated

    (Since version 3.1.0) The convertEquivalenceToBToAConversionConstraint method has been deprecated and will be removed in a future version of ScalaTest. It is no longer needed now that the deprecation period of ConversionCheckedTripleEquals has expired. It will not be replaced.

  5. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @Deprecated @deprecated @throws( classOf[java.lang.Throwable] )
    Deprecated

    (Since version ) see corresponding Javadoc for more information.

  6. def lowPriorityConversionCheckedConstraint[A, B](implicit equivalenceOfB: Equivalence[B], cnv: (A) ⇒ B): CanEqual[A, B]

    Permalink
    Definition Classes
    TripleEquals → TripleEqualsSupport
    Annotations
    @deprecated
    Deprecated

    (Since version 3.1.0) The lowPriorityConversionCheckedConstraint method has been deprecated and will be removed in a future version of ScalaTest. It is no longer needed now that the deprecation period of ConversionCheckedTripleEquals has expired. It will not be replaced.

  7. final val styleName: String

    Permalink
    Definition Classes
    AnyFunSuiteLike → Suite
    Annotations
    @deprecated
    Deprecated

    (Since version 3.1.0) The styleName lifecycle method has been deprecated and will be removed in a future version of ScalaTest with no replacement.

Inherited from DataFrameSuiteBase

Inherited from DataFrameSuiteBaseLike

Inherited from SharedSparkContext

Inherited from SparkContextProvider

Inherited from BeforeAndAfterAll

Inherited from SuiteMixin

Inherited from TestSuite

Inherited from TestSuiteLike

Inherited from AnyFunSuite

Inherited from AnyFunSuiteLike

Inherited from Documenting

Inherited from Alerting

Inherited from Notifying

Inherited from Informing

Inherited from TestRegistration

Inherited from org.scalatest.TestSuite

Inherited from Suite

Inherited from Serializable

Inherited from Serializable

Inherited from Assertions

Inherited from TripleEquals

Inherited from TripleEqualsSupport

Inherited from AnyRef

Inherited from Any

Ungrouped