org.apache.spark.sql.execution.python.streaming.benchmark
BenchmarkTransformWithStateInPySparkStateServer
object BenchmarkTransformWithStateInPySparkStateServer extends App
This spins up standalone TransformWithStateInPySparkStateServer with in-memory state implementations. This is useful for understanding the performance of state intercommunication, since the logic of state processing is really lightweight (compared to the actual implementation leveraging RocksDB).
The instruction to run this benchmark:
1. Build Spark with ./dev/make-distribution.sh
2. cd dist
3. java -classpath "./jars/*" org.apache.spark.sql.execution.python.streaming.benchmark.BenchmarkTransformWithStateInPySparkStateServer
To run this with Unix Domain Socket, set the environment variable PYSPARK_UDS_MODE=true
The app will show the port number of the server, which is needed to connect to the server.
- Alphabetic
- By Inheritance
- BenchmarkTransformWithStateInPySparkStateServer
- App
- DelayedInit
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def args: Array[String]
- Attributes
- protected
- Definition Classes
- App
- val arrowTransformWithStateInPySparkMaxRecordsPerBatch: Int
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- val errorOnDuplicatedFieldNames: Boolean
- final val executionStart: Long
- Definition Classes
- App
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- val groupingKeySchema: StructType
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- val isUnixDomainSock: Boolean
- val largeVarTypes: Boolean
- final def main(args: Array[String]): Unit
- Definition Classes
- App
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- var serverSocketChannel: ServerSocketChannel
- var sockPath: File
- val spark: SparkSession
- val sqlConf: RuntimeConfig
- val stateHandleImpl: InMemoryStatefulProcessorHandleImpl
- val stateServer: TransformWithStateInPySparkStateServer
- var stateServerSocketPort: Int
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- val timeZoneId: String
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def delayedInit(body: => Unit): Unit
- Definition Classes
- App → DelayedInit
- Annotations
- @deprecated
- Deprecated
(Since version 2.11.0) the delayedInit mechanism will disappear
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)