Packages

package benchmark

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. Protected

Type Members

  1. class InMemoryStatefulProcessorHandleImpl extends StatefulProcessorHandleImplBase

    This is a benchmark purposed implementation of StatefulProcessorHandleImplBase that stores state in memory.

    This is a benchmark purposed implementation of StatefulProcessorHandleImplBase that stores state in memory. This leverages Scala collection types.

    NOTE: TTL is not supported in this implementation since it complicates the thing a lot and this is the benchmark purposed implementation.

Value Members

  1. object BenchmarkTransformWithStateInPySparkStateServer extends App

    This spins up standalone TransformWithStateInPySparkStateServer with in-memory state implementations.

    This spins up standalone TransformWithStateInPySparkStateServer with in-memory state implementations. This is useful for understanding the performance of state intercommunication, since the logic of state processing is really lightweight (compared to the actual implementation leveraging RocksDB).

    The instruction to run this benchmark: 1. Build Spark with ./dev/make-distribution.sh 2. cd dist 3. java -classpath "./jars/*" org.apache.spark.sql.execution.python.streaming.benchmark.BenchmarkTransformWithStateInPySparkStateServer To run this with Unix Domain Socket, set the environment variable PYSPARK_UDS_MODE=true

    The app will show the port number of the server, which is needed to connect to the server.

Ungrouped