Packages

class StreamingQueryListenerBus extends SparkListener with ListenerBus[StreamingQueryListener, Event]

A bus to forward events to StreamingQueryListeners. This one will send received StreamingQueryListener.Events to the Spark listener bus. It also registers itself with Spark listener bus, so that it can receive StreamingQueryListener.Events and dispatch them to StreamingQueryListeners.

Note 1: Each bus and its registered listeners are associated with a single SparkSession and StreamingQueryManager. So this bus will dispatch events to registered listeners for only those queries that were started in the associated SparkSession.

Note 2: To rebuild Structured Streaming UI in SHS, this bus will be registered into org.apache.spark.scheduler.ReplayListenerBus. We check sparkListenerBus defined or not to determine how to process StreamingQueryListener.Event. If false, it means this bus is used to replay all streaming query event from eventLog.

Linear Supertypes
ListenerBus[StreamingQueryListener, Event], Logging, SparkListener, SparkListenerInterface, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. StreamingQueryListenerBus
  2. ListenerBus
  3. Logging
  4. SparkListener
  5. SparkListenerInterface
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new StreamingQueryListenerBus(sparkListenerBus: Option[LiveListenerBus])

Type Members

  1. implicit class LogStringContext extends AnyRef
    Definition Classes
    Logging

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def addListener(listener: StreamingQueryListener): Unit
    Definition Classes
    ListenerBus
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  7. def doPostEvent(listener: StreamingQueryListener, event: Event): Unit

    Dispatch events to registered StreamingQueryListeners.

    Dispatch events to registered StreamingQueryListeners. Only the events associated queries started in the same SparkSession as this ListenerBus will be dispatched to the listeners.

    Attributes
    protected
    Definition Classes
    StreamingQueryListenerBus → ListenerBus
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  10. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  11. def getTimer(listener: StreamingQueryListener): Option[Timer]
    Attributes
    protected
    Definition Classes
    ListenerBus
  12. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  13. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  14. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  15. def isIgnorableException(e: Throwable): Boolean
    Attributes
    protected
    Definition Classes
    ListenerBus
  16. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  17. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  18. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  19. def logDebug(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  20. def logDebug(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  21. def logDebug(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  22. def logDebug(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  23. def logError(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  24. def logError(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  25. def logError(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  26. def logError(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  27. def logInfo(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  28. def logInfo(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  29. def logInfo(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  30. def logInfo(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  31. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  32. def logTrace(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  33. def logTrace(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  34. def logTrace(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  35. def logTrace(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  36. def logWarning(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  37. def logWarning(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  38. def logWarning(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  39. def logWarning(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  40. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  41. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  42. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  43. def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  44. def onApplicationStart(applicationStart: SparkListenerApplicationStart): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  45. def onBlockManagerAdded(blockManagerAdded: SparkListenerBlockManagerAdded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  46. def onBlockManagerRemoved(blockManagerRemoved: SparkListenerBlockManagerRemoved): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  47. def onBlockUpdated(blockUpdated: SparkListenerBlockUpdated): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  48. def onEnvironmentUpdate(environmentUpdate: SparkListenerEnvironmentUpdate): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  49. def onExecutorAdded(executorAdded: SparkListenerExecutorAdded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  50. def onExecutorBlacklisted(executorBlacklisted: SparkListenerExecutorBlacklisted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  51. def onExecutorBlacklistedForStage(executorBlacklistedForStage: SparkListenerExecutorBlacklistedForStage): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  52. def onExecutorExcluded(executorExcluded: SparkListenerExecutorExcluded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  53. def onExecutorExcludedForStage(executorExcludedForStage: SparkListenerExecutorExcludedForStage): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  54. def onExecutorMetricsUpdate(executorMetricsUpdate: SparkListenerExecutorMetricsUpdate): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  55. def onExecutorRemoved(executorRemoved: SparkListenerExecutorRemoved): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  56. def onExecutorUnblacklisted(executorUnblacklisted: SparkListenerExecutorUnblacklisted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  57. def onExecutorUnexcluded(executorUnexcluded: SparkListenerExecutorUnexcluded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  58. def onJobEnd(jobEnd: SparkListenerJobEnd): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  59. def onJobStart(jobStart: SparkListenerJobStart): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  60. def onNodeBlacklisted(nodeBlacklisted: SparkListenerNodeBlacklisted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  61. def onNodeBlacklistedForStage(nodeBlacklistedForStage: SparkListenerNodeBlacklistedForStage): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  62. def onNodeExcluded(nodeExcluded: SparkListenerNodeExcluded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  63. def onNodeExcludedForStage(nodeExcludedForStage: SparkListenerNodeExcludedForStage): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  64. def onNodeUnblacklisted(nodeUnblacklisted: SparkListenerNodeUnblacklisted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  65. def onNodeUnexcluded(nodeUnexcluded: SparkListenerNodeUnexcluded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  66. def onOtherEvent(event: SparkListenerEvent): Unit
    Definition Classes
    StreamingQueryListenerBus → SparkListener → SparkListenerInterface
  67. def onResourceProfileAdded(event: SparkListenerResourceProfileAdded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  68. def onSpeculativeTaskSubmitted(speculativeTask: SparkListenerSpeculativeTaskSubmitted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  69. def onStageCompleted(stageCompleted: SparkListenerStageCompleted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  70. def onStageExecutorMetrics(executorMetrics: SparkListenerStageExecutorMetrics): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  71. def onStageSubmitted(stageSubmitted: SparkListenerStageSubmitted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  72. def onTaskEnd(taskEnd: SparkListenerTaskEnd): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  73. def onTaskGettingResult(taskGettingResult: SparkListenerTaskGettingResult): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  74. def onTaskStart(taskStart: SparkListenerTaskStart): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  75. def onUnpersistRDD(unpersistRDD: SparkListenerUnpersistRDD): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  76. def onUnschedulableTaskSetAdded(unschedulableTaskSetAdded: SparkListenerUnschedulableTaskSetAdded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  77. def onUnschedulableTaskSetRemoved(unschedulableTaskSetRemoved: SparkListenerUnschedulableTaskSetRemoved): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  78. def post(event: Event): Unit

    Post a StreamingQueryListener event to the added StreamingQueryListeners.

    Post a StreamingQueryListener event to the added StreamingQueryListeners. Note that only the QueryStarted event is posted to the listener synchronously. Other events are dispatched to Spark listener bus. This method is guaranteed to be called by queries in the same SparkSession as this listener.

  79. def postToAll(event: Event): Unit

    Override the parent postToAll to remove the query id from activeQueryRunIds after all the listeners process QueryTerminatedEvent.

    Override the parent postToAll to remove the query id from activeQueryRunIds after all the listeners process QueryTerminatedEvent. (SPARK-19594)

    Definition Classes
    StreamingQueryListenerBus → ListenerBus
  80. final def removeAllListeners(): Unit
    Definition Classes
    ListenerBus
  81. final def removeListener(listener: StreamingQueryListener): Unit
    Definition Classes
    ListenerBus
  82. def removeListenerOnError(listener: StreamingQueryListener): Unit
    Definition Classes
    ListenerBus
  83. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  84. def toString(): String
    Definition Classes
    AnyRef → Any
  85. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  86. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  87. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  88. def withLogContext(context: Map[String, String])(body: => Unit): Unit
    Attributes
    protected
    Definition Classes
    Logging

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from ListenerBus[StreamingQueryListener, Event]

Inherited from Logging

Inherited from SparkListener

Inherited from SparkListenerInterface

Inherited from AnyRef

Inherited from Any

Ungrouped