Package org.apache.beam.runners.spark.translation

Class SingleEmitInputDStream<T>

  • Type Parameters:
    T - The type of elements in the RDD
    All Implemented Interfaces:
    java.io.Serializable, org.apache.spark.internal.Logging, scala.Serializable
    public class SingleEmitInputDStream<T>
extends org.apache.spark.streaming.dstream.ConstantInputDStream<T>
    A specialized ConstantInputDStream that emits its RDD exactly once. Alternative to QueueInputDStream when checkpointing is required.

    Features:

    • Supports checkpointing
    • Guarantees single emission of data
    • Returns empty RDD after first emission
    See Also:
    Serialized Form

    • Nested Class Summary

      • Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

        org.apache.spark.internal.Logging.SparkShellLoggingFilter

    • Constructor Summary

      Constructors 
      Constructor Description
      SingleEmitInputDStream​(org.apache.spark.streaming.StreamingContext ssc, org.apache.spark.rdd.RDD<T> rdd)  

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      scala.Option<org.apache.spark.rdd.RDD<T>> compute​(org.apache.spark.streaming.Time validTime)  

      • Methods inherited from class org.apache.spark.streaming.dstream.ConstantInputDStream

        start, stop

      • Methods inherited from class org.apache.spark.streaming.dstream.InputDStream

        baseScope, dependencies, id, isTimeValid, lastValidTime, lastValidTime_$eq, name, rateController, slideDuration

      • Methods inherited from class org.apache.spark.streaming.dstream.DStream

        cache, checkpoint, checkpointData, checkpointDuration, checkpointDuration_$eq, clearCheckpointData, clearMetadata, context, count, countByValue, countByValue$default$1, countByValue$default$2, countByValueAndWindow, countByValueAndWindow$default$3, countByValueAndWindow$default$4, countByWindow, createRDDWithLocalProperties, creationSite, filter, flatMap, foreachRDD, foreachRDD, generatedRDDs, generatedRDDs_$eq, generateJob, getOrCompute, glom, graph, graph_$eq, initialize, initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isInitialized, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, map, mapPartitions, mapPartitions$default$2, mustCheckpoint, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, parentRememberDuration, persist, persist, print, print, reduce, reduceByWindow, reduceByWindow, register, remember, rememberDuration, rememberDuration_$eq, repartition, restoreCheckpointData, saveAsObjectFiles, saveAsObjectFiles$default$2, saveAsTextFiles, saveAsTextFiles$default$2, setContext, setGraph, slice, slice, ssc, ssc_$eq, storageLevel, storageLevel_$eq, toPairDStreamFunctions, toPairDStreamFunctions$default$4, transform, transform, transformWith, transformWith, union, updateCheckpointData, validateAtStart, window, window, zeroTime, zeroTime_$eq

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

      • SingleEmitInputDStream

        public SingleEmitInputDStream​(org.apache.spark.streaming.StreamingContext ssc,
                              org.apache.spark.rdd.RDD<T> rdd)

    • Method Detail

      • compute

        public scala.Option<org.apache.spark.rdd.RDD<T>> compute​(org.apache.spark.streaming.Time validTime)
        Overrides:
        compute in class org.apache.spark.streaming.dstream.ConstantInputDStream<T>