Package org.apache.beam.runners.spark.translation.streaming

Class ParDoStateUpdateFn<KeyT,ValueT,InputT extends org.apache.beam.sdk.values.KV<KeyT,ValueT>,OutputT>

java.lang.Object
scala.runtime.AbstractFunction3<ByteArray,scala.Option<byte[]>,org.apache.spark.streaming.State<StateAndTimers>,List<scala.Tuple2<org.apache.beam.sdk.values.TupleTag<?>,byte[]>>>
org.apache.beam.runners.spark.translation.streaming.ParDoStateUpdateFn<KeyT,ValueT,InputT,OutputT>
Type Parameters:
KeyT - The type of the key in the input KV pairs
ValueT - The type of the value in the input KV pairs
InputT - The input type, must be a KV of KeyT and ValueT
OutputT - The output type produced by the DoFn
All Implemented Interfaces:
Serializable, scala.Function3<ByteArray,scala.Option<byte[]>,org.apache.spark.streaming.State<StateAndTimers>,List<scala.Tuple2<org.apache.beam.sdk.values.TupleTag<?>,byte[]>>>
public class ParDoStateUpdateFn<KeyT,ValueT,InputT extends org.apache.beam.sdk.values.KV<KeyT,ValueT>,OutputT> extends scala.runtime.AbstractFunction3<ByteArray,scala.Option<byte[]>,org.apache.spark.streaming.State<StateAndTimers>,List<scala.Tuple2<org.apache.beam.sdk.values.TupleTag<?>,byte[]>>> implements Serializable
A function to handle stateful processing in Apache Beam's SparkRunner. This class processes stateful DoFn operations by managing state updates in a Spark streaming context.

Current Implementation Status:

See Also:

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static class 
    ParDoStateUpdateFn.SparkTimerInternalsIterator
    An iterator implementation that processes timers from SparkTimerInternals.

  • Constructor Summary

    Constructors
    Constructor
    Description
    ParDoStateUpdateFn(MetricsContainerStepMapAccumulator metricsAccum, String stepName, org.apache.beam.sdk.transforms.DoFn<InputT,OutputT> doFn, org.apache.beam.sdk.coders.Coder<KeyT> keyCoder, org.apache.beam.sdk.values.WindowedValues.FullWindowedValueCoder<ValueT> wvCoder, org.apache.beam.runners.core.construction.SerializablePipelineOptions options, org.apache.beam.sdk.values.TupleTag<?> mainOutputTag, List<org.apache.beam.sdk.values.TupleTag<?>> additionalOutputTags, org.apache.beam.sdk.coders.Coder<InputT> inputCoder, Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.coders.Coder<?>> outputCoders, Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.values.KV<org.apache.beam.sdk.values.WindowingStrategy<?,?>,SideInputBroadcast<?>>> sideInputs, org.apache.beam.sdk.values.WindowingStrategy<?,?> windowingStrategy, org.apache.beam.sdk.transforms.DoFnSchemaInformation doFnSchemaInformation, Map<String,org.apache.beam.sdk.values.PCollectionView<?>> sideInputMapping, Map<Integer,GlobalWatermarkHolder.SparkWatermarks> watermarks, List<Integer> sourceIds, boolean useStreamingSideInput)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    List<scala.Tuple2<org.apache.beam.sdk.values.TupleTag<?>,byte[]>>
    apply(ByteArray serializedKey, scala.Option<byte[]> serializedValue, org.apache.spark.streaming.State<StateAndTimers> state)
     

    Methods inherited from class scala.runtime.AbstractFunction3

    curried, toString, tupled

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

  • Constructor Details

    • ParDoStateUpdateFn

      public ParDoStateUpdateFn(MetricsContainerStepMapAccumulator metricsAccum, String stepName, org.apache.beam.sdk.transforms.DoFn<InputT,OutputT> doFn, org.apache.beam.sdk.coders.Coder<KeyT> keyCoder, org.apache.beam.sdk.values.WindowedValues.FullWindowedValueCoder<ValueT> wvCoder, org.apache.beam.runners.core.construction.SerializablePipelineOptions options, org.apache.beam.sdk.values.TupleTag<?> mainOutputTag, List<org.apache.beam.sdk.values.TupleTag<?>> additionalOutputTags, org.apache.beam.sdk.coders.Coder<InputT> inputCoder, Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.coders.Coder<?>> outputCoders, Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.values.KV<org.apache.beam.sdk.values.WindowingStrategy<?,?>,SideInputBroadcast<?>>> sideInputs, org.apache.beam.sdk.values.WindowingStrategy<?,?> windowingStrategy, org.apache.beam.sdk.transforms.DoFnSchemaInformation doFnSchemaInformation, Map<String,org.apache.beam.sdk.values.PCollectionView<?>> sideInputMapping, Map<Integer,GlobalWatermarkHolder.SparkWatermarks> watermarks, List<Integer> sourceIds, boolean useStreamingSideInput)

  • Method Details

    • apply

      public List<scala.Tuple2<org.apache.beam.sdk.values.TupleTag<?>,byte[]>> apply(ByteArray serializedKey, scala.Option<byte[]> serializedValue, org.apache.spark.streaming.State<StateAndTimers> state)
      Specified by:
      apply in interface scala.Function3<KeyT,ValueT,InputT extends org.apache.beam.sdk.values.KV<KeyT,ValueT>,OutputT>