Package org.apache.beam.runners.spark.translation.streaming

Class ParDoStateUpdateFn<KeyT,ValueT,InputT extends org.apache.beam.sdk.values.KV<KeyT,ValueT>,OutputT>

java.lang.Object

scala.runtime.AbstractFunction3<ByteArray,scala.Option<byte[]>,org.apache.spark.streaming.State<StateAndTimers>,java.util.List<scala.Tuple2<org.apache.beam.sdk.values.TupleTag<?>,byte[]>>>

org.apache.beam.runners.spark.translation.streaming.ParDoStateUpdateFn<KeyT,ValueT,InputT,OutputT>

Type Parameters:

KeyT - The type of the key in the input KV pairs

ValueT - The type of the value in the input KV pairs

InputT - The input type, must be a KV of KeyT and ValueT

OutputT - The output type produced by the DoFn

All Implemented Interfaces:

java.io.Serializable, scala.Function3<ByteArray,scala.Option<byte[]>,org.apache.spark.streaming.State<StateAndTimers>,java.util.List<scala.Tuple2<org.apache.beam.sdk.values.TupleTag<?>,byte[]>>>

public class ParDoStateUpdateFn<KeyT,ValueT,InputT extends org.apache.beam.sdk.values.KV<KeyT,ValueT>,OutputT>
extends scala.runtime.AbstractFunction3<ByteArray,scala.Option<byte[]>,org.apache.spark.streaming.State<StateAndTimers>,java.util.List<scala.Tuple2<org.apache.beam.sdk.values.TupleTag<?>,byte[]>>>
implements java.io.Serializable

A function to handle stateful processing in Apache Beam's SparkRunner. This class processes stateful DoFn operations by managing state updates in a Spark streaming context.

Current Implementation Status:

• State: Fully implemented and supported through SparkStateInternals
• Timers: Processing time timers are now supported through SparkTimerInternals.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	ParDoStateUpdateFn.SparkTimerInternalsIterator	An iterator implementation that processes timers from SparkTimerInternals.

Constructor Summary

Constructors

Constructor

Description

ParDoStateUpdateFn(MetricsContainerStepMapAccumulator metricsAccum, java.lang.String stepName, org.apache.beam.sdk.transforms.DoFn<InputT,OutputT> doFn, org.apache.beam.sdk.coders.Coder<KeyT> keyCoder, org.apache.beam.sdk.values.WindowedValues.FullWindowedValueCoder<ValueT> wvCoder, org.apache.beam.runners.core.construction.SerializablePipelineOptions options, org.apache.beam.sdk.values.TupleTag<?> mainOutputTag, java.util.List<org.apache.beam.sdk.values.TupleTag<?>> additionalOutputTags, org.apache.beam.sdk.coders.Coder<InputT> inputCoder, java.util.Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.coders.Coder<?>> outputCoders, java.util.Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.values.KV<org.apache.beam.sdk.values.WindowingStrategy<?,?>,SideInputBroadcast<?>>> sideInputs, org.apache.beam.sdk.values.WindowingStrategy<?,?> windowingStrategy, org.apache.beam.sdk.transforms.DoFnSchemaInformation doFnSchemaInformation, java.util.Map<java.lang.String,org.apache.beam.sdk.values.PCollectionView<?>> sideInputMapping, java.util.Map<java.lang.Integer,GlobalWatermarkHolder.SparkWatermarks> watermarks, java.util.List<java.lang.Integer> sourceIds, boolean useStreamingSideInput)

Method Summary

Modifier and Type	Method	Description
java.util.List<scala.Tuple2<org.apache.beam.sdk.values.TupleTag<?>,byte[]>>	apply(ByteArray serializedKey, scala.Option<byte[]> serializedValue, org.apache.spark.streaming.State<StateAndTimers> state)

Methods inherited from class scala.runtime.AbstractFunction3

curried, toString, tupled

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

ParDoStateUpdateFn

public ParDoStateUpdateFn(MetricsContainerStepMapAccumulator metricsAccum,
                          java.lang.String stepName,
                          org.apache.beam.sdk.transforms.DoFn<InputT,OutputT> doFn,
                          org.apache.beam.sdk.coders.Coder<KeyT> keyCoder,
                          org.apache.beam.sdk.values.WindowedValues.FullWindowedValueCoder<ValueT> wvCoder,
                          org.apache.beam.runners.core.construction.SerializablePipelineOptions options,
                          org.apache.beam.sdk.values.TupleTag<?> mainOutputTag,
                          java.util.List<org.apache.beam.sdk.values.TupleTag<?>> additionalOutputTags,
                          org.apache.beam.sdk.coders.Coder<InputT> inputCoder,
                          java.util.Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.coders.Coder<?>> outputCoders,
                          java.util.Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.values.KV<org.apache.beam.sdk.values.WindowingStrategy<?,?>,SideInputBroadcast<?>>> sideInputs,
                          org.apache.beam.sdk.values.WindowingStrategy<?,?> windowingStrategy,
                          org.apache.beam.sdk.transforms.DoFnSchemaInformation doFnSchemaInformation,
                          java.util.Map<java.lang.String,org.apache.beam.sdk.values.PCollectionView<?>> sideInputMapping,
                          java.util.Map<java.lang.Integer,GlobalWatermarkHolder.SparkWatermarks> watermarks,
                          java.util.List<java.lang.Integer> sourceIds,
                          boolean useStreamingSideInput)

Method Detail

apply

public java.util.List<scala.Tuple2<org.apache.beam.sdk.values.TupleTag<?>,byte[]>> apply(ByteArray serializedKey,
                                                                                               scala.Option<byte[]> serializedValue,
                                                                                               org.apache.spark.streaming.State<StateAndTimers> state)

Specified by:

apply in interface scala.Function3<KeyT,ValueT,InputT extends org.apache.beam.sdk.values.KV<KeyT,ValueT>,OutputT>