Package org.apache.beam.runners.spark.translation

Class MultiDoFnFunction<InputT,OutputT>

java.lang.Object
org.apache.beam.runners.spark.translation.MultiDoFnFunction<InputT,OutputT>
Type Parameters:
InputT - Input type for DoFunction.
OutputT - Output type for DoFunction.
All Implemented Interfaces:
Serializable, org.apache.spark.api.java.function.PairFlatMapFunction<Iterator<org.apache.beam.sdk.values.WindowedValue<InputT>>,org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.values.WindowedValue<?>>
public class MultiDoFnFunction<InputT,OutputT> extends Object implements org.apache.spark.api.java.function.PairFlatMapFunction<Iterator<org.apache.beam.sdk.values.WindowedValue<InputT>>,org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.values.WindowedValue<?>>
DoFunctions ignore outputs that are not the main output. MultiDoFunctions deal with additional outputs by enriching the underlying data with multiple TupleTags.
See Also:

  • Constructor Summary

    Constructors
    Constructor
    Description
    MultiDoFnFunction(MetricsContainerStepMapAccumulator metricsAccum, String stepName, org.apache.beam.sdk.transforms.DoFn<InputT,OutputT> doFn, org.apache.beam.runners.core.construction.SerializablePipelineOptions options, org.apache.beam.sdk.values.TupleTag<OutputT> mainOutputTag, List<org.apache.beam.sdk.values.TupleTag<?>> additionalOutputTags, org.apache.beam.sdk.coders.Coder<InputT> inputCoder, Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.coders.Coder<?>> outputCoders, Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.values.KV<org.apache.beam.sdk.values.WindowingStrategy<?,?>,SideInputBroadcast<?>>> sideInputs, org.apache.beam.sdk.values.WindowingStrategy<?,?> windowingStrategy, boolean stateful, org.apache.beam.sdk.transforms.DoFnSchemaInformation doFnSchemaInformation, Map<String,org.apache.beam.sdk.values.PCollectionView<?>> sideInputMapping, boolean useBoundedConcurrentOutput, boolean useStreamingSideInput)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    Iterator<scala.Tuple2<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.values.WindowedValue<?>>>
    call(Iterator<org.apache.beam.sdk.values.WindowedValue<InputT>> iter)
     

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • MultiDoFnFunction

      public MultiDoFnFunction(MetricsContainerStepMapAccumulator metricsAccum, String stepName, org.apache.beam.sdk.transforms.DoFn<InputT,OutputT> doFn, org.apache.beam.runners.core.construction.SerializablePipelineOptions options, org.apache.beam.sdk.values.TupleTag<OutputT> mainOutputTag, List<org.apache.beam.sdk.values.TupleTag<?>> additionalOutputTags, org.apache.beam.sdk.coders.Coder<InputT> inputCoder, Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.coders.Coder<?>> outputCoders, Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.values.KV<org.apache.beam.sdk.values.WindowingStrategy<?,?>,SideInputBroadcast<?>>> sideInputs, org.apache.beam.sdk.values.WindowingStrategy<?,?> windowingStrategy, boolean stateful, org.apache.beam.sdk.transforms.DoFnSchemaInformation doFnSchemaInformation, Map<String,org.apache.beam.sdk.values.PCollectionView<?>> sideInputMapping, boolean useBoundedConcurrentOutput, boolean useStreamingSideInput)
      Parameters:
      metricsAccum - The Spark AccumulatorV2 that backs the Beam metrics.
      doFn - The DoFn to be wrapped.
      options - The SerializablePipelineOptions.
      mainOutputTag - The main output TupleTag.
      additionalOutputTags - Additional output tags.
      inputCoder - The coder for the input.
      outputCoders - A map of all output coders.
      sideInputs - Side inputs used in this DoFn.
      windowingStrategy - Input WindowingStrategy.
      stateful - Stateful DoFn.
      useBoundedConcurrentOutput - If it should use bounded output for processing.

  • Method Details

    • call

      public Iterator<scala.Tuple2<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.values.WindowedValue<?>>> call(Iterator<org.apache.beam.sdk.values.WindowedValue<InputT>> iter) throws Exception
      Specified by:
      call in interface org.apache.spark.api.java.function.PairFlatMapFunction<Iterator<org.apache.beam.sdk.values.WindowedValue<InputT>>,org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.values.WindowedValue<?>>
      Throws:
      Exception