Package org.apache.beam.runners.spark.translation

Class SparkCombineFn<InputT,ValueT,AccumT,OutputT>

java.lang.Object
org.apache.beam.runners.spark.translation.SparkCombineFn<InputT,ValueT,AccumT,OutputT>
All Implemented Interfaces:
Serializable
public class SparkCombineFn<InputT,ValueT,AccumT,OutputT> extends Object implements Serializable
A CombineFnBase.GlobalCombineFn with a CombineWithContext.Context for the SparkRunner.
See Also:

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static interface 
    SparkCombineFn.WindowedAccumulator<InputT,ValueT,AccumT,ImplT extends SparkCombineFn.WindowedAccumulator<InputT,ValueT,AccumT,ImplT>>
    Accumulator of WindowedValues holding values for different windows.

  • Method Summary

    Modifier and Type
    Method
    Description
    Stream<org.apache.beam.sdk.values.WindowedValue<OutputT>>
    extractOutputStream(SparkCombineFn.WindowedAccumulator<?,?,AccumT,?> accumulator)
    Extracts the stream of accumulated values.
    static <InputT, AccumT, OutputT>
    SparkCombineFn<InputT,InputT,AccumT,OutputT>
    globally(org.apache.beam.sdk.transforms.CombineWithContext.CombineFnWithContext<InputT,AccumT,OutputT> combineFn, org.apache.beam.runners.core.construction.SerializablePipelineOptions options, Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.values.KV<org.apache.beam.sdk.values.WindowingStrategy<?,?>,SideInputBroadcast<?>>> sideInputs, org.apache.beam.sdk.values.WindowingStrategy<?,?> windowingStrategy)
     
    static <K, V, AccumT, OutputT>
    SparkCombineFn<org.apache.beam.sdk.values.KV<K,V>,V,AccumT,OutputT>
    keyed(org.apache.beam.sdk.transforms.CombineWithContext.CombineFnWithContext<V,AccumT,OutputT> combineFn, org.apache.beam.runners.core.construction.SerializablePipelineOptions options, Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.values.KV<org.apache.beam.sdk.values.WindowingStrategy<?,?>,SideInputBroadcast<?>>> sideInputs, org.apache.beam.sdk.values.WindowingStrategy<?,?> windowingStrategy)
     

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Method Details

    • keyed

      public static <K, V, AccumT, OutputT> SparkCombineFn<org.apache.beam.sdk.values.KV<K,V>,V,AccumT,OutputT> keyed(org.apache.beam.sdk.transforms.CombineWithContext.CombineFnWithContext<V,AccumT,OutputT> combineFn, org.apache.beam.runners.core.construction.SerializablePipelineOptions options, Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.values.KV<org.apache.beam.sdk.values.WindowingStrategy<?,?>,SideInputBroadcast<?>>> sideInputs, org.apache.beam.sdk.values.WindowingStrategy<?,?> windowingStrategy)

    • globally

      public static <InputT, AccumT, OutputT> SparkCombineFn<InputT,InputT,AccumT,OutputT> globally(org.apache.beam.sdk.transforms.CombineWithContext.CombineFnWithContext<InputT,AccumT,OutputT> combineFn, org.apache.beam.runners.core.construction.SerializablePipelineOptions options, Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.values.KV<org.apache.beam.sdk.values.WindowingStrategy<?,?>,SideInputBroadcast<?>>> sideInputs, org.apache.beam.sdk.values.WindowingStrategy<?,?> windowingStrategy)

    • extractOutputStream

      public Stream<org.apache.beam.sdk.values.WindowedValue<OutputT>> extractOutputStream(SparkCombineFn.WindowedAccumulator<?,?,AccumT,?> accumulator)
      Extracts the stream of accumulated values.