Package org.apache.beam.runners.spark.translation

Class GroupCombineFunctions

java.lang.Object
org.apache.beam.runners.spark.translation.GroupCombineFunctions
public class GroupCombineFunctions extends Object
A set of group/combine functions to apply to Spark RDDs.

  • Constructor Summary

    Constructors
    Constructor
    Description
    GroupCombineFunctions()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    static <InputT, OutputT, AccumT>
    SparkCombineFn.WindowedAccumulator<InputT,InputT,AccumT,?>
    combineGlobally(org.apache.spark.api.java.JavaRDD<org.apache.beam.sdk.values.WindowedValue<InputT>> rdd, SparkCombineFn<InputT,InputT,AccumT,OutputT> sparkCombineFn, org.apache.beam.sdk.coders.Coder<AccumT> aCoder, org.apache.beam.sdk.values.WindowingStrategy<?,?> windowingStrategy)
    Apply a composite Combine.Globally transformation.
    static <K, V, AccumT>
    org.apache.spark.api.java.JavaPairRDD<K,SparkCombineFn.WindowedAccumulator<org.apache.beam.sdk.values.KV<K,V>,V,AccumT,?>>
    combinePerKey(org.apache.spark.api.java.JavaRDD<org.apache.beam.sdk.values.WindowedValue<org.apache.beam.sdk.values.KV<K,V>>> rdd, SparkCombineFn<org.apache.beam.sdk.values.KV<K,V>,V,AccumT,?> sparkCombineFn, org.apache.beam.sdk.coders.Coder<K> keyCoder, org.apache.beam.sdk.coders.Coder<V> valueCoder, org.apache.beam.sdk.coders.Coder<AccumT> aCoder, org.apache.beam.sdk.values.WindowingStrategy<?,?> windowingStrategy)
    Apply a composite Combine.PerKey transformation.
    static <K, V> org.apache.spark.api.java.JavaRDD<org.apache.beam.sdk.values.KV<K,Iterable<org.apache.beam.sdk.values.WindowedValue<V>>>>
    groupByKeyOnly(org.apache.spark.api.java.JavaRDD<org.apache.beam.sdk.values.WindowedValue<org.apache.beam.sdk.values.KV<K,V>>> rdd, org.apache.beam.sdk.coders.Coder<K> keyCoder, org.apache.beam.sdk.values.WindowedValues.WindowedValueCoder<V> wvCoder, @Nullable org.apache.spark.Partitioner partitioner)
    An implementation of GroupByKeyViaGroupByKeyOnly.GroupByKeyOnly for the Spark runner.
    static <T> org.apache.spark.api.java.JavaRDD<org.apache.beam.sdk.values.WindowedValue<T>>
    reshuffle(org.apache.spark.api.java.JavaRDD<org.apache.beam.sdk.values.WindowedValue<T>> rdd, org.apache.beam.sdk.values.WindowedValues.WindowedValueCoder<T> wvCoder)
    An implementation of Reshuffle for the Spark runner.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • GroupCombineFunctions

      public GroupCombineFunctions()

  • Method Details

    • groupByKeyOnly

      public static <K, V> org.apache.spark.api.java.JavaRDD<org.apache.beam.sdk.values.KV<K,Iterable<org.apache.beam.sdk.values.WindowedValue<V>>>> groupByKeyOnly(org.apache.spark.api.java.JavaRDD<org.apache.beam.sdk.values.WindowedValue<org.apache.beam.sdk.values.KV<K,V>>> rdd, org.apache.beam.sdk.coders.Coder<K> keyCoder, org.apache.beam.sdk.values.WindowedValues.WindowedValueCoder<V> wvCoder, @Nullable org.apache.spark.Partitioner partitioner)
      An implementation of GroupByKeyViaGroupByKeyOnly.GroupByKeyOnly for the Spark runner.

    • combineGlobally

      public static <InputT, OutputT, AccumT> SparkCombineFn.WindowedAccumulator<InputT,InputT,AccumT,?> combineGlobally(org.apache.spark.api.java.JavaRDD<org.apache.beam.sdk.values.WindowedValue<InputT>> rdd, SparkCombineFn<InputT,InputT,AccumT,OutputT> sparkCombineFn, org.apache.beam.sdk.coders.Coder<AccumT> aCoder, org.apache.beam.sdk.values.WindowingStrategy<?,?> windowingStrategy)
      Apply a composite Combine.Globally transformation.

    • combinePerKey

      public static <K, V, AccumT> org.apache.spark.api.java.JavaPairRDD<K,SparkCombineFn.WindowedAccumulator<org.apache.beam.sdk.values.KV<K,V>,V,AccumT,?>> combinePerKey(org.apache.spark.api.java.JavaRDD<org.apache.beam.sdk.values.WindowedValue<org.apache.beam.sdk.values.KV<K,V>>> rdd, SparkCombineFn<org.apache.beam.sdk.values.KV<K,V>,V,AccumT,?> sparkCombineFn, org.apache.beam.sdk.coders.Coder<K> keyCoder, org.apache.beam.sdk.coders.Coder<V> valueCoder, org.apache.beam.sdk.coders.Coder<AccumT> aCoder, org.apache.beam.sdk.values.WindowingStrategy<?,?> windowingStrategy)
      Apply a composite Combine.PerKey transformation.

      This aggregation will apply Beam's Combine.CombineFn via Spark's JavaPairRDD.combineByKey(Function, Function2, Function2) aggregation. For streaming, this will be called from within a serialized context (DStream's transform callback), so passed arguments need to be Serializable.

    • reshuffle

      public static <T> org.apache.spark.api.java.JavaRDD<org.apache.beam.sdk.values.WindowedValue<T>> reshuffle(org.apache.spark.api.java.JavaRDD<org.apache.beam.sdk.values.WindowedValue<T>> rdd, org.apache.beam.sdk.values.WindowedValues.WindowedValueCoder<T> wvCoder)
      An implementation of Reshuffle for the Spark runner.