Package org.apache.beam.runners.spark.translation

Class TranslationUtils

java.lang.Object
org.apache.beam.runners.spark.translation.TranslationUtils
public final class TranslationUtils extends Object
A set of utilities to help translating Beam transformations into Spark transformations.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static class 
    TranslationUtils.CombineGroupedValues<K,InputT,OutputT>
    A SparkCombineFn function applied to grouped KVs.
    static final class 
    TranslationUtils.TupleTagFilter<V>
    A utility class to filter TupleTags.

  • Method Summary

    Modifier and Type
    Method
    Description
    static void
    checkpointIfNeeded(org.apache.spark.streaming.dstream.DStream<?> dStream, org.apache.beam.runners.core.construction.SerializablePipelineOptions options)
    Checkpoints the given DStream if checkpointing is enabled in the pipeline options.
    static <T1, T2> org.apache.spark.streaming.api.java.JavaDStream<T2>
    dStreamValues(org.apache.spark.streaming.api.java.JavaPairDStream<T1,T2> pairDStream)
    Transform a pair stream into a value stream.
    static <T> org.apache.spark.api.java.function.VoidFunction<T>
    emptyVoidFunction()
     
    static <InputT, OutputT>
    org.apache.spark.api.java.function.FlatMapFunction<Iterator<InputT>,OutputT>
    functionToFlatMapFunction(org.apache.spark.api.java.function.Function<InputT,OutputT> func)
    A utility method that adapts Function to a FlatMapFunction with an Iterator input.
    static Long
    getBatchDuration(org.apache.beam.runners.core.construction.SerializablePipelineOptions options)
    Retrieves the batch duration in milliseconds from Spark pipeline options.
    static Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.values.KV<org.apache.beam.sdk.values.WindowingStrategy<?,?>,SideInputBroadcast<?>>>
    getSideInputs(Iterable<org.apache.beam.sdk.values.PCollectionView<?>> views, org.apache.spark.api.java.JavaSparkContext context, SparkPCollectionView pviews)
    Create SideInputs as Broadcast variables.
    static Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.values.WindowedValue<?>>>
    getTupleTagCoders(Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.values.PCollection<?>> outputs)
    Utility to get mapping between TupleTag and a coder.
    static org.apache.spark.api.java.function.PairFunction<scala.Tuple2<org.apache.beam.sdk.values.TupleTag<?>,ValueAndCoderLazySerializable<org.apache.beam.sdk.values.WindowedValue<?>>>,org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.values.WindowedValue<?>>
    getTupleTagDecodeFunction(Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.values.WindowedValue<?>>> coderMap)
    Returns a pair function to convert bytes to value via coder.
    static org.apache.spark.api.java.function.PairFunction<scala.Tuple2<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.values.WindowedValue<?>>,org.apache.beam.sdk.values.TupleTag<?>,ValueAndCoderLazySerializable<org.apache.beam.sdk.values.WindowedValue<?>>>
    getTupleTagEncodeFunction(Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.values.WindowedValue<?>>> coderMap)
    Returns a pair function to convert value to bytes via coder.
    static boolean
    hasEventTimers(org.apache.beam.sdk.transforms.DoFn<?,?> doFn)
    Checks if the given DoFn uses event time timers.
    static boolean
    hasTimers(org.apache.beam.sdk.transforms.DoFn<?,?> doFn)
    Checks if the given DoFn uses any timers.
    static <T, K, V> org.apache.spark.api.java.function.PairFlatMapFunction<Iterator<T>,K,V>
    pairFunctionToPairFlatMapFunction(org.apache.spark.api.java.function.PairFunction<T,K,V> pairFunction)
    A utility method that adapts PairFunction to a PairFlatMapFunction with an Iterator input.
    static void
    rejectStateAndTimers(org.apache.beam.sdk.transforms.DoFn<?,?> doFn)
    Reject state and timers DoFn.
    static <T, W extends org.apache.beam.sdk.transforms.windowing.BoundedWindow>
    boolean
    skipAssignWindows(org.apache.beam.sdk.transforms.windowing.Window.Assign<T> transform, EvaluationContext context)
    Checks if the window transformation should be applied or skipped.
    static <K, V> org.apache.spark.api.java.function.PairFunction<org.apache.beam.sdk.values.WindowedValue<org.apache.beam.sdk.values.KV<K,V>>,ByteArray,org.apache.beam.sdk.values.WindowedValue<org.apache.beam.sdk.values.KV<K,V>>>
    toPairByKeyInWindowedValue(org.apache.beam.sdk.coders.Coder<K> keyCoder)
    Extract key from a WindowedValue KV into a pair.
    static <K, V> org.apache.spark.api.java.function.PairFlatMapFunction<Iterator<org.apache.beam.sdk.values.KV<K,V>>,K,V>
    toPairFlatMapFunction()
    KV to pair flatmap function.
    static <K, V> org.apache.spark.api.java.function.PairFunction<org.apache.beam.sdk.values.KV<K,V>,K,V>
    toPairFunction()
    KV to pair function.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Method Details

    • skipAssignWindows

      public static <T, W extends org.apache.beam.sdk.transforms.windowing.BoundedWindow> boolean skipAssignWindows(org.apache.beam.sdk.transforms.windowing.Window.Assign<T> transform, EvaluationContext context)
      Checks if the window transformation should be applied or skipped.

      Avoid running assign windows if both source and destination are global window or if the user has not specified the WindowFn (meaning they are just messing with triggering or allowed lateness).

      Type Parameters:
      T - PCollection type.
      W - BoundedWindow type.
      Parameters:
      transform - The Window.Assign transformation.
      context - The EvaluationContext.
      Returns:
      if to apply the transformation.

    • dStreamValues

      public static <T1, T2> org.apache.spark.streaming.api.java.JavaDStream<T2> dStreamValues(org.apache.spark.streaming.api.java.JavaPairDStream<T1,T2> pairDStream)
      Transform a pair stream into a value stream.

    • toPairFunction

      public static <K, V> org.apache.spark.api.java.function.PairFunction<org.apache.beam.sdk.values.KV<K,V>,K,V> toPairFunction()
      KV to pair function.

    • toPairFlatMapFunction

      public static <K, V> org.apache.spark.api.java.function.PairFlatMapFunction<Iterator<org.apache.beam.sdk.values.KV<K,V>>,K,V> toPairFlatMapFunction()
      KV to pair flatmap function.

    • toPairByKeyInWindowedValue

      public static <K, V> org.apache.spark.api.java.function.PairFunction<org.apache.beam.sdk.values.WindowedValue<org.apache.beam.sdk.values.KV<K,V>>,ByteArray,org.apache.beam.sdk.values.WindowedValue<org.apache.beam.sdk.values.KV<K,V>>> toPairByKeyInWindowedValue(org.apache.beam.sdk.coders.Coder<K> keyCoder)
      Extract key from a WindowedValue KV into a pair.

    • getSideInputs

      public static Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.values.KV<org.apache.beam.sdk.values.WindowingStrategy<?,?>,SideInputBroadcast<?>>> getSideInputs(Iterable<org.apache.beam.sdk.values.PCollectionView<?>> views, org.apache.spark.api.java.JavaSparkContext context, SparkPCollectionView pviews)
      Create SideInputs as Broadcast variables.
      Parameters:
      views - The PCollectionViews.
      context - The JavaSparkContext.
      pviews - The SparkPCollectionView.
      Returns:
      a map of tagged SideInputBroadcasts and their WindowingStrategy.

    • getBatchDuration

      public static Long getBatchDuration(org.apache.beam.runners.core.construction.SerializablePipelineOptions options)
      Retrieves the batch duration in milliseconds from Spark pipeline options.
      Parameters:
      options - The serializable pipeline options containing Spark-specific settings
      Returns:
      The checkpoint duration in milliseconds as specified in SparkPipelineOptions

    • hasTimers

      public static boolean hasTimers(org.apache.beam.sdk.transforms.DoFn<?,?> doFn)
      Checks if the given DoFn uses any timers.
      Parameters:
      doFn - the DoFn to check for timer usage
      Returns:
      true if the DoFn uses timers, false otherwise

    • hasEventTimers

      public static boolean hasEventTimers(org.apache.beam.sdk.transforms.DoFn<?,?> doFn)
      Checks if the given DoFn uses event time timers.
      Parameters:
      doFn - the DoFn to check for event time timer usage
      Returns:
      true if the DoFn uses event time timers, false otherwise. Note: Returns false if the DoFn has no timers at all.

    • checkpointIfNeeded

      public static void checkpointIfNeeded(org.apache.spark.streaming.dstream.DStream<?> dStream, org.apache.beam.runners.core.construction.SerializablePipelineOptions options)
      Checkpoints the given DStream if checkpointing is enabled in the pipeline options.
      Parameters:
      dStream - The DStream to be checkpointed
      options - The SerializablePipelineOptions containing configuration settings including batch duration

    • rejectStateAndTimers

      public static void rejectStateAndTimers(org.apache.beam.sdk.transforms.DoFn<?,?> doFn)
      Reject state and timers DoFn.
      Parameters:
      doFn - the DoFn to possibly reject.

    • emptyVoidFunction

      public static <T> org.apache.spark.api.java.function.VoidFunction<T> emptyVoidFunction()

    • pairFunctionToPairFlatMapFunction

      public static <T, K, V> org.apache.spark.api.java.function.PairFlatMapFunction<Iterator<T>,K,V> pairFunctionToPairFlatMapFunction(org.apache.spark.api.java.function.PairFunction<T,K,V> pairFunction)
      A utility method that adapts PairFunction to a PairFlatMapFunction with an Iterator input. This is particularly useful because it allows to use functions written for mapToPair functions in flatmapToPair functions.
      Type Parameters:
      T - the input type.
      K - the output key type.
      V - the output value type.
      Parameters:
      pairFunction - the PairFunction to adapt.
      Returns:
      a PairFlatMapFunction that accepts an Iterator as an input and applies the PairFunction on every element.

    • functionToFlatMapFunction

      public static <InputT, OutputT> org.apache.spark.api.java.function.FlatMapFunction<Iterator<InputT>,OutputT> functionToFlatMapFunction(org.apache.spark.api.java.function.Function<InputT,OutputT> func)
      A utility method that adapts Function to a FlatMapFunction with an Iterator input. This is particularly useful because it allows to use functions written for map functions in flatmap functions.
      Type Parameters:
      InputT - the input type.
      OutputT - the output type.
      Parameters:
      func - the Function to adapt.
      Returns:
      a FlatMapFunction that accepts an Iterator as an input and applies the Function on every element.

    • getTupleTagCoders

      public static Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.values.WindowedValue<?>>> getTupleTagCoders(Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.values.PCollection<?>> outputs)
      Utility to get mapping between TupleTag and a coder.
      Parameters:
      outputs - - A map of tuple tags and pcollections
      Returns:
      mapping between TupleTag and a coder

    • getTupleTagEncodeFunction

      public static org.apache.spark.api.java.function.PairFunction<scala.Tuple2<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.values.WindowedValue<?>>,org.apache.beam.sdk.values.TupleTag<?>,ValueAndCoderLazySerializable<org.apache.beam.sdk.values.WindowedValue<?>>> getTupleTagEncodeFunction(Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.values.WindowedValue<?>>> coderMap)
      Returns a pair function to convert value to bytes via coder.
      Parameters:
      coderMap - - mapping between TupleTag and a coder
      Returns:
      a pair function to convert value to bytes via coder

    • getTupleTagDecodeFunction

      public static org.apache.spark.api.java.function.PairFunction<scala.Tuple2<org.apache.beam.sdk.values.TupleTag<?>,ValueAndCoderLazySerializable<org.apache.beam.sdk.values.WindowedValue<?>>>,org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.values.WindowedValue<?>> getTupleTagDecodeFunction(Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.values.WindowedValue<?>>> coderMap)
      Returns a pair function to convert bytes to value via coder.
      Parameters:
      coderMap - - mapping between TupleTag and a coder
      Returns:
      a pair function to convert bytes to value via coder