Package org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions

Class SparkSideInputReader

java.lang.Object
org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions.SparkSideInputReader
All Implemented Interfaces:
Serializable, org.apache.beam.runners.core.SideInputReader
public class SparkSideInputReader extends Object implements org.apache.beam.runners.core.SideInputReader, Serializable
SideInputReader using broadcasted SideInputValues.
See Also:

  • Method Summary

    Modifier and Type
    Method
    Description
    <T> boolean
    contains(org.apache.beam.sdk.values.PCollectionView<T> view)
     
    static org.apache.beam.runners.core.SideInputReader
    create(Map<String,org.apache.spark.broadcast.Broadcast<SideInputValues<?>>> sideInputs)
    Creates a SideInputReader for Spark from a map of PCollectionView tag ids and the corresponding broadcasted SideInputValues.
    static org.apache.beam.runners.core.SideInputReader
    empty()
     
    <T> @Nullable T
    get(org.apache.beam.sdk.values.PCollectionView<T> view, org.apache.beam.sdk.transforms.windowing.BoundedWindow window)
     
    boolean
    isEmpty()
     
    static void
    validateMaterializations(Iterable<org.apache.beam.sdk.values.PCollectionView<?>> views)
     

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Method Details

    • empty

      public static org.apache.beam.runners.core.SideInputReader empty()

    • create

      public static org.apache.beam.runners.core.SideInputReader create(Map<String,org.apache.spark.broadcast.Broadcast<SideInputValues<?>>> sideInputs)
      Creates a SideInputReader for Spark from a map of PCollectionView tag ids and the corresponding broadcasted SideInputValues.

      Note, the materialization of respective PCollectionViews should be validated ahead of time before any costly creation and broadcast of SideInputValues.

    • validateMaterializations

      public static void validateMaterializations(Iterable<org.apache.beam.sdk.values.PCollectionView<?>> views)

    • get

      public <T> @Nullable T get(org.apache.beam.sdk.values.PCollectionView<T> view, org.apache.beam.sdk.transforms.windowing.BoundedWindow window)
      Specified by:
      get in interface org.apache.beam.runners.core.SideInputReader

    • contains

      public <T> boolean contains(org.apache.beam.sdk.values.PCollectionView<T> view)
      Specified by:
      contains in interface org.apache.beam.runners.core.SideInputReader

    • isEmpty

      public boolean isEmpty()
      Specified by:
      isEmpty in interface org.apache.beam.runners.core.SideInputReader