Package org.apache.beam.runners.spark.translation

Interface SparkPortablePipelineTranslator<T extends SparkTranslationContext>

All Known Implementing Classes:
SparkBatchPortablePipelineTranslator, SparkStreamingPortablePipelineTranslator
public interface SparkPortablePipelineTranslator<T extends SparkTranslationContext>
Interface for portable Spark translators. This allows for a uniform invocation pattern for pipeline translation between streaming and batch runners.

  • Method Summary

    Modifier and Type
    Method
    Description
    T
    createTranslationContext(org.apache.spark.api.java.JavaSparkContext jsc, SparkPipelineOptions options, org.apache.beam.runners.fnexecution.provisioning.JobInfo jobInfo)
     
    Set<String>
    knownUrns()
     
    void
    translate(org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline pipeline, T context)
    Translates the given pipeline.

  • Method Details

    • knownUrns

      Set<String> knownUrns()

    • translate

      void translate(org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline pipeline, T context)
      Translates the given pipeline.

    • createTranslationContext

      T createTranslationContext(org.apache.spark.api.java.JavaSparkContext jsc, SparkPipelineOptions options, org.apache.beam.runners.fnexecution.provisioning.JobInfo jobInfo)