Package org.apache.beam.runners.spark

Interface SparkCommonPipelineOptions

All Superinterfaces:
org.apache.beam.sdk.options.ApplicationNameOptions, org.apache.beam.sdk.options.FileStagingOptions, org.apache.beam.sdk.transforms.display.HasDisplayData, org.apache.beam.sdk.options.PipelineOptions, org.apache.beam.sdk.options.StreamingOptions
All Known Subinterfaces:
SparkContextOptions, SparkPipelineOptions, SparkPortableStreamingPipelineOptions, SparkStructuredStreamingPipelineOptions, TestSparkPipelineOptions
public interface SparkCommonPipelineOptions extends org.apache.beam.sdk.options.PipelineOptions, org.apache.beam.sdk.options.StreamingOptions, org.apache.beam.sdk.options.ApplicationNameOptions, org.apache.beam.sdk.options.FileStagingOptions
Spark runner PipelineOptions handles Spark execution-related configurations, such as the master address, and other user-related knobs.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Interface
    Description
    static class 
    SparkCommonPipelineOptions.StorageLevelFactory
    Returns Spark's default storage level for the Dataset or RDD API based on the respective runner.
    static class 
    SparkCommonPipelineOptions.TmpCheckpointDirFactory
    Returns the default checkpoint directory of /tmp/${job.name}.

    Nested classes/interfaces inherited from interface org.apache.beam.sdk.options.PipelineOptions

    org.apache.beam.sdk.options.PipelineOptions.AtomicLongFactory, org.apache.beam.sdk.options.PipelineOptions.CheckEnabled, org.apache.beam.sdk.options.PipelineOptions.DirectRunner, org.apache.beam.sdk.options.PipelineOptions.JobNameFactory, org.apache.beam.sdk.options.PipelineOptions.UserAgentFactory

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final String
    DEFAULT_MASTER_URL
     

  • Method Summary

    Modifier and Type
    Method
    Description
    String
    getCheckpointDir()
     
    Boolean
    getEnableSparkMetricSinks()
     
    Boolean
    getPreferGroupByKeyToHandleHugeValues()
     
    String
    getSparkMaster()
     
    String
    getStorageLevel()
     
    static void
    prepareFilesToStage(SparkCommonPipelineOptions options)
    Classpath contains non jar files (eg.
    void
    setCheckpointDir(String checkpointDir)
     
    void
    setEnableSparkMetricSinks(Boolean enableSparkMetricSinks)
     
    void
    setPreferGroupByKeyToHandleHugeValues(Boolean preferGroupByKeyToHandleHugeValues)
     
    void
    setSparkMaster(String master)
     
    void
    setStorageLevel(String storageLevel)
     

    Methods inherited from interface org.apache.beam.sdk.options.ApplicationNameOptions

    getAppName, setAppName

    Methods inherited from interface org.apache.beam.sdk.options.FileStagingOptions

    getFilesToStage, setFilesToStage

    Methods inherited from interface org.apache.beam.sdk.transforms.display.HasDisplayData

    populateDisplayData

    Methods inherited from interface org.apache.beam.sdk.options.PipelineOptions

    as, getGbek, getJobName, getOptionsId, getRunner, getStableUniqueNames, getTempLocation, getUserAgent, outputRuntimeOptions, revision, setGbek, setJobName, setOptionsId, setRunner, setStableUniqueNames, setTempLocation, setUserAgent

    Methods inherited from interface org.apache.beam.sdk.options.StreamingOptions

    getUpdateCompatibilityVersion, isStreaming, setStreaming, setUpdateCompatibilityVersion

  • Field Details

  • Method Details

    • getSparkMaster

      @String("local[4]") String getSparkMaster()

    • setSparkMaster

      void setSparkMaster(String master)

    • getCheckpointDir

      @InstanceFactory(TmpCheckpointDirFactory.class) String getCheckpointDir()

    • setCheckpointDir

      void setCheckpointDir(String checkpointDir)

    • getStorageLevel

      @InstanceFactory(StorageLevelFactory.class) String getStorageLevel()

    • setStorageLevel

      void setStorageLevel(String storageLevel)

    • getEnableSparkMetricSinks

      @Boolean(true) Boolean getEnableSparkMetricSinks()

    • setEnableSparkMetricSinks

      void setEnableSparkMetricSinks(Boolean enableSparkMetricSinks)

    • getPreferGroupByKeyToHandleHugeValues

      @Boolean(false) Boolean getPreferGroupByKeyToHandleHugeValues()

    • setPreferGroupByKeyToHandleHugeValues

      void setPreferGroupByKeyToHandleHugeValues(Boolean preferGroupByKeyToHandleHugeValues)

    • prepareFilesToStage

      @Internal static void prepareFilesToStage(SparkCommonPipelineOptions options)
      Classpath contains non jar files (eg. directories with .class files or empty directories) will cause exception in running log. Though the SparkContext can handle this when running in local master, it's better not to include non-jars files in classpath.