Package org.apache.beam.runners.spark.structuredstreaming.translation

Interface PipelineTranslator.TranslationState

All Superinterfaces:

EncoderProvider

All Known Implementing Classes:

TransformTranslator.Context

Enclosing class:

PipelineTranslator

public static interface PipelineTranslator.TranslationState
extends EncoderProvider

Shared, mutable state during the translation of a pipeline and omitted afterwards.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderProvider

EncoderProvider.Factory<T>

Method Summary

Modifier and Type	Method	Description
<T> org.apache.spark.sql.Dataset<org.apache.beam.sdk.values.WindowedValue<T>>	getDataset(org.apache.beam.sdk.values.PCollection<T> pCollection)
org.apache.beam.sdk.options.PipelineOptions	getOptions()
Supplier<org.apache.beam.sdk.options.PipelineOptions>	getOptionsSupplier()
<T> org.apache.spark.broadcast.Broadcast<SideInputValues<T>>	getSideInputBroadcast(org.apache.beam.sdk.values.PCollection<T> pCollection, SideInputValues.Loader<T> loader)
org.apache.spark.sql.SparkSession	getSparkSession()
boolean	isLeaf(org.apache.beam.sdk.values.PCollection<?> pCollection)
default <T> void	putDataset(org.apache.beam.sdk.values.PCollection<T> pCollection, org.apache.spark.sql.Dataset<org.apache.beam.sdk.values.WindowedValue<T>> dataset)
<T> void	putDataset(org.apache.beam.sdk.values.PCollection<T> pCollection, org.apache.spark.sql.Dataset<org.apache.beam.sdk.values.WindowedValue<T>> dataset, boolean cache)
<InT, OutT> void	putUnresolved(org.apache.beam.sdk.values.PCollection<OutT> out, PipelineTranslator.UnresolvedTranslation<InT,OutT> unresolved)

Methods inherited from interface org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderProvider

encoderFactory, encoderOf, encoderOf, keyEncoderOf, kvEncoderOf, valueEncoderOf

Method Details

getDataset

<T>
org.apache.spark.sql.Dataset<org.apache.beam.sdk.values.WindowedValue<T>> getDataset(org.apache.beam.sdk.values.PCollection<T> pCollection)

isLeaf

boolean isLeaf(org.apache.beam.sdk.values.PCollection<?> pCollection)

putUnresolved

<InT,
OutT> void putUnresolved(org.apache.beam.sdk.values.PCollection<OutT> out,
 PipelineTranslator.UnresolvedTranslation<InT,OutT> unresolved)

putDataset

<T> void putDataset(org.apache.beam.sdk.values.PCollection<T> pCollection,
 org.apache.spark.sql.Dataset<org.apache.beam.sdk.values.WindowedValue<T>> dataset,
 boolean cache)

putDataset

default <T> void putDataset(org.apache.beam.sdk.values.PCollection<T> pCollection,
 org.apache.spark.sql.Dataset<org.apache.beam.sdk.values.WindowedValue<T>> dataset)

getSideInputBroadcast

<T>
org.apache.spark.broadcast.Broadcast<SideInputValues<T>> getSideInputBroadcast(org.apache.beam.sdk.values.PCollection<T> pCollection,
 SideInputValues.Loader<T> loader)

getOptionsSupplier

Supplier<org.apache.beam.sdk.options.PipelineOptions> getOptionsSupplier()

getOptions

org.apache.beam.sdk.options.PipelineOptions getOptions()

getSparkSession

org.apache.spark.sql.SparkSession getSparkSession()