Package org.apache.beam.runners.spark.structuredstreaming.translation

Interface PipelineTranslator.UnresolvedTranslation<InT,T>

Enclosing class:

PipelineTranslator

public static interface PipelineTranslator.UnresolvedTranslation<InT,T>

Unresolved translation, allowing to optimize the generated Spark DAG.

An unresolved translation can - in certain cases - be fused together with following transforms. Currently this is only the case for ParDos with linear linage.

Method Summary

Modifier and Type	Method	Description
<T2> PipelineTranslator.UnresolvedTranslation<InT,T2>	fuse(PipelineTranslator.UnresolvedTranslation<T,T2> next)
org.apache.beam.sdk.values.PCollection<InT>	getInput()
org.apache.spark.sql.Dataset<org.apache.beam.sdk.values.WindowedValue<T>>	resolve(Supplier<org.apache.beam.sdk.options.PipelineOptions> options, org.apache.spark.sql.Dataset<org.apache.beam.sdk.values.WindowedValue<InT>> input)

Method Details

getInput

org.apache.beam.sdk.values.PCollection<InT> getInput()

fuse

<T2>
PipelineTranslator.UnresolvedTranslation<InT,T2> fuse(PipelineTranslator.UnresolvedTranslation<T,T2> next)

resolve

org.apache.spark.sql.Dataset<org.apache.beam.sdk.values.WindowedValue<T>> resolve(Supplier<org.apache.beam.sdk.options.PipelineOptions> options,
 org.apache.spark.sql.Dataset<org.apache.beam.sdk.values.WindowedValue<InT>> input)