All Classes and Interfaces
Class
Description
Abstract base class for iterators that process Spark input data and produce corresponding output
values.
Holds an RDD or values for deferred conversion to an RDD if needed.
Serializable byte array.
SideInputReader that caches results for costly
Materializations.SideInputReader that caches materialized views.Checkpoint data to make it available in future pipeline runs.
Checkpoint dir tree.
A
Sink for Spark's
metric system reporting metrics (including Beam step metrics) to a CSV file.A
Sink for Spark's
metric system reporting metrics (including Beam step metrics) to Graphite.Serialization utility class.
Serialization utility class.
A function for converting a byte array pair to a key-value pair.
Print to console.
Write to console.
PTransform writing PCollection to the console.Create an input stream from Queue.
Spark streaming overrides for various view (side input) transforms.
Creates a primitive
PCollectionView.A
Sink for Spark's
metric system reporting metrics (including Beam step metrics) to a CSV file.Holder for Spark RDD/DStream.
DoFnRunner decorator which registers
MetricsContainerImpl.Passing null values to Spark's Java API may cause problems because of Guava preconditions.
Encoders utility class.Encoder / expression utils that are called from generated code.
The
EvaluationContext is the result of a pipeline translation and can be used to evaluate / run the pipeline.The EvaluationContext allows us to define pipeline instructions and translate between
PObject<T>s or PCollection<T>s and Ts or DStreams/RDDs of Ts.A store to hold the global watermarks for a micro-batch.
A
GlobalWatermarkHolder.SparkWatermarks holds the watermarks and batch time relevant to a micro-batch input
from a specific source.Advance the WMs onBatchCompleted event.
A
Sink for Spark's
metric system reporting metrics (including Beam step metrics) to Graphite.A FlatMap function that groups by windows in batch mode using
ReduceFnRunner.Traverses the pipeline to populate the candidates for group by key.
A set of group/combine functions to apply to Spark
RDDs.Functions for GroupByKey with Non-Merging windows translations to Spark.
For resilience,
Accumulators are required to be wrapped in a Singleton.AccumulatorV2 for Beam metrics captured in MetricsContainerStepMap.Spark Listener which checkpoints
MetricsContainerStepMap values for fault-tolerance.AccumulatorV2 implementation for MetricsContainerStepMap.A
Source that accommodates Spark's micro-batch oriented nature and wraps an UnboundedSource.DoFunctions ignore outputs that are not the main output.
A
StepContext for Spark Batch Runner execution.A function to handle stateful processing in Apache Beam's SparkRunner.
An iterator implementation that processes timers from
SparkTimerInternals.The pipeline translator translates a Beam
Pipeline into a Spark correspondence, that can
then be evaluated.Shared, mutable state during the translation of a pipeline and omitted afterwards.
Unresolved translation, allowing to optimize the generated Spark DAG.
PipelineTranslator for executing a Pipeline in Spark in batch mode.Simple
Function to bring the windowing information into the value from the implicit
background representation of the PCollection.Utilities for easier interoperability with the Spark Scala API.
Broadcast helper for side inputs.
Metadata class for side inputs in Spark runner.
Utility class for creating and managing side input readers in the Spark runner.
SideInputValues serves as a Kryo serializable container that contains a materialized view
of side inputs.General
SideInputValues for BoundedWindows in two possible
states.Specialized
SideInputValues for use with the GlobalWindow in two possible
states.Factory function for load
SideInputValues from a Dataset.A specialized
ConstantInputDStream that emits its RDD exactly once.Classes implementing Beam
Source RDDs.A
SourceRDD.Unbounded is the implementation of a micro-batch in a SourceDStream.An implementation of
Window.Assign for the Spark runner.Translates a bounded portable pipeline into a Spark job.
Predicate to determine whether a URN is a Spark native transform.
A Spark
Source that is tailored to expose a SparkBeamMetric, wrapping an
underlying MetricResults instance.A Spark
Source that is tailored to expose a SparkBeamMetric, wrapping an
underlying MetricResults instance.A
CombineFnBase.GlobalCombineFn with a CombineWithContext.Context for the SparkRunner.Accumulator of WindowedValues holding values for different windows.
Type of the accumulator.
Spark runner
PipelineOptions handles Spark execution-related configurations, such as the
master address, and other user-related knobs.Returns Spark's default storage level for the Dataset or RDD API based on the respective
runner.
Returns the default checkpoint directory of /tmp/${job.name}.
A custom
PipelineOptions to work with properties related to JavaSparkContext.Returns an empty list, to avoid handling null.
Singleton class that contains one
ExecutableStageContext.Factory per job.An implementation of
GroupByKeyViaGroupByKeyOnly.GroupAlsoByWindow logic for grouping by windows and controlling
trigger firings and pane accumulation.Processes Spark's input data iterators using Beam's
DoFnRunner.Creates a job invocation to manage the Spark runner's execution of a portable pipeline.
Driver program that starts a job server for the Spark runner.
Spark runner-specific Configuration for the jobServer.
Pipeline visitor for translating a Beam pipeline into equivalent Spark operations.
SparkPCollectionView is used to pass serialized views to lambdas.
Type of side input.
Spark runner
PipelineOptions handles Spark execution-related configurations, such as the
master address, batch-interval, and other user-related knobs.Represents a Spark pipeline execution result.
Runs a portable pipeline on Apache Spark.
Translator to support translation between Beam transformations and Spark transformations.
Interface for portable Spark translators.
Pipeline options specific to the Spark portable runner running a streaming job.
Holds current processing context for
SparkInputDataProcessor.The SparkRunner translate operations defined on a pipeline to a representation executable by
Spark, and then submitting the job to Spark to be executed.
Evaluator on the pipeline.
Pipeline runner which translates a Beam pipeline into equivalent Spark operations, without
running them.
PipelineResult of running a
Pipeline using SparkRunnerDebugger Use SparkRunnerDebugger.DebugSparkPipelineResult.getDebugString() to get a String representation of the Pipeline translated into
Spark native operations.Custom
KryoRegistrators for Beam's Spark runner needs and registering used class in spark
translation for better serialization performance.Registers the
SparkPipelineOptions.Registers the
SparkRunner.A
JavaStreamingContext factory for resilience.KryoRegistrator for Spark to serialize broadcast variables used for side-inputs.SideInputReader using broadcasted
SideInputValues.A
SideInputReader for the SparkRunner.An implementation of
StateInternals for the SparkRunner.Translates an unbounded portable pipeline into a Spark job.
Translation context used to lazily store Spark datasets during streaming portable pipeline
translation and compute them after translation.
Spark runner
PipelineOptions handles Spark execution-related configurations, such as the
master address, and other user-related knobs.A Spark runner build on top of Spark's SQL Engine (Structured
Streaming framework).
Contains the
PipelineRunnerRegistrar and PipelineOptionsRegistrar for the SparkStructuredStreamingRunner.Registers the
SparkStructuredStreamingPipelineOptions.Registers the
SparkStructuredStreamingRunner.An implementation of
TimerInternals for the SparkRunner.PTransform overrides for Spark runner.Translation context used to lazily store Spark data sets during portable pipeline translation and
compute them after translation.
A "composite" InputDStream implementation for
UnboundedSources.A metadata holder for an input stream partition.
State and Timers wrapper.
A specialized evaluator for ParDo operations in Spark Streaming context that is invoked when
stateful streaming is detected in the DoFn.
A class containing
StateSpec mappingFunctions.Supports translation between a Beam transform, and Spark's operations on DStreams.
Registers classes specialized by the Spark runner.
Translator matches Beam transformation with the appropriate evaluator.
A
SparkPipelineOptions for tests.A factory to provide the default watermark to stop a pipeline that reads from an unbounded
source.
The SparkRunner translate operations defined on a pipeline to a representation executable by
Spark, and then submitting the job to Spark to be executed.
Utility class for handling timers in the Spark runner.
A marker class used to identify timer keys and values in Spark transformations.
Describe a
PTransform evaluator.A
TransformTranslator provides the capability to translate a specific primitive or
composite PTransform into its Spark correspondence.Supports translation between a Beam transform, and Spark's operations on RDDs.
Translator matches Beam transformation with the appropriate evaluator.
A set of utilities to help translating Beam transformations into Spark transformations.
A SparkCombineFn function applied to grouped KVs.
A utility class to filter
TupleTags.DStream holder Can also crate a DStream from a supplied queue of values, but mainly for testing.
Kryo serializer for
ValueAndCoderLazySerializable.A holder object that lets you serialize an element with a Coder with minimal wasted space.
A
and
MetricRegistry decorator-like that supports
invalid reference
AggregatorMetricSparkBeamMetric as Gauges.A
MetricRegistry decorator-like that supports BeamMetricSets as Gauges.Duplicated from beam-examples-java to avoid dependency.
Duplicated from beam-examples-java to avoid dependency.
A PTransform that converts a PCollection containing lines of text into a PCollection of
formatted word counts.
A PTransform that converts a PCollection containing lines of text into a PCollection of
formatted word counts.
A SimpleFunction that converts a Word and Count into a printable string.
A SimpleFunction that converts a Word and Count into a printable string.
Options supported by
WordCount.Options supported by
WordCount.