Package org.apache.beam.runners.spark.io

Class MicrobatchSource<T,CheckpointMarkT extends org.apache.beam.sdk.io.UnboundedSource.CheckpointMark>

java.lang.Object

org.apache.beam.sdk.io.Source<T>

org.apache.beam.runners.spark.io.MicrobatchSource<T,CheckpointMarkT>

All Implemented Interfaces:

Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData

public class MicrobatchSource<T,CheckpointMarkT extends org.apache.beam.sdk.io.UnboundedSource.CheckpointMark>
extends org.apache.beam.sdk.io.Source<T>

A Source that accommodates Spark's micro-batch oriented nature and wraps an UnboundedSource.

See Also:

• Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
class	MicrobatchSource.Reader	Mostly based on BoundedReadFromUnboundedSource's UnboundedToBoundedSourceAdapter, with some adjustments for Spark specifics.

Method Summary

Modifier and Type	Method	Description
static void	clearCache()
boolean	equals(@Nullable Object o)
org.apache.beam.sdk.coders.Coder<CheckpointMarkT>	getCheckpointMarkCoder()
String	getId()
org.apache.beam.sdk.io.Source.Reader<T>	getOrCreateReader(org.apache.beam.sdk.options.PipelineOptions options, CheckpointMarkT checkpointMark)
org.apache.beam.sdk.coders.Coder<T>	getOutputCoder()
int	hashCode()
void	validate()

Methods inherited from class org.apache.beam.sdk.io.Source

getDefaultOutputCoder, populateDisplayData

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Method Details

getOrCreateReader

public org.apache.beam.sdk.io.Source.Reader<T> getOrCreateReader(org.apache.beam.sdk.options.PipelineOptions options,
 CheckpointMarkT checkpointMark)
                                                          throws IOException

Throws:

IOException

validate

public void validate()

Overrides:

validate in class org.apache.beam.sdk.io.Source<T>

getOutputCoder

public org.apache.beam.sdk.coders.Coder<T> getOutputCoder()

Overrides:

getOutputCoder in class org.apache.beam.sdk.io.Source<T>

getCheckpointMarkCoder

public org.apache.beam.sdk.coders.Coder<CheckpointMarkT> getCheckpointMarkCoder()

getId

public String getId()

equals

public boolean equals(@Nullable Object o)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

clearCache

public static void clearCache()