Package org.apache.beam.runners.spark.io

Class MicrobatchSource.Reader

java.lang.Object

org.apache.beam.sdk.io.Source.Reader<T>

org.apache.beam.runners.spark.io.MicrobatchSource.Reader

All Implemented Interfaces:

AutoCloseable

Enclosing class:

MicrobatchSource<T,CheckpointMarkT extends org.apache.beam.sdk.io.UnboundedSource.CheckpointMark>

public class MicrobatchSource.Reader
extends org.apache.beam.sdk.io.Source.Reader<T>

Mostly based on BoundedReadFromUnboundedSource's UnboundedToBoundedSourceAdapter, with some adjustments for Spark specifics.

This Reader reads until one of the following thresholds has been reached:

1. max records (per batch)
2. max read duration (per batch)

Method Summary

Modifier and Type	Method	Description
boolean	advance()
void	close()
CheckpointMarkT	getCheckpointMark()
T	getCurrent()
org.apache.beam.sdk.io.Source<T>	getCurrentSource()
org.joda.time.Instant	getCurrentTimestamp()
org.joda.time.Instant	getWatermark()
boolean	start()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

start

public boolean start()
              throws IOException

Specified by:

start in class org.apache.beam.sdk.io.Source.Reader<T>

Throws:

IOException

advance

public boolean advance()
                throws IOException

Specified by:

advance in class org.apache.beam.sdk.io.Source.Reader<T>

Throws:

IOException

getCurrent

public T getCurrent()
             throws NoSuchElementException

Specified by:

getCurrent in class org.apache.beam.sdk.io.Source.Reader<T>

Throws:

NoSuchElementException

getCurrentTimestamp

public org.joda.time.Instant getCurrentTimestamp()
                                          throws NoSuchElementException

Specified by:

getCurrentTimestamp in class org.apache.beam.sdk.io.Source.Reader<T>

Throws:

NoSuchElementException

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in class org.apache.beam.sdk.io.Source.Reader<T>

Throws:

IOException

getCurrentSource

public org.apache.beam.sdk.io.Source<T> getCurrentSource()

Specified by:

getCurrentSource in class org.apache.beam.sdk.io.Source.Reader<T>

getCheckpointMark

public CheckpointMarkT getCheckpointMark()

getWatermark

public org.joda.time.Instant getWatermark()