Package org.apache.flink.streaming.connectors.gcp.pubsub

Class PubSubSource<OUT>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.source.RichSourceFunction<OUT>

org.apache.flink.streaming.connectors.gcp.pubsub.PubSubSource<OUT>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.api.java.typeutils.ResultTypeQueryable<OUT>, org.apache.flink.streaming.api.checkpoint.ListCheckpointed<AcknowledgeIdsForCheckpoint<String>>, org.apache.flink.streaming.api.functions.source.ParallelSourceFunction<OUT>, org.apache.flink.streaming.api.functions.source.SourceFunction<OUT>

public class PubSubSource<OUT>
extends org.apache.flink.streaming.api.functions.source.RichSourceFunction<OUT>
implements org.apache.flink.api.java.typeutils.ResultTypeQueryable<OUT>, org.apache.flink.streaming.api.functions.source.ParallelSourceFunction<OUT>, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.checkpoint.ListCheckpointed<AcknowledgeIdsForCheckpoint<String>>

PubSub Source, this Source will consume PubSub messages from a subscription and Acknowledge them on the next checkpoint. This ensures every message will get acknowledged at least once.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	PubSubSource.DeserializationSchemaBuilder	Part of PubSubSource.PubSubSourceBuilder to set required fields.
static interface	PubSubSource.ProjectNameBuilder<OUT>	Part of PubSubSource.PubSubSourceBuilder to set required fields.
static class	PubSubSource.PubSubSourceBuilder<OUT>	Builder to create PubSubSource.
static interface	PubSubSource.SubscriptionNameBuilder<OUT>	Part of PubSubSource.PubSubSourceBuilder to set required fields.

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T extends Object>

Field Summary

Fields

Modifier and Type	Field	Description
protected AcknowledgeOnCheckpoint<String>	acknowledgeOnCheckpoint
protected org.apache.flink.streaming.connectors.gcp.pubsub.PubSubSource.AcknowledgeOnCheckpointFactory	acknowledgeOnCheckpointFactory
protected com.google.auth.Credentials	credentials
protected PubSubDeserializationSchema<OUT>	deserializationSchema
protected boolean	isRunning
protected int	messagePerSecondRateLimit
protected PubSubSubscriberFactory	pubSubSubscriberFactory
protected org.apache.flink.api.common.io.ratelimiting.FlinkConnectorRateLimiter	rateLimiter
protected PubSubSubscriber	subscriber

Method Summary

Modifier and Type	Method	Description
void	cancel()
void	close()
org.apache.flink.api.common.typeinfo.TypeInformation<OUT>	getProducedType()
static PubSubSource.DeserializationSchemaBuilder	newBuilder()
void	notifyCheckpointAborted(long checkpointId)
void	notifyCheckpointComplete(long checkpointId)
void	open(org.apache.flink.configuration.Configuration configuration)
void	restoreState(List<AcknowledgeIdsForCheckpoint<String>> state)
void	run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<OUT> sourceContext)
List<AcknowledgeIdsForCheckpoint<String>>	snapshotState(long checkpointId, long timestamp)

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

deserializationSchema

protected final PubSubDeserializationSchema<OUT> deserializationSchema

pubSubSubscriberFactory

protected final PubSubSubscriberFactory pubSubSubscriberFactory

credentials

protected final com.google.auth.Credentials credentials

acknowledgeOnCheckpointFactory

protected final org.apache.flink.streaming.connectors.gcp.pubsub.PubSubSource.AcknowledgeOnCheckpointFactory acknowledgeOnCheckpointFactory

rateLimiter

protected final org.apache.flink.api.common.io.ratelimiting.FlinkConnectorRateLimiter rateLimiter

messagePerSecondRateLimit

protected final int messagePerSecondRateLimit

acknowledgeOnCheckpoint

protected transient AcknowledgeOnCheckpoint<String> acknowledgeOnCheckpoint

subscriber

protected transient PubSubSubscriber subscriber

isRunning

protected transient volatile boolean isRunning

Method Detail

open

public void open(org.apache.flink.configuration.Configuration configuration)
          throws Exception

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

run

public void run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<OUT> sourceContext)
         throws Exception

Specified by:

run in interface org.apache.flink.streaming.api.functions.source.SourceFunction<OUT>

Throws:

Exception

close

public void close()
           throws Exception

Specified by:

close in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

close in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

cancel

public void cancel()

Specified by:

cancel in interface org.apache.flink.streaming.api.functions.source.SourceFunction<OUT>

getProducedType

public org.apache.flink.api.common.typeinfo.TypeInformation<OUT> getProducedType()

Specified by:

getProducedType in interface org.apache.flink.api.java.typeutils.ResultTypeQueryable<OUT>

newBuilder

public static PubSubSource.DeserializationSchemaBuilder newBuilder()

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)
                              throws Exception

Specified by:

notifyCheckpointComplete in interface org.apache.flink.api.common.state.CheckpointListener

Throws:

Exception

notifyCheckpointAborted

public void notifyCheckpointAborted(long checkpointId)

Specified by:

notifyCheckpointAborted in interface org.apache.flink.api.common.state.CheckpointListener

snapshotState

public List<AcknowledgeIdsForCheckpoint<String>> snapshotState(long checkpointId,
                                                               long timestamp)
                                                        throws Exception

Specified by:

snapshotState in interface org.apache.flink.streaming.api.checkpoint.ListCheckpointed<OUT>

Throws:

Exception

restoreState

public void restoreState(List<AcknowledgeIdsForCheckpoint<String>> state)
                  throws Exception

Specified by:

restoreState in interface org.apache.flink.streaming.api.checkpoint.ListCheckpointed<OUT>

Throws:

Exception