Package org.apache.flink.streaming.connectors.rabbitmq

Class RMQSource<OUT>

  • Type Parameters:
    OUT - The type of the data read from RabbitMQ.
    All Implemented Interfaces:
    Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.api.java.typeutils.ResultTypeQueryable<OUT>, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.source.SourceFunction<OUT>
    public class RMQSource<OUT>
extends org.apache.flink.streaming.api.functions.source.MultipleIdsMessageAcknowledgingSourceBase<OUT,​String,​Long>
implements org.apache.flink.api.java.typeutils.ResultTypeQueryable<OUT>
    RabbitMQ source (consumer) which reads from a queue and acknowledges messages on checkpoints. When checkpointing is enabled, it guarantees exactly-once processing semantics.

    RabbitMQ requires messages to be acknowledged. On failures, RabbitMQ will re-resend all messages which have not been acknowledged previously. When a failure occurs directly after a completed checkpoint, all messages part of this checkpoint might be processed again because they couldn't be acknowledged before failure. This case is handled by the MessageAcknowledgingSourceBase base class which deduplicates the messages using the correlation id.

    RabbitMQ's Delivery Tags do NOT represent unique ids / offsets. That's why the source uses the Correlation ID in the message properties to check for duplicate messages. Note that the correlation id has to be set at the producer. If the correlation id is not set, messages may be produced more than once in corner cases.

    This source can be operated in three different modes:

    1) Exactly-once (when checkpointed) with RabbitMQ transactions and messages with unique correlation IDs. 2) At-least-once (when checkpointed) with RabbitMQ transactions but no deduplication mechanism (correlation id is not set). 3) No strong delivery guarantees (without checkpointing) with RabbitMQ auto-commit mode.

    Users may overwrite the setupConnectionFactory() method to pass their setup their own ConnectionFactory in case the constructor parameters are not sufficient.

    See Also:
    Serialized Form

    • Nested Class Summary

      • Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

        org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T extends Object>

    • Field Summary

      Fields 
      Modifier and Type Field Description
      protected boolean autoAck  
      protected com.rabbitmq.client.Channel channel  
      protected com.rabbitmq.client.Connection connection  
      protected org.apache.flink.streaming.connectors.rabbitmq.QueueingConsumer consumer  
      protected RMQDeserializationSchema<OUT> deliveryDeserializer  
      protected String queueName  

      • Fields inherited from class org.apache.flink.streaming.api.functions.source.MultipleIdsMessageAcknowledgingSourceBase

        sessionIds, sessionIdsPerSnapshot

      • Fields inherited from class org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase

        pendingCheckpoints

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      protected void acknowledgeSessionIDs​(List<Long> sessionIds)  
      void cancel()  
      void close()  
      org.apache.flink.api.common.typeinfo.TypeInformation<OUT> getProducedType()  
      void open​(org.apache.flink.configuration.Configuration config)  
      void run​(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<OUT> ctx)  
      protected com.rabbitmq.client.Connection setupConnection()
      Initializes the connection to RMQ using the default connection factory from setupConnectionFactory().
      protected com.rabbitmq.client.ConnectionFactory setupConnectionFactory()
      Initializes the connection to RMQ with a default connection factory.
      protected void setupQueue()
      Sets up the queue.

      • Methods inherited from class org.apache.flink.streaming.api.functions.source.MultipleIdsMessageAcknowledgingSourceBase

        acknowledgeIDs, snapshotState

      • Methods inherited from class org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase

        addId, initializeState, notifyCheckpointAborted, notifyCheckpointComplete

      • Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

        getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

    • Field Detail

      • queueName

        protected final String queueName

      • connection

        protected transient com.rabbitmq.client.Connection connection

      • channel

        protected transient com.rabbitmq.client.Channel channel

      • consumer

        protected transient org.apache.flink.streaming.connectors.rabbitmq.QueueingConsumer consumer

      • autoAck

        protected transient boolean autoAck

    • Constructor Detail

      • RMQSource

        public RMQSource​(RMQConnectionConfig rmqConnectionConfig,
                 String queueName,
                 org.apache.flink.api.common.serialization.DeserializationSchema<OUT> deserializationSchema)
        Creates a new RabbitMQ source with at-least-once message processing guarantee when checkpointing is enabled. No strong delivery guarantees when checkpointing is disabled.

        For exactly-once, please use the constructor RMQSource(RMQConnectionConfig, String, boolean, DeserializationSchema).

        Parameters:
        rmqConnectionConfig - The RabbiMQ connection configuration RMQConnectionConfig.
        queueName - The queue to receive messages from.
        deserializationSchema - A DeserializationSchema for turning the bytes received into Java objects.

      • RMQSource

        public RMQSource​(RMQConnectionConfig rmqConnectionConfig,
                 String queueName,
                 boolean usesCorrelationId,
                 org.apache.flink.api.common.serialization.DeserializationSchema<OUT> deserializationSchema)
        Creates a new RabbitMQ source. For exactly-once, you must set the correlation ids of messages at the producer. The correlation id must be unique. Otherwise the behavior of the source is undefined. If in doubt, set usesCorrelationId to false. When correlation ids are not used, this source has at-least-once processing semantics when checkpointing is enabled.
        Parameters:
        rmqConnectionConfig - The RabbiMQ connection configuration RMQConnectionConfig.
        queueName - The queue to receive messages from.
        usesCorrelationId - Whether the messages received are supplied with a unique id to deduplicate messages (in case of failed acknowledgments). Only used when checkpointing is enabled.
        deserializationSchema - A DeserializationSchema for turning the bytes received into Java objects.

      • RMQSource

        public RMQSource​(RMQConnectionConfig rmqConnectionConfig,
                 String queueName,
                 boolean usesCorrelationId,
                 RMQDeserializationSchema<OUT> deliveryDeserializer)
        Creates a new RabbitMQ source. For exactly-once, you must set the correlation ids of messages at the producer. The correlation id must be unique. Otherwise the behavior of the source is undefined. If in doubt, set usesCorrelationId to false. When correlation ids are not used, this source has at-least-once processing semantics when checkpointing is enabled.

        It also uses the provided RMQDeserializationSchema to parse both the correlationID and the message.

        Parameters:
        rmqConnectionConfig - The RabbiMQ connection configuration RMQConnectionConfig.
        queueName - The queue to receive messages from.
        usesCorrelationId - Whether the messages received are supplied with a unique id to deduplicate messages (in case of failed acknowledgments). Only used when checkpointing is enabled.
        deliveryDeserializer - A RMQDeserializationSchema for parsing the RMQDelivery.

    • Method Detail

      • setupConnectionFactory

        protected com.rabbitmq.client.ConnectionFactory setupConnectionFactory()
                                                                throws Exception
        Initializes the connection to RMQ with a default connection factory. The user may override this method to setup and configure their own ConnectionFactory.
        Throws:
        Exception

      • setupConnection

        @VisibleForTesting
protected com.rabbitmq.client.Connection setupConnection()
                                                  throws Exception
        Initializes the connection to RMQ using the default connection factory from setupConnectionFactory(). The user may override this method to setup and configure their own Connection.
        Throws:
        Exception

      • setupQueue

        @VisibleForTesting
protected void setupQueue()
                   throws IOException
        Sets up the queue. The default implementation just declares the queue. The user may override this method to have a custom setup for the queue (i.e. binding the queue to an exchange or defining custom queue parameters)
        Throws:
        IOException

      • open

        public void open​(org.apache.flink.configuration.Configuration config)
          throws Exception
        Specified by:
        open in interface org.apache.flink.api.common.functions.RichFunction
        Overrides:
        open in class org.apache.flink.streaming.api.functions.source.MultipleIdsMessageAcknowledgingSourceBase<OUT,​String,​Long>
        Throws:
        Exception

      • close

        public void close()
           throws Exception
        Specified by:
        close in interface org.apache.flink.api.common.functions.RichFunction
        Overrides:
        close in class org.apache.flink.streaming.api.functions.source.MultipleIdsMessageAcknowledgingSourceBase<OUT,​String,​Long>
        Throws:
        Exception

      • run

        public void run​(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<OUT> ctx)
         throws Exception
        Specified by:
        run in interface org.apache.flink.streaming.api.functions.source.SourceFunction<OUT>
        Throws:
        Exception

      • cancel

        public void cancel()
        Specified by:
        cancel in interface org.apache.flink.streaming.api.functions.source.SourceFunction<OUT>

      • acknowledgeSessionIDs

        protected void acknowledgeSessionIDs​(List<Long> sessionIds)
        Specified by:
        acknowledgeSessionIDs in class org.apache.flink.streaming.api.functions.source.MultipleIdsMessageAcknowledgingSourceBase<OUT,​String,​Long>

      • getProducedType

        public org.apache.flink.api.common.typeinfo.TypeInformation<OUT> getProducedType()
        Specified by:
        getProducedType in interface org.apache.flink.api.java.typeutils.ResultTypeQueryable<OUT>