Package org.apache.beam.sdk.extensions.ordered

Class OrderedProcessingHandler<EventT,​KeyT,​StateT extends MutableState<EventT,​?>,​ResultT>

  • Type Parameters:
    EventT - type of events to be processed
    KeyT - type of keys which will be used to group the events
    StateT - type of internal State which will be used for processing
    ResultT - type of the result of the processing which will be output
    All Implemented Interfaces:
    java.io.Serializable
    Direct Known Subclasses:
    OrderedProcessingHandler.OrderedProcessingGlobalSequenceHandler
    public abstract class OrderedProcessingHandler<EventT,​KeyT,​StateT extends MutableState<EventT,​?>,​ResultT>
extends java.lang.Object
implements java.io.Serializable
    Parent class for Ordered Processing configuration handlers.

    There are two types of processing - when the sequence numbers are contiguous per key and these sequences per keys are independent of each other, and when there is a global sequence shared by all keys. In case of the global sequence processing the custom handler must extend from OrderedProcessingHandler.OrderedProcessingGlobalSequenceHandler.

    See Also:
    Serialized Form

    • Field Detail

      • DEFAULT_MAX_ELEMENTS_TO_OUTPUT

        public static final int DEFAULT_MAX_ELEMENTS_TO_OUTPUT
        See Also:
        Constant Field Values

    • Constructor Detail

      • OrderedProcessingHandler

        public OrderedProcessingHandler​(java.lang.Class<EventT> eventTClass,
                                java.lang.Class<KeyT> keyTClass,
                                java.lang.Class<StateT> stateTClass,
                                java.lang.Class<ResultT> resultTClass)
        Provide concrete classes which will be used by the ordered processing transform.
        Parameters:
        eventTClass - class of the events
        keyTClass - class of the keys
        stateTClass - class of the state
        resultTClass - class of the results

    • Method Detail

      • getEventExaminer

        public abstract @NonNull EventExaminer<EventT,​StateT> getEventExaminer()
        Returns the event examiner instance which will be used by the transform.

      • getEventCoder

        public @NonNull org.apache.beam.sdk.coders.Coder<EventT> getEventCoder​(org.apache.beam.sdk.Pipeline pipeline,
                                                                       org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.values.KV<KeyT,​org.apache.beam.sdk.values.KV<java.lang.Long,​EventT>>> inputCoder)
                                                                throws org.apache.beam.sdk.coders.CannotProvideCoderException
        Provide the event coder.

        The default implementation of the method will use the event coder from the input PCollection. If the input PCollection doesn't use KVCoder, it will attempt to get the coder from the pipeline's coder registry.

        Parameters:
        pipeline - of the transform
        inputCoder - input coder of the transform
        Returns:
        event coder
        Throws:
        org.apache.beam.sdk.coders.CannotProvideCoderException - if the method can't determine the coder based on the above algorithm.

      • getStateCoder

        public @NonNull org.apache.beam.sdk.coders.Coder<StateT> getStateCoder​(org.apache.beam.sdk.Pipeline pipeline)
                                                                throws org.apache.beam.sdk.coders.CannotProvideCoderException
        Provide the state coder.

        The default implementation will attempt to get the coder from the pipeline's code registry.

        Parameters:
        pipeline - of the transform
        Returns:
        the state coder
        Throws:
        org.apache.beam.sdk.coders.CannotProvideCoderException

      • getKeyCoder

        public @NonNull org.apache.beam.sdk.coders.Coder<KeyT> getKeyCoder​(org.apache.beam.sdk.Pipeline pipeline,
                                                                   org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.values.KV<KeyT,​org.apache.beam.sdk.values.KV<java.lang.Long,​EventT>>> inputCoder)
                                                            throws org.apache.beam.sdk.coders.CannotProvideCoderException
        Provide the key coder.

        The default implementation of the method will use the event coder from the input PCollection. If the input PCollection doesn't use KVCoder, it will attempt to get the coder from the pipeline's coder registry.

        Parameters:
        pipeline -
        inputCoder -
        Returns:
        Throws:
        org.apache.beam.sdk.coders.CannotProvideCoderException - if the method can't determine the coder based on the above algorithm.

      • getResultCoder

        public @NonNull org.apache.beam.sdk.coders.Coder<ResultT> getResultCoder​(org.apache.beam.sdk.Pipeline pipeline)
                                                                  throws org.apache.beam.sdk.coders.CannotProvideCoderException
        Provide the result coder.

        The default implementation will attempt to get the coder from the pipeline's code registry.

        Parameters:
        pipeline -
        Returns:
        result coder
        Throws:
        org.apache.beam.sdk.coders.CannotProvideCoderException

      • getStatusUpdateFrequency

        public @Nullable org.joda.time.Duration getStatusUpdateFrequency()
        Determines the frequency of emission of the OrderedProcessingStatus elements.

        Default is 5 seconds.

        Returns:
        the frequency of updates. If null is returned, no updates will be emitted on a scheduled basis.

      • setStatusUpdateFrequency

        public void setStatusUpdateFrequency​(org.joda.time.Duration statusUpdateFrequency)
        Changes the default status update frequency. Updates will be disabled if set to null.
        Parameters:
        statusUpdateFrequency -

      • setProduceStatusUpdateOnEveryEvent

        public void setProduceStatusUpdateOnEveryEvent​(boolean value)
        Sets the indicator of whether the status notification needs to be produced on every event.
        Parameters:
        value -

      • getMaxOutputElementsPerBundle

        public int getMaxOutputElementsPerBundle()
        Returns the maximum number of elements which will be output per each bundle. The default is 10,000 elements.

        This is used to limit the amount of data produced for each bundle - many runners have limitations on how much data can be output from a single bundle. If many events arrive out of sequence and are buffered then at some point a single event can cause processing of a large number of buffered events.

        Returns:

      • setMaxOutputElementsPerBundle

        public void setMaxOutputElementsPerBundle​(int maxOutputElementsPerBundle)
        Overrides the default value.
        Parameters:
        maxOutputElementsPerBundle -