org.apache.hudi.common.util.queue

Class BaseHoodieQueueBasedExecutor<I,O,E>

java.lang.Object

org.apache.hudi.common.util.queue.BaseHoodieQueueBasedExecutor<I,O,E>

All Implemented Interfaces:

HoodieExecutor<E>

Direct Known Subclasses:

BoundedInMemoryExecutor, DisruptorExecutor

public abstract class BaseHoodieQueueBasedExecutor<I,O,E>
extends Object
implements HoodieExecutor<E>

Base class for multi-threaded queue-based executor which

• Can be ingesting instances of type I from multiple HoodieProducers into the HoodieMessageQueue
• Can be ingesting instances of type I into an (optional) HoodieConsumer from the internal HoodieMessageQueue (when no consumer is provided records are simply accumulated into internal queue)

Such executors are allowing to setup an ingestion pipeline w/ N:1 configuration, where data is ingested from multiple sources (ie producers) into a singular sink (ie consumer), using an internal queue to stage the records ingested from producers before these are consumed

Field Summary

Fields

Modifier and Type	Field and Description
protected Option<HoodieConsumer<O,E>>	consumer
protected HoodieMessageQueue<I,O>	queue

Constructor Summary

Constructors

Constructor and Description
BaseHoodieQueueBasedExecutor(List<HoodieProducer<I>> producers, Option<HoodieConsumer<O,E>> consumer, HoodieMessageQueue<I,O> queue, Runnable preExecuteRunnable)

Method Summary

Modifier and Type	Method and Description
boolean	awaitTermination() Allows to gracefully await the termination of the executor
protected abstract void	doConsume(HoodieMessageQueue<I,O> queue, HoodieConsumer<O,E> consumer)
protected void	doProduce(HoodieMessageQueue<I,O> queue, HoodieProducer<I> producer)
E	execute() Main API to run both production and consumption.
boolean	isRunning()
protected void	setUp()
void	shutdownNow() Shuts executor down immediately, cleaning up any allocated resources
CompletableFuture<Void>	startProducingAsync() Start producing

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

queue

protected final HoodieMessageQueue<I,O> queue

consumer

protected final Option<HoodieConsumer<O,E>> consumer

Constructor Detail

BaseHoodieQueueBasedExecutor

public BaseHoodieQueueBasedExecutor(List<HoodieProducer<I>> producers,
                                    Option<HoodieConsumer<O,E>> consumer,
                                    HoodieMessageQueue<I,O> queue,
                                    Runnable preExecuteRunnable)

Method Detail

doProduce

protected void doProduce(HoodieMessageQueue<I,O> queue,
                         HoodieProducer<I> producer)

doConsume

protected abstract void doConsume(HoodieMessageQueue<I,O> queue,
                                  HoodieConsumer<O,E> consumer)

setUp

protected void setUp()

startProducingAsync

public final CompletableFuture<Void> startProducingAsync()

Start producing

awaitTermination

public final boolean awaitTermination()

Description copied from interface: HoodieExecutor

Allows to gracefully await the termination of the executor

Specified by:

awaitTermination in interface HoodieExecutor<E>

shutdownNow

public final void shutdownNow()

Description copied from interface: HoodieExecutor

Shuts executor down immediately, cleaning up any allocated resources

Specified by:

shutdownNow in interface HoodieExecutor<E>

isRunning

public boolean isRunning()

execute

public E execute()

Main API to run both production and consumption.

Specified by:

execute in interface HoodieExecutor<E>