Package org.apache.druid.frame.processor

Class FrameChannelMerger

java.lang.Object

org.apache.druid.frame.processor.FrameChannelMerger

All Implemented Interfaces:

FrameProcessor<Long>

public class FrameChannelMerger
extends Object
implements FrameProcessor<Long>

Processor that merges already-sorted inputChannels and writes a fully-sorted stream to a single outputChannel. Frames from input channels must be FrameType.isRowBased(). Output frames will be row-based as well. Optionally supports combining adjacent rows with identical sort keys via a FrameCombiner, which reduces intermediate data volume during merge-sort. All combine-specific state is held in FrameChannelMerger.CombineState. For unsorted output, use FrameChannelMixer instead.

Constructor Summary

Constructors

Constructor	Description
FrameChannelMerger(List<ReadableFrameChannel> inputChannels, FrameReader frameReader, WritableFrameChannel outputChannel, FrameWriterFactory frameWriterFactory, List<KeyColumn> sortKey, FrameCombiner combiner, ClusterByPartitions partitions, long rowLimit)

Method Summary

Modifier and Type	Method	Description
void	cleanup()	Closes resources used by this worker.
List<ReadableFrameChannel>	inputChannels()	List of input channels.
List<WritableFrameChannel>	outputChannels()	List of output channels.
ReturnOrAwait<Long>	runIncrementally(it.unimi.dsi.fastutil.ints.IntSet readableInputs)	Runs some of the algorithm, without blocking, and either returns a value or a set of input channels to wait for.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

FrameChannelMerger

public FrameChannelMerger(List<ReadableFrameChannel> inputChannels,
 FrameReader frameReader,
 WritableFrameChannel outputChannel,
 FrameWriterFactory frameWriterFactory,
 List<KeyColumn> sortKey,
 @Nullable
 FrameCombiner combiner,
 @Nullable
 ClusterByPartitions partitions,
 long rowLimit)

Parameters:

inputChannels - readable frame channels. Each channel must be sorted (i.e., if all frames in the channel are concatenated, the concatenated result must be fully sorted).

frameReader - reader for frames

outputChannel - writable channel to receive the merge-sorted data

frameWriterFactory - writer for frames

sortKey - sort key for input and output frames

combiner - optional combiner for merging rows with identical sort keys

partitions - partitions for output frames. If non-null, output frames are written with partition numbers set according to this parameter

rowLimit - maximum number of rows to write to the output channel

Method Details

inputChannels

public List<ReadableFrameChannel> inputChannels()

Description copied from interface: FrameProcessor

List of input channels. The positions of channels in this list are used to build the readableInputs set provided to FrameProcessor.runIncrementally(it.unimi.dsi.fastutil.ints.IntSet).

Specified by:

inputChannels in interface FrameProcessor<Long>

outputChannels

public List<WritableFrameChannel> outputChannels()

Description copied from interface: FrameProcessor

List of output channels.

Specified by:

outputChannels in interface FrameProcessor<Long>

runIncrementally

public ReturnOrAwait<Long> runIncrementally(it.unimi.dsi.fastutil.ints.IntSet readableInputs)
                                     throws IOException

Description copied from interface: FrameProcessor

Runs some of the algorithm, without blocking, and either returns a value or a set of input channels to wait for. This method is called by FrameProcessorExecutor.runFully(org.apache.druid.frame.processor.FrameProcessor<T>, java.lang.String) when all output channels are writable. Therefore, it is guaranteed that each output channel can accept at least one frame. This method must not read more than one frame from each readable input channel, and must not write more than one frame to each output channel.

Specified by:

runIncrementally in interface FrameProcessor<Long>

Parameters:

readableInputs - channels from FrameProcessor.inputChannels() that are either finished or ready to read. That is: either ReadableFrameChannel.isFinished() or ReadableFrameChannel.canRead() are true.

Returns:

either a final return value or a set of input channels to wait for. Must be nonnull.

Throws:

IOException

cleanup

public void cleanup()
             throws IOException

Description copied from interface: FrameProcessor

Closes resources used by this worker. Exact same concept as Closeable.close(). This interface does not extend Closeable, in order to make it easier to find all places where cleanup happens. (Static analysis tools can lose the thread when Closeables are closed in generic ways.) Implementations typically call ReadableFrameChannel.close() and WritableFrameChannel.close() on all input and output channels, as well as releasing any additional resources that may be held, such as FrameWriter. In cases of cancellation, this method may be called even if FrameProcessor.runIncrementally(it.unimi.dsi.fastutil.ints.IntSet) has not yet returned a result via ReturnOrAwait.returnObject(T).

Specified by:

cleanup in interface FrameProcessor<Long>

Throws:

IOException