Package org.apache.flink.runtime.io.disk.iomanager

Class IOManager

    • Field Detail

      • LOG

        protected static final org.slf4j.Logger LOG

    • Constructor Detail

      • IOManager

        protected IOManager​(String[] tempDirs,
                    ExecutorService executorService)
        Constructs a new IOManager.
        Parameters:
        tempDirs - The basic directories for files underlying anonymous channels.

    • Method Detail

      • createChannel

        public FileIOChannel.ID createChannel()
        Creates a new FileIOChannel.ID in one of the temp directories. Multiple invocations of this method spread the channels evenly across the different directories.
        Returns:
        A channel to a temporary directory.

      • createChannelEnumerator

        public FileIOChannel.Enumerator createChannelEnumerator()
        Creates a new FileIOChannel.Enumerator, spreading the channels in a round-robin fashion across the temporary file directories.
        Returns:
        An enumerator for channels.

      • deleteChannel

        public static void deleteChannel​(FileIOChannel.ID channel)
        Deletes the file underlying the given channel. If the channel is still open, this call may fail.
        Parameters:
        channel - The channel to be deleted.

      • getSpillingDirectories

        public File[] getSpillingDirectories()
        Gets the directories that the I/O manager spills to.
        Returns:
        The directories that the I/O manager spills to.

      • getSpillingDirectoriesPaths

        public String[] getSpillingDirectoriesPaths()
        Gets the directories that the I/O manager spills to, as path strings.
        Returns:
        The directories that the I/O manager spills to, as path strings.

      • createBlockChannelWriter

        public BlockChannelWriter<org.apache.flink.core.memory.MemorySegment> createBlockChannelWriter​(FileIOChannel.ID channelID)
                                                                                        throws IOException
        Creates a block channel writer that writes to the given channel. The writer adds the written segment to its return-queue afterwards (to allow for asynchronous implementations).
        Parameters:
        channelID - The descriptor for the channel to write to.
        Returns:
        A block channel writer that writes to the given channel.
        Throws:
        IOException - Thrown, if the channel for the writer could not be opened.

      • createBlockChannelWriter

        public abstract BlockChannelWriter<org.apache.flink.core.memory.MemorySegment> createBlockChannelWriter​(FileIOChannel.ID channelID,
                                                                                                        LinkedBlockingQueue<org.apache.flink.core.memory.MemorySegment> returnQueue)
                                                                                                 throws IOException
        Creates a block channel writer that writes to the given channel. The writer adds the written segment to the given queue (to allow for asynchronous implementations).
        Parameters:
        channelID - The descriptor for the channel to write to.
        returnQueue - The queue to put the written buffers into.
        Returns:
        A block channel writer that writes to the given channel.
        Throws:
        IOException - Thrown, if the channel for the writer could not be opened.

      • createBlockChannelWriter

        public abstract BlockChannelWriterWithCallback<org.apache.flink.core.memory.MemorySegment> createBlockChannelWriter​(FileIOChannel.ID channelID,
                                                                                                                    RequestDoneCallback<org.apache.flink.core.memory.MemorySegment> callback)
                                                                                                             throws IOException
        Creates a block channel writer that writes to the given channel. The writer calls the given callback after the I/O operation has been performed (successfully or unsuccessfully), to allow for asynchronous implementations.
        Parameters:
        channelID - The descriptor for the channel to write to.
        callback - The callback to be called for
        Returns:
        A block channel writer that writes to the given channel.
        Throws:
        IOException - Thrown, if the channel for the writer could not be opened.

      • createBlockChannelReader

        public BlockChannelReader<org.apache.flink.core.memory.MemorySegment> createBlockChannelReader​(FileIOChannel.ID channelID)
                                                                                        throws IOException
        Creates a block channel reader that reads blocks from the given channel. The reader pushed full memory segments (with the read data) to its "return queue", to allow for asynchronous read implementations.
        Parameters:
        channelID - The descriptor for the channel to write to.
        Returns:
        A block channel reader that reads from the given channel.
        Throws:
        IOException - Thrown, if the channel for the reader could not be opened.

      • createBlockChannelReader

        public abstract BlockChannelReader<org.apache.flink.core.memory.MemorySegment> createBlockChannelReader​(FileIOChannel.ID channelID,
                                                                                                        LinkedBlockingQueue<org.apache.flink.core.memory.MemorySegment> returnQueue)
                                                                                                 throws IOException
        Creates a block channel reader that reads blocks from the given channel. The reader pushes the full segments to the given queue, to allow for asynchronous implementations.
        Parameters:
        channelID - The descriptor for the channel to write to.
        returnQueue - The queue to put the full buffers into.
        Returns:
        A block channel reader that reads from the given channel.
        Throws:
        IOException - Thrown, if the channel for the reader could not be opened.

      • createBulkBlockChannelReader

        public abstract BulkBlockChannelReader createBulkBlockChannelReader​(FileIOChannel.ID channelID,
                                                                    List<org.apache.flink.core.memory.MemorySegment> targetSegments,
                                                                    int numBlocks)
                                                             throws IOException
        Creates a block channel reader that reads all blocks from the given channel directly in one bulk. The reader draws segments to read the blocks into from a supplied list, which must contain as many segments as the channel has blocks. After the reader is done, the list with the full segments can be obtained from the reader.

        If a channel is not to be read in one bulk, but in multiple smaller batches, a BlockChannelReader should be used.

        Parameters:
        channelID - The descriptor for the channel to write to.
        targetSegments - The list to take the segments from into which to read the data.
        numBlocks - The number of blocks in the channel to read.
        Returns:
        A block channel reader that reads from the given channel.
        Throws:
        IOException - Thrown, if the channel for the reader could not be opened.