Package org.apache.flink.connector.file.src.reader

Class SimpleStreamFormat<T>

  • Type Parameters:
    T - The type of records created by this format reader.
    All Implemented Interfaces:
    Serializable, org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>, StreamFormat<T>
    Direct Known Subclasses:
    TextLineInputFormat
    @PublicEvolving
public abstract class SimpleStreamFormat<T>
extends Object
implements StreamFormat<T>
    A simple version of the StreamFormat, for formats that are not splittable.

    This format makes no difference between creating readers from scratch (new file) or from a checkpoint. Because of that, if the reader actively checkpoints its position (via the StreamFormat.Reader.getCheckpointedPosition() method) then the checkpointed offset must be a byte offset in the file from which the stream can be resumed as if it were the beginning of the file.

    For all other details, please check the docs of StreamFormat.

    See Also:
    Serialized Form

    • Constructor Detail

      • SimpleStreamFormat

        public SimpleStreamFormat()

    • Method Detail

      • createReader

        public abstract StreamFormat.Reader<T> createReader​(org.apache.flink.configuration.Configuration config,
                                                    org.apache.flink.core.fs.FSDataInputStream stream)
                                             throws IOException
        Creates a new reader. This method is called both for the creation of new reader (from the beginning of a file) and for restoring checkpointed readers.

        If the reader previously checkpointed an offset, then the input stream will be positioned to that particular offset. Readers checkpoint an offset by returning a value from the method StreamFormat.Reader.getCheckpointedPosition() method with an offset other than CheckpointedPosition.NO_OFFSET).

        Throws:
        IOException

      • getProducedType

        public abstract org.apache.flink.api.common.typeinfo.TypeInformation<T> getProducedType()
        Gets the type produced by this format. This type will be the type produced by the file source as a whole.
        Specified by:
        getProducedType in interface org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>
        Specified by:
        getProducedType in interface StreamFormat<T>

      • isSplittable

        public final boolean isSplittable()
        This format is always not splittable.
        Specified by:
        isSplittable in interface StreamFormat<T>

      • createReader

        public final StreamFormat.Reader<T> createReader​(org.apache.flink.configuration.Configuration config,
                                                 org.apache.flink.core.fs.FSDataInputStream stream,
                                                 long fileLen,
                                                 long splitEnd)
                                          throws IOException
        Description copied from interface: StreamFormat
        Creates a new reader to read in this format. This method is called when a fresh reader is created for a split that was assigned from the enumerator. This method may also be called on recovery from a checkpoint, if the reader never stored an offset in the checkpoint (see StreamFormat.restoreReader(Configuration, FSDataInputStream, long, long, long) for details.

        If the format is splittable, then the stream is positioned to the beginning of the file split, otherwise it will be at position zero.

        The fileLen is the length of the entire file, while splitEnd is the offset of the first byte after the split end boundary (exclusive end boundary). For non-splittable formats, both values are identical.

        Specified by:
        createReader in interface StreamFormat<T>
        Throws:
        IOException

      • restoreReader

        public final StreamFormat.Reader<T> restoreReader​(org.apache.flink.configuration.Configuration config,
                                                  org.apache.flink.core.fs.FSDataInputStream stream,
                                                  long restoredOffset,
                                                  long fileLen,
                                                  long splitEnd)
                                           throws IOException
        Description copied from interface: StreamFormat
        Restores a reader from a checkpointed position. This method is called when the reader is recovered from a checkpoint and the reader has previously stored an offset into the checkpoint, by returning from the StreamFormat.Reader.getCheckpointedPosition() a value with non-negative offset. That value is supplied as the restoredOffset.

        If the format is splittable, then the stream is positioned to the beginning of the file split, otherwise it will be at position zero. The stream is NOT positioned to the checkpointed offset, because the format is free to interpret this offset in a different way than the byte offset in the file (for example as a record index).

        If the reader never produced a CheckpointedPosition with a non-negative offset before, then this method is not called, and the reader is created in the same way as a fresh reader via the method StreamFormat.createReader(Configuration, FSDataInputStream, long, long) and the appropriate number of records are read and discarded, to position to reader to the checkpointed position.

        Having a different method for restoring readers to a checkpointed position allows readers to seek to the start position differently in that case, compared to when the reader is created from a split offset generated at the enumerator. In the latter case, the offsets are commonly "approximate", because the enumerator typically generates splits based only on metadata. Readers then have to skip some bytes while searching for the next position to start from (based on a delimiter, sync marker, block offset, etc.). In contrast, checkpointed offsets are often precise, because they were recorded as the reader when through the data stream. Starting a reader from a checkpointed offset may hence not require and search for the next delimiter/block/marker.

        The fileLen is the length of the entire file, while splitEnd is the offset of the first byte after the split end boundary (exclusive end boundary). For non-splittable formats, both values are identical.

        Specified by:
        restoreReader in interface StreamFormat<T>
        Throws:
        IOException