Package org.apache.iotdb.tsfile.read

Class TsFileSequenceReader

    • Field Detail

      • fileMetadataPos

        protected long fileMetadataPos

      • fileMetadataSize

        protected int fileMetadataSize

    • Constructor Detail

      • TsFileSequenceReader

        public TsFileSequenceReader​(String file)
                     throws IOException
        Create a file reader of the given file. The reader will read the tail of the file to get the file metadata size.Then the reader will skip the first TSFileConfig.MAGIC_STRING.getBytes().length + TSFileConfig.NUMBER_VERSION.getBytes().length bytes of the file for preparing reading real data.
        Parameters:
        file - the data file
        Throws:
        IOException - If some I/O error occurs

      • TsFileSequenceReader

        public TsFileSequenceReader​(String file,
                            boolean loadMetadataSize)
                     throws IOException
        construct function for TsFileSequenceReader.
        Parameters:
        file - -given file name
        loadMetadataSize - -whether load meta data size
        Throws:
        IOException

      • TsFileSequenceReader

        public TsFileSequenceReader​(String file,
                            boolean loadMetadata,
                            boolean cacheDeviceMetadata)
                     throws IOException
        Throws:
        IOException

      • TsFileSequenceReader

        public TsFileSequenceReader​(TsFileInput input)
                     throws IOException
        Create a file reader of the given file. The reader will read the tail of the file to get the file metadata size.Then the reader will skip the first TSFileConfig.MAGIC_STRING.getBytes().length + TSFileConfig.NUMBER_VERSION.getBytes().length bytes of the file for preparing reading real data.
        Parameters:
        input - given input
        Throws:
        IOException

      • TsFileSequenceReader

        public TsFileSequenceReader​(TsFileInput input,
                            boolean loadMetadataSize)
                     throws IOException
        construct function for TsFileSequenceReader.
        Parameters:
        input - -given input
        loadMetadataSize - -load meta data size
        Throws:
        IOException

      • TsFileSequenceReader

        public TsFileSequenceReader​(TsFileInput input,
                            long fileMetadataPos,
                            int fileMetadataSize)
        construct function for TsFileSequenceReader.
        Parameters:
        input - the input of a tsfile. The current position should be a marker and then a chunk Header, rather than the magic number
        fileMetadataPos - the position of the file metadata in the TsFileInput from the beginning of the input to the current position
        fileMetadataSize - the byte size of the file metadata in the input

    • Method Detail

      • getFileMetadataPos

        public long getFileMetadataPos()

      • getTsFileMetadataSize

        public int getTsFileMetadataSize()

      • getFileMetadataSize

        public long getFileMetadataSize()
                         throws IOException
        Return the tsfile meta data size of this tsfile.
        Throws:
        IOException

      • getAllMetadataSize

        public long getAllMetadataSize()
                        throws IOException
        Return the whole meta data size of this tsfile, including ChunkMetadata, TimeseriesMetadata and etc.
        Throws:
        IOException

      • readTailMagic

        public String readTailMagic()
                     throws IOException
        this function does not modify the position of the file reader.
        Throws:
        IOException

      • isComplete

        public boolean isComplete()
                   throws IOException
        whether the file is a complete TsFile: only if the head magic and tail magic string exists.
        Throws:
        IOException

      • readHeadMagic

        public String readHeadMagic()
                     throws IOException
        this function does not modify the position of the file reader.
        Throws:
        IOException

      • readVersionNumber

        public byte readVersionNumber()
                       throws IOException
        this function reads version number and checks compatibility of TsFile.
        Throws:
        IOException

      • readBloomFilter

        public BloomFilter readBloomFilter()
                            throws IOException
        this function does not modify the position of the file reader.
        Throws:
        IOException - io error

      • readDeviceMetadata

        public Map<String,​TimeseriesMetadata> readDeviceMetadata​(String device)
                                                        throws IOException
        this function reads measurements and TimeseriesMetaDatas in given device Thread Safe
        Parameters:
        device - name
        Returns:
        the map measurementId -> TimeseriesMetaData in one device
        Throws:
        IOException - io error

      • binarySearchInTimeseriesMetadataList

        protected int binarySearchInTimeseriesMetadataList​(List<TimeseriesMetadata> timeseriesMetadataList,
                                                   String key)

      • getAllDevicesIteratorWithIsAligned

        public TsFileDeviceIterator getAllDevicesIteratorWithIsAligned()
                                                        throws IOException
        Returns:
        an iterator of "device, isAligned" list, in which names of devices are ordered in dictionary order, and isAligned represents whether the device is aligned. Only read devices on one device leaf node each time to save memory.
        Throws:
        IOException

      • getDevicesAndEntriesOfOneLeafNode

        public void getDevicesAndEntriesOfOneLeafNode​(Long startOffset,
                                              Long endOffset,
                                              Queue<Pair<String,​long[]>> measurementNodeOffsetQueue)
                                       throws IOException
        Get devices and first measurement node offset.
        Parameters:
        startOffset - start offset of device leaf node
        endOffset - end offset of device leaf node
        measurementNodeOffsetQueue - device -> first measurement node offset
        Throws:
        IOException

      • readChunkMetadataInDevice

        public Map<String,​List<ChunkMetadata>> readChunkMetadataInDevice​(String device)
                                                                throws IOException
        read all ChunkMetaDatas of given device
        Parameters:
        device - name
        Returns:
        measurement -> ChunkMetadata list
        Throws:
        IOException - io error

      • getAllPaths

        public List<Path> getAllPaths()
                       throws IOException
        this function return all timeseries names
        Returns:
        list of Paths
        Throws:
        IOException - io error

      • getPathsIterator

        public Iterator<List<Path>> getPathsIterator()
                                      throws IOException
        Returns:
        an iterator of timeseries list, in which names of timeseries are ordered in dictionary order
        Throws:
        IOException - io error

      • isAlignedDevice

        public boolean isAlignedDevice​(MetadataIndexNode measurementNode)
        Check whether the deivce is aligned or not.
        Parameters:
        measurementNode - the next measurement layer node of specific device node

      • getTimeseriesMetadataOffsetByDevice

        public Map<String,​Pair<List<IChunkMetadata>,​Pair<Long,​Long>>> getTimeseriesMetadataOffsetByDevice​(MetadataIndexNode measurementNode,
                                                                                                                    Set<String> excludedMeasurementIds,
                                                                                                                    boolean needChunkMetadata)
                                                                                                             throws IOException
        Get the measurements of current device by its first measurement node. Also get the chunk metadata list and timeseries metadata offset.
        Parameters:
        measurementNode - first measurement node of the device
        excludedMeasurementIds - do not deserialize chunk metadatas whose measurementId is in the set. Notice: It only takes effect when the needChunkMetadata parameter is true.
        needChunkMetadata - need to deserialize chunk metadatas or not
        Returns:
        measurement -> chunk metadata list -> timeseries metadata
        Throws:
        IOException

      • getChunkMetadataListByTimeseriesMetadataOffset

        public List<IChunkMetadata> getChunkMetadataListByTimeseriesMetadataOffset​(long startOffset,
                                                                           long endOffset)
                                                                    throws IOException
        Get chunk metadata list by the start offset and end offset of the timeseries metadata.
        Parameters:
        startOffset - the start offset of timeseries metadata
        endOffset - the end offset of timeseries metadata
        Throws:
        IOException

      • getDeviceTimeseriesMetadata

        public void getDeviceTimeseriesMetadata​(List<TimeseriesMetadata> timeseriesMetadataList,
                                        MetadataIndexNode measurementNode,
                                        Set<String> excludedMeasurementIds,
                                        boolean needChunkMetadata)
                                 throws IOException
        Get timeseries metadata under the measurementNode and put them into timeseriesMetadataList. Skip timeseries whose measurementId is in the excludedMeasurementIds.
        Parameters:
        measurementNode - next layer measurement node of specific device leaf node
        excludedMeasurementIds - skip timeseries whose measurementId is in the set
        Throws:
        IOException

      • getMetadataAndEndOffset

        protected Pair<MetadataIndexEntry,​Long> getMetadataAndEndOffset​(MetadataIndexNode metadataIndex,
                                                                      String name,
                                                                      boolean isDeviceLevel,
                                                                      boolean exactSearch)
                                                               throws IOException
        Get target MetadataIndexEntry and its end offset
        Parameters:
        metadataIndex - given MetadataIndexNode
        name - target device / measurement name
        isDeviceLevel - whether target MetadataIndexNode is device level
        exactSearch - whether is in exact search mode, return null when there is no entry with name; or else return the nearest MetadataIndexEntry before it (for deeper search)
        Returns:
        target MetadataIndexEntry, endOffset pair
        Throws:
        IOException

      • readChunkGroupHeader

        public ChunkGroupHeader readChunkGroupHeader()
                                      throws IOException
        read data from current position of the input, and deserialize it to a CHUNK_GROUP_FOOTER.
        This method is not threadsafe.
        Returns:
        a CHUNK_GROUP_FOOTER
        Throws:
        IOException - io error

      • readChunkGroupHeader

        public ChunkGroupHeader readChunkGroupHeader​(long position,
                                             boolean markerRead)
                                      throws IOException
        read data from current position of the input, and deserialize it to a CHUNK_GROUP_FOOTER.
        Parameters:
        position - the offset of the chunk group footer in the file
        markerRead - true if the offset does not contains the marker , otherwise false
        Returns:
        a CHUNK_GROUP_FOOTER
        Throws:
        IOException - io error

      • readChunkHeader

        public ChunkHeader readChunkHeader​(byte chunkType)
                            throws IOException
        read data from current position of the input, and deserialize it to a CHUNK_HEADER.
        This method is not threadsafe.
        Returns:
        a CHUNK_HEADER
        Throws:
        IOException - io error

      • readChunk

        public ByteBuffer readChunk​(long position,
                            int dataSize)
                     throws IOException
        notice, this function will modify channel's position.
        Parameters:
        dataSize - the size of chunkdata
        position - the offset of the chunk data
        Returns:
        the pages of this chunk
        Throws:
        IOException

      • readMarker

        public byte readMarker()
                throws IOException
        read one byte from the input.
        this method is not thread safe
        Throws:
        IOException

      • getFileName

        public String getFileName()

      • readData

        protected ByteBuffer readData​(long position,
                              int totalSize)
                       throws IOException
        read data from tsFileInput, from the current position (if position = -1), or the given position.
        if position = -1, the tsFileInput's position will be changed to the current position + real data size that been read. Other wise, the tsFileInput's position is not changed.
        Parameters:
        position - the start position of data in the tsFileInput, or the current position if position = -1
        totalSize - the size of data that want to read
        Returns:
        data that been read.
        Throws:
        IOException

      • readData

        protected ByteBuffer readData​(long start,
                              long end)
                       throws IOException
        read data from tsFileInput, from the current position (if position = -1), or the given position.
        Parameters:
        start - the start position of data in the tsFileInput, or the current position if position = -1
        end - the end position of data that want to read
        Returns:
        data that been read.
        Throws:
        IOException

      • readRaw

        public int readRaw​(long position,
                   int length,
                   ByteBuffer target)
            throws IOException
        notice, the target bytebuffer are not flipped.
        Throws:
        IOException

      • selfCheck

        public long selfCheck​(Map<Path,​IMeasurementSchema> newSchema,
                      List<ChunkGroupMetadata> chunkGroupMetadataList,
                      boolean fastFinish)
               throws IOException
        Self Check the file and return the position before where the data is safe.
        Parameters:
        newSchema - the schema on each time series in the file
        chunkGroupMetadataList - ChunkGroupMetadata List
        fastFinish - if true and the file is complete, then newSchema and chunkGroupMetadataList parameter will be not modified.
        Returns:
        the position of the file that is fine. All data after the position in the file should be truncated.
        Throws:
        IOException

      • getChunkMetadataList

        public List<ChunkMetadata> getChunkMetadataList​(Path path,
                                                boolean ignoreNotExists)
                                         throws IOException
        get ChunkMetaDatas of given path, and throw exception if path not exists
        Parameters:
        path - timeseries path
        Returns:
        List of ChunkMetaData
        Throws:
        IOException

      • getDeviceNameInRange

        public List<String> getDeviceNameInRange​(long start,
                                         long end)
                                  throws IOException
        get device names which has valid chunks in [start, end)
        Parameters:
        start - start of the partition
        end - end of the partition
        Returns:
        device names in range
        Throws:
        IOException

      • getMetadataIndexNode

        public MetadataIndexNode getMetadataIndexNode​(long startOffset,
                                              long endOffset)
                                       throws IOException
        get metadata index node
        Parameters:
        startOffset - start read offset
        endOffset - end read offset
        Returns:
        MetadataIndexNode
        Throws:
        IOException

      • getMinPlanIndex

        public long getMinPlanIndex()

      • getMaxPlanIndex

        public long getMaxPlanIndex()

      • getMeasurementChunkMetadataListMapIterator

        public Iterator<Map<String,​List<ChunkMetadata>>> getMeasurementChunkMetadataListMapIterator​(String device)
                                                                                           throws IOException
        Returns:
        An iterator of linked hashmaps ( measurement -> chunk metadata list ). When traversing the linked hashmap, you will get chunk metadata lists according to the lexicographic order of the measurements. The first measurement of the linked hashmap of each iteration is always larger than the last measurement of the linked hashmap of the previous iteration in lexicographic order.
        Throws:
        IOException

      • readMetadataIndexNode

        public MetadataIndexNode readMetadataIndexNode​(long start,
                                               long end)
                                        throws IOException
        Read MetadataIndexNode by start and end offset.
        Parameters:
        start - the start offset of the MetadataIndexNode
        end - the end offset of the MetadataIndexNode
        Returns:
        MetadataIndexNode
        Throws:
        IOException - IOException

      • hashCode

        public int hashCode()
        Overrides:
        hashCode in class Object