Package org.apache.iotdb.tsfile.write.writer

Class TsFileIOWriter

java.lang.Object

org.apache.iotdb.tsfile.write.writer.TsFileIOWriter

All Implemented Interfaces:

AutoCloseable

Direct Known Subclasses:

ForceAppendTsFileWriter, RestorableTsFileIOWriter

public class TsFileIOWriter
extends Object
implements AutoCloseable

TsFileIOWriter is used to construct metadata and write data stored in memory to output stream.

Field Summary

Fields

Modifier and Type	Field	Description
protected boolean	canWrite
static String	CHUNK_METADATA_TEMP_FILE_SUFFIX
protected List<ChunkGroupMetadata>	chunkGroupMetadataList
protected List<ChunkMetadata>	chunkMetadataList
protected File	chunkMetadataTempFile
protected static TSFileConfig	config
protected ChunkMetadata	currentChunkMetadata
protected long	currentChunkMetadataSize
protected boolean	enableMemoryControl
protected LinkedList<Long>	endPosInCMTForDevice
protected File	file
protected boolean	hasChunkMetadataInDisk
protected static byte[]	MAGIC_STRING_BYTES
protected long	maxMetadataSize
protected TsFileOutput	out
protected int	pathCount
protected LocalTsFileOutput	tempOutput
static byte	VERSION_NUMBER_BYTE

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	TsFileIOWriter()	empty construct function.
	TsFileIOWriter(File file)	for writing a new tsfile.
	TsFileIOWriter(File file, boolean enableMemoryControl, long maxMetadataSize)	for write with memory control
	TsFileIOWriter(TsFileOutput output)	for writing a new tsfile.
	TsFileIOWriter(TsFileOutput output, boolean test)	for test only

Method Summary

Modifier and Type	Method	Description
boolean	canWrite()
int	checkMetadataSizeAndMayFlush()	Check if the size of chunk metadata in memory is greater than the given threshold.
void	close()	close the outputStream or file channel without writing FileMetadata.
void	endChunkGroup()	end chunk and write some log.
void	endCurrentChunk()	end chunk and write some log.
void	endFile()	write TSFileMetaData to output stream and close it.
void	filterChunks(Map<Path,List<Long>> chunkStartTimes)	Remove such ChunkMetadata that its startTime is not in chunkStartTimes
void	flush()
List<ChunkGroupMetadata>	getChunkGroupMetadataList()
List<ChunkMetadata>	getChunkMetadataListOfCurrentDeviceInMemory()	This method should be called before flushing chunk group metadata list, otherwise, it will return null.
String	getCurrentChunkGroupDeviceId()
Map<String,List<ChunkMetadata>>	getDeviceChunkMetadataMap()
Map<String,List<TimeseriesMetadata>>	getDeviceTimeseriesMetadataMap()	this function is for Upgrade Tool and Split Tool.
File	getFile()
TsFileOutput	getIOWriterOut()	this function is only for Test.
long	getMaxPlanIndex()
long	getMinPlanIndex()
long	getPos()	get the length of normal OutputStream.
TsFileOutput	getTsFileOutput()
boolean	isWritingChunkGroup()	For TsFileReWriteTool / UpgradeTool.
void	mark()
void	reset()
void	setFile(File file)
void	setMaxPlanIndex(long maxPlanIndex)
void	setMinPlanIndex(long minPlanIndex)
protected int	sortAndFlushChunkMetadata()	Sort the chunk metadata by the lexicographical order and the start time of the chunk, then flush them to a temp file.
int	startChunkGroup(String deviceId)
protected void	startFile()
void	startFlushChunk(String measurementId, CompressionType compressionCodecName, TSDataType tsDataType, TSEncoding encodingType, Statistics<? extends Serializable> statistics, int dataSize, int numOfPages, int mask)	start a ChunkMetaData.
void	truncate(long offset)
void	writeBytesToStream(PublicBAOS bytes)	Writes given bytes to output stream.
void	writeChunk(Chunk chunk)
void	writeChunk(Chunk chunk, ChunkMetadata chunkMetadata)	Write a whole chunk in another file into this file.
void	writeEmptyValueChunk(String measurementId, CompressionType compressionType, TSDataType tsDataType, TSEncoding encodingType, Statistics<? extends Serializable> statistics)	Write an empty value chunk into file directly.
void	writePlanIndices()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAGIC_STRING_BYTES

protected static final byte[] MAGIC_STRING_BYTES

VERSION_NUMBER_BYTE

public static final byte VERSION_NUMBER_BYTE

config

protected static final TSFileConfig config

out

protected TsFileOutput out

canWrite

protected boolean canWrite

file

protected File file

currentChunkMetadata

protected ChunkMetadata currentChunkMetadata

chunkMetadataList

protected List<ChunkMetadata> chunkMetadataList

chunkGroupMetadataList

protected List<ChunkGroupMetadata> chunkGroupMetadataList

maxMetadataSize

protected long maxMetadataSize

currentChunkMetadataSize

protected long currentChunkMetadataSize

chunkMetadataTempFile

protected File chunkMetadataTempFile

tempOutput

protected LocalTsFileOutput tempOutput

hasChunkMetadataInDisk

protected volatile boolean hasChunkMetadataInDisk

pathCount

protected int pathCount

enableMemoryControl

protected boolean enableMemoryControl

endPosInCMTForDevice

protected LinkedList<Long> endPosInCMTForDevice

CHUNK_METADATA_TEMP_FILE_SUFFIX

public static final String CHUNK_METADATA_TEMP_FILE_SUFFIX

See Also:

Constant Field Values

Constructor Detail

TsFileIOWriter

protected TsFileIOWriter()

empty construct function.

TsFileIOWriter

public TsFileIOWriter(File file)
               throws IOException

for writing a new tsfile.

Parameters:

file - be used to output written data

Throws:

IOException - if I/O error occurs

TsFileIOWriter

public TsFileIOWriter(TsFileOutput output)
               throws IOException

for writing a new tsfile.

Parameters:

output - be used to output written data

Throws:

IOException

TsFileIOWriter

public TsFileIOWriter(TsFileOutput output,
                      boolean test)

for test only

TsFileIOWriter

public TsFileIOWriter(File file,
                      boolean enableMemoryControl,
                      long maxMetadataSize)
               throws IOException

for write with memory control

Throws:

IOException

Method Detail

writeBytesToStream

public void writeBytesToStream(PublicBAOS bytes)
                        throws IOException

Writes given bytes to output stream. This method is called when total memory size exceeds the chunk group size threshold.

Parameters:

bytes - - data of several pages which has been packed

Throws:

IOException - if an I/O error occurs.

startFile

protected void startFile()
                  throws IOException

Throws:

IOException

startChunkGroup

public int startChunkGroup(String deviceId)
                    throws IOException

Throws:

IOException

endChunkGroup

public void endChunkGroup()
                   throws IOException

end chunk and write some log. If there is no data in the chunk group, nothing will be flushed.

Throws:

IOException

isWritingChunkGroup

public boolean isWritingChunkGroup()

For TsFileReWriteTool / UpgradeTool. Use this method to determine if needs to start a ChunkGroup.

Returns:

isWritingChunkGroup

startFlushChunk

public void startFlushChunk(String measurementId,
                            CompressionType compressionCodecName,
                            TSDataType tsDataType,
                            TSEncoding encodingType,
                            Statistics<? extends Serializable> statistics,
                            int dataSize,
                            int numOfPages,
                            int mask)
                     throws IOException

start a ChunkMetaData.

Parameters:

measurementId - - measurementId of this time series

compressionCodecName - - compression name of this time series

tsDataType - - data type

statistics - - Chunk statistics

dataSize - - the serialized size of all pages

mask - - 0x80 for time chunk, 0x40 for value chunk, 0x00 for common chunk

Throws:

IOException - if I/O error occurs

writeChunk

public void writeChunk(Chunk chunk,
                       ChunkMetadata chunkMetadata)
                throws IOException

Write a whole chunk in another file into this file. Providing fast merge for IoTDB.

Throws:

IOException

writeEmptyValueChunk

public void writeEmptyValueChunk(String measurementId,
                                 CompressionType compressionType,
                                 TSDataType tsDataType,
                                 TSEncoding encodingType,
                                 Statistics<? extends Serializable> statistics)
                          throws IOException

Write an empty value chunk into file directly. Only used for aligned timeseries.

Throws:

IOException

writeChunk

public void writeChunk(Chunk chunk)
                throws IOException

Throws:

IOException

endCurrentChunk

public void endCurrentChunk()

end chunk and write some log.

endFile

public void endFile()
             throws IOException

write TSFileMetaData to output stream and close it.

Throws:

IOException - if I/O error occurs

getPos

public long getPos()
            throws IOException

get the length of normal OutputStream.

Returns:

- length of normal OutputStream

Throws:

IOException - if I/O error occurs

getDeviceChunkMetadataMap

public Map<String,List<ChunkMetadata>> getDeviceChunkMetadataMap()

canWrite

public boolean canWrite()

mark

public void mark()
          throws IOException

Throws:

IOException

reset

public void reset()
           throws IOException

Throws:

IOException

close

public void close()
           throws IOException

close the outputStream or file channel without writing FileMetadata. This is just used for Testing.

Specified by:

close in interface AutoCloseable

Throws:

IOException

getFile

public File getFile()

setFile

public void setFile(File file)

filterChunks

public void filterChunks(Map<Path,List<Long>> chunkStartTimes)

Remove such ChunkMetadata that its startTime is not in chunkStartTimes

writePlanIndices

public void writePlanIndices()
                      throws IOException

Throws:

IOException

truncate

public void truncate(long offset)
              throws IOException

Throws:

IOException

getIOWriterOut

public TsFileOutput getIOWriterOut()

this function is only for Test.

Returns:

TsFileOutput

getChunkMetadataListOfCurrentDeviceInMemory

public List<ChunkMetadata> getChunkMetadataListOfCurrentDeviceInMemory()

This method should be called before flushing chunk group metadata list, otherwise, it will return null.

getDeviceTimeseriesMetadataMap

public Map<String,List<TimeseriesMetadata>> getDeviceTimeseriesMetadataMap()

this function is for Upgrade Tool and Split Tool.

Returns:

DeviceTimeseriesMetadataMap

getMinPlanIndex

public long getMinPlanIndex()

setMinPlanIndex

public void setMinPlanIndex(long minPlanIndex)

getMaxPlanIndex

public long getMaxPlanIndex()

setMaxPlanIndex

public void setMaxPlanIndex(long maxPlanIndex)

checkMetadataSizeAndMayFlush

public int checkMetadataSizeAndMayFlush()
                                 throws IOException

Check if the size of chunk metadata in memory is greater than the given threshold. If so, the chunk metadata will be written to a temp files. Notice! If you are writing a aligned device in row, you should make sure all data of current writing device has been written before this method is called. For writing not aligned series or writing aligned series in column, you should make sure that all data of one series is written before you call this function.

Throws:

IOException

sortAndFlushChunkMetadata

protected int sortAndFlushChunkMetadata()
                                 throws IOException

Sort the chunk metadata by the lexicographical order and the start time of the chunk, then flush them to a temp file.

Throws:

IOException

getCurrentChunkGroupDeviceId

public String getCurrentChunkGroupDeviceId()

getChunkGroupMetadataList

public List<ChunkGroupMetadata> getChunkGroupMetadataList()

flush

public void flush()
           throws IOException

Throws:

IOException

getTsFileOutput

public TsFileOutput getTsFileOutput()