Package org.apache.druid.java.util.common.io.smoosh

Class FileSmoosher

java.lang.Object

org.apache.druid.java.util.common.io.smoosh.FileSmoosher

All Implemented Interfaces:

Closeable, AutoCloseable, SegmentFileBuilder

public class FileSmoosher
extends Object
implements SegmentFileBuilder

A class that concatenates files together into configurable sized chunks, works in conjunction with the SmooshedFileMapper to provide access to the individual files.

It does not split input files among separate output files, instead the various "chunk" files will be varying sizes and it is not possible to add a file of size greater than Integer.MAX_VALUE.

This class is not thread safe.

This class allows writing multiple files even if main smoosh file writer is open. If main smoosh file writer is already open, it delegates the write into temporary file on the file system which is later copied on to the main smoosh file and underlying temporary file will be cleaned up.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	FileSmoosher.Outer

Constructor Summary

Constructors

Constructor	Description
FileSmoosher(File baseDir)
FileSmoosher(File baseDir, int maxChunkSize)
FileSmoosher(File baseDir, int maxChunkSize, String outputFileName)

Method Summary

Modifier and Type	Method	Description
void	abort()	cleanup any open resources in the event of an exception while building the segment files
void	add(String name, File fileToAdd)	Add a File to the segment file as the specified name
void	add(String name, ByteBuffer bufferToAdd)	Add a ByteBuffer to a segment file as the specified name
void	addColumn(String name, ColumnDescriptor columnDescriptor)	Add a column to the metadata of this segment file
SegmentFileChannel	addWithChannel(String name, long size)	Creates a SegmentFileChannel to write data to the segment file as the specified name.
SmooshedWriter	addWithSmooshedWriter(String name, long size)
void	close()	Close the segment file builder, writing the file(s) to the destination
List<SegmentFileContainerMetadata>	getContainers()
Map<String,SegmentInternalFileMetadata>	getInternalFiles()
List<File>	getOutFiles()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.druid.segment.file.SegmentFileBuilder

getExternalBuilder

Constructor Details

FileSmoosher

public FileSmoosher(File baseDir)

FileSmoosher

public FileSmoosher(File baseDir,
 int maxChunkSize)

FileSmoosher

public FileSmoosher(File baseDir,
 int maxChunkSize,
 @Nullable
 String outputFileName)

Method Details

getOutFiles

public List<File> getOutFiles()

getContainers

public List<SegmentFileContainerMetadata> getContainers()

getInternalFiles

public Map<String,SegmentInternalFileMetadata> getInternalFiles()

addColumn

public void addColumn(String name,
 ColumnDescriptor columnDescriptor)

Description copied from interface: SegmentFileBuilder

Add a column to the metadata of this segment file

Specified by:

addColumn in interface SegmentFileBuilder

add

public void add(String name,
 File fileToAdd)
         throws IOException

Description copied from interface: SegmentFileBuilder

Add a File to the segment file as the specified name

Specified by:

add in interface SegmentFileBuilder

Throws:

IOException

add

public void add(String name,
 ByteBuffer bufferToAdd)
         throws IOException

Description copied from interface: SegmentFileBuilder

Add a ByteBuffer to a segment file as the specified name

Specified by:

add in interface SegmentFileBuilder

Throws:

IOException

addWithChannel

public SegmentFileChannel addWithChannel(String name,
 long size)
                                  throws IOException

Description copied from interface: SegmentFileBuilder

Creates a SegmentFileChannel to write data to the segment file as the specified name. Callers must be sure to write the amount of data specified by the size parameter.

Specified by:

addWithChannel in interface SegmentFileBuilder

Throws:

IOException

abort

public void abort()

Description copied from interface: SegmentFileBuilder

cleanup any open resources in the event of an exception while building the segment files

Specified by:

abort in interface SegmentFileBuilder

addWithSmooshedWriter

public SmooshedWriter addWithSmooshedWriter(String name,
 long size)
                                     throws IOException

Throws:

IOException

close

public void close()
           throws IOException

Description copied from interface: SegmentFileBuilder

Close the segment file builder, writing the file(s) to the destination

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Specified by:

close in interface SegmentFileBuilder

Throws:

IOException