org.apache.gobblin.util.io

Class StreamUtils

java.lang.Object

org.apache.gobblin.util.io.StreamUtils

public class StreamUtils
extends Object

Utility class of input/output stream helpers.

Constructor Summary

Constructors

Constructor and Description
StreamUtils()

Method Summary

Modifier and Type	Method and Description
static void	byteBufferToOutputStream(ByteBuffer in, OutputStream out) Reads the full contents of a ByteBuffer and writes them to an OutputStream.
static org.apache.hadoop.fs.FSDataInputStream	convertStream(InputStream in) Convert an instance of InputStream to a FSDataInputStream that is Seekable and PositionedReadable.
static long	copy(InputStream is, OutputStream os) Copies an InputStream to and OutputStream using Channels.
static long	copy(ReadableByteChannel inputChannel, WritableByteChannel outputChannel) Copies a ReadableByteChannel to a WritableByteChannel.
static boolean	isCompressed(byte[] bytes)
static void	tar(org.apache.hadoop.fs.FileSystem sourceFs, org.apache.hadoop.fs.FileSystem destFs, org.apache.hadoop.fs.Path sourcePath, org.apache.hadoop.fs.Path destPath) Similiar to tar(FileSystem, Path, Path) except the source and destination FileSystem can be different.
static void	tar(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path sourcePath, org.apache.hadoop.fs.Path destPath) Creates a tar gzip file using a given Path as input and a given Path as a destination.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StreamUtils

public StreamUtils()

Method Detail

convertStream

public static org.apache.hadoop.fs.FSDataInputStream convertStream(InputStream in)
                                                            throws IOException

Convert an instance of InputStream to a FSDataInputStream that is Seekable and PositionedReadable.

Throws:

IOException

See Also:

SeekableFSInputStream

copy

public static long copy(InputStream is,
                        OutputStream os)
                 throws IOException

Copies an InputStream to and OutputStream using Channels.

Note: The method does not close the InputStream and OutputStream. However, the ReadableByteChannel and WritableByteChannels are closed

Returns:

Total bytes copied

Throws:

IOException

copy

public static long copy(ReadableByteChannel inputChannel,
                        WritableByteChannel outputChannel)
                 throws IOException

Copies a ReadableByteChannel to a WritableByteChannel.

Note: The ReadableByteChannel and WritableByteChannels are NOT closed by the method

Returns:

Total bytes copied

Throws:

IOException

tar

public static void tar(org.apache.hadoop.fs.FileSystem fs,
                       org.apache.hadoop.fs.Path sourcePath,
                       org.apache.hadoop.fs.Path destPath)
                throws IOException

Creates a tar gzip file using a given Path as input and a given Path as a destination. If the given input is a file then only that file will be added to tarball. If it is a directory then the entire directory will be recursively put into the tarball.

Parameters:

fs - the FileSystem the input exists, and the the output should be written to.

sourcePath - the Path of the input files, this can either be a file or a directory.

destPath - the Path that tarball should be written to.

Throws:

IOException

tar

public static void tar(org.apache.hadoop.fs.FileSystem sourceFs,
                       org.apache.hadoop.fs.FileSystem destFs,
                       org.apache.hadoop.fs.Path sourcePath,
                       org.apache.hadoop.fs.Path destPath)
                throws IOException

Similiar to tar(FileSystem, Path, Path) except the source and destination FileSystem can be different.

Throws:

IOException

See Also:

tar(FileSystem, Path, Path)

isCompressed

public static boolean isCompressed(byte[] bytes)

byteBufferToOutputStream

public static void byteBufferToOutputStream(ByteBuffer in,
                                            OutputStream out)
                                     throws IOException

Reads the full contents of a ByteBuffer and writes them to an OutputStream. The ByteBuffer is consumed by this operation; eg in.remaining() will be 0 after it completes successfully.

Parameters:

in - ByteBuffer to write into the OutputStream

out - Destination stream

Throws:

IOException - If there is an error writing into the OutputStream