com.google.cloud.hadoop.gcsio

Class GoogleCloudStorageReadChannel

java.lang.Object

com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, java.nio.channels.ByteChannel, java.nio.channels.Channel, java.nio.channels.ReadableByteChannel, java.nio.channels.SeekableByteChannel, java.nio.channels.WritableByteChannel

Direct Known Subclasses:

InMemoryObjectReadChannel

public class GoogleCloudStorageReadChannel
extends java.lang.Object
implements java.nio.channels.SeekableByteChannel

Provides seekable read access to GCS.

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_BACKOFF_INITIAL_INTERVAL_MILLIS
static int	DEFAULT_BACKOFF_MAX_ELAPSED_TIME_MILLIS
static int	DEFAULT_BACKOFF_MAX_INTERVAL_MILLIS
static double	DEFAULT_BACKOFF_MULTIPLIER
static double	DEFAULT_BACKOFF_RANDOMIZATION_FACTOR

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	GoogleCloudStorageReadChannel() Constructs an instance of GoogleCloudStorageReadChannel.
protected	GoogleCloudStorageReadChannel(GoogleCloudStorageReadOptions readOptions) Constructs an instance of GoogleCloudStorageReadChannel.
	GoogleCloudStorageReadChannel(com.google.api.services.storage.Storage gcs, java.lang.String bucketName, java.lang.String objectName, ApiErrorExtractor errorExtractor, ClientRequestHelper<com.google.api.services.storage.model.StorageObject> requestHelper) Constructs an instance of GoogleCloudStorageReadChannel.
	GoogleCloudStorageReadChannel(com.google.api.services.storage.Storage gcs, java.lang.String bucketName, java.lang.String objectName, ApiErrorExtractor errorExtractor, ClientRequestHelper<com.google.api.services.storage.model.StorageObject> requestHelper, GoogleCloudStorageReadOptions readOptions) Constructs an instance of GoogleCloudStorageReadChannel.

Method Summary

Modifier and Type	Method and Description
void	close() Closes this channel.
protected void	closeReadChannel() Closes the underlying ReadableByteChannel.
protected com.google.api.services.storage.Storage.Objects.Get	createRequest()
protected static com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.FileEncoding	getEncoding(com.google.api.services.storage.model.StorageObject metadata) Returns the FileEncoding of a file given its metadata.
protected com.google.api.services.storage.model.StorageObject	getMetadata() Retrieve the object's metadata from GCS.
boolean	isOpen() Tells whether this channel is open.
protected java.io.InputStream	openStreamAndSetMetadata(long newPosition) Opens the underlying stream, sets its position to the given value and sets size based on stream content size.
long	position() Returns this channel's current position.
java.nio.channels.SeekableByteChannel	position(long newPosition) Sets this channel's position.
int	read(java.nio.ByteBuffer buffer) Reads from this channel and stores read data in the given buffer.
void	setMaxRetries(int maxRetries) Sets the number of times to automatically retry by re-opening the underlying readChannel whenever an exception occurs while reading from it.
protected void	setSize(com.google.api.client.http.HttpResponse response, long offset) Set the size of the content.
protected void	setSize(long size) Sets size of this channel to the given value.
long	size() Returns size of the object to which this channel is connected.
java.nio.channels.SeekableByteChannel	truncate(long size)
protected void	validatePosition(long newPosition) Validates that the given position is valid for this channel.
int	write(java.nio.ByteBuffer src)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_BACKOFF_INITIAL_INTERVAL_MILLIS

public static final int DEFAULT_BACKOFF_INITIAL_INTERVAL_MILLIS

See Also:

Constant Field Values

DEFAULT_BACKOFF_RANDOMIZATION_FACTOR

public static final double DEFAULT_BACKOFF_RANDOMIZATION_FACTOR

See Also:

Constant Field Values

DEFAULT_BACKOFF_MULTIPLIER

public static final double DEFAULT_BACKOFF_MULTIPLIER

See Also:

Constant Field Values

DEFAULT_BACKOFF_MAX_INTERVAL_MILLIS

public static final int DEFAULT_BACKOFF_MAX_INTERVAL_MILLIS

See Also:

Constant Field Values

DEFAULT_BACKOFF_MAX_ELAPSED_TIME_MILLIS

public static final int DEFAULT_BACKOFF_MAX_ELAPSED_TIME_MILLIS

See Also:

Constant Field Values

Constructor Detail

GoogleCloudStorageReadChannel

public GoogleCloudStorageReadChannel(com.google.api.services.storage.Storage gcs,
                                     java.lang.String bucketName,
                                     java.lang.String objectName,
                                     ApiErrorExtractor errorExtractor,
                                     ClientRequestHelper<com.google.api.services.storage.model.StorageObject> requestHelper)
                              throws java.io.IOException

Constructs an instance of GoogleCloudStorageReadChannel.

Parameters:

gcs - storage object instance

bucketName - name of the bucket containing the object to read

objectName - name of the object to read

requestHelper - a ClientRequestHelper used to set any extra headers

Throws:

java.io.FileNotFoundException - if the given object does not exist

java.io.IOException - on IO error

GoogleCloudStorageReadChannel

public GoogleCloudStorageReadChannel(com.google.api.services.storage.Storage gcs,
                                     java.lang.String bucketName,
                                     java.lang.String objectName,
                                     ApiErrorExtractor errorExtractor,
                                     ClientRequestHelper<com.google.api.services.storage.model.StorageObject> requestHelper,
                                     GoogleCloudStorageReadOptions readOptions)
                              throws java.io.IOException

Constructs an instance of GoogleCloudStorageReadChannel.

Parameters:

gcs - storage object instance

bucketName - name of the bucket containing the object to read

objectName - name of the object to read

requestHelper - a ClientRequestHelper used to set any extra headers

readOptions - fine-grained options specifying things like retry settings, buffering, etc.

Throws:

java.io.FileNotFoundException - if the given object does not exist

java.io.IOException - on IO error

GoogleCloudStorageReadChannel

protected GoogleCloudStorageReadChannel()
                                 throws java.io.IOException

Constructs an instance of GoogleCloudStorageReadChannel. Used for unit testing only. Do not use elsewhere.

Throws:

java.io.IOException - on IO error

GoogleCloudStorageReadChannel

protected GoogleCloudStorageReadChannel(GoogleCloudStorageReadOptions readOptions)
                                 throws java.io.IOException

Constructs an instance of GoogleCloudStorageReadChannel. Used for unit testing only. Do not use elsewhere.

Throws:

java.io.IOException - on IO error

Method Detail

setMaxRetries

public void setMaxRetries(int maxRetries)

Sets the number of times to automatically retry by re-opening the underlying readChannel whenever an exception occurs while reading from it. The count of attempted retries is reset whenever at least one byte is successfully read, so this number of retries refers to retries made without achieving any forward progress.

read

public int read(java.nio.ByteBuffer buffer)
         throws java.io.IOException

Reads from this channel and stores read data in the given buffer.

On unexpected failure, will attempt to close the channel and clean up state.

Specified by:

read in interface java.nio.channels.ReadableByteChannel

Specified by:

read in interface java.nio.channels.SeekableByteChannel

Parameters:

buffer - buffer to read data into

Returns:

number of bytes read or -1 on end-of-stream

Throws:

java.io.IOException - on IO error

truncate

public java.nio.channels.SeekableByteChannel truncate(long size)
                                               throws java.io.IOException

Specified by:

truncate in interface java.nio.channels.SeekableByteChannel

Throws:

java.io.IOException

write

public int write(java.nio.ByteBuffer src)
          throws java.io.IOException

Specified by:

write in interface java.nio.channels.SeekableByteChannel

Specified by:

write in interface java.nio.channels.WritableByteChannel

Throws:

java.io.IOException

isOpen

public boolean isOpen()

Tells whether this channel is open.

Specified by:

isOpen in interface java.nio.channels.Channel

Returns:

a value indicating whether this channel is open

closeReadChannel

protected void closeReadChannel()

Closes the underlying ReadableByteChannel.

Catches and ignores all exceptions as there is not a lot the user can do to fix errors here and a new connection will be needed. Especially SSLExceptions since the there's a high probability that SSL connections would be broken in a way that causes Channel.close() itself to throw an exception, even though underlying sockets have already been cleaned up; close() on an SSLSocketImpl requires a shutdown handshake in order to shutdown cleanly, and if the connection has been broken already, then this is not possible, and the SSLSocketImpl was already responsible for performing local cleanup at the time the exception was raised.

close

public void close()
           throws java.io.IOException

Closes this channel.

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface java.nio.channels.Channel

Throws:

java.io.IOException - on IO error

position

public long position()
              throws java.io.IOException

Returns this channel's current position.

Specified by:

position in interface java.nio.channels.SeekableByteChannel

Returns:

this channel's current position

Throws:

java.io.IOException

position

public java.nio.channels.SeekableByteChannel position(long newPosition)
                                               throws java.io.IOException

Sets this channel's position.

Specified by:

position in interface java.nio.channels.SeekableByteChannel

Parameters:

newPosition - the new position, counting the number of bytes from the beginning.

Returns:

this channel instance

Throws:

java.io.FileNotFoundException - if the underlying object does not exist.

java.io.IOException - on IO error

size

public long size()
          throws java.io.IOException

Returns size of the object to which this channel is connected.

Specified by:

size in interface java.nio.channels.SeekableByteChannel

Returns:

size of the object to which this channel is connected

Throws:

java.io.IOException - on IO error

setSize

protected void setSize(long size)

Sets size of this channel to the given value.

validatePosition

protected void validatePosition(long newPosition)
                         throws java.io.IOException

Validates that the given position is valid for this channel.

Throws:

java.io.IOException

getMetadata

protected com.google.api.services.storage.model.StorageObject getMetadata()
                                                                   throws java.io.IOException

Retrieve the object's metadata from GCS.

Throws:

java.io.IOException - on IO error.

getEncoding

protected static com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.FileEncoding getEncoding(com.google.api.services.storage.model.StorageObject metadata)

Returns the FileEncoding of a file given its metadata. Currently supports GZIPPED and OTHER.

Parameters:

metadata - the object's metadata.

Returns:

FileEncoding.GZIPPED if the response from GCS will have gzip encoding or FileEncoding.OTHER otherwise.

setSize

protected void setSize(com.google.api.client.http.HttpResponse response,
                       long offset)
                throws java.io.IOException

Set the size of the content.

First, we examine the Content-Length header. If neither exists, we then look for and parse the Content-Range header. If there is no way to determine the content length, an exception is thrown. If the Content-Length header is present, then the offset is added to this value (i.e., offset is the number of bytes that were not requested).

Parameters:

response - response to parse.

offset - the number of bytes that were not requested.

Throws:

java.io.IOException - on IO error.

openStreamAndSetMetadata

protected java.io.InputStream openStreamAndSetMetadata(long newPosition)
                                                throws java.io.IOException

Opens the underlying stream, sets its position to the given value and sets size based on stream content size.

If the file encoding in GCS is gzip (and therefore the HTTP client will attempt to decompress it), the entire file is always requested and we seek to the position requested. If the file encoding is not gzip, only the remaining bytes to be read are requested from GCS.

Parameters:

newPosition - position to seek into the new stream.

Throws:

java.io.IOException - on IO error

createRequest

protected com.google.api.services.storage.Storage.Objects.Get createRequest()
                                                                     throws java.io.IOException

Throws:

java.io.IOException