org.apache.flink.runtime.blob

Class BlobServer

java.lang.Object

java.lang.Thread

org.apache.flink.runtime.blob.BlobServer

All Implemented Interfaces:

Closeable, AutoCloseable, Runnable, BlobService, BlobWriter, PermanentBlobService, TransientBlobService, GloballyCleanableResource, LocallyCleanableResource

public class BlobServer
extends Thread
implements BlobService, BlobWriter, PermanentBlobService, TransientBlobService, LocallyCleanableResource, GloballyCleanableResource

This class implements the BLOB server. The BLOB server is responsible for listening for incoming requests and spawning threads to handle these requests. Furthermore, it takes care of creating the directory structure to store the BLOBs or temporarily cache them.

Nested Class Summary

Nested classes/interfaces inherited from class java.lang.Thread

Thread.State, Thread.UncaughtExceptionHandler

Field Summary

Fields inherited from class java.lang.Thread

MAX_PRIORITY, MIN_PRIORITY, NORM_PRIORITY

Constructor Summary

Constructors

Constructor and Description
BlobServer(org.apache.flink.configuration.Configuration config, File storageDir, BlobStore blobStore)
BlobServer(org.apache.flink.configuration.Configuration config, org.apache.flink.util.Reference<File> storageDir, BlobStore blobStore) Instantiates a new BLOB server and binds it to a free network port.

Method Summary

Modifier and Type	Method and Description
void	close() Shuts down the BLOB server.
protected BlobClient	createClient()
boolean	deleteFromCache(org.apache.flink.api.common.JobID jobId, TransientBlobKey key) Deletes the file associated with the blob key in the local storage of the blob server.
boolean	deleteFromCache(TransientBlobKey key) Deletes the (job-unrelated) file associated with the blob key in the local storage of the blob server.
boolean	deletePermanent(org.apache.flink.api.common.JobID jobId, PermanentBlobKey key) Delete the uploaded data with the given JobID and PermanentBlobKey.
File	getFile(org.apache.flink.api.common.JobID jobId, PermanentBlobKey key) Returns the path to a local copy of the file associated with the provided job ID and blob key.
File	getFile(org.apache.flink.api.common.JobID jobId, TransientBlobKey key) Retrieves the local path of a file associated with a job and a blob key.
File	getFile(TransientBlobKey key) Retrieves the local path of a (job-unrelated) file associated with a job and a blob key.
int	getMinOffloadingSize() Returns the configuration used by the BLOB server.
PermanentBlobService	getPermanentBlobService() Returns a BLOB service for accessing permanent BLOBs.
int	getPort() Returns the port on which the server is listening.
File	getStorageDir()
File	getStorageLocation(org.apache.flink.api.common.JobID jobId, BlobKey key) Returns a file handle to the file associated with the given blob key on the blob server.
TransientBlobService	getTransientBlobService() Returns a BLOB service for accessing transient BLOBs.
CompletableFuture<Void>	globalCleanupAsync(org.apache.flink.api.common.JobID jobId, Executor executor) Removes all BLOBs from local and HA store belonging to the given JobID.
boolean	isShutdown() Tests whether the BLOB server has been requested to shut down.
CompletableFuture<Void>	localCleanupAsync(org.apache.flink.api.common.JobID jobId, Executor cleanupExecutor) Deletes locally stored artifacts for the job represented by the given JobID.
PermanentBlobKey	putPermanent(org.apache.flink.api.common.JobID jobId, byte[] value) Uploads the data of the given byte array for the given job to the BLOB server and makes it a permanent BLOB.
PermanentBlobKey	putPermanent(org.apache.flink.api.common.JobID jobId, InputStream inputStream) Uploads the data from the given input stream for the given job to the BLOB server and makes it a permanent BLOB.
TransientBlobKey	putTransient(byte[] value) Uploads the (job-unrelated) data of the given byte array to the BLOB server.
TransientBlobKey	putTransient(InputStream inputStream) Uploads the (job-unrelated) data from the given input stream to the BLOB server.
TransientBlobKey	putTransient(org.apache.flink.api.common.JobID jobId, byte[] value) Uploads the data of the given byte array for the given job to the BLOB server.
TransientBlobKey	putTransient(org.apache.flink.api.common.JobID jobId, InputStream inputStream) Uploads the data from the given input stream for the given job to the BLOB server.
void	retainJobs(Collection<org.apache.flink.api.common.JobID> jobsToRetain, Executor ioExecutor)
void	run()

Methods inherited from class java.lang.Thread

activeCount, checkAccess, clone, countStackFrames, currentThread, destroy, dumpStack, enumerate, getAllStackTraces, getContextClassLoader, getDefaultUncaughtExceptionHandler, getId, getName, getPriority, getStackTrace, getState, getThreadGroup, getUncaughtExceptionHandler, holdsLock, interrupt, interrupted, isAlive, isDaemon, isInterrupted, join, join, join, resume, setContextClassLoader, setDaemon, setDefaultUncaughtExceptionHandler, setName, setPriority, setUncaughtExceptionHandler, sleep, sleep, start, stop, stop, suspend, toString, yield

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.blob.BlobWriter

offloadWithException, serializeAndTryOffload, tryOffload

Methods inherited from interface org.apache.flink.runtime.blob.PermanentBlobService

readFile

Constructor Detail

BlobServer

@VisibleForTesting
public BlobServer(org.apache.flink.configuration.Configuration config,
                                     File storageDir,
                                     BlobStore blobStore)
                              throws IOException

Throws:

IOException

BlobServer

public BlobServer(org.apache.flink.configuration.Configuration config,
                  org.apache.flink.util.Reference<File> storageDir,
                  BlobStore blobStore)
           throws IOException

Instantiates a new BLOB server and binds it to a free network port.

Parameters:

config - Configuration to be used to instantiate the BlobServer

storageDir - storage directory for the blobs

blobStore - BlobStore to store blobs persistently

Throws:

IOException - thrown if the BLOB server cannot bind to a free network port or if the (local or distributed) file storage cannot be created or is not usable

Method Detail

getStorageDir

public File getStorageDir()

getStorageLocation

@VisibleForTesting
public File getStorageLocation(@Nullable
                                                  org.apache.flink.api.common.JobID jobId,
                                                  BlobKey key)
                                           throws IOException

Returns a file handle to the file associated with the given blob key on the blob server.

This is only called from BlobServerConnection or unit tests.

Parameters:

jobId - ID of the job this blob belongs to (or null if job-unrelated)

key - identifying the file

Returns:

file handle to the file

Throws:

IOException - if creating the directory fails

run

public void run()

Specified by:

run in interface Runnable

Overrides:

run in class Thread

close

public void close()
           throws IOException

Shuts down the BLOB server.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

createClient

protected BlobClient createClient()
                           throws IOException

Throws:

IOException

getFile

public File getFile(TransientBlobKey key)
             throws IOException

Retrieves the local path of a (job-unrelated) file associated with a job and a blob key.

The blob server looks the blob key up in its local storage. If the file exists, it is returned. If the file does not exist, it is retrieved from the HA blob store (if available) or a FileNotFoundException is thrown.

Specified by:

getFile in interface TransientBlobService

Parameters:

key - blob key associated with the requested file

Returns:

file referring to the local storage location of the BLOB

Throws:

IOException - Thrown if the file retrieval failed.

getFile

public File getFile(org.apache.flink.api.common.JobID jobId,
                    TransientBlobKey key)
             throws IOException

Retrieves the local path of a file associated with a job and a blob key.

The blob server looks the blob key up in its local storage. If the file exists, it is returned. If the file does not exist, it is retrieved from the HA blob store (if available) or a FileNotFoundException is thrown.

Specified by:

getFile in interface TransientBlobService

Parameters:

jobId - ID of the job this blob belongs to

key - blob key associated with the requested file

Returns:

file referring to the local storage location of the BLOB

Throws:

IOException - Thrown if the file retrieval failed.

getFile

public File getFile(org.apache.flink.api.common.JobID jobId,
                    PermanentBlobKey key)
             throws IOException

Returns the path to a local copy of the file associated with the provided job ID and blob key.

We will first attempt to serve the BLOB from the local storage. If the BLOB is not in there, we will try to download it from the HA store.

Specified by:

getFile in interface PermanentBlobService

Parameters:

jobId - ID of the job this blob belongs to

key - blob key associated with the requested file

Returns:

The path to the file.

Throws:

FileNotFoundException - if the BLOB does not exist;

IOException - if any other error occurs when retrieving the file

putTransient

public TransientBlobKey putTransient(byte[] value)
                              throws IOException

Description copied from interface: TransientBlobService

Uploads the (job-unrelated) data of the given byte array to the BLOB server.

Specified by:

putTransient in interface TransientBlobService

Parameters:

value - the buffer to upload

Returns:

the computed BLOB key identifying the BLOB on the server

Throws:

IOException - thrown if an I/O error occurs while uploading the data to the BLOB server

putTransient

public TransientBlobKey putTransient(org.apache.flink.api.common.JobID jobId,
                                     byte[] value)
                              throws IOException

Description copied from interface: TransientBlobService

Uploads the data of the given byte array for the given job to the BLOB server.

Specified by:

putTransient in interface TransientBlobService

Parameters:

jobId - the ID of the job the BLOB belongs to

value - the buffer to upload

Returns:

the computed BLOB key identifying the BLOB on the server

Throws:

IOException - thrown if an I/O error occurs while uploading the data to the BLOB server

putTransient

public TransientBlobKey putTransient(InputStream inputStream)
                              throws IOException

Description copied from interface: TransientBlobService

Uploads the (job-unrelated) data from the given input stream to the BLOB server.

Specified by:

putTransient in interface TransientBlobService

Parameters:

inputStream - the input stream to read the data from

Returns:

the computed BLOB key identifying the BLOB on the server

Throws:

IOException - thrown if an I/O error occurs while reading the data from the input stream or uploading the data to the BLOB server

putTransient

public TransientBlobKey putTransient(org.apache.flink.api.common.JobID jobId,
                                     InputStream inputStream)
                              throws IOException

Description copied from interface: TransientBlobService

Uploads the data from the given input stream for the given job to the BLOB server.

Specified by:

putTransient in interface TransientBlobService

Parameters:

jobId - ID of the job this blob belongs to

inputStream - the input stream to read the data from

Returns:

the computed BLOB key identifying the BLOB on the server

Throws:

IOException - thrown if an I/O error occurs while reading the data from the input stream or uploading the data to the BLOB server

putPermanent

public PermanentBlobKey putPermanent(org.apache.flink.api.common.JobID jobId,
                                     byte[] value)
                              throws IOException

Description copied from interface: BlobWriter

Uploads the data of the given byte array for the given job to the BLOB server and makes it a permanent BLOB.

Specified by:

putPermanent in interface BlobWriter

Parameters:

jobId - the ID of the job the BLOB belongs to

value - the buffer to upload

Returns:

the computed BLOB key identifying the BLOB on the server

Throws:

IOException - thrown if an I/O error occurs while writing it to a local file, or uploading it to the HA store

putPermanent

public PermanentBlobKey putPermanent(org.apache.flink.api.common.JobID jobId,
                                     InputStream inputStream)
                              throws IOException

Description copied from interface: BlobWriter

Uploads the data from the given input stream for the given job to the BLOB server and makes it a permanent BLOB.

Specified by:

putPermanent in interface BlobWriter

Parameters:

jobId - ID of the job this blob belongs to

inputStream - the input stream to read the data from

Returns:

the computed BLOB key identifying the BLOB on the server

Throws:

IOException - thrown if an I/O error occurs while reading the data from the input stream, writing it to a local file, or uploading it to the HA store

deleteFromCache

public boolean deleteFromCache(TransientBlobKey key)

Deletes the (job-unrelated) file associated with the blob key in the local storage of the blob server.

Specified by:

deleteFromCache in interface TransientBlobService

Parameters:

key - blob key associated with the file to be deleted

Returns:

true if the given blob is successfully deleted or non-existing; false otherwise

deleteFromCache

public boolean deleteFromCache(org.apache.flink.api.common.JobID jobId,
                               TransientBlobKey key)

Deletes the file associated with the blob key in the local storage of the blob server.

Specified by:

deleteFromCache in interface TransientBlobService

Parameters:

jobId - ID of the job this blob belongs to

key - blob key associated with the file to be deleted

Returns:

true if the given blob is successfully deleted or non-existing; false otherwise

deletePermanent

public boolean deletePermanent(org.apache.flink.api.common.JobID jobId,
                               PermanentBlobKey key)

Delete the uploaded data with the given JobID and PermanentBlobKey.

Specified by:

deletePermanent in interface BlobWriter

Parameters:

jobId - ID of the job this blob belongs to

key - the key of this blob

localCleanupAsync

public CompletableFuture<Void> localCleanupAsync(org.apache.flink.api.common.JobID jobId,
                                                 Executor cleanupExecutor)

Deletes locally stored artifacts for the job represented by the given JobID. This doesn't touch the job's entry in the BlobStore to enable recovering.

Specified by:

localCleanupAsync in interface LocallyCleanableResource

Parameters:

jobId - The JobID of the job that is subject to cleanup.

cleanupExecutor - The fallback executor for IO-heavy operations.

Returns:

The cleanup result future.

globalCleanupAsync

public CompletableFuture<Void> globalCleanupAsync(org.apache.flink.api.common.JobID jobId,
                                                  Executor executor)

Removes all BLOBs from local and HA store belonging to the given JobID.

Specified by:

globalCleanupAsync in interface GloballyCleanableResource

Parameters:

jobId - ID of the job this blob belongs to

executor - The fallback executor for IO-heavy operations.

Returns:

The cleanup result future.

retainJobs

public void retainJobs(Collection<org.apache.flink.api.common.JobID> jobsToRetain,
                       Executor ioExecutor)
                throws IOException

Throws:

IOException

getPermanentBlobService

public PermanentBlobService getPermanentBlobService()

Description copied from interface: BlobService

Returns a BLOB service for accessing permanent BLOBs.

Specified by:

getPermanentBlobService in interface BlobService

Returns:

BLOB service

getTransientBlobService

public TransientBlobService getTransientBlobService()

Description copied from interface: BlobService

Returns a BLOB service for accessing transient BLOBs.

Specified by:

getTransientBlobService in interface BlobService

Returns:

BLOB service

getMinOffloadingSize

public final int getMinOffloadingSize()

Returns the configuration used by the BLOB server.

Specified by:

getMinOffloadingSize in interface BlobWriter

Returns:

configuration

getPort

public int getPort()

Returns the port on which the server is listening.

Specified by:

getPort in interface BlobService

Returns:

port on which the server is listening

isShutdown

public boolean isShutdown()

Tests whether the BLOB server has been requested to shut down.

Returns:

True, if the server has been requested to shut down, false otherwise.