org.apache.flink.runtime.blob

Class AbstractBlobCache

java.lang.Object

org.apache.flink.runtime.blob.AbstractBlobCache

All Implemented Interfaces:

Closeable, AutoCloseable

Direct Known Subclasses:

PermanentBlobCache, TransientBlobCache

public abstract class AbstractBlobCache
extends Object
implements Closeable

Abstract base class for permanent and transient BLOB files.

Field Summary

Fields

Modifier and Type	Field and Description
protected Configuration	blobClientConfig Configuration for the blob client like ssl parameters required to connect to the blob server.
protected BlobView	blobView Blob store for distributed file storage, e.g.
protected org.slf4j.Logger	log The log object used for debugging.
protected int	numFetchRetries The number of retries when the transfer fails.
protected ReadWriteLock	readWriteLock Lock guarding concurrent file accesses.
protected InetSocketAddress	serverAddress
protected Thread	shutdownHook Shutdown hook thread to ensure deletion of the local storage directory.
protected AtomicBoolean	shutdownRequested
protected Reference<File>	storageDir Root directory for local file storage.
protected AtomicLong	tempFileCounter Counter to generate unique names for temporary files.

Constructor Summary

Constructors

Constructor and Description
AbstractBlobCache(Configuration blobClientConfig, Reference<File> storageDir, BlobView blobView, org.slf4j.Logger logger, InetSocketAddress serverAddress)

Method Summary

Modifier and Type	Method and Description
protected abstract void	cancelCleanupTask() Cancels any cleanup task that subclasses may be executing.
void	close()
protected File	getFileInternal(JobID jobId, BlobKey blobKey) Returns local copy of the file for the BLOB with the given key.
int	getPort() Returns the port the BLOB server is listening on.
File	getStorageDir()
void	setBlobServerAddress(InetSocketAddress blobServerAddress) Sets the address of the BlobServer.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

protected final org.slf4j.Logger log

The log object used for debugging.

tempFileCounter

protected final AtomicLong tempFileCounter

Counter to generate unique names for temporary files.

storageDir

protected final Reference<File> storageDir

Root directory for local file storage.

blobView

protected final BlobView blobView

Blob store for distributed file storage, e.g. in HA.

shutdownRequested

protected final AtomicBoolean shutdownRequested

shutdownHook

protected final Thread shutdownHook

Shutdown hook thread to ensure deletion of the local storage directory.

numFetchRetries

protected final int numFetchRetries

The number of retries when the transfer fails.

blobClientConfig

protected final Configuration blobClientConfig

Configuration for the blob client like ssl parameters required to connect to the blob server.

readWriteLock

protected final ReadWriteLock readWriteLock

Lock guarding concurrent file accesses.

serverAddress

@Nullable
protected volatile InetSocketAddress serverAddress

Constructor Detail

AbstractBlobCache

public AbstractBlobCache(Configuration blobClientConfig,
                         Reference<File> storageDir,
                         BlobView blobView,
                         org.slf4j.Logger logger,
                         @Nullable
                         InetSocketAddress serverAddress)
                  throws IOException

Throws:

IOException

Method Detail

getStorageDir

public File getStorageDir()

getFileInternal

protected File getFileInternal(@Nullable
                               JobID jobId,
                               BlobKey blobKey)
                        throws IOException

Returns local copy of the file for the BLOB with the given key.

The method will first attempt to serve the BLOB from its local cache. If the BLOB is not in the cache, the method will try to download it from this cache's BLOB server via a distributed BLOB store (if available) or direct end-to-end download.

Parameters:

jobId - ID of the job this blob belongs to (or null if job-unrelated)

blobKey - The key of the desired BLOB.

Returns:

file referring to the local storage location of the BLOB.

Throws:

IOException - Thrown if an I/O error occurs while downloading the BLOBs from the BLOB server.

getPort

public int getPort()

Returns the port the BLOB server is listening on.

Returns:

BLOB server port or -1 if no server address

setBlobServerAddress

public void setBlobServerAddress(InetSocketAddress blobServerAddress)

Sets the address of the BlobServer.

Parameters:

blobServerAddress - address of the BlobServer.

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

cancelCleanupTask

protected abstract void cancelCleanupTask()

Cancels any cleanup task that subclasses may be executing.

This is called during close().