public class TransientBlobCache extends AbstractBlobCache implements TransientBlobService
BlobServer
.
TODO: make this truly transient by returning file streams to a local copy with the remote being removed upon retrieval and the local copy being deleted at the end of the stream.
blobClientConfig, blobView, log, numFetchRetries, readWriteLock, serverAddress, shutdownHook, shutdownRequested, storageDir, tempFileCounter
Constructor and Description |
---|
TransientBlobCache(Configuration blobClientConfig,
File storageDir,
InetSocketAddress serverAddress) |
TransientBlobCache(Configuration blobClientConfig,
Reference<File> storageDir,
InetSocketAddress serverAddress)
Instantiates a new BLOB cache.
|
Modifier and Type | Method and Description |
---|---|
protected void |
cancelCleanupTask()
Cancels any cleanup task that subclasses may be executing.
|
boolean |
deleteFromCache(JobID jobId,
TransientBlobKey key)
Deletes the file associated with the provided job ID and blob key from the local cache.
|
boolean |
deleteFromCache(TransientBlobKey key)
Deletes the (job-unrelated) file associated with the provided blob key from the local cache.
|
File |
getFile(JobID jobId,
TransientBlobKey key)
Returns the path to a local copy of the file associated with the provided job ID and blob
key.
|
File |
getFile(TransientBlobKey key)
Returns the path to a local copy of the (job-unrelated) file associated with the provided
blob key.
|
protected File |
getFileInternal(JobID jobId,
BlobKey blobKey)
Returns local copy of the file for the BLOB with the given key.
|
File |
getStorageLocation(JobID jobId,
BlobKey key)
Returns a file handle to the file associated with the given blob key on the blob server.
|
TransientBlobKey |
putTransient(byte[] value)
Uploads the (job-unrelated) data of the given byte array to the BLOB server.
|
TransientBlobKey |
putTransient(InputStream inputStream)
Uploads the (job-unrelated) data from the given input stream to the BLOB server.
|
TransientBlobKey |
putTransient(JobID jobId,
byte[] value)
Uploads the data of the given byte array for the given job to the BLOB server.
|
TransientBlobKey |
putTransient(JobID jobId,
InputStream inputStream)
Uploads the data from the given input stream for the given job to the BLOB server.
|
close, getPort, getStorageDir, setBlobServerAddress
@VisibleForTesting public TransientBlobCache(Configuration blobClientConfig, File storageDir, @Nullable InetSocketAddress serverAddress) throws IOException
IOException
public TransientBlobCache(Configuration blobClientConfig, Reference<File> storageDir, @Nullable InetSocketAddress serverAddress) throws IOException
blobClientConfig
- global configurationstorageDir
- storage directory for the cached blobsserverAddress
- address of the BlobServer
to use for fetching files from or
null
if none yetIOException
- thrown if the (local or distributed) file storage cannot be created or is
not usablepublic File getFile(TransientBlobKey key) throws IOException
TransientBlobService
getFile
in interface TransientBlobService
key
- blob key associated with the requested fileFileNotFoundException
- when the path does not exist;IOException
- if any other error occurs when retrieving the filepublic File getFile(JobID jobId, TransientBlobKey key) throws IOException
TransientBlobService
getFile
in interface TransientBlobService
jobId
- ID of the job this blob belongs tokey
- blob key associated with the requested fileFileNotFoundException
- when the path does not exist;IOException
- if any other error occurs when retrieving the fileprotected File getFileInternal(@Nullable JobID jobId, BlobKey blobKey) throws IOException
AbstractBlobCache
The method will first attempt to serve the BLOB from its local cache. If the BLOB is not in the cache, the method will try to download it from this cache's BLOB server via a distributed BLOB store (if available) or direct end-to-end download.
getFileInternal
in class AbstractBlobCache
jobId
- ID of the job this blob belongs to (or null if job-unrelated)blobKey
- The key of the desired BLOB.IOException
- Thrown if an I/O error occurs while downloading the BLOBs from the BLOB
server.public TransientBlobKey putTransient(byte[] value) throws IOException
TransientBlobService
putTransient
in interface TransientBlobService
value
- the buffer to uploadIOException
- thrown if an I/O error occurs while uploading the data to the BLOB serverpublic TransientBlobKey putTransient(JobID jobId, byte[] value) throws IOException
TransientBlobService
putTransient
in interface TransientBlobService
jobId
- the ID of the job the BLOB belongs tovalue
- the buffer to uploadIOException
- thrown if an I/O error occurs while uploading the data to the BLOB serverpublic TransientBlobKey putTransient(InputStream inputStream) throws IOException
TransientBlobService
putTransient
in interface TransientBlobService
inputStream
- the input stream to read the data fromIOException
- thrown if an I/O error occurs while reading the data from the input
stream or uploading the data to the BLOB serverpublic TransientBlobKey putTransient(JobID jobId, InputStream inputStream) throws IOException
TransientBlobService
putTransient
in interface TransientBlobService
jobId
- ID of the job this blob belongs toinputStream
- the input stream to read the data fromIOException
- thrown if an I/O error occurs while reading the data from the input
stream or uploading the data to the BLOB serverpublic boolean deleteFromCache(TransientBlobKey key)
TransientBlobService
deleteFromCache
in interface TransientBlobService
key
- associated with the file to be deletedpublic boolean deleteFromCache(JobID jobId, TransientBlobKey key)
TransientBlobService
deleteFromCache
in interface TransientBlobService
jobId
- ID of the job this blob belongs tokey
- associated with the file to be deleted@VisibleForTesting public File getStorageLocation(@Nullable JobID jobId, BlobKey key) throws IOException
jobId
- ID of the job this blob belongs to (or null if job-unrelated)key
- identifying the fileIOException
- if creating the directory failsprotected void cancelCleanupTask()
AbstractBlobCache
This is called during AbstractBlobCache.close()
.
cancelCleanupTask
in class AbstractBlobCache
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.