public class BlobServer extends Thread implements BlobService, BlobWriter, PermanentBlobService, TransientBlobService, LocallyCleanableResource, GloballyCleanableResource
Thread.State, Thread.UncaughtExceptionHandler
MAX_PRIORITY, MIN_PRIORITY, NORM_PRIORITY
Constructor and Description |
---|
BlobServer(Configuration config,
File storageDir,
BlobStore blobStore) |
BlobServer(Configuration config,
Reference<File> storageDir,
BlobStore blobStore)
Instantiates a new BLOB server and binds it to a free network port.
|
Modifier and Type | Method and Description |
---|---|
void |
close()
Shuts down the BLOB server.
|
protected BlobClient |
createClient() |
boolean |
deleteFromCache(JobID jobId,
TransientBlobKey key)
Deletes the file associated with the blob key in the local storage of the blob server.
|
boolean |
deleteFromCache(TransientBlobKey key)
Deletes the (job-unrelated) file associated with the blob key in the local storage of the
blob server.
|
boolean |
deletePermanent(JobID jobId,
PermanentBlobKey key)
Delete the uploaded data with the given
JobID and PermanentBlobKey . |
File |
getFile(JobID jobId,
PermanentBlobKey key)
Returns the path to a local copy of the file associated with the provided job ID and blob
key.
|
File |
getFile(JobID jobId,
TransientBlobKey key)
Retrieves the local path of a file associated with a job and a blob key.
|
File |
getFile(TransientBlobKey key)
Retrieves the local path of a (job-unrelated) file associated with a job and a blob key.
|
int |
getMinOffloadingSize()
Returns the configuration used by the BLOB server.
|
PermanentBlobService |
getPermanentBlobService()
Returns a BLOB service for accessing permanent BLOBs.
|
int |
getPort()
Returns the port on which the server is listening.
|
File |
getStorageDir() |
File |
getStorageLocation(JobID jobId,
BlobKey key)
Returns a file handle to the file associated with the given blob key on the blob server.
|
TransientBlobService |
getTransientBlobService()
Returns a BLOB service for accessing transient BLOBs.
|
CompletableFuture<Void> |
globalCleanupAsync(JobID jobId,
Executor executor)
Removes all BLOBs from local and HA store belonging to the given
JobID . |
boolean |
isShutdown()
Tests whether the BLOB server has been requested to shut down.
|
CompletableFuture<Void> |
localCleanupAsync(JobID jobId,
Executor cleanupExecutor)
Deletes locally stored artifacts for the job represented by the given
JobID . |
PermanentBlobKey |
putPermanent(JobID jobId,
byte[] value)
Uploads the data of the given byte array for the given job to the BLOB server and makes it a
permanent BLOB.
|
PermanentBlobKey |
putPermanent(JobID jobId,
InputStream inputStream)
Uploads the data from the given input stream for the given job to the BLOB server and makes
it a permanent BLOB.
|
TransientBlobKey |
putTransient(byte[] value)
Uploads the (job-unrelated) data of the given byte array to the BLOB server.
|
TransientBlobKey |
putTransient(InputStream inputStream)
Uploads the (job-unrelated) data from the given input stream to the BLOB server.
|
TransientBlobKey |
putTransient(JobID jobId,
byte[] value)
Uploads the data of the given byte array for the given job to the BLOB server.
|
TransientBlobKey |
putTransient(JobID jobId,
InputStream inputStream)
Uploads the data from the given input stream for the given job to the BLOB server.
|
void |
retainJobs(Collection<JobID> jobsToRetain,
Executor ioExecutor) |
void |
run() |
activeCount, checkAccess, clone, countStackFrames, currentThread, destroy, dumpStack, enumerate, getAllStackTraces, getContextClassLoader, getDefaultUncaughtExceptionHandler, getId, getName, getPriority, getStackTrace, getState, getThreadGroup, getUncaughtExceptionHandler, holdsLock, interrupt, interrupted, isAlive, isDaemon, isInterrupted, join, join, join, resume, setContextClassLoader, setDaemon, setDefaultUncaughtExceptionHandler, setName, setPriority, setUncaughtExceptionHandler, sleep, sleep, start, stop, stop, suspend, toString, yield
equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
offloadWithException, serializeAndTryOffload, tryOffload
readFile
@VisibleForTesting public BlobServer(Configuration config, File storageDir, BlobStore blobStore) throws IOException
IOException
public BlobServer(Configuration config, Reference<File> storageDir, BlobStore blobStore) throws IOException
config
- Configuration to be used to instantiate the BlobServerstorageDir
- storage directory for the blobsblobStore
- BlobStore to store blobs persistentlyIOException
- thrown if the BLOB server cannot bind to a free network port or if the
(local or distributed) file storage cannot be created or is not usablepublic File getStorageDir()
@VisibleForTesting public File getStorageLocation(@Nullable JobID jobId, BlobKey key) throws IOException
This is only called from BlobServerConnection
or unit tests.
jobId
- ID of the job this blob belongs to (or null if job-unrelated)key
- identifying the fileIOException
- if creating the directory failspublic void close() throws IOException
close
in interface Closeable
close
in interface AutoCloseable
IOException
protected BlobClient createClient() throws IOException
IOException
public File getFile(TransientBlobKey key) throws IOException
The blob server looks the blob key up in its local storage. If the file exists, it is
returned. If the file does not exist, it is retrieved from the HA blob store (if available)
or a FileNotFoundException
is thrown.
getFile
in interface TransientBlobService
key
- blob key associated with the requested fileIOException
- Thrown if the file retrieval failed.public File getFile(JobID jobId, TransientBlobKey key) throws IOException
The blob server looks the blob key up in its local storage. If the file exists, it is
returned. If the file does not exist, it is retrieved from the HA blob store (if available)
or a FileNotFoundException
is thrown.
getFile
in interface TransientBlobService
jobId
- ID of the job this blob belongs tokey
- blob key associated with the requested fileIOException
- Thrown if the file retrieval failed.public File getFile(JobID jobId, PermanentBlobKey key) throws IOException
We will first attempt to serve the BLOB from the local storage. If the BLOB is not in there, we will try to download it from the HA store.
getFile
in interface PermanentBlobService
jobId
- ID of the job this blob belongs tokey
- blob key associated with the requested fileFileNotFoundException
- if the BLOB does not exist;IOException
- if any other error occurs when retrieving the filepublic TransientBlobKey putTransient(byte[] value) throws IOException
TransientBlobService
putTransient
in interface TransientBlobService
value
- the buffer to uploadIOException
- thrown if an I/O error occurs while uploading the data to the BLOB serverpublic TransientBlobKey putTransient(JobID jobId, byte[] value) throws IOException
TransientBlobService
putTransient
in interface TransientBlobService
jobId
- the ID of the job the BLOB belongs tovalue
- the buffer to uploadIOException
- thrown if an I/O error occurs while uploading the data to the BLOB serverpublic TransientBlobKey putTransient(InputStream inputStream) throws IOException
TransientBlobService
putTransient
in interface TransientBlobService
inputStream
- the input stream to read the data fromIOException
- thrown if an I/O error occurs while reading the data from the input
stream or uploading the data to the BLOB serverpublic TransientBlobKey putTransient(JobID jobId, InputStream inputStream) throws IOException
TransientBlobService
putTransient
in interface TransientBlobService
jobId
- ID of the job this blob belongs toinputStream
- the input stream to read the data fromIOException
- thrown if an I/O error occurs while reading the data from the input
stream or uploading the data to the BLOB serverpublic PermanentBlobKey putPermanent(JobID jobId, byte[] value) throws IOException
BlobWriter
putPermanent
in interface BlobWriter
jobId
- the ID of the job the BLOB belongs tovalue
- the buffer to uploadIOException
- thrown if an I/O error occurs while writing it to a local file, or
uploading it to the HA storepublic PermanentBlobKey putPermanent(JobID jobId, InputStream inputStream) throws IOException
BlobWriter
putPermanent
in interface BlobWriter
jobId
- ID of the job this blob belongs toinputStream
- the input stream to read the data fromIOException
- thrown if an I/O error occurs while reading the data from the input
stream, writing it to a local file, or uploading it to the HA storepublic boolean deleteFromCache(TransientBlobKey key)
deleteFromCache
in interface TransientBlobService
key
- blob key associated with the file to be deletedpublic boolean deleteFromCache(JobID jobId, TransientBlobKey key)
deleteFromCache
in interface TransientBlobService
jobId
- ID of the job this blob belongs tokey
- blob key associated with the file to be deletedpublic boolean deletePermanent(JobID jobId, PermanentBlobKey key)
JobID
and PermanentBlobKey
.deletePermanent
in interface BlobWriter
jobId
- ID of the job this blob belongs tokey
- the key of this blobpublic CompletableFuture<Void> localCleanupAsync(JobID jobId, Executor cleanupExecutor)
JobID
. This
doesn't touch the job's entry in the BlobStore
to enable recovering.localCleanupAsync
in interface LocallyCleanableResource
jobId
- The JobID
of the job that is subject to cleanup.cleanupExecutor
- The fallback executor for IO-heavy operations.public CompletableFuture<Void> globalCleanupAsync(JobID jobId, Executor executor)
JobID
.globalCleanupAsync
in interface GloballyCleanableResource
jobId
- ID of the job this blob belongs toexecutor
- The fallback executor for IO-heavy operations.public void retainJobs(Collection<JobID> jobsToRetain, Executor ioExecutor) throws IOException
IOException
public PermanentBlobService getPermanentBlobService()
BlobService
getPermanentBlobService
in interface BlobService
public TransientBlobService getTransientBlobService()
BlobService
getTransientBlobService
in interface BlobService
public final int getMinOffloadingSize()
getMinOffloadingSize
in interface BlobWriter
public int getPort()
getPort
in interface BlobService
public boolean isShutdown()
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.