public class AsyncFsStateBackend extends AbstractStateBackend
The state backend has one core directory into which it puts all checkpoint data. Inside that
directory, it creates a directory per job, inside which each checkpoint gets a directory, with
files for each state, for example:
hdfs://namenode:port/flink-checkpoints/<job-id>/chk-17/6ba7b810-9dad-11d1-80b4-00c04fd430c8
Modifier and Type | Field and Description |
---|---|
static int |
DEFAULT_FILE_STATE_THRESHOLD
By default, state smaller than 1024 bytes will not be written to files, but
will be stored directly with the metadata
|
Constructor and Description |
---|
AsyncFsStateBackend(Path checkpointDataUri)
Creates a new state backend that stores its checkpoint data in the file system and location
defined by the given URI.
|
AsyncFsStateBackend(String checkpointDataUri)
Creates a new state backend that stores its checkpoint data in the file system and location
defined by the given URI.
|
AsyncFsStateBackend(URI checkpointDataUri)
Creates a new state backend that stores its checkpoint data in the file system and location
defined by the given URI.
|
AsyncFsStateBackend(URI checkpointDataUri,
int fileStateSizeThreshold)
Creates a new state backend that stores its checkpoint data in the file system and location
defined by the given URI.
|
Modifier and Type | Method and Description |
---|---|
<K> AbstractKeyedStateBackend<K> |
createKeyedStateBackend(Environment env,
JobID jobID,
String operatorIdentifier,
TypeSerializer<K> keySerializer,
int numberOfKeyGroups,
KeyGroupRange keyGroupRange,
TaskKvStateRegistry kvStateRegistry)
Creates a new
AbstractKeyedStateBackend that is responsible for keeping keyed state
and can be checkpointed to checkpoint streams. |
CheckpointStreamFactory |
createStreamFactory(JobID jobId,
String operatorIdentifier)
Creates a
CheckpointStreamFactory that can be used to create streams
that should end up in a checkpoint. |
Path |
getBasePath()
Gets the base directory where all state-containing files are stored.
|
String |
toString() |
static Path |
validateAndNormalizeUri(URI checkpointDataUri)
Checks and normalizes the checkpoint data URI.
|
createOperatorStateBackend
public static final int DEFAULT_FILE_STATE_THRESHOLD
public AsyncFsStateBackend(String checkpointDataUri) throws IOException
A file system for the file system scheme in the URI (e.g., 'file://', 'hdfs://', or 'S3://')
must be accessible via FileSystem.get(URI)
.
For a state backend targeting HDFS, this means that the URI must either specify the authority (host and port), or that the Hadoop configuration that describes that information must be in the classpath.
checkpointDataUri
- The URI describing the filesystem (scheme and optionally authority),
and the path to the checkpoint data directory.IOException
- Thrown, if no file system can be found for the scheme in the URI.public AsyncFsStateBackend(Path checkpointDataUri) throws IOException
A file system for the file system scheme in the URI (e.g., 'file://', 'hdfs://', or 'S3://')
must be accessible via FileSystem.get(URI)
.
For a state backend targeting HDFS, this means that the URI must either specify the authority (host and port), or that the Hadoop configuration that describes that information must be in the classpath.
checkpointDataUri
- The URI describing the filesystem (scheme and optionally authority),
and the path to the checkpoint data directory.IOException
- Thrown, if no file system can be found for the scheme in the URI.public AsyncFsStateBackend(URI checkpointDataUri) throws IOException
A file system for the file system scheme in the URI (e.g., 'file://', 'hdfs://', or 'S3://')
must be accessible via FileSystem.get(URI)
.
For a state backend targeting HDFS, this means that the URI must either specify the authority (host and port), or that the Hadoop configuration that describes that information must be in the classpath.
checkpointDataUri
- The URI describing the filesystem (scheme and optionally authority),
and the path to the checkpoint data directory.IOException
- Thrown, if no file system can be found for the scheme in the URI.public AsyncFsStateBackend(URI checkpointDataUri, int fileStateSizeThreshold) throws IOException
A file system for the file system scheme in the URI (e.g., 'file://', 'hdfs://', or 'S3://')
must be accessible via FileSystem.get(URI)
.
For a state backend targeting HDFS, this means that the URI must either specify the authority (host and port), or that the Hadoop configuration that describes that information must be in the classpath.
checkpointDataUri
- The URI describing the filesystem (scheme and optionally authority),
and the path to the checkpoint data directory.fileStateSizeThreshold
- State up to this size will be stored as part of the metadata,
rather than in filesIOException
- Thrown, if no file system can be found for the scheme in the URI.public Path getBasePath()
public CheckpointStreamFactory createStreamFactory(JobID jobId, String operatorIdentifier) throws IOException
AbstractStateBackend
CheckpointStreamFactory
that can be used to create streams
that should end up in a checkpoint.createStreamFactory
in class AbstractStateBackend
jobId
- The JobID
of the job for which we are creating checkpoint streams.operatorIdentifier
- An identifier of the operator for which we create streams.IOException
public <K> AbstractKeyedStateBackend<K> createKeyedStateBackend(Environment env, JobID jobID, String operatorIdentifier, TypeSerializer<K> keySerializer, int numberOfKeyGroups, KeyGroupRange keyGroupRange, TaskKvStateRegistry kvStateRegistry) throws Exception
AbstractStateBackend
AbstractKeyedStateBackend
that is responsible for keeping keyed state
and can be checkpointed to checkpoint streams.createKeyedStateBackend
in class AbstractStateBackend
Exception
public static Path validateAndNormalizeUri(URI checkpointDataUri) throws IOException
If the URI does not include an authority, but the file system configured for the URI has an authority, then the normalized path will include this authority.
checkpointDataUri
- The URI to check and normalize.IllegalArgumentException
- Thrown, if the URI misses scheme or path.IOException
- Thrown, if no file system can be found for the URI's scheme.Copyright © 2014–2017 The Apache Software Foundation. All rights reserved.