Constructor and Description |
---|
DistributedCache(Map<String,Future<Path>> cacheCopyTasks) |
Constructor and Description |
---|
AbstractRuntimeUDFContext(TaskInfo taskInfo,
ClassLoader userCodeClassLoader,
ExecutionConfig executionConfig,
Map<String,Accumulator<?,?>> accumulators,
Map<String,Future<Path>> cpTasks,
MetricGroup metrics) |
RuntimeUDFContext(TaskInfo taskInfo,
ClassLoader userCodeClassLoader,
ExecutionConfig executionConfig,
Map<String,Future<Path>> cpTasks,
Map<String,Accumulator<?,?>> accumulators,
MetricGroup metrics) |
Modifier and Type | Field and Description |
---|---|
protected Path |
FileInputFormat.filePath
Deprecated.
Please override
FileInputFormat.supportsMultiPaths() and
use FileInputFormat.getFilePaths() and FileInputFormat.setFilePaths(Path...) . |
protected Path |
FileOutputFormat.outputFilePath
The path of the file to be written.
|
Modifier and Type | Method and Description |
---|---|
Path |
FileInputFormat.getFilePath()
Deprecated.
Please use getFilePaths() instead.
|
Path[] |
FileInputFormat.getFilePaths()
Returns the paths of all files to be read by the FileInputFormat.
|
Path |
FileOutputFormat.getOutputFilePath() |
Modifier and Type | Method and Description |
---|---|
boolean |
GlobFilePathFilter.filterPath(Path filePath) |
abstract boolean |
FilePathFilter.filterPath(Path filePath)
Returns
true if the filePath given is to be
ignored when processing a directory, e.g. |
boolean |
FilePathFilter.DefaultFilter.filterPath(Path filePath) |
protected FileInputFormat.FileBaseStatistics |
FileInputFormat.getFileStats(FileInputFormat.FileBaseStatistics cachedStats,
Path[] filePaths,
ArrayList<FileStatus> files) |
protected FileInputFormat.FileBaseStatistics |
FileInputFormat.getFileStats(FileInputFormat.FileBaseStatistics cachedStats,
Path filePath,
FileSystem fs,
ArrayList<FileStatus> files) |
void |
FileInputFormat.setFilePath(Path filePath)
Sets a single path of a file to be read.
|
void |
FileInputFormat.setFilePaths(Path... filePaths)
Sets multiple paths of files to be read.
|
void |
FileOutputFormat.setOutputFilePath(Path path) |
Constructor and Description |
---|
DelimitedInputFormat(Path filePath,
Configuration configuration) |
FileInputFormat(Path filePath) |
FileOutputFormat(Path outputPath) |
GenericCsvInputFormat(Path filePath) |
Modifier and Type | Method and Description |
---|---|
Path |
CsvReader.getFilePath() |
Constructor and Description |
---|
CsvInputFormat(Path filePath) |
CsvOutputFormat(Path outputPath)
Creates an instance of CsvOutputFormat.
|
CsvOutputFormat(Path outputPath,
String fieldDelimiter)
Creates an instance of CsvOutputFormat.
|
CsvOutputFormat(Path outputPath,
String recordDelimiter,
String fieldDelimiter)
Creates an instance of CsvOutputFormat.
|
CsvReader(Path filePath,
ExecutionEnvironment executionContext) |
PojoCsvInputFormat(Path filePath,
PojoTypeInfo<OUT> pojoTypeInfo) |
PojoCsvInputFormat(Path filePath,
PojoTypeInfo<OUT> pojoTypeInfo,
boolean[] includedFieldsMask) |
PojoCsvInputFormat(Path filePath,
PojoTypeInfo<OUT> pojoTypeInfo,
int[] includedFieldsMask) |
PojoCsvInputFormat(Path filePath,
PojoTypeInfo<OUT> pojoTypeInfo,
String[] fieldNames) |
PojoCsvInputFormat(Path filePath,
PojoTypeInfo<OUT> pojoTypeInfo,
String[] fieldNames,
boolean[] includedFieldsMask) |
PojoCsvInputFormat(Path filePath,
PojoTypeInfo<OUT> pojoTypeInfo,
String[] fieldNames,
int[] includedFieldsMask) |
PojoCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
PojoTypeInfo<OUT> pojoTypeInfo) |
PojoCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
PojoTypeInfo<OUT> pojoTypeInfo,
boolean[] includedFieldsMask) |
PojoCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
PojoTypeInfo<OUT> pojoTypeInfo,
int[] includedFieldsMask) |
PojoCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
PojoTypeInfo<OUT> pojoTypeInfo,
String[] fieldNames) |
PojoCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
PojoTypeInfo<OUT> pojoTypeInfo,
String[] fieldNames,
boolean[] includedFieldsMask) |
PojoCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
PojoTypeInfo<OUT> pojoTypeInfo,
String[] fieldNames,
int[] includedFieldsMask) |
PrimitiveInputFormat(Path filePath,
Class<OT> primitiveClass) |
PrimitiveInputFormat(Path filePath,
String delimiter,
Class<OT> primitiveClass) |
RowCsvInputFormat(Path filePath,
TypeInformation[] fieldTypes) |
RowCsvInputFormat(Path filePath,
TypeInformation[] fieldTypes,
boolean emptyColumnAsNull) |
RowCsvInputFormat(Path filePath,
TypeInformation[] fieldTypes,
int[] selectedFields) |
RowCsvInputFormat(Path filePath,
TypeInformation[] fieldTypes,
String lineDelimiter,
String fieldDelimiter) |
RowCsvInputFormat(Path filePath,
TypeInformation[] fieldTypes,
String lineDelimiter,
String fieldDelimiter,
int[] selectedFields) |
RowCsvInputFormat(Path filePath,
TypeInformation[] fieldTypeInfos,
String lineDelimiter,
String fieldDelimiter,
int[] selectedFields,
boolean emptyColumnAsNull) |
TextInputFormat(Path filePath) |
TextOutputFormat(Path outputPath) |
TextOutputFormat(Path outputPath,
String charset) |
TextValueInputFormat(Path filePath) |
TupleCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
TupleTypeInfoBase<OUT> tupleTypeInfo) |
TupleCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
TupleTypeInfoBase<OUT> tupleTypeInfo,
boolean[] includedFieldsMask) |
TupleCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
TupleTypeInfoBase<OUT> tupleTypeInfo,
int[] includedFieldsMask) |
TupleCsvInputFormat(Path filePath,
TupleTypeInfoBase<OUT> tupleTypeInfo) |
TupleCsvInputFormat(Path filePath,
TupleTypeInfoBase<OUT> tupleTypeInfo,
boolean[] includedFieldsMask) |
TupleCsvInputFormat(Path filePath,
TupleTypeInfoBase<OUT> tupleTypeInfo,
int[] includedFieldsMask) |
Constructor and Description |
---|
ScalaCsvOutputFormat(Path outputPath)
Creates an instance of CsvOutputFormat.
|
ScalaCsvOutputFormat(Path outputPath,
String fieldDelimiter)
Creates an instance of CsvOutputFormat.
|
ScalaCsvOutputFormat(Path outputPath,
String recordDelimiter,
String fieldDelimiter)
Creates an instance of CsvOutputFormat.
|
Modifier and Type | Method and Description |
---|---|
static Path |
Path.fromLocalFile(File file)
Creates a path for the given local file.
|
Path |
LimitedConnectionsFileSystem.getHomeDirectory() |
Path |
SafetyNetWrapperFileSystem.getHomeDirectory() |
abstract Path |
FileSystem.getHomeDirectory()
Returns the path of the user's home directory in this file system.
|
Path |
Path.getParent()
Returns the parent of a path, i.e., everything that precedes the last separator
or
null if at root. |
Path |
FileInputSplit.getPath()
Returns the path of the file containing this split's data.
|
Path |
FileStatus.getPath()
Returns the corresponding Path to the FileStatus.
|
Path |
LimitedConnectionsFileSystem.getWorkingDirectory() |
Path |
SafetyNetWrapperFileSystem.getWorkingDirectory() |
abstract Path |
FileSystem.getWorkingDirectory()
Returns the path of the file system's current working directory.
|
Path |
Path.makeQualified(FileSystem fs)
Returns a qualified path object.
|
Path |
Path.suffix(String suffix)
Adds a suffix to the final name in the path.
|
Modifier and Type | Method and Description |
---|---|
FSDataOutputStream |
FileSystem.create(Path f,
boolean overwrite)
Deprecated.
Use
FileSystem.create(Path, WriteMode) instead. |
FSDataOutputStream |
LimitedConnectionsFileSystem.create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize)
Deprecated.
|
FSDataOutputStream |
SafetyNetWrapperFileSystem.create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize) |
FSDataOutputStream |
FileSystem.create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize)
Deprecated.
Deprecated because not well supported across types of file systems.
Control the behavior of specific file systems via configurations instead.
|
FSDataOutputStream |
LimitedConnectionsFileSystem.create(Path f,
FileSystem.WriteMode overwriteMode) |
FSDataOutputStream |
SafetyNetWrapperFileSystem.create(Path f,
FileSystem.WriteMode overwrite) |
abstract FSDataOutputStream |
FileSystem.create(Path f,
FileSystem.WriteMode overwriteMode)
Opens an FSDataOutputStream to a new file at the given path.
|
boolean |
LimitedConnectionsFileSystem.delete(Path f,
boolean recursive) |
boolean |
SafetyNetWrapperFileSystem.delete(Path f,
boolean recursive) |
abstract boolean |
FileSystem.delete(Path f,
boolean recursive)
Delete a file.
|
boolean |
LimitedConnectionsFileSystem.exists(Path f) |
boolean |
SafetyNetWrapperFileSystem.exists(Path f) |
boolean |
FileSystem.exists(Path f)
Check if exists.
|
FileStatus |
LimitedConnectionsFileSystem.getFileStatus(Path f) |
FileStatus |
SafetyNetWrapperFileSystem.getFileStatus(Path f) |
abstract FileStatus |
FileSystem.getFileStatus(Path f)
Return a file status object that represents the path.
|
boolean |
SafetyNetWrapperFileSystem.initOutPathDistFS(Path outPath,
FileSystem.WriteMode writeMode,
boolean createDirectory) |
boolean |
FileSystem.initOutPathDistFS(Path outPath,
FileSystem.WriteMode writeMode,
boolean createDirectory)
Initializes output directories on distributed file systems according to the given write mode.
|
boolean |
SafetyNetWrapperFileSystem.initOutPathLocalFS(Path outPath,
FileSystem.WriteMode writeMode,
boolean createDirectory) |
boolean |
FileSystem.initOutPathLocalFS(Path outPath,
FileSystem.WriteMode writeMode,
boolean createDirectory)
Initializes output directories on local file systems according to the given write mode.
|
FileStatus[] |
LimitedConnectionsFileSystem.listStatus(Path f) |
FileStatus[] |
SafetyNetWrapperFileSystem.listStatus(Path f) |
abstract FileStatus[] |
FileSystem.listStatus(Path f)
List the statuses of the files/directories in the given path if the path is
a directory.
|
boolean |
LimitedConnectionsFileSystem.mkdirs(Path f) |
boolean |
SafetyNetWrapperFileSystem.mkdirs(Path f) |
abstract boolean |
FileSystem.mkdirs(Path f)
Make the given file and all non-existent parents into directories.
|
FSDataInputStream |
LimitedConnectionsFileSystem.open(Path f) |
FSDataInputStream |
SafetyNetWrapperFileSystem.open(Path f) |
abstract FSDataInputStream |
FileSystem.open(Path f)
Opens an FSDataInputStream at the indicated Path.
|
FSDataInputStream |
LimitedConnectionsFileSystem.open(Path f,
int bufferSize) |
FSDataInputStream |
SafetyNetWrapperFileSystem.open(Path f,
int bufferSize) |
abstract FSDataInputStream |
FileSystem.open(Path f,
int bufferSize)
Opens an FSDataInputStream at the indicated Path.
|
boolean |
LimitedConnectionsFileSystem.rename(Path src,
Path dst) |
boolean |
SafetyNetWrapperFileSystem.rename(Path src,
Path dst) |
abstract boolean |
FileSystem.rename(Path src,
Path dst)
Renames the file/directory src to dst.
|
Constructor and Description |
---|
FileInputSplit(int num,
Path file,
long start,
long length,
String[] hosts)
Constructs a split with host information.
|
Path(Path parent,
Path child)
Resolve a child path against a parent path.
|
Path(Path parent,
String child)
Resolve a child path against a parent path.
|
Path(String parent,
Path child)
Resolve a child path against a parent path.
|
Modifier and Type | Method and Description |
---|---|
Path |
LocalFileSystem.getHomeDirectory() |
Path |
LocalFileStatus.getPath() |
Path |
LocalFileSystem.getWorkingDirectory() |
Modifier and Type | Method and Description |
---|---|
FSDataOutputStream |
LocalFileSystem.create(Path filePath,
FileSystem.WriteMode overwrite) |
boolean |
LocalFileSystem.delete(Path f,
boolean recursive) |
boolean |
LocalFileSystem.exists(Path f) |
FileStatus |
LocalFileSystem.getFileStatus(Path f) |
FileStatus[] |
LocalFileSystem.listStatus(Path f) |
boolean |
LocalFileSystem.mkdirs(Path f)
Recursively creates the directory specified by the provided path.
|
FSDataInputStream |
LocalFileSystem.open(Path f) |
FSDataInputStream |
LocalFileSystem.open(Path f,
int bufferSize) |
boolean |
LocalFileSystem.rename(Path src,
Path dst) |
Modifier and Type | Method and Description |
---|---|
Path |
FileCopyTask.getPath() |
Constructor and Description |
---|
FileCopyTask(Path path,
String relativePath) |
Constructor and Description |
---|
AvroInputFormat(Path filePath,
Class<E> type) |
AvroOutputFormat(Path filePath,
Class<E> type) |
Constructor and Description |
---|
GraphCsvReader(Path edgePath,
ExecutionEnvironment context) |
GraphCsvReader(Path edgePath,
MapFunction<K,VV> mapper,
ExecutionEnvironment context) |
GraphCsvReader(Path vertexPath,
Path edgePath,
ExecutionEnvironment context) |
Modifier and Type | Method and Description |
---|---|
URL |
MesosArtifactServer.addPath(Path path,
Path remoteFile)
Adds a path to the artifact server.
|
void |
MesosArtifactServer.removePath(Path remoteFile) |
scala.Option<URL> |
MesosArtifactServer.resolve(Path remoteFile) |
scala.Option<URL> |
MesosArtifactResolver.resolve(Path remoteFile) |
Constructor and Description |
---|
VirtualFileServerHandler(Path path) |
Modifier and Type | Method and Description |
---|---|
static List<PermanentBlobKey> |
BlobClient.uploadJarFiles(InetSocketAddress serverAddress,
Configuration clientConfig,
JobID jobId,
List<Path> jars)
Uploads the JAR files to the
PermanentBlobService of the BlobServer at the
given address with HA as configured. |
Modifier and Type | Field and Description |
---|---|
Path |
ContainerSpecification.Artifact.dest |
Path |
ContainerSpecification.Artifact.Builder.dest |
Path |
ContainerSpecification.Artifact.source |
Path |
ContainerSpecification.Artifact.Builder.source |
Modifier and Type | Method and Description |
---|---|
ContainerSpecification.Artifact.Builder |
ContainerSpecification.Artifact.Builder.setDest(Path dest) |
ContainerSpecification.Artifact.Builder |
ContainerSpecification.Artifact.Builder.setSource(Path source) |
Constructor and Description |
---|
Artifact(Path source,
Path dest,
boolean executable,
boolean cachable,
boolean extract) |
Constructor and Description |
---|
KeytabOverlay(Path keytab) |
Krb5ConfOverlay(Path krb5Conf) |
Modifier and Type | Method and Description |
---|---|
Map<String,Future<Path>> |
Environment.getDistributedCacheEntries() |
Modifier and Type | Method and Description |
---|---|
Future<Path> |
FileCache.createTmpFile(String name,
DistributedCache.DistributedCacheEntry entry,
JobID jobID)
If the file doesn't exists locally, it will copy the file to the temp directory.
|
Modifier and Type | Method and Description |
---|---|
static void |
FileCache.copy(Path sourcePath,
Path targetPath,
boolean executable) |
Modifier and Type | Method and Description |
---|---|
Path |
HadoopFileSystem.getHomeDirectory() |
Path |
HadoopFileStatus.getPath() |
Path |
HadoopFileSystem.getWorkingDirectory() |
Modifier and Type | Method and Description |
---|---|
HadoopDataOutputStream |
HadoopFileSystem.create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize) |
HadoopDataOutputStream |
HadoopFileSystem.create(Path f,
FileSystem.WriteMode overwrite) |
boolean |
HadoopFileSystem.delete(Path f,
boolean recursive) |
boolean |
HadoopFileSystem.exists(Path f) |
FileStatus |
HadoopFileSystem.getFileStatus(Path f) |
FileStatus[] |
HadoopFileSystem.listStatus(Path f) |
boolean |
HadoopFileSystem.mkdirs(Path f) |
HadoopDataInputStream |
HadoopFileSystem.open(Path f) |
HadoopDataInputStream |
HadoopFileSystem.open(Path f,
int bufferSize) |
boolean |
HadoopFileSystem.rename(Path src,
Path dst) |
Constructor and Description |
---|
FsNegativeRunningJobsRegistry(FileSystem fileSystem,
Path workingDirectory)
Creates a new registry that writes its files to the given FileSystem at
the given working directory path.
|
FsNegativeRunningJobsRegistry(Path workingDirectory)
Creates a new registry that writes to the FileSystem and working directory
denoted by the given path.
|
Modifier and Type | Method and Description |
---|---|
static Path |
FsJobArchivist.archiveJob(Path rootPath,
AccessExecutionGraph graph)
Deprecated.
only kept for legacy reasons
|
static Path |
FsJobArchivist.archiveJob(Path rootPath,
JobID jobId,
Collection<ArchivedJson> jsonToArchive)
Writes the given
AccessExecutionGraph to the FileSystem pointed to by
JobManagerOptions.ARCHIVE_DIR . |
Modifier and Type | Method and Description |
---|---|
static Path |
FsJobArchivist.archiveJob(Path rootPath,
AccessExecutionGraph graph)
Deprecated.
only kept for legacy reasons
|
static Path |
FsJobArchivist.archiveJob(Path rootPath,
JobID jobId,
Collection<ArchivedJson> jsonToArchive)
Writes the given
AccessExecutionGraph to the FileSystem pointed to by
JobManagerOptions.ARCHIVE_DIR . |
static Collection<ArchivedJson> |
FsJobArchivist.getArchivedJsons(Path file)
Reads the given archive file and returns a
Collection of contained ArchivedJson . |
Modifier and Type | Method and Description |
---|---|
List<Path> |
JobGraph.getUserJars()
Gets the list of assigned user jar paths.
|
Modifier and Type | Method and Description |
---|---|
void |
JobGraph.addJar(Path jar)
Adds the path of a JAR file required to run the job on a task manager.
|
Constructor and Description |
---|
DistributedRuntimeUDFContext(TaskInfo taskInfo,
ClassLoader userCodeClassLoader,
ExecutionConfig executionConfig,
Map<String,Future<Path>> cpTasks,
Map<String,Accumulator<?,?>> accumulators,
MetricGroup metrics) |
Modifier and Type | Field and Description |
---|---|
protected Path |
SnapshotDirectory.directory
This path describes the underlying directory for the snapshot.
|
Modifier and Type | Method and Description |
---|---|
Path |
SnapshotDirectory.getDirectory() |
Path |
DirectoryStateHandle.getDirectory() |
Modifier and Type | Method and Description |
---|---|
static SnapshotDirectory |
SnapshotDirectory.permanent(Path directory)
Creates a permanent snapshot directory for the given path, which will not delete the underlying directory in
SnapshotDirectory.cleanup() after SnapshotDirectory.completeSnapshotAndGetHandle() was called. |
static SnapshotDirectory |
SnapshotDirectory.temporary(Path directory)
Creates a temporary snapshot directory for the given path.
|
Constructor and Description |
---|
DirectoryStateHandle(Path directory) |
RetrievableStreamStateHandle(Path filePath,
long stateSize) |
Modifier and Type | Method and Description |
---|---|
protected static Path |
AbstractFsCheckpointStorage.createCheckpointDirectory(Path baseDirectory,
long checkpointId)
Creates the directory path for the data exclusive to a specific checkpoint.
|
static Path |
AbstractFsCheckpointStorage.decodePathFromReference(CheckpointStorageLocationReference reference)
Decodes the given reference into a path.
|
Path |
FsStateBackend.getBasePath()
Deprecated.
Deprecated in favor of
FsStateBackend.getCheckpointPath() . |
Path |
FsCheckpointStorageLocation.getCheckpointDirectory() |
protected static Path |
AbstractFsCheckpointStorage.getCheckpointDirectoryForJob(Path baseCheckpointPath,
JobID jobId)
Builds directory into which a specific job checkpoints, meaning the directory inside which
it creates the checkpoint-specific subdirectories.
|
Path |
FsStateBackend.getCheckpointPath()
Gets the base directory where all the checkpoints are stored.
|
Path |
AbstractFileStateBackend.getCheckpointPath()
Gets the checkpoint base directory.
|
Path |
FsCheckpointStorage.getCheckpointsDirectory() |
Path |
AbstractFsCheckpointStorage.getDefaultSavepointDirectory()
Gets the default directory for savepoints.
|
Path |
FileStateHandle.getFilePath()
Gets the path where this handle's state is stored.
|
Path |
FsCheckpointStorageLocation.getMetadataFilePath() |
Path |
AbstractFileStateBackend.getSavepointPath()
Gets the directory where savepoints are stored by default (when no custom path is given
to the savepoint trigger command).
|
Path |
FsCheckpointStorageLocation.getSharedStateDirectory() |
Path |
FsCheckpointStorageLocation.getTaskOwnedStateDirectory() |
Modifier and Type | Method and Description |
---|---|
protected static Path |
AbstractFsCheckpointStorage.createCheckpointDirectory(Path baseDirectory,
long checkpointId)
Creates the directory path for the data exclusive to a specific checkpoint.
|
protected CheckpointStorageLocation |
FsCheckpointStorage.createSavepointLocation(FileSystem fs,
Path location) |
protected abstract CheckpointStorageLocation |
AbstractFsCheckpointStorage.createSavepointLocation(FileSystem fs,
Path location) |
static CheckpointStorageLocationReference |
AbstractFsCheckpointStorage.encodePathAsReference(Path path)
Encodes the given path as a reference in bytes.
|
protected static Path |
AbstractFsCheckpointStorage.getCheckpointDirectoryForJob(Path baseCheckpointPath,
JobID jobId)
Builds directory into which a specific job checkpoints, meaning the directory inside which
it creates the checkpoint-specific subdirectories.
|
Constructor and Description |
---|
AbstractFileStateBackend(Path baseCheckpointPath,
Path baseSavepointPath)
Creates a backend with the given optional checkpoint- and savepoint base directories.
|
AbstractFileStateBackend(Path baseCheckpointPath,
Path baseSavepointPath,
Configuration configuration)
Creates a new backend using the given checkpoint-/savepoint directories, or the values defined in
the given configuration.
|
AbstractFsCheckpointStorage(JobID jobId,
Path defaultSavepointDirectory)
Creates a new checkpoint storage.
|
FileBasedStateOutputStream(FileSystem fileSystem,
Path path) |
FileStateHandle(Path filePath,
long stateSize)
Creates a new file state for the given file path.
|
FsCheckpointMetadataOutputStream(FileSystem fileSystem,
Path metadataFilePath,
Path exclusiveCheckpointDir) |
FsCheckpointStateOutputStream(Path basePath,
FileSystem fs,
int bufferSize,
int localStateThreshold) |
FsCheckpointStorage(Path checkpointBaseDirectory,
Path defaultSavepointDirectory,
JobID jobId,
int fileSizeThreshold) |
FsCheckpointStorageLocation(FileSystem fileSystem,
Path checkpointDir,
Path sharedStateDir,
Path taskOwnedStateDir,
CheckpointStorageLocationReference reference,
int fileStateSizeThreshold) |
FsCheckpointStreamFactory(FileSystem fileSystem,
Path checkpointDirectory,
Path sharedStateDirectory,
int fileStateSizeThreshold)
Creates a new stream factory that stores its checkpoint data in the file system and location
defined by the given Path.
|
FsCompletedCheckpointStorageLocation(FileSystem fs,
Path exclusiveCheckpointDir,
FileStateHandle metadataFileHandle,
String externalPointer) |
FsStateBackend(Path checkpointDataUri)
Creates a new state backend that stores its checkpoint data in the file system and location
defined by the given URI.
|
FsStateBackend(Path checkpointDataUri,
boolean asynchronousSnapshots)
Creates a new state backend that stores its checkpoint data in the file system and location
defined by the given URI.
|
Modifier and Type | Method and Description |
---|---|
protected CheckpointStorageLocation |
MemoryBackendCheckpointStorage.createSavepointLocation(FileSystem fs,
Path location) |
Constructor and Description |
---|
MemoryBackendCheckpointStorage(JobID jobId,
Path checkpointsBaseDirectory,
Path defaultSavepointLocation,
int maxStateSize)
Creates a new MemoryBackendCheckpointStorage.
|
PersistentMetadataCheckpointStorageLocation(FileSystem fileSystem,
Path checkpointDir,
int maxStateSize)
Creates a checkpoint storage persists metadata to a file system and stores state
in line in state handles with the metadata.
|
Modifier and Type | Method and Description |
---|---|
Map<String,Future<Path>> |
RuntimeEnvironment.getDistributedCacheEntries() |
Constructor and Description |
---|
RuntimeEnvironment(JobID jobId,
JobVertexID jobVertexId,
ExecutionAttemptID executionId,
ExecutionConfig executionConfig,
TaskInfo taskInfo,
Configuration jobConfiguration,
Configuration taskConfiguration,
ClassLoader userCodeClassLoader,
MemoryManager memManager,
IOManager ioManager,
BroadcastVariableManager bcVarManager,
TaskStateManager taskStateManager,
AccumulatorRegistry accumulatorRegistry,
TaskKvStateRegistry kvStateRegistry,
InputSplitProvider splitProvider,
Map<String,Future<Path>> distCacheEntries,
ResultPartitionWriter[] writers,
InputGate[] inputGates,
TaskEventDispatcher taskEventDispatcher,
CheckpointResponder checkpointResponder,
TaskManagerRuntimeInfo taskManagerInfo,
TaskMetricGroup metrics,
Task containingTask) |
Modifier and Type | Method and Description |
---|---|
static Path |
WebMonitorUtils.validateAndNormalizeUri(URI archiveDirUri)
Checks and normalizes the given URI.
|
Constructor and Description |
---|
FileSystemStateStorageHelper(Path rootPath,
String prefix) |
Constructor and Description |
---|
TimestampedFileInputSplit(long modificationTime,
int num,
Path file,
long start,
long length,
String[] hosts)
Creates a
TimestampedFileInputSplit based on the file modification time and
the rest of the information of the FileInputSplit , as returned by the
underlying filesystem. |
Constructor and Description |
---|
PythonEnvironmentFactory(String localTmpPath,
Path tmpDistributedDir,
String scriptName) |
Modifier and Type | Method and Description |
---|---|
static void |
TestStreamEnvironment.setAsContext(JobExecutor jobExecutor,
int parallelism,
Collection<Path> jarFiles,
Collection<URL> classpaths)
Sets the streaming context environment to a TestStreamEnvironment that runs its programs on
the given cluster with the given default parallelism and the specified jar files and class
paths.
|
Constructor and Description |
---|
TestStreamEnvironment(JobExecutor jobExecutor,
int parallelism,
Collection<Path> jarFiles,
Collection<URL> classPaths) |
Modifier and Type | Method and Description |
---|---|
static void |
TestEnvironment.setAsContext(JobExecutor jobExecutor,
int parallelism,
Collection<Path> jarFiles,
Collection<URL> classPaths)
Sets the current
ExecutionEnvironment to be a TestEnvironment . |
Constructor and Description |
---|
TestEnvironment(JobExecutor jobExecutor,
int parallelism,
boolean isObjectReuseEnabled,
Collection<Path> jarFiles,
Collection<URL> classPaths) |
Modifier and Type | Method and Description |
---|---|
static boolean |
FileUtils.deletePathIfEmpty(FileSystem fileSystem,
Path path)
Deletes the path if it is empty.
|
Modifier and Type | Field and Description |
---|---|
protected Path |
YarnHighAvailabilityServices.haDataDirectory
The directory for HA persistent data.
|
protected Path |
YarnHighAvailabilityServices.workingDirectory
The working directory of this YARN application.
|
Copyright © 2014–2019 The Apache Software Foundation. All rights reserved.