Constructor and Description |
---|
DistributedCache(Map<String,Future<Path>> cacheCopyTasks) |
Constructor and Description |
---|
AbstractRuntimeUDFContext(TaskInfo taskInfo,
ClassLoader userCodeClassLoader,
ExecutionConfig executionConfig,
Map<String,Accumulator<?,?>> accumulators,
Map<String,Future<Path>> cpTasks,
MetricGroup metrics) |
RuntimeUDFContext(TaskInfo taskInfo,
ClassLoader userCodeClassLoader,
ExecutionConfig executionConfig,
Map<String,Future<Path>> cpTasks,
Map<String,Accumulator<?,?>> accumulators,
MetricGroup metrics) |
Modifier and Type | Field and Description |
---|---|
protected Path |
FileInputFormat.filePath
The path to the file that contains the input.
|
protected Path |
FileOutputFormat.outputFilePath
The path of the file to be written.
|
Modifier and Type | Method and Description |
---|---|
Path |
FileInputFormat.getFilePath() |
Path |
FileOutputFormat.getOutputFilePath() |
Modifier and Type | Method and Description |
---|---|
boolean |
GlobFilePathFilter.filterPath(Path filePath) |
abstract boolean |
FilePathFilter.filterPath(Path filePath)
Returns
true if the filePath given is to be
ignored when processing a directory, e.g. |
boolean |
FilePathFilter.DefaultFilter.filterPath(Path filePath) |
protected FileInputFormat.FileBaseStatistics |
FileInputFormat.getFileStats(FileInputFormat.FileBaseStatistics cachedStats,
Path filePath,
FileSystem fs,
ArrayList<FileStatus> files) |
void |
FileInputFormat.setFilePath(Path filePath) |
void |
FileOutputFormat.setOutputFilePath(Path path) |
Constructor and Description |
---|
DelimitedInputFormat(Path filePath,
Configuration configuration) |
FileInputFormat(Path filePath) |
FileOutputFormat(Path outputPath) |
GenericCsvInputFormat(Path filePath) |
Modifier and Type | Method and Description |
---|---|
Path |
CsvReader.getFilePath() |
Constructor and Description |
---|
AvroInputFormat(Path filePath,
Class<E> type) |
AvroOutputFormat(Path filePath,
Class<E> type) |
CsvInputFormat(Path filePath) |
CsvOutputFormat(Path outputPath)
Creates an instance of CsvOutputFormat.
|
CsvOutputFormat(Path outputPath,
String fieldDelimiter)
Creates an instance of CsvOutputFormat.
|
CsvOutputFormat(Path outputPath,
String recordDelimiter,
String fieldDelimiter)
Creates an instance of CsvOutputFormat.
|
CsvReader(Path filePath,
ExecutionEnvironment executionContext) |
PojoCsvInputFormat(Path filePath,
PojoTypeInfo<OUT> pojoTypeInfo) |
PojoCsvInputFormat(Path filePath,
PojoTypeInfo<OUT> pojoTypeInfo,
boolean[] includedFieldsMask) |
PojoCsvInputFormat(Path filePath,
PojoTypeInfo<OUT> pojoTypeInfo,
int[] includedFieldsMask) |
PojoCsvInputFormat(Path filePath,
PojoTypeInfo<OUT> pojoTypeInfo,
String[] fieldNames) |
PojoCsvInputFormat(Path filePath,
PojoTypeInfo<OUT> pojoTypeInfo,
String[] fieldNames,
boolean[] includedFieldsMask) |
PojoCsvInputFormat(Path filePath,
PojoTypeInfo<OUT> pojoTypeInfo,
String[] fieldNames,
int[] includedFieldsMask) |
PojoCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
PojoTypeInfo<OUT> pojoTypeInfo) |
PojoCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
PojoTypeInfo<OUT> pojoTypeInfo,
boolean[] includedFieldsMask) |
PojoCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
PojoTypeInfo<OUT> pojoTypeInfo,
int[] includedFieldsMask) |
PojoCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
PojoTypeInfo<OUT> pojoTypeInfo,
String[] fieldNames) |
PojoCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
PojoTypeInfo<OUT> pojoTypeInfo,
String[] fieldNames,
boolean[] includedFieldsMask) |
PojoCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
PojoTypeInfo<OUT> pojoTypeInfo,
String[] fieldNames,
int[] includedFieldsMask) |
PrimitiveInputFormat(Path filePath,
Class<OT> primitiveClass) |
PrimitiveInputFormat(Path filePath,
String delimiter,
Class<OT> primitiveClass) |
RowCsvInputFormat(Path filePath,
TypeInformation[] fieldTypes) |
RowCsvInputFormat(Path filePath,
TypeInformation[] fieldTypes,
boolean emptyColumnAsNull) |
RowCsvInputFormat(Path filePath,
TypeInformation[] fieldTypes,
int[] selectedFields) |
RowCsvInputFormat(Path filePath,
TypeInformation[] fieldTypes,
String lineDelimiter,
String fieldDelimiter) |
RowCsvInputFormat(Path filePath,
TypeInformation[] fieldTypes,
String lineDelimiter,
String fieldDelimiter,
int[] selectedFields) |
RowCsvInputFormat(Path filePath,
TypeInformation[] fieldTypeInfos,
String lineDelimiter,
String fieldDelimiter,
int[] selectedFields,
boolean emptyColumnAsNull) |
TextInputFormat(Path filePath) |
TextOutputFormat(Path outputPath) |
TextOutputFormat(Path outputPath,
String charset) |
TextValueInputFormat(Path filePath) |
TupleCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
TupleTypeInfoBase<OUT> tupleTypeInfo) |
TupleCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
TupleTypeInfoBase<OUT> tupleTypeInfo,
boolean[] includedFieldsMask) |
TupleCsvInputFormat(Path filePath,
String lineDelimiter,
String fieldDelimiter,
TupleTypeInfoBase<OUT> tupleTypeInfo,
int[] includedFieldsMask) |
TupleCsvInputFormat(Path filePath,
TupleTypeInfoBase<OUT> tupleTypeInfo) |
TupleCsvInputFormat(Path filePath,
TupleTypeInfoBase<OUT> tupleTypeInfo,
boolean[] includedFieldsMask) |
TupleCsvInputFormat(Path filePath,
TupleTypeInfoBase<OUT> tupleTypeInfo,
int[] includedFieldsMask) |
Constructor and Description |
---|
ScalaCsvOutputFormat(Path outputPath)
Creates an instance of CsvOutputFormat.
|
ScalaCsvOutputFormat(Path outputPath,
String fieldDelimiter)
Creates an instance of CsvOutputFormat.
|
ScalaCsvOutputFormat(Path outputPath,
String recordDelimiter,
String fieldDelimiter)
Creates an instance of CsvOutputFormat.
|
Modifier and Type | Method and Description |
---|---|
Path |
SafetyNetWrapperFileSystem.getHomeDirectory() |
abstract Path |
FileSystem.getHomeDirectory()
Returns the path of the user's home directory in this file system.
|
Path |
Path.getParent()
Returns the parent of a path, i.e., everything that precedes the last separator
or
null if at root. |
Path |
FileStatus.getPath()
Returns the corresponding Path to the FileStatus.
|
Path |
FileInputSplit.getPath()
Returns the path of the file containing this split's data.
|
Path |
SafetyNetWrapperFileSystem.getWorkingDirectory() |
abstract Path |
FileSystem.getWorkingDirectory()
Returns the path of the file system's current working directory.
|
Path |
Path.makeQualified(FileSystem fs)
Returns a qualified path object.
|
Path |
Path.suffix(String suffix)
Adds a suffix to the final name in the path.
|
Modifier and Type | Method and Description |
---|---|
FSDataOutputStream |
SafetyNetWrapperFileSystem.create(Path f,
boolean overwrite) |
abstract FSDataOutputStream |
FileSystem.create(Path f,
boolean overwrite)
Opens an FSDataOutputStream at the indicated Path.
|
FSDataOutputStream |
SafetyNetWrapperFileSystem.create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize) |
abstract FSDataOutputStream |
FileSystem.create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize)
Opens an FSDataOutputStream at the indicated Path.
|
boolean |
SafetyNetWrapperFileSystem.delete(Path f,
boolean recursive) |
abstract boolean |
FileSystem.delete(Path f,
boolean recursive)
Delete a file.
|
boolean |
SafetyNetWrapperFileSystem.exists(Path f) |
boolean |
FileSystem.exists(Path f)
Check if exists.
|
FileStatus |
SafetyNetWrapperFileSystem.getFileStatus(Path f) |
abstract FileStatus |
FileSystem.getFileStatus(Path f)
Return a file status object that represents the path.
|
boolean |
SafetyNetWrapperFileSystem.initOutPathDistFS(Path outPath,
FileSystem.WriteMode writeMode,
boolean createDirectory) |
boolean |
FileSystem.initOutPathDistFS(Path outPath,
FileSystem.WriteMode writeMode,
boolean createDirectory)
Initializes output directories on distributed file systems according to the given write mode.
|
boolean |
SafetyNetWrapperFileSystem.initOutPathLocalFS(Path outPath,
FileSystem.WriteMode writeMode,
boolean createDirectory) |
boolean |
FileSystem.initOutPathLocalFS(Path outPath,
FileSystem.WriteMode writeMode,
boolean createDirectory)
Initializes output directories on local file systems according to the given write mode.
|
FileStatus[] |
SafetyNetWrapperFileSystem.listStatus(Path f) |
abstract FileStatus[] |
FileSystem.listStatus(Path f)
List the statuses of the files/directories in the given path if the path is
a directory.
|
boolean |
SafetyNetWrapperFileSystem.mkdirs(Path f) |
abstract boolean |
FileSystem.mkdirs(Path f)
Make the given file and all non-existent parents into directories.
|
FSDataInputStream |
SafetyNetWrapperFileSystem.open(Path f) |
abstract FSDataInputStream |
FileSystem.open(Path f)
Opens an FSDataInputStream at the indicated Path.
|
FSDataInputStream |
SafetyNetWrapperFileSystem.open(Path f,
int bufferSize) |
abstract FSDataInputStream |
FileSystem.open(Path f,
int bufferSize)
Opens an FSDataInputStream at the indicated Path.
|
boolean |
SafetyNetWrapperFileSystem.rename(Path src,
Path dst) |
abstract boolean |
FileSystem.rename(Path src,
Path dst)
Renames the file/directory src to dst.
|
Constructor and Description |
---|
FileInputSplit(int num,
Path file,
long start,
long length,
String[] hosts)
Constructs a split with host information.
|
Path(Path parent,
Path child)
Resolve a child path against a parent path.
|
Path(Path parent,
String child)
Resolve a child path against a parent path.
|
Path(String parent,
Path child)
Resolve a child path against a parent path.
|
Modifier and Type | Method and Description |
---|---|
Path |
LocalFileSystem.getHomeDirectory() |
Path |
LocalFileStatus.getPath() |
Path |
LocalFileSystem.getWorkingDirectory() |
Modifier and Type | Method and Description |
---|---|
FSDataOutputStream |
LocalFileSystem.create(Path f,
boolean overwrite) |
FSDataOutputStream |
LocalFileSystem.create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize) |
boolean |
LocalFileSystem.delete(Path f,
boolean recursive) |
FileStatus |
LocalFileSystem.getFileStatus(Path f) |
FileStatus[] |
LocalFileSystem.listStatus(Path f) |
boolean |
LocalFileSystem.mkdirs(Path f)
Recursively creates the directory specified by the provided path.
|
FSDataInputStream |
LocalFileSystem.open(Path f) |
FSDataInputStream |
LocalFileSystem.open(Path f,
int bufferSize) |
boolean |
LocalFileSystem.rename(Path src,
Path dst) |
Modifier and Type | Method and Description |
---|---|
Path |
FileCopyTask.getPath() |
Constructor and Description |
---|
FileCopyTask(Path path,
String relativePath) |
Constructor and Description |
---|
GraphCsvReader(Path edgePath,
ExecutionEnvironment context) |
GraphCsvReader(Path edgePath,
MapFunction<K,VV> mapper,
ExecutionEnvironment context) |
GraphCsvReader(Path vertexPath,
Path edgePath,
ExecutionEnvironment context) |
Modifier and Type | Method and Description |
---|---|
URL |
MesosArtifactServer.addPath(Path path,
Path remoteFile)
Adds a path to the artifact server.
|
void |
MesosArtifactServer.removePath(Path remoteFile) |
scala.Option<URL> |
MesosArtifactServer.resolve(Path remoteFile) |
scala.Option<URL> |
MesosArtifactResolver.resolve(Path remoteFile) |
Constructor and Description |
---|
VirtualFileServerHandler(Path path) |
Modifier and Type | Method and Description |
---|---|
Path |
AbstractFileStateHandle.getFilePath()
Deprecated.
Gets the path where this handle's state is stored.
|
Constructor and Description |
---|
AbstractFileStateHandle(Path filePath)
Deprecated.
Creates a new file state for the given file path.
|
AbstractFsStateSnapshot(TypeSerializer<K> keySerializer,
TypeSerializer<N> namespaceSerializer,
TypeSerializer<SV> stateSerializer,
SD stateDesc,
Path filePath)
Deprecated.
|
FileSerializableStateHandle(Path filePath)
Deprecated.
Creates a new FileSerializableStateHandle pointing to state at the given file path.
|
FileStreamStateHandle(Path filePath)
Deprecated.
Creates a new FileStreamStateHandle pointing to state at the given file path.
|
Snapshot(TypeSerializer<K> keySerializer,
TypeSerializer<N> namespaceSerializer,
TypeSerializer<ACC> stateSerializer,
FoldingStateDescriptor<T,ACC> stateDescs,
Path filePath) |
Snapshot(TypeSerializer<K> keySerializer,
TypeSerializer<N> namespaceSerializer,
TypeSerializer<ArrayList<V>> stateSerializer,
ListStateDescriptor<V> stateDescs,
Path filePath) |
Snapshot(TypeSerializer<K> keySerializer,
TypeSerializer<N> namespaceSerializer,
TypeSerializer<V> stateSerializer,
ReducingStateDescriptor<V> stateDescs,
Path filePath) |
Snapshot(TypeSerializer<K> keySerializer,
TypeSerializer<N> namespaceSerializer,
TypeSerializer<V> stateSerializer,
ValueStateDescriptor<V> stateDescs,
Path filePath) |
Modifier and Type | Method and Description |
---|---|
static List<BlobKey> |
BlobClient.uploadJarFiles(ActorGateway jobManager,
scala.concurrent.duration.FiniteDuration askTimeout,
Configuration clientConfig,
List<Path> jars)
Retrieves the
BlobServer address from the JobManager and uploads
the JAR files to it. |
static List<BlobKey> |
BlobClient.uploadJarFiles(InetSocketAddress serverAddress,
Configuration clientConfig,
List<Path> jars)
Uploads the JAR files to a
BlobServer at the given address. |
Modifier and Type | Field and Description |
---|---|
Path |
ContainerSpecification.Artifact.dest |
Path |
ContainerSpecification.Artifact.Builder.dest |
Path |
ContainerSpecification.Artifact.source |
Path |
ContainerSpecification.Artifact.Builder.source |
Modifier and Type | Method and Description |
---|---|
ContainerSpecification.Artifact.Builder |
ContainerSpecification.Artifact.Builder.setDest(Path dest) |
ContainerSpecification.Artifact.Builder |
ContainerSpecification.Artifact.Builder.setSource(Path source) |
Constructor and Description |
---|
Artifact(Path source,
Path dest,
boolean executable,
boolean cachable,
boolean extract) |
Constructor and Description |
---|
KeytabOverlay(Path keytab) |
Krb5ConfOverlay(Path krb5Conf) |
Modifier and Type | Method and Description |
---|---|
Map<String,Future<Path>> |
Environment.getDistributedCacheEntries() |
Modifier and Type | Method and Description |
---|---|
Future<Path> |
FileCache.createTmpFile(String name,
DistributedCache.DistributedCacheEntry entry,
JobID jobID)
If the file doesn't exists locally, it will copy the file to the temp directory.
|
Modifier and Type | Method and Description |
---|---|
static void |
FileCache.copy(Path sourcePath,
Path targetPath,
boolean executable) |
Modifier and Type | Method and Description |
---|---|
Path |
HadoopFileSystem.getHomeDirectory() |
Path |
HadoopFileStatus.getPath() |
Path |
HadoopFileSystem.getWorkingDirectory() |
Modifier and Type | Method and Description |
---|---|
HadoopDataOutputStream |
HadoopFileSystem.create(Path f,
boolean overwrite) |
HadoopDataOutputStream |
HadoopFileSystem.create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize) |
boolean |
HadoopFileSystem.delete(Path f,
boolean recursive) |
FileStatus |
HadoopFileSystem.getFileStatus(Path f) |
FileStatus[] |
HadoopFileSystem.listStatus(Path f) |
boolean |
HadoopFileSystem.mkdirs(Path f) |
HadoopDataInputStream |
HadoopFileSystem.open(Path f) |
HadoopDataInputStream |
HadoopFileSystem.open(Path f,
int bufferSize) |
boolean |
HadoopFileSystem.rename(Path src,
Path dst) |
Modifier and Type | Method and Description |
---|---|
Path |
MapRFileSystem.getHomeDirectory() |
Path |
MapRFileSystem.getWorkingDirectory() |
Modifier and Type | Method and Description |
---|---|
FSDataOutputStream |
MapRFileSystem.create(Path f,
boolean overwrite) |
FSDataOutputStream |
MapRFileSystem.create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize) |
boolean |
MapRFileSystem.delete(Path f,
boolean recursive) |
FileStatus |
MapRFileSystem.getFileStatus(Path f) |
FileStatus[] |
MapRFileSystem.listStatus(Path f) |
boolean |
MapRFileSystem.mkdirs(Path f) |
FSDataInputStream |
MapRFileSystem.open(Path f) |
FSDataInputStream |
MapRFileSystem.open(Path f,
int bufferSize) |
boolean |
MapRFileSystem.rename(Path src,
Path dst) |
Modifier and Type | Method and Description |
---|---|
void |
JobGraph.addJar(Path jar)
Adds the path of a JAR file required to run the job on a task manager.
|
Constructor and Description |
---|
DistributedRuntimeUDFContext(TaskInfo taskInfo,
ClassLoader userCodeClassLoader,
ExecutionConfig executionConfig,
Map<String,Future<Path>> cpTasks,
Map<String,Accumulator<?,?>> accumulators,
MetricGroup metrics) |
Constructor and Description |
---|
RetrievableStreamStateHandle(Path filePath,
long stateSize) |
Modifier and Type | Method and Description |
---|---|
Path |
FsStateBackend.getBasePath()
Gets the base directory where all state-containing files are stored.
|
Path |
FileStateHandle.getFilePath()
Gets the path where this handle's state is stored.
|
static Path |
FsStateBackend.validateAndNormalizeUri(URI checkpointDataUri)
Checks and normalizes the checkpoint data URI.
|
Constructor and Description |
---|
FileStateHandle(Path filePath,
long stateSize)
Creates a new file state for the given file path.
|
FsCheckpointStateOutputStream(Path basePath,
FileSystem fs,
int bufferSize,
int localStateThreshold) |
FsCheckpointStreamFactory(Path checkpointDataUri,
JobID jobId,
int fileStateSizeThreshold)
Creates a new state backend that stores its checkpoint data in the file system and location
defined by the given URI.
|
FsStateBackend(Path checkpointDataUri)
Creates a new state backend that stores its checkpoint data in the file system and location
defined by the given URI.
|
Modifier and Type | Method and Description |
---|---|
Path |
AsyncFsStateBackend.getBasePath()
Gets the base directory where all state-containing files are stored.
|
static Path |
AsyncFsStateBackend.validateAndNormalizeUri(URI checkpointDataUri)
Checks and normalizes the checkpoint data URI.
|
Constructor and Description |
---|
AsyncFsStateBackend(Path checkpointDataUri)
Creates a new state backend that stores its checkpoint data in the file system and location
defined by the given URI.
|
Modifier and Type | Method and Description |
---|---|
Map<String,Future<Path>> |
RuntimeEnvironment.getDistributedCacheEntries() |
Constructor and Description |
---|
RuntimeEnvironment(JobID jobId,
JobVertexID jobVertexId,
ExecutionAttemptID executionId,
ExecutionConfig executionConfig,
TaskInfo taskInfo,
Configuration jobConfiguration,
Configuration taskConfiguration,
ClassLoader userCodeClassLoader,
MemoryManager memManager,
IOManager ioManager,
BroadcastVariableManager bcVarManager,
AccumulatorRegistry accumulatorRegistry,
TaskKvStateRegistry kvStateRegistry,
InputSplitProvider splitProvider,
Map<String,Future<Path>> distCacheEntries,
ResultPartitionWriter[] writers,
InputGate[] inputGates,
CheckpointResponder checkpointResponder,
TaskManagerRuntimeInfo taskManagerInfo,
TaskMetricGroup metrics,
Task containingTask) |
Constructor and Description |
---|
FileSystemStateStorageHelper(Path rootPath,
String prefix) |
Constructor and Description |
---|
TimestampedFileInputSplit(long modificationTime,
int num,
Path file,
long start,
long length,
String[] hosts)
Creates a
TimestampedFileInputSplit based on the file modification time and
the rest of the information of the FileInputSplit , as returned by the
underlying filesystem. |
Modifier and Type | Method and Description |
---|---|
static boolean |
FileUtils.deletePathIfEmpty(FileSystem fileSystem,
Path path)
Deletes the path if it is empty.
|
Copyright © 2014–2017 The Apache Software Foundation. All rights reserved.