@Internal public class SafetyNetWrapperFileSystem extends FileSystem implements WrappingProxy<FileSystem>, PathsCopyingFileSystem
WrappingProxy
around FileSystem
which (i) wraps all opened streams as
ClosingFSDataInputStream
or ClosingFSDataOutputStream
and (ii) registers them to
a SafetyNetCloseableRegistry
.
Streams obtained by this are therefore managed by the SafetyNetCloseableRegistry
to
prevent resource leaks from unclosed streams.
FileSystem.FSKey, FileSystem.WriteMode
PathsCopyingFileSystem.CopyRequest
Constructor and Description |
---|
SafetyNetWrapperFileSystem(FileSystem unsafeFileSystem,
SafetyNetCloseableRegistry registry) |
Modifier and Type | Method and Description |
---|---|
boolean |
canCopyPaths(Path source,
Path destination)
Tells if this
FileSystem supports an optimised way to directly copy between given
paths. |
void |
copyFiles(List<PathsCopyingFileSystem.CopyRequest> requests,
ICloseableRegistry closeableRegistry)
List of
PathsCopyingFileSystem.CopyRequest to copy in batch by this PathsCopyingFileSystem . |
FSDataOutputStream |
create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize)
Opens an FSDataOutputStream at the indicated Path.
|
FSDataOutputStream |
create(Path f,
FileSystem.WriteMode overwrite)
Opens an FSDataOutputStream to a new file at the given path.
|
RecoverableWriter |
createRecoverableWriter()
Creates a new
RecoverableWriter . |
boolean |
delete(Path f,
boolean recursive)
Delete a file.
|
boolean |
exists(Path f)
Check if exists.
|
long |
getDefaultBlockSize()
Return the number of bytes that large input files should be optimally be split into to
minimize I/O time.
|
BlockLocation[] |
getFileBlockLocations(FileStatus file,
long start,
long len)
Return an array containing hostnames, offset and size of portions of the given file.
|
FileStatus |
getFileStatus(Path f)
Return a file status object that represents the path.
|
Path |
getHomeDirectory()
Returns the path of the user's home directory in this file system.
|
FileSystemKind |
getKind()
Gets a description of the characteristics of this file system.
|
URI |
getUri()
Returns a URI whose scheme and authority identify this file system.
|
Path |
getWorkingDirectory()
Returns the path of the file system's current working directory.
|
FileSystem |
getWrappedDelegate() |
boolean |
initOutPathDistFS(Path outPath,
FileSystem.WriteMode writeMode,
boolean createDirectory)
Initializes output directories on distributed file systems according to the given write mode.
|
boolean |
initOutPathLocalFS(Path outPath,
FileSystem.WriteMode writeMode,
boolean createDirectory)
Initializes output directories on local file systems according to the given write mode.
|
boolean |
isDistributedFS()
Returns true if this is a distributed file system.
|
FileStatus[] |
listStatus(Path f)
List the statuses of the files/directories in the given path if the path is a directory.
|
boolean |
mkdirs(Path f)
Make the given file and all non-existent parents into directories.
|
FSDataInputStream |
open(Path f)
Opens an FSDataInputStream at the indicated Path.
|
FSDataInputStream |
open(Path f,
int bufferSize)
Opens an FSDataInputStream at the indicated Path.
|
boolean |
rename(Path src,
Path dst)
Renames the file/directory src to dst.
|
create, get, getDefaultFsUri, getLocalFileSystem, getUnguardedFileSystem, initialize, initialize
public SafetyNetWrapperFileSystem(FileSystem unsafeFileSystem, SafetyNetCloseableRegistry registry)
public void copyFiles(List<PathsCopyingFileSystem.CopyRequest> requests, ICloseableRegistry closeableRegistry) throws IOException
PathsCopyingFileSystem
PathsCopyingFileSystem.CopyRequest
to copy in batch by this PathsCopyingFileSystem
. In case
of an exception some files might have been already copied fully or partially. Caller should
clean this up. Copy can be interrupted by the CloseableRegistry
.copyFiles
in interface PathsCopyingFileSystem
IOException
public boolean canCopyPaths(Path source, Path destination) throws IOException
IFileSystem
FileSystem
supports an optimised way to directly copy between given
paths. In other words if it implements PathsCopyingFileSystem
.
At least one of, either source or destination belongs to this IFileSystem
. One of
them can point to the local file system. In other words this request can correspond to
either: downloading a file from the remote file system, uploading a file to the remote file
system or duplicating a file in the remote file system.
canCopyPaths
in interface IFileSystem
canCopyPaths
in interface PathsCopyingFileSystem
source
- The path of the source file to duplicatedestination
- The path where to duplicate the source fileIFileSystem
can perform this operation more quickly compared to
the generic code path of using streams.IOException
public Path getWorkingDirectory()
IFileSystem
getWorkingDirectory
in interface IFileSystem
getWorkingDirectory
in class FileSystem
public Path getHomeDirectory()
IFileSystem
getHomeDirectory
in interface IFileSystem
getHomeDirectory
in class FileSystem
public URI getUri()
IFileSystem
getUri
in interface IFileSystem
getUri
in class FileSystem
public FileStatus getFileStatus(Path f) throws IOException
IFileSystem
getFileStatus
in interface IFileSystem
getFileStatus
in class FileSystem
f
- The path we want information fromFileNotFoundException
- when the path does not exist; IOException see specific
implementationIOException
public RecoverableWriter createRecoverableWriter() throws IOException
IFileSystem
RecoverableWriter
. A recoverable writer creates streams that can
persist and recover their intermediate state. Persisting and recovering intermediate state is
a core building block for writing to files that span multiple checkpoints.
The returned object can act as a shared factory to open and recover multiple streams.
This method is optional on file systems and various file system implementations may not
support this method, throwing an UnsupportedOperationException
.
createRecoverableWriter
in interface IFileSystem
createRecoverableWriter
in class FileSystem
IOException
- Thrown, if the recoverable writer cannot be instantiated.public BlockLocation[] getFileBlockLocations(FileStatus file, long start, long len) throws IOException
IFileSystem
getFileBlockLocations
in interface IFileSystem
getFileBlockLocations
in class FileSystem
IOException
public FSDataInputStream open(Path f, int bufferSize) throws IOException
IFileSystem
open
in interface IFileSystem
open
in class FileSystem
f
- the file name to openbufferSize
- the size of the buffer to be used.IOException
public FSDataInputStream open(Path f) throws IOException
IFileSystem
open
in interface IFileSystem
open
in class FileSystem
f
- the file to openIOException
public long getDefaultBlockSize()
FileSystem
getDefaultBlockSize
in class FileSystem
public FileStatus[] listStatus(Path f) throws IOException
IFileSystem
listStatus
in interface IFileSystem
listStatus
in class FileSystem
f
- given pathIOException
public boolean exists(Path f) throws IOException
IFileSystem
exists
in interface IFileSystem
exists
in class FileSystem
f
- source fileIOException
public boolean delete(Path f, boolean recursive) throws IOException
IFileSystem
delete
in interface IFileSystem
delete
in class FileSystem
f
- the path to deleterecursive
- if path is a directory and set to true
, the directory is
deleted else throws an exception. In case of a file the recursive can be set to either
true
or false
true
if delete is successful, false
otherwiseIOException
public boolean mkdirs(Path f) throws IOException
IFileSystem
mkdirs
in interface IFileSystem
mkdirs
in class FileSystem
f
- the directory/directories to be createdtrue
if at least one new directory has been created, false
otherwiseIOException
- thrown if an I/O error occurs while creating the directorypublic FSDataOutputStream create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize) throws IOException
FileSystem
This method is deprecated, because most of its parameters are ignored by most file systems. To control for example the replication factor and block size in the Hadoop Distributed File system, make sure that the respective Hadoop configuration file is either linked from the Flink configuration, or in the classpath of either Flink or the user code.
create
in class FileSystem
f
- the file name to openoverwrite
- if a file with this name already exists, then if true, the file will be
overwritten, and if false an error will be thrown.bufferSize
- the size of the buffer to be used.replication
- required block replication for the file.blockSize
- the size of the file blocksIOException
- Thrown, if the stream could not be opened because of an I/O, or because a
file already exists at that path and the write mode indicates to not overwrite the file.public FSDataOutputStream create(Path f, FileSystem.WriteMode overwrite) throws IOException
IFileSystem
If the file already exists, the behavior depends on the given WriteMode
. If the
mode is set to FileSystem.WriteMode.NO_OVERWRITE
, then this method fails with an
exception.
create
in interface IFileSystem
create
in class FileSystem
f
- The file path to write tooverwrite
- The action to take if a file or directory already exists at the given
path.IOException
- Thrown, if the stream could not be opened because of an I/O, or because a
file already exists at that path and the write mode indicates to not overwrite the file.public boolean rename(Path src, Path dst) throws IOException
IFileSystem
rename
in interface IFileSystem
rename
in class FileSystem
src
- the file/directory to renamedst
- the new name of the file/directorytrue
if the renaming was successful, false
otherwiseIOException
public boolean initOutPathLocalFS(Path outPath, FileSystem.WriteMode writeMode, boolean createDirectory) throws IOException
IFileSystem
Files contained in an existing directory are not deleted, because multiple instances of a DataSinkTask might call this function at the same time and hence might perform concurrent delete operations on the file system (possibly deleting output files of concurrently running tasks). Since concurrent DataSinkTasks are not aware of each other, coordination of delete and create operations would be difficult.
initOutPathLocalFS
in interface IFileSystem
initOutPathLocalFS
in class FileSystem
outPath
- Output path that should be prepared.writeMode
- Write mode to consider.createDirectory
- True, to initialize a directory at the given path, false to prepare
space for a file.IOException
- Thrown, if any of the file system access operations failed.public boolean initOutPathDistFS(Path outPath, FileSystem.WriteMode writeMode, boolean createDirectory) throws IOException
IFileSystem
WriteMode.NO_OVERWRITE & parallel output: - A directory is created if the output path does not exist. - An existing file or directory raises an exception.
WriteMode.NO_OVERWRITE & NONE parallel output: - An existing file or directory raises an exception.
WriteMode.OVERWRITE & parallel output: - A directory is created if the output path does not exist. - An existing directory and its content is deleted and a new directory is created. - An existing file is deleted and replaced by a new directory.
WriteMode.OVERWRITE & NONE parallel output: - An existing file or directory is deleted and replaced by a new directory.
initOutPathDistFS
in interface IFileSystem
initOutPathDistFS
in class FileSystem
outPath
- Output path that should be prepared.writeMode
- Write mode to consider.createDirectory
- True, to initialize a directory at the given path, false otherwise.IOException
- Thrown, if any of the file system access operations failed.public boolean isDistributedFS()
IFileSystem
isDistributedFS
in interface IFileSystem
isDistributedFS
in class FileSystem
public FileSystemKind getKind()
FileSystem
getKind
in interface IFileSystem
getKind
in class FileSystem
public FileSystem getWrappedDelegate()
getWrappedDelegate
in interface WrappingProxy<FileSystem>
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.