Class SafetyNetWrapperFileSystem

    • Method Detail

      • canCopyPaths

        public boolean canCopyPaths​(Path source,
                                    Path destination)
                             throws IOException
        Description copied from interface: IFileSystem
        Tells if this FileSystem supports an optimised way to directly copy between given paths. In other words if it implements PathsCopyingFileSystem.

        At least one of, either source or destination belongs to this IFileSystem. One of them can point to the local file system. In other words this request can correspond to either: downloading a file from the remote file system, uploading a file to the remote file system or duplicating a file in the remote file system.

        Specified by:
        canCopyPaths in interface IFileSystem
        Specified by:
        canCopyPaths in interface PathsCopyingFileSystem
        Parameters:
        source - The path of the source file to duplicate
        destination - The path where to duplicate the source file
        Returns:
        true, if this IFileSystem can perform this operation more quickly compared to the generic code path of using streams.
        Throws:
        IOException
      • getHomeDirectory

        public Path getHomeDirectory()
        Description copied from interface: IFileSystem
        Returns the path of the user's home directory in this file system.
        Specified by:
        getHomeDirectory in interface IFileSystem
        Specified by:
        getHomeDirectory in class FileSystem
        Returns:
        the path of the user's home directory in this file system.
      • getUri

        public URI getUri()
        Description copied from interface: IFileSystem
        Returns a URI whose scheme and authority identify this file system.
        Specified by:
        getUri in interface IFileSystem
        Specified by:
        getUri in class FileSystem
        Returns:
        a URI whose scheme and authority identify this file system
      • createRecoverableWriter

        public RecoverableWriter createRecoverableWriter()
                                                  throws IOException
        Description copied from interface: IFileSystem
        Creates a new RecoverableWriter. A recoverable writer creates streams that can persist and recover their intermediate state. Persisting and recovering intermediate state is a core building block for writing to files that span multiple checkpoints.

        The returned object can act as a shared factory to open and recover multiple streams.

        This method is optional on file systems and various file system implementations may not support this method, throwing an UnsupportedOperationException.

        Specified by:
        createRecoverableWriter in interface IFileSystem
        Overrides:
        createRecoverableWriter in class FileSystem
        Returns:
        A RecoverableWriter for this file system.
        Throws:
        IOException - Thrown, if the recoverable writer cannot be instantiated.
      • getFileBlockLocations

        public BlockLocation[] getFileBlockLocations​(FileStatus file,
                                                     long start,
                                                     long len)
                                              throws IOException
        Description copied from interface: IFileSystem
        Return an array containing hostnames, offset and size of portions of the given file. For a nonexistent file or regions, null will be returned. This call is most helpful with DFS, where it returns hostnames of machines that contain the given file. The FileSystem will simply return an elt containing 'localhost'.
        Specified by:
        getFileBlockLocations in interface IFileSystem
        Specified by:
        getFileBlockLocations in class FileSystem
        Throws:
        IOException
      • getDefaultBlockSize

        public long getDefaultBlockSize()
        Description copied from class: FileSystem
        Return the number of bytes that large input files should be optimally be split into to minimize I/O time.
        Overrides:
        getDefaultBlockSize in class FileSystem
        Returns:
        the number of bytes that large input files should be optimally be split into to minimize I/O time
      • delete

        public boolean delete​(Path f,
                              boolean recursive)
                       throws IOException
        Description copied from interface: IFileSystem
        Delete a file.
        Specified by:
        delete in interface IFileSystem
        Specified by:
        delete in class FileSystem
        Parameters:
        f - the path to delete
        recursive - if path is a directory and set to true, the directory is deleted else throws an exception. In case of a file the recursive can be set to either true or false
        Returns:
        true if delete is successful, false otherwise
        Throws:
        IOException
      • mkdirs

        public boolean mkdirs​(Path f)
                       throws IOException
        Description copied from interface: IFileSystem
        Make the given file and all non-existent parents into directories. Has the semantics of Unix 'mkdir -p'. Existence of the directory hierarchy is not an error.
        Specified by:
        mkdirs in interface IFileSystem
        Specified by:
        mkdirs in class FileSystem
        Parameters:
        f - the directory/directories to be created
        Returns:
        true if at least one new directory has been created, false otherwise
        Throws:
        IOException - thrown if an I/O error occurs while creating the directory
      • create

        public FSDataOutputStream create​(Path f,
                                         boolean overwrite,
                                         int bufferSize,
                                         short replication,
                                         long blockSize)
                                  throws IOException
        Description copied from class: FileSystem
        Opens an FSDataOutputStream at the indicated Path.

        This method is deprecated, because most of its parameters are ignored by most file systems. To control for example the replication factor and block size in the Hadoop Distributed File system, make sure that the respective Hadoop configuration file is either linked from the Flink configuration, or in the classpath of either Flink or the user code.

        Overrides:
        create in class FileSystem
        Parameters:
        f - the file name to open
        overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.
        bufferSize - the size of the buffer to be used.
        replication - required block replication for the file.
        blockSize - the size of the file blocks
        Throws:
        IOException - Thrown, if the stream could not be opened because of an I/O, or because a file already exists at that path and the write mode indicates to not overwrite the file.
      • create

        public FSDataOutputStream create​(Path f,
                                         FileSystem.WriteMode overwrite)
                                  throws IOException
        Description copied from interface: IFileSystem
        Opens an FSDataOutputStream to a new file at the given path.

        If the file already exists, the behavior depends on the given WriteMode. If the mode is set to FileSystem.WriteMode.NO_OVERWRITE, then this method fails with an exception.

        Specified by:
        create in interface IFileSystem
        Specified by:
        create in class FileSystem
        Parameters:
        f - The file path to write to
        overwrite - The action to take if a file or directory already exists at the given path.
        Returns:
        The stream to the new file at the target path.
        Throws:
        IOException - Thrown, if the stream could not be opened because of an I/O, or because a file already exists at that path and the write mode indicates to not overwrite the file.
      • rename

        public boolean rename​(Path src,
                              Path dst)
                       throws IOException
        Description copied from interface: IFileSystem
        Renames the file/directory src to dst.
        Specified by:
        rename in interface IFileSystem
        Specified by:
        rename in class FileSystem
        Parameters:
        src - the file/directory to rename
        dst - the new name of the file/directory
        Returns:
        true if the renaming was successful, false otherwise
        Throws:
        IOException
      • initOutPathLocalFS

        public boolean initOutPathLocalFS​(Path outPath,
                                          FileSystem.WriteMode writeMode,
                                          boolean createDirectory)
                                   throws IOException
        Description copied from interface: IFileSystem
        Initializes output directories on local file systems according to the given write mode.
        • WriteMode.NO_OVERWRITE & parallel output:
          • A directory is created if the output path does not exist.
          • An existing directory is reused, files contained in the directory are NOT deleted.
          • An existing file raises an exception.
        • WriteMode.NO_OVERWRITE & NONE parallel output:
          • An existing file or directory raises an exception.
        • WriteMode.OVERWRITE & parallel output:
          • A directory is created if the output path does not exist.
          • An existing directory is reused, files contained in the directory are NOT deleted.
          • An existing file is deleted and replaced by a new directory.
        • WriteMode.OVERWRITE & NONE parallel output:
          • An existing file or directory (and all its content) is deleted

        Files contained in an existing directory are not deleted, because multiple instances of a DataSinkTask might call this function at the same time and hence might perform concurrent delete operations on the file system (possibly deleting output files of concurrently running tasks). Since concurrent DataSinkTasks are not aware of each other, coordination of delete and create operations would be difficult.

        Specified by:
        initOutPathLocalFS in interface IFileSystem
        Overrides:
        initOutPathLocalFS in class FileSystem
        Parameters:
        outPath - Output path that should be prepared.
        writeMode - Write mode to consider.
        createDirectory - True, to initialize a directory at the given path, false to prepare space for a file.
        Returns:
        True, if the path was successfully prepared, false otherwise.
        Throws:
        IOException - Thrown, if any of the file system access operations failed.
      • initOutPathDistFS

        public boolean initOutPathDistFS​(Path outPath,
                                         FileSystem.WriteMode writeMode,
                                         boolean createDirectory)
                                  throws IOException
        Description copied from interface: IFileSystem
        Initializes output directories on distributed file systems according to the given write mode.

        WriteMode.NO_OVERWRITE & parallel output: - A directory is created if the output path does not exist. - An existing file or directory raises an exception.

        WriteMode.NO_OVERWRITE & NONE parallel output: - An existing file or directory raises an exception.

        WriteMode.OVERWRITE & parallel output: - A directory is created if the output path does not exist. - An existing directory and its content is deleted and a new directory is created. - An existing file is deleted and replaced by a new directory.

        WriteMode.OVERWRITE & NONE parallel output: - An existing file or directory is deleted and replaced by a new directory.

        Specified by:
        initOutPathDistFS in interface IFileSystem
        Overrides:
        initOutPathDistFS in class FileSystem
        Parameters:
        outPath - Output path that should be prepared.
        writeMode - Write mode to consider.
        createDirectory - True, to initialize a directory at the given path, false otherwise.
        Returns:
        True, if the path was successfully prepared, false otherwise.
        Throws:
        IOException - Thrown, if any of the file system access operations failed.
      • isDistributedFS

        public boolean isDistributedFS()
        Description copied from interface: IFileSystem
        Returns true if this is a distributed file system. A distributed file system here means that the file system is shared among all Flink processes that participate in a cluster or job and that all these processes can see the same files.
        Specified by:
        isDistributedFS in interface IFileSystem
        Specified by:
        isDistributedFS in class FileSystem
        Returns:
        True, if this is a distributed file system, false otherwise.