Package org.apache.flink.fs.s3.common.writer

Interface S3AccessHelper

  • All Known Implementing Classes:
    HadoopS3AccessHelper
    @Internal
public interface S3AccessHelper
    An interface that abstracts away the Multi-Part Upload (MPU) functionality offered by S3, from the specific implementation of the file system. This is needed so that we can accommodate both Hadoop S3 and Presto.

    Multipart uploads are convenient for large object. These will be uploaded in multiple parts and the mutli-part upload is the equivalent of a transaction, where the upload with all its parts will be either committed or discarded.

    • Method Summary

      All Methods Instance Methods Abstract Methods 
      Modifier and Type Method Description
      com.amazonaws.services.s3.model.CompleteMultipartUploadResult commitMultiPartUpload​(String key, String uploadId, List<com.amazonaws.services.s3.model.PartETag> partETags, long length, AtomicInteger errorCount)
      Finalizes a Multi-Part Upload.
      boolean deleteObject​(String key)
      Deletes the object associated with the provided key.
      long getObject​(String key, File targetLocation)
      Gets the object associated with the provided key from S3 and puts it in the provided targetLocation.
      com.amazonaws.services.s3.model.ObjectMetadata getObjectMetadata​(String key)
      Fetches the metadata associated with a given key on S3.
      com.amazonaws.services.s3.model.PutObjectResult putObject​(String key, File inputFile)
      Uploads an object to S3.
      String startMultiPartUpload​(String key)
      Initializes a Multi-Part Upload.
      com.amazonaws.services.s3.model.UploadPartResult uploadPart​(String key, String uploadId, int partNumber, File inputFile, long length)
      Uploads a part and associates it with the MPU with the provided uploadId.

    • Method Detail

      • startMultiPartUpload

        String startMultiPartUpload​(String key)
                     throws IOException
        Initializes a Multi-Part Upload.
        Parameters:
        key - the key whose value we want to upload in parts.
        Returns:
        The id of the initiated Multi-Part Upload which will be used during the uploading of the parts.
        Throws:
        IOException

      • uploadPart

        com.amazonaws.services.s3.model.UploadPartResult uploadPart​(String key,
                                                            String uploadId,
                                                            int partNumber,
                                                            File inputFile,
                                                            long length)
                                                     throws IOException
        Uploads a part and associates it with the MPU with the provided uploadId.
        Parameters:
        key - the key this MPU is associated with.
        uploadId - the id of the MPU.
        partNumber - the number of the part being uploaded (has to be in [1 ... 10000]).
        inputFile - the (local) file holding the part to be uploaded.
        length - the length of the part.
        Returns:
        The result of the attempt to upload the part.
        Throws:
        IOException

      • putObject

        com.amazonaws.services.s3.model.PutObjectResult putObject​(String key,
                                                          File inputFile)
                                                   throws IOException
        Uploads an object to S3. Contrary to the uploadPart(String, String, int, File, long) method, this object is not going to be associated to any MPU and, as such, it is not subject to the garbage collection policies specified for your S3 bucket.
        Parameters:
        key - the key used to identify this part.
        inputFile - the (local) file holding the data to be uploaded.
        Returns:
        The result of the attempt to stage the incomplete part.
        Throws:
        IOException

      • commitMultiPartUpload

        com.amazonaws.services.s3.model.CompleteMultipartUploadResult commitMultiPartUpload​(String key,
                                                                                    String uploadId,
                                                                                    List<com.amazonaws.services.s3.model.PartETag> partETags,
                                                                                    long length,
                                                                                    AtomicInteger errorCount)
                                                                             throws IOException
        Finalizes a Multi-Part Upload.
        Parameters:
        key - the key identifying the object we finished uploading.
        uploadId - the id of the MPU.
        partETags - the list of ETags associated with this MPU.
        length - the size of the uploaded object.
        errorCount - a counter that will be used to count any failed attempts to commit the MPU.
        Returns:
        The result of the attempt to finalize the MPU.
        Throws:
        IOException

      • deleteObject

        boolean deleteObject​(String key)
              throws IOException
        Deletes the object associated with the provided key.
        Parameters:
        key - The key to be deleted.
        Returns:
        true if the resources were successfully freed, false otherwise (e.g. the file to be deleted was not there).
        Throws:
        IOException

      • getObject

        long getObject​(String key,
               File targetLocation)
        throws IOException
        Gets the object associated with the provided key from S3 and puts it in the provided targetLocation.
        Parameters:
        key - the key of the object to fetch.
        targetLocation - the file to read the object to.
        Returns:
        The number of bytes read.
        Throws:
        IOException

      • getObjectMetadata

        com.amazonaws.services.s3.model.ObjectMetadata getObjectMetadata​(String key)
                                                          throws IOException
        Fetches the metadata associated with a given key on S3.
        Parameters:
        key - the key.
        Returns:
        The associated ObjectMetadata.
        Throws:
        IOException