org.apache.flink.fs.s3.common.writer

Interface S3AccessHelper

All Known Implementing Classes:

HadoopS3AccessHelper

@Internal
public interface S3AccessHelper

An interface that abstracts away the Multi-Part Upload (MPU) functionality offered by S3, from the specific implementation of the file system. This is needed so that we can accommodate both Hadoop S3 and Presto.

Multipart uploads are convenient for large object. These will be uploaded in multiple parts and the mutli-part upload is the equivalent of a transaction, where the upload with all its parts will be either committed or discarded.

Method Summary

Modifier and Type	Method and Description
com.amazonaws.services.s3.model.CompleteMultipartUploadResult	commitMultiPartUpload(String key, String uploadId, List<com.amazonaws.services.s3.model.PartETag> partETags, long length, AtomicInteger errorCount) Finalizes a Multi-Part Upload.
boolean	deleteObject(String key) Deletes the object associated with the provided key.
long	getObject(String key, File targetLocation) Gets the object associated with the provided key from S3 and puts it in the provided targetLocation.
com.amazonaws.services.s3.model.ObjectMetadata	getObjectMetadata(String key) Fetches the metadata associated with a given key on S3.
com.amazonaws.services.s3.model.PutObjectResult	putObject(String key, File inputFile) Uploads an object to S3.
String	startMultiPartUpload(String key) Initializes a Multi-Part Upload.
com.amazonaws.services.s3.model.UploadPartResult	uploadPart(String key, String uploadId, int partNumber, File inputFile, long length) Uploads a part and associates it with the MPU with the provided uploadId.

Method Detail

startMultiPartUpload

String startMultiPartUpload(String key)
                     throws IOException

Initializes a Multi-Part Upload.

Parameters:

key - the key whose value we want to upload in parts.

Returns:

The id of the initiated Multi-Part Upload which will be used during the uploading of the parts.

Throws:

IOException

uploadPart

com.amazonaws.services.s3.model.UploadPartResult uploadPart(String key,
                                                            String uploadId,
                                                            int partNumber,
                                                            File inputFile,
                                                            long length)
                                                     throws IOException

Uploads a part and associates it with the MPU with the provided uploadId.

Parameters:

key - the key this MPU is associated with.

uploadId - the id of the MPU.

partNumber - the number of the part being uploaded (has to be in [1 ... 10000]).

inputFile - the (local) file holding the part to be uploaded.

length - the length of the part.

Returns:

The result of the attempt to upload the part.

Throws:

IOException

putObject

com.amazonaws.services.s3.model.PutObjectResult putObject(String key,
                                                          File inputFile)
                                                   throws IOException

Uploads an object to S3. Contrary to the uploadPart(String, String, int, File, long) method, this object is not going to be associated to any MPU and, as such, it is not subject to the garbage collection policies specified for your S3 bucket.

Parameters:

key - the key used to identify this part.

inputFile - the (local) file holding the data to be uploaded.

Returns:

The result of the attempt to stage the incomplete part.

Throws:

IOException

commitMultiPartUpload

com.amazonaws.services.s3.model.CompleteMultipartUploadResult commitMultiPartUpload(String key,
                                                                                    String uploadId,
                                                                                    List<com.amazonaws.services.s3.model.PartETag> partETags,
                                                                                    long length,
                                                                                    AtomicInteger errorCount)
                                                                             throws IOException

Finalizes a Multi-Part Upload.

Parameters:

key - the key identifying the object we finished uploading.

uploadId - the id of the MPU.

partETags - the list of ETags associated with this MPU.

length - the size of the uploaded object.

errorCount - a counter that will be used to count any failed attempts to commit the MPU.

Returns:

The result of the attempt to finalize the MPU.

Throws:

IOException

deleteObject

boolean deleteObject(String key)
              throws IOException

Deletes the object associated with the provided key.

Parameters:

key - The key to be deleted.

Returns:

true if the resources were successfully freed, false otherwise (e.g. the file to be deleted was not there).

Throws:

IOException

getObject

long getObject(String key,
               File targetLocation)
        throws IOException

Gets the object associated with the provided key from S3 and puts it in the provided targetLocation.

Parameters:

key - the key of the object to fetch.

targetLocation - the file to read the object to.

Returns:

The number of bytes read.

Throws:

IOException

getObjectMetadata

com.amazonaws.services.s3.model.ObjectMetadata getObjectMetadata(String key)
                                                          throws IOException

Fetches the metadata associated with a given key on S3.

Parameters:

key - the key.

Returns:

The associated ObjectMetadata.

Throws:

IOException