org.apache.flink.fs.s3hadoop

Class HadoopS3AccessHelper

java.lang.Object

org.apache.flink.fs.s3hadoop.HadoopS3AccessHelper

All Implemented Interfaces:

S3AccessHelper

public class HadoopS3AccessHelper
extends Object
implements S3AccessHelper

An implementation of the S3AccessHelper for the Hadoop S3A filesystem.

Constructor Summary

Constructors

Constructor and Description
HadoopS3AccessHelper(org.apache.hadoop.fs.s3a.S3AFileSystem s3a, Configuration conf)

Method Summary

Modifier and Type	Method and Description
com.amazonaws.services.s3.model.CompleteMultipartUploadResult	commitMultiPartUpload(String destKey, String uploadId, List<com.amazonaws.services.s3.model.PartETag> partETags, long length, AtomicInteger errorCount) Finalizes a Multi-Part Upload.
boolean	deleteObject(String key) Deletes the object associated with the provided key.
long	getObject(String key, File targetLocation) Gets the object associated with the provided key from S3 and puts it in the provided targetLocation.
com.amazonaws.services.s3.model.ObjectMetadata	getObjectMetadata(String key) Fetches the metadata associated with a given key on S3.
com.amazonaws.services.s3.model.PutObjectResult	putObject(String key, File inputFile) Uploads an object to S3.
String	startMultiPartUpload(String key) Initializes a Multi-Part Upload.
com.amazonaws.services.s3.model.UploadPartResult	uploadPart(String key, String uploadId, int partNumber, File inputFile, long length) Uploads a part and associates it with the MPU with the provided uploadId.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopS3AccessHelper

public HadoopS3AccessHelper(org.apache.hadoop.fs.s3a.S3AFileSystem s3a,
                            Configuration conf)

Method Detail

startMultiPartUpload

public String startMultiPartUpload(String key)
                            throws IOException

Description copied from interface: S3AccessHelper

Initializes a Multi-Part Upload.

Specified by:

startMultiPartUpload in interface S3AccessHelper

Parameters:

key - the key whose value we want to upload in parts.

Returns:

The id of the initiated Multi-Part Upload which will be used during the uploading of the parts.

Throws:

IOException

uploadPart

public com.amazonaws.services.s3.model.UploadPartResult uploadPart(String key,
                                                                   String uploadId,
                                                                   int partNumber,
                                                                   File inputFile,
                                                                   long length)
                                                            throws IOException

Description copied from interface: S3AccessHelper

Uploads a part and associates it with the MPU with the provided uploadId.

Specified by:

uploadPart in interface S3AccessHelper

Parameters:

key - the key this MPU is associated with.

uploadId - the id of the MPU.

partNumber - the number of the part being uploaded (has to be in [1 ... 10000]).

inputFile - the (local) file holding the part to be uploaded.

length - the length of the part.

Returns:

The result of the attempt to upload the part.

Throws:

IOException

putObject

public com.amazonaws.services.s3.model.PutObjectResult putObject(String key,
                                                                 File inputFile)
                                                          throws IOException

Description copied from interface: S3AccessHelper

Uploads an object to S3. Contrary to the S3AccessHelper.uploadPart(String, String, int, File, long) method, this object is not going to be associated to any MPU and, as such, it is not subject to the garbage collection policies specified for your S3 bucket.

Specified by:

putObject in interface S3AccessHelper

Parameters:

key - the key used to identify this part.

inputFile - the (local) file holding the data to be uploaded.

Returns:

The result of the attempt to stage the incomplete part.

Throws:

IOException

commitMultiPartUpload

public com.amazonaws.services.s3.model.CompleteMultipartUploadResult commitMultiPartUpload(String destKey,
                                                                                           String uploadId,
                                                                                           List<com.amazonaws.services.s3.model.PartETag> partETags,
                                                                                           long length,
                                                                                           AtomicInteger errorCount)
                                                                                    throws IOException

Description copied from interface: S3AccessHelper

Finalizes a Multi-Part Upload.

Specified by:

commitMultiPartUpload in interface S3AccessHelper

Parameters:

destKey - the key identifying the object we finished uploading.

uploadId - the id of the MPU.

partETags - the list of ETags associated with this MPU.

length - the size of the uploaded object.

errorCount - a counter that will be used to count any failed attempts to commit the MPU.

Returns:

The result of the attempt to finalize the MPU.

Throws:

IOException

deleteObject

public boolean deleteObject(String key)
                     throws IOException

Description copied from interface: S3AccessHelper

Deletes the object associated with the provided key.

Specified by:

deleteObject in interface S3AccessHelper

Parameters:

key - The key to be deleted.

Returns:

true if the resources were successfully freed, false otherwise (e.g. the file to be deleted was not there).

Throws:

IOException

getObject

public long getObject(String key,
                      File targetLocation)
               throws IOException

Description copied from interface: S3AccessHelper

Gets the object associated with the provided key from S3 and puts it in the provided targetLocation.

Specified by:

getObject in interface S3AccessHelper

Parameters:

key - the key of the object to fetch.

targetLocation - the file to read the object to.

Returns:

The number of bytes read.

Throws:

IOException

getObjectMetadata

public com.amazonaws.services.s3.model.ObjectMetadata getObjectMetadata(String key)
                                                                 throws IOException

Description copied from interface: S3AccessHelper

Fetches the metadata associated with a given key on S3.

Specified by:

getObjectMetadata in interface S3AccessHelper

Parameters:

key - the key.

Returns:

The associated ObjectMetadata.

Throws:

IOException