org.apache.flink.streaming.api.functions.sink.filesystem

Interface BucketWriter<IN,BucketID>

All Known Implementing Classes:

BulkBucketWriter, HadoopPathBasedPartFileWriter.HadoopPathBasedBucketWriter, RowWiseBucketWriter

@Internal
public interface BucketWriter<IN,BucketID>

An interface for factories that create the different writers.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static interface	BucketWriter.PendingFile This represents the file that can not write any data to.

Method Summary

Modifier and Type	Method and Description
boolean	cleanupInProgressFileRecoverable(InProgressFileWriter.InProgressFileRecoverable inProgressFileRecoverable) Frees up any resources that were previously occupied in order to be able to recover from a (potential) failure.
WriterProperties	getProperties()
default CompactingFileWriter	openNewCompactingFile(CompactingFileWriter.Type type, BucketID bucketID, Path path, long creationTime) Used to create a new CompactingFileWriter of the requesting type.
InProgressFileWriter<IN,BucketID>	openNewInProgressFile(BucketID bucketID, Path path, long creationTime) Used to create a new InProgressFileWriter.
BucketWriter.PendingFile	recoverPendingFile(InProgressFileWriter.PendingFileRecoverable pendingFileRecoverable) Recovers a pending file for finalizing and committing.
InProgressFileWriter<IN,BucketID>	resumeInProgressFileFrom(BucketID bucketID, InProgressFileWriter.InProgressFileRecoverable inProgressFileSnapshot, long creationTime) Used to resume a InProgressFileWriter from a InProgressFileWriter.InProgressFileRecoverable.

Method Detail

openNewInProgressFile

InProgressFileWriter<IN,BucketID> openNewInProgressFile(BucketID bucketID,
                                                        Path path,
                                                        long creationTime)
                                                 throws IOException

Used to create a new InProgressFileWriter.

Parameters:

bucketID - the id of the bucket this writer is writing to.

path - the path this writer will write to.

creationTime - the creation time of the file.

Returns:

the new InProgressFileWriter

Throws:

IOException - Thrown if creating a writer fails.

openNewCompactingFile

default CompactingFileWriter openNewCompactingFile(CompactingFileWriter.Type type,
                                                   BucketID bucketID,
                                                   Path path,
                                                   long creationTime)
                                            throws IOException

Used to create a new CompactingFileWriter of the requesting type. Requesting a writer of an unsupported type will result in UnsupportedOperationException. By default, only RECORD_WISE type is supported, for which a InProgressFileWriter will be created.

Parameters:

type - the type of this writer.

bucketID - the id of the bucket this writer is writing to.

path - the path this writer will write to.

creationTime - the creation time of the file.

Returns:

the new InProgressFileWriter

Throws:

IOException - Thrown if creating a writer fails.

UnsupportedOperationException - Thrown if the bucket writer doesn't support the requesting type.

resumeInProgressFileFrom

InProgressFileWriter<IN,BucketID> resumeInProgressFileFrom(BucketID bucketID,
                                                           InProgressFileWriter.InProgressFileRecoverable inProgressFileSnapshot,
                                                           long creationTime)
                                                    throws IOException

Used to resume a InProgressFileWriter from a InProgressFileWriter.InProgressFileRecoverable.

Parameters:

bucketID - the id of the bucket this writer is writing to.

inProgressFileSnapshot - the state of the part file.

creationTime - the creation time of the file.

Returns:

the resumed InProgressFileWriter

Throws:

IOException - Thrown if resuming a writer fails.

getProperties

WriterProperties getProperties()

Returns:

the property of the BucketWriter

recoverPendingFile

BucketWriter.PendingFile recoverPendingFile(InProgressFileWriter.PendingFileRecoverable pendingFileRecoverable)
                                     throws IOException

Recovers a pending file for finalizing and committing.

Parameters:

pendingFileRecoverable - The handle with the recovery information.

Returns:

A pending file

Throws:

IOException - Thrown if recovering a pending file fails.

cleanupInProgressFileRecoverable

boolean cleanupInProgressFileRecoverable(InProgressFileWriter.InProgressFileRecoverable inProgressFileRecoverable)
                                  throws IOException

Frees up any resources that were previously occupied in order to be able to recover from a (potential) failure.

NOTE: This operation should not throw an exception, but return false if the cleanup did not happen for any reason.

Parameters:

inProgressFileRecoverable - the InProgressFileWriter.InProgressFileRecoverable whose state we want to clean-up.

Returns:

true if the resources were successfully freed, false otherwise (e.g. the file to be deleted was not there for any reason - already deleted or never created).

Throws:

IOException - if an I/O error occurs