org.apache.flink.formats.hadoop.bulk

Class HadoopPathBasedPartFileWriter.HadoopPathBasedBucketWriter<IN,BucketID>

java.lang.Object

org.apache.flink.formats.hadoop.bulk.HadoopPathBasedPartFileWriter.HadoopPathBasedBucketWriter<IN,BucketID>

All Implemented Interfaces:

BucketWriter<IN,BucketID>

Enclosing class:

HadoopPathBasedPartFileWriter<IN,BucketID>

public static class HadoopPathBasedPartFileWriter.HadoopPathBasedBucketWriter<IN,BucketID>
extends Object
implements BucketWriter<IN,BucketID>

Factory to create HadoopPathBasedPartFileWriter. This writer does not support snapshotting the in-progress files. For pending files, it stores the target path and the staging file path into the state.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter

BucketWriter.PendingFile

Constructor Summary

Constructors

Constructor and Description
HadoopPathBasedBucketWriter(Configuration configuration, HadoopPathBasedBulkWriter.Factory<IN> bulkWriterFactory, HadoopFileCommitterFactory fileCommitterFactory)

Method Summary

Modifier and Type	Method and Description
boolean	cleanupInProgressFileRecoverable(InProgressFileWriter.InProgressFileRecoverable inProgressFileRecoverable) Frees up any resources that were previously occupied in order to be able to recover from a (potential) failure.
WriterProperties	getProperties()
HadoopPathBasedPartFileWriter<IN,BucketID>	openNewInProgressFile(BucketID bucketID, Path flinkPath, long creationTime) Used to create a new InProgressFileWriter.
BucketWriter.PendingFile	recoverPendingFile(InProgressFileWriter.PendingFileRecoverable pendingFileRecoverable) Recovers a pending file for finalizing and committing.
InProgressFileWriter<IN,BucketID>	resumeInProgressFileFrom(BucketID bucketID, InProgressFileWriter.InProgressFileRecoverable inProgressFileSnapshot, long creationTime) Used to resume a InProgressFileWriter from a InProgressFileWriter.InProgressFileRecoverable.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopPathBasedBucketWriter

public HadoopPathBasedBucketWriter(Configuration configuration,
                                   HadoopPathBasedBulkWriter.Factory<IN> bulkWriterFactory,
                                   HadoopFileCommitterFactory fileCommitterFactory)

Method Detail

openNewInProgressFile

public HadoopPathBasedPartFileWriter<IN,BucketID> openNewInProgressFile(BucketID bucketID,
                                                                        Path flinkPath,
                                                                        long creationTime)
                                                                 throws IOException

Description copied from interface: BucketWriter

Used to create a new InProgressFileWriter.

Specified by:

openNewInProgressFile in interface BucketWriter<IN,BucketID>

Parameters:

bucketID - the id of the bucket this writer is writing to.

flinkPath - the path this writer will write to.

creationTime - the creation time of the file.

Returns:

the new InProgressFileWriter

Throws:

IOException - Thrown if creating a writer fails.

recoverPendingFile

public BucketWriter.PendingFile recoverPendingFile(InProgressFileWriter.PendingFileRecoverable pendingFileRecoverable)
                                            throws IOException

Description copied from interface: BucketWriter

Recovers a pending file for finalizing and committing.

Specified by:

recoverPendingFile in interface BucketWriter<IN,BucketID>

Parameters:

pendingFileRecoverable - The handle with the recovery information.

Returns:

A pending file

Throws:

IOException - Thrown if recovering a pending file fails.

getProperties

public WriterProperties getProperties()

Specified by:

getProperties in interface BucketWriter<IN,BucketID>

Returns:

the property of the BucketWriter

resumeInProgressFileFrom

public InProgressFileWriter<IN,BucketID> resumeInProgressFileFrom(BucketID bucketID,
                                                                  InProgressFileWriter.InProgressFileRecoverable inProgressFileSnapshot,
                                                                  long creationTime)

Description copied from interface: BucketWriter

Used to resume a InProgressFileWriter from a InProgressFileWriter.InProgressFileRecoverable.

Specified by:

resumeInProgressFileFrom in interface BucketWriter<IN,BucketID>

Parameters:

bucketID - the id of the bucket this writer is writing to.

inProgressFileSnapshot - the state of the part file.

creationTime - the creation time of the file.

Returns:

the resumed InProgressFileWriter

cleanupInProgressFileRecoverable

public boolean cleanupInProgressFileRecoverable(InProgressFileWriter.InProgressFileRecoverable inProgressFileRecoverable)

Description copied from interface: BucketWriter

Frees up any resources that were previously occupied in order to be able to recover from a (potential) failure.

NOTE: This operation should not throw an exception, but return false if the cleanup did not happen for any reason.

Specified by:

cleanupInProgressFileRecoverable in interface BucketWriter<IN,BucketID>

Parameters:

inProgressFileRecoverable - the InProgressFileWriter.InProgressFileRecoverable whose state we want to clean-up.

Returns:

true if the resources were successfully freed, false otherwise (e.g. the file to be deleted was not there for any reason - already deleted or never created).