public static class HadoopPathBasedPartFileWriter.HadoopPathBasedBucketWriter<IN,BucketID> extends Object implements BucketWriter<IN,BucketID>
HadoopPathBasedPartFileWriter
. This writer does not support
snapshotting the in-progress files. For pending files, it stores the target path and the
staging file path into the state.BucketWriter.PendingFile
Constructor and Description |
---|
HadoopPathBasedBucketWriter(org.apache.hadoop.conf.Configuration configuration,
HadoopPathBasedBulkWriter.Factory<IN> bulkWriterFactory,
HadoopFileCommitterFactory fileCommitterFactory) |
Modifier and Type | Method and Description |
---|---|
boolean |
cleanupInProgressFileRecoverable(InProgressFileWriter.InProgressFileRecoverable inProgressFileRecoverable)
Frees up any resources that were previously occupied in order to be able to recover from a
(potential) failure.
|
WriterProperties |
getProperties() |
HadoopPathBasedPartFileWriter<IN,BucketID> |
openNewInProgressFile(BucketID bucketID,
Path flinkPath,
long creationTime)
Used to create a new
InProgressFileWriter . |
BucketWriter.PendingFile |
recoverPendingFile(InProgressFileWriter.PendingFileRecoverable pendingFileRecoverable)
Recovers a pending file for finalizing and committing.
|
InProgressFileWriter<IN,BucketID> |
resumeInProgressFileFrom(BucketID bucketID,
InProgressFileWriter.InProgressFileRecoverable inProgressFileSnapshot,
long creationTime)
Used to resume a
InProgressFileWriter from a InProgressFileWriter.InProgressFileRecoverable . |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
openNewCompactingFile
public HadoopPathBasedBucketWriter(org.apache.hadoop.conf.Configuration configuration, HadoopPathBasedBulkWriter.Factory<IN> bulkWriterFactory, HadoopFileCommitterFactory fileCommitterFactory)
public HadoopPathBasedPartFileWriter<IN,BucketID> openNewInProgressFile(BucketID bucketID, Path flinkPath, long creationTime) throws IOException
BucketWriter
InProgressFileWriter
.openNewInProgressFile
in interface BucketWriter<IN,BucketID>
bucketID
- the id of the bucket this writer is writing to.flinkPath
- the path this writer will write to.creationTime
- the creation time of the file.InProgressFileWriter
IOException
- Thrown if creating a writer fails.public BucketWriter.PendingFile recoverPendingFile(InProgressFileWriter.PendingFileRecoverable pendingFileRecoverable) throws IOException
BucketWriter
recoverPendingFile
in interface BucketWriter<IN,BucketID>
pendingFileRecoverable
- The handle with the recovery information.IOException
- Thrown if recovering a pending file fails.public WriterProperties getProperties()
getProperties
in interface BucketWriter<IN,BucketID>
BucketWriter
public InProgressFileWriter<IN,BucketID> resumeInProgressFileFrom(BucketID bucketID, InProgressFileWriter.InProgressFileRecoverable inProgressFileSnapshot, long creationTime)
BucketWriter
InProgressFileWriter
from a InProgressFileWriter.InProgressFileRecoverable
.resumeInProgressFileFrom
in interface BucketWriter<IN,BucketID>
bucketID
- the id of the bucket this writer is writing to.inProgressFileSnapshot
- the state of the part file.creationTime
- the creation time of the file.InProgressFileWriter
public boolean cleanupInProgressFileRecoverable(InProgressFileWriter.InProgressFileRecoverable inProgressFileRecoverable)
BucketWriter
NOTE: This operation should not throw an exception, but return false if the cleanup did not happen for any reason.
cleanupInProgressFileRecoverable
in interface BucketWriter<IN,BucketID>
inProgressFileRecoverable
- the InProgressFileWriter.InProgressFileRecoverable
whose state we want to clean-up.true
if the resources were successfully freed, false
otherwise (e.g.
the file to be deleted was not there for any reason - already deleted or never created).Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.