IN
- The type of input elements.@Internal public class FileWriter<IN> extends Object implements SinkWriter<IN,FileSinkCommittable,FileWriterBucketState>, Sink.ProcessingTimeService.ProcessingTimeCallback
SinkWriter
implementation for FileSink
.
It writes data to and manages the different active buckes
in the
FileSink
.
SinkWriter.Context
Constructor and Description |
---|
FileWriter(Path basePath,
SinkWriterMetricGroup metricGroup,
BucketAssigner<IN,String> bucketAssigner,
FileWriterBucketFactory<IN> bucketFactory,
BucketWriter<IN,String> bucketWriter,
RollingPolicy<IN,String> rollingPolicy,
OutputFileConfig outputFileConfig,
Sink.ProcessingTimeService processingTimeService,
long bucketCheckInterval)
A constructor creating a new empty bucket manager.
|
Modifier and Type | Method and Description |
---|---|
void |
close() |
void |
initializeState(List<FileWriterBucketState> bucketStates)
Initializes the state after recovery from a failure.
|
void |
onProcessingTime(long time)
This method is invoked with the time which the callback register for.
|
List<FileSinkCommittable> |
prepareCommit(boolean flush)
Prepare for a commit.
|
List<FileWriterBucketState> |
snapshotState(long checkpointId) |
void |
write(IN element,
SinkWriter.Context context)
Add an element to the writer.
|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
snapshotState, writeWatermark
public FileWriter(Path basePath, SinkWriterMetricGroup metricGroup, BucketAssigner<IN,String> bucketAssigner, FileWriterBucketFactory<IN> bucketFactory, BucketWriter<IN,String> bucketWriter, RollingPolicy<IN,String> rollingPolicy, OutputFileConfig outputFileConfig, Sink.ProcessingTimeService processingTimeService, long bucketCheckInterval)
basePath
- The base path for our buckets.metricGroup
- SinkWriterMetricGroup
to set sink writer specific metrics.bucketAssigner
- The BucketAssigner
provided by the user.bucketFactory
- The FileWriterBucketFactory
to be used to create buckets.bucketWriter
- The BucketWriter
to be used when writing data.rollingPolicy
- The RollingPolicy
as specified by the user.public void initializeState(List<FileWriterBucketState> bucketStates) throws IOException
During this process:
bucketStates
- the state holding recovered state about active buckets.IOException
- if anything goes wrong during retrieving the state or
restoring/committing of any in-progress/pending part filespublic void write(IN element, SinkWriter.Context context) throws IOException
SinkWriter
write
in interface SinkWriter<IN,FileSinkCommittable,FileWriterBucketState>
element
- The input recordcontext
- The additional information about the input recordIOException
- if fail to add an element.public List<FileSinkCommittable> prepareCommit(boolean flush) throws IOException
SinkWriter
This will be called before we checkpoint the Writer's state in Streaming execution mode.
In case the sink has no explicit committer, this method is still called to allow the writer to implement a 1-phase commit protocol.
prepareCommit
in interface SinkWriter<IN,FileSinkCommittable,FileWriterBucketState>
flush
- Whether flushing the un-staged data or notIOException
- if fail to prepare for a commit.public List<FileWriterBucketState> snapshotState(long checkpointId) throws IOException
snapshotState
in interface SinkWriter<IN,FileSinkCommittable,FileWriterBucketState>
IOException
- if fail to snapshot writer's state.public void close()
close
in interface AutoCloseable
public void onProcessingTime(long time) throws IOException
Sink.ProcessingTimeService.ProcessingTimeCallback
onProcessingTime
in interface Sink.ProcessingTimeService.ProcessingTimeCallback
time
- The time this callback was registered for.IOException
Copyright © 2014–2023 The Apache Software Foundation. All rights reserved.