org.apache.flink.connector.file.sink.writer

Class FileWriter<IN>

java.lang.Object

org.apache.flink.connector.file.sink.writer.FileWriter<IN>

Type Parameters:

IN - The type of input elements.

All Implemented Interfaces:

AutoCloseable, ProcessingTimeService.ProcessingTimeCallback, SinkWriter<IN>, StatefulSink.StatefulSinkWriter<IN,FileWriterBucketState>, TwoPhaseCommittingSink.PrecommittingSinkWriter<IN,FileSinkCommittable>

@Internal
public class FileWriter<IN>
extends Object
implements StatefulSink.StatefulSinkWriter<IN,FileWriterBucketState>, TwoPhaseCommittingSink.PrecommittingSinkWriter<IN,FileSinkCommittable>, SinkWriter<IN>, ProcessingTimeService.ProcessingTimeCallback

A SinkWriter implementation for FileSink.

It writes data to and manages the different active buckes in the FileSink.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.api.connector.sink2.SinkWriter

SinkWriter.Context

Constructor Summary

Constructors

Constructor and Description
FileWriter(Path basePath, SinkWriterMetricGroup metricGroup, BucketAssigner<IN,String> bucketAssigner, FileWriterBucketFactory<IN> bucketFactory, BucketWriter<IN,String> bucketWriter, RollingPolicy<IN,String> rollingPolicy, OutputFileConfig outputFileConfig, ProcessingTimeService processingTimeService, long bucketCheckInterval) A constructor creating a new empty bucket manager.

Method Summary

Modifier and Type	Method and Description
void	close()
void	flush(boolean endOfInput) Called on checkpoint or end of input so that the writer to flush all pending data for at-least-once.
void	initializeState(Collection<FileWriterBucketState> bucketStates) Initializes the state after recovery from a failure.
void	onProcessingTime(long time) This method is invoked with the time which the callback register for.
Collection<FileSinkCommittable>	prepareCommit() Prepares for a commit.
List<FileWriterBucketState>	snapshotState(long checkpointId)
void	write(IN element, SinkWriter.Context context) Adds an element to the writer.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.connector.sink2.SinkWriter

writeWatermark

Constructor Detail

FileWriter

public FileWriter(Path basePath,
                  SinkWriterMetricGroup metricGroup,
                  BucketAssigner<IN,String> bucketAssigner,
                  FileWriterBucketFactory<IN> bucketFactory,
                  BucketWriter<IN,String> bucketWriter,
                  RollingPolicy<IN,String> rollingPolicy,
                  OutputFileConfig outputFileConfig,
                  ProcessingTimeService processingTimeService,
                  long bucketCheckInterval)

A constructor creating a new empty bucket manager.

Parameters:

basePath - The base path for our buckets.

metricGroup - SinkWriterMetricGroup to set sink writer specific metrics.

bucketAssigner - The BucketAssigner provided by the user.

bucketFactory - The FileWriterBucketFactory to be used to create buckets.

bucketWriter - The BucketWriter to be used when writing data.

rollingPolicy - The RollingPolicy as specified by the user.

Method Detail

initializeState

public void initializeState(Collection<FileWriterBucketState> bucketStates)
                     throws IOException

Initializes the state after recovery from a failure.

During this process:

1. we set the initial value for part counter to the maximum value used before across all tasks and buckets. This guarantees that we do not overwrite valid data,
2. we commit any pending files for previous checkpoints (previous to the last successful one from which we restore),
3. we resume writing to the previous in-progress file of each bucket, and
4. if we receive multiple states for the same bucket, we merge them.

Parameters:

bucketStates - the state holding recovered state about active buckets.

Throws:

IOException - if anything goes wrong during retrieving the state or restoring/committing of any in-progress/pending part files

write

public void write(IN element,
                  SinkWriter.Context context)
           throws IOException,
                  InterruptedException

Description copied from interface: SinkWriter

Adds an element to the writer.

Specified by:

write in interface SinkWriter<IN>

Parameters:

element - The input record

context - The additional information about the input record

Throws:

IOException - if fail to add an element.

InterruptedException

flush

public void flush(boolean endOfInput)
           throws IOException,
                  InterruptedException

Description copied from interface: SinkWriter

Called on checkpoint or end of input so that the writer to flush all pending data for at-least-once.

Specified by:

flush in interface SinkWriter<IN>

Throws:

IOException

InterruptedException

prepareCommit

public Collection<FileSinkCommittable> prepareCommit()
                                              throws IOException

Description copied from interface: TwoPhaseCommittingSink.PrecommittingSinkWriter

Prepares for a commit.

This method will be called after SinkWriter.flush(boolean) and before StatefulSink.StatefulSinkWriter.snapshotState(long).

Specified by:

prepareCommit in interface TwoPhaseCommittingSink.PrecommittingSinkWriter<IN,FileSinkCommittable>

Returns:

The data to commit as the second step of the two-phase commit protocol.

Throws:

IOException - if fail to prepare for a commit.

snapshotState

public List<FileWriterBucketState> snapshotState(long checkpointId)
                                          throws IOException

Specified by:

snapshotState in interface StatefulSink.StatefulSinkWriter<IN,FileWriterBucketState>

Returns:

The writer's state.

Throws:

IOException - if fail to snapshot writer's state.

close

public void close()

Specified by:

close in interface AutoCloseable

onProcessingTime

public void onProcessingTime(long time)
                      throws IOException

Description copied from interface: ProcessingTimeService.ProcessingTimeCallback

This method is invoked with the time which the callback register for.

Specified by:

onProcessingTime in interface ProcessingTimeService.ProcessingTimeCallback

Parameters:

time - The time this callback was registered for.

Throws:

IOException