Interface SupportsPreCommitTopology<WriterResultT,CommittableT>
-
- All Known Implementing Classes:
FileSink
@Experimental public interface SupportsPreCommitTopology<WriterResultT,CommittableT>
Allows expert users to implement a custom topology afterSinkWriter
and beforeCommitter
.It is recommended to use immutable committables because mutating committables can have unexpected side-effects.
It's important that all
CommittableMessage
s are modified appropriately, such that all messages with the same subtask id will also be processed by the sameCommitter
subtask and theCommittableSummary
matches the respective count. If committables are combined or split in any way, the summary needs to be adjusted.There is also no requirement to keep the subtask ids of the writer, they can be changed as long as there are no two summaries with the same subtask ids (and corresponding
CommittableWithLineage
). Subtask ids don't need to be consecutive or small. The global committer will useCommittableSummary.getNumberOfSubtasks()
to determine if all committables have been received, so that number needs to correctly reflect the number of distinct subtask ids. The easiest way to guarantee all of this is to useRuntimeContext.getTaskInfo()
.
-
-
Method Summary
All Methods Instance Methods Abstract Methods Modifier and Type Method Description DataStream<CommittableMessage<CommittableT>>
addPreCommitTopology(DataStream<CommittableMessage<WriterResultT>> committables)
Intercepts and modifies the committables sent on checkpoint or at end of input.SimpleVersionedSerializer<WriterResultT>
getWriteResultSerializer()
Returns the serializer of the WriteResult type.
-
-
-
Method Detail
-
addPreCommitTopology
DataStream<CommittableMessage<CommittableT>> addPreCommitTopology(DataStream<CommittableMessage<WriterResultT>> committables)
Intercepts and modifies the committables sent on checkpoint or at end of input. Implementers need to ensure to modify allCommittableMessage
s appropriately.- Parameters:
committables
- the stream of committables.- Returns:
- the custom topology before
Committer
.
-
getWriteResultSerializer
SimpleVersionedSerializer<WriterResultT> getWriteResultSerializer()
Returns the serializer of the WriteResult type.
-
-