Interface SupportsPreCommitTopology<WriterResultT,​CommittableT>

  • All Known Implementing Classes:
    FileSink

    @Experimental
    public interface SupportsPreCommitTopology<WriterResultT,​CommittableT>
    Allows expert users to implement a custom topology after SinkWriter and before Committer.

    It is recommended to use immutable committables because mutating committables can have unexpected side-effects.

    It's important that all CommittableMessages are modified appropriately, such that all messages with the same subtask id will also be processed by the same Committer subtask and the CommittableSummary matches the respective count. If committables are combined or split in any way, the summary needs to be adjusted.

    There is also no requirement to keep the subtask ids of the writer, they can be changed as long as there are no two summaries with the same subtask ids (and corresponding CommittableWithLineage). Subtask ids don't need to be consecutive or small. The global committer will use CommittableSummary.getNumberOfSubtasks() to determine if all committables have been received, so that number needs to correctly reflect the number of distinct subtask ids. The easiest way to guarantee all of this is to use RuntimeContext.getTaskInfo().